Recently, products such as Meta's Muse and OpenAI's Dot have been launched in succession, making the Personal Agent (personal intelligence) once again the focus of the larger model. Compared to traditional dialogue assistants, these agents are no longer limited to answering questions, but are able to combine user's personal background, historical memory and application context, actively execute tasks and continue to advance long-term goals. A more dynamic, long-term companion AI assistant is becoming a reality.
However, long-term memory is just the beginning of a Personal Agent. Remembering what a user has done in the past does not mean being able to understand what the user needs right now. In real situations, users often don't clearly state every need. Repeatedly review the steps when preparing meals, encounter difficulties when operating software, or temporarily change plans during tasks. These subtle behaviors and environmental changes can all imply a new need for assistance. How to capture these signals from a constantly changing environment, understand the intentions of the user that have not yet been expressed, and decide when and how to provide help is a key issue that Personal Agent needs to address in the real world.
To address this issue, research teams such as Google have proposed StreamVibe, a proactive video intelligence system focused on dynamic environments. Through continuous video observation and interaction, the study enables the Agent to not only identify user's apparent needs, but also discover the underlying intent in the behavior, and to update its understanding of user goals in a timely manner as the task progresses. The team further constructed an assessment and training system covering explicit, implicit, and dynamic evolutionary needs, exploring how to move Agent from passive waiting instructions to an intelligent partner that can continue to understand users, intervene actively and flexibly adapt to assist.
The article is titled: StreamVibe: Proactive Video Agents for Evolving User Intent
Address of the paper:https://arxiv.org/pdf/2610.34662
The main page of the project:https://stream-vibe.github.io/
User intentions in real-world interactions are constantly changing
The boiling water opens to remind my mom and dad to help me cook this dish, and the requirements for the assistant are not the same. The former has given clear triggering conditions, the latter only describes the goal, the need for the assistant to combine what's going on, decide when to intervene, what help to provide.
The actual task will also continue to generate new information. The user may have completed a step that has just been suggested as a useful reminder; or it may be that the request has been temporarily modified and the original plan has failed. Even if the complete history is preserved, the assistant still needs to distinguish: which requirements are still valid, which steps have been completed, and what the new situation means.
StreamVibe therefore integrates active video interaction into the ever-changing user interface environment. The model must understand the current visual evidence, as well as combine previous interactions and new feedback from users to update judgments about tasks and intentions.
Figure 1: StreamVibe research overview. The following shows the ongoing interactions between explicit, implicit and dynamic intentions, including dependence on explicit requests, insufficient status tracking and difficulty in updating intentions.
This also poses a more specific ability requirement for the Personal Agent: to remember the past while continuously obtaining evidence from new interactions to form an effective understanding of current tasks.
Putting the three categories of user intent into the same set of ratings
The team built the StreamVibe-Bench, covering both real-world activities and digital interface tasks, and dividing it into three levels according to the degree of clarity and change intended.
The explicit intention is to tell me when a user specifies an event or condition, such as a boiler burning, to cover single events, state transitions, and repeated events. The implicit intention only gives an overall goal, such as helping me finish this menu, where the assistant needs to deduce the help time based on the process, covering guidance, learning, monitoring, and tool use.
Dynamic intent further considers changes in requirements in tasks: Revision changes to existing requirements, Composition superimposes new objectives or constraints, Interruption inserts issues that need to be dealt with immediately. Assistants need to update the current status of tasks while continuing to maintain valid needs.
Figure 2: Structure and statistics of the StreamVibe-Bench, including candidate help point marking, artificial verification, three types of intent and scenario distribution, and point-to-point and trajectory evaluation.
The benchmark includes a total of 4,100 assessment samples, 20,546 help points and 183.4 hours of video, with a sample size of 600, 2,000 and 1,500 for the three categories of intentions respectively. Each candidate help point includes a response time window with reference responses that have been independently verified by four reviewers, all of which are retained after approval.
Evaluation simultaneously examines the local judgement of a single moment and the continuous interaction in the entire video stream. The latter allows the model to assume its own historical output to continue running, exposing repeated reminders, leaks, accumulations of errors and intent to track errors.
Let the model learn to respond actively and keep track of tasks
StreamVibe divides the problem into two interdependent levels: when and how to help a trained model; when to run it based on new observations and feedback, continuously updating intentions, task progress and related memories.
Strategy learning uses two stages of training. First, adaptive flow-supervised fine tuning, balancing silence and response decisions based on the training distribution, while learning to respond to content, avoiding models that remain silent or overreact. Subsequently, local trajectory reinforcement training is conducted to construct a short-term causal rollout around help points, combining GRPO to optimize response timing, content quality and consistency with continuous interaction.
Figure 3: Framework of the StreamVibe method. Continuous maintenance of current state and associated memory during reasoning; combination of adaptive flow SFT and local trajectory RL during training.
Training data included SFT samples related to 70K-article tasks and 15K RL seed tasks, collectively known as StreamVibe-85K; SFT also mixed with 768K-article general-stream video samples. Reinforcement learning both assessed the time and content of single responses and constrained omissions, repetitions and inconsistencies in the entire output.
When running, StreamVibe-Harness preserves current active intent, task and environmental progress, and responses already given. Requests can be independently added, modified, or terminated after new information arrives; layered memory retains recent details, compresses earlier history, and retrieves relevant evidence according to current tasks.
After this information is delivered to the model, the model decides whether to open it and the specific content; the interaction results are rewritten and the memory serves as a follow-up decision. Thus, memory and interaction with the current support a continuous understanding of user needs.
The results of the experiment
The team compared StreamVibe with a closed-source model, an open-source offline model and an open-source flow model. The mainframe is at a time matching capacity difference of ±2 seconds, while reporting response time and content quality, and distinguishing between point-by-point evaluation based on standard history and complete trajectory evaluation using its own history output.
Table 1: The complete main experimental results of the StreamVibe-Bench are from Table 2 of the paper. P, R, F1 measures response time, FPR measures false triggers, ANS and C-F1 measures content performance.
In the full trajectory evaluation, Content F1 on the three categories of intentions, 10.83 ̊ 19.13 and 8.91, is higher than the open source baseline listed in the paper; compared to MMDuet2, the scores are 4.95 ̊ 2.62 and 1.44 respectively.
In point-to-point evaluation, the model reached Content F1 on implicit intent at 61.28, and dynamic intent at 39.31. The two protocols reflect the ability under different conditions: point-to-point judgement can be made using standard history, and the complete trajectory needs to take on its own output, so the two types of values cannot be directly mixed.
Capacity also migrates to the external active interaction benchmark: the macro score of EgoProactive is increased from 46.2 to 59.5 at the base, and the Understanding score of IPIBench is increased from 9.8 to 20.4; general video understanding is retained at the same time.
Continued interaction remains a major challenge
The complete results table also suggests that local judgments are correct, which does not mean that the assistant can work with complete reliability. The paper further analyzes the impact of disagreements with task types, wrong triggers, autogeneration history and interaction time on model performance.
Figure 4: Results of active interaction diagnostics, corresponding paper Figure 5. Includes intent and subtype differences, recall and mis-triggered relationships, and the impact of spontaneous history and complete trajectory on performance.
First, there is still a significant improvement in the quality of content for implicit and dynamic needs. First, the assistant needs to identify the needs that the user does not say directly, but also to keep track of the changes brought about by revision, overlapping and disruption. Second, higher calls are often accompanied by more false triggers, indicating that it is still difficult to balance between covering the opportunity to help and avoiding interference.
Self-generated history adds to the problem: previous mistakes may affect subsequent judgments, resulting in a decrease in content quality. Compared to only examining independent moments, the complete trajectory exposes this gap and is closer to the actual requirements of ongoing assistance.
What Strategic Learning and State Tracking Bring
The paper further distinguishes the role of the two types of improvement by comparing training and state management. SFTs make it easier for models to detect helpful opportunities, but may also increase unnecessary responses; RLs further improve response selection, reduce error triggers, and improve response quality.
Figure 5: Interaction length, impact of strategic learning and state management, corresponding paper Figure 6. The difference between the time observed in the left column and the forward and back half of the dynamic intent, the middle column compares SFT to RL, and the right column analyzes changes in Harness.
In addition to Harness when the strategy is fixed, paper analysis shows that unnecessary responses are further reduced and dynamic intent improvements are more apparent. However, different indicators are still balanced, and state management does not mean that all indicators are rising simultaneously.
The average of the six models in the chart also shows that the longer the video, the lower the timing of F1; the more noticeable the decline in the second half of Revision. This shows that when users continue to modify requests, the assistant needs to repeatedly update which intentions are still valid, rather than just keeping the history as it was.
For Personal Agents, these two pathways complement each other: learning strategies to improve judgments of the evidence at hand, and managing status and memory so that subsequent decisions always revolve around current effective needs.
What is the meaning of this verse?
To truly achieve personalization, personal agents need not only to remember the user's past information, but also to constantly understand the user's current needs in actual interaction. The future personal agent should form the continuous interaction closure of the user interface: understanding the needs through user behavior and feedback, judging the progress of tasks through environmental changes, and constantly adjusting the understanding and assistance of user intentions in the interaction process.
StreamVibe provides initial exploration in this direction. Through learning interactive strategies and continuous status management, Agents are able to track the progress of tasks and adjust assistance strategies according to changing user needs. In the future, there are still many questions to be studied about how to extend this capability to longer ranges, more complex real-world scenarios, and make the most of feedback from users during interactions.
© THE