Imagine a home scene like this: when you wear an AI lens in the kitchen to prepare dinner and cut dishes with a knife, it immediately reminds you to pay attention to safety; when you develop the habit of holding long videos on your phone for several days in a row, it actively suggests a more economical way of recording. The former requires an AI assistant to serve the user according to the current scenario, while the latter needs to call on memories spanning several days to summarize the user's lifestyle.
However, the current first-person video assistant is largely confined to two categories: passive-response, reactive, single-use query, model, single-response, semi-active, semi-proactive, pre-directed, model-monitoring, pre-defined events, and triggering responses. Both cannot be based on the user's history, habits, and current activity to independently judge whether "the user really needs to be bothered right now".
To address this gap, a team from the University of Technology, Alaya Lab, and the University of Tokyo has proposed Vinci2 to formalize Proactive Assistance as a shared "decision + generation" mission based on continuous video streaming: the intelligent body must not only perceive what is happening at every moment, but also decide whether and how to intervene at the moment based on the context accumulated over a long period of time.
Title: Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos
Authors: Shinshu, Shinshu, Shinkansen, Shinkansen, Shinkansen, Shinkansen, Shinkansen, Shinkansen, Shinkansen, Shinkansen and Shinkansen
Institutions: Indiana University of Technology, Alaya Lab, and the University of Tokyo
Link to the paper:https://arxiv.org/pdf/2607.11523
The paper's webpage:
Currently, the work has been received by the Computer Vision Summit, ECCV 2026, whose code and dataset labels are open source.
EgoServe: The first first-person streaming video-activated service rating benchmark
The existing video comprehension benchmark is difficult to assess for "initiativeness": the offline video questionnaire benchmark only examines the understanding of pre-division segments, the streaming video comprehension benchmark does not involve long-term memory and intervention decisions, and there are already active dialogue benchmarks that rely on explicit instructions given in advance by the user.
EgoServe's core design is to organize active services into four levels, 10 sub-categories, according to the time span of memory required by the service and the application scenario of the service itself:
Instant service: only current observation and instant memory, such as security alerts, tools and reminders;
Short-term service: requiring the most recent context, such as error detection and correction, user resource alerts, next-step guidance;
Scenario services: need to go through the current task (from a few minutes to a few hours), such as unfinished task reminder, phase memory retrieval;
Long-term service: requiring cross-sectional, cross-sectional contexts such as user habit counseling, scheduling optimization, and long-term memory associations.
The labeling process of the data sets uses semi-automatic flow waterlines: based on existing artificial labeling of the data sets, it drives the LLM to generate candidate service instances using customization prompts for categories, such as Gemini; for long-term services, it introduces a "stream trace capture strategy for models based on intensive artificial annotation of trace accumulated events, ensuring that long-term services are based on real multi-day behavioral patterns. All labeling experiences. The evaluation examines the timing of the school while examining whether the intervention falls within the trigger window, whether the category matches, reporting the categories F1) and response quality (LLM scores).
EgoMemo: Untrained memory enhances active service intelligence
The team constructed the self-trained EgoMemo, which continues to process first-person videos, streamlined the creation of multi-mode memories and retrieval enhanced reasoning for the input video chips.
During the flow memory building phase, the EgoMemo enhancement maintains three complementary memory characteristics:
(1) Multi-dimensional sequence memory: first, uniform sampling of the input video clips and the production of time-marked detail descriptions by VLM, and then the detail description of the layer-by-layer organization into a three-layer summary of the segment-level sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequence sequ
(2) Evolutionary knowledge graphs: Using LLM to extract entities and relationships from segment-level descriptions and to increase their merging, to obtain a comprehensive knowledge graph across the video as a whole, to establish structured indexes for semantic associations across time.
(3) Visual embedded archives: In order to supplement visual clues that may be missing from text memory, ImageBind is used to store key cryptographic encodings to supplement text with difficult-to-describe information on the appearance and spatial layout of objects.
In the retrieval-enhanced reasoning phase, the EgoMemo stream describes video clips generated prior to receipt. In the active service mode, the model needs to evaluate whether there is a need for active intervention in the current scenario. In the passive response mode, the model responds directly to a user's question at a given moment. In both modes, the reasoning model determines whether deep retrieval is needed and, if necessary, generates retrieval queries and activates the triple parallel retrieval mechanism.
(1) Multidimensional sequence retrieval: Directly embed the retrieval query encoding as a dense one and look for top-k similarity descriptions.
(2) Graph-based semantic retrieval: first extracting keywords from the retrieval query, then coding the keywords and finding nodes with high similarities in the knowledge graph, and then extracting the adjacent nodes and intermediate relationships of those nodes to map the entities in the nodes as a segmental description.
(3) Visual Similarity Retrieval: First use the LLM to rewrite the retrieval query to include visual properties, and then encode and extract the highest-similarity key metrics through ImageBind to extract the corresponding segmental description.
(4) VLM Description Restructuring: Re-injecting a segmental description based on graphs and visuals and its corresponding video arrays into VLM, prompting it to re-generate a new description based on the retrieval query, thereby integrating the original deviation from the query description into a coherent narrative of the query.
Finally, the description of the reconstructed and the original fragmentary descriptions retrieved in sequence are re-injected into the reasoning model in the context above and output the intervention decision or problem response.
The EgoMemo architecture not only supports both active service and passive query modes at the same time, but also seamlessly migrates to offline video understanding tasks through perceptual strategies "from rough to fine grain", providing universal information while maintaining access to evidence of detail.
Experimental verification: achieving SOTA in active services and multi-video understanding benchmarks
The team evaluated EgoMemo on EgoServe and several online and offline video understanding benchmarks, respectively, and obtained multiple SOTA results.
The results of the experiments on EgoServe show that:
(1) EgoMemo relies heavily on its multi-scale memory and reasoning retrieval mechanisms to outperform the closed-source models Qwen3-VL-Plus and GPT-5-mini, with the most significant improvements in scenario services and long-term services (ML reaching 4.9, RO reaching 11.8).
(2) The dissolution experiment describes the greatest contribution of reconstruction and diagram retrieval, and the three retrieval pathways are complementary and irreplaceable.
EgoMemo's architecture is equally well generalized to existing video rating benchmarks: on the OVO-Bench, the average accuracy of real-time perception class problems is 75.15, significantly exceeding GPT-4o[64.46); on the ESTP-Bench explicit activation task, 27.6 over the trained EyeWO[23.6); on the offline benchmark EgoSchema, 74.8 over the previous best 7.2 points, and QAEgo4D, with a best score of 68.0.
What is the meaning of this verse?
For the first time, Vinci2 introduces "whether to intervene, when to intervene, how to intervene" as an explicit reasoning question called the first-person video assistant, which defines the evaluation criteria with EgoServe to demonstrate the feasibility of non-training routes with EgoMemo. The absolute scores of all current methods are still low, precisely indicating that active service is a long-term, deep-dive direction. Both the benchmark and the code are open source, and the community is welcome to explore follow-up directions on this basis to intervene in time learning, audio context and multi-user scenarios.
Author's note
The first author of this article is a Ph.D. student of the University of Technology Luhankawa team, and the first three co-authors are interns of Alaya Lab. The main research direction of the University of Technology Luhankawa team is tracking and detecting significant targets. The Alaya Lab focuses on the study of world models and game intelligence.