AI real-time interactive video is one of the hottest technology trends this year.
For most of the past year, the model has been heavily circulated. Ali launched HappyOyster, Kuan Yew released Matrix-Game 4.0, Epic Technologies released PixVerse R2, Biometric Technologies launched Vidu S1 in July, and Vidu S2 in September...
Capital is also on the run, with many companies completing a new round of financing, world models, digital people, real-time editing...
But behind the heat, the track is in a state of chaos.
The confusion comes first from the definition. Similarly to real-time interaction video, some call it the "interaction world model", others call it the "real-time interaction multimodal model", and some understand it as a third line independent of video generation and world models.
The existing assessment tools are still using the old framework to give the model a hint, generate a video, see images that are not clear, movements that are not logical, style not in line with description, and mainstream benchmarks such as VBench, EvalCrafter, but for real-time interactive video systems, this method is not the hardest part of testing.
The core challenge of real-time interaction video is that the user's instructions are continuous, and the model needs to respond to new instructions in a timely manner, without interruption, while maintaining the identity of the person, the scene, and all previous effective modifications.
Recently, China University, NVIDIA, CMU, UC Berkeley, Princeton, NUS, and other overseas universities have jointly released a new rating benchmark called StreamEval.
Leaderboard:
Paper:
Github Code:
Benchmark Data on Huggingface:
Benchmark Data on ModelScope:
It is the industry's first global streamlined interactive video-generation rating list, covering the five categories of digital conversations, virtual human performances, world generation, instructional editing and reference editing tasks, with a total of 300 test use cases covering 38 models.
Compared to existing benchmarks, StreamEval not only measures "generation quality", but also "interaction capability", incorporating instruction execution, state consistency, visual quality and sound image coordination into the same quantifiable assessment system, allowing models of different technical routes to complete tests under uniform conditions.
Currently, all code, data sets and scoring rules are open source, and developers can independently review and verify the scoring process according to an open process. The list is constantly updated and any developer can access their own models and participate in testing.
What exactly is StreamEval and how is it measured?
Assessing real-time interaction systems, it must be able to run in close to real conditions, orders to arrive in batches, scenarios to evolve continuously, and models to respond continuously in the flow of time.
StreamEval is based on this logical design, covering five categories of tasks: digital conversation, virtual human performance, world generation, instructional editing, and reference editing, each with 60 test use cases covering 15 seconds, 30 seconds, 60 seconds, 120 seconds, and 300 seconds, respectively. The measurement dimensions are very comprehensive. Each use case is run independently of the same model three times to reduce the impact of random factors on the results.
StreamEval's overall framework covers five categories of tasks, including digital human interaction, world generation and real-time video editing.
The complete set of data contains 1347 online events, voice, text instructions, reference charts and source video clips are released gradually according to the time axis, the model can only see the inputs that have already been released, cannot know the follow-up requirements in advance, and cannot reach the expected answers for scoring.
Composition of the benchmark dataset: (a) task and functional distribution; (b) preset input type; (c) number of input events per test session; (d) time interval between adjacent preset input events. The chart shows 200 test use cases containing a total of 562 online input events, excluding initialized input and continuous input.
There are five main categories of tasks:
the ability of the digital person dialogue examination model to understand the user's speech and respond to the context, while maintaining the identity of the person and the status of the conversation;
Virtual characters perform actions such as gaze, expression, gesture, and gesture based on real-time instructions;
The world generates attention to continuously controlled scene changes, lens movement and spatial continuity;
Instructional editing and reference editing require the model to complete modifications based on text or reference graphics while the source video continues to play, and retain unspecified content.
The scope and evaluation time of the five categories of tasks. Dlg. indicates a digital person's conversation, Perf. indicates a virtual person's performance, World indicates world generation, Instr./Ref. indicates real-time video editing based on text instructions and reference images respectively.✓ indicates that the task is included, and x indicates that the task is not included.
StreamEval will break down the performance into four dimensions and ten indicators.
Task execution focuses on conversational performance, instruction tracking and complete editing; status consistency examines image stability, time continuity and non-targeted content retention; visual accuracy covers image quality, personality and reference image matching; voice image coordination mainly evaluates sound image synchronization.
Each indicator uses 0 to 100 points. Each task only calculates the applicable indicator and receives an integrated score within the task. Digital human tasks also add 5 points to systems that support self-defined images. The five categories of tasks are ranked separately, not merged into a cross-task overall list.
38 model rolls, consecutive instructions the most difficult to score
The paper evaluated a total of 38 models, forming 49 model task configurations. The results showed that image quality, personality retention and real-time control are several relatively independent abilities, allowing the model to stably generate video rather than being able to reliably execute constantly changing instructions.
In the Digital Man Conversation mission, the Basic Comprehensive Score of the Vidu S2-Avatar (Q-Overall) is 88.98, and after 5 points of custom image ability, the Comprehensive Score of the Vidu S2-Avatar (Overall) is 93.98, the highest score in a rating model that supports custom images.
Results of the evaluation of the Digital People Dialog mission.
The Vidu S2-Avatar also achieved its highest overall score at 81.71, with synchronization, stability and identity consistency of 99.60, 97.25 and 86.12 respectively. However, its instructions followed only 36.90.
This means that the character can remain stable, the mouth shape can keep up with the sound, and the specific movements may still not be completed as requested. Continuous interaction poses a more complex requirement for the model to both hear the instructions and make changes to the right object at the right time.
Global generation also demonstrates a distinct ability to differentiate.
MemFlow scored 33.36, Krea Realtime scored 25.93, and Matrix-Game 3.5 scored 34.50. The graphics and stability of HappyOyster 1.0 Adventure were 58.34 and 56.67, respectively, the highest scores for the task, but the instruction followed with only 5.56, and a continuity score of only 2.78.
World generating. Model configuration uniquely participates in ranking under text conditions and image conditions; when the same model receives two input-condition evaluations simultaneously, it reports only its evaluations under text conditions.
These results show that image perception, instruction response and long-term continuity need to be judged separately, and that a model looks more stable, not that it listens more to commands.
In the real-time editing section, Vidu S2-Editing scored an overall score of 45.43 on instructional editing tasks, with instruction tracking and editing success rates of 72.10 and 57.20, respectively; JoyAI-Video-Edit scored 42.39, but content preservation scored 83.33, higher than Vidu S2-Editing's 60.93.
Vidu S2-Editing has a higher Overall score of 39.02, Preservation score of 64.44, and Edit Success score of 15.00.
Reference to image switching, modification of range controls, maintaining the existing state and revoking restoration, needs to be done together in the same continuous generation. Models can successfully change the appearance of the characters, destroying the background; or they can retain the original video content, but without adopting a new reference image. Only all conditions are met at the same time, to obtain a complete edit score.
For example, by systematically loading the red rainbow veil, switching the 10th to the golden veil, and requiring the removal of the editors in the 20th second, the model must be changed in time, while maintaining the character movements and background, and finally accurately restore the original state of the source video.
Reference chart editing in the process of input changes. Reference chart A is pre-loaded, reference chart B is entered in the 10th second and the instructions to remove the reference chart are sent out in the 20th second. Each column pairs the source video footage of the corresponding meeting time points with the model output. Local enlargement of the chart shows the chart area selected by the box.
Therefore, the focus of StreamEval is not on who is the overall leader in all tasks. Different models have advantages, the challenges faced by the industry together, how to steadily execute continuous instructions in the continuous generation process, and correctly maintain the status quo.
What is the meaning of this verse?
The value of StreamEval is more than just a model ranking.
It is backed by leading academic and industrial institutions such as Qingdao University, Ingrid University, CMU, UC Berkeley, Princeton, NUS, etc. The collaboration of a multidisciplinary research force has determined that it is a public assessment framework for the entire industry from the outset that serves the common needs of the field.
More importantly, it is establishing a set of reproducible, sustainable evolutionary evaluation standards for real-time interactive video. Open test data and scoring rules allow developers to have a shortcut to accurately locate models, thus providing a clear and verifiable basis for the progress of different technological routes.
The amount of money in the list depends on how many models are involved, whether the evaluation will continue to be updated. StreamEval plans to maintain the list for a long time, continuously incorporating new models and the latest evaluation results each month. As more research teams and developers join, test records will continue to accumulate, helping the entire industry to better determine where technology has progressed and what issues remain unresolved.
StreamEval participates directly, in three steps.
The first step is to get the code and data. Developers can view the evaluation process via GitHub, access examples and model evaluation rules, and download the necessary media files for evaluation through open datasets. The project provides complete task definitions, test timelines and expected results, supporting repeat evaluation under uniform conditions.
Second, access your own model. The project provides clear model interface specifications and currently supports both in-process model interface and WebSocket service access forms, specific technical requirements can be found in the Add a Model documentation on GitHub.
The third step is to submit model information and evaluation results. Research institutions, enterprise model teams and individual developers can submit model information via the StreamEval official website ("your model" entrance. Participants who have completed tests locally in accordance with an open process can also attach their own evaluation results and technical feedback.
The next step in real-time interactive video requires continuous model breakthroughs, as well as a set of verifiable evaluation criteria. If you're developing a digital person, interactive world or real-time video editing model, you might want to personally go through StreamEval and get your own model tested and get more real results into this constantly updated list to add value to the industry's progress.
GitHub：
© THE