From retrieving knowledge of action, searching for character references, to simulating movement and connecting lenses, how much improvement can a reinforced-learning smart body bring to video generation?
If a video model generates a person's tone, the picture can be very fluid. But if the request is specific to a particular trick, how should the hand be raised, how should the focus be shifted, can the model still do the right thing?
Add a bit of difficulty: specify a character to complete the action; or require two objects to collide under given conditions, and then divide a story into three successive shots.
These requirements take video generation to a more subtle level. Motion requires knowledge, characters need visual references, movement requires physical constraints, and multiple shots need to be ordered and continuous.
What would happen if the system could look at the data first, look for references, run a simulation when necessary, and then decide how to generate the video?
VideoGen-Agent attempts to pass these steps on to a trainable multi-modal intelligent body. It uses Qwen3-VL-8B-Instruct to call out the tools for retrieval, generation, and verification in multiple rounds of interaction, and to continue to make decisions based on the results returned by the tools. The research team first supervises micromanagement to build tool usability and then improves the decision-making process through multitasking reinforcement learning.
On VABench, which contains 600 pointers and covers six classes of capabilities, VideoGen-Agent's Toolset 1 configuration scored 75.6 points, up 19.1 points from 56.5 points for basic video generators. The score was further increased to 86.1 points after keeping the smart body parameters unchanged and upgrading the generation tools. In human comparison, evaluators preferred to upgrade configuration generated videos to the results of the strongest single model baseline in 84.3% of judgments.
VideoGen-Agent: Reinforcing video generation agents
Link to the paper:https://arxiv.org/abs/2609.24997
The main page of the project:https://andyca111.github.io/VideoGen_Agent/
VideoGen-Agent is a tool for selecting different generating requirements: retrieving motion knowledge, obtaining visual references, simulating physical movements, verifying scene structures, and generating continuous frames in stages.
Let's look at three examples of what the tool specifically adds.
In the first line of Figure 1, a single model has generated white-clothed practitioners and courtiers, and also shows martial movements, but the gesture shown is closer to the generalized arm, receiver, and does not clearly reflect the specific gesture details. The agent first retrieves the gesture instructions, adds the focus after movement, the right leg knee and front leg movements to the generating input. The right side presents a step-by-step lifting arm, forming a high one-handed position.
The second example is a scene in which a specific game character holds a hat. In the second line of Figure 1, a single model grabs "character" and "characters", but generates a writing style hat holder that does not present character shapes in the reference material. Agents use image retrieval to obtain visual references and then assign them to condition-generating tools. The right-hand result is the reference chosen in the figure that is closer to the hairstyle, clothing and overall character style. This comparison shows that character names alone can only generate a person that is semantically close to a person, while visual references can implement identification requirements for visible appearance characteristics.
The third example requires a 12-second video to be unfolded in three time stages. In Figure 1, the last line of the prompts specifies three stages of content: first showing a heavy-pressure rain hole, then the bracket is cracked, and the final rain hole is laid down. Compared to a single model, the time structure in the Agent prompt disassembles the video into three sets of continuous lenses and generates the prompt according to the corresponding time period respectively, while using the end column of the previous one as the next one's visual condition.
The three sets of samples correspond to the knowledge of the action, the appearance of the subject and the sequence of events respectively. They show the specific differences that are generated in the diagram; critical analysis cannot independently prove the accuracy and timing of the entire video, and the overall effect needs to be combined with subsequent VABench and human assessments.
The first thing to do is to get the information you need to generate this video.
VideoGen-Agent's work begins with understanding missions.
When encountering a motion description with expert knowledge, the intelligent body can first retrieve text information, adding the action steps and key details to the generating input. When encountering a specified character, landmark or branded object, it can retrieve images, select reference materials, and then call images or video generating tools under multiple reference graphics conditions.
Physical tasks provide a different tool for use. For a collision/fall scenario with a given initial condition, the system can simulate the conditions of motion and pass those conditions to a video generator that supports motion control. Thus, the generation process can use external computed motion information.
Tools can also play a role after video is generated. For example, when a video requires multiple objects to appear in a specific location relationship, the system can use target detection and depth estimation to examine the subject and spatial structure, and re-generate based on feedback decisions.
Multi-lens tasks require processing of front-and-back welding. The intelligent body can be mastered in the first paragraph, extracting the endpoints as a condition for the next paragraph, gradually completing the follow-up shot. The actual output of the first paragraph becomes the input of the next decision.
These capabilities share a smart-body strategy. They provide tool constraints and typical processes in system prompts, specifically when implemented, the strategy is based on user requirements and already observed selection tools, organizing parameters, and deciding whether to continue to verify or generate.
Intelligent entities revolve around user input and interaction history, continuously selecting tools and reading feedback. The tools are divided into three groups: information enhancement, video generation and results verification. The action-to-video expansion interface for robotic exploration experiments in the diagram, excluding the six-task main assessment.
From simulating tool calls to learning based on results generated
After connecting the tools, how can intelligent agents learn how to use them?
The team constructed 24K instructional tips and teacher interaction tracks for six classes of tasks. Among them, 16K instructional tracks were used to monitor microthoning, allow the model to learn how to select tools, fill in parameters, and use the information returned by the tools.
Supervised mitigation gives an example of what can be learned. Reinforcement learning further compares the performance of different execution trajectories under the same task, giving more effective decision-making a stronger training signal.
The difficulty is that video production is difficult based on a simple "yes or wrong" judgment. The tool calls the format correctly, does not mean that the tool is chosen correctly; the tool is chosen correctly, and does not guarantee that the final video meets the requirements.
Therefore, training simultaneously checks three aspects: whether the call is effective, whether the tool is used to meet the task requirements, and the video quality and consistency of the prompts. The weight of the three rewards is 0.1, 0.4 and 0.5 respectively.
This design connects the training signal to the entire execution process. For example, it is more meaningful than merely recording a retrieval call after seeing a reference image and evaluating separately whether the image reflects a specified action after generating a stream stream.
The scoring distribution of the six classes of tasks is not the same. After the study is also relatively unified in the group under the same tip, the advantages of joining the task category are unified, so that the learning signals of the different categories are on a comparable scale.
16K instructor trajectories are used to oversee micro tuning, 8K independent tips are used to reinforce learning. Mixed rewards combined with call effectiveness, task-related tool use and video quality guide sharing strategy learning.
600 questions to check whether the video has been completed
To assess the effectiveness of tool use, the research team proposed VABench: a complex task benchmark for video-generating smart objects. It contains 600 untrained tips, 100 tasks in six categories, and focuses on whether the system meets specific requirements for knowledge, identity, physics, scenarios, and timing.
Candidate suggestions were repeatedly checked, VLM screened and audited, with 100 items selected from 150 candidates in each category. The evaluation used Gemini 3.1 Pro, rated by the rating dimensions and weight of the video for each category, and combined to 0×100 points; an equal-weighted average of six categories of assignment scores overall.
The six categories of tasks will explore procedural knowledge, maintaining single-subject identity, maintaining multi-subject identity, physical consistency, scene combinations and multi-lens timing structures respectively.
These tasks are intended to highlight different difficulties: knowledge tasks focus on professional movements and processes; identity tasks check the visual characteristics of specified subjects; scene combinations check the relationships between subjects; multi-lens tasks focus on whether the required phases occur; whether the sequence is correct. Physical tasks are limited in scope to collisions and falls, while multi-lens tasks contain two to three equal shots of no more than five seconds each.
The following table lists the performance of four open-source video generators in the paper's main table, four commercial video generators, and two sets of VideoGen-Agent tools configured for six tasks.
Table 1: VABench Complete Results. All scores are uniformly converted to 0×100 points. Overall (compound score) is a simple average of six categories of scores without additional weighting. In each column, the gross and bottom lines represent first and second place respectively. When upgraded from Toolset 1 to Toolset 2, Agent's strategy remained unchanged and no additional training.
In the table, PK, SI, MI, PS, CS, MS are the corresponding procedural knowledge, single-subject identity, multi-subject identity, physical consistency, scene combinations and multi-lens structure respectively.
The Basic Generator scored 56.5, VideoGen-Agent's Toolset 1 configuration reached 75.6, up 19.1 points; Toolset 2 after the tool upgrade reached 86.1.
Human evaluation provides a different perspective. The study used the same protocol to make parallel comparisons between two groups: each group randomly extracted 100 pairs of videos from VABench, and each group received 400 judgments from four evaluators, each selecting a more preferred outcome.
VideoGen-Agent-Toolset 2 has a human preference rate of 84.3% against Seedance 2.0.
VideoGen-Agent-Toolset 1 has a human preference rate of 88.0% against Seedance 1.0 Pro Fast.
The two digits correspond to different contrasting models.
The study also examined separately whether the automated evaluation matched human judgments. Each of the six categories of tasks drew 100 pairs of videos, compared independently by three evaluators, and contrasted the majority of human opinions with the Gemini 3.1 Pro selection. If the two videos had the same automated rating, the two videos were handed over to VLM, forcing them to select a better one according to the same evaluation guidelines.
In this verification of 600 videos, the overall consistency of the automated assessment with the majority of human opinions was 92.0%. The six categories of tasks were 72.0%, 98.0%, 100.0%, 94.0%, 88.0%, and 100.0%, respectively; the minimum procedural knowledge indicates that professional actions and processes are still more difficult to judge. This consistency test and the previous four preference experiments are two independent analyses.
Tools are upgraded and strategies learned can be used.
Will the continuous updating of the video-generating model make a trained intelligence system obsolete?
The paper was tested with replacement tools. Keeping the smart body strategy, information enhancement tools and verification tools unchanged, the generating tools were upgraded from Toolset 1 to Toolset 2, and the total VABench score was raised from 75.6 to 86.1, with all six classes of tasks being upgraded.
Among them, the most noticeable change in maintaining multi-subjective identity is from 65.3 to 86.7. The system improved by 25.7 points on multi-lens tasks after using Seedance 2.0 relatively directly, and 14.1 points and 17.7 points respectively on maintaining procedural knowledge and multi-subjective identity.
The increase in scenario combinations is smaller, at 2.9 points. This also indicates that the benefits of enhanced tools are related to specific needs: some tasks rely more on external knowledge, reference materials or clear time structures; some tasks can already be handled better by powerful generators.
This group of experiments proved a practical versatility: Under interface-compatible conditions, people who have learned the strategy of organizing the generation process can continue to use stronger underlying generation tools without the need for additional training of an intelligent body.
Where did the uplifting come from?
The overall results show the effectiveness of the system. In order to further understand where the improvement came from, the paper compares suggestions for rewriting, fixed tooling processes, supervision of fine tuning, reinforcement learning and different reward configurations.
Table 2. Results of the dilution experiment on VABench. All agent-related variables use Toolset 1. Comprehensive scores. Overall is a simple average of six category scores, with no additional weighting.
After accessing the fixed-tool process, the score was raised to 64.1; after supervision, the smart body further reached 69.2; and reinforcement learning was improved by 6.4 points on this basis, with all six tasks improved.
The single-task variable trains six SMBs, each using only the corresponding category data for SFT and RL, and then summarizes the results of their respective responsible categories. The overall score is 76.3, slightly higher than the shared multitasking strategy of 75.6; the value of the shared strategy is that it covers the six tasks using the same model, rather than more than the strategies trained separately on each indicator.
Rewards decomposition also gave corresponding evidence: removing video quality rewards, the score dropped to 73.3; removing tool use rewards, dropping to 71.2. The results showed that in this training set, both final video feedback and tool use feedback are helpful for effective learning strategies.
Fixed processes here use the same basic intelligence body and Toolset 1, by which model organizations retrieve and generate inputs and code control processes. Most tasks execute a pre-set process with a maximum of three rounds of ReAct interactions; multi-subject tasks retrieve reference charts for each specified subject and call multiple reference charts to generate them. The number of calls generated does not match the RL trajectory, so these scores reflect the effect of the corresponding system configuration and cannot be directly explained by the efficiency comparison under the same generating expense.
Putting two demands together, can a smart body reassemble tools?
Real needs often contain multiple constraints at the same time. Have you learned "step by step actions" and "keep specified roles" to generate videos that "specify roles to complete professional actions"?
The paper constructed an additional test called Procedural Identity ((PI)), which selected 20 professional actions or processes from a pool of reservation prompts with 20 specified subjects, combined, weighted and filtered out 100 prompts, putting together procedural knowledge and single-subject identity requirements. These prompts did not overlap with those used by SFT,RL and VABench, nor did the training contain a combination of the two requirements.
Testing continued with Toolset 1 without additional training. The system instructed to retain the reference process for each task with guidance on a combination of general tools, but did not write out the execution process separately for PI. Scores were used to independently collect, reference the action text shared with the subject image for all methods; aesthetics, text alignment and image alignment weighed 0.10, 0.45 and 0.45 respectively.
Table 3. Zero-sample evaluation on unseen PK + SI combinations. The evaluation uses 100 remaining test cues. The tool call rate indicates whether the Agent has called the tool at least once during a complete execution process and does not represent the success rate of video generation.
With no additional training, the reinforced-learning smartphone scored 73.2 points, which was 57.6 points higher than SFT, and 72.4 points closer to the fixed flow of explicit combination tools. It summoned text and image retrieval at the same time in 84.0% of execution trajectories, while SFT's correspondence ratio was 40.0%.[3]
Fixed processes in combined tasks require sequential execution of text retrieval, image retrieval and reference condition generation, with each prompt executed once, the tool performs errors using the same retrieval upper limit, and the number of times generated is also not matched to the RL trajectory.
A description of the changes in the call of the tool indicates how the training affects the behavior when the model is required to process the combination. Scores provide further evidence that this change can improve the video results in the test. However, it is a specific combination that is verified here; the recall call rate itself is not the same as the video success rate.
Expanded to Robot Video: Adding a new toolchain
The paper also explores the scenarios of robot operation. Given initial image and text instructions, the intelligent body first summons π0.5 to predict the trajectory of the joint action, and then hands over the action to Ctrl-World to generate video. The study starts from the intelligent body after RL, using an additional 1,000 robot operation samples to fine-tune, obtained independently of the model evaluated by the six-mission master.
Qualitative examples, such as removing blocks, folding towels and moving towels, show that mitigated strategies can simultaneously call for action prediction and video generation tools. This expansion shows the possibility of accessing new tool categories; it uses additional training and differs in timing, torque and resolution from videos, and does not constitute a quantitative verification of the success rate of robot missions or actual robot execution.
Making the video generation process a learning decision
VideoGen-Agent clarifies a question that deserves further study: Can the system determine what information is missing before creating a video and then supplement it with tools?
Evidence of this work is currently coming from six class assignments and an additional combination test. More complex longer videos, more open physical processes, and call costs and wait times are still in need of further testing. But the results that have been observed show that training the intelligent body to organize retrieval, generation and verification can be an effective pathway to improving the quality of video production.