He is a PhD student at Peking University, whose main research areas are controllable video generation, audio video co-generation, world models, understanding the generation of unified models and multimodal large models, and related results have been published in international conferences such as ECCV, ICCV, CVPR.
Over the past few years, video generation models have made continuous improvements in clarity, motion performance, and image texture. By entering a word, the model has been able to generate video with a cinematic feel.
But when the task goes from "creating a beautiful piece" to "filming a complete play", the problem is different.
In the first shot, the lady says a sentence; the picture is cut to the man, who pauses to respond; then the lens is cut back to the woman, while the man continues to talk outside the picture. To complete this, the model not only needs to know what's in the picture, but also to execute a series of narrative constraints: when to cut the lens? when to open the mouth? can the white match the shape of the mouth and movement? can the face and voice of the character still be consistent after cutting the mirror?
Even if these requirements have been written into the script, existing models may still have the misinterpretation of the mirror, the premature or delayed whitening, and the stringing of faces and strings in multiple scenes. The question is not only whether the story is clear, but whether the model has the ability to organize audio video content along complex timelines.
To this end, a joint proposal was made by Peking University, the Kool-Aid team, the Institute of Automation of the Chinese Academy of Sciences and Tsinghua University.MAVIN, a multi-lens audio video co-generation model that supports customised narrative control。
Currently, the paper has been accepted by ECCV 2026 in the form of Oral.
The title of the paper is: MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control
Author's note:Beijing University, the Corning team, the Institute of Automation of the Chinese Academy of Sciences, the University of Tsinghua
Address of the paper:https://arxiv.org/abs/2606.29473
The main page of the project:https://liukqchoco.github.io/MAVIN/
One is to generate film-sensitive images.
Why is it that "I can't finish a play"?
Multi-lens audio and video narratives face at least three challenges.
The model may know which characters and events are in the script, but cannot tell which time period they should appear: the footage has been switched, the voice of one of the characters is still continuing; the characters have not yet appeared, and the whites are responding in advance.
The second is identity confusion. When the same character appears across the camera, not only should the face, hairstyle and clothing be consistent, but also the sound be stable; when multiple characters alternate in conversation, it is also necessary to avoid mistakes in facial, syntactic and speaker bonds.
In addition, the user typically enters only a story narrative, rarely fully marking the lens boundaries, white space, character appearance and sound events. The system needs to complete a complete executable script, but it needs to actually be generated according to that script.
Therefore, MAVIN is not about simply "producing multiple shots", but about the model learning to navigate a unified narrative timeline, scheduling images, contrasts, actions and character identities.
Second, the core idea:
You can actually write a narrative timeline into a generative model.
MAVIN uses a structured script of three levels: overall, footage, and characters, uniquely describing the overall plot and background music, the timing of each footage's start, the content and character movements, the white content and its time zones, and the visual appearance and sound identity of the characters.
The model builds on the dual-tower video co-generation architecture, modeling video and audio in each of its branches, and exchanging information through cross-modal attention. MAVIN proposes two core mechanisms around "when it happens" and "by whom it happens".
1. Boundary perceptive attention: Accurately position the lens and the contrast in the narrative timeline
Video generation models typically allow the entire text to influence the location of all audio videos. This means that the model can also read the description of subsequent footage when it generates the first shot, easily causing the next character to appear in advance, the previous sequence to appear in the next shot, and so on.
MAVIN divides the video characteristics according to the boundary of the lens, the audio characteristics according to the white area, and then through dynamic token routing, to the specific scope of action for different information: the description of the lens only affects the corresponding segment, the appearance of the character only applies in its appearance, and the sound conditions only work in the corresponding white area.
As a result, lens switching, character movements, and speaking time are no longer just "soft requests" in the prompts, but are time constraints that can be executed within the model.
2. Identity perception dissemination: footage can be changed, actors cannot "off line".
MAVIN can simultaneously receive character reference images and reference audio, and extract "who's talking" from reference voices through a dedicated sound encoder, rather than copying the reference voice "what's saying".
On this basis, identity perception masks further restrict the transmission of information between different characters. Each shot reads only the visual reference of the actual appearing character, each segment reads only the voice characteristics of the current speaker; the words, images and sounds of the same character are linked to each other, while the separation between the different characters is maintained.
This is equivalent to setting separate visual and sound focus points for each character, thus reducing identity drift after multiple interviews, as well as string faces and strings in multi-person conversations.
3. Multi-Intelligent Screenplay Streamline: Complete the story with a "can be shot" screenplay
Faced with free user input, MAVIN also designed a lightweight multi-smart body flow line.
Structural analysis is responsible for identifying scenes, characters and contrasts, and dividing frames and contrasts; identification matches characters with reference images and reference sounds; narrative fine-tuning is responsible for completing the action, visual language and environmental sound, ultimately forming a three-tiered script for the whole scene, lens and character.
It is responsible for "writing the shooting plan clearly", while border perception attention and identity perception dissemination are responsible for "really shooting as planned".
Three, MAVINSet:
Let the model learn from the data how to shoot a play.
In order for the model to learn how to execute footage, training data must also contain a complete narrative structure. However, existing audio-video data typically provide only full content descriptions, lack of footage boundaries, footage time, and cross-lens character information.
For this purpose, the team built a multi-lens audio and video data set.MAVINSetIn contrast to regular video subtitles, MAVINSet provides three-dimensional labels for each piece of content, covering story progression, footage content, character movements, white text and time, and character appearance and sound characteristics.
For the contrast of cross-sectional lens points, the data also records their continuous relationship between different lenses. Thus, what the model sees is no longer just an audio video, but a complete "split-screen" containing images, sounds, characters, and time information.
Four: The results of the experiment:
It's not just the timing, it's the characters, it's the strings.
The team compared MAVIN with two types of representational methods: one type of MAVIN into multi-lens video, and the other type of MAVIN is the end-to-end audio video co-generation model.
Visual contrast
In the same three-shots snowboard dialogue, existing methods are more likely to cause character appearance drift, camera sequence distortion or inappropriate white-to-character correspondence. In contrast, MAVIN is able to perform shots switching more fully and maintain character identity and scene style between different shots while making white-to-character correspondence more accurate.
Quantitative indicators
MAVIN achieved the best results on 13 indicators covering audio video quality, semantic consistency, synchronization of audio images and multi-lens controls. Of these, the STA for lens switching accuracy of 0.9897 and the TAMS for white-time alignment of 0.8104 indicate that the model is not only capable of producing consistent audio video content, but also of more accurate execution of time and identity constraints in scripts.
From modeling skills to creative interfaces:
MAVIN expands the boundaries of multi-lens narration
Once the boundaries of the footage, the white space and the identity of the characters are unified, MAVIN is no longer merely a one-time production of the film according to the script. The creator can directly modify the timing, character settings and local narrative conditions, thus deriving a variety of real-world applications. The following are two representative application scenarios:
The story is told through a series of flashbacks and flashbacks.
By adjusting the lens boundary and the white space, users can change the narrative rhythm of the entire piece of content. Compressing the length of the lens and the white space allows the image to move forward with the character's expression more tightly; extending the corresponding space creates a more comfortable rhythm. Models reorganize movements, patterns and sounds according to a new timetable, rather than simply mechanizing changes in the existing video.
It's the first time I've done this.
MAVIN puts the character's visual appearance and sound identity as an independent editable condition. Users can either replace reference images or reference audio, make the same person speak in different tones, or have different characters share the same tone. The creators can freely combine the character's appearance and sound, while maintaining the correct visual and audible identity over and over again, while keeping the story, lens and white content unchanged.
Six, from generating images to organizing narratives.
MAVIN's focus is not only on how to produce a clearer, more natural single lens, but also on how the model can begin to process the lens, the time-space relationship between the image and the character's visual identity. The video-generated control particle also moves from "what is generated in the image" to "when the content occurs, by whom it is completed, and how it unfolds across multiple lenses".
This structured narrative control provides new technical pathways for scenarios such as multi-character shorts, digital people's performances, video previews, and interactive content creation, and allows audio-video models to be generated from a single segment to a more complete narrative organization.
When the model is no longer just completing a picture, but is able to organize and shoot a play along a timeline, AI video really begins to move towards a controllable narrative creation.
For more methods and details of the experiment, see the original paper and the project homepage.