The TaoMate-H3 is an audio-video co-streaming generation model developed by Alibaba's TaoLive AIGC team. Based on the MiniMax H3, the TaoMate-H3 combines three-step LoRA reasoning with self-return generation, capable of sequentially generating text-based videos containing characters to white, vocals or ambient sound, supporting minute-level rewriting, and 480p, 768p, 1080p Yandex screen output.
Compared to the production method of repeating video noise, the TaoMate-H3 was developed in small segments with only three steps for each segment to make noise without waiting for the entire video to be generated. This significantly shortened the production time of the first segment and provided a new technical basis for live broadcasting, virtual role-playing and interactive video applications.
Currently, the TaoMate-H3 reasoning code with LoRA weight is officially open. Developers can download the experience via GitHub and Hugging Face to conduct application development and streamlined generation research on this basis.
This is the first time I've ever heard of such a thing.
Weight of the model: huggingface.co/TaoLiveAIGC/TaoMate-H3
What is TaoMate-H3?
The TaoMate-H3 is geared towards creative scenarios that require continuous sound and image creation. Users can describe the entire scene using a cue word, or arrange the plot, action and plot by paragraph; the model continues to generate based on historical context, allowing adjacent paragraphs to share person, sound and scene information.
This release revolves around three core capabilities.
Three-step flow generation, reducing waiting time
Through low-level LoRA, the TaoMate-H3 compresses the de-noise process of each small segment into three steps. The model prioritizes the completion of the current segment and plays back content without having to wait for the entire video to complete the noise. For content production that requires timely feedback, this step-by-step output helps shorten the wait and provides the basis for edge-generating edge playback.
Audio-video co-production, rich character and scene performances
The TaoMate-H3 handles sound and images throughout its lifetime, supporting both character speech, singing and character contrast, as well as surroundings and motion sound effects such as waves, vehicle movement, explosions.
Continuous writing at the minute level, supporting segmental content sorting
With continuously updated audio video KV caching and segment audio guidance, the model is able to transcend the word boundary to continue the previous content. Creators can adjust the narrative focus or performance content in stages, and organize more complete product introductions and character narratives.
Effects of Application
E-commerce explains: From product presentation to full introduction
Commodity presentations typically include multiple links, such as visual presentations, functional details, and use scenarios. The TaoMate-H3 supports the use of segmented prompts to organize the content, promote the narrative in continuous footage, and apply to short videos, product presentations, and brand content production.
The exhibition includes a one-minute introduction to the backpack and a side-by-side presentation of the wooden backpack. The two sets of exhibits showcase the effects of personality displays and merchandise displayed in different paintings, with the backpack video retaining a complete one-minute generating result.
The backpack explainsWhat is the meaning of this verse?
Around the same commodity, creators can also customize the narrative content for different populations, such as highlighting commute receipts, travel carrying or everyday use needs.
Singing and role-playing: voice, expression and action combined
In addition to commercial voice broadcasting, the TaoMate-H3 also supports performance scenes that contain vocals and music. The case of the Window Concerts showcases the singing process of writing real characters, organizing melodies, moods and physical movements in the same video.
Singing by the Window
Anime contrast: expanding stylized character creation
Audio-video co-generation also applies to anime content. In the case of Moonlight Girl and White Fox, the stylized characters, scenes, and characters together form a short drama clip. The creator can set the character image, atmosphere, and plot by way of prompts, and then step-by-step advance the story.
Girl of the Moon and White Fox
Environment and features: Making the scene have its own voice
The production range of the TaoMate-H3 also includes scenes where no characters are white. The case of the Sunset Coast depicts a space atmosphere with waves and ambient acoustics; the case of the film explosion combines character movement, building explosion and shock sound effects to produce a stronger visual and audio rhythm.
Sunset Coast and Sea Gate
People explode in front and behind buildings
These cases cover different types of content, such as merchandise narration, character singing, anime coloring, landscapes and film specialties. The models provide a horizontal picture frame and multi-layer resolution to accommodate mobile content with a side screen display.
The TaoMate-H3 overall process
From the prompts to the final video, the TaoMate-H3's reasoning process consists mainly of four components:
The content of the paragraphs is organized by reading the sequence of prompts according to the length of the target, identifying the scene, action and sound description of each paragraph.
Prepare audio routing: generate audio noise trajectories for the current paragraph and use sound information at the end of the previous paragraph to link the tone to the tone.
Synchronous stream generation: dismantles the paragraphs into small segments, each segment performs three-step audio video noise, and continuously updates the historical KV cache.
Decode output: The generated latent decodes the image and sound through VAE, completing the media output. Open logic input automatically runs the entire process and saves the final video.
Latent is an audio-video compression representation used in the model generation process. The model first completes the production in this representation space, and then decodes the viewable and audible content.
The TaoMate-H3 is a key technology
Three-step LoRA: Reduce production costs for each segment
The TaoMate-H3 adapts to the low-level LoRA, compressing the flow logic to three de-noise ranges. In the current five-second cue word configuration, each paragraph is divided into four chunk, the subject chunk is about 1.4 seconds, and the tail segments are shorter. The model completes these segments in succession and continues to generate them using historical context.
Three-step update progresses along 0→16→33→49 time state. After each segment is completed, the model updates the KV cache for use in the next segment based on the final audio video state generated.
Self Forcing: Self-return training for continuous writing
During the training phase, the TaoMate-H3 uses Self Forcing's self-returning training concept to condition the model on its self-generated historical segments and continue to generate follow-up content. This allows the training process to access historical states in actual retranscriptions, mitigating the differences between training and reasoning distributions that only depend on true historical training.
This method of training, along with step-by-step reasoning and KV caching, forms the basis for continuous writing: the model learns both the generation of the current segment and how to use the preceding output sequences of characters, actions, and scenes.
Audio video KV cache: support for continuous long video memory
To maintain the consistency of the long video, the TaoMate-H3 continuous duplicate audio video KV cache passes the content generated in context to subsequent clips. Character status, recent actions and sound information can therefore be continued over time, while still participating in subsequent generation while switching prompt words.
Caching is organized in the "initial visual reference + context on recent audio video". Initial reference retains the initial information of the person and scene, and recent context rolls updates as the progress is generated, enabling the model to keep up with the latest movements and sounds to continue creating.
This way, the history cache is controlled on a fixed scale, avoiding its increasing occupancy with the length of the video, and provides stable upstream context support for minute-level continuous generation.
Continuous time coding: improving cross-tweet word syntax
In long video, the prompts can change, and the media timeline still needs to be continuous. The TaoMate-H3 uses universal continuous RoPE location encoding, moves the time coordinates of the current text as the audio video progresses, and aligns the text's right border to the starting point of the current media, so that the new prompts match the content being generated.
When switching the prompts, the system retains existing audio video KVs, while distinguishing the newly generated content from the background on the tail required to decode the code. Historical content is only used for synchronization, not reproduction. For brightness and color changes in long-running retranscriptions, the model also uses the visual statistics of the first segment as a reference, balancing the mean to the aspect ratio of the subsequent video latent, reducing the displacement of the image.
Audio trajectory guides: Stabilizing rhythm and tone
Stream generation breaks down the video into short clips, but the pauses, speed of speech and ending positions of a sentence need to be arranged within the entire paragraph. TaoMate-H3 is designed internally to cover the audio trajectory of the current prompts, and then uses the latent audio of the corresponding time position to guide the small clips, so that local generation always has a paragraph-level sound reference.
Three de-noise updates align the three states of the audio trajectory respectively, with the final audio latent being consistent with the clean end of the guiding trajectory. The audio guides through the production process, helping to stabilize the rhythm of the lyrics, reducing the repetition of short clips when rewritten.
In order to combine the tone and tone of the different paragraphs, the system uses the clean audio of the last second or so as a read-only reference. New audio continues to be generated under this reference, and the reference itself remains unchanged.
Performance performance
On the same 8×NVIDIA H20 96 GB machine, we tested the original MiniMax H3 and TaoMate-H3 with an output of 480×864, 10 seconds of video.
The TaoMate-H3's pure DiT computation time has been reduced from 169,572 seconds to 14,810 seconds, accelerating 11.45 times; the final latency of the first phase has been reduced from 170,052 seconds to 6,148 seconds, accelerating 27.66 times.
In the aforementioned ten-second configuration, the TaoMate-H3 produced eight small segments together, performing 24 forward-generation and 8 KV updates. The single eight cards used TP2 × Ulysses 4 in parallel, and combined high-efficiency attention, calculation convergence and selective low-precision computation of partial linear layers, reducing reasoning costs.
These optimizations both increased the production efficiency of the entire content and shortened the production time of the first content, providing the basis for block decoding, playback and interactive application development.
A quick experience
Environment and Hardware
The project supports Linux、Python 3.10/3.11、CUDA 12.8 and PyTorch 2.8, providing a single 4-card or 8-card logic input. Verified running configuration is 8 × H20 96 GB, see README warehouse for full installation steps.
Running examples
Get the code:git clonehttps://github.com/TaoLiveAIGC/TaoMate-H3.git cd TaoMate-H3
Following the completion of the README Environmental Installation and the download of the MiniMax H3 Base Weight, a 10-second example can be run in warehouses. The TaoMate-H3 LoRA Weight is automatically downloaded when first used:
python -m taomate_h3 \--model-root models/MiniMax-H3 \--prompt-json examples/prompts_10s.json \--duration 10 \--resolution 768x1376 \--gpus 8 \--output outputs/demo_10s
Results are stored in outputs/demo_10s/video.mp4.[13] Alternative prompts can be used to modify scenes, lyrics, and movements; longer videos can be generated through time and resolution parameters, or a banner image can be switched.
Open source and follow-up plans
This release includes logic codes, LoRA weights and example hints, welcomes developers to download the experience, and welcomes interaction with us around streamlined logic, audio video generation and application integration.
The project is based on MiniMax H3 and follows the MiniMax H3 Community License. Thanks to contributors to MiniMax H3 and related open source tools.
The team is continuing to optimize the speeds of reasoning, further shortening the wait time for first paragraph content and subsequent responses. We also plan to launch and open up new FL2AV stream models in the near future, extending further to Ref2AV, and simultaneously opening up corresponding weights to support more rich audio and video creation methods.
Looking forward to working with the community to advance streaming audio and video generation, bringing this technology into more practical creative scenarios.