When one hears "Raise your hands above your head and then move forward", one can complete an entire movement in one's brain; when one sees a silent human trajectory, one can judge whether a person is walking, turning or walking. Continuous movement can be described both linguistically and as a condition for generating, predicting and editing.
Today's large multi-modal models are able to understand and generate text and images, but rarely use "how people continue to move" as a standalone model. Movements from a single stroke to a few seconds later, the model still needs to process order, speed, direction, focus shift and coordination with the whole body.
Researchers from the University of Beijing, the University of Taipei and the South China University of Technology have proposed UniMotion, which will incorporate continuous motion into the Unified Multimodal Model (UMM), connecting Motion、Text with RGB in the same framework. The paper has been accepted by ECCV 2026.
Figure 1 UniMotion covers seven tasks such as understanding, generating, predicting and editing movements within the same framework.
UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation
Link to the paper:https://arxiv.org/abs/2603.22282
The main page of the project:https://wangzy01.github.io/UniMotion
The code link:https://github.com/wangzy01/UniMotion
Movement model gaps in the unified multi-mode model
In the field of human movement, "single-handedness" can describe a certain type of human body structure, and "double-handedness" can record how the body structure evolves over time. A step-by-step gesture seems reasonable, and does not mean that it is still natural after connecting the sequence: the feet continue to slide flat after landing, the angles of the joints change to produce vibration, and the sequence of the movements in the stages of error can destroy the whole meaning.
The existing Motion Text method often quantifies trajectories as discrete tokens, allowing language models to be accessed, but may lose particle change and time continuity; human modeling of visual models is also developed mainly around static Pose. UniMotion allows continuous motion to be integrated into the process of understanding and generating.
UniMotion is the only way to cover the seven tasks in the table.
Continuous movement interfaces aligned with cross-modal
UniMotion is based on Show-o2 1.5B. Text, Motion and RGB enter the shared LLM backbone; text continues to use discrete tokens and self-return generation, while Motion and RGB are encoded as continuously latent and generated by their respective flow heads. Unification occurs in the semantic and reasoning mainstream, while the modular proprietary interface is retained.
Image 2: Motion accesses the shared skeleton through parallel continuous paths with RGB, while Text retains the back-to-back interface.
Motion latent through semantic and generating dual path embedders: the former generalizes the meanings and phases of the movement, while the latter retains local junctions and time positions; Hybrid Attention is responsible for coordinating the different modeling needs of continuous signals and text with the LoRA, which routes by model.
Cross-Modal Aligned Motion VAE ((CMA-VAE) uses motion coding as a continuous latent. When training, DPA will bring the image semantics backwards to a motion encoder that only reads motion; when reasoning, the visual fusion encoder is removed, and the pure motion task requires no additional images.
Figure 3 shows that the two encoded pathways share a Motion Decoder.
The LRA uses motion latent as a density condition to allow the flow head to rebuild the target latent from noise, co-calibrate the embedders, the main branch and the generating head. Motion-to-Motion provides only pre-training signals to improve the ability to perform downstream tasks in a simple but clear way.
Figure 4 LRA uses the Motion-to-Motion self-monitoring mission pre-calibration movement to generate pathways.
Ability to understand, generate and edit
The text generates a sequence of movements: "a person throws something and then moves forward" on the left, and "a man jumps and brings both arms above his head as he spreads his legs and then moves them back into the original position".
Text-to-Motion requires the language to be developed in a complete chronological order, while Motion-to-Text generalizes the meanings of the movements and the sequence of stages from continuous trajectories.
Image 6 UniMotion generates an English description based on continuous action.
Motion Editing simultaneously receives the source movement with the text instructions. In the diagram below, the source movement "waving" is changed to "drinking coffee", and the meaning of the movement changes and the trajectory remains continuous.
Figure 7 The source action was edited for coffee under the "not waving drink coffee" instruction.
Motion Prediction continues the future trajectory according to the observed forecast. The sample provides 1.05 seconds of motion, and the model continues to predict subsequent 4.35 seconds of change.
In Figure 8, the model predicts a subsequent 4.35-second trajectory based on a 1.05-second action forecast.
The same interface also supports recovering human posture from images, generating descriptions for images or videos, and allowing reference movements to participate in character image editing. Movements can thus serve as input, output, and cross-mode generating conditions.
The results of the experiment
The experiments covered data sets such as HumanML3D, MotionFix, Human 3.6M, and MoVid. The radar map shows that UniMotion is the only model in the comparison to cover all mission directions.
Figure 9 Summary of cross-mission outcomes of seven missions
The paper reported competitive results on tasks such as Text-to-Motion, Motion-to-Text, motion prediction, motion editing, and visual human body recovery respectively; where the distribution indicators for Text-to-Motion were not all optimum, and dedicated human body recovery models were still less error-prone.
Text-to-Motion results from HumanML3D on page 2.
The project page also gives a qualitative comparison under the same complex tips. In the case of "a person stomps up some stairs", the movements require continuous lifting of legs, shifting of weight and demonstrating step strength; UniMotion performs these steps more fully.
MotionGPT and UniMotion generate results under the same English prompts.
UniMotion demonstrates a way for unified multi-modal models to access continuous, structured non-native models: sharing semantics and reasoning capabilities while maintaining proprietary representation and generating interfaces. This idea serves to generate general understanding of human behavior, digital and content creation, and design references for other continuous signals.
Introduction by the author
Wang Liu, Wang Liu, PhD student at the University of Beijing in Shenzhen, where his research focuses on multi-modal models and human movement; Chen Shui, PhD student at East China University, where his research focuses on world models; Liu, professor at Shandong Technological University, where his research focuses on robotic autonomous operation, edge intelligence and learning, and others; Dream Source, Beijing University in Shenzhen, where his research focuses on human behavior understanding and learning of robotic skills.