Moving a box from the ground to a designated location is a simple task for humans. For humanoid robots, it is a whole body collaboration.
When approaching, adjust the position of the station, when descending, maintain the balance, keep your hands firmly in contact with the box, and when getting up and moving, coordinate your legs, knees, and arms. Any link that goes wrong can cause the box to slip and even cause the robot to lose its balance.
One way is to prepare a complete reference action in advance so that the robot can track it step by step. But if only the target location of the box is given, how should the intermediate action be generated?
Is it possible to use the same controller to accurately track both reference movements and non-reference movements, depending on perception and mission objectives?
This is exactly what ULTRA is trying to solve.
We presented ULTRA (Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation) and validated it on the Unitree G1 humanoid robot. The work has been accepted by IROS 2026 and nominated for the IROS 2026 Mobile Manipulation Paper Award.
A controller that accepts different "degree of detail" instructions
We first distinguish between the two types of control needs.
Motion tracking provides a step-by-step guide to how the body moves, how objects move, and how they have a clear trajectory.
Target-driven control only determines the desired outcome, such as the object's target position and direction, leaving the intermediate movements to the controller. This is called "rare target": it does not specify step by step how the robot steps, bends, and raises its hands.
ULTRA puts these requirements into the same control framework: these capabilities are realized by the same set of strategic parameters. Controllers can also use different perceptual inputs, including the state of objects provided by external motion capture systems, and information obtained by the robot's own depth camera.
"Autonomous" here refers specifically to the generation of movements under the conditions of a given goal and perceptible perception. The goal still needs to be provided externally, and robots no longer need others to dictate in advance how each piece should be moved. Human movements cannot "copy-paste" before training data is physically consistent.Where did this ability to collaborate throughout the body come from?
However, the body proportions and joint structure of humans and robots are different. The same movements of the box, mapped directly to the robot, may cause problems of not reaching the box, slipping under the feet, or unreasonable contact. The gesture looks similar and does not represent a steady execution of the movement.
ULTRA introduces physical-driven neural redirection for this purpose.
It can be understood as: allowing robots to learn how to "use their own bodies" in physical simulation to complete the actions in the demonstration. By reinforcing learning, redirecting strategies, while trying to conserve human movement and operation intent, considering dynamics and contact constraints, generate executable robot trajectories in the simulation.
Once the training is complete, the same retargeting strategy can handle new moves and expand the data by adjusting movement range and object size without having to retrain a set of strategies for each new trajectory.
In this way, human movements are transformed into training materials that can be learned as follow-up control strategies.
The following video focuses on how the same human movement is transformed into a robot's full body movement, and how the coordinates between hands, feet and objects are adjusted after changes in size and range of motion.
It takes two steps to learn from "following" to "achieving".The next step is to train the controller to work under different input conditions.
The first step is to pass on control skills through teacher strategies.
In training, we first train a "Teacher Strategy" to read the full simulation status and reference movements, learn how to coordinate the body, maintain contact and complete operations.
Students need to adapt to different information combinations: sometimes with complete references, sometimes with partial objectives; sometimes with accurate object states, and sometimes with first-person perception.
For this purpose, ULTRA uses an input availability buffer. It is like a list of information that tells the strategy which inputs are available this time so that the same model generates actions based on existing information.
The second step is to improve the ability to deal with biases by reinforcing the mitigation of learning.
After learning to imitate, the strategy also needs to deal with situations beyond the demonstration: a change in the initial location, a new location of the target, or a deviation in the execution process.
ULTRA therefore further refines reinforcement learning in simulation, allowing the strategy to learn how to adjust movements and approach objectives through attempts and task feedback. The scope of training focuses also on reproducing reference movements and expanding to actual accomplishments after a given goal.
It's a robot's way of looking at things.
Beyond control, there is a real question: How do robots know where the boxes are?
In the laboratory, external motion-capture systems can provide precise location of objects. But to move into a wider use scenario, robots also need to use their sensors to understand the immediate environment.
ULTRA supports first perspective deep perception: extracting object point clouds from a depth chart, that is, describing the shape and position of objects using a set of spatial points, and then providing them to control strategies.
We tested complete motion tracking, fine-grain control, longer-term goal tracking, and goal-driven control based on first-point depth perception on G1. When we used first-point cloud input, we found that reinforcement learning mitigation greatly improved success rates.
"How to move in the middle" and give it to the controller
Human motion data provides a wealth of body-to-body motion experiences. ULTRA attempts to translate these experiences into more flexible control capabilities, enabling a set of strategies to accept different levels of detailed mission instructions and use perceptual information from different sources for action.
From tracking a given trajectory to generating movements after a given target, humanoid robots need to cross multiple barriers between data, control and perception. ULTRA shows a viable path to connect these links.
When people only need to specify the results they want to achieve, robots have to learn to add "how to move in the middle".
The paper
ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body
Loco-Manipulation
The article links to:https://arxiv.org/abs/2603.03279
The main page of the project:https://ultra-humanoid.github.io/
Author