In the past few years, the field of robotics operations has been almost re-brushed by a series of new concepts.
LLM Planner, VLA, Diffusion Policy, Flow Matching, Latent Action, World Model, Imitation Learning, Reinforcement Learning... more and more methods are being used to describe how robots understand tasks, perceive the world, and ultimately make moves.
But one problem that is easily overlooked is:
These concepts are not actually on the same coordinate axis.
VLA focuses more on how multimodal information, such as visual, language, enters action models; Diffusion Policy focuses on how action is generated; Imitation Learning describes how models learn through sample data; and LLM Planner typically works at a higher task planning stage.
When these approaches are put together in a simple way, it is easy to confuse fuzzy systems-level fuzzy models with fuzzy learning strategies.
This is also a new problem facing robotics research in the Foundation Model era: there are more and more models, but we need to rethink the question of exactly where these models are located in robotic systems and what problems they are solving.
Recently, the paper Embodied Robot Manipulation in the Era of Foundation Models: Planning and Learning Perspectives was received by IEEE Transactions on Robotics (T-RO). The work attempts to re-establish the analytical coordinate system of robotics research, and to use High-Level Planning and Low-Level Action Modeling as two main lines for the development of robotics in the Foundation Model era.
The review was completed jointly by researchers from Xi'an University of Transportation, Hong Kong University of Science and Technology, Guangzhou, Chinese Academy of Sciences, West Lake University, Zhejiang University, University of Sydney, Beijing Institute of Intelligent Artificial Intelligence, Beijing University, and others. The first authors of the paper included Wei Weiwei University of Transportation, Hong Kong University of Science and Technology, Guangzhou University of Communication, Hong Kong University of Science and Technology, Guangzhou University of Technology, Li Lijiang University, Beijing University of Science and Technology, and Xi'an University of Transportation, Cheng Dong.
Embodied Robot Manipulation in the Era of Foundation Models: Planning and Learning Perspectives
Link to the paper:https://arxiv.org/abs/2512.22983
Project warehouse:
1. A diagram of the system boundaries of robot operation
Today's large-scale model is easy to give the impression that robots could eventually become a huge end-to-end network, inputting images and language, and outputting actions.
But from the robot system itself, there are still several different levels of problems between understanding the task task task and the task task task of getting the engine to really move.
The paper summarizes this process as follows:
High-Level Planning → Low-Level Action Modeling → Actuation-Level Control
The three levels each answer three different questions.
High-Level Planning: What should robots do? This layer is responsible for forming structured planning information such as task plans, procedures, geometric constraints, Affordance Prior or 3D Representation.
Low-Level Action Modeling: How exactly should robots move? This layer further transforms environmental observations, mission conditions and intermediate representations into executable actions or trajectories.
Actuation-Level Control: How are these actions performed steadily? Eventually, the actions need to be further implemented as real robot control signals such as position, speed, torque, and resistance.
It is not these three names that are truly remarkable, but rather they provide a set of coordinates that redefine today's understanding of robotics.
For example:
| Common Concepts | It answers the main questions |
LLM / MLLM Planner | How to form high-level mission planning and structured guidance? |
VLA | How do we use visual, language, and other multi-form information to model robot actions? |
Latent Action | Is there a better way to learn between perception and action? |
Diffusion / Flow Matching | How should action or trajectory be generated? |
Imitation / Reinforcement Learning | How do models learn signals by monitoring or feedback? |
Therefore, VLA, Diffusion Policy, and Imitation Learning are not three parallel crawling robot routes.
They're actually answering different questions.
That's why it's becoming increasingly difficult to organize robot-operated research just by following a popular model family.
2.Foundation Model
What really changed the plan?
The Foundation Model had the most direct impact on robotic planning, not by allowing the LLM to directly produce joint controls, but by allowing the robot to have a stronger semantic understanding, open world knowledge and multi-modal reasoning capabilities for the first time.
The question then becomes:
How can these capabilities be passed on to real robots?
The paper understands this process as generating different forms of planning artifacts.
These planning artifacts can be:
Planning Artifact | What the Foundation Model Provides | How to Use Downloader |
Task Plan | Task breakdown, skill ranking, and sub-goals | Skills call and further planning |
Program | Conditions, cycles and task logic | Call sensing and control interfaces |
Geometric Constraint | Positions, key points and spatial relationships | Optimization of sports planning and tracks |
Affordance Prior | Interactive regional and functional relationships | Get in touch and plan |
3D Representation | Space matches, picking up candidates, optimizing goals | Planning and action generation |
One of the most typical contradictions is that rational rational rational is not the same as physically viable rational.
Work such as SayCan, Inner Monologue demonstrates LLM's ability to break down tasks and plan long-term sequences, but robots end up living in continuous physical space. A language model can correctly say that a monkey puts a cup in a closet without knowing whether a mechanical arm can reach it, in what posture the cup should enter, or if there is a collision in the process.
Thus, another group of work began to look for a more reliable interface between semantic reasoning and physical execution.
VoxPoser translates language understanding into a spatial value map, ReKep translates operational tasks into critical point relationship constraints, and GeoManip further uses geometric constraints as the interface between semantic reasoning and motion generation.
The Foundation Model is not responsible for the complete trajectory here, but instead answers: What kind of spatial relationships should be satisfied?
Another change comes from Affordance and 3D Representation.
The traditional visual perception is more concerned with what objects are like, and what robots really need to understand further is:
How can this object be manipulated?
Where should cups be grabbed? Which part of the door can be rotated? What areas can be reached? What spatial relationships need to be maintained?
Studies such as Affordance, Gaussian Splatting, Neural Descriptor Field and 3D Feature Field are essentially attempting to turn visual perception further into action-relevant representation.
So, from this point of view, the real question that comes with the Foundation Model when it comes to high-level planning is not: can the LLM control the robot? but: what kind of intermediate structure is the most effective way to transfer the understanding and reasoning of the Foundation Model to the robotic execution system?
Why is VLA, Diffusion Policy and Imitation Learning not the same thing?
If high-level planning solves what you want, then low-level action modeling really cares about how you want to move.
Traditional robot operations have developed mature non-learning methods such as Sampling-based Planning, Trajectory Optimization, Model Predictive Control; with the development of robot learning, more and more studies have begun to learn motion models through data-driven methods, and further adopted different learning strategies such as Imitation Learning, Reinforcement Learning, and Auxiliary-task Learning to obtain and optimize robot behavior.
The paper does not simply classify by model names such as VLA, Diffusion, but follows the process of building an action model by breaking down learning-based lower-level action models into three questions:
Input Modeling → Latent Learning → Policy Learning
The first question is: what information does a robot use to make a move?
This corresponds to Input Modeling.
Early Vision-Action Policy primarily used RGB images or multi-point vision, and combined with robots' own state prediction actions.
But vision does not answer all questions.
Two-dimensional images do not naturally contain a clear three-dimensional structure, so more and more methods are beginning to introduce depth, dot clouds and other 3D representations; at the same time, natural language provides more open task conditions for robots.
Important changes in VLA are occurring here. It enables robotic motion models to simultaneously use information such as visual environment, language instructions, and robot state to allow the mouse to see what mouse and mouse are doing to enter the same motion modeling process.
The paper further develops VLAs from 2D/3D inputs and dimensional models such as model-oriented/model-agnostic.
The borders of robotic inputs are continuing to expand.
Vision can see objects, but cannot directly judge whether there is a slip; language can describe tasks, but cannot tell if the robot's contact force is too great.
As a result, signals such as touch, sensation, and sound began to enter the robot's model of action.
This means that the problem of Input Modeling has been solved by:
What did the robot see?
It gradually became:
What kind of information can a robot use to understand the real physical world?
The second question: What kind of middle ground is needed between perception and action?
This corresponds to Latent Learning.
Direct mapping of high-dimensional images, language, and robot states into complex sequential actions is not easy, so many approaches have begun to look for more compact intermediate representations.
One type of work focuses on Pretrained Latent Learning. Researchers use generic images, human first-person video, and large-scale robotic interaction data pre-training visuals or multi-mode representations to give robots more generic environmental characteristics.
Another category goes further into the movement itself, studying Latent Action Learning. Complex continuous movements can be compressed into discrete action tokens, represented as continuous latent vectors, latent dynamics, or even combined with predictive representations of the world model.
These methods are actually looking for the same thing:
Connecting high-dimensional perception with complex physical movements is more closely linked.
The third question: How did the final action come about?
This is Policy Learning.
Policy Learning focuses on how to truly transform input and potential representations into robotic actions.
Early methods could directly use MLP regression movements; Transformer provided stronger sequence modeling capabilities; Diffusion Policy handled complex, multi-peak motion distributions through generative modeling; and Flow Matching has become a new model for motion generation in recent years.
This is where an often-confused question becomes clear:
VLA and Diffusion Policy are not two equivalent technical routes.
VLA describes more about how visual, language, and action are organized and modeled, while Diffusion or Flow Matching focuses on how action is ultimately generated.
A VLA can fully use Transformer, Diffusion or Flow Matching as an action generating module.
Similarly, imitation learning and reinforcement learning also fall under another coordinate axis.
They are more concerned with:
How do models learn these behaviors through monitoring and feedback signals?
This is why:
VLA describes how fluid inputs and movements organize fluid, Latent Learning describes how fluid is formed in the middle of the fluid, Diffusion/Flow Matching describes how fluid movements generate fluid, and IL/RL describes how fluid models learn fluid.
These concepts are only put back into their respective dimensions, and today's rapidly expanding robot-operated research graphs will only become clear again.
What is missing from General Robot Brain? 4.
When planning and action modeling are put back into the same system framework, the next stage of robotics becomes even clearer.
The paper argues that there are at least four key gaps away from the truly universal Robot Brain.
The first is the real ability to push.
Many robot models today have been able to achieve good generalization effects in similar environments, new objects or task combinations, but a significant portion of them are still close to the training distribution plug. Truly universal robots need to face completely new task structures, scenarios, objects and even the robot itself, while also being able to continue learning and accumulating experience.
The second gap comes from data and evaluation.
Real robot data is expensive, slow to collect, different benchmarks often cover only a fraction of the capabilities. Scoring platforms such as Calvin, Libero, SimplerEnv focus on long time sequences, multitasking, or real-to-sim, but the entire field still lacks a truly unified and highly reproducible scoring system.
The third gap is real physical perception.
Human-operated objects do not rely solely on vision; we also rely on touch, sensation, sound, and judgments about materials and dynamics.
For complex objects such as fabrics, cables, fluids, particles, and more, just adding a camera or a larger visual model is far from enough. Future Robot Brain will need to understand more about contact, materials, transformation, and the physical processes themselves.
The last issue is security and cooperation.
The success rate is no longer the only indicator when robots actually enter homes, factories and environments where people share with robots.
Robots must be able to continuously monitor their behavior, recover in the event of failure, reduce risk in the event of uncertainty, and at the same time meet safety constraints such as kinetics, kinetics, contact force, and energy.
This also means that the real robot system of the future is unlikely to simply replace everything from end to end.
While learning models provide generalization and adaptability, traditional methods such as rules, planners, and MPC may still play an important role in reliability and safety.
5. From the competition to the robotic system.
The Foundation Model is rapidly changing the way robots operate.
LLM gave robots greater semantic reasoning capabilities, VLA began connecting language, vision and motion, Diffusion and Flow Matching changed the way motion is generated, and World Model and Latent Action also attempted to establish a more compact middle representation.
However, as the number of models grew, a more important issue emerged:
How should these capabilities be combined to truly create robotic intelligence?
This T-RO review does not attempt to re-create a new model family, but attempts to go back to the robot system itself, with High-Level Planning and Low-Level Action Modeling as the main lines, to put the approaches that have emerged rapidly in recent years back to the real problems they solve.
From this point of view, the ultimate question that General Robot needs to answer is perhaps not how big the next model should be, but a more basic question:
How do robots form an understanding of tasks and the world and reliably translate this understanding into executable, generalizable behaviour in the real physical world?
