Multimodal basic models can learn to "see" and "speak" through Internet-sized images and language, but with physical intelligence there are no shortcuts. Robots must further understand physical states and dynamics, determine how actions change the environment, and perform continuous actions in the real world; this requires training data to simultaneously connect observation, state, action and physical consequences.
At the heart of the question of "what data supports the basic model of physical hardware", several organizations in the United States have proposed specific definitions of the physical hardware data pyramid, organizing the existing data ecosystem into five complementary levels: machine data, UMI-like data, self-centered and external viewpoint data, simulation data, and general visual language data.
The framework focuses on the tension between the "degree of scalability" and the "degree of robotic alignment", and analyzes the value and boundaries of each layer of data in four additional dimensions, from quality, diversity, reproducibility, and physical integrity. On this basis, the researchers further explore the value and boundaries of each layer of data.
The topic of the paper is: Data Pyramid for Embodied Manipulation
Collaborating institutions: Beijing University, South China University of Technology, Hong Kong University of Technology, the National University of Singapore, the Chinese University of Hong Kong, and the University of Hong Kong
Link to the paper:https://arxiv.org/abs/2607.24744v1
The main page of the project:https://jasper-aaa.github.io/embodied-data-pyramid/
Open source warehouses:
1. Research background and challenges:
A person with physical intelligence cannot directly replicate "Internet pre-training".
In recent years, multi-modal base models have gained a wide range of perceptual, semantic, and reasoning capabilities, relying on mass images, video, and language data. When this paradigm is extended to physical intelligence, the problem has changed fundamentally: robots must not only recognize scenarios and understand instructions, but also understand the state of objects, contact relationships, and environmental dynamics, and transform higher-level intentions into executable control sequences.
Accordingly, the paper poses two questions that go through the entire text: first, how to collect, organize, compare and integrate heterogeneous data in terms of scalability, degree of robotic alignment, physical fidelity, and significant migration differences; and second, how to select and combine these data to target different capability goals for physically-brained, VLA and world-moving models.
Figure 1 provides a comprehensive framework for the paper. Work begins with the construction of a five-tiered physical data pyramid, followed by analysis of the use of heterogeneous data in the physical brain, VLA and the world's motion models, and concludes with a summary of the future direction of data collection, alignment and formula design.
2. The data pyramid:
Five categories of data organized on a scale aligned with robots
Data pyramids are organized by two core axes: "scalability" measures the expansion efficiency of a data in terms of hardware dependency, human input, environmental replacement, security oversight, and marginal generation costs; and "robot alignment" measures how directly its observation, expression, and monitoring signals support the learning and execution of physical robots.
The two are often stressed: the trajectory of the machine is most consistent with the target platform, but requires real hardware, operators, and repetitive replication; general network data is most easily expanded, but cannot provide direct movement and contact monitoring.
In addition to the above two main axes, this article further describes the practical use of data in terms of quality, diversity, reliability, and physical certainty. Quality focuses on trajectory, marking, and synchronization as reliable and informative; diversity focuses on coverage of tasks, objects, scenarios, perspectives, entities, sensors, behaviors, and outcomes; reliability focuses on whether data transcends tasks, environments, entities, and model migrations; and physical security focuses on whether contact, friction, sequencing, sensing noise, control delays, and object motion are actually recorded.
From the top to the bottom of the pyramid, the 5 categories of data are real-machine data, UMI-style data, self-centered and external viewpoint data, simulation data, and general data. This sequence reflects the data's gradual transition from stronger physical landing capabilities and executable monitoring to higher accessibility and scale. It should be emphasized that the sequence is the overall result of six dimensions, which does not mean that all individual properties are strictly monogamous.
Various data sources are expanding rapidly, but their "scale" has different meanings: robots, UMI and simulation data are usually measured by trajectory, self-centered data is often measured by hours, and general data can be described by queries, images, or labels.
The core logic of the pyramid
Moving downward is not simply "reducing the quality of data", but at the same time loosening some sort of robotic clutter in exchange for greater coverage. UMI removes robots in the collection link but retains end-user supervision; human video removes the claw, but retains interaction with the open world; simulation restores executable actions and privilege tags, but approximates real physics; and general data further abandons robotic actions, taking over the network's semantics, perception and reasoning coverage.
3. Key insights into the five categories of data
3.1 Real machine data: the most direct executable supervision, costing hardware and collecting closed-loop
Real machine data is located at the top of the pyramid. It records visual or multimodal observations, robot states, control movements and execution results in the same physical enclosure, so that it can directly reflect real sensing noise, control latency, touch dynamics and hardware constraints. No other data source can provide an equivalent level of target platform alignment for learning deployable operating strategies.
Its core bottleneck is the cost of acquisition. Each trajectory relies on real hardware, operator or automation strategies, scenario replication, maintenance and security oversight, and is difficult to scale at the same low cost as network data or simulation data. Different robots differ in form, sensor, coordinate system, and motion space, making cross-dataset integration much more difficult than a single file format.
The paper further emphasizes that data diversity is often more critical than the number of tracks themselves. Even if the same task has a large number of samples, if the initial state, path, contact sequence and execution speed are highly repeated, the state-of-the-art motion distribution of the model is still limited. Therefore, the development of real-machine data should focus simultaneously on the expansion of the task, the scene, the object and the model, and the diversity of trajectory levels within the task.
Towards the future, machine-to-machine acquisition needs to be advanced along three paths: a more intuitive, smooth and retaining fine-particle feedback remote-operation interface; field acquisition covering open environments, objects and conditions changes; and a shift from pure human demonstration to autonomous strategy rolling with human intervention, so that the failure of the strategy itself can be transformed into error-correcting and restoring supervision.
Figure 3 visualizes the trajectory diversity of the representative data sets. The spatial distribution of the key action packs reveals that even with large amounts of data, there may still be significant differences in the coverage of different data sets in the workspace, trajectory aggregation and changes in tasks.
3.2 UMI data: real-world demonstration of "approaching robotic action" outside of robots
UMI-like data uses portable, robotic-free operating interfaces to capture real-world representations. Typical systems integrate wrist cameras, posture tracking, claw states, and optional motion or touch sensations into handheld tools, allowing humans to operate naturally in everyday environments while recording visual and motion structures that are nearly consistent with end-user execution.
But UMI has not eliminated physical gaps. Camera perspective, end-geometry, kinetic constraints, dynamics and contact conditions still need to be addressed through labeling, inversion and redirection; visual tracking may fail in shadowy, fast-moving or low-textured environments; and data often lacks target robot-specific physical states and executor dynamics.
3.3 Ego-centric perspective data: access to the open world and hands-on experience with humans as intelligent agents
Ego-centric data comes from long-term human activity in the real world. The key value of this layer is not only "more video", but the interaction pioneers produced by humans as mature intelligences. Key points in the hand, wrist trajectories, contact sequences, selection-grabbing, tool use strategies and task decomposition have the potential to be replicated across robots. For example, human five-finger trajectories can provide crawling and approaching directions for two fingernails, and can provide higher-dimensional movements and structure contact for skillful hands-learning.
Its main contradiction is the balance between scale and marking accuracy. Video that relies solely on RGB is easy to scale, but there are problems of overshadowing, motion blurring, visual drift, and the lack of precise gestures or contact tags; wearable tracking, IMU, tactile sensing, and EMG can provide more direct geometry and interaction measurements, but increase cost, marking burden, and interference with natural operations.
Figure 5 shows the collection and monitoring of self-centered and external perspective data. Vision, motion tracking, touch, EMG and external cameras can be further transformed into semantic, geometric, polymorphic and robotic monitoring.
3.4 Simulation data: The most scalable robots are aligned, but "generating more" is not the same as "covering more".
Simulation data is the most scalable and cost-effective source of robotic alignment in the pyramid. Simulators can synchronize, repeatedly generate observations, actions, and task results, and automatically provide privileged tags such as object positions, splits, contacts, physical parameters, successful signals, and stealth rewards. Automation processes can repeatedly collect precise snippets for decomposable atomic operations, allowing models to focus on learning a skill.
The value of simulation data depends on a complete data infrastructure, including robotic and sensor models, scene and object assets, physical and rendering back-end, and collection processes such as human remote operation, trajectory playback, rule experts, game planning, reinforcement learning strategies, or big model auxiliary generation. In recent years, world models have also begun to take on the role of learning simulators for strategic training, evaluation, and synthesis of interactive data.
Its core limitations are physical fidelity. Real deployment requires strategies to master the laws of motion of the physical world, such as contact force, and the physical motion produced by force. These factors cannot be fully reproduced by existing simulation dynamics. Furthermore, observing the difference in texture, lighting, depth of noise and sensory response at the level can also further form a sim-to-real gap.
3.5 General data: the basics of semantics, space, time and reasoning for robots
General data includes images, video, language, visual language, and three-dimensional and reasoning data of a network size. Their main tasks are not to teach a robot how to move, but to build the cognitive basis that a robot needs before it learns its movements: identifying objects and attributes, understanding instructions and common sense, positioning interactive areas, understanding three-dimensional space, remembering events in videos, breaking down task steps, and judging whether physical results are logical.
Therefore, this article does not rank general data according to the name of the dataset, but according to the ability to organize it into categories such as visual language, division and positioning, three-dimensional space, planning, video timing, physics and failure reasoning, crawling, etc. Different oversight is not repetitive: language data provides semantics and common sense, location data brings concepts to regions and components, three-dimensional data provides scale and accessibility, video provides state changes and sensory memory, planning data provides target substructures, and physical and failure data helps models judge outcomes.
The advantages of general data are scale, coverage, and cross-bodied versatility, but it barely includes the state of the robot's body, the dynamics of the executor, the contact power, and the real consequences of the action. Automatically generated labels, plans, and queries may also have hallucinations, ambiguities, or physical impossibilities. Therefore, this layer is best suited to be seen as a broad perceptual and reasoning precedent, and then implemented into physical execution by higher-level robotically aligned data.
Figure 7 The ability to use general data constitutes the ability to supervise visual language, OCR, sequence, space and 3D, planning, physical reasoning, positioning and crawling, and together form the cognitive basis of a basic robot model.
4. From data classification to data formulation:
How different basic models use five layers of data
The data pyramid provides a static category organization, but the body-based model really faces the dynamic data formulation problem: which data is used for pre-training and which is used for post-training; how different levels of data are sampled, aligned and mixed; and how different operating spaces, coordinate systems and entities enter the same model.
The evolution of the paper's model theory shows that the training process is moving gradually from real-machine trajectory to a heterogeneous combination of real-machine, simulation, generalization, self-centered and UMI data.
This trend does not mean that "the more data sources the better". Some models that rely heavily on robotic data still perform strongly, while current research lacks the convergence of category levels under the Unified Computing Budget, and it is not yet possible to confirm that a fixed formula has a universal advantage.
A more reliable conclusion is that different layers of data provide complementary capabilities: general data supporting semantics and reasoning, self-centered and UMI data expansion scenarios, perspective and human behavior coverage, simulation data reinforcing controllable skills, machine data completing executable actions and real dynamic landings.
Ego-centric data is becoming a particularly significant growth direction. It is both closer to robotic perspective and object interaction than normal network video, and more easily expandable than direct collection of real-machine data, thus gradually transforming from assistive data into an important pre-training base for connecting general knowledge to robotic activity.
At the same time, some new work has begun to reassess the value of low-quality trajectories or failed trajectories, showing that future data usage may shift from simple filtering to explicit modeling quality, intent and results.
Early systems were more dependent on real-machine trajectories, more recent models gradually explored the combined training of real-machine, general, simulation, self-centered and UMI data, and expanded from VLA to a world model enhanced with a unified world action model.
4.1 Movement space alignment: a unified quantity is not equivalent to a unified semantics
Asynchronous data joint training first needs to deal with movement space differences. This article generalizes the existing methods into three categories: maintaining the native control interfaces of the various robots and sharing the main sequence representation through a specific projection or movement head; using fixed-dimensional movement vectors to zero-fill and mask missing dimensions; establishing semantic movement slots with fixed physical meanings to allow different entities to share control semantics on corresponding dimensions. The latter method provides a stronger cross-entity consistency than simple shapes.
The geometrical coordinate system constitutes a secondary coordination problem. The same end movements are given different numerical values in the robot's base, camera or wrist coordinate system. The robot center indicates that the controller is easy to execute, but that the platform installation method is easy to combine with the workspace; the camera center indicates that it helps to map visually similar movements to more consistent numerical values; the wrist center indicates that the local finger movement can be separated from the overall position of the wrist.
Regardless of the characteristics chosen, the coordinate origin, axis direction, right hand system, tool center point, absolute motion or incremental motion, rotational parameterization and physical units should be used as first-order metadata.
4.2 Data roles in three types of models are not the same
Body-brain models are primarily focused on perception, spatial and temporal reasoning, availability, memory, and high-level planning. Large amounts of non-action-labelled general multimodal data and self-centered video can help the model establish an understanding of the physical world; interactive signals in real-machine, UMI, and simulation data are often converted into available positions, position graphs, trajectory paths, motion boundaries, and subtask labels, which are used to implement cognitive abilities into physical structures.
5. Challenges and future directions:
From "expansion" to "enhanced information coverage".
The paper describes future questions as three groups of questions: "What to collect, how to collect, how to use". Together, they show that the focus of physical data research is shifting from the simple pursuit of a larger scale to improved data coverage, efficiency and efficient use of heterogeneous experiences.
Data vision for the next phase
The question of data in robotic learning is moving from "is there more trajectory" to "is the data covering critical states, whether it includes contact and failure, whether it can be replicated across the body, and whether it can be properly sampled at the right training stage". Scale is still important, but only when the quantity of information, alignment and model objectives are taken into account, is it transformed into deployability.
6. Summary
The Data Pyramid for Embodied Manipulation provides a scalable and robotically aligned data center framework that divides real-machine data, UMI data, self-centered and external viewpoint data, simulation data, and general data organization into five complementary levels, explaining the value and boundaries of each layer with quality, diversity, versatility, and physical reliability.
Based on this, the paper combines isomorphic data with body-based models, discussing how different data support perception and reasoning, motion generation and world prediction, and how motion spaces and coordinates influence cross-body joint training.
Its core conclusion is not to look for a data model that can replace other sources, but to build a data portfolio based on the capability, architecture and training phases.