To enter the home, the robot faces not only cups and boxes that can be moved as a whole. The cabinet doors, drawers, overhangs and spinners have movement joints, the robot must first find the right parts, make contact, and then complete the movement along the chain, track or rotating axis. There is something wrong with grasping the position, direction of movement or the state of the object, and the movement may stop in the middle.
Over the past few years, robotic operating models have gradually shifted from single-action to multi-tasking and long-time sequence control. The model has begun to process language instructions, spatial relationships, and the state changes brought about by movements, and the ability to extend from being unable to grab a mouse to being able to complete a thing. Doors, drawers, lids, and spins in the home have thus become a new test: robots must not only find operational parts, but also maintain contact during the movement of objects, and make movements adjust as the state changes.
This type of operation also requires more training data. Data must cover different objects and components, but also retain the joint state, contact position and results of the interaction process. Real robots are costly to collect one at a time, human samples must be mapped to robots; objects change a gesture, axis or resistance, and the original samples cannot be directly replicated.
Gamma Robotics, the Chinese Institute of Telecommunications and Artificial Intelligence (CITAI) proposed Gamma-Art, to construct a large-scale chain manipulation (Articulated Manipulation) data synthesization of Agentic links, using the ability of the Agent to organize objects, tasks, and interaction conditions into bulk-generated training samples.
01.
Overlooked welding operations
Imagine a robot opening a cabinet door. It must first find its hand in the picture, then place its claw in the appropriate position, establish contact, and finally rotate along the chain. The hand is too straight, and the door will overhang; the angle of the wrist is incorrect, and the claw will slide over the hand. When pulling, the end must move backwards along the slide; when rotating the rotor, the finger must continue to press around the center of rotation. The task instructs the claw to open the cabinet door or open the rotor, but the execution process involves a series of positions, directions and adjustments to the contact.
The body grabs the position and posture of the main concern of the object as a whole, and the progress of the task of the welding object is directly reflected in the state of the joint shaft. Whether the cabinet door is opened to the position, see the angle of opening of the chain shaft; whether the drawer is pulled to the target position, see the position in the direction of the slide; whether the rotor is completed, see the angle around the shaft. A cabinet may have a door, a drawer and a lid at the same time, and the task state must also record the axis, range of motion and current position of each part separately.
Unlike rigid operating tasks, in welding operations, the mission semantics directly determine which components the robot should contact and the operating trajectory to follow. When opening the cabinet door, the use of the handle is usually more efficient than starting from the edge of the door; when pulling the handle, it is easier to hold the hand in the direction of the slide; when rotating the rotor, it is easier to form a steady cutting motion near the center of rotation. The choice of the part is just beginning, the robot must also determine the position of the contact, the direction of the movement, and when to maintain contact. Once the contact is established, the trajectory must obey several constraints of the joint axis: the door moves along a circular arc, the slide axis moves along the slide axis, the rotor rotates around the line.
In existing operational capability verification, welding objects are often not given enough attention: only a small proportion of the tasks involved in the assessment, and the categories of objects involved are relatively limited. The model's success on a door can only indicate that it completes this object and this opening; changing the handle, axle direction or resistance, the original contact position and trajectory may fail.
In order for the model to learn to connect object operations, the data needs to cover different categories and components, joints and their states, contact locations and trajectories, as well as to record the meaning of the task, contact constraints and success conditions.
02.
Data source difficulties in training the ability to handle welded objects
These data requirements will soon come down to a real question: where the data comes from and how to maintain the constraints of welding operations while collecting on a large scale? The most direct data source is real remote operation. The operator needs to make a decision based on the image, the state of the joints and the force feedback, whether to pull, push, turn or press, then coordinate the wrist, mechanical arm and bottom, maintain end contact and avoid collision and joint boundaries. The operator controls another body, and human intuition does not directly transmit it to the robot; the trajectory is slightly tortuous, which can increase the interaction power, disrupt the contact, or allow the card to stay. After the object's position, chain linking direction, resistance, or initial shape changes, the original trajectory may also fail and fail to return to the rule.
The scale of objects and scenes will continue to drive up the cost of collection. Models and structures of attached objects are plentiful, requiring more real equipment to be purchased, transported, and maintained to cover more doors, drawers, lids, and rotors; the same object changes position, initial angle, or resistance, and may need to be retrieved. Demonstrations need to be adapted again after changes in the body, camera, and workspace of the robot.
UMI, FastUMI, and first-person data collection have reduced the reliance on targeted robotic devices. Handheld devices record the movement of a person's wrists and tools, first-person video recorders observe and interact with the process, and work such as EgoMimic, EgoDex, and EgoScale has also begun to use these data for operating model training. However, these data still lack the joint state of the target robot, clavicle cleavage and contact power, and the need to complete position estimation of the hand or tool to the end-user, re-orientation and timing of the movements. The arm length, perspective and range of activity of the person is different from that of the robot, and the trajectory after mapping may collide with the environment and may even deviate from the chain axis.
In simulation, privileged information such as joint axis, range of motion, position of components and contact status can be directly obtained, and the position of objects, robot structure, lighting, material, friction, and camera parameters can also be changed in bulk; the same tasks can be replicated and failed trajectories can also be filtered before rendering. InternData-A1 and MolmoBoT have attempted to train operational models through large-scale asset, scene, and trajectory generation, but the task of attached objects is still limited, and the functional characteristics of axis joints, continuous components, and contact operations are not specifically addressed.
Gamma-Art began by synthesizing simulated data, attempting to synthesize trajectory generation and domain randomization through the synthesis of sensed assets, component markings, trajectory generations, and synthesis of training data covering different objects, components, joints, and contact conditions, and enabling models to learn to migrate from these interactive trajectories.
03.
Construct executable operational tasks in simulation
In order to efficiently construct and validate welding object operation tasks in simulation, the team introduced Gamma-Sim: asset libraries, component level labels, constrained sensing action generation, welding sensing domains randomization, and task generation organizations in the same simulation platform.
Fund of assets
Gamma-Sim includes 2,507 welding objects with motion structures such as chain-driven, sliding track-driven, 88 functional categories, and 1,452 sets of indoor scenes and more than 10,000 4K indoor backgrounds. Assets require maintaining structural relationships between components, allowing cabinet doors to rotate, pull energy to slide, and allowing robots to find their hands in scenes, judge the direction of the chain, and estimate the range of motion.
Component-level asset labelling
Gamma-Sim uses visual language models to filter operational components from a positively-intercepted projection view, re-marking and calibrating each component's joint type, range of motion, and operation meaning. The system uses DINO-X positioning components to track the position of surface points in the object's coordinates by light, then combine alternate corrosion and two-dimensional to three-dimensional anti-projection point extraction action coordinates.
The coordinate system of action defines a unified direction:+ZThe axis indicates the direction of approach.+YThe axis represents a parallel direction, where the point of origin is the point of contact of the clavicle with the object. The parallel axis of the rotating joint is aligned with the rotating axis; the parallel axis of the moving joint is located perpendicular to the plane of the horizontal direction. The system samples the coordinates of movement around the point of contact according to the type of joint, so that the same object produces different interaction postures. The visual language model also identifies the material of the components for subsequent visual and physical randomization use.
Binding perceptual action generated
In free space, Gamma-Sim uses cuRobo for GPU-accelerated batch trajectory optimization and collision sensing programming and disruption of the trajectory. After entering the contact process, the end-executor must attach components along its joint-binding motion, so that trajectory generation explicitly uses the type of joint, the location of the joint and the effective range of motion.
Based on these constraints, Gamma-Sim defines 23 replicable skills that cover push, pull, slide, press, open, rotate, turn, rotate, and manipulate, as well as grab, place and align. Each skill has corresponding constraints and parameters, and can be broken down into multiple sub-processes to select pattern-planning or constraint-guided trajectory generation according to the interactive structure. Control points join random disruption, increasing trajectory changes, while maintaining mission semantics and contact relationships.
Randomization of the field of welding perception
Gamma-Sim randomized object positioning, robot initial configuration and camera external input. For container type attached objects, the system first transforms the container and internal objects as a whole, then randomizes the internal objects and their relative locations respectively.
Friction, density, joint stiffness and blockage affect the contact response of the moving parts, so the system randomizes the friction and density of the parts and disrupts the joint stiffness and blockage in a small range. Visual randomization is also performed by parts, from the preset library independent of sampling materials and textures, retaining the structure of the parts of the object. The system also randomizes the location, intensity, color and color temperature of the local light and the overall local light, and pasts the sample environment from the HDR environment library.
Textures of materials
Randomized display of material textures
Lighting conditions
Lighting conditions randomized display
Space configuration
Space Configuration Randomized Display
Cameras outside
Randomized display of cameras outside
Intelligent-powered welding object operation tasks generated
In order to improve the build efficiency of simulation-interconnected object-operating tasks, Gamma-Art creates intelligent objects that convert natural language tasks into simulation configurations, writing scenes, objects, robots, skill sequences, initial states, success conditions, and rendering, physics, randomization, and parallel execution parameters.
First, search the database for robots, objects and scenarios, then determine the identity of objects, initial status and position, query the operational labels of the target components, and finally, combine the skills according to the task semantics. Thus, the same skill set can be adapted to different objects, tasks and robot configurations.
Intelligent bodies also select rendering patterns based on the material of the object, transparent or reflective objects can use path tracking; the expected task length is used to adjust the number of parallel environments. Task generation starts with natural language and ends with executable simulation tasks with specific components, connectivity status and successful conditions.
04.
Building scalable data synthesis systems
A training demonstration requires the trajectory of the mechanical arm, the state of the robot, language instructions, and multiple viewpoints. Once simulation, quality screening, visual rendering, and format conversion have been completed, the sample can enter the training set.
These links use different resources. Trajectory generates the main CPU consumption, visual rendering relies on GPUs, and format conversion requires both CPU and high frequency storage I/O. Serial execution causes different stages to wait for each other.
Gamma-Art-Hive breaks down the process into three different workloads: trajectory generation, verification, visual rendering, data format conversion, and builds a distributed simulation data synthesis task scheduling system.- Trajectory generation and verification. System batch generation of trajectory variables, pre-filtering collisions, joint limitations, contact loss, oversized movements, and missions failures. Hundreds of thousands of simulation environments can run simultaneously, with failed trajectories terminating early, and successful trajectories entering the next stage.
- Visual rendering. The rendering phase only processes trajectories that have been verified, reproduces saved random seeds, and simultaneously generates multi-track camera images, and then writes observations, robot states, and actions into the intermediate data.
- Data format conversion. The CPU process reads intermediate results from shared storage, converts stream to general formats such as LeRobot, encodes multi-camera videos and aligns fields of different robots.
Shared storage connects in three stages, with the modulator responsible for breaking down long tasks, monitoring status and retesting failures. Failed samples do not take up rendering computation, and each stage can be expanded according to CPU, GPU and I/O requirements respectively. A server with 8 RTX 4090s can produce about 144 hours of robot-operated demonstration data per day. This scalable, high-efficiency data production makes it possible to synthesize large-scale simulated robot-operated data sets.
05.
Gamma-Art-Data: Operating data from multi-million-level welded objects
Gamma-Art-Data collects more than 1 million tracks of welding objects, a total duration of 4,600+ hours, about 500 million tonnes of images, and covers dozens of everyday objects such as cabinet doors, refrigerators, drawers, and spindles.
Gamma-Art-Data consists of 44 classes of tasks and 2,507 binding objects. Single-step atomic movements accounted for 69.4%, combined tasks accounted for 20.9%, and long-time tasks accounted for 9.7%. The latter two classes combined multiple operating stages, bringing the robot closer to continuous completion of a household task.
Each trajectory contains multiple viewpoints, the state of the robot's body, language instructions and sequences of movements. The object's position, lighting, material, friction, size, initial posture and camera parameters change simultaneously, and long sequence tasks also connect multiple contact stages. Gamma-Art-Data is about 2.4 times RoboCasa365 and 8 times MolmoB0T in the number of trajectories that connect objects.
06.
Do you know how to use a real machine to learn how to simulate?
In order to verify whether simulation synthesis data allows the operating model to learn the ability to manipulate movable connecting objects, Gamma-Art uses Gamma-Art-Data to train Gamma-Art-VLA. The model enters camera images and language instructions, exits the movements of the mechanical arms and claws.
In several robot simulation benchmarks, Gamma-Art-VLA training with purely synthetic data reaches or exceeds models that use some real-world data training. In tasks such as opening cabinet doors, pull pull pulling and pulling, the success rate of the model is still higher than that of the same period compared to the model. Training data covers different positions, postures and contact states, so the interaction between visual input and motion is no longer dependent on a fixed scene.
Gamma-Art-VLA was deployed on three RealMan RM75, R1Pro, and ARX AC-1 twin-armed robots to test drag, shut microwave doors, rotary valves, pressure distributors, open toolbox caps, etc. The assessment included 12 platforms and task combinations, covering 8 home operations. The overall average success rate on the three robots was 76.7%, compared to the average success rate of models using real robot data pre-training, which was 69.4%, down 7.3 percentage points.
In AC-1, the researchers further altered the plane and altitude combinations in which the task was located, and the test model performed under spatial relationships not seen in training. In this setting, the success rate of Gamma-Art-VLA decreased only slightly, while the model's success rate decreased significantly, with multiple missions failing. This showed that the simulation data synthesis phase randomized the object's position, robot posture and contact conditions, and indeed reduced the model's reliance on fixed spatial relationships.
The researchers also compared the actual performance of individual tasks using different numbers of synthetic samples. After the number of samples increased gradually, the success rate of the model continued to increase; at 500 points, the success rate of lifting the barrel handle, closing the microwave oven door, pressing the baking bread press, pulling the towel and rotating the microwave rotor in these five expansion tasks exceeded 80%.
07.
Let's move welding objects to reality
Gamma-Art provides a viable route for training robots to manipulate attached objects: it moves data production into simulation. It first generates interactive scenarios and operation tasks in batches, then uses automated labeling and skill libraries to generate executable trajectories, adding to changes in object position, lighting, material, friction and robot posture through field randomization, and ultimately forming a trainable multimodal demonstration that enables models to be deployed directly to real robots after exercising their skills in virtual environments.
Gamma-Art-VLA has completed 12 platform and mission combination tests on three two-armed robots, covering eight classes of home operations, with an overall average success rate of 76.7%.
Simulation can also continue to generate new furniture, home appliances, tasks, and contact conditions. The more data rich the models can practice repeatedly in virtual environments and then test their more mature abilities in real families. In recent studies of artificial intelligence, the idea of expanding data with simulation and moving to the virtual machine is becoming a trend.
When the diverse operating experience gained in simulation can continue to migrate to reality, robots will not be far away from the normal family.
The main page of the project:
This is not the case.
[1] J. Ao, S. Jiang, Y. Zhong, Y. Liu, Y. Gao, J. Zhao, Y. Zhang, S. Zhu, C. Bai, X. Li*, “SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining”, arXiv:2610.07652.
[2] Zhang et al., “PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations”, arXiv:2604.27472.
[3] Tian et al., “InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policy”, arXiv:2511.16651.
[4] Deshpande et al., “MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation”, arXiv:2603.16861.
The team's brief:
The Gamma-Art series of simulation platforms was launched by the team of the Chinese Telecommunications Artificial Intelligence Institute (TeleAI) and Gamma Robotics (Gamma Robotics), led by the team of leading scientists of TeleAI, the founder of Gamma Intelligent, Dr. Whitehead, the team conducted full-scale research and development around the evolution of the human-like robot brain, including general operating models, navigation models, high-dynamic brain, full-body data capabilities, and combined with the Chinese telecommunications industry scene to provide local cases.