When a robot has multiple cameras, does the world it sees remain consistent?
The head camera sees a robot grabbing an object, but the wrist camera shows that the mechanical arm has not yet moved; the object in one perspective has been placed and the object in the other perspective remains in place. These seemingly subtle deviations mean that the model generated may not be the same real world.
A new problem also arose:
How do you judge whether a world model really understands the world robots live in?
Recently, Beijing University, Tsinghua University, Beijing Aerospace University, Shanghai University of Transportation, and Chinese University of Science and Technology have jointly launched the TriWorldBench Challenge, officially launching the first rating list for three-dimensional world models aimed at robots.
How to register / Address:
The list focuses on the ability to generate and understand multi-point video in a robot-operated scene head view (head view) left-wrist view (left-wrist view) right-wrist view (right-wrist view) with the aim of establishing a more comprehensive world model evaluation standard that will move the body world model from the blink of an eye to the blink of an eye.
From creating video clips to understanding the world of video clips: A model of the world with a body needs new evaluation criteria
In the past, evaluations of video-generated models have been more focused on:
If the picture is clear;
whether the content conforms to the description;
If the video is streaming continuously.
But for a model of the physical world, generating a video that looks like a real video is far from enough.
What robots need to face is a dynamic, continuous world with physical laws.
In a real robot system, a task is usually observed together by multiple cameras:
The head camera is responsible for providing comprehensive environmental information to help the robot understand the status of the mission;
The wrist cameras are close to the operating area and can capture details such as grab-and-touch.
It is not a simple repetition of information between different points of view, but rather a comprehensive understanding of the world by robots.
Therefore, a truly reliable model of the physical world requires not only the creation of a reasonable video from a single perspective, but also the assurance of:
Many cameras see the same world.
This has also become a new problem to be solved in the current world model assessment system.
TriWorldBench: A three-point assessment system for a model of the physical world
The TriWorldBench Challenge addresses the core challenge of multivariate consistency by proposing a comprehensive assessment system for robotic scenarios.
As opposed to traditional single-point video evaluation methods, TriWorldBench focuses more on whether the model is capable of understanding the real world.
01 Multipurpose Consistency: Are three cameras depicting the same world?
TriWorldBench focuses primarily on the interaction between three-way perspectives.
The video of the head, left and right wrist generated by the model should not only be reasonable, but also guaranteed:
the interaction between the active mechanical arms and the phases of action;
Keep the progress of the grasp, contact and action consistent;
There is no conflict between the category, appearance and status of the target object.
In other words, three-way videos can't just look reasonable, but they also need to share a complete, coherent process of robot operation.
02 Task Execution Ability: Did the robot really complete the task?
For robots, video generation is not the ultimate goal, but the completion of tasks is central.
TriWorldBench further evaluates the model's ability to understand robotic processes, including:
the correct execution of the order;
whether the movement of the mechanical arm is in line with the expectations;
It is also important to know whether the operation of capturing, moving, releasing, etc. is justified.
Of these, the head point of view is primarily used to judge mission instructions, trajectory of the mechanical arm and final results; the wrist point of view is used to further check whether the grip is stable, whether there is slippage, and whether the local interaction between the claws and objects is normal.
This means that the image looks clear enough, smooth enough, and doesn't mean that the robot has actually completed the task. TriWorldBench hopes to make the world model evaluate from the look of a mouse to the real use of a mouse.
03 Understanding the physical world: Do models really understand 3D space?
Robots are facing a three-dimensional physical world.
Therefore, the model of the world needs not only to generate visual content, but also to understand:
the positioning relationship between objects;
the process of change of action;
Space-related relationships from different points of view.
TriWorldBench hopes to further explore the world model's ability to understand the real environment through multiple perspective assessments.
Not just ranking: TriWorldBench wants to be a diagnostic tool to help research teams optimize models.
For a list, eliminating ranking is only the first step. More importantly, it can explain why the model scores high or low and provide a clear direction for improvement to the researcher. TriWorldBench therefore does not compress three-way videos into a simple average, but builds a complete evaluation link from synchronous data, action states, to multi-layered results.
01 From data to indicators: how triworldbench comprehensively evaluates model capabilities
The benchmark consists of 500 three-dimensional synchronous episodes covering 50 robot-operated missions. The system revolves around three-dimensional consistency, task alignment, physics and 3D consistency, motion quality, sequence consistency and visual quality in six dimensions, summarizing 19 evaluation signals, and using TWB-Score as the total score of the list.
02 How Three Perspectives Together Judge the Status of Robot Missions
The three-point alignment is placed at the center of the entire system. Each generated point of view is controlled by the corresponding true-value camera, followed by semantic judgement of the head-wrist and combined three-point queries to check the state of the mechanical arm, the stage of action, the contact relationship and the compatibility of the target object.
At the same time, the indicators will be assigned to the most reliable cameras for the evidence. The head view is better suited to judging mission instructions, overall progress, mechanical arm trajectories and final results; the wrist view is better suited to observing contact, grabbing stability, slippage and local geometry. This will neither obscure the wrist to affect overall mission judgments, nor will clear head images obscure failures near the claws.
03 STATE Let the evaluation know when to move, when to stay stable
TriWorldBench constructs STATE tags from reference robot trajectories, identifying the current stage of movement, the active mechanical arm, and whether each perspective is expected to be in motion or stationary. This moving wrist is also recognized if a long period of freezing is interrupted and excess motion occurs on the camera that should not be in motion.
04 Total score for ranking and fine particulate results for diagnosis
Graphic quality and aesthetic scores are not excluded from the task alone. When head trajectory errors, or three-way video conflicts in the state of robots and objects, the system makes task-bound corrections to visual quality, avoiding a subtle but erroneous video segment by gaining an unreasonable advantage.
In the VLM semantic evaluation, the evaluation protocol is aligned with human experts' scores and rankings, and the rules are then frozen for formal testing. In addition to reporting the TWB-Score, the list also retains six dimensions, single indicators, per-angle, head-wrist matching and combined triangle results. The research team can therefore not only see the ranking, but also determine whether the problem arises in the task, motion, image quality, or the state of the world between the three cameras.
This makes TriWorldBench more like a set of research-oriented screening reports: it not only answers who is stronger, but also helps explain where the model should be improved next.
Open enrollment for global teams to explore the next phase of a world model with bodies
The TriWorldBench Challenge is open to all relevant research teams worldwide.
Participants include, but are not limited to:
A world model research team;
A team with physical intelligence;
Video generating model teams;
Multilingual generation and understanding of teams.
Whether focusing on video generation capabilities or exploring models of robotic interaction with spatial intelligence, participants can participate in the challenge through a unified data and assessment system.
Through open listing, TriWorldBench hopes to provide a platform for fair comparison of different technology routes while promoting a more unified, scientific evaluation standard in the field of model worlds.
The robots of the future will not only need to see the world, but also to understand the world, predict the world and perform complex tasks in the real world.
The TriWorldBench Challenge aims to drive the industry together to answer one key question:
How should a truly intelligent model of the world be judged?
Looking forward to more research teams joining the TriWorldBench to explore the next phase in the development of a model of the physical world together.