When the GPT-6-Astra was released last month, OpenAI called it "the world's smartest and most responsive model to human intentions".
"Where is the physical capacity limit of GPT-6-Astra and what is the next step for the physical community?"
Recently, a team of researchers from Masashi Robotics, Ropedia, the South China University of Technology, Singapore, and the University of Tokyo published a paper titled Has GPT-6-Astra Achieved Embodied Intelligence? that covered 12 simulation assessments and systematic analysis of eight real robot platforms.
They also analyzed how the GPT-6-Astra collaborated with the underlying robotic control system to build a body-assessment prototype closer to the real needs of the home.
Address of the technical report:
Interestingly, the same GPT-6-Astra, after simply lifting the restrictions, scored or near-scored in several traditional body ratings: 100% on the LIBERO-Long++ long-range combined operating rating, 99.3% on the LIBERO-Pro, which examines robustness and generalization, 98.8% on the SIMPLE, and 81.9% on the RoboDojo.
"Put a pen in a pencil, a task that can be done in a few seconds, takes more than four hours and costs $55".
"I thought the eggs were cooked well, but I couldn't put the fire out in time, so I burned the eggs; I opened the fridge and put the milk in it, and I turned around and moved the fruit, leaving the fridge door open for a few minutes".
"The operation of inserting the plug has been cross-checked 11 times on multiple real machines, with no success at all".
Combined with high scores and errors, the apparent discrepancies reveal a key question: How far away is the ability to plan, observe, and correct a general model of action from a reliable distance?
The report concludes that the GPT-6-Astra has largely crossed the threshold of physical capability, but there is still a significant gap between the physical intelligence that can be deployed in practice. The cost and time-consuming issues exposed in the experiment, the unexpected errors, illustrate precisely where the next hardest bone to be hardened is.
It's worth noting that after the release of GPT-5 last year, the same team did a review of their spatial intelligence article titled Has GPT-5 Achieved Spatial Intelligence, which was followed by the release of the SenseNova-SI series model.
From full to less than 30 percent
What's the worst success rate?
Currently, the field of robotics operations is widely used for specially trained control strategies, as well as technical routes such as the visual-language-action model VLA. These models typically use real or simulated robot-operated data to train robots to execute seizure, movement, placement, and other actions according to visual information and mission instructions.
The emergence of the GPT-6-Astra provides another possibility for physical intelligence. The general large model can combine visual information, robot status and task requirements, directly generate operating instructions for subsequent execution and adjustment.
To systematically study this capability, SenseNova and its collaborative research team selected 12 representative assessments covering five scenarios: desktop operations, agile operations, human-robot mobile operations, home long-range tasks, and visual language navigation. In the assessment, GPT-6-Astra uses the same assessment thinking to read observed information in different environments, generate robot actions, and proceed with decisions based on execution results.
GPT-6-Astra achieved an 87.3% success rate on DexJoCo and an 81.9% success rate on RoboDojo. This indicates that GPT-6-Astra achieved or exceeded the best performance of the then-published method in most assessments.
Figure 1: Comparison of the report's 12 assessments with the results published at the time.
Behind these high scores, the research team devised two key conditions:
The first is the up and down literature study (ICL): before the assessment, GPT-6-Astra can look at the success examples allowed to be used in the training set, from which to organize task descriptions, action trajectories, visual references, auxiliary code and failure recovery methods, to form a knowledge package.
This is equivalent to allowing the model to study the operational data before entering the test site. When the robot encounters the current scenario, it can retrieve nearby demonstration movements and adjust them to the position of the object in sight. It uses mission experience, but does not learn these movements by re-training the model weight.
Figure 2: ICL training paradigm.
The second condition is to increase the number of execution steps. Traditional body measurements typically limit the maximum number of moves a robot can take, and once the upper limit is reached, the task will be deemed a failure even if only the last few moves are taken.
On the basis of a standard execution budget of 1×, the research team allows the model to use up to 10 times the number of action steps. When the robot catches failures, object deviations or operations do not meet expectations, it can continue to observe the environment, adjust the operating program and try again.
The effects of this adjustment are striking, as seen in the RoboDojo benchmark, where the success rate increased from 27.6% under the standard budget to 81.9% under the 10x budget. Bench2Dex also increased from 25.6% to 61.5%. GPT-6-Astra has a certain failure recovery capability for tasks that require repeated adjustments in positioning, hand-to-hand collaboration and touch accuracy.
Figure 4: GPT-6-Astra's mission success rate varies as the number of operational steps allowed to be performed increases.
However, this improvement has not occurred on all benchmarks. The success rate is only 37.0% on the family long-range benchmark BEHAVIOR. The success rate is only 28.9% on the long-range navigation benchmark LHPR-VLN, even with expanded execution budgets.
Even if the mistake is corrected, the worse it can be.
One of the most representative instances of failure compiled by SenseNova and its collaborative research team was that the robot needed to hang the cup on the hanger. GPT-6-Astra observed through the head camera that the cup handle appeared to have matched with the hanger, so the operation was successful. But from the other camera's point of view, the cup did not actually hang, but instead fell on the table. The model then tried to retrieve it, but the mechanical claw shutter mistakenly thought it had touched the cup, causing the subsequent operation to go wrong.
A similar situation occurs in the puzzle task. GPT-6-Astra judges that the four pieces of the puzzle have already formed the target shape, but in fact some of the positions are still not aligned. Since it is unable to identify where the problem arose, it constantly adjusts the position of the puzzle, but again confuses the parts that were close to completion.
These two examples show that even if the model is able to understand the objective of the task, it may not be able to accurately determine whether the physical operation is truly successful. In language or software tasks, many results can be verified by text, program output, or test feedback. But in robotic operations, an object appears to be placed in the correct position and does not mean that it is in a stable contact, support, alignment, or fixed position.
In the RoboDojo's reverse liquid task, the GPT-6-Astra operated by knocking the cup over and placing the liquid on the table. The model then straightened the cup and tried to continue pouring into the liquid, but the resulting part could not be recovered by a simple retest. In the DexJoCo installation task, a part slipped from a mechanical hand to the ground.
This explains why increasing the number of execution steps does not necessarily solve all the problems. If the robot only grabs a slight deviation from its position, it can re-observe and adjust the claws. But if the object is damaged, the liquid has leaked out, or the part has fallen into an untouchable position, more operational attempts are not necessarily useful.
Figure 6: (a) (b) and (c) Misjudgments of the conditions of physical success respectively, the consequences of irrevocable actions and the destruction of the state of completion in long missions.
These failures are easily overlooked. Once the robot completes a step, the environment will not always remain the same. Once the conditions are broken, subsequent tasks may be affected. This means that as the model becomes better at planning and correcting errors, verification and recovery in the process of physical execution becomes a new difficulty.
We have a stronger brain.
The cost of execution is a big problem
While some assessments were almost solved, time and Token costs were staggering. The success rate of LIBERO-Long++ has reached 97.5% under the standard budget, but each test cycle still consumes an average of about 1 million Tokens, with reasoning and tooling taking nearly an hour. On the longer LIBERO-Ultra-Long task, the average Token consumption reaches about 9 million and takes nearly two hours. In RoboDojo's expansion budget test, a single cycle takes an average of even more than three hours.
Much of this time is not spent on robotic movements, with large amounts of time spent on model reasoning, checking execution results and calling tools. The more difficult the task is, the more the cost of thinking and trying repeatedly cannot be ignored.
Figure 5: (a) Average use of tokens per task, (b) Average time spent per task.
Since increasing the number of attempts resulted in expensive time and Token expenses, Business SenseNova and its collaborative research team began exploring how to reduce unnecessary model calls to make reasoning and error correction more effective each time. For this, they turned their attention to Harness, the execution system for connecting models, environmental observation and robotic control interfaces.
On the RoboDojo benchmark, let GPT-6-Astra serve as System 2, which is responsible for reasoning and action review, let π0.5 VLA, which has been tailored to the assessment task, serve as System 1, and compare different collaborative methods.
In open-loop collaboration, GPT-6-Astra gives a sub-target, with VLA producing and executing movements with a success rate of only 5.83%. Closed-loop collaboration converts the VLA's proposed candidate movements into end-user movements, allowing GPT-6-Astra to decide whether to accept, modify or reject prior to execution. The success rate increases to 25.83% each time four candidate movements are audited.
Figure 7: On the RoboDojo benchmark, the cost-to-success ratio for different System 1/System 2 collaboration strategies and action interfaces is compared.
In addition to the collaboration between the model and the VLA, Harness needed to solve a problem: how to transform the decisions of a large model into actions that robots can execute accurately.
In experiments with GPT-6-Astra directly controlled robots, the team found that having the model directly specify the angles of the joints of the mechanical arms had a success rate of only 9.72%; changing to specify the position and gesture of the end of the mechanical arm, the success rate increased to 15.83%; further adopting relative direction and moving distance describing movements had a success rate of 18.06%.
Another factor that affects the success rate and cost is the model's call frequency. Each execution step recalls the model, although it can be adjusted in time, but the cost is higher. Continuous execution of fixed multi-step movements may also miss environmental changes. They let the model choose to execute 1 to 4 steps at a time depending on the situation. The success rate increased from 15.83% to 20.00%, while the token consumption decreased from 256 million to 114.6 million.
These experiments point to an important tradeoff in Harness design: large models participate in judging and correcting errors when necessary, while minimizing unnecessary calls as much as possible.
Eight virtual platforms
It's easy to get caught, you can walk, you can work fine, and you can hit a wall.
In order to verify the real capabilities of the GPT-6-Astra, SenseNova and its joint research team deployed it to eight real robot platforms, covering five different shapes, including Franka 3, Franka Panda, single-armed mechanical arm, two-armed Piper robot, sleek-armed jet M6S, wheeled mobile operating robot, and Yu-Gi-Oh! G1 humanoid robot.
Unlike previous simulation experiments, this set of real-world experiments did not provide a successful mission demonstration for the GPT-6-Astra, nor did they provide additional training for robots. The model received structured human-written information, including robot interfaces, link restrictions, camera positioning, security constraints, and instructions on some platforms.
The results showed that under these conditions, the same GPT-6-Astra robot was indeed able to control a series of operations in different shapes: 18 successful tests on rigid object-grabbing and placement missions, 17 times. On the Utrecht G1, the GPT-6-Astra completed two of the three mechanical arm operations through the bottom control interface, and all three turn, walk and stop missions.
However, when it comes to precision contact and long-range tasks, the success rate drops significantly. In the Franka 3 fabric folding task, four attempts were successful twice. In the three robotic platform plug-in tasks, a total of 11 attempts failed altogether. In the AgileX Tracer long-range navigation and search task, 23 tests were successful only nine times.
Figure 8: Zero sample real robot control: from simple to complex robot forms.
These results reflect that the robot's ability to grasp rigid objects does not mean that it can handle flexible object operations, precise contact and long-range autonomous tasks. But a surprising phenomenon has also emerged in real-machine experiments.
In the plowing task, the robot needs to grasp the stems and insert the flowers into the plowing bottle. When attempting to execute the single-arm Franka 3, the wrist camera can only see the stems that are grabbed by the claw and cannot see the opening of the bottle below. Due to the lack of a critical angle, the robot ultimately fails. After replacing the Piper, the situation changes. GPT-6-Astra controls one mechanical arm to grasp the stems while moving the other, allowing its wrist camera to gradually approach the opening of the plowing bottle.
This behavior demonstrates the ability to "activate vision". GPT-6-Astra, able to actively change perspective without a mission demonstration, provides a valuable example for studying general model understanding and utilizing the robot's own perceptual capabilities.
Figure 9: What GPT-6-Astra can do depends on what it can see.
The real world also exposes problems that are difficult to fully reflect in the simulation environment. First, safety. In some operations, the mechanical arm triggers the protection stop due to contact with the desktop or a collision with the joint, causing a direct interruption of the task. Second, the mechanical load.
The decision-making speed of the model is also a problem. In experiments with the Yuri G1, GPT-6-Astra spends about 90% of its running time thinking about decisions, and an operational experiment can take more than an hour.
For robot manufacturers, making accurate model judgments translate into physical movements and stability in different tasks and environments is an important test of product landing.
Physical evaluation is near completion
What's next?
With GPT-6-Astra achieving near-total scores in multiple fitness assessments, Business SenseNova and its collaborative research team began to rethink a question: How much of these high scores reflect the capabilities of robots in real life?
While physical evaluation often revolves around a clear task goal, life's tasks are often not that simple. Suppose a user tells a robot, "Help me prepare breakfast". If the user leaves after 15 minutes, preparing eggs and bread in 10 minutes may be more valuable than spending an hour making a rich breakfast. The robot also has to remember to turn off the lights, close the refrigerator door, and not mess up the kitchen because it is busy cooking.
According to some existing rules of assessment, as long as the food reaches a specified maturity, even after it is burned, it may still meet the conditions of success. They pointed to the limitations of this assessment method, for example, with the cooking task of BEHAVIOR-1K.
Figure 10: Contradiction between traditional physical evaluation and new methods of evaluation. The former mainly examines whether the task achieves its predefined objectives, the latter further considers the time taken to complete, the actual effect, and the safety requirements to be observed during the operation.
They believe that future personality assessments need to make some adjustments. Tasks should be closer to the actual needs of humans, allowing robots to decide how to do it themselves, and considering the value of the end result to the user. Testing should also not by default provide success examples, allowing robots to explore themselves in an unfamiliar environment. At the same time, the time of reasoning needs to be included in the assessment.
To validate these ideas, the research team used NVIDIA Isaac Sim to construct an interactive single-bedroom home environment, equipped with household appliances such as refrigerators, sinks, boilers, and air conditioners, simulating object contact, fluid flow, and heating processes. GPT-6-Astra uses a simulated robot AgiBot G2's camera and robot status information to observe the environment, trying to complete tasks such as preparing breakfast, organizing tables, and collecting items without a mission demonstration.
GPT-6-Astra exposes a number of problems. (Fig. 12) While cleaning the table, the mechanical arm knocks over other objects and even twists the display. When receiving food, it turns the milk into the refrigerator and treats the fruit, while the refrigerator door remains open. Faced with an unknown faucet, the robot repeatedly pushes the handle that should have been rotated, failing to turn off the water in time, causing the sink to overflow.
These family experiments are currently only small-scale explorations and do not represent the actual deployment of the robot. But from the eggs cooked and burned after cooking, to the milk put in the refrigerator and forgot to close the door, it is easy to miss out on the problems that occur during the execution process, only to see if the task is completed.
These are exactly the problems that provide clues to further improvements in robotic systems. If a complete execution record of each failure is left, researchers will have the opportunity to find where the problem is occurring and validate different improvement schemes.
Learning from Failure
Robot systems are beginning to improve themselves
Looking back at this study, we can find a pervasive phenomenon. GPT-6-Astra's performance in robotic tasks is correlated with the level of use that has already been experienced and the level of collaboration between the model and the execution system. Providing task demonstration, adjusting the operation interface, and increasing feedback checks can significantly change success rates without updating the model weight.
This paved the way for another direction of research: how to allow robots to accumulate experience from past failures and avoid making the same mistakes as much as possible in the next execution. In this direction, the company conducted a SenseNova-RoboRSI study to try to get Harness to improve mission-based feedback.
Address of the project:https://github.com/OpenSenseNova/SenseNova-RoboRSI
Specifically, developers can connect different models, robotic control tools, skill modules, and execution environments to the same operating system, and observe the entire execution process through actual task feedback. Here Harness is responsible for managing what information the model receives, when to call the tool, how to generate the action, and how to proceed after execution fails.
On this basis, Harness should also have the ability to continuously improve. For example, after a robot grabs a task that fails, the system uses trajectory and feedback to perform information analysis, adjusting the mode of action generation, planning strategy, feedback checking and memory organization, and then returning to the corresponding assessment task to verify. Effective improvements will be maintained until later versions, with the entire process without updating the base model weight, which is in line with the idea of recurrent self-evolution (RSI).
In a previously disclosed RoboDojo-Sim experiment, Mercer showed preliminary results in this direction. Compared to the GPT-6-Astra baseline, the average score of SenseNova-RoboRSI increased from 28.97 to 60.08, and the success rate increased from 22.48% to 54.17%.
In addition to RoboDojo-Sim, SenseNova-RoboRSI has also performed well in other fitness ratings. In LIBERO-Pro, its success rate is 94.50%, higher than the RPent method of the same GPT-6-Astra (92.63%). In the target kitchen environment of RoboCasa365, the combination of task memory achieves a success rate of 64.60%, higher than Paimon-0 (58.10%) and other methods.
SenseNova-RoboRSI-27B-Preview further explores the effects of the smaller scale model with Harness' combined RSI, and the initial assessment results are equally prominent.
On RoboDojo, the 27B model received an average score of 26.13 from VLA-assisted execution, 2.84 points lower than the GPT-6-Astra baseline. On LIBERO-Pro, the VLA-assisted execution and VLM direct control success rates were 95.0% and 96.1%, respectively, second only to GPT-6-Astra. On RoboCasa 365, after RSI training, the model success rate increased from 26.0% to 46.5%, the average Token output decreased by 73.3%, and the task performance and Token output efficiency doubled.
Subsequently, SenseNova-RoboRSI will continue to optimize the issues surrounding GPT-6-Astra experimental exposure, including transferring operational experience to other objects or robotic platforms, using accumulated cases to more quickly select appropriate operating strategies, reducing unnecessary large model calls, and using reasoning resources in key decision-making roles.
GPT-6-Astra demonstrates the ability of cross-robot formats to perform complex operations, with RoboRSI focusing on the self-evolution of the executive system: by analyzing failures, improving Harness, and making robots do better next time. However, this route is currently being verified mainly in a simulation environment, and whether the gains will extend to real robots is yet to be verified.
There is also a distance from real-world reliable execution. Users won't forget to turn off the fire when they accept it for breakfast because the robot runs nice.
© THE