From visual observation to strategy generation, to physical execution and completion of judgments, CodeActionBench assesses the robotic operational capabilities of the general multi-mode Agent under uniform conditions. In the two-arm simulation assessment, the GPT-6 Astra (Codex CLI) trial success rate was 73.3%, with Claude Opus 5 scoring 49.3% and 45.3% respectively in the Reference harness and Claude Code respectively.
CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
The main page of the project:
GitHub：
arXiv：
Figure 1 shows Astra completing stacking tasks by generating code by observation, then by performing feedback adjustment and withdrawal operations.
Figure 2 Astra re-observes and corrects the grabbing posture after the unsuccessful grabbing, then completes the intersection and placement of the arms.
General Multimodal models can understand images, generate code, and modify programs based on tool feedback. When these capabilities come into robotic operation, can they support a complete capture, transfer, and placement together?
Researchers and collaborators from the Southern Ocean University of Technology proposed the CodeActionBench, which built 25 two-arms simulation missions around this problem, with 675 attempts on 9 models and operating framework configurations. The success rate of each configuration varies from 2.7% to 73.3%, while the execution trajectory reveals a difference in ability behind overall scores.
"Allow General Agent to transform visual observations into operational strategies and test the process with physical results".
The agent selects visual evidence, estimates three-dimensional targets, writes and corrects procedures, evaluates at the same time the results of tasks, intermediate progress and completes judgments.
Share the interface, leave assignment decisions to Agent
How much help the interface provides directly affects the ability to evaluate. If the system has already given an object's position, to grab the candidate or complete skills, the model's task is significantly different from the task of constructing an action goal by itself from observation.
Figure 3 shows the shared robot interface and evaluation process of CodeActionBench.
CodeActionBench is based on RoboTwin 2.0 and SAPIEN and uses ALOHA-AgileX two-armed robots. It does not evaluate additional task-specific mitigation, nor does it provide task demonstration, specialized sensing or crawling module output, privileged scenarios, and default task strategies.
Shared APIs provide RGB images, camera markup, robot status, contact feedback, and geometric computation and limited motion execution. Agents independently choose to observe perspective and visual objectives, create three-dimensional estimates, and reconstruct operational positions and gestures. Reverse motion, motion planning, and low-level controls are undertaken by shared infrastructure.
"Tools are responsible for computing and executing, task decisions are left to the Agent". The Agent can call the tool directly, or run_code to write calculations, branches, cycles, and multiple actions into a Python program, and then review, modify, or adjust the actions based on actual feedback.
Both success rates and mission coverage
Each task fixes a scenario instance, each configuration tries three times on each task, for a total of 75 attempts. The task conditions, robot interfaces and resource budget are consistent. Seven configurations use the Reference harness, the other two use the Claude Code and Codex CLI.
An independent hidden verifier checks the physical results and scores do not return to the Agent. The agent completes the alternative record and verifies the actual results.
Figure 4 shows the success rate of attempts and mission coverage of nine agents deployed.
Figure 5: The top three are assigned to representative execution segments on 25 missions, prioritizing successful attempts.
Of the seven configurations of the shared Reference harness, Opus led, with Gemini 3.6 Flash coming in second with a 20.0% attempt success rate.
The success rate of the two Opus configurations differed by 4 percentage points, with the task coverage being 19/25 and 14/25 respectively.
Astra's performance demonstrates the potential of code-building strategies
Astra completed 55/75 attempts, succeeding at least once in 22 missions, 15 of which were successful all three times.
Figure 6 shows the number of successful attempts on each assignment in three configurations.
"Without additional training in VLA action heads, General Agent can also complete operational tasks with code". In this group of experiments, Agent will use existing visual, reasoning, and programming capabilities to estimate targets, construct action, and modify strategies, generate procedures during reasoning, and execute through shared robot interfaces.
How the program is organized is also related to performance. A single program's submission can combine multiple tool calls, occupying only one external call volume, and internal movements still consume simulation time. The program submits a proportion of all budgeted tool calls through run_code. The four most successful configurations are 85.6% to 92.6%, the other five are 15.6% to 35.2%. Astra has the highest and lowest code error rates, at only 0.46% (10/2,167).
Figure 7 shows the ratio of program usage to code error rates for each configuration.
"Here's the programming ability to understand robotic tools and combine them into effective operating strategies". Compared to Opus (Reference), Astra's total API equalization costs are 15.1%, single-trial average takes 60.0%, and 477.2 seconds. These comparisons are based on the same fixed scenarios, involving time-consuming model waiting and environment execution.
From "was it successful" to "what steps did you take?"
"In addition to the final success, it is also necessary to measure the progress of the execution process". CodeActionBench completes two levels of analysis of the execution trajectory from space arrival and sub-target through intermediate checkpoints: space checkpoints record whether the end of the robot enters the relevant operating area and sub-target checkpoints recording phase requirements. Both types of coverage are task-equivalent statistics.
The spatial coverage of the two Opus configurations is 73.6% and 73.4%, almost the same, with different success rates and task coverage. Relative spatial progress can correspond to different operational outcomes. Combined with object movement, release and target state, it is possible to continue checking at what stage the operation is blocked.
Sub-targets that have never been completed or subsequently failed, combined with space location, contact, object movement, and release records, fall into the earliest evidence-supported phase of the study. Of the 491 failures, 122 have reached the relevant region without completing or maintaining the corresponding sub-targets.
Figure 8 Distribution of the results of the missions in each configuration with failure phases.
"Finished", according to Agent himself.
Autonomous operation also includes deciding when to stop. Agents need to judge whether or not the task has been completed, or has not been able to continue, based on observation and execution of feedback, then actively terminate through done and report success or failure.
Figure 9 The first three assigned autonomous termination and completion judgments.
Astra made 68 correct judgments out of 75 attempts, including correctly reporting success or failure. The most distinguishing configuration is post-failure judgments: 14 of Astra's 20 failed attempts were explicitly reported, and 3 of the 38 failed attempts by Opus (Reference) were 14 and 13 respectively.
"To execute the strategy, it is also necessary to identify whether the strategy is working". This allows the evaluation to consider further whether the agent can judge the results of the operation based on feedback and decide when to stop.
CodeActionBench links task outcomes, execution progress, and completed judgments, allowing researchers to compare overall performance and how the trajectory checking strategy is formed and where it stops working. Subsequent methods can be used to further test spatial estimates, organizational and feedback utilization improvements under the same interface and protocol.
Information from the author
The authors include Yiheng Lyu (first author), Xueying Jiang, Wenhao Li, Shijian Lu (communication author), and independent researcher Gongjie Zhang (communication author, CodeActionBench project manager).
© THE