The big model is increasingly "seeing" space: it can judge which side of an object is on the mechanical arm, which is the target of the operation, and understand how the task is done. But if the mechanical arm is actually given to it, the question becomes more specific: how many millimeters should it move?
VA-Bench tested 12 major multi-modal models. Several of the best performing models reached 100% on target identification and positioning (TL) and operational semantic understanding (MS) reached 99.6% and 100%; but the overall mission success rate was only about half: Qwen 3.8-max, Opus-5 and GPT-5.6-sol at 53.93% and 52.86% and 51.55% respectively.
If the goal is right and the task is right, why is it really moving, or is it only about half successful? In these situations, does the model really "understand" space?
This is exactly what the VA-Bench wants to test.
VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control
Link to the paper:https://arxiv.org/pdf/2609.19554
The main page of the project:https://github.com/zhangzhongbo2213/VABench
Huggingface paper:
Spatial intelligence cannot stop at "yes".
Many spatial reasoning Benchmarks have been able to evaluate depth, distance, three-dimensional relationships, and multiple viewpoints, but it is usually the evaluator who decides in advance "what the model looks at, what it answers". After actually entering the robotic environment, the model must decide for itself whether the current information is enough; if not enough, it decides what is missing, where to continue to observe.
Therefore, space intelligence that is truly geared toward physical action should not stop at answering a well-defined space question. Around this, we have proposed the VA-Bench (Vision-Action Benchmark), which puts space intelligence into a complete closed loop:
Observe → Reason → Act → Revise, which means "observe → reason → action → correction".
Figure 1: The contrast between VA-Bench and traditional spatial assessments.
The VA-Bench requires that the model actually enter the interaction process of the robot with the environment. Before the test, the model looks at the successful demonstration video and understands how the task is done, but does not obtain object coordinates, expert trajectories or precise motion parameters; in the test, it can actively switch angles, but will consume the interaction steps, and must make a trade-off between continuing observation and starting action.
Finally, the spatial judgment must really fall on the motion. The model needs to directly give specific moving distances and angles of rotation and decide when to open or close the clavicle.
Figure 2: Examples of active observation and execution processes.
Class 14 missions, 280 basic scenarios
The VA-Bench comprises 14 types of robotic operations, including 11 types of single-arms and 3 types of dual-arms, with 20 physically verified scenarios in each category for a total of 280 base scenarios.
In addition, geometric migration and long-time sequence combination tests were set up to ask further questions: would the model be able to re-judge how to change a spatial layout, would it perform a number of independent operations, or would it mean being able to stably sequence them into longer tasks?
Table 1: Set the basic tasks of the VA-Bench.
The goal is to find out what is right and why is wrong.
From the overall results, the best performing groups of models have only about half the success rate of complete tasks, but the subtask completion rate is 65.9% and 68.6% respectively. This indicates that many failures are not that the model doesn't know how to do it at all, but that it has completed many of the steps and has not been able to stabilize the entire task.
Further dismantling the model's execution process, the current difficulties are mainly concentrated in three areas: from target identification to precise spatial execution, from error detection to online correction, and from single-arm operation to two-arm coordination.
Figure 3: The main results of the VA-Bench experiment.
Finding a goal is not the same as being able to move accurately to the goal.
The first three models all reached 100% on target identification and positioning (TL) and 99.6% on operational semantic understanding (MS) but the results began to decline significantly at a more precise spatial execution stage, such as finite particle analysis (FG) before contact, and spatial relationship judgement (SR) between targets.
A model may be perfectly accurate in finding a bottle but closing in advance before the claw is lowered to the appropriate height; or it may understand the "need to put an object in the container" but not be able to determine the exact height, depth and contact position between the object and the container.
It's not about finding mistakes, it's not about coming back in time.
Qwen 3.8-max, GPT-5.6-sol and Opus-5 scored 73.6%, 70.8% and 69.9%, respectively, on the error detection of ED, but only 46.7%, 27.3% and 32.4% respectively, respectively, on the true online correction of OC. The model is increasingly able to detect "where wrong", but is not yet good at quickly translating new observations into effective corrections.
One mechanical arm is no longer easy, two more difficult.
This difficulty is further amplified as the mission moves from single-arm to double-arm collision. Qwen 3.8-max has an average success rate of 65.61% on single-arm missions, but only 11.11% on double-arm missions; Opus-5 has dropped from 64.09% to 11.67%.
Table 2: Single-arm, arm and overall mission success rates.
The difficulty of this type of task is not simply to "multi-control one mechanical arm". The model also needs to determine which mechanical arm is responsible for which object, how the relative position between the two mechanical arms should be maintained, and in what order the different movements should be joined. Space judgments at any one link may be biased, which may prevent subsequent operations from continuing.
Do you think a few points of view can solve a problem?
For this purpose, VA-Bench set up a set of contrast experiments: in one case, the model sees only the current viewpoint at a time, but can actively switch the direction of observation according to the current task; in another case, the model directly obtains five fixed viewpoints, but cannot continue to independently select a new observation position.
Table 3: Contrast the results of active observation with fixed five viewpoints.
The results were very consistent: 4 models lost the ability to actively observe and the success rate decreased entirely.
What's really important about proactive observation may not be to simply let the model "see more", but to let it know what it's missing and where to look for answers next.
Change the layout, extend the task, how much capacity is left.
Geometric migration tests keep mission objectives and operation processes unchanged, changing only the shape of objects, containers, or spatial layouts. The success rate for Qwen 3.8-max has dropped from 77.86% to 67.86%, while GPT-5.6-sol has dropped from 80.00% to 47.86%.
Knowing how to perform an operation does not mean that a model can re-establish the correct spatial relationship after entering a new spatial layout.
Table 4: Geometric migration and long sequence combined test results.
Similar problems were further amplified in the longer execution process. The VA-Bench set a long-time combination task of five objects, with Qwen 3.8-max eventually completing 61/100 object placements and GPT-5.6-sol completing 53/100.
Some models are able to complete a number of intermediate steps, but the entire task has not been completed by any model once. Doing several independent operations does not mean that the model can continue to perform steadily over long tasks.
From "understanding" to real space intelligence.
What VA-Bench sees is not a simple "would" or "wouldn't" but a more specific set of capabilities: the model can find targets but not move to them accurately; it can detect errors but not correct them in time; it can get more images but not know where to look most at the moment; it can complete several independent steps but not stably complete longer tasks.
Today's multi-modal models have increasingly "understood" the physical world, but there is still a clear gap between understanding and reliable action.
What VA-Bench wants to do is measure this distance.