In the past, the basic model was primarily responsible for understanding and describing the physical world. Now, the boundary is moving: the model begins to enter the robotic control box, participating in deciding what a body will do next.
Recently, Google DeepMind has added a layer of physical reasoning to the full-body control model, while Anthropic brings the front-end model to different levels of robotic control interfaces. The two routes are implemented differently, but present similar system assignments: the basic model is responsible for high-level decision-making, and the pre-trained controller is responsible for balancing, tracking, and execution of joints.
This also raises an unanswered question. When a low-level controller is able to reliably perform movements, does a visual language model really know what the body needs to do next? It needs to not only see the target, but also continuously judge when to explore, how to approach, where to stop, and when to complete the interaction from the first point of view.
Human CLAW: Can Vision-Language Models Act Through a Body?
It's the first time I've seen this.
The code is fully open source:https://github.com/Human-CLAW/HumanCLAW
Researchers from Meta, South Pacific University of Technology, Washington University, Brown University, and Northwestern University have proposed HumanCLAW, which separates upper action selection from lower motion control, while maintaining continuous motion, collision, contact, and gravity. Nine front-end VLMs underwent the same set of tests, with the best models achieving complete interaction with a success rate of "16.8%".
Figure 1: Where action intelligence is located (Figure 1)
Understanding the world is not the same as being able to act physically.
HumanCLAW calls this ability "Action Intelligence": the model needs to continuously determine what the body should do next based on current observations and the results of previous actions. It studies "what to do" rather than "how to do it" specifically for each joint.
Figure 2: The task of climbing the stairs: deciding once every 0.5 seconds
This distinction is important. Scripting environments typically take action directly abstractly as discrete commands, so there is no real physical consequence; real robots and end-to-end vision, fluent language, fluent action models, and VLAs combine decision-making, balance, and joint controls. When robots fail, it's difficult to tell whether the model chooses the wrong action or the controller doesn't execute the action properly.
The evaluation principle of HumanCLAW is therefore clear: "retain the physical consequences of each decision while stripping the lower levels of control errors from the results". Only in this way can the operational decision-making capacity of the basic model itself be measured separately.
How can action decisions be measured separately from movement control?
HumanCLAW organizes long-range tasks into a closed loop of observation, decision making, observation, action, and feedback. The model makes one decision every 0.5 seconds, with the results of the action moving immediately to the next round of observation. The entire framework consists of three parts:
Figure 3: HumanCLAW closed-loop framework
1. "VLM decision": A frozen existing VLM, starting from the first point of view, can only select one of the atomic skills with parameters forwards, backwards, and downwards at each step. Without a quick instruction such as "Sit on the couch", looking for, approaching, adjusting towards, it must disintegrate itself.
2. "Continuous motion generation": The motion generator converts atomic skills into continuous full-body movements; the new skills require access to lightweight adapters and no need to re-train the motion model or VLM.
Figure 4: Skill Movement Generation (spread predecessor + one lightweight ControlNet adapter for each skill)
3. "Semi-physical execution": Physical movements are reliably completed, never falling, but the walls get stuck in the way, the feet hit the furniture, the objects are touched, and the model has to face the new situation it has just created.
From the model's perspective, it's like a first-person game: control keys are not up and down, but whole-body skills; after each selection, both the body and the environment really change.
The real challenge begins when you see the goal.
HumanCLAW-Bench constructed 1,218 long-range missions in 41 indoor scenarios. Each task had to be completed in three stages: first finding a target, then bringing the body closer, and finally completing a real interaction.
Figure 5: 3D labeling of task difficulty
In addition to the success rate, the benchmark also records process indicators such as collisions and disruptions, which not only judge success, but also track specific failures.
The nine front-end VLMs were tested under exactly the same body, tasks, and indicators, and the decision-making model was completely frozen, with no task-oriented training. The best performing models had a success rate of "64.9% → 42.5% → 16.8%" in three phases; four of the nine models had a complete interaction success rate of no more than 0.2%.
Figure 6: Main result: 64.9% → 42.5% → 16.8% ((FindSR: Confirmed discovery; NavSR: Stopping within close to 20 cm; InteractSR: Sitting to form contact)
Loss of capability occurs mainly after a plan has been developed into dozens or even hundreds of decisions in succession: models are often able to identify targets and give reasonable plans, but keep falling behind in gradual execution. Between "understanding a scenario" and "continuing to change the scenario through the body", they remain at a level of immature closed-loop decision-making ability.
The model sees the environment but continues to lose its state.
Misunderstandings have further broken the gap:
1. "Identification is usually not a problem once the goal is in sight". A more common failure to find the stage is inefficient exploration: the model does not bring the goal into sight at all.
2. "After finding the target, the body still doesn't get there". 68% of these tracks fail to complete navigation, most of which are related to first-person self-space judgments: not knowing how far away from the target or when to stop.
3. "After contact with the target, interaction is not necessarily complete". The success rate of different models completing the sitting varies from 90% to 3.5%.
Figure 7: Analysis of failure patterns
The collision distribution further confirms this: the legs collide at 28 per cent to 45 per cent of their movements, compared to less than 7 per cent of the head near the centre of vision. The model can hit areas where it is difficult to see, and even after correctly describing the position of the wall, the next step is still to push the legs into the wall.
Figure 8: Distribution of physical collisions
These failures together point to a more basic gap: "embodied self-awareness". The current VLM, which describes the world to come, is unable to sustain online estimates of where, to what, and just what the body has touched. The problem is that the model does not see its own body and the consequences of its actions as a constantly updated state.
Figure 9: Typical failure: Sitting in the air
Structured reasoning is more important than long history
The dilution experiment gives a concrete clue: the closed-loop operation does not require a longer context, but a structured reasoning process.
1. Close structured memory, mid-level targets and short context verifiers will improve results. Eliminate mid-level targets, and the overall interaction success rate decreases from 18.9% to zero; eliminate verifiers, and the navigation success rate decreases from 27% to 2%.
The key is not to remember more, but to organize information around the state of the body: what has just been done, what has happened to the world, and what needs to be confirmed most now.
Figure 10: Melting: Structural reasoning vs. Longer history
This also points to the bigger question behind HumanCLAW: Will action intelligence grow alongside the general reasoning capabilities of the basic model?
Will action intelligence go along with the basic model?
HumanCLAW is concerned not only with whether today's models can complete a set of indoor tasks, but also with the longer-term question of whether these advances can naturally be transformed into operational capabilities when the basic model acquires a stronger visual understanding and reasoning capability. Continuous comparison of different basic models in the same set of assessments has given a remarkable signal: "The performance of the strongest open source model is close to the best closed source model".
As a starting point, the future can move forward in three directions:
Expand the body shape, skills, object interaction, and task length, and test the current findings to see if it can be promoted to a more open environment.
From simulation to real robots: how much of the decision-making capability of the study model can be preserved by bringing high-level decision-making access to learned controllers, reintroducing hardware noise, controlling errors and real contact.
Forming a continuous estimate of one's own body: introducing one's own state and contact signals, and training against one's own state, further studying the perception of oneself from observation, memory, or specialized learning mechanisms.
If actionable intelligence could grow alongside basic models, the advancement of body systems could no longer be entirely dependent on the collection of a large number of instructional, training special strategies for each task; the role of the basic model would also change: from an observer of the physical world to a decision-maker who can act through the body in it.
HumanCLAW provides a way for the answers to be continuously observed: if general reasoning can migrate to action, the benchmark records this change; if not, it can also help determine what is lacking. The "basic model can become a decision-making model with physical intelligence" is therefore no longer just a vision, but a question of research that can be repeatedly tested with model development.