From Computer-Use to Robot-Use: An interface that allows VLM to directly "play" real robots

• Show-Harness: Just a VLM Agent Can Play Robots

• Written by Yanzhe Chen*, Zechen Bai*, Zhijun Cao*, Wenzheng Zeng*, Kevin Qinghong Lin, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Qiming Huang, Mike Zheng Shou†

• Unit: Showlab of the National University of Singapore

• The main page of the project:https://showlab.github.io/Show-Harness/

• Link to the paper:https://arxiv.org/abs/2609.10522

• Open source code:https://github.com/showlab/Show-Harness

• The open source model:https://huggingface.co/showlab/Show-Harness-VLMs

• Open source data:

Now that today's Agent can see the screen, understand the task, and perform tasks in the digital world by clicking, typing, scrolling and so on, what's the worst thing about it being away from a robot in the real world?

Over the past few years, more and more efforts have been made to involve frontier models such as GPT, Claude, Gemini and others in real robot controls. There is a very direct judgment behind this: when we discuss physical intelligence, the emphasis is often placed on physical intelligence, but the core of it is still tangible intelligence. Today, the strongest intelligence, including visual understanding, spatial reasoning and task decomposition, is increasingly concentrated in frontier foundation models.

The problem is that these capabilities do not naturally turn into robot movements. Past studies have attempted different routes to take advantage of the foundation model. Traditional VLAs tend to allow models based on images and task instructions to return directly to continuous controls; other types of systems allow larger models to be responsible for planning only and physically executed by independent controllers. The former require repeated adaptation to different robots and data distributions, while the latter stretches the distance between semantic decisions and actual movements.

Researchers at the National University of Singapore's Show Lab suggested that the starting point for Show-Harness was not to train a larger robotic strategy, but to ask: If a foundation model were given a physical interface that it could understand, combine and continuously correct, would its existing intelligence go more directly into the physical world?

Figure 1: Show-Harness demonstrates real robots on different tasks, environments, robot shapes and models

Robot-Use also needs interfaces

Research related to Computer-Use provides a very intuitive reference. On computers, Agent does not need to output an underlying drive signal; it faces a clear, combineable interface such as click, type, scroll, which can be validated by observing the results. The model thus forms the closure of observe → reason → act.

Robotic control can also be reinterpreted from a similar perspective. The real challenge is not just to let the mouse model determine an action loop, but to design a layer of interface: it should both allow the VLM to map visual and spatial understanding to the intent of the action, and be sufficiently fine-grained to complete approximation, alignment, grappling, placement and correction in the real world.

As an embodied harness specifically designed for physical intelligence, the core of Show-Harness is a set of semantic action interfaces between the model and the robot control: for the model, the action is a readable, reasoned semantic unit; for the robot, the action is converted by their respective interpreters into local control commands that are bound by security boundaries.

Figure 2: Show-Harness overview. Different VLMs are connected to different robot shapes through the same semantic interface.

I'm not going to let the VLM guess the continuity of the controls.

It's not just about making decisions, it's about making decisions.

Show-Harness's motion space contains a compact set of semantic motion units. For example, models can choose to move forward, left, up and down relative to the current reference point of view, or to rotate, grab, release, or complete tasks around a specific axis.

There are two key points to this statement:

• Semantically understandable. The model does not have to directly output the node or endpoint posture values, but rather to make a decision about which direction the target should approach the node next or whether the node has already aligned the node. This clearer semantic expression is easier to understand and reason on the model.

• Physically playable. Each semantic unit corresponds to a small and distinct physical change. Different robots use their own motion interpreters to map the same motion intent as their own motion control targets.

Thus, the model-side interface does not change with the shape of the robot; when changing the body platform, the system replaces the underlying motion interpreter rather than requiring the VLM to relearn a set of articulated and coordinate-oriented motion languages. For the zero-sample frontier agent, the trans-body switching in the paper is done only by replacing the underlying robot control interpreter.

What's more, it's not a one-time output-long plan to blindly execute the puzzle. Harness describes multiple viewpoints, the state of the robot's body, the short-term history of action and the organizational cycle of the task: the model observes the current state, breaks down or updates the subtasks, makes semantic actions, and then executes and then continues to modify according to new vision and state feedback.

Figure 3:Show-Harness's perceive-reason-act closed loop. The perception, reasoning and execution modules revolve around the organization of shared semantic interfaces.

GUMI: A GUI that serves both humans and Computer-Use Agents

If the semantic action itself is readable, understandable, and operable, it should not be limited to the model. The Show-Harness team further built the GUMI (GUI-based Manipulation Interface) by presenting the same set of robot semantic action units as graphical controls and keyboard shortcuts in the browser.

This is a very natural unification:

• Human operators can play with robots remotely via a keyboard without the need for additional dedicated remote operating hardware;

• A Computer-Use agent can operate the same GUI as a web page to control a robot;

• The VLM robot agent can directly predict the same set of semantic actions without having to click through the GUI.

Each step before the operation and the semantic action selected are recorded as an observation, action data pair. Because the action interpreter retains the corresponding low-level commands and trajectories, a demonstration data can be used both to train action strategies based on Show-Harness, and can be converted into continuous control policy data; the same process can also cover single-arms, two-arms, simulations, and real machines, and allows people to intervene in the execution process of modifications by the agent.

Figure 4: GUMI translates semantic actions into a GUI. On the left is a two-armed robot control interface, and on the right is a computer-use agent operating the same interface through a browser.

In the paper, the team collected 164 pieces of remote control data on real robots through GUMI, about 7.8K decision steps: with Franka at 101 steps (5.0K) and AgileX at 63 steps (2.8K). The simulation portion collected 230 pieces of remote control data, 13.5K decision steps, covering both ManiSkill and RoboLab environments.

From a higher point of view, GUMI is more than just a data collection tool. It aligns human control, GUI agent control and VLM direct control into the same action semantics, allowing GUMI to demonstrate how the GUMI Agent operates.

The same set of interfaces, across models, across bodies, across tasks.

The goal of this set of semantic interfaces is not to bind to a specific model, but to become a common layer between the foundation model and the physical system. The paper first shows the scope of the same interface from the three dimensions of the model, the robot's body and the task.

Cross-models. The closed-source frontier VLM, Show-Harness, can be accessed from the actual robot's closing loop without micronization; small open-source VLM can be lightly adapted in the same operating space. The paper uses Qwen 3.5-2B by default, adding only rank-64 LoRA to the linear layer of the language model, freezing visual encoders and multi-modal projection layers, updating about 3% parameters.

The experiments in the paper covered 7 degrees Franka and 6 degrees Agile X (with two-arm scenarios). The model was faced with the same set of semantic action interfaces, the bottom of which was landed by the action interpreters of each platform. It should be emphasized that: the zero-shot mode switched the robot's body by replacing the bottom control interpreter; the lightweight post-training mode used demonstration data collected from the two platforms for joint training.

Inter-tasking and environment. Real-machine mission presentation extends from everyday pick-and-place to alien organisms, background/lighting/perceptions/interference changes, spatial reasoning, and hand-dragging operations. Scenes of writing erasure, hand-breaking clothes, pencil insertion, cutting cakes, etc. in the foreground also show that this set of movements abstraction can accommodate a task process that is richer than standard crawling.

Figure 5: A generalized presentation of real robots, including new objects, environmental changes, spatial reasoning, and two-armed operation.

The three layers of the real machine

Furthermore, Show-Harness assesses generalization in real-machine experiments from three dimensions: cross-task, cross-environment, and cross-body. ZS for zero-sample frontier VLM, FT for lightweight post-training Qwen 3.5-2B; average success rates for each corresponding setup.

In addition, in sim-to-real experiments, the FT model uses only 230 simulation simulation exercises collected through the same interface to complete 13/20 tasks after moving to the real Franka; the trainable VLA contrast method in the paper is 0/20 in this setting.

Of course, the success rate figures for these tasks do not mean that the robot controls have been solved. A more accurate interpretation is that, in a paper-tested operational task and distribution change, an interface that is semantically clear enough and physically fine enough to translate the existing capabilities of the frontier model into executable robot controls, or that allows smaller models to obtain stronger migration at lower cost.

Figure 6: The paper provides further analysis of precise controls, spin-offs, combinations of movements, changes in workspace, multi-arm collaboration, reasoning tasks and in-context learning.

A more difficult question is: to what extent does Show-Harness enable the foundation model to adapt to new physical and semantic needs? For this purpose, as shown above, the team examined two complementary adaptations through a series of targeted scenarios: physical adaptation, which requires no retraining strategies to respond to movement accuracy, combinations of movements, workspaces and personality changes; and semantic adaptation, which requires processing of reasoned or in-context learning tasks from new demonstrations.

Figure 7: A 2×2 control variable experiment with motion naming and physics convention.

Another group of experiments further explained the role of the interface: replacing the intuitive action name with AF, but maintaining a clear physical agreement, the success rate still reaches 95%; maintaining only the semantic name, without interpreting the agreement, can also reach 90%; when the name and agreement are removed simultaneously, the success rate decreases to 5%.

This is the first time I've ever heard of an embodied harness

It's the baseline model for entering the physical world.

From Computer-Use to Robot-Use, the change is not just about the environment: the former is about the digital objects on the screen, while the latter faces the physical world of uncertainty, contact and security constraints.

Show-Harness offers a specific attempt: not to simply compress the intelligence of the foundation model into a continuous set of controls, nor to leave it stuck in high-level planning; but to join an Embodied Harness between the model and the robot, which continues the process of observing, reasoning, semantic decision-making, interpreting and executing feedback organizations from multiple perspectives.

Perhaps the future of Computer Agent and Robot Agent will not be two completely independent systems, but a foundation model of two forms of interaction with the digital world and the physical world through different interfaces.

Read more: From Show-Harness

To the Multimodal Embodied Agent

Following Show-Harness, the Show Lab team released a more comprehensive Survey on Multimodal Embodied Agents: From Computer-Use to Robot-Use to further broaden the field of view from a single system, discussing what changes have occurred in the capacity structure and research problems as Multimodal Agents move from the digital environment to the physical world.

The review is presented in a unified framework with Perceive → Anticipate → Plan → Act → Verify (PAPAV), a multi-modal Agent, robotic system, and integrated research map; the companion Awesome Repo will also continue to capture relevant work in this rapidly evolving direction.

• Awesome Repo：

• The main page of the project:https://showlab.github.io/Awesome-Multimodal-Embodied-Agent/

The author's note:

The study was led by a team from the National University of Singapore's ShowLab.

Co-authored by Yanzhe Chen Chen, Ph.D., Zechen Bai, Ph.D., Zhijun Cao, M.S., and Wenzheng Zeng, Ph.D., both from ShowLab@NUS, with a long-term focus on multi-modal understanding and cognitive intelligence-related research.

The project is headed by Mike Zheng Shou Su-Sung, a young assistant professor at the National University of Singapore.