I see, I touch, I understand.
Human beings know an object and never just "see" it. When we see a box, we may pick it up and touch it and judge how much it weighs; we may shake and hear if there is anything inside; or we may pinch and try to make it soft. The process of perceiving the world is not to passively accept existing sensory information, but to actively interact with it when we discover that we don't understand the things around us enough, complementing the physical understanding of the whole world based on the information that interacts.
ROMA (Real-World Object-Centric Multi-Sensory Active Perception), proposed by researchers from institutions such as the GeWu-Lab, the Institute for Intelligent Resources, and AresoX, aims to empower multi-sensory robots with the ability to perceive the physical world in a proactive way. This work is based entirely on data sets with Benchmarks, multi-modal reasoning models and physical interaction systems, with the goal of paving the way for the next generation of proactive agents.
Figure 1 is the real-world multi-mode active perception system ROMA.
The title of the paper is ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception.
Address of the paper:https://arxiv.org/abs/2610.06955
The main page of the project:https://gewu-lab.github.io/ROMA/
Data sets and models:https://huggingface.co/datasets/GeWuLab/ROMA
Open source code:https://github.com/GeWu-Lab/ROMA
If the current information is not enough to complete the task,
What should robots do?
Imagine a scene where humans and robots say, "Help me find a cup of tea". The robot needs to first understand that human intent is to find an empty cup, and then find that empty cup from a variety of possible candidate objects. It may need to be shaken, touched, and touched to determine which cup is empty.
ROMA revolves around this critical issue, offering a complete set of solutions from data sets and benchmarks to reasoning models and physical systems. It integrates the perception of the four most commonly used visual-to-audible models into the ROMA-7B model and teaches the robot to judge for itself what information is missing, determine which objects interact with it, how to interact with it, focus on which models are sufficient to complete tasks, and smoothly execute these interactions to obtain multi-modal feedback, forming a complete reasoning-interaction-feedback cycle.
ROMI-2K: Nearly 2,000 real objects, and it's the first of its kind in the world.
Multi-sensory interaction data
In order for the Robotics Society to interact and explore proactively, a core challenge is to understand what the different physical interactions bring. For this, the team constructed a large real-world multi-modal object interaction dataset ROMI-2K, covering nearly 2,000 everyday objects and combinations of objects, and detailed physical attributes such as 6 basic physical interactions (lifting, pressing, colliding, pinching, shaking, and rotating simultaneously) around handheld devices and real-world desktop mechanical arm scenes. The interaction of each object synchronized collected observational information from four models: visual, auditory, tactile, and power, detailing the physical properties of objects, as well as evidence, roughness, texture, rigidity, and content.
Figure two: the real-world multi-modal object interaction data set ROMI-2K.
ROMA Bench: From Interaction to the Perception Chain
In the process of active perception, the modular observations obtained at one interaction will affect the choice of the next interaction. For example, by shaking to determine whether the object is empty when the object emits a clear inner-content sound, there is no longer any need to interact. Without emitting a sound, it cannot be ruled out that the inner-content fills the object's interior and needs to be re-touched by a single weight to confirm. This forms a continuous chain of perception created by reasoning-interaction-feedback. Based on this important concept of the sensor chain, the team further constructed the ROMA Bench on the real desktop suite of ROMI-2K, which contains 2,100 scene-level active perception tasks, covering: 1) Single-Chain: single-attributed induction; and 2) M-Culti-attributed induction; 3) multi-attributed induction; 3) complexity-attributed induction; 3) complexity-attributed induction; 3) complexity-attributed induction; and complex-attributed selection.
Figure 3. ROMA Bench is an example of a multi-modal active perception task.
ROMA system: ROMA-7B interacts with physical interfaces that actively determine interaction
The ROMA Bench provides an opportunity to explore whether models of robotic drivers "know how to explore", while the ROMA system further introduces this capability into the physical world. The research team built a real-world multi-modal physical interaction system that connects the four sensory models of vision, hearing, touch, and power to the mechanical arm, and designed a unified physical interaction interface that enables the model to directly output interaction instructions to real objects.
Specifically, ROMA will program the six basic interactions in ROMI-2K, and use AnyGrasp-based crawling strategies to combine well-designed point-to-point filtering, crawling and enhancement mechanisms, enhance real-world stable crawling capabilities, and reliably translate interactive decision making into real-world physical actions.
Based on this, the research team, based on Qwen 2.5-Omni, trained the ROMA-7B Active Perception model through multi-sensor combined alignment and active sensing scenarios. It combines current mission objectives with existing observational information, actively decides which objects to grab, what interactions to execute, what patterns to use, and when to stop exploring, and dynamically adjusts subsequent interactions based on new data acquired from each interaction, forming a "reasoning-interaction-feedback-reasoning" closure that enables robots to actively improve their understanding of the physical world by interacting with the world.
Figure 4. ROMA is a real-world, multi-modal object perception system diagram.
Multimodal active perception capabilities overall flatten the GPT-6 Astra, leading the way in multiple front-end models
The ROMA-7B achieved an overall task success rate of 72.9% on the 2100 path multiple choice, judgement, and sequencing of ROMA Bench's multi-modal proactivity perception. The ROMA-7B performed essentially on par with the latest GPT-6 Astra and significantly outperformed the GPT-5.4 and Gemini 3.5 Flash. The GPT-6 Astra demonstrated stronger performance in single-step, visually deductible properties with stronger visual understanding and general reasoning capabilities, while the ROMA-7B demonstrated stronger advantages in continuous interaction, integrated multi-modal feedback, especially in audio-based Multi-Chain tasks, and significantly outperformed the GPT-6 Astra.
In contrast, Qwen 2.5-Omni and Qwen 3-Omni, which are relatively small in parameters and are not ROMI-2K-threatened, are significantly inadequate in terms of target selection, interaction instruction following and multi-modal dynamic reasoning when faced with the ROMA Bench. This also reflects the core value of the ROMI-2K dataset: Synchronous vision, audio, touch and power data when interacting with large-scale real objects, providing a critical training basis for learning multi-modal dynamic sensing capabilities, enabling the ROMA-7B to achieve multi-modal dynamic sensing capabilities that are close to those of the GPT-6 Astra at only a 7B model scale, and outperforming other cutting-ed closed source models.
Table 1 -- ROMA Bench Multimodal Active Perception Testing
In eight multi-object scenarios in real-world experiments, the ROMA-7B still achieved an overall success rate of 61.4%, close to GPT-6 Astra and still significantly ahead of other leading-edge models, demonstrating that learning about active perception through ROMI-2K can be maintained in real-world scenarios rather than confined to the fixed form of tasks in benchmark tests.
Tables 2 and 2 are real-world multimodal active perception measurements.
Figure 5: The Single-Chain mission completes the presentation of the ROMA, fully automated, 2.5 times faster.
Figure 6. The Multi-Chain mission completes the demonstration -- fully automatic ROMA, 2.5 times faster.
Figure 7. The Intent-Driven mission completes the demonstration of the ROMA, fully automated, 2.5 times faster.
I'm from Rome.
Moving towards more proactive multi-modal AI
ROMA's current research focuses on six basic physical interactions, but this is only a starting point. As a fundamental systematic work, ROMA can further inspire and explore the future: more interactive Skill, the unification of active perception and manipulation strategies, multi-phase dialogue with long-time sequence reasoning, persistent object memory across scenarios, active perception towards different robotic forms, etc. ROMA hopes that robots can also interact with the world and truly understand the world through one-off exploration.