It's the first time in the history of the world.
AI has been able to hear, see and respond.
But in the face of ongoing change in dialogue, how does it respond while maintaining understanding and reasoning?
When you have the AI analyze the graphs on the screen and suddenly add "Wait, look at that curve on the right", the system needs to stop the old answer, update the target, and check if the background calculations are still working.
The difficulty of real-time interaction is in these moments.
A recent review by researchers at the University of Illinois at Urbana-Champaign, Carnegie Mellon University, Columbia University, and others suggests connecting full-mode intelligence and interactivity.Unified Conceptual Framework，Systemic cleaningFrom streamlined representation, model architecture, training methods to real-time deployment and evaluation of technical links, and further summarizing the key balances between intelligence and interaction and future research directions.
Address of the paper:https://doi.org/10.13140/RG.2.2.20245.69602
Address of the project:https://github.com/ShenYifanS/Awesome-Interactive-Omni
Figure 1 summarizes a total of 349 works and presents a unified conceptual framework that uses all-modal intelligence and interactive capabilities as two interconnected components: sequential representation, architecture, training, deployment and evaluation. Source: paper Figure 1.
Real-time interaction with full-mode intelligence
Omni AI (also known as Omni AI) is the pursuit of a unified understanding, reasoning, and generation of multiple modules of text, images, audio, and video. However, having multi-modular capabilities does not necessarily mean having the ability to interact naturally.
The review will beReal-time interactive full-mode intelligenceIt is divided into two interrelated components:
Omni Intelligence: focused on cross-modal understanding, reasoning and generation.
Omni Interactivity: focuses on response timing, shift management, disruption processing and dynamic adjustment.
The paper further summarizes three key characteristics: flow input, internal state of continuous evolution, and online response control. This means that models need to continuously receive information, update judgments, and dynamically determine when to respond, wait, or modify outputs.
Image 2 Full-duplex interaction allows simultaneous listening and requires processing of addition, overlap and interruption. "Native" further emphasizes that timing, incremental processing and interaction control are incorporated into the model architecture and training itself. Simply reducing response delays is not enough to cover these requirements. Source: paper Figure 2.
Two technical routes
Within the Unified Conceptual Framework, the overview describes two main technological pathways.
One is adaptation on the basis of existing models. Through flow-processing, modular modules, post-training and external sorting, the real-time interaction capability of pre-trained language models or full-modular models is increased. The advantage is the ability to replicate already, and the challenge lies in delay, state synchronization and disruption control.
The second is to build a native interaction model. Continuous multi-modal processing, time structure and interaction controls will be integrated into model architecture and training to achieve closer perception and interaction synergy, but higher requirements for training data and capability retention.
When do you open your mouth and how do you deal with interruptions?
Real-time interaction requires models to accurately determine when to respond, wait, and how to deal with interruptions.
Around these issues, the comprehensive system addresses response time control, turn management, and disruption recovery mechanisms, and analyzes how existing methods combine voice, vision, semantics, and task states to make dynamic interaction decisions.
At the training level, the article summarizes how to use interactive signals such as pause, attachment, voice overlap, correction and silence to teach the model not only what to answer, but also when to answer.
The review further summarizes six key elements for the recovery of interaction after disruption:
Detect: Identify whether a user has actually initiated a disruption and distinguish between effective instructions and background sounds.
Yield: Stop or reduce current voice output in a timely manner.
Rollback: To align what users actually hear.
Cancel: Cancel outdated tasks.
Revise: Updated intentions and tasks in accordance with the new instructions.
Resume: Continued interaction based on updated status.
Consistent with intelligence and interaction
In the face of complex reasoning and tool calls, early responses to models can lead to a lack of information, while waiting for the computation to be completed can affect the interactive experience.
Interactivity Trade-off: The training phase focuses on the impact of interaction adaptation on original capabilities, while the running phase requires a balance between response speed and response quality.
Figure 3: Balance between intelligence and interaction: the focus at the training stage is on the ability to retain, the focus at the running stage is on response times and the quality of task completion. Source: paper Figure 14.
In order to address this balance, the review discussed further.Multi-Rate Interactive Intelligence (MIRI) is a multi-level interactive intelligence (MIRI) that can be used in a variety of ways.By coordinating rapid response with slower reasoning, retrieval and tool calling, while simultaneously interacting in real-time with complex task processing capabilities.
One way to implement the idea is the front-end collaboration, the front-end responsible for continuous perception and rapid response, the back-end asynchronously performs complex reasoning and tooling tasks, and avoids using outdated computational results through state synchronization.
Figure 4: The front end maintains low latency interaction, while the rear end performs asymmetrical retrieval, tool calling, reasoning, and memory operations. Source: paper Figure 6.
Evaluation and prospects
For the existing evaluation of the difficulty of simultaneously measuring responses to quality and interaction behaviour, a summary of six dimensional systems from latency, timing quality, content quality, robustness, session quality, and repairability compiles existing evaluation methods, emphasizing the need for comparable interaction scenarios and resource budgets, while measuring intelligence and interaction performance.
The review further summarizes four future directions worth looking at:
Multi-time scale synergy: coordination of perception, reasoning and interaction under different time scales.
Continuous status and long-term memory: Maintaining mission status and memory in long-term interaction.
Active and social interaction: Judging the timing of active responses, understanding roles and powers in multiple interactions.
Safety and efficiency in an open environment: with safety, computing costs and system reliability combined.
From "hearing, seeing, speaking", to continuous understanding, timely response and dynamic adjustment of actions, it is an important research direction for the next phase of real-time interactive full-mode intelligence.