Do you remember Samantha from the movie "Her Mummy"?
In 2013, director Spike Jonze portrayed a fascinating future in this film: an AI operating system without an entity that can act like a real person, and the protagonist, Theodore, speaks voluptuously.
She would make jokes, be proactive in opening the conversation, be aware of her husband's feelings and intentions, and be naturally conversational. What was even more amazing was that she could not only chat, but also communicate, process emails, arrange schedules, and participate in Theodore's daily life.
More than a decade ago, we finally seem to be one step closer to being away from AI.
Over the past few years, we've seen advances in AI speech technology. From voice recognition (ASR) to voice synthesis (TTS), models can not only hear what people are saying, but also generate more and more natural, expressive sounds.
But if you really think of her as Samantha to talk a few words, the problem will soon be exposed, because she doesn't always know when to speak and when to shut up.
You haven't finished, and it may be in a hurry to respond; you're just saying "oh", and it thinks you want to interrupt; you're halfway through the conversation and it's helping to find out something, and the conversation that's been going smoothly can get stuck waiting.
After all, AI is becoming more and more "talking" like humans, but it's not very good at "conversing" with people in real time and naturally.
This is not only a question of speech recognition and generation, but also whether AI can understand meaning in continuous communication, judge the rhythm of the conversation, manage context and handle ongoing tasks.
Just today, MOSS-Live, a full-featured real-time interaction system released by MindSmart, is trying to go further in this direction.
MOSS-Live is able to listen continuously while speaking, judging when to respond and when to wait based on its semantic understanding. Even if it interrupts and switches the topic several times in the middle, it can remember where the conversation was before.
Furthermore, MOSS-Live is capable of real-time video and screen sharing, combined with imagery to understand the dialogue scene; calls can be executed asynchronously with the background through tools, and it can query information, plan routes, complete complex tasks while maintaining the dialogue, and actively feedback when the results return.
According to the official Benchmark, MOSS-Live is the leading real-time voice system such as GPT Realtime 2.1, GPT-Live-1, Doubao SeedDuplex on multiple real-time voice interaction indicators.
Currently, MOSS-Live is open for online experience and API access, and provides free access to the API for a limited time.
API access and experience:
This is the first time I've heard of this.
So how does MOSS-Live do that? What kind of technical architecture is needed to allow AI to chat while handling complex tasks? How does MOSS-Live actually perform in real-world scenarios like multi-person meetings, driving, traveling, cross-lingual real-time communication?
01 How to realise real full-time work?
In order to understand the changes in MOSS-Live, we first need to figure out what is full-fledged work.
In the field of communication, dual work means both parties can send and receive messages at the same time. When making a call, two people can open their mouths at the same time; using conventional speakers, it is usually necessary to speak in turn. Today, there are many ways of interacting with voice AI, still closer to the latter.
The system may mechanically judge whether a user has finished a speech, and then identify, understand, and respond. When the user says half a pause to think, the AI may mistakenly think that a sentence has ended; the user may add a word when the AI answers, and the system may use it as an interruption instruction.
What MOSS-Live wants to accomplish is obviously more than just allowing AI to receive voices while speaking. The real difficulty is getting the system to understand exactly what those voices mean in the current conversation.
MOSS-Live needs to continually combine semantics and conversational states to make judgments: when to continue speaking, when to remain silent, when to accept interruption, and which voices do not need to respond.
Being able to communicate naturally is only half the problem. Some conversations may involve external tool calls, or multiple rounds of searching and reasoning. If the AI has to wait for all queries to end before continuing to talk, it's easy to get caught up in a smooth conversation. But how do you ensure that the two don't interfere with each other if you let it communicate and perform tasks at the same time?
The MOSS-Live program is a set of front-end collaboration architectures.
Among them, the front-end interaction model is responsible for real-time perception and dialogue. It continues to receive speech, images, and surrounding sounds, tracking the current state of the conversation, deciding when to listen, respond, wait, or accept interruption. The back-end model is responsible for the execution of complex reasoning and long-range tasks. When users make requests for deeper thinking or long-range complex tasks, the front-end can assign tasks along with the relevant context to the back-end asynchronous processing, without having to suspend the current dialog waiting for the results.
In addition, MOSS-Live can also enable proactive interaction according to user-defined rules. For example, users can ask the AI to correct errors in reciting in a timely manner, or to observe specific matters during a meeting discussion, and actively alert them after conditions are met, without having to wait for explicit instructions each time.
02 What is the real performance of MOSS-Live, which is leading the way in more than two industry indicators?
In the past, AI assessments focused on accuracy, quality of voice generation, response speed, and whether the model can correctly answer a question.
But when AI starts engaging in continuous conversations, these metrics are clearly not enough. Real-time voice interaction systems need to face a new set of evaluation dimensions: semantic shutdown, disrupted processing, unrelated voice rejection, context recovery, and responsiveness and reliability in task execution.
So what is the actual performance of MOSS-Live on these capabilities?
MOSS-Live and GPT Realtime 2.1, GPT-Live-1, Doubao SeedDuplex, Qwen Audio 3.0 Plus/Flash have been released by Intel, covering a wide range of dimensions including interaction, voice understanding, multi-round conversations, and tool calling.
Let's take a look at the full-fledged capabilities that have the most direct impact on the conversation experience.
In the FDB v1 test, the stop-gap rate for MOSS-Live was 1.44%, significantly lower than 3.83% for GPT Realtime 2.1, 10.37% for GPT-Live-1 and 14.29% for Doubao SeedDuplex.
In other words, a normal pause occurs when a user speaks, and it is easier for MOSS-Live to assume that the other person has finished the expression, thus making a call in advance.
In the unrelated voice misinterpretation test of FDB v1.5, the MOSS-Live misinterpretation rate was only 2.00%; in the same test, this indicator ranged from 68% to 88% for other commentary systems.
A similar advantage was seen in the τ-Voice anti-interference test.
In the face of user-generated synchronization, MOSS-Live's synchronization and anti-interference rate is 98.17%; in the oral Zen and non-speech interference tests, its anti-interference rate is 82.16% and 78.90%, respectively, ranked first in the rating system.
MOSS-Live has also shown some competitiveness in the assessment of voice understanding and multilingual dialogue.
In the above context welding test of FDB v1, the quality of the response received 4.88 points (5 points) after the break, ranking first in the rating system; in the scenario handling test of FDB v2, the combined score for entity tracking, error correction and security was 3.63 points (5 points), also achieving the highest score.
In addition to this, tool calling is also an important ability to measure whether a real-time voice intelligent body can actually "chatting alongside".
In the FDB v3 tool call rating, MOSS-Live has a success rate of 51%, with Doubao SeedDuplex ranking highest, while in the τ-Voice multitasking rating, MOSS-Live has a success rate of 57.93%, second only to GPT-Live-1.
From these indicators, it can be seen that the competition for real-time voice AI is no longer limited to whether a single answer is accurate, but has begun to extend to whether the entire conversation process can be kept natural, continuous, and tasks can be completed reliably.
So what experience does it bring when these abilities are put into specific life and work scenarios?
03 When AI really begins to integrate into real life
In order to more intuitively demonstrate the interactive capabilities of MOSS-Live, MindSmart has also released several demos involving multi-person meetings, ride-hailing assistants, real-time video guides and cross-lingual communication.
In an event preparation meeting, MOSS-Live participates in a number of discussions as an AI participant. It not only distinguishes between casual attachment and actual disruption, but it also proactively inquires about the progress of the discussion, the weather in Shanghai, Hangzhou and large events during the same period, joins the discussion at the right time, and at the end of the meeting organizes the conclusions and work to be done.
In another driving scene presentation, the user lets MOSS-Live plan a two-day self-driving trip, changing preferences in the middle and temporarily, intercepting questions about boat tickets, parking fees and accommodation options. MOSS-Live continues to consult the hotel in the backseat according to the new requirements.
MOSS-Live also provides direct access to more environmental information through video calls, screen sharing, etc. In another Shanghai tourism demo, a foreign visitor communicates with MOSS-Live via a mobile phone. AI first plans a tour route through the lobby, along Nanjing Road, based on the visitor's location. When the visitor reaches the lobby, the visitor picks up the mobile phone and asks what the building is in front of him.
In addition, MOSS-Live also supports streamlined synophonic interpretation. While the system can continue to receive speech in the source language and output the target language in real time, cross-language communication does not always have to wait for the other to complete the entire expression, but its actual latency performance will still be affected by factors such as networking.
Of course, from demonstration to large-scale practical use, further testing of the system's stability in complex noise, long-term continuous interaction, multiple speakers, etc. is required.
Looking back on developments in AI interaction in recent years, voice AI is attempting to shift to real-time intelligent entities that can continue to engage in communication, understand scenarios, and perform tasks.
For developers, this change also means that the next generation of voice applications may change the way they develop. In the past, developers may have only needed to synthesize ASR, Big Language Models, and TTS to build a basic voice Q&A system.
But for AI to actually participate in a meeting, take on the role of a real-time assistant, or continue to perform tasks in a video call, it also needs to handle a range of issues such as interruptions, concurrent tasks, context synchronization, multi-module input and executive status management.
Currently, MOSS-Live is open to online experience and API access, and offers a limited amount of free API access. Developers can further explore its applications for real-time voice assistants, conference collaboration, multilingual communication, and multi-modal smart bodies.
API access and experience:
This is the first time I've heard of this.
It is worth noting that in addition to MOSS-Live, MOSS intelligence has also been active in recent years.
In mid-October, Intel announced the completion of a more than 1 billion yuan Pre-A round financed by the China Control Fund. Two weeks ago, the company had just launched the 5B-parameters image generation model MOSS-Image, focusing on exploring more accurate word rendering and layout controls, an explosion in the community of in-house developers and AI creators, and over 3 million downloads of the Hugging Face model; its new generation MOSS-TTS 2.0 is also entering the Preview internal testing phase, which will officially go online, further exploring voice cloning, emotional control, and other capabilities, leading to more natural, expressive, and controllable voice generation.
On the one hand, the focus is on how images are generated and controlled, on the other hand, the exploration of how sounds are expressed more naturally, while MOSS-Live will allow AI to continue to perceive, communicate and act in real environments. Although they are oriented to different tasks, when they are put together, we can see that mind-minded intelligence is interacting around vision, speech and real-time, gradually expanding its multi-modal modeling capabilities, and promoting the development and capacity building of full-modal real-time interactive intelligence.
Looking at the industry as a whole, this is perhaps the next step in the development of AI: as the generation and understanding of a single model matures, how to further synchronize these capabilities, supporting the natural, coherent, intelligent interaction of chat and task execution, will be the focus of the next stage.
© THE