It's the first time in the history of the world.
When humans say "I'm not a wolf", the multicolored model can both see faces and hear voices. But does it really know that it's true?
Recently, a research team from the University of Tokyo has proposed the Multimodal Interactive Deception Assessment (MIDA) benchmark for multi-modal interactive deception assessment, which tests the ability of open-source and closed-source models such as GPT, Gemini, Claude, Qwen, DeepSeek to judge deception in multiple social interactions.
The results show that even large multimodal models with stronger current capabilities find it difficult to tell the truth from lies in a reliable region. They can understand images, sounds and text without necessarily figuring out what a person at the time "knew, believed, and tried to make others believe".
The relevant research has been received by the EMNLP 2026 Main Conference.
Link to the paper: https://arxiv.org/abs/2511.16221
The main page of the project: https://ut-vision-social.github.io/
Code address: https://github.com/kkkcx/Can-MLLMs-Read-the-Room
Address of data collection: https://huggingface.co/datasets/Ukpkmkkk/MIDA
The authors of the paper, Kang Kang Xin, Huang Lifei, Yuan Yan, Zhang Yimou, Liu Sheng, and Soto Yang, are all from the University of Tokyo. The team focused on multimodal understanding, video analysis, and social intelligence, and the study focused primarily on fraudulent judgement and mental reasoning in complex multi-person interactions.
Figure 1: MIDA's overall framework. On the left is a process for building game records, rules and labels based on artificial markup; in the middle is a strategy classification and cheating judgement task; on the right is an evaluation model and two reasoning modules for SoCoT, DSEM.
Killed by a werewolfA standard answer to "lies".
Previous fraud detection studies have largely revolved around single-person statements or two-person conversations. However, lies in reality often do not arise in isolation, and they tend to permeate interrogation, stationery, defense, and group pressure, and change with new information.
The more tricky question is whether a sentence in the real world is a lie, and there is usually no absolute reliable standard answer.
The team chose "One Night's Ultimate Werewolf" as an experimental setting because the game solved both of the above problems simultaneously.
In the game, each player has a hidden identity, night actions determine which information the player knows, and day speech reflects his judgments and tactics. Because of the traces of both characters, actions and game rules, researchers can judge whether a sentence violates the information that the speaker had at the time.
Thus, MIDA does not judge a person who "does not seem to be lying", but who clearly knows what he says.
The MIDA benchmark
MIDA contains 2,360 speeches, of which 819 are from 40 games in the Ego4D subgenre, 1541 are from 151 YouTube video games.
The two sub-episodes form a natural difficulty ladder: the participants in Ego4D are mostly beginners, while the players in YouTube videos are more experienced and better at using sophisticated deception strategies.
Each sample contains video, audio, conversation history, and game rules, and is marked as true, false, or neutral. "Neutrality" here does not refer to the neutrality of a sentence, but it does not contain facts that can be verified based on the speaker's current state of knowledge, such as questions, conjectures, opinions, or assumptions.
In order to obtain verifiable labels, the research team first artificially recorded the game's night actions, then combined the characters, rules, and dialogue content to generate and calibrate the labels for each speech.
It should be pointed out that the characters, night actions, and voting results are only used to construct verifiable value labels and are not provided to the model during the testing phase. Models still need to independently infer the information that the speaker has and its true intent based on video, audio, and conversation history. Therefore, the rules of the game provide a controllable, verifiable evaluation environment, rather than a shortcut to help the model get answers directly.
In addition to making judgments that are false, MIDA also requires models to identify persuasive strategies for speaking, including identity statements, accusations, defenses, evidence, inquiries, and calls to action.
Figure 2: Proportion of different persuasive strategies and distribution of TRUE-,FALSE-,NEUTRAL labels in the two data subclusters. Statements of identity and statements of evidence carry more verifiable information, while statements of questioning and unclear strategies mostly belong to NEUTRAL.
The results of the experimentOverall accuracy may be misleading
The study covered models such as GPT-4o, GPT-5, Gemini, Claude, and Llama, InternVL, Qwen, and DeepSeek.
From an overall accuracy point of view, some models don't seem to perform poorly. In the Ego4D subset, the overall accuracy of GPT-4o is 74.0%, and Qwen2.5 - VL-7B is 76.4%.
But there are a lot of neutral speeches in MIDA. If the model chooses "neutrality" frequently, it can get a good-looking total score.
When researchers kept only samples that had to be judged to be false, the results changed significantly: GPT-4o had a secondary accuracy of only 27.6%, and GPT-4o-mini only 39.4%. Qwen 2.5-VL-7B had the highest overall accuracy, but its F1 value for "false" was 0.
In other words, a model may most of the time answer "unjudgeable" and thus obtain a higher overall accuracy rate but still fail to truly identify lies.
Table 1: Major ratings of different models on MIDA-Ego4D and MIDA-YouTube. There is a significant gap between overall accuracy and binary accuracy for TRUE and FALSE samples alone.
The most socially rational speeches are also the worst-performing parts of the model.
In the "identity statement" category of speeches, the average accuracy of each model is only 15.7% and 25.3% respectively in the two sub-sets; the average accuracy is less than 40% for speeches containing evidence. The model may be able to recognize "this sentence is about accusing someone else", but may not necessarily know whether the accusation is likely to be true.
Experienced gamers also increase the difficulty. The percentage of YouTube players who give false testimony is 36.8%, nearly three times that of new players; identity lies account for 51.6%, which is much higher than 18.9% of new players.
The history of dialogue is more important than "microspective".
The team analyzed the effects of different information through digestion experiments.
GPT-4o-mini's accuracy on fake samples dropped from 39.4% to 13.4% after removing the previous conversation history.
This suggests that identifying a sentence as a question, a defense or an accusation may only be necessary by looking at the local text; judging that the sentence is true or false, it is necessary to remember who said what, who knew what, and whether the claims of different players are mutually valid.
After adding one image, the model's macro average F1 increased by 2.5 percentage points; after increasing to three, the gain decreased to 2.1 percentage points instead.
More images bring more expression and movement, and more noise. Although models "see" these clues, they don't necessarily know which clues are really important.
Table 2: Meltdown experiments of dialogue history and visual acuity. Following the removal of dialogue history, the MIDA secondary accuracy rate decreased from 39.4% to 13.4%; the increase in visual acuity also did not lead to a sustained increase.
Research methods
To address these issues, the research team designed two modules for social thinking chain SoCoT and dynamic social cognitive memory DSEM.
Figure 3: SoCoT transforms the face, body, and voice into structured clues; DSEM maintains each participant's changing knowledge, speech, and relationship status over time.
SoCoT requires the model to first describe facial expressions, physical gestures and vocal characteristics, then speculate on the intended and psychological state of the speaker, and finally give false judgments.
This process is equivalent to breaking down the intuition that "there may be a problem when he speaks" into verifiable reasoning steps.
However, non-language clues do not always improve judgement. In the example of GPT-4o-mini, when analyzing only facial clues, the overall accuracy of the model increased by 8 percentage points, while the true-false categorical accuracy decreased by 7.1 percentage points.
Tensions, avoidance of sight, and other "suspicious" behaviours are not the same as lying. Even if models succeed in identifying these behaviours, they may misinterpret them.
DSEM uses another approach: maintaining a continuously updated "memory board" for each player, recording their character statements, actions, insights, known facts, personality relationships, and event timelines.
In the face of a new statement, the model no longer analyzes the current picture, but continues to verify: what should this person know right now?
In the experiment, the improvement brought about by DSEM was more stable. It improved the secondary accuracy of the GPT-4o-mini by 2.3 percentage points, the macro average F1 improved by 3.3 percentage points, and the macro average accuracy improved by 7.6 percentage points on Qwen 2.5-VL-7B.
Instead of looking for "cracks" in an instant's expression and continually tracking everyone's knowledge and position, it seems to be closer to the way social reasoning is actually used by humans.
Table 3: Effects of SoCoT and DSEM on closed-source, open-source model baselines. The effects of SoCoT on different indicators are not consistent, while DSEM has steadily improved the accuracy of false classification in both models.
Conclusions
MIDA is not limited to killing wolves. In situations such as conference collaboration, online discussion, human-computer interaction, intelligent customer service and multi-intelligent collaboration, models also need to track the information held by different participants, their changing positions and their interactions.
Killing wolves provides a controllable and sufficiently complex experimental environment for this capability: participants have different private information, associate, question, defend, and consciously influence the judgment of others. Through the gap in the ability to expose models in this environment, MIDA provides a quantifiable starting point for broader multidisciplinary social reasoning research.
The authors summarize the current multimodal models as powerful "knowledge engines" rather than mature "social intelligences". They may lack not only stronger visual encoders, but the ability to continuously model the knowledge, beliefs, and intentions of others - what is commonly referred to as "mental theory".
When models leave clear visual responses and enter environments where different information is acquired, views are constantly changing, or even actively influence the judgment of others, it is still difficult to truly "read the air" in today's strongest multimodal models.