Let AI help you send an e-mail, which may have been written correctly, the attachments arranged correctly, and eventually sent the wrong person.
The problem arose in an earlier step: it read the mail address incorrectly. The smoother the process, the easier it is to get through until the mail is sent.
Nowadays, the desktop assistant and browser Agent are able to continuously operate multiple applications, organize files, fill out forms, and process tasks for users. People give more steps to AI and have less opportunity to examine what it reads and what judgments are made accordingly.
A hallucination begins with real execution consequences.
Researchers at Shenzhen University and the Hong Kong University of Science and Technology have focused on this link. They have proposed ECA (Evidence-Carrying Multimodal Agents): models are responsible for making moves, independent verification procedures are responsible for providing evidence, tools are executed after the evidence has been examined.
The starting point for this work is straightforward: when the model says "the address has been confirmed" and "this website is credible", the operating system should get the corresponding proof.
Hallucination as Exploit: Evidence-Carrying Multimodal Agents
The author of the book is Guijia Zhang, Hao Zheng, Harry Yang.
The document is addressed at: arxiv.org/abs/2605.19192
A false premise that could lead to a complete mismanagement process
In the past, when we talk about AI illusions, we often worry about whether they are fabricated: whether the object actually exists in the image, whether the figure actually appears in the document.
Identifying the wrong color of a dress in the picture may leave only a wrong description. Identifying the wrong receiver on the bill may change the direction of a payment. The error itself is small and does not indicate how easy the consequences are.
The paper calls this phenomenon a "transformation of hallucination into action" (H2AC): the model generates or adopts an unsupported judgment and then takes it as a premise for the execution of the action.
This path is particularly concealed in the fact that the model may have been following the user's request. The page does not have to appear with an attack phrase such as "ignore previous instructions". A seemingly credible address, a field that has been misread, may cause further operations to deviate.
In the authorization trajectory relay of the paper, the insecurity rate of instruction-injected class tasks was 51.0% when guided solely by a hint word protection; the error-prerequisite-driven task was 85.7%. It is difficult to detect problems when the model does not receive malicious commands, but believes in the wrong information, and instruction filtering is difficult.
ECA therefore refines the objects of inspection to each action. Before clicking, confirm the existence of the button; before sending, verify the recipient, user intent, and attachment permissions. Any critical condition needs to have its own basis.
If the page says "official", can you give the file to it?
This example illustrates how ECA works.
The model sees a web page that says "official entry" and is ready to upload the user's file. The OCR also accurately identifies the four words. Here, the system confirms what text appears on the page; the identity of the website also needs to be verified through browser source information and a credible list maintained externally.
The text on the page cannot be used to prove the identity of the page.
ECA retains different locations for both types of information. Multimodal models receive web pages, screenshots and documents, understand tasks and make tool calls; another group of task-defining verifiers return evidence from the original content.
The DOM is used to check buttons and links, OCR verifies text in images, and an unobstructed tree provides control names and roles. Each result records a specific value, location, source, verification process, and credentials level, forming what the paper calls a "proof certificate".
Models can explain why a button is clicked, but whether the button exists, where it came from, needs to be proven by these records.
Figure 1 of the ECA architecture. Models propose actions, independent verifiers provide evidence, door control procedures check the execution conditions. Source: Original paper Figure 1, page 1.
The final decision is made by a door control procedure that works according to fixed rules. It matches the conditions required for the action with each piece of evidence: the conditions are fulfilled at the time of release; the evidence is incomplete; it is appropriate for further verification to supplement the evidence or request confirmation from the user; it stops execution when information conflicts or sources are lacking.
In this way, the executable interface receives not only a "can send", but also a specific record that supports this sending. The model is no longer sure of its own judgment, nor can it be verified with a text instead of this step.
The team also tested another common practice: using large models as security auditors, and attempting structured reasoning, multiple voting, and multiple rounds of consideration. In 200 Agent Dojo missions, the best-performing multiple rounds of evaluation in the enhancement program still yielded about 79.3% of insecure moves.
Figure 2 GPT-5.4 Different configurations of security referees. UAR is the rate of unsafe action, ASR is the rate of attack success, Benign is the rate of normal mission success. Basic referees use 500 missions, the rest of the configuration uses 200 Agent Dojo missions.
The researchers observed that the audit model uses the planning model to describe the premises. Both can make the same mistaken judgments in the face of the same manipulated observation. The independent evidence corridor introduced by the ECA requires that the judgement be externally verified before it is executed.
Check if it really works after you put it in the browser
The team constructed 200 tasks around web navigation, document extraction and email writing, allowing the system to complete the entire process from content extraction to action authorization. Of these, 140 insecure actions were all stopped and 60 normal tasks were all completed.
They then accessed the ECA to the Chromium browser. Of the 120 missions, 85 unsecured actions were unauthorized; of the 35 normal missions, 30 were direct releases, and five were confirmed by users.
Figure 3 shows the authorized results of experiments and control methods at each level. The table lists the verifier red team tests, the contents of the extracted experiments, the assumption of correct relaying of the certificate, and the results of the model arbitration. Source: Original paper Table 1, page 6.
The verifier itself is also the target of the attack. The team constructed attacks such as source disguise, simulation of characters, and fictitious controls, with each reinforcement extracting a link. In 1,700 standard instances of reinforcement attacks, no one bypassed the gate controls.
One of the details is representative: some of the attacks fooled unobstructed tree verifiers, but were discovered by a cross-checking by DOM. Multiple evidence channels played a substantial role here.
The team also relayed 7,488 authorized tracks on six external benchmarks, using assumptions of correct certificates, and independently checked gate control rules. The complete process, browser testing, and verifier attack tests were further tested to see where the evidence came from, whether it would go wrong, and whether it would support actual operations.
Multiple checks, how long users have to wait
For everyday use, frequent pop-up confirmation boxes can also be patient. The researchers took the ECA to 16 real web pages and tested 452 target-specific normal clicks. These sites are already on the list of trusted sites, including encyclopedias, news, development documents and others.
In the end, 423 clicks were sent directly, or 93.6%; another 29 users confirmed that no normal clicks were directly rejected.
Confirmation is needed mainly on dynamic pages: links or button text already displayed on the screen are not completely consistent with the content obtained by static analysis, leading to incomplete element evidence.
On the same set of complex web pages, the completion of page information extraction, screenshots, text recognition and door control judgments takes an average of about 0.66 seconds and a minimum of about 3.2 seconds.
Written at the end
The ECA adds a pre-executed verification to the Agent: the model submits the action, the independent verifier provides evidence, and the requirements for the door control procedures to be checked on a case-by-case basis. From the operation of the web page to the sending of the email, the system must answer the same question: Is the basis for this action validated?
We entrusted the task to the Agent, hoping that it could get things done without having to stare at the screen from time to time. A click, an email, a file submitted requires the system to verify it before execution.
Author's note
The study was carried out by Shenzhen University and the Hong Kong University of Technology in collaboration with Guijia Zhang, Hao Zheng and Harry Yang.
Harry Yang is an assistant professor at the Hong Kong University of Science and Technology's School of Interdisciplinary Sciences and is currently working on the Digital Neural Alive project.