Let an intelligent body analyze a legal case, it can retrieve the regulations, the facts of the case, list the reasons, and finally give a seemingly perfectly correct answer.
The problem may arise at an earlier stage: it refers to an incorrect legal source, calculates the program deadline incorrectly, or misinterpretes the content returned by the tool. The more fluid the subsequent reasoning, the more easily this error is obscured by the final answer.
Nowadays, the legal model is moving from a single question-and-answer system to a smart one. The system no longer generates just a text, but plans tasks, calls out retrieval tools, reads materials, and continues to act based on intermediate results.
It's a hallucination that starts with a multi-step process.
The research team at the Hong Kong University of Science and Technology focused on this execution chain. They proposed the LexAgentHallu IQ hallucination benchmark, extending the observed objects from the final answer to the complete trajectory, and positioning errors to specific steps.
The starting point for this work is straightforward: when an intelligent organism says the hypothesis is based on the hypothesis that the hypothesis rules have already been found to apply to the current hypothesis, the assessment does not only look at the conclusion that it is wrong, but also at whether the reasons behind the hypothesis are true, accurate and consistent.
LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents
Institution: Hong Kong University of Technology and Technology
Address of the paper:https://arxiv.org/abs/2609.09754
A wrong source of law
It's not just a matter of law, it's a matter of law.
In the past, judging large legal models, the most common practice was to give a question and then check whether the final answer was correct. In the case of an intelligent body, the same error is to continue to be traced back and forth to see what searches, citations and judgments it affects.
A misspelling of the name of a regulation may leave only one text error. A misspelling of a regulation as an existing law or a downstream regulation as an upstream law may change the choice of subsequent rules, their actual application and recommendations for relief.
The risk for an intelligent organism is therefore not only that it can find genuine materials but use them incorrectly; call the right tools but fill in the wrong parameters; see the relevant results but misread the scope supported by the results.
One illusion is no longer just the wrong text, but could be a prerequisite for the next legal judgment.
LexAgentHallu, therefore, does not consider the final answer to be the only evaluation object. It examines the execution trajectory of the intelligent organism gradually: how the system plans, what tools it calls, what results it reads, what information it retains, and how to connect facts and rules.
Illusion of the Intelligent Body
Where exactly is it?
LexAgentHallu has established a two-tiered classification system.
The first layer focuses on the content of the law itself, including errors such as legal sources, legal principles, procedural rules and the applicability of facts. The second layer focuses on the execution process of the intelligent body, including planning and reasoning, memory, and errors such as tool calling and understanding the results of observations. The two layers combined contain 7 intermediate categories and 27 sub-particle classes.
The purpose of this classification is not to change the name of a set of errors, but to allow the problem to be positioned. Law source falsification requires verification of the citation, the fact that the error of the string rules requires re-checking of the element, tool parameter errors require constraining the call interface, and memory deviation refers to context management.
The construction of the benchmark also revolves around the complexity of the case and the verifiability of the hypothesis. The research team collects data from various legal tasks, filters easy samples, then verifies the answers, legal sources and reasoning trajectories by markers with legal training backgrounds, and finally completes the classification of illusions with expert reviews.
Figure 1 shows the four stages of constructing the process of LexAgentHallu. Source: Original paper Figure 1, page 4.
In the end, LexAgentHallu collected 3,414 instances, covering 17 legal categories and 6 task categories. It not only answers how many times the pollution system made a mistake, but also tries to explain where the error occurred, to which category it belonged, and whether it is possible to continue to contaminate the next steps.
The best configuration
And 89% of the tracks are hallucinatory.
The paper evaluated 18 types of closed-source and open-source intelligent body configurations. The results showed that high hallucinations are not a marginal phenomenon of individual systems.
Even with the best configuration, 89% of execution tracks still have at least one hallucination. The hallucination frequency of all systems at the legal content level is not less than 87.5%.
This set of results does not mean that all legal responses are at the same risk. It shows that in this set of assessments that emphasize difficult samples and complete execution trajectories, it is difficult for the current system to maintain a clean, verifiable chain of reasoning from start to finish.
Models, tools and methodologies also change the distribution of errors. ReAct-like action loop observations have more advantages over overall hallucinations and legal content hallucinations; legal specialized workflows reduce process errors; and methods of pre-planning and execution do not show a steady lead in the experiment.
Therefore, when deploying intelligent objects, models, tools and processes cannot be evaluated separately. Stronger underlying models may still make mistakes in inappropriate retrieval processes; smaller models may also achieve more stable performance through clearer tool protocols and process constraints.
The answer is right.
Why the Reasons Are Still Unbelievable
The paper proposes a very intuitive indicator: RAWR, or Right-Answer-Wrong-Reason. It only observes samples of the final correct answers and then checks whether there are still hallucinations in the execution trajectory of those samples.
The results show that, on average, 68% of tracks still contain at least one legal content hallucination, and 37% of tracks contain at least one intelligent process hallucination.
Figure 2 The RAWR measures the ratio of responses to but not reasoned errors. Red is the error of legal content and blue is the error of intelligent processes. Source: Original paper Figure 4, page 7.
The correct answer only explains the conclusion of the proposition, and does not prove that the chain of reasoning is pure.
This distinction is particularly important for applications such as legal advice, contract review and legal aid. Users need not only a sentence that looks right, but also a traceable, verifiable and questionable chain of reasoning.
If the system only gives conclusions at the output end but does not expose the quality of the citation, reasoning, and tool calling, the error may persist if the answer is correctly rendered.
Which tasks are easier to solve
The frequency of hallucinations in different tasks is not uniform. Open, long-chain tasks that require a comprehensive set of facts and rules make it easier to expose legal content and process errors.
Figure 3 illustrates the frequency of hallucinations under different task types. Open tasks such as judgment analysis, case analysis are high. Source: Original paper Figure 5, page 8.
On the task type, the legal content hallucination frequency for judgment analysis is 1.00, and case analysis is 0.95; legal reasoning, legal consulting, legal knowledge questionnaires, and judgment prediction are also highly ranked.
Processor hallucinations are presented in different order. Judgment analysis reaches 0.83, and judgment predictions are 0.71. Shorter outputs are not necessarily safe: as long as the task requires multi-step evidence integration, deadline judgement or rules apply, planning and memory may still be at risk.
The real determinant of the risk is not just how difficult the topic looks, but how long the system needs to complete the hashing process. Any one of the links deviates, and subsequent steps can proceed on the wrong premise.
Are the hallucinations going to happen along the chain?
The paper also analyzed co-existing relationships between 27 subclasses of finite-dimensional hallucinations. The results showed that these errors are not random scattering points, but form structured combinations.
Figure 4 illustrates the symmetry matrix of the illusion subclass. Red represents the symmetry and blue represents the symmetry. Source: Original paper Figure 6, page 8.
The co-existence improvement for procedural deadline errors and remedial path errors is 2.82, and the error of procedural steps and the error of procedural consequences is 2.59. The strongest cross-group signal appears between the error at the law source level and the confusion of the legal position, with the improvement reaching 7.07.
This means that many seemingly decentralized errors may share the same upstream cause. At the legal source level, once a mistake is judged, the position of the subsequent rules and the facts apply may diverge together; once a procedural clock is calculated incorrectly, the timing, steps and remedial paths may also continue to cause problems.
For system governance, high-risk errors can be viewed as early signals. Once anomalies are detected at the law source level, in the timeframe calculation, in the process steps, or in the mapping of fact elements, the system should trigger a second retrieval, rule checking, or artificial replication, rather than waiting for the final answer to be generated and then passively correcting the error.
From the evaluation benchmark
How to get to the product line
The direct value of LexAgentHallu is to dismantle a lot of hallucinations into a type of defect that can be handled. To put this idea into a product, a smart body needs at least three checks.
Prior to creation, clear boundaries. The system needs to confirm the scope of the applicable law, the scope of the task and the extent to which it can be answered. When jurisdiction, timing or procedural paths are unclear, definitive judgments should not be given directly.
In execution, verification bases - retrieval results, reference sources, critical facts and tool returns - need to be continuously verified. An intermediate result cannot be taken to the next step just because it looks reasonable.
Conclusion and reasoning should be mutually supportive, especially in verifying the effectiveness of legal sources, procedural deadlines, constituent elements and remedial pathways. The final answer is correct and cannot replace this step of checking.
Furthermore, the evaluation subjects should be extended from a single model to a complete workflow. How the system retrieves, when to reflect, how to maintain the intermediate state, under what conditions to stop, are all part of reliability.
The paper also reminds us to be cautious. LexAgentHallu is primarily based on the Chinese legal system and focuses on single-entity scenarios. When migrating to common law, multi-jurisdictional retrieval, or multi-intelligent collaboration, law authority, precedent, and procedural rules need to be redefined.
Written at the end
LexAgentHallu adds a more realistic way of evaluating deployments for smart bodies: not only to see what the final answer is, but also to track how the answer is formed.
We entrusted the legal task to the intelligent body in the hope that it would be able to handle the searches, the fact-finding, and the rules that apply, rather than allowing users to scrutinise a seemingly smooth chain of reasoning afterwards. A reference, a deadline judgement, a tool call, all need to be validated before going to the next step.
It reminds us that the credibility of an intelligent organism cannot be proven by the final answer alone. A truly reliable system requires that errors be exposed sooner and that errors are made more difficult to propagate along the execution chain.
Author's note
The paper's first co-authors, Zhou Jing, Liu Ming, and Cao Chu, are PhD students at the Hong Kong University of Science and Technology's Interdisciplinary School of Artificial Intelligence. Professors from Hong Kong University of Science and Technology and Professors from Hong Kong University of Technology and Guangzhou, whose research focuses on legal AI, big model security alignment with intelligent agents, and the results published at top international conferences such as ICML, EMNLP, ACL.