When an AI assistant says your new job is wrong, it's easy to conclude that the model is in a "false eight".
But did the mistake happen at the moment of answering?
Imagine that you've told an assistant that you've just been promoted. It may have missed a position while organizing a conversation, or it may have kept an old record, or it may not have completed an update, or it may have gotten the correct memory but changed it to another title when answering.
It's the same "wrong answer", but it could be a completely different kind of memory failure.
Researchers such as the Chinese Telecommunications Institute and MemTensor in Shanghai have proposed HaluMem: the first operating-level hallucination assessment benchmark for intelligent memory systems. It will assess in-depth the three links of memory extraction, memory update and memory questionnaires, tracking exactly where errors occur and how they affect subsequent responses.
This work has been accepted by the EMNLP 2026 Conference, the top conference in the field of natural language processing.
If a note is written incorrectly and then proficiently reviewed, it may just be more accurate to recover the error. To determine whether AI's memory is reliable, we need to check how the "note" is written, modified, and used.
HaluMem: Evaluating Hallucinations in Memory Systems of Agents
It was written by Ding Chen, Simin Niu, Kehang Li, Peng Liu, Xiangping Zheng, Bo Tang, Xinchi Li, Feiyu Xiong, Zhiyu Li.
Open source projects:
ArXiv paper:
1. A question obscured by the final answer:
Is AI wrong, or has it already been wrong?
The memory system allows the AI to retain user information between conversations: what you do, what you like, what changes you've been through recently.
Existing memory assessments mainly use end-to-end questions: input a series of conversations into the system, ask questions, and compare answers to standard answers. This can measure how the system ultimately responds, but it's difficult to explain why it's wrong.
HaluMem's entry point is a verifiable standard answer for intermediate operations.
Figure 1: Comparison of HaluMem with existing assessment methods. In addition to checking the final answer, it is also necessary to check whether memory bias has occurred during extraction and update.
For example, if a user says "I recently started to like chicken, I really didn't like it before", the system takes it as the "user doesn't like chicken". The error has already occurred in writing the memory. If the user explicitly states that his health has deteriorated, the system updates the health status to "good" according to the assistant's incorrect memory.
There is also a more concealed situation: details that the assistant himself compiles are considered by the memory system to be user facts. In conversation, the AI may casually mention a friend or experience that does not exist; the user does not explicitly confirm it, but the system stores it in long-term memory. An unconfirmed expression may become the "base" for the next answer.
HaluMem therefore examines three issues separately:
Memory extraction: Is the information recorded in the memory complete and accurate?
Memory updates: Are old memories correctly modified after new information appears? Are there any leaks, changes or conflicts left?
Memory questions: Can the system ultimately use memory to give the right answers, the wrong answers and the lack of information that can't?
These checks run in the order of the conversation time and are triggered immediately after the relevant session processing is completed. Therefore, the evaluation is no longer only given a total score, but can also see the specific problems of the different operating stages.
2. To give AI a "life" spanning more than a decade,
Add to that the interference of a million tokens
To check for errors in memory, one must first know: what exactly should the system remember at each moment?
Real chat records often lack such a complete tag. Users' preferences change, old information becomes obsolete, and some facts are hidden in the context. It is difficult to recover the whole process of memory evolution after completing a few questions after the end of a conversation.
HaluMem has designed a six-stage user-centric data-building process: starting with user images, building life development skeletons, expanding into event streams, generating corresponding session summaries and memory points, and finally generating multi-round conversations and evaluation questions.
Figure 2: The six stages of HaluMem build a process. From user images, life skeletons and events, the memory points to be recorded will be associated with actual conversations and evaluation questions.
The process simulates 10 to 20 years of user experience. Career changes, health status, interest preferences and relationships evolve over time, forming three types of memory: images, events, and relationships. When updates occur, the data retains the relationship between new and old information, so that "how to change" can also be checked.
When creating a conversation, researchers also add interference such as assistant misinterpretations or fictitious details, as well as implicit expressions and references. Both the system finds information worth preserving and determines which statements have not been confirmed by the user.
On this basis, HaluMem offers two versions:
HaluMem-Medium: covers 20,073 rounds of conversations, with an average of about 160,000 tokens per user, and contains 14,948 memory points and 3,467 questions.
HaluMem-Long: Preserves the same valid memory, inserting unrelated conversations between sessions and within sessions, extending the context of each user to a million token quantity level, with total conversations reaching 53,516 rounds.
The key to this group's design is that: The Long version mainly increases the burden of interference with the context, rather than the core facts that need to be remembered. Therefore, the differences in the two sets of results can help observe whether the system can continue to retain effective information in longer, noisier interactions.
The problem is not limited to "remembering but not remembering". The six questions cover basic fact-remembering, multivariate reasoning, dynamic updating, memory boundaries, generalization with applications, memory conflicts. For example, both knowing the current state of the user and identifying the wrong premises in the problem, or acknowledging that information has never been provided.
To check the quality of the data, the researchers performed an artificial evaluation of 700 sessions on Medium, covering more than half of the sessions, and verified the memory points and questions in them. The accuracy of the tests reached 95.70%.
3. Six systems tested:
Remembering more and changing less can cover up the problem.
The study evaluated six memory systems: Mem0, Mem0-Graph, Memobase, MemOS, Supermemory, and Zep, and tested them separately on two contextual scales.
For standardized evaluation conditions, updated task retrieval verifies related memories of the first 10 items, query task retrieval of the first 20 items, and unifies using GPT-4o to generate responses. The following values are derived from the paper's main experimental setup; Zep cannot access session-level memory retrieval records due to its official interface, and does not report memory retrieval indicators.
Table 1: Results of six systems in memory retrieval, update and query tasks. Put multiple indicators together to distinguish information coverage, correctness and omission issues.
Discovering one: Remembering more doesn't mean remembering more reliably.
On Medium, Mem0 and Mem0-Graph are recalled at 42.91% and 43.28%, respectively, and Memobase at 14.55%.
After extending to Long in the above context, the recall rate of the three further decreased to 3.23%, 2.24% and 6.18% respectively. The core memory remained unchanged, and the addition of unrelated conversations was enough to cause some systems to miss the vast majority of target information.
MemOS on Long has a recall rate of 81.90%, and SuperMemory is 53.02%; both cover more target memories while also exposing issues that interfere with insufficient memory.
HaluMem uses False Memory Resistance (FMR) to measure whether the system can ignore the interference memory mentioned by the assistant but not confirmed by the user. The higher the indicator, the better. On Long, the FMR of MemOS and Supermemory is 28.85% and 36.86% respectively.
This means that expanding information coverage does not automatically lead to stronger authentication capabilities.
A reliable memory needs to do two things at once: the memory cannot be leaked, and the message cannot be stored. It is easy to ignore the cost of the other side by looking at the recall rate or the memory rule.
Discovering two: Why is it still unreliable to update hallucinations below 1.2 percent?
If you look only at the rate of refreshing hallucinations, the results seem quite optimistic: in the main experiment, all systems in both versions had this indicator below 1.2%.
But if we put aside the rate of misses, the situation is completely different.
On Long, five of the six sets of systems have an update miss rate of more than 60%. Of these, Mem0 and Mem0-Graph reached 98.51% and 98.40%, respectively, with correct updates of only 1.45% and 1.47% respectively. MemOS has the highest correct updates of 65.25%, but still missed about a third of the target updates.
Why do "little mistakes" happen at the same time as "a lot of missteps"?
Because the update hallucination rate is measured by the percentage of errors that occur during the target update. When a large number of updates are not completed at all, the percentage of errors can also be very low.
Furthermore, the update also relies on previous extraction: if old facts are never stored in memory, there is a lack of subsequent objects that can be updated. Operating-level evaluation thus reveals an important correlation.
Discover three: Upstream memory questions that will eventually appear in the answers
In the Unified Q&A setting, the accuracy of all measurement systems did not reach 70%. MemOS reached 67.23% and 64.44% respectively on Medium and Long, respectively, for the two sets of highest values, but there is still significant room for stability and reliability.
For some systems, the decline in context was particularly prominent. Mem0's correct answer rate dropped from 53.02% to 28.11%, while the missed answer rate rose from 27.81% to 54.60%.
Figure 3: The accuracy of the responses of the six memory systems in the six types of subjects, respectively showing their performance in the two sets of assessments. Different subjects are used to examine the system's ability to respond in different memory use scenarios, and the impact of the long-term context on the different types of capabilities.
Different topics also reveal differences in capabilities: most systems perform relatively well in identifying unknown information or erroneous premises, but there is still a clear shortcut in the face of multiple hypotheses, dynamic updates and generalized applications.
The ultimate answer is a common product of the entire memory processing process. To improve the response, it is necessary to simultaneously check whether the facts have been memorized, whether changes have been updated, and whether the information retrieved can be used correctly.
The paper also randomly selected 10 users and reviewed them using a GPT-5.4-mini replacement main experimental referee. The specific scores vary, but the overall ranking of the memory system, main performance trends, the swapping between extracted coverage and interference resistance, and the core findings of updated missing severity remain consistent.
4. What is missing from "remembering" to "remembering reliably"?
The value of HaluMem is to make improvements in memory systems more specific.
If the problem is being extracted, the coverage of the information and the reliability of the source must be checked at the same time; if the problem is being updated, the association of the new old memory must be checked, and whether the modifications to be made have actually been completed; if the problem appears in a query, the upstream record must be combined with the results of the retrieval to continue the analysis.
This also explains why a nice single indicator isn't enough: high recalls may be accompanied by false messages being written, low refresh hallucinations may be accompanied by a large number of omissions, and ultimate Q&A scores may obscure the failure of various internal links.
Of course, there is still room for future expansion in this work. Current data revolves around user-simulated images, events, and relationship memories, and other forms of memory such as tools, skills, have not yet been comprehensively covered. Open interfaces of different systems will also affect the observability of internal operations.
But it has provided a replicable evaluation idea: one-by-one verifying the memory operations behind the final answer, making "where went wrong" a measurable problem.
Going back to the assistant at the beginning who said the new job was wrong. Next time, in addition to asking "why did you answer wrong?" we can also continue to ask: was this information correct at first?
When these questions can be answered one by one, AI has a clearer path to improvement away from truly reliable long-term memory.