After doing so, it is attributed to online auditing, which identifies critical errors before missions fail
A team from the University of Texas at Austin and Purdue University proposed Agent Foresight to allow a 7B model to be continuously audited in the course of missions. In the AFTraj-2K test kit, it reached the critical error step identification indicator Exact-F1 of 66.44, which is 19.88 points higher than the strongest general model baseline, and only 2.37% higher for successful trajectory reporting.
Would it be more reliable to have one agent plan, one agent search, and then another agent report?
Imagine a scenario like this: a user wants to buy a pair of noise-reducing headphones for $200, but the filter agent sets the budget limit to $300. Next, compare the parameters, generate reports, give recommendations, and every step can be a mess.
The following agents do not necessarily make mistakes in their sub-tasks; they merely make the mistake of taking the previous step as a prerequisite for continuing their work.
After the recommendation is generated, asking "which agent, which step went wrong" will help, of course, but is it possible to remind the system when the budget has just been changed, instead of letting the next agent continue to follow this erroneous premise?
This is the starting point for Agent Foresight: early warning when a local error has occurred but the mission has not yet been communicated.
For this reason, it attributes the aftermath to online auditing: with each step completed, the external auditor reads the execution records up to this point, decides whether to proceed with the alert, and seeks time for further intervention.
The paper is titled AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems.
Written by: Boxuan Zhang, Jianing Zhu, Zeru Shi, Dongfang Liu, Ruixiang Tang.
The following institutions: Rutgers University, The University of Texas at Austin, and Purdue University.
Link to the paper:https://arxiv.org/abs/2605.08715
The main page of the project:https://zbox1005.github.io/agent-foresight/
The code repository:https://github.com/ZBox1005/AgentForesight
The data collection:https://huggingface.co/datasets/ZBox008003/AFTraj
You haven't seen the end yet, why stop?
In the past, many intelligence failures have been attributed to the use of post-traumatic analysis: first reading a known failed trajectory, then judging the responsibility of the Agent and the critical error steps.
Online auditing lacks these "retrospectives". AgentForesight can only see the current trajectory, i.e. the records that have taken place so far; future actions, unreported tool results and ultimate successes are not within its scope.
Figure 1 is attributed to online auditing after being engaged. Figure 1 is attributed to post-audit errors after mission failure; Figure 1 is attributed to post-audit errors after mission failure; Figure 1 is attributed to post-audit errors after mission failure; Figure 1 is attributed to post-audit errors after mission failure.
The difficulty, therefore, is not simply to "find an error", but to distinguish: is the system still under reasonable investigation, or has there been a decisive error affecting the success of the mission?
The auditors should also point out the steps in which the error occurred, by which Agent was introduced, and give a brief justification to give a clear focus for further processing.
Not only do we collect failures, but we also know when to let go.
To move the attribution to the internet, the training data must also be changed. Only the failed trajectory, the model does not learn when to continue the task; only the success label of the entire task, and cannot tell it exactly from which point the problem started.
For this purpose, the team constructed AFTraj-2K, covering code generation, mathematical reasoning, and Agentic tasks involving tools and retrieval. The data set contains 2,272 labeling tracks, 158 verified success tracks, and 1,114 failed tracks.
However, the "final answer" is not the same as "every step of the process is reliable". The trajectory corrected by other agents after a mid-way error cannot be distinguished as a usable success sample. The team therefore simultaneously checks the final results, the full effectiveness of the tool call, and the consistency of each step with the current sub-goal, to identify the successful trajectory that can be used for forward supervision.
Figure 2 Definitive Error Location Distribution: The horizontal axis of the wrong position is proportional to the length of the trajectory, the three mission domains have different distributions; the total covers 1,114 failed trajectories. The critical errors are not fixed at the beginning or end.
Failure data comes from two paths: one injecting errors into the verified success trajectory, checking whether the modification is effective; the other proposing a candidate position from natural failure, followed by multiple independent checks to check whether the error is real; whether it substantially affects the task; whether it is decisive; and whether it is the earliest decisive error.
What we're looking for here is not the most noticeable exception, but a key step that can change the outcome of the task: after fixing it, is there a path to move the task from failure to success? This makes the label from "the entire task failed" to "which step, which agent introduced the decisive error".
Learn to recognize the boundaries of failure and identify mistakes
Because the general model faces two interconnected thresholds: first to determine whether the current record has crossed the boundary of failure, and then to locate the specific steps and responsibilities of the Agent in the wrong record.
It requires that it acquire all of its capabilities at once, and the reward signals for accurate attribution are too scarce, and training can be degraded into a response to "safety". AgentForesight therefore uses two stages of training: first to establish sensitivity to the limits of failure and then to fine-tune this risk judgment into accurate attribution.
Figure 3 Overview of AgentForesight methods. Left: Screening success trajectories, marking construction steps by injecting controlled errors with natural failure verification. Right: First learning the failure boundaries with BPPO, then finishing the audit with three-axis reward refinement.
Stage 1: Agile capture of key shifts from "continue" to "alert".
The focus of the first phase is to make the online audit model sensitive to the critical change from "can still go on" to "should be alerted". The team draws two adjacent fronts from the same failed trajectory: one should continue until a decisive error is made; the other should alert only the step that has just gone wrong.
In this way, the context of the task and the history of previous execution remain unchanged, but judgments must be reversed with a critical step. The training signal thus focuses on the content that causes the change in risk, rather than on the full exposure of the consequences of failure, to recognize that "the task has failed".
This is Boundary-Pair Preference Optimization (BPPO): improving the preference for correct answers versus incorrect answers under each lead and combining samples from both sides of the boundary. The model is thus able to predict the risk of failing critical points, providing a starting point for the next phase of precision determination.
Stage two: From "There's a problem" to "Which step, which agent".
Following the establishment of a risk pre-determination, the alert should also be sufficiently specific. In the second phase, the first phase model is used as a starting point to reward the three dimensions of the audit conclusions.
The step reward isn't just "full match/full error": the closer the predicted position is to the mark, the higher the reward, and the smoother the signal for the gradual identification of the error.
Training also uses the first stage model as a reference to the KL constraint, limiting the second stage's deviation from previously learned risk judgments. The two stages are not simply synchronized: the latter is to be refined into "this step, this agent is in trouble" while maintaining a failed boundary sensitivity.
Also, just look at the current record, how are the 7B auditors doing?
The team evaluated the audit capability on the AFTraj-2K's 332-item test trajectory. AgentForesight and the general model in the table gradually read the forecasts and count the judgments in the first alert. Exact-F1 also examines the detection and accuracy of critical errors, while the ASS measures the average deviation of the error position from the markings in the report.
AFTraj-2K left the main test result selection. The higher up, the lower down.
The Exact-F1 of Agent Foresight-7B reached 66.44 points, 19.88 points higher than the strongest generic baseline DeepSeek-V4-Pro on the index; an improvement of 21.05 points compared to its base model Qwen2.5-7B-Instruct.
The ASS decreased from 1.77 on the DeepSeek-V4-Pro to 0.59, about one-third. It measures the attribution position deviation on the failed trajectory detected, rather than the wait time from error to alarm.
Exact-F1 for mathematics, code, and Agentic tasks is 77.36, 78.87, and 48.70, respectively; Exact-F1 and ASS are the leading control methods in the main table in all three domains.
On the external Who&When benchmark for non-trainees, AgentForesight-7B's critical steps and responsible Agent recognition accuracy rates of 57.69% and 73.08%, respectively, and ASS of 1.62, are the best results in the comparison models listed in the paper.
What is the complement of the two stages?
The dilution experiment gave direct contrast: the total Exact-F1 of the basic model was 21.05, with only 35.63 BPPO training in the first phase, 50.42 in the second phase, and 66.44 in the full program training.
Figure 4 shows the results of the two-stage training. The complete program is better than any single stage in the three mission domains and in the Exact-F1 as a whole; the column-top numbers are four to five.
Further illustrating the problem is the Agentic task: With only the second stage, Exact-F1 was 19.05, down from 31.58 with only the first stage; it was upgraded to 48.70 after the two stages were combined.
This is consistent with the two-stage breakdown. Critical errors in mathematics and code are relatively easy to locate, and positioning rewards have resulted in better performance; while in retrieval, tool calling and multi-character interaction, it is more difficult to determine "whether a decisive error has already been made" and it is especially important to first establish the boundary of failure.
Judging the boundaries of failure provides a starting point for a more accurate error attribution.
This also explains why you can't just look at the ASS: the model only alerts in a few accurate instances and can get very small position biases.
Don't interrupt your normal tasks unless you're on target.
The other half of the test of online auditing lies in the path of success: it does not judge a "success" at the end of a task, but to stop the rational process at every moment that has not yet been completed.
Figure 5 The more to the left, the better the balance between success trajectory error reporting and failure trajectory step accuracy. The light green area is the analytical reference range selected for the paper: FAR not more than 20%, Step Accuracy not less than 50%.
The FAR for Agent Foresight-7B was 2.37%, the Step Accuracy for Agent Foresight-7B was 59.51%; and the DeepSeek-V4-Pro was 43.20% and 53.99%, respectively.
2.37% was calculated on the basis of the entire trajectory: of the 169 successful trajectories of the test set, only 4 trajectories triggered false reports during the phase-audit. A trajectory is considered false only if it is alerted once at an arbitrary forefront, rather than averaging the error with a large number of normal steps.
Therefore, this group of results is not only "less alarms", but also low misreporting and better critical error recognition. Auditors should both remain sensitive to risk and leave room for normal exploration.
How can a wrong answer become a "consensus" between multiple agents?
One site in the paper questioned the trajectory and demonstrated this risk. The system needed to find an old industrial town near Rivington Hall Barn; the case marked the answer as Bolton, the search agent returned to Horwich, and the manager followed that result and eventually submitted the wrong answer.
Figure 6 shows different judgments in the same query trajectory. AgentForesight attributes the error to the steps of search_agent returning to the wrong location; Gemini-3-Flash points to earlier planning steps, and DeepSeek-V4-Pro gives Safe.
Two baseline errors are exactly the opposite: Gemini-3-Flash decides that the steps that are still in the planning stages are decisive errors; DeepSeek-V4-Pro accepts the sequence of errors but builds on the actual execution process.
Subsequent Agent repetition of an answer does not make it a more reliable piece of evidence. Online auditing should distinguish between "information still needs to be supplemented" and "critical errors have already been based on" states, both of which seem to continue to be executed on and off.
Agent's next teammate, possibly an independent auditor
Agent Foresight explores an independent division of labor: the executive agent is responsible for advancing the task, and the external auditor specializes in the process of determining whether a critical bias has occurred. It does not require the re-training of the underlying executive system, and execution and audit capabilities can be optimized, respectively.
Such audit conclusions can serve as a basis for suspending, replanning, switching execution paths or transferring man-made inspections. Multi-Agent systems do not have to wait for the end of the task to receive feedback on the execution process for the first time.
The experimental focus of this work remains on online identification and attribution. What actions can be taken after the alert, how many failures can be saved, and whether to improve the success rate of the final mission, needs to be verified in conjunction with specific intervention mechanisms.
The auditors themselves would also be wrong: the paper observed that the process of self-correction by the verifiable Agent could still trigger misreporting, and that the errors detected could also lead to location biases. Gradual audits will increase the cost of calls, and performance in longer workflows, rigorous and open-ended scientific research tasks will also need to be further evaluated.
The implication of this work is that improving the reliability of multiple intelligences will not only rely on a stronger execution model, but also allow a specially trained model to identify and point out critical errors in operation.
Multi-Agent collaboration requires not only teammates to execute, but also an auditor to know when to remind "this step is wrong".
Information from the author
Dr. Zhang Boon, Rutgers University (Rutgers University) PhD student in computer science, taught by Professor Don Riqiang. His research areas include credible intelligence, training and content production after big-language model detection, and related results published in international conferences and journals such as NeurIPS, ACL, IJCV. Currently, the focus is on online audit, fail-alert and reliability evaluation of smart entities, dedicated to improving the safety and reliability of big-language model intelligence in complex tasks.
This is the first time I've seen this.
The main page of the lab: