After BenchJack, what is missing from Reward Hacking detection?
An agent receives high scores on a benchmark without actually completing a task. It may also use vulnerabilities in the rating or reward mechanism to bypass title requirements to improve scores. This behavior is commonly referred to as reward hacking.
Prior to this, BenchJack conducted system audits of 10 Agent benchmarks, identified 219 vulnerabilities, and constructed a way to use them to get close to full scores on most benchmarks without solving actual tasks. The research team categorized common problems into eight types of vulnerability patterns, organizing them into an Agent-Eval Checklist for benchmark designers to inspect and modify rating systems.
Once a vulnerability has been detected, it is necessary to determine a specific operation: did the agent use the vulnerability? did its operation affect the final score? a compliance execution process may also occur in a task with a vulnerability, and a test response may also come from a way the task is not allowed to access.
BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
Link to the paper:https://arxiv.org/abs/2609.11028v1
Researchers from the University of California, Berkeley, and institutions such as BenchFlow AI have proposed BenchShield, a paper titled BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure.
BenchShield will extend the inspection to the complete process of producing scores: first specifying the requirements to be met at each stage, then using static analysis to look for potential use paths and, based on running evidence, judging whether Agent attempted or actually used these paths.
Why would an isolated verifier still give a full score?
Bypassing type inspection, allowing proof of error to pass verification
One of the paper's Lean theorem proving tasks is to isolate the verifier responsible for checking the submitted results from the operating environment of the Agent. The Agent submitted a proof of error, but still received a full score ((reward = 1.00))).
The reason for this is that the submitted Lean source code enables debug.skipKernelTC, shutting down the type checks in the Lean kernel. Verifier applies patches, rebuilds projects, and this setting is also effective. Although Agent cannot directly modify the verifier in the isolation environment, it can change the verification process by submitting the code.
Directly download standard answers and pass all 10 tests
The other task also used an independent verification environment. The agent downloaded a set of test labels from the open repository of the upstream project, which corresponded directly to the standard answer. It then converted the labels into the required output format and passed all 10 tests.
The verifier has not been tampered with, and the path and format for submitting the file are also required. The problem lies in the source of the answer: both the normal query and the moving label can get the same output, and only verifying the final answer cannot distinguish between the two processes.
Figure 1: Two types of reward hacking can still be completed after verifier isolation. The left side closes the kernel type check by submitting Lean code; the right side generates the answer after downloading the test set tags.
From the classification of vulnerabilities to the evaluation of the entire process
BenchJack organises audits according to a common loophole pattern. BenchShield further tracks the process of generating scores: which boundaries are crossed, which operations may affect scores, and what evidence is needed to determine whether a run is breached.
The paper summarizes this requirement with reward integrity: the scores should reflect how the Agent has completed the task within the agreed rules. The check thus covers environment initialization, task execution, result submission, result calculation, score reading, and log and feedback output; whether the environment remains in a round state after reset, and within the scope.
The framework first specifies repeatable inspection rules, and then maps the resources, permissions, and submissions of each task into the rules. It focuses on how the impact of violations enters the assessment process, rather than just adding a patch to a known vulnerability.
Six structural requirements and one semantic check
During the task execution and result submission phase, the system checks whether the protected information is read by the Agent (I1); whether the data or state at the end of the evaluation is controlled by the Agent (I2); and whether the content generated by the Agent can only be accessed through a pre-arranged submission (I3).
In the result calculation and rating phase, scores must come from credible evaluation output (I4), crashes, overtime or formatting errors cannot be considered as passing through (I5); logs, feedback and environmental reset also need to avoid leaking protected information or bringing a further round of residual states (I6).
I7 Separately discusses semantic adequacy: the evidence accepted by the evaluator is sufficient to prove that the task has been completed as requested. For example, a downloaded test set label can also be checked through the answer, but it cannot prove that the Agent has completed the reasoning for the subject requirement.
The research team uses TLA+ to build a limited-state model that examines structural requirements from I1 to I6 through TLC and records semantic issues that need to be verified separately in I7. Model checking is not the same as proving that the entire benchmark backend or Agent is correct; the conclusion of a specific operation still needs to be supported by task configuration, actual records, and the necessary semantic audit.
How did BenchShield detect the vulnerability and determine if Agent exploited the vulnerability?
This design draws on the practice of process analysis. Just as tools such as LLVM can insert monitoring codes into the critical linkages of the rating system by plugging in the behavior of the process, BenchShield adds monitoring codes to the critical linkages of the rating system, matching the running of the record with the control rules without replacing the original back end of the benchmark.
Figure 2: BenchShield workflow. After mapping the task configuration to a formalized model, the system first analyzes the potential use path; when running, it combines event recording and semantic auditing to make a determination.
Static analysis: What pathways could affect the score?
Before running, BenchShield builds a capability graph of the task, recording the relationship between resources, permissions, and operations. The stain analysis then tracks these relationships to check if Agent controlled content can be entered into the verification or rating link.
The analysis also takes into account the dissemination of protected information, failed status and historical residual status, and distinguishes the assessment stages in which they occur. The report identifies which inspection requirements are involved in reaching the critical nodes of the score, showing how a local vulnerability may be combined with other problems to affect the final outcome.
Technical explanation: The paper calls this analysis phase-aware taint analysis; reward-relevant sink refers to key nodes related to evaluation conclusions or scores. The system judges whether the impact can reach those nodes along the path, rather than just whether there is some questionable limitation.
Running time verification: What exactly happened?
When running, BenchShield records the actual operations received by the rating system, file submissions, verifiers, inputs, and score sources. The verifier controls the same set of process models in event order to verify that the Agent has crossed the assigned boundary.
These records supplement information beyond the Agent's visible trajectory. For example, the Agent says that he modified a file, and cannot state that the modification was ultimately read by the verifier. Every time the audit evidence was judged to depend on encryption, the same evidence was entered correspondingly to the same result.
There are other questions that need to be combined with the mission meaning judgement: is the downloaded document the answer to the data that is allowed to be used, or is it the answer to the leak? For this, the system will provide relevant and fixed versions of the evidence to a dedicated audit agent.
Figure 3: Running time verification and semantic audit. Structured events are used to verify processes, and the evidence that needs to be interpreted is handled by the audit module. Event types include Expose, resource visibility, Mutate, modify, or control resources, Handoff, submit results, Verify, use evaluation inputs, Reward, read scores, and Release, output logs and feedback.
Four judgments, each answering different questions
BenchShield maintains the benchmark's original scores and passing status, and the alternative report checks the conclusions, avoiding confusion between bugging tasks with bugs and bugging agents for breaches.
Checked: task configuration, execution records and necessary evidence to meet current enabled checks.
VectorExposed: Detected a potential reward hacking path, but this record shows no evidence that Agent used it.
Agent Violation: Evidence that the Agent attempted or used violation paths, including unsuccessful violation attempts.
Inconclusive: Lack of evidence or inconsistency, or the configuration and execution mode of a task that does not meet the system requirements, cannot be determined for the time being.
These labels rely on the scope of the inspection of the declaration and on the available evidence. In particular, Checked only indicates that it satisfies the enabled inspection and cannot be inferred from this that the rating standards fully express the intent of the task.
What questions did the experiments answer?
Based on more than 31,000 Agent run records, the team screened and manually reviewed 456 execution tracks. Of these, 314 were labeled reward hacking and contained a total of 419 exploitation snippets. Data sets from Terminal-Bench 3, SkillsBench and ClawsBench contained naturally occurring and routed exploitations; this ratio does not represent the rate at which all source runs occurred.
When did Agent start exploiting the vulnerability?
The researchers unified the operational steps of each execution trajectory. The median position for the first attempt to exploit the vulnerability was 0.60, and the median position for the first successful exploitation was 0.76, which is about 60% and 76% of the operational sequence respectively.
An agent often has completed some normal operation before attempting to exploit the vulnerability. A pre-run scan can detect potential risks, but it cannot tell whether a run later used the vulnerability; the final answer may not be to restore the mid-range operation.
Figure 4 (c) 4 (e): The location of the exploitation of the vulnerability in the execution process. The left diagram shows the cumulative distribution of the first attempted (314 trajectories) and the first successful observed (40 trajectories), with the null-line being the center number; the right diagram shows the distribution of each exploitation stage.
Another separate dilution experiment provided only a task description to the large model, Agent's visible execution trajectory and final results, allowing it to determine whether reward hacking occurred, with a single test accuracy of about 36%. These materials reflected Agent's attempts, but to determine whether operations affected scores, it was also necessary to know what the evaluation system actually accepted, what the verifier read.
How many known use chains can be detected before running?
In Terminal-Bench 3, the team used the same large models and task packages, as opposed to BenchShield and BenchJack. After reconciling the findings of the five experiments, the full utilization chain recall rates for both were 77% and 23%, respectively.
BenchShield costs $2.05 and BenchJack $5.91 per model for a static audit. BenchShield's model costs come from semantic analysis in static auditing, with no need to call Agent for structured checks themselves.
Table 3: Static test results for BenchShield and BenchJack. Summary statistics of recall rates for five trials, with mean and standard deviations for single trials in parentheses; model call fees for each audit.
Is it possible to distinguish between vulnerability exposures and actual breaches when operating?
Running-time experiments are organized according to clutch task × test case clutch, with a total of 144 clutches of run-able tests. BenchShield gave a clear judgment to 87.5% of them, with an accuracy of about 96%; the remainder returning Inconclusive.
There has never been a single test for the use of a known vulnerability to be classified as Checked. However, there are still cases where it is classified as VectorExposed or there is insufficient evidence, so this does not mean that all exploits are accurately attributed to AgentViolation.
Table 4: Running accuracy results. Accuracy rates only statistically give clearly defined tests, Inconclusive single columns; use tests for known vulnerabilities have not been classified as Checked.
How many problems can isolate verifiers solve?
The team performed a counterfactual analysis based on existing cases: which use paths would be blocked if the verifier were placed in an independent environment? For I1 and I4 statistics, respectively, 82% to 95% of the taskpacks with the corresponding risk existed to block this type of use path through an independent verification environment. This is an analysis of existing cases, not a decrease in overall reward hacking after deployment.
The isolation measures evaluated in the study did not eliminate the problems of I5 and I7. The former included fail-open, i.e. still releasing after checking the error; the latter involved whether the accepted results actually met the task's objectives. They required checking abnormal processing rules and verification standards for the task, which cannot be solved alone in isolation environments.
Figure 8: Scope of protection of the six isolation measures. Figure shows the blocking effect in the formalized model and the proportion of tasks that eliminate the corresponding risk in case analysis.
In addition to scores, what else should be retained in the evaluation?
BenchJack's system audits reveal vulnerabilities that can be exploited in multiple benchmarks. BenchShield extends this research route to the determination of each run: clearly evaluate boundaries, connect operations to scores, and retain evidence to support the conclusion.
For benchmark designers, this means explaining exactly what the task allows the Agent to do, how to submit the results to the evaluation, and what records can be reviewed. For users of the evaluation results, scores should also be used to see the scope of the evaluation, whether the evidence is sufficient, and the semantics of the task that has not yet been solved.