On August 13, DeepSeek's open-source Agent Harness dsh pushed the concept of what was originally a biased engineering context to the center of discussion across the AI industry.
In addition to the model, why do we need Harness?
The answer actually comes from the significant changes that are taking place in Agent.
The ultimate capability of an agent today is no longer simply equivalent to the capabilities of the underlying model. In addition to the model, there is context management, tool calling, memory, task demolition, environment execution, permission management, and results verification. The real determinant of whether an agent can perform complex tasks steadily is whether these components can be organized into a stable, executable workflow system.
This is what Harness describes as this layer of ability.
Clearly, understanding Harness as a coding agent as a product form would seriously underestimate its significance. Almost all complex intelligent processes require a system layer that is responsible for planning, coordinating and verifying.
Recently, MirroS, a collaboration between research institutes such as Ningbo, Berkeley, MIT, xbench, and Ingrid, released HarnessEval, which further incorporates the concept of Harness into assessment systems.
If the AI being evaluated has gone from a model to a complex system, what kind of evaluation is effective enough?
Blog:
Code:
Project Page:
HarnessEval's answer is not to add a set of indicators or introduce a larger Judge model. It is to build its own Harness for Evaluation.
If Agent Harness is responsible for organizing the model to complete the task, then Evaluation Harness is responsible for organizing the evaluation of how to form credible judgments: understanding cases, planning paths, selecting skills, calling tools, gathering evidence, validating reasoning, and making each score traceable.
From Metric to Harness, from static benchmarks to executable rating systems.
When the model becomes the system,
Evaluation must also be a system.
In the past, we used to think of models as a single function: input prompt, get output, and then use a set of indicators to compare results. This evaluation method is suitable for borderline clarity, answer-determined tasks. But Agent and interactive generating systems are no longer a one-time input-output mapping. They can read files, retrieve networks, execute code, pass between multiple tools; they can break down a long-term task into dozens of stages and continuously modify plans based on environmental feedback; they can also schedule multiple sub-agents, each responsible for planning, execution, and verification.
At this point, the evaluation faces not only the question of whether the final answer is right or wrong, but a dynamic process:
Did the system understand the task correctly?
Is it using the right tools?
Is the median state consistent?
Is there a correct causal relationship between actions and outcomes?
Is it a failure of the ability to model, or of the tools, observation, or evaluation itself?
Traditional benchmarks often pre-defined a set of relatively fixed assessment processes: a set of test data, a set of fixed rubrics, a number of indicators, and the final aggregate score. For complex tasks, this approach is increasingly easy to encounter a fundamental contradiction: different cases do not really need to be examined in the same way.
In the face of a complex case, we usually first understand what happened and then decide the question that is most worth checking out; to locate the problem based on the goal of finding the evidence, and to call on additional tools when necessary; and to continue the investigation if the evidence is insufficient, rather than giving a definite answer with a blink of an eye.
In other words, evaluation itself is also a process that needs to be understood, planned, investigated and verified.
The starting point of HarnessEval is to demonstrate the process that was originally implicit in human expert judgement: dismantling assessments into programmable, summoned, verifiable modules and handing them over to intelligent body systems to execute.
Evaluation thus no longer merely executes a set of pre-written rules, but begins to have its own reasoning process.
This is not a longer rubric.
It's a workflow assessment.
The core of HarnessEval is not how the LLM replaces a conventional indicator, but how the benchmark is executed. In each test case, the evaluating intelligent body first understands the context and the intent to evaluate the case, and then selects the skills that are truly applicable from the replicable Skill Library. Each high-level problem will continue to be broken down into measurable subproblems, handed over to different sub-agents or diagnostic tools to execute.
The whole process can be summarized in four stages.
Plan to understand the case and then decide what to measure: evaluate the intelligence to read the initial state, type of action tasks and evaluate objectives, generate evaluation plans that correspond to the current case.
Route is used to select the applicable skills rather than running all the indicators: the system calls the skill from the Skill Library for the current case and records why one check is enabled and why another is skipped.
Decompose also breaks down abstract judgments into verifiable evidence: high-level problems are broken down into sub-problems of target positioning, state tracking, timing changes, cause sequence, structure maintenance, and so on, and then handled by specialized sub-agents or tools respectively.
Verify first audits the evidence and then delivers the scores: the main intelligence experience validates the quality of evidence and the logical relationship between the branches. The final output is not just a metric score, but a complete evidence tree.
This tree of evidence records what was measured, why it was measured, what tools were used, what evidence was found, and how the evidence supports the final conclusions.
Therefore, HarnessEval delivers two layers of results:
A layer is a quantitative score used to compare models with rankings;
The other layer is the verifiable, reproducible, accountable evidence and reasoning structure.
Benchmark is no longer just an offline calculator, but a truly functioning rating system.
How to make the world model
As the first test site for HarnessEval
HarnessEval first landed on the interactive world model because it was a high-pressure testing ground for Evaluation Harness:
It is much more difficult to evaluate a generated world than it is to judge whether a text is correct. A video can portray a beautiful image but perform the wrong moves; it can complete the changes in sight but forget about the entire scene after leaving the camera; it can produce movement that seems reasonable but violates contact relationships, speed changes or causation sequences; it can even add people, objects or events in space without a hint. Humans are able to locate objects naturally, track the state, understand the intent of the movement, and judge whether the world is consistent in time.
Addressing these highly context-dependent, time-space-evident, and specific failure patterns, Mirros opens HarnessEval-w: evaluating three-dimensional organizations from observing quality, state transfer correctness, and world persistence, and selecting, combining, and validating different Skill based on case dynamics.
Agents can choose which skills to use depending on the dynamics of the scene
The focus here is not on establishing a longer list of indicators, but on breaking down each Skill into a series of sub-questions that are easier to verify and explain. For example, in the face of a change in the state of a specified object, the system may check successively: whether the target exists and is clearly visible, whether the change occurs on the correct object, whether the expected state is finally established, whether the key scenarios are maintained, and whether there are irrelevant changes in the process. In the face of a physical collision, it may call for target tracking, time overlap verification, speed estimation and cause sequence checks.
Each skill derives a series of subagents to solve complex problems and measure model capabilities from different angles.
The ultimate goal of HarnessEval-W is not just to produce a numerical score. We hope to eventually form a layered evidence tree, complete records of what is being measured, which specific tools provide visual evidence, and a complete logical chain that supports the final score. With this executable, traceable, verifiable evidence chain, researchers can not only identify the source of location errors, but also improve model design training at the next stage.
What does Harness for Eval mean?
HarnessEval-w is not just a set of new world model benchmarks, it represents an upgrade of the rating form: the core assets of the benchmark will be expanded from data and metrics to skill routing, tool calling, evidence verification, and a continuously growing Skill Library.
Models can be upgraded through test-time scaling, and the evaluator should also put in more computation, more in-depth search and verification. At the same time, different tasks require different assessments of Skill, and by combining expertise on demand, Eval can cover complex scenarios such as code, search, robots, and world models.
The evaluation results will also be traceable from a single score to evidence. When the system encounters problems beyond the limits of capability, it should also actively expose its own skills and evidence gaps. Therefore, the benchmark is no longer a fixed set of rules, but a living system that can continuously expand and improve itself.
I'm not sure if I'm going to be able to do that.
The next generation of Eval shouldn't be just a foot.
Harness' redemption means that the industry is realizing the value of systems beyond models. However, the Harness era belongs not only to Model but also to Eval.
When AI gradually moves towards recursive self-improvement (RSI), evaluation is no longer just a metric outside of the model, but becomes the key feedback in the self-improvement closure. What evaluation focuses on, the model optimizes in what direction; what evaluation ignores, the system may amplify in the iterative iteration.
The World Model is MirroS' first test site and an important step towards the Physical RSI. In the future, Eval will not only understand context, planning evaluation, combinational skills, scheduling tools and verification evidence, but also examine itself, discover capability gaps, and evolve with the evaluated model.
This is exactly what the HarnessEval is all about: from Metric to Harness, from fixed rubrics to executable workflows, from index numbers to transparent evidence trees.
Let evaluation not only measure intelligence, but also become a system that promotes the continuous evolution of intelligence.
Eval's Harness era is beginning.