In the development of the Agent application, everyone must have been entangled with the choice of Harness: the same model accesses different Harness, tool calls, context management and error recovery methods can change, and the end result can be very different.
So, which Harness to choose for different tasks? Is there a general scheme that is stable enough to accommodate multiple tasks? Is the native Harness offered by model manufacturers, or is it necessarily better suited to your own model?
A team of researchers from the South Pacific University of Technology, Singapore, conducted a systematic study on these issues in their latest report, Finding the Right Fit: Model Harness Interactions across Agent Tasks.
The report is linked to:
Experimental code stores:https://github.com/liyix/finding-the-right-fit
The data collection:https://huggingface.co/datasets/yixuanli97/finding-the-right-fit
This report does not make a simple Harness list, but compares the model to the Harness as a whole. The experiment covers the four configurable Harness sets OpenHands, DSH, PI, and openJiuwen, and pairs the two groups of Codex GPT and Claude Code as a reference. The report concludes with a very practical conclusion: The good and bad of the Harness are not fixed attributes, but depend on the specific combination between the model and the task.
01 Experimental background
To examine the impact of Harness on Agent performance, the study first selected four freely configurable Harness models: OpenHands, DeepSeek Harness, DSH, PI, and openJiuwen.
The experiments included Claude Opus 5, GPT-6 Astra, GLM-5.3, Kimi K3 and DeepSeek V4 Pro. The cross-design allowed to observe the performance changes of the same model after replacing the Harness and to compare whether the same Harness remained stable in the face of different models and tasks.
In addition, the study included two groups of natives paired as references: Claude Code and Codex GPT. Because Claude Code and Codex only accessed their own models, they did not participate in complete cross-experiments.
In terms of tasks, the study selected three different task sets: TUA-Bench focused on general terminal operations, ALE-CLI focused on professional workflows, and Terminal-Bench 4 contained higher-difficulty command lines.
Explanation of the three task sets and their evaluation methods
Because the assignments for the three sets differed in composition and scoring criteria, the study did not combine the results into a composite score, but instead compared them separately. The four configurable Harness models, five models and three assignment sets made up a total of 60 4×5×3 results; combined with the 6 native matches of Codex GPT and Claude Code and Claude in the three sets of assignments, the results eventually formed a comparison matrix containing 66 records.
02 Experiment configuration and data
2.1 Harness version and running configuration
The experiment involved four configurable harnesses, as well as two native harnesses from Codex and Claude Code. All harnesses were called through the OpenRouter model and routed to the official servers of each model; the reasoning intensity was unified into high, up-context management, retesting, and turnaround limits to maintain the default settings of each harness. TUA-Bench and Terminal-Bench 4 run on Harbor v0.22.0, theALE-CLI set uses a task autonomous operator, and on ALE-CLI all harnesses access the 14 computer operating tools provided by the task set.
The following are the versions of each Harness:
OpenHands: openhands-tools version 1.44.1.
DeepSeek Harness ((DSH): Version 0.1.1-rc.2.
PI: v0.84.4 release;
OpenJiuwen: version 0.1.18
Codex: 0.150.1, only available with GPT;
Claude Code:2.1.251 is only available with Claude.
2.2 Method of counting
In order to unify the methods of counting different combinations as much as possible, the study used a fixed task set divider:
TUA-Bench: Mission 120;
ALE-CLI:99 mission;
Terminal-Bench 4:63 mission.
The three task sets are scored according to the fixed number of tasks (TUA-Bench 120,ALE-CLI 99,Terminal-Bench 4 63), TUA-Bench and ALE-CLI allow for partial scores, and Terminal-Bench 4 is scored according to passing or not passing.
2.3 Experimental data
Below are the full results of the three mission sets under the aforementioned qualities:
Figure 1 Full results of the TUA-Bench
OpenJiuwen has the widest advantage over the TUA-Bench: among the four configurable Harness models, it scores the highest on five models; even if it joins the native Harness, it still leads on four of them. It is noteworthy that the differences in lead scores between different combinations are evident: OpenJiuwen is only slightly ahead of OpenHands with GLM and significantly ahead of PI with Kimi.
Figure 2 Results of the 99 local Docker missions of the ELA-CLI
At ALE-CLI, the advantages of Harness began to differentiate: Claude's highest score came from Claude Code, GPT-6 Astra was better suited for PI, Kimi still worked best with openJiuwen, and GLM and DeepSeek's highest score came from OpenHands.
Results of 63 non-H100 missions from Terminal-Bench 4
In the more challenging Command Line Task Set Terminal-Bench 4, the model's adaptation differences with Harness are more pronounced: OpenHands is best suited for Claude Opus 5, PI is best suited for GPT-6 Astra, DSH leads on GLM and DeepSeek, while openJiuwen continues to score highest for Kimi.
These three tables are not a simple Harness chart. It is more important to consider whether the original leadership relationship can continue when the model, Harness or task changes.
03 Analysis of the results of experiments
The results of the 66 experiments show two seemingly contradictory phenomena: on the one hand, replacing the Harness may change the ranking of the model, and the best Harness of the same model will also change the task; on the other hand, the combination of some models and Harness can maintain an advantage over multiple task sets.
This means that it is difficult for Harness to break away from specific models and tasks to form a fixed ranking.
3.1 Harness Significantly Affects Model Performance
Terminal-Bench 4 shows a typical ranking reversal.
Under the fixed counting range of 63 missions, Claude Opus 5 completed 36 missions through OpenHands, GPT-6 Astra completed 31 missions, Claude leading by 7.94 points. However, after replacing PI, Claude completed only 19 missions, GPT completed 38 missions, and GPT in turn led by 30.16 points.
Figure 4 shows the performance of Claude and GPT on Terminal-Bench 4.
From OpenHands to PI, Claude Minor completed 17 tasks, while GPT completed 7 more. Harness's change did not cause the two models to rise or fall simultaneously, but had the opposite effect on them.
Therefore, the advantages of a model observed in a set of Harnesses cannot be directly extended to other Harnesses. The model ranking describes not only the model itself, but also the adaptive relationship between the model and the tool interface, context management and execution processes.
3.2 Best Harness Changes with Tasks
Even if the model remains unchanged, its best Harness for different tasks may vary.
Figure 5 Harness the highest score of each model in three sets of missions
Of the five models, the four highest scores for Harness changed with the task set. GLM-5.3 and DeepSeek V4 Pro were the most noticeable changes: on TUA-Bench,ALE-CLI and Terminal-Bench 4, the highest scores for both came from openJiuwen, OpenHands and DSH respectively.
The GPT-6 Astra has the highest scores on the TUA-Bench with openJiuwen, and is better suited to PI for the other two sets of missions. The highest scores on the TUA-Bench and ALE-CLI are from Claude Code to Terminal-Bench 4, where OpenHands performs better.
Kimi K3 was the only exception. It achieved the highest score of the model in all three mission sets combined with open Jiuwen, leading the other configurations rated at 5.61, 6.91 and 11.11 points, respectively, showing a more stable adaptation advantage.
3.3 Original mating does not mean the best choice
The native Harness offered by model manufacturers is usually thought to be more familiar with their own models, but experimental results have not shown that this advantage can be steadily sustained.
FIGURE 6 Harness comparison with the highest score of the same model
Claude Code scored Claude's highest score on TUA-Bench and ALE-CLI, but lagged behind OpenHands on Terminal-Bench 4.
These results suggest that the native Harness is not necessarily the best match for the model itself. The same model can perform better on some tasks after accessing other Harnesses. Therefore, the selection of the model does not only look at whether the harness is suitable for the kit, but also requires a comprehensive comparison of task completion and running costs. Harness's scalability is also an important consideration, especially when it is necessary to switch models, access more tools or cover multiple classes of tasks.
3.4 Higher costs don't necessarily lead to better performance
Harness affects not only task scores, but also changes in the number of model calls, Token consumption and overall running costs. The paper, according to OpenRouter's Unified Price Calculation Model API cost, finds that higher inputs do not stabilize and replace better performance.
For example, GPT-6 Astra on Terminal-Bench 4, the total cost of PI is about $293.30, with a score of 60.32%; the total cost of DSH is $1,256.48, which is about 4.3 times PI, with a score of only 52.38%.
Figure 7 Harness combination of scores on three missions versus cost of single missions
The cost difference is mainly due to the way Harness organises model calls and the context. Lower costs are usually accompanied by fewer model calls and uncached Token inputs, but do not necessarily mean fewer model outputs.
There is also a significant difference in the contextual replicability of Harness: openJiuwen has 94% of the input tokens from caching, while the other Harness has a caching ratio of 49% to 77%.
Therefore, the Harness evaluation does not only look at the final score or model single price, but also combines the number of calls, uncached inputs, cached utilization and task completion to compare the actual input output of the target task in a complete configuration.
04 Why is this happening:
From the trajectory, see what Harness did.
The scores show only the results, but do not explain why the difference is. For this reason, the report further analyzes the same tasks and models, only Harness's different matching trajectories, and observes how the model responds to failure signals.
The study selected 10 sets of OpenHands CPI pairs on Terminal-Bench 4, covering five models and 192 failures, and further analyzed the trajectories of six sets of Kimi K3 on openJiuwen and under PI.
Of the 192 cases, 180 response actions were initiated by the model actively. Diagnosis or fixed point repair is the most common treatment method, with 133 successes, 116 of which were successful. The real effect is often not whether the model will be repaired, but whether Harness will turn the failure into feedback that the model can use.
4.1 Mistakes are not scary, and the key is whether feedback is accurate.
TUA-Bench's 056-move-textbox-left task provides a typical example. Kimi K3 made the same mistake under openJiuwen and PI: omitting the exit command when executing the GIMP script through the pipeline caused the command to wait for input, with the difference that openJiuwen responded to the error in a timely manner and eventually fixed the error in the next round of tasks.
Under PI, the shell default is no overtime, and the model is not actively set, so the command continues to hang. 39.8 minutes after the task reaches the deadline, no final file is generated, scoring 0.
In openJiuwen, the error message was returned to the model after the first two GIMP calls failed. When the third call was hung up, the shell returned overtime after 300 seconds. The model found that the script was missing the exit command and confirmed that the image had been generated, and then verified the size of the canvas, background color, and text location. The entire task took 11 minutes to complete, scoring 1.
Figure 8 shows the mating trajectory of the Kimi K3 on TUA-Bench mission 056-move-textbox-left.
As can be seen from the diagram above, openJiuwen returns overtime after the command is hung, and the model completes the diagnosis and recovery accordingly; the PI does not return the signal, running until the final deadline. The default overtime is not unique to openJiuwen, and OpenHands and DSH also limit the order duration. This case illustrates not which Harness is absolutely stronger, but whether the failure can effectively return feedback, which directly affects whether the model has a chance to recover.
4.2 Harness adaptivity is associated with model habits
The same design can have completely different effects on different models. GPT-6 Astra has 56% and 67% of Shell calls in PI that actively set overtime, so PI has no default overtime and little impact on it. GPT-6 Astra scored the highest score of the model on Terminal-Bench 4 and ALE-CLI with PI.
How much Harness should intervene cannot be categorized simply according to the model's strengths and weaknesses. The model is already able to handle the link itself, so reducing the intervention may be more appropriate; the link that the model is easy to miss requires Harness to provide the necessary support.
4.3 Reasons for stable adaptation of models and harness
Kimi K3 is the only model in all three missions to have the highest score from the same Harness: openJiuwen. Per-mission comparisons show that this advantage does not arise from a few missions: on TUA-Bench,ALE-CLI and Terminal-Bench 4, openJiuwen scored higher on 22, 25, and 8 missions respectively. Even if the top three missions are eliminated, they still lead by an average of 3.11 points, 3.88 points, and 6.35 points.
Figure 9 provides a comparison of the Kimi K3 with the task-by-task configuration of each task cluster.
The trajectory analysis gives three possible reasons.
Tool Interface: In OpenHands, Kimi repeatedly sends editing calls that lack the necessary parameters, triggering terminations after repeated failures. These terminations occur 55 times, 48 of which are from Kimi.
Overtime commands: openJiuwen returns overtime information when the command is unresponsive for a long time, making the pending state return to model-processable feedback.
Interruption: When the output is interrupted due to length limitations, openJiuwen retains the existing reasoning and suggests that the model continue. In the 6 groups of Kimi paired trajectories that were analyzed in the report, this mechanism played a decisive role in 2 of them.
These results show that the stable performance of openJiuwen and the Kimi K3 is not the result of a single mechanism, but rather a combination of tool interfaces, error feedback and rewriting strategies.
05 Summary: The model should be evaluated together with Harness
The report's most direct reference to the Agent selection model is to evaluate the model and Harness as a whole on targeted missions.
If the model has been determined, it is necessary to compare its actual performance after accessing different Harness; if Harness has been determined, it cannot be directly applied to the model ranking under other Harness. When the Agent moves from one business to another business, the original combination should also be re-verified.
For Agent systems that need to handle multitasking tasks, retaining model and Harness configuration space is valuable. However, allowing a mouse to select a mouse and a mouse to automatically select a mouse is not the same thing. The best configuration in the report is the resulting comparison, and does not prove that the system is able to decide in advance which Harness should be used in the face of new tasks.
If automated selection or routing strategy is to be verified, configuration selection must also be completed on the development set, and then tested on unseen tasks, and compared to the best fixed Harness selected on the same basis as the development set. During actual deployment, delays, call costs and tool execution costs should also be taken into consideration along with task completion.
The evaluation unit of the Agent should not be just a model or Harness, but should be the complete configuration of the model, Harness and the task together.
When a task changes, model upgrades, or Harness updates, this adaptation relationship is worth re-evaluating. Instead of viewing the model or Harness rankings separately, directly testing the performance of the entire configuration in the target workload is closer to the real deployment needs of the Agent.