This article is about Harness's evolutionary design.

AutoDesign：

Let Harness Learn to Self-Evolve

The development of future multi-intelligent systems, which require model parameters to evolve in sync with Harness, cannot ignore either end.

Multimodal input-output tasks require a long-range Agent System: using coding as the basic operational capability, combining tool calling, visual perception and feedback, environmental interaction, and ongoing self-modification and iteration, to gradually reach the level of human experts dealing with real complex tasks. In this system, model parameters provide the basic ability to think, reason, and act; Harness is responsible for organizing processes, constraining behavior, and enabling models to accomplish long-range tasks more efficiently and accurately.

Currently, model training has formed a more mature industrialized pipeline: through mass data learning parameters, then post-training and reinforcement learning constraining behavior. In contrast, Harness's iterative optimization still has many unanswered questions: which modules and components should be modified each generation; how large the scope of change should be; what mechanisms and standards should be accepted for an update; how should an evaluator be designed; what indicators should be used in the face of subjective tasks; how to locate and process performance reversals; how to come up with new solutions and ensure that the system continues to evolve in the right direction?

AutoDesign chose multi-modal design, a long and complex task to study the evolution of Harness.

• The technical report is titled: AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design.

• The main page of the project:

• The code repository:https://github.com/Yaxin9Luo/AutoDesign

• Technical report:

• Demo：

Since the open source community previously lacked a set of posters Benchmarks that integrated multiple metrics and interdisciplinary standards, we built PosterBench and a complete Evaluation Protocol to generate results from seven-dimensional evaluations: authenticity of the source, content coverage, information density, visual quality, editing accuracy, overall readability and aesthetic performance.

The rating system uses Hybrid Evaluation: subjective quality is assessed by VLM Judge, cross-sectional, text overlap, and elemental proportions are calculated explicitly, and tested by Rule-based Python algorithms. PosterBench contains 100 papers and provides PosterBench-mini, which consists of 10 papers, for quick testing and comparison.

In addition to the Design Agent main track, we also open Source the Fixed DesignHarness CodingHarness track and control variables through different sub-tracks, evaluating CodingHarness, the model itself and the quality of DesignHarness, respectively.

Based on our proposed MetaHarness Optimization method, AutoDesign's scores in the Paper-to-Poster task continued to improve with iteration: 49 → 53.2 → 68.8 → 69.7 → 73.5 → 76.3 → 78.6 → 80.9

After multiple models joined DesignHarness, the average improvement was 12.4 points; under the same Agent and model configuration, AutoDesign was 7.45 points higher than Claude Design. In a complete poster design, Agent summoned 253 tools, performed 11 rounds of editing cycles, and controlled the total cost within $3 USD. This further demonstrated that the automated DesignHarness can be generalized to different models on different Coding Harness models.

AutoDesign continued to evolve DesignHarness through self-optimization and manual guidance, bringing the Coding Agent of all types up 5.0 to 19.6 points to a high of 81.5 points.

AutoDesign methodology details

DesignHarness consists of five components: context and memory, tools and specification, running time, task sorting, and evaluation and feedback. Together, they determine how the system reads multi-module sources, calls tools, generates work, discovers problems and completes deliveries.

AutoDesign evolves through embedded internal and external cycles. In the internal cycles, the Designer calls the tools from the source context, generates editable HTML, and makes local revisions in combination with rendering results, rule verification, and feedback from VLM Critic.

In the outgoing cycle, MetaHarnessOptimizer summarizes the Rollout trajectory, tool calling, rendering diagnosis, and Evaluator feedback in multiple tasks to identify repeated failure patterns. Subsequently, the Optimizer Code Agent introduces bounded code updates to a specific Harness component. To ensure that benefits can be attributed, each outgoing cycle only allows one component to be modified; candidate updates will only be accepted and written to the next version of DesignHarness when the training task performance improves and the independent Development Set does not degrade.

Changing only one component at a time and using independent development kits are two important principles that we have summarized from a large number of experiments. The first is to avoid multiple changes occurring simultaneously, leading to Credit Assignment confusion; the second is to prevent the system from remembering only training tasks.

As a result, the resulting DesignHarness is not a fixed set of Prompt, but a replicable design production system. It is capable of constructing a context on the source, coordinating tool calls, generating and checking editable outputs, while retaining traceable execution evidence to support subsequent revisions and the next round of system evolution.

Design Harness makes use of Critic to generate and modify the work, forming an internal loop; the external loop, by running the Harness in the design task, evaluating the output, proposing updates to individual components, and deciding whether to accept or reject candidate updates, continuously optimizes the Design Harness. The entire process can be run autonomously or selectively incorporated into manual instruction.

Why did you choose the paper to the academic poster as a verification scenario?

We use Paper-to-Poster as the primary verification task. Unlike commercial posters that focus on style and visual ambiance, academic posters need to compress long, multi-form papers into a single-page dissemination medium, while ensuring fact traceability, core results completeness, graphics and numerical accuracy, editable and editable content. A poster can't handle scientific dissemination, even with visual refinement, if it misses a numerical value or omits a core contribution.

Well, it's not good, you can't just look good on your face.

In order to use an optimised system of unified, independent standard evaluation, we published PosterBench. The main track covers 100 papers in five disciplines, while PosterBench-mini offers a low-cost test set of 10 papers. The evaluations measure the results from seven dimensions: factual fidelity, content coverage, information density, visual evidence, layout, readability and aesthetic performance.

PosterBench uses the rules algorithm in conjunction with the Rubric-based VLM Judge. The rules algorithm is responsible for checking programmatic problems such as text overflow, element overlap, spatial utilization and numerical consistency; the VLM Judge combines the source of the paper, evaluates visual expression and the quality of scientific dissemination. In addition, we also systematically blindly evaluate humans to test automatically the consistency of ratings with human preferences.

PosterBench is a set of freeze evaluation protocols used after optimization has been completed, and is not directly used as a scanner to drive internal and external cycling updates. This isolation can reduce the risk of systems ultimately evaluating over-fitness.

Benchmark's rules drive visualization of rating algorithms, and which parts and how they are monitored respectively.

AutoDesign also provides a client that can be deployed locally. After the agent produces content, the system loads the results into a visualized canvas, allowing users to continue editing in three ways, and after the editing is complete, the system supports exporting formats such as PPTX, PDF, etc.

1. Multiple rounds of conversations with the Coding Agent to modify content in real time;

2. Select and circle the areas that need to be adjusted to make a request for modification to the Agent in the form of multiple comments;

3. Use manual editing functions in traditional design software to adjust colors, colors, fonts and spaces, or insert or remove images.

Under the same Claude Code + Claude 4.8 setup as the PosterBench Main Track, AutoDesign scored 78.32 points, more than the closed-source commercial system Claude Design 7.45 points. After learning that DesignHarness had access to seven group control variables experiments, Coding Agents, the average score increased from 54.99 to 67.39, an average of 12.4 points.

Future prospects: Starting with a poster

But more than posters

AutoDesign is the first project in our research pathway, and in the future, we will continue to expand from posters to more modular content such as PPT, web pages and academic lecture videos. The current system has been able to turn papers into these four media, but there is still room for optimization in methodology and effects.

The longer-term question is: Can a multimodal-in, multimodal-out Agent System continue to accumulate human preferences and mission experience into system-level capabilities and achieve Recursive Self-Improvement?

Welcome to experience code, issues, and PRs to participate in the next phase of the system's collaboration.