Harness: AI itself is blowing up AI

  It's the first time in the history of the world.

After OpenAI, the old man is gone!

Eventually, the company launched its own Harness engine, the SoL-Pi.

It uses Pi as a backdrop to re-create a Harness Efficiency Layer that allows AI to self-replicate and optimize.

In this rigorous set of automated pipelines, AI itself has become a researcher.

They observe how the Agent works, figure out what steps are taken to burn the token in white and white, propose the program, modify the bug, and then put the program to the test.

The entire search for AI began with 152 candidate directions, ultimately leaving behind four killer architectural mechanisms.

The results of the tests were astonishing.

Token consumption is up to 64%, while the cost of API calls drops 50%-54%.

GitHub is the gateway:

In a professional research scenario, direct savings can range from $8.75 to $13.5 per hour.

The configuration is very simple, requiring only one line of code: "pi install git:github.com/NVlabs/SoL-Pi", which can be installed on an existing Pi.

Can we let it pay the bills of self-improvement before allowing AI to go back to improving itself?

Harness, the complete operating framework used when the model is running, is the best solution at the moment.

Harness is responsible for reasoning, and Harness is responsible for organizing tools, context, feedback, and task processes so that the model can read files, change code, run tests, and proceed according to the results.

The same model, placed in different harnesses, can make the work more efficient.

The current problem is that the task of programming intelligent agents is increasingly long.

From adding a few lines of code, to fixing issues across warehouses, and to working independently for long periods of time, single tasks can last for hours.

During this time, unusual tokens will be wasted and accumulated.

After completing the file, the next step is to run the test, and the model has another round of reasoning. A large file has already been read and subsequent requests are still repeatedly carrying it.

In thousands of diaries, only a few lines are likely to actually influence decision-making, and expensive models need to be read from scratch.

Returning to self-improvement (RSI), it's also not worth the expense. Every time AI tries to improve a system, it costs Token.

SoL-Pi is a problem that needs to be solved.

The final results showed that Sol-Pi used 45% to 49% fewer tokens compared to baseline Pi, costing about one-third less, and maintained an average score of about 94%.

Compared to the original harness of the Codex, the tokens are 35% to 64% less expensive and 50% to 54% less expensive.

So what is so amazing about SoL-Pi? It has to be broken down in detail, its core four mechanisms.

Ingrid's flow of water ultimately left four mechanisms, the knife to avoid mainstream, and the dictatorship to repeat labor.

First layer: action integration, one call to complete editing and verification

Action Fusion is a direct link between two tool calls, and the remaining model decision-making link.

The most common sequence in a basic Pi runtime is to modify a file, receive results, and then call commands to test or build.

The following instructions are very clear, and the "one at a time" in the middle of the model has a great deal of compression space.

Motion Fusion keeps an edit and its subsequent commands in a local sequence, saving the middle model decision link.

Action Fusion directly packs editing and follow-up commands into the same local execution sequence.

Harness completes the modification, runs the command at the bottom, and returns the result after the merger once and for all. The test is normally performed and the results are normally obtained, but the model request in the middle is successfully saved.

This is equivalent to combining two decentralized requests into one efficient closed loop operation.

Second layer: Compression of online context, dynamic billing of costs per subtask

The Online Context Compact addresses the core pain of "when to trigger compression".

The longer the context, the easier it is for historical material to become a burden on mathematics and finance.

However, compression also comes at a cost: rewriting the context may disrupt the reuse of existing KV-caches, requiring re-payment of processing costs.

Therefore, premature compression does not necessarily save money, and past strategies have often been to delay compression actions as much as possible.

On-line context compression will view completed subtasks as potential compression points, and then execute them while waiting for subsequent requests to be able to repay the cost of rewriting.

SoL-Pi reconstructed the time of judgement, dividing major tasks into sub-tasks and re-evaluating each step after completion.

Its core logic lies in precise calculations: only when the "expenditures expected to be saved in the future" can cover the "cost of this rewriting" will the system actually perform compression operations.

Third layer: Output filing and indexing, large files retrieved on demand

ObservationPack, which deals specifically with the issue of "repeat billing" caused by large-scale tool output.

If a large file or a very long result is read for the first time and stays in the previous context, then each subsequent round of requests carries it as before, continuing to consume cache resources.

ObservationPack replaces repeated complete output with a stable handle, while retaining the original content for partitioning.

SoL-Pi's practice is to archive the entire content directly to the local disk, leaving only a stable short sentence handle and a small extract in the context.

This is like archiving the original after reading a long report, leaving only the index and the key summary. When the model needs to look at the details, it can be retrieved directly from the index by page.

Level 4: Small model start-up logs, introducing rigorous evidence verification mechanisms

At the heart of Evidence-Preserving Reducer is the task of assigning the first reading of a long log to a lower-cost model.

Building and testing logs can be tens of thousands of words, and only a few lines of reporting can really affect the next decision. The front-end model is extremely costly to read the full text each time; it can be given directly to a small model to summarize, and it is easy to introduce hallucinations.

Evidence retention returns are subject to rigorous diagnostic feedback only when verifiable archival logs verify their evidence

In the middle, the SoL-Pi adds a rigorous proof-of-confirmation test. After the auxiliary model reads the log, a rigorous diagnostic response must be issued.

The system takes this retrospective and compares it with the original logs of the archives. Only a strictly synchronized version of the original is given to the front-end model.

This ensures that the main model only receives verified information and disconnects error transmission from the source.

This is about the real ambition of Ingrid, the RSI (return to self-improvement).

Since AI can change code, it should be able to change the "systems that produce AI".

But the RSI is burning so much money that it has to pay a token fee every time it tries to make a mistake.

The efficiency of tokens is more difficult to cheat than in direct sequencing. Brushing task scores are easily over-adapted to write rules for specific topics; however, removing repetition over context, compression tooling output, and merging ineffective decisions can migrate these optimizations to different tasks and models.

So a stream of "Agent Research Agent" started:

The team first created 535 verifiable environments, then had AI propose 152 optimization directions, followed by three rounds of screening.

Using historical trajectories to estimate profits without the potential for direct elimination; then allowing AI to modify the code, run experiments, and be punched by the Reviewer Agent;

Finally, the freezing program, validated on a completely isolated held-out mission, cannot lose capacity and efficiency must be maintained.

There were 152 ideas, and only four of them survived, with an average of about 40 coming up with one.

So the core method of SoL-Pi is becoming clear: allowing AI to make a lot of assumptions, then automate experiments, eliminate, verify, and come up with a handful of really effective mechanisms.

SoL-Pi is backed by the Efficient AI team led by MIT Assistant Professor, Han Chang, Director of Research at INVIDIA.

For them, how many tokens are currently saved by SoL-Pi is just a step-by-step step.

What they really care about is how big this AI research aircraft can be.

On the homepage of the official project, the team also buried two highly ambitious longline pencils:

One is "Pretraining the harness" ("Pretraining the harness").

The current 535 environments are still man-made, and in the future the Agency itself will collect tasks, build environments, verify and update its own harness all over the web.

Once computational and environmental diversity are lifted, harness is likely to run out of its own scaling law.

Another is "Efficiency for efficiency".

Using a modified harness to run a larger, less costly, automated research cycle, thereby finding more efficient mechanisms to create cost-compressed profitability effects.

In this aircraft, humans are only responsible for providing initial guidance; once they enter the loop, from research to verification, the entire process of zero intervention; when AI runs the mechanism, people come back to find out what AI has discovered, and reconstitute the rough code generated by the machine into a version that meets industry standards.

Now, the scaling law of the model is still debated; but Harness's scaling law has already begun to be robbed.

And the one who runs crazy on the track is the AI itself.