In the past, LlamaFactory allowed everyone to micro-enlarge the model; now, PenguinHarness wants everyone to build an Evolutionary Agent.
PenguinHarness is the world's first to support multi-agent autonomous Harness:
0.2 yuan to build a new Agent from scratch. It takes only half the time of the Codex, costs less than 1/200 of the Codex, and results closer to delivery.
With less than a penny, the Agent has become stronger. No manual tweaking is required, and the Agent's accuracy rate has increased from 50% to 90%.
More than 1000 models, cheaper and faster to run. Support multi-mode, trajectory observation, cost contrast, and multiple users simultaneously.
This is an open-source address:
Link to the main page:
Let the agent build the agent automatically.
The author of LlamaFactory, Jay-Way, recently launched a new project called PenguinHarness, which allows you to automate the construction, evaluation and continuous improvement of Agents, moving from "Manual Optimization Agent" to "Agent Self-Generation".
PenguinHarness is a native self-developing Agent Harness that is primarily lightweight, open-source, and easy to use. It is based on the original Agent kernel, both GPT-5.6 and DeepSeek V4.
The project was initially conceived as a collaboration between AI4AI, which is hotly anticipated for 2026, and RSI, which aims to return to self-evolution.
Recursive, founded by Meta's former research director, Honda, focused on exploring the potential for AI's regression to self-improvement, and its first blog has proven that Agent can train models and optimize GPUs.
XYZ AI Lab, founded by MSRA's former vice-president Jiang, has also successfully explored a path to training after completing the Deep Research model by running hundreds of agents.
PenguinHarness is committed to making the best use of open source Harness, allowing Agent to evolve automatically without the need to manually optimize Agent.
Its first capability is to allow the Agent to automatically build another Agent. We compare the real-world performance of PenguinHarness and OpenAI Codex with one of the most classic examples: "to allow the AI to create a RAG application, implement block retrieval, streamline returns to commonly understood answers, and bring with it references".
From the video, we can clearly see that PenguinHarness not only takes less time and costs tens of times less than Codex, but also produces RAG Agent applications that can land almost directly.
The left hand side responds with fluent language, fluent output, and source citations. The results of Codex are mixed in Chinese and English, lacking fluent output.
In the past, the development of such an Agent application required framework selection, model deployment, access tools, writing hints, repeated testing and modification, with each application starting almost from scratch and taking a team of weeks of effort.
Even with development tools like Claude Code, which are unfamiliar to the existing Agent framework, it is difficult to do as easily as developing front-end web pages.
For PenguinHarness, you just need to be clear about the needs, and it can start right away: generate a scripting shelf, Prompt, install and optimize Skill, and build the front end until you have a fully-running Agent app delivered to you.
Jay-Z also reveals the underlying principles behind PenguinHarness: it puts the file system as the only source of truth, Agent is a document, hints are a document, and conversation history is a document.
The framework is solely responsible for packing the file into a workable Agent object, while creating a new Agent with no duplicate pasting of the file.
While running, PenguinMessage serves as the only messaging protocol that, like a web document, switches between models, environments, and users. This unified and lightweight interface allows PenguinHarness to easily access any code.
PenguinHarness is both an Agent framework and an Agent entity, and no one knows himself better than it.
Locally replicable Agent Evolutionary Closed Circles
PenguinHarness's long-term goal is for everyone to have a locally evolved Agent.
Building an Agent is from 0 to 1, optimizing an Agent is from 1 to 100, which requires many challenges.
「Chaning agents is easy; improving them is hard.」
In a discussion about evolution, Liu Wei argued that an agent is essentially a set of codes and documents that are easy to modify and difficult to improve. The big model is already a random probability function, and the agent is more of an autonomous system full of uncertainties.
The author of ReAct, Suzanne Hu, has said that evaluation is the second half of the big model. For Agent evolution, the phrase also holds true that a good set of evaluation criteria is fundamental to evolution.
In order to explore the exact method of evaluation, Liu Wei took more than half a year to lead the team. The biggest obstacle was that the existing evaluation was mostly a test set rather than a training set.
To do this, the team even created a new evaluation benchmark, GDPRevo, specifically designed to test Agent's ability to evolve, covering six real-world scenarios: medical, financial, and legal.
PenguinHarness incorporates the core idea of GDPevo to evaluate and optimize the Agent's skills for use in the box. After calling Skill, PenguinHarness launches multiple smart entities to complete the Agent's optimization process, which is called "automation".
Think of it this way: being an Agent can help you predict the game more accurately, come up with an investment strategy, deal with the e-commerce after sales, but this Agent doesn't have a high initial accuracy rate. In the past, you needed to manually adjust the tip words and build the workflow to improve accuracy, and now PenguinHarness can automatically help you complete Agent optimization.
As you can see from the video, PenguinHarness has completed multiple iterations, and the score of Agent has increased from 53 points before optimization to 95 points after optimization. Using the official version of DeepSeek V4 Flash, the entire process costs only 0.5 yuan of Token.
Behind this is a multi-intelligent collaboration process. Mr. PenguinHarness developed a Benchmark Builder with the task of generating a set of questions and answers around the ability to target and calibrate the difficulty of the questions through repeated testing.
After entering the optimization phase, the three Agents, who assume different roles, will form an automated loop, repeating the following four-step process:
1. Organize Evaluations: Optimizer Agent organizes multiple Evaluator Agents in parallel according to the number of titles and the number of runs;
2. Independent score: Each Evaluator Agent launches a separate evaluated Agent answer and returns the score to the Optimizer Agent based on the Rubrics that only the Evaluator can see;
3. Analysis Optimization: Optimizer Agent summarizes the evaluation results by viewing and analyzing trajectories, locating causes of misalignment, modifying the test Agent's cue words, Skill, or configuration, and generating the next version of the test Agent for the candidate;
4. Verification iteration: Evaluator Agent re-evaluates the candidate version, Optimizer Agent decides whether to accept the candidate version based on the results, and proceeds to the next iteration.
The entire process is as shown in the figure below, and during the entire optimization process, only the candidate version receives a strictly higher score, and the Optimizer Agent returns to the previous version if the score remains flat or decreases.
From one-time delivery to continuous evolution, this is the answer PenguinHarness gives to the next generation of Agent Harness.
And the contract between the Evolutionary Agent
In order to make Agent Evolution more stable, PenguinHarness has designed a framework that must be adhered to.
The contract specifies where the boundaries of Agent evolution lie, and we're going to focus on a few points here.
1. Restrict the range of modifications to the Agent to hints and Skills that do not touch the Harness kernel. This is to prevent the Agent from creating security risks in the course of evolution, such as violating the original authorization auditing rules.
2.Agent automation should be able to take snapshots and roll back, and if negative effects occur, the framework should be able to return the Agent version to its previous state.
3. Target Agent can only see the titles during evolution, not Rubrics. This is to prevent Agent from taking shortcuts, getting higher scores by answering backwards or catching scores, and preventing Reward Hacking.
4. All optimization processes must be documented and users have the right to audit the optimization process and provide automated explainability.
The contract was also formalized by PenguinHarness as a CONTRACT.md file, which is easy to copy and distribute.
Other interesting features of PenguinHarness
While the team is exploring Agent automation paradigms, PenguinHarness has also installed a number of interesting capabilities.
First, PenguinHarness has built-in LlamaFactory, vLLM, Ollama, etc. Skills related to model training and deployment, and you can use PenguinHarness to try to start model training and deployment using a method similar to Vibe Coding.
Second, PenguinHarness integrates a unified model gateway that supports access to more than 1,000 online and local models, and is fully compatible with the latest models such as Kimi K3, Gemini 3.6, and Ling 3.0 Flash.
Additionally, PenguinHarness also has an eye for DeepSeek. You can configure a proxy model with a visual model, implement DeepSeek as a main driver, and a visual proxy model as a co-driver, so that DeepSeek can also see your self-generated web pages and PPT and make adjustments based on visual feedback.
Not only that, PenguinHarness itself also supports fine-grained behavioral trajectory analysis by observing timelines and analyzing Agent performance cardpoints.
In addition, you can use PenguinHarness as a blank Agent template similar to the Pi Agent, the built-in Agent of PenguinHarness maintains a minimalist design, uses Shell as a general interface, completes tasks with very few tools, optimizes features such as Subagent as a core performance target, and system prompts are only a tenth of Claude Code (1,300 vs 15,000).
On a rigorous benchmark, PenguinHarness performed best with its DeepSeek model, while costing only a tenth of other products.
In the process of development, the team has actually deployed PenguinHarness in two production scenarios. One medical examiner used PenguinHarness to produce reports similar to those of medical experts to verify Agent, a report that used to take 30 minutes to verify, and now only takes a few tens of seconds.
One manufacturer applied several of the Agents built by PenguinHarness to streamline operations, keeping the equipment up to date and recovering automatically, reducing the downtime of the production line by 65% and improving output by almost double.
PenguinHarness is open source under the Apache 2.0 protocol, supports Linux, Mac, and Windows systems with one-click installation, can be deployed to local machines, can be placed on servers remotely via browser access, provides multi-user isolation, and supports team collaboration, can be used both as an Agent auto-build platform, and can be used as a local alternative to Codex to complete tasks at a lower cost.
Looking forward to more attention being paid to the PenguinHarness open source project to build a variety of interesting Agent applications. PenguinHarness will work as an Agent production engine so that everyone can enjoy the productivity boost that comes with the evolutionary Agent.
The team behind PenguinHarness
PenguinHarness was founded in 2025 by the PrismShadow team, and focuses on building an agent-based platform that continues to learn in real-world business, transforming business knowledge, workflows, and business feedback into deployable, measurable, and sustainably optimized agents.
PrismShadow's founding team includes:
LlamaFactory, the author of the open source framework GitHub 70,000 stars, focuses on building more user-friendly models and Agent infrastructure.
Dr. Davis, former IBM senior researcher T.J. Waston, was a professor at the University of California, Riverside.
Shen Shui: Formerly a founding member of the Baidu NLP division, he served as Senior Director of Alibaba and Vice President of the Jindong Mathematics Department.
Hu Jun Ho: A PhD student at Peking University, who proposed that the location is unrelated to cache, has been involved in the research and development of the MiMo V2 model, the Hy3 series model.
Chen Shei: Professor at the New York University School of Business, and Dr. Carnegie Mellon University, who was the chief scientist at Amazon.
From LlamaFactory to PenguinHarness, the team's mission has always been to transform sophisticated AI capabilities into truly easy-to-use, reliable, and efficient crowdsourcing tools.