As the capabilities of open-source models continue to improve, they have been able to generate multiple high-quality candidates at extremely low cost, and further use the models themselves to verify, score, and filter these results. The latest experiment by the Stanford team found that using DeepSeek V4 Flash + LLM-as-a-Verifier to "self-verify" could outperform Claude Fable 5 on Terminal-Bench 2.1 at about 11 times the overall cost.
Specifically, the team first allowed the DeepSeek V4 Flash to generate five candidate Agent trajectories for the same mission. Subsequently, without introducing any more capable closed-source model, the team continued to use the same DeepSeek V4 Flash to verify, score, and sort these candidate results through the LLM-as-a-Verifier verification framework.
It's worth noting that the cost here isn't just the cost of verification, but it also includes two parts: the cost of generating five candidates, and the cost of using DeepSeek V4 Flash for verification and screening. "Auto-verification" does consume more tokens, but because the Token cost of the open-source model is low enough that even with the addition of additional sampling and verification, the overall single-mission cost remains significantly lower than the closed-source edge model.
In terms of latency, automated verification does not mean waiting for five complete generations in a row. Multiple candidate programs can be generated in parallel, and most verification processes can also be executed in parallel. In the case of abundant concurrent resources, the overall delay is roughly equivalent to: the longest Agent rollout + a few rounds of verification. Therefore, an increase in the number of candidates does not cause end-to-end delays to increase proportionally.
With the release of the results of the experiment, LLM-as-a-Verifier sparked widespread discussion on X, while also hitting GitHub Tr.