It's the first time in the history of the world.
AI has been able to write code, run experiments, and even update parts of human records by mass testing.
But how far can the best model go from "continuing to raise a score" to "developing a truly better machine learning approach"?
A new project covering 12 fields and 140 real-world research missionsMLS-BenchThis is not an optimistic answer.
Even with the full implementation of the most powerful human method and the permission for repeated experiments, the current front-end models do not demonstrate reliable method innovation capabilities. The so-called new solutions they offer are basically optimizations and recombinations of known methods.
It's even more remarkable to see thatMLS-BenchThe rating has been added to the official distribution list of Kimi K3 and Qwen 3.8-Max.
After the code evaluation has gradually waned, leading model manufacturers are pointing the next capability frontier to the same problem:
AICan we really do research?
Link to the paper:https://arxiv.org/abs/2605.08678
The main page of the project:
The code link:https://github.com/Imbernoulli/MLS-Bench
MLS-Bench-Lite appears alongside software engineering, terminal operations, and general intelligence assessment in the official rating tables of Kimi K3 and Qwen 3.8-Max.
The images were taken fromKimi K3 Tech BlogwithQwen 3.8-Max official blog。
MLS-Bench-Lite selected 30 tasks from a comprehensive assessment, covering all 12 areas of machine learning, to specially examine whether models can come up with and implement better approaches to real research problems.
In June 2026, Kimi K2.7-Code first incorporated it into official release ratings; a month later, Kimi K3 adopted it again; and then Qwen 3.8-Max wrote it into official release results as well. For a newly proposed research review, this speed of adoption demonstrates that the industry's concern is not just that the Agency can't do more coding tasks, but that it can't push AI itself forward.
After a strong approachModels are still rearranging answers
The paper's main experiment evaluated five cutting-edge models. The model can either directly submit the first program or access the actual code library, modify the code multiple times, run experiments, and then submit the final implementation.
The team also handed the code of the strong human baseline directly to the model, which included the accepted SOTA method for each problem. In other words, the model did not guess the answer from scratch in the absence of information, but stood on top of the current best method, and the task was to go one step further.
The conclusions of the paper's main experiment were clear: of the five models evaluated at the time, none of the models proposed could outperform Human SOTA in overall performance. This step did not happen even if the model had achieved the realization of Human SOTA and was allowed to run multiple rounds of experiments.
As stronger models enter the list, aggregate scores will continue to rise, but the lead scores themselves remain low and the assessment is far from consistent. At the same time, once the aggregate exceeds a baseline, it does not automatically mean that the model discovers a new mechanism.
The columns of the web page show the first program, multiple rounds of Agents and Human, by area.SOTAIn the paper's main experiment, the overall performance of the model method did not exceed Human SOTA.
The paper further examined what the model was actually doing. The result was that when the prompts were changed from "discovering a new method" to "optimizing an existing solution", the model did better. Experts also observed that after submitting one-by-one analyses, the model often rearranged the combinations of already existing modules and packaged the combinations into new methods; very few, even occasional, truly new mechanisms often lacked insightful assumptions that could explain their effectiveness.
Experts' case analysis shows that models submit often close-up baseline components that have already been seen.
The model is able to read code libraries, identify the components of multiple baselines, and can combine different components into a functional system.
It's already a powerful capability from an engineering execution standpoint; but from a scientific discovery standpoint, it still doesn't answer the most critical questions: why do we need a new mechanism, what fundamental flaws in existing methods do we have to fix, and how do we keep working on new data and scale?
When a model lacks this judgment, multiple rounds of experiments can easily degenerate into local climbing hills. Based on the results just seen, it minimizes weights, combines modules, modifies training details, and can occasionally gain an increase, but it is difficult to form an explanatory, portable research proposition.
It's not a simple "score isn't high enough". Current models are still far from the real approach: they modify, engineer, piece together known components, but they don't actually come up with useful algorithmic innovations like good human researchers do.
Extraterrestrial discoveries
In the past year, Auto-Research and recurring self-improvement have been warming up rapidly. AlphaEvolve has found improvements on issues such as circle packing, and nanoGPT speedrun has allowed Agents to keep testing around fixed targets.
At the same time, "automation" is becoming an ever-expanding label - allowing Agents to write more project code, automatically generate skills, update memory, clean up data, synthesize training samples, and even optimize training infrastructure - to be put into the narrative of AI improving AI.
Anthropic paves a very intuitive evolutionary path in When AI builds itself: the model is a chatbot that provides snippets of code, then becomes a programming agent that can run the code itself, and then develops into a multi-agent system that can allocate long-range tasks. Its internal data shows that the code Claude wrote already accounts for more than 80% of the merged code, and the amount of merged code is also growing significantly.
This is an image from Anthropic.When AI builds itself。
A similar narrative has emerged domestically. The official release of the MiniMax M2.7, which calls for models to create reinforced learning skills, update memory and optimize their own learning processes, is called the "self-evolution of the opening model". These advances do increase the degree of automation of AI research and development, but they mainly answer the question of whether models can participate more deeply in existing processes.
Even more critical questions remain: Can AI find ways that humans haven't yet?
These studies have shown that large models can be powerful search engines, but they usually have one thing in common: cheaper verification, quick feedback, and a clear score for results. The system can generate a large number of candidates at a time and then select the best answer from them.
That's not how the real way to change machine learning was born. The value of attention, residual connectivity, integration, and Adam is not that they won multiple points in a single experiment, but that they still exist after changing to different data, models, and scales.
This also echoes the standards set by Richard Sutton in his bitter lessonbook. A truly effective, often universal, long-term approach that can continue to work with computational resources; we need AI that can complete the discovery process, rather than just loading systems that humans have already discovered.
From GAN, Seq2seq, VAE, BatchNorm, to ResNet, Transformer, Spread Models, LoRA, RMSNorm, and PPO, the boom of machine learning communities has come from a series of time-consuming, spatial and scale-testing approaches.
Many engineering tasks combine method design, hyperparameters, data processing and implementation techniques; some open-source research tasks look only at the final score and cannot determine whether improvements come from algorithmic innovation, or shortcuts such as expanding models, modifying meters.
One type of task is essentially a data mining competition: give training sets and test sets, allow the model to clean up the data, combine the existing algorithms, and finally submit the predictions. Another type of task requires Agent to generate a research program or paper, and then separate the language model from dimensions such as novelty, completeness, and feasibility, but not directly running and comparing it to the approach it proposes. The former is closer to the project, the latter closer to the text evaluation, and neither is the correct answer method has actually improved.
The MLS-Bench must be locked, which is the most difficult axis of ability to measure.
140 questionsIt's a good idea to take the "innovation method" alone.
The MLS-Bench covers 12 main areas of language model pre-training and post-training, visual generation, reinforcement learning, robotics, machine learning systems, AI for Science, optimization and theory, causal inference, time sequence and credible learning, and includes 140 research tasks from real code libraries.
This range of coverage determines that it is not a specialized questionnaire built around a specific model capability. Language model tasks ask for training objectives, optimizers, and routing mechanisms to be improved; visual generation tasks involve sampling and generation processes; reinforcement learning and robotic tasks require methods to maintain yields in different environments; system direction puts the algorithm back into real throughput, resources, and execution constraints.
Specific tasks can range from KV cache compression to time-space traffic forecasting, black hole image reconstruction, antibody-antigen combining, robot world models and training quantization. They come from different research communities and use completely different metrics. This breadth itself is an important part of the work: it expands "what else AI can do to drive machine learning" from a few popular cases into a cross-disciplinary research map.
In other words, MLS-Bench is not looking for a field specifically suited to Agent and then announces that AI has already studied it. It puts a number of important issues in today's machine learning research into a single set of executable assessments, hoping to see if a capability can cross the boundaries of the field: whether the model can identify the substantive limitations of existing methods and propose improvements that are still established under multiple conditions.
Each problem is not a replicative paper, nor is it a replicative result. Models need to revolve around a clear problem, design new optimizers, train targets, attention variables, sample strategies or routing rules, and submit code that can actually work.
Each mission contains at least three test environments to test migration capabilities, and at least three strong human baselines, including field-recognized SOTAs. All baselines are re-implemented and calibrated by the research team in the same set of codes and assessment processes.
The "strong baseline" here is not a line of numbers copied from the paper. The research team needs to re-access at least three representative human approaches to the task to confirm that they can recover and perform properly in a unified code library. Only then will the model ultimately exceed or lag behind the baseline without differences from different training codes, data versions, or quality achievement.
This process also calibrates the boundaries of tasks. If the strongest human methods cannot be expressed within the limits allowed for modification, the limits are too strict; if models can change data, train budgets or rating processes at will, the limits are also relaxed. Each task must find a verifiable boundary between the two: it both embraces true method innovation and excludes choices that are not related to research problems.
Models are compared to human baselines in the same code library, the same training protocol and multiple test environments.
To ensure the detection of the target research problem, the team freezes data pipelines, meters, and shared training protocols, opening only the components that need to be studied. If the task involves the model structure, the system will also check the parametric scale, preventing the model from relying on increasing the capacity to exchange points.
Different tasks test this migration in different ways. Some replace the data set, some replace the model scale, some replace the environment or training conditions. Models cannot write a specific instance for immediate feedback, nor can they score by remembering the rules of a public set of tests. A candidate method must continue to bring benefits under at least three conditions in order to qualify as a method-level advance.
The team also did a direct anti-cheat dissolution: removing the model scale check. As a result, the tested model gains higher scores by increasing parameters, and some results can even be easily surpassed by strong human methods. After reopening the check, the shortcut disappears. The experiment explains that if variables such as capacity are not locked, a seemingly bright automatic study scoreboard is likely to detect a larger network.
This rigorous set of controls is critical. It allows the end result to answer a question that has been confusing in the past: Are scores rising, are methods improving, or is the engineering space expanding?
More samples are usefulBut beyond the threshold of finding a way
If we keep raising the budget, can we solve the problem?
More prototyping and large-scale sampling can bring benefits, but improvements will soon come to a head.
Experiments have shown that increasing the number of iterations and sampling during testing can indeed improve the results, but the main role is to continue searching and grinding in the right direction. The gains on simple tasks are quickly matched; continuing search may also over-adapt to the visible results when the model can only see feedback from a partial environment, instead harming performance in an unopen environment.
This poses a specific boundary to the current popular "reasoning time algorithm can solve everything". Testing extensions are good at increasing attempts in already-defined candidate spaces, but the method finds that the model first needs to create a new space worth searching for. The former can improve the success rate with more samples, while the latter needs to form new judgments about the problem structure.
The team then placed the model in a scenario that was closer to a real pre-training study. The algorithms that were originally available for three complete 345M parameter training sessions were converted into a freely available experimental budget. The model could decide on its own the size of the proxy model, the number of training tokens, the sequence of experiments, and when to perform expensive full verification.
Some models actively consume budgets and ultimately reduce performance; the only models that achieve improvement use little computing power.
More computational opportunity does not mean that the model will automatically select more informative experiments.
This reveals a deeper bottleneck than "not thinking of new points". Scientific research also needs to judge which assumptions are worth verifying, which small experiments have the most information, when enough evidence is available, and when a wrong path should be stopped. Current models lack not only method innovation but also the broader scientific judgment needed to build and verify evidence.
The team also tested a seemingly direct remedy: to give the model more external knowledge. The model could use web searches to obtain detailed derivations of baseline papers and related theoretical contexts. However, the overall benefits of these additional information were still limited and in many cases did not significantly exceed ordinary multi-cycle experiments.
Increasing web search, derivation and theoretical context has only brought limited improvements.
This means that bottlenecks are not just "models haven't read this paper". Knowing more terms, formulas, and related methods is not the same as being able to make an insightful hypothesis; proposing a hypothesis that sounds reasonable, or being able to design experiments that can distinguish between multiple explanations. Knowledge retrieval solves information availability, and scientific judgement solves what to believe and verify next.
Automated AIThe next step is not just a pile of searches.
The MLS-Bench ultimately poses a more fundamental paradigm problem.
From AlphaGo, AlphaZero's codebase search, to AlphaTensor's transformation of mathematical structures into searchable objects, to FunSearch and AlphaEvolve, which allow language models to generate code candidates, the core cycle is gradually stabilized: a large number of programs are proposed, automatically executed, scored, and the next round of candidates are generated using the winners.
From AlphaGo to AlphaEvolve, object search has gradually expanded from chess tables to programs and games.AlgorithmsSources: The authors report.
AlphaEvolve not only solves mathematical constructs, but also optimizes data center modeling, chip design and Gemini training processes, showing that this paradigm is not without real value.
When the verifier is inexpensive and clear, expanding the sample size is an effective route. However, verification in real scientific discoveries is often expensive, delayed, multi-phase, and only partial results can be observed. Models cannot infinitely call on an inexpensive scorer, and can only make intuitive assumptions to obtain information from limited experiments and determine which paths deserve more computational input.
The paper summarizes the challenge as follows:Scalable scientific discoveries can be achieved under conditions that are difficult to verify and scale.
Conventional code evaluation is gradually fading, and AI for Research is likely to be the next to truly bridge the model gap. Kimi K3 and Qwen 3.8-Max have already incorporated MLS-Bench-Lite into official releases, just the beginning of this long-term measurement.
The real return to self-improvement is not to allow AI to search faster for the space that humans have already defined, but to allow it to one day independently come up with, validate, and establish the ways that really drive AI forward.