The paper team consists of 28 researchers from 10 colleges and universities, the core authors of which include Kim Jong-un at the University of California, Berkeley, Han Yang, Zhang Chen of Purdue University, Li Starhan at the University of Washington, and Jiangsu University of China, Huangzhou, Xu Jia, Zhang Hua, and Zhang Chen. The team members cover a wide range of fields, including big language models, reinforcement learning, MLS, computer vision, optimization theory, and more.
When an AI Agent gets higher scores in a machine learning experiment, where does the improvement come from?
It may have proposed new optimizers or training goals, or it may have simply adjusted learning rates, expanded models, modified data processing, or even found flaws in the evaluation process.
This is also the core attribution problem faced by existing Auto-Research evaluations. Code generation and experimental execution have become common capabilities in cutting-edge models, but "performing the research process" is not the same as "discovering better ways". Without controlling data, capacity, training budgets, and evaluation protocols, any score improvements can hardly be reliably attributed to the algorithm itself.
Researchers from several institutions such as Berkeley, Princeton, and Chiang Mai have proposed the MLS-Bench, which attempts to measure this capability separately in executable environments. It contains 140 real-world research missions in 12 areas of machine learning, each setting multiple generalization conditions, and reproduces at least three powerful human approaches in a unified codebase, including the field-recognized SOTA.
The paper's analysis of the first five front-end models found that even with the implementation of these powerful methods, the models were still better at optimizing and combining components already present than at completing the actual method discovery.
MLS-Bench's first paper:
The paper:
The main page of the project:
GitHub：
Current assessments of why measurement methods are difficult to find
Over the past year, many advances have been made in automated research systems. One system connects code generation, automated verification, and evolutionary search to continuously improve indicators in circle packing, kernel optimization, or fixed training tasks; another tries to get agents to read papers, come up with ideas, write implementations, and generate research reports.
The two types of routes, respectively, demonstrate that large models can be searched on a large scale and that they can perform an increasingly complete research process. But they still leave a common problem: when the results change, we don't know exactly what the model finds.
Specifically, the MLE-Bench class of tasks inherits the form of a data mining competition: models acquire training sets and test interfaces, improve predictive performance on individual datasets through data cleaning, characterization engineering, and a combination of existing algorithms. This is suitable for assessing machine learning projects, but single-point results do not indicate whether candidate methods can be migrated. Another class of end-to-end research evaluates Agent to produce programs and papers, and then evaluates language models from dimensions such as novelty, completeness, and feasibility, but does not directly compare the actual performance of candidate methods.
There are other tasks that are closer to method search, but the focus range is narrower. For example, activation functions can be found to validate large amounts of expressions on a small number of fixed data and models; nanoGPT speedrun can continue to absorb new optimizers; attention to implementation and training techniques. They can drive specific systems but do not provide uniform measurements for cross-domain methods.
In tasks with inexpensive verifiers, the system can generate a large number of candidates, which can be optimized by selecting a single score. This setting is suitable for measuring search efficiency, but does not require the creation of candidate methods across data, models or scale. End-to-end research tasks cover a broader range, but allow for more links to model modification. As long as data processing, training resources, model capacity or rating logic are not locked, local engineering gains may be mistaken for method advances.
The value of machine learning methods typically comes from portability. Residual connectivity, attention, integration, or Adam are not only effective in a fixed instance; they solve more general problems and continue to benefit when conditions change. Therefore, a set of method-oriented findings requires at least two questions to be answered:
Can the improvement be attributed to the target research component rather than other engineering variables?
Can the same improvement span multiple data, environments, models, or scales?
From this perspective, the difference between engineering optimization and scientific method discoveries is not whether to write code, but whether to optimize objects. Engineering tasks allow multiple techniques to be used around a final indicator; method discoveries require improving a specific research component and verifying that it still exists under multiple downstream conditions and on a larger scale.
MLS-Bench's mission design revolves around these two issues.
Building 140 executable tasks from real-world research problems
The MLS-Bench covers 12 areas including language model pre-training, language model post-training, visual generation, reinforcement learning, robotics, machine learning systems, AI for Science, optimization and theory, causal inference, time sequence and credible learning, and a total of 140 tasks.
Task ranges include KV cache compression, active learning, time-space traffic forecasting, black hole image inversion problems, antibody to antigen combining problems, robot world models, training quantification, and optimization and theoretical problems. A complete evaluation requires about 700 H100 hours to run a candidate program; MLS-Bench-Lite for 30 questions will reduce single evaluations to about 90 H100 hours. This cost itself reflects the difference between authentication methods and low-cost code scores.
These tasks do not require a model to summarize or reproduce existing experiments. Each task starts with a real code library and specific research problems, requiring the Agent to modify a definite research component. For example, candidate submission may be a new training goal, optimizer, attention variable, sampling strategy, or routing mechanism.
The MLS-Bench focuses on whether the method continues to be effective under multiple conditions rather than on single-point improvements in fixed instances.
The entire task consists of five key elements:
A real code library for specific research issues and their correspondence;
a modifiable scope that is calibrated by experts in the field;
At least three test conditions for testing migration capability;
At least three strong human baselines, including field-recognised SOTA;
Baseline results reproduced in the same code library, in the same training and evaluation process.
The team also selected 30 missions to form MLS-Bench-Lite. This subset covers all 12 directions for cost-effective continuous evaluation.
The images are from the official blog of Kimi K3 Tech Blog and Qwen 3.8-Max respectively.
How to isolate algorithms from the scope of controlled modifications
There is a direct tension between openness and attributability. Allowing models to modify entire warehouses, closer to free research, but difficult to judge sources of gain; only allowing a small amount of code to be modified, and possibly excluding truly valuable methods.
MLS-Bench calibrates this boundary by reproducing human baselines. For each task, the research team first re-accesses at least three representative methods to the same code library and confirms that the SOTA in it can restore reference performance. If a strong method cannot be expressed within a given range, the task boundary is narrow; if the target component is not modified, the results can be significantly improved, indicating that the boundary remains wide.
After calibration, the model can only modify parts directly related to the target problem. Data pipelines, scores and shared training protocols remain fixed when studying the target; when studying the model structure, the system checks the size of the parameters to avoid exchanging performance by increasing the capacity; tasks involving different computing budgets are also subject to unified resource constraints.
The training scale used in some primitive studies is too large to be directly incorporated into continuous evaluation. MLS-Bench uses smaller models or less data for this type of task, but requires all human baselines to maintain the original performance order on a scaled-down scale. That is, reducing the cost of validation does not change the basic judgment of the task that the method is superior. All 140 tasks need to pass this calibration.
Compare strong human methods with models submitted in the same code base and in the same protocol.
Multiple test conditions play a different role. Models may find examples in visible feedback, but candidate methods need to remain effective in unused locally optimized datasets, environments, model sizes, or training conditions. A patch that improves only a single instance, leaves the current condition and fails, will not be counted as a general method progression.
Different tasks contain a different number of conditions and indicators, and the paper uses a three-tiered unified rating. The original indicators are first transformed with the human baseline as the focal point, then aggregated under the same test conditions, and finally get the task score. For single indicators, the weakest baseline corresponds to 0 points, the strongest baseline corresponds to 50 points; theoretically existing indicators still retain continuous upward space.
Because scores are fixed on individual indicators, then aggregated across test conditions and tasks, the final score is not a predetermined "human and linear" score. It is suitable for the overall performance of comparing candidate programs, but it cannot individually prove that a candidate contains a new methodology mechanism.
This design brings the MLS-Bench closer to controlled experiments. The final variable is not how much engineering detail the model can modify, but whether it can make a migratory algorithmic change around the problem of targeted research.
It's not just that the model doesn't run the experiment.
The main experiment offers two working methods for the five front-end models. The first is the direct evaluation of the model for the first time; the second allows the model to enter the code library, run multiple rounds of experiments and modify the implementation, and finally submit the optimized version.
At the same time, the model obtained strong human baseline codes, including the best method currently recognized. Therefore, the experiment did not measure whether the model could be derived from zero-replicate literature, but whether it could continue to discover better mechanisms when methods are clearly available.
The column-shaped graph of the page shows the model's first schematics, the results of multiple rounds of experiments, and Human SOTA respectively.
In the paper's main experiment, multiple rounds of experiments generally improved the overall score, but the methodology proposed by the five models evaluated at the time did not exceed the overall performance of Human SOTA. The total continued to rise as stronger models entered the public list. However, the absolute level of current leading scores remained low, and the assessment was far from consistent. More important than static ranking at a given point in time was how the model achieved these scores.
More importantly, it is how the model is improved. Meltdowns and expert analysis of the first five models show that the model mainly revolves around local modifications, parameter adjustments and recombinations of known components, rarely proposing new mechanisms that can explain existing failures and continue to be established under multiple conditions.
Control experiments with model-scale checks further illustrate the need for strict attribution. After removing this check, all models will improve performance by expanding the parameters, and some candidates will even exceed the human baseline. On the surface, Agent has found a better solution, and the actual change is in capacity rather than the target algorithm.
This gap cannot simply be explained by the model not using code tools. Agents are able to read and write warehouses, run training, view indicators, and iterate. Problems arise in upstream approaches: they often make local modifications around known components, but rarely come up with new mechanisms that can explain existing failures and generate steady gains.
The analysis of fusion differentiates between "optimization" and "discovery".
The team further altered the task hints. When the model was asked to "find a new method", the results showed weakness when describing the goal as "optimizing existing methods". This showed that the current model's advantage was closer to engineering search than method discovery.
Expert case analysis supports this. Many candidate submissions can run or combine a variety of known technologies, but the core structure is usually derived from the already provided baseline. Models combine the loss function of one method, the modules of another method, and the scheduling strategy of a third method, and then perform local parameter adjustments. Real new mechanisms are rare; even if they appear, there is often not enough insightful assumption to explain why it should be cross-conditionally effective.
Model outputs are usually closer to recombination of known components.
This distinction is important. Combinations of already existing modules may produce a practical system and may improve the performance of specific tasks, but it is not the same as proposing a new, scalable approach. MLS-Bench's experimental results show that the current model is already stronger on the former, while there are still significant gaps in the latter.
Expand to improve scores while testing but without changing behavior patterns
The paper examines how to calculate when testing at different sizes.
Increasing the number of attempts can improve the simplicity of the task, but the profits quickly fade.
More iterations and large-scale sampling can usually improve the results, but gains are rapidly declining, with models still primarily conducting local searches around existing candidate spaces. Continuous searches may also over-adapt to these visible conditions when the model only receives feedback from partial test conditions, causing candidate methods to deteriorate under unopened conditions.
This indicates that search scale and method innovation are not variables that can be directly interchanged. More samples can more fully explore the defined candidate space, but it cannot guarantee that the model offers a better explanation of the problem, nor can it automatically create new algorithmic space.
It's harder to plan experiments than to think.
The paper also set up a calculator-bound pre-trained research experiment to see if the model has more complete scientific judgments.
The fixed scheme originally allowed for three full 345M parametric training sessions. The research team converted the resources from the first two sessions into a freely allocated budget, allowing the model to choose the size of the proxy model, the number of training tokens, the sequence of experiments, and when to complete the verification.
This is equivalent to giving the model more freedom to study. It can first test the hypothesis with a cheap experimental test, eliminate the wrong direction, and then put the remaining resources into the most promising scheme. However, more options usually do not lead to better results. A model consumes a large amount of budget after performance decreases; the only model that gets upgraded uses only a small amount of resources.
Current models still have difficulty allocating limited computational power to the most informative experiments.
The gap between models thus reflects not only the ability to come up with new ideas, but also the ability to form insightful assumptions, choose effective proxy experiments, update judgments based on incomplete feedback, and determine when it is worth expensive verification.
Adding the model to the web search, baseline thesis derivation and theoretical context has only led to limited improvements. More information does not automatically translate into better experimental decisions. This suggests that the current bottleneck is not a mere lack of knowledge, but the need to build the broader scientific judgment needed to verify the evidence.
From cheap verification to complete scientific discovery
The current large-scale search pattern dates back to a clear technical path. AlphaGo and AlphaZero conduct deep searches in well-defined games; AlphaTensor rewrites matrix multiples as searchable mathematical objects; FunSearch allows language models to generate functions and retain better candidates through automatic scoring; and AlphaEvolve extends the same loop to more complex programs and code libraries.
Language models generate candidate programs, automatic scores are executed and scored, and the library decides the starting point for the next round of searches. Source: Google DeepMind.
Self-improvement work based on large-scale searches typically relies on fast, precise verifiers. Each candidate gets scores instantly, so expanding the sample increases the likelihood of finding a good answer.
Verification in scientific research is often expensive, delayed, multi-phase and can only be partially observed. The failure of an experiment can be due to methods, implementation, data, scale or randomness; researchers need to distinguish between multiple explanations using small-scale experiments before deciding whether to put in full training. The real limit is not how many candidates can be generated, but how much computation can be converted into valid evidence.
The paper summarizes the objective as: achieving scalable scientific discoveries under conditions that are difficult to verify and scale.
This also gives Auto-Research a stricter evaluation standard. A system not only produces a candidate method, but also explains where the improvement comes from, verifies whether it exists under multiple conditions, and completes a research process from hypothesis to evidence with limited resources.
As traditional code evaluation gradually fades, AI for Research is becoming the frontrunner of new capabilities. Kimi K3 and Qwen 3.8-Max have put MLS-Bench-Lite into official release evaluation. The next step is worth continuing to observe, not only if the model can perform the research, but when it can independently discover and verify how it really drives machine learning forward.
Related links
The MLS-Bench paper:
MLS-Bench project home page:
MLS-Bench GitHub：
Kimi K3's official blog:
The official blog of Qwen 3.8-Max:
Anthropic《When AI builds itself》：
Richard Sutton《The Bitter Lesson》：
Google DeepMind AlphaEvolve：