If there is a super-intelligence dawn,
We're almost there before dawn.
The sky was not yet clear, but there was light on the horizon.
What ASI-Bench wanted to do was to create a scale that would measure "how far away is AI from the limits of human intelligence" before dawn, and measure how far the amount of AI has gone.
ASI-Bench is a benchmark test designed to measure the scientific autonomy of AI systems, developed by a team of global scientists from over a dozen institutions, including MIT, Harvard, Carnegie Mellon University, China University of Science and Technology, and the Microsoft Research Institute.
Address of the paper:https://arxiv.org/abs/2608.17271
Contributions to entry:
GitHub：
Currently, ASI-Bench is open to more researchers to contribute:
Why ASI is Important
AI has been strong in the last few years, largely based on one thing: learning what humans already know, and then calling it more quickly and accurately. The model can score high on more and more tests and benchmarks, but these high scores largely explain the same thing.
But the boundaries of knowledge are not pushed this way, nor are they pushed by more skillfully invoking what is known.
Research is essentially a series of combinations. Truly transformative discoveries often involve not only a more sophisticated rearrangement of already knowledgeable areas, but finding unknown regions of extremely low probability density that have never been previously explored in areas that have not been observed, conceptualized, or even recognized as problematic.
In the face of these areas, AI has to decide for itself: how to study this problem, what methods to use, what phenomena to observe, and if the first step goes wrong, can it find a way to change it?
It's not just the ability to reason, it's a combination of innovation and execution. This step from mastery of the known to exploration of the unknown is why ASI is so important.
Scientific research is the most natural scenario for observing this ability. Scientific research itself explores the unknown under uncertain conditions, poses questions, selects methods, performs experiments, understands results, corrects directions. It naturally requires knowledge, judges and executes at the same time. If AI can independently advance a research topic, it will begin to approach autonomous intelligence.
But the problem is: if you can't evaluate, you can't judge how far AI has gone.
What is ASI-Bench?
ASI (Artificial Superintelligence) is not a science-fiction concept in this context, but a concrete definition: AI moves from grasping and executing knowledge already available to humans to self-exploring the unknown, creating new knowledge and actually executing it.
General intelligence
ASI is not only strong in mathematics, code or a certain vertical field. It should be able to deal with problems in different fields, understand new scientific goals, and transfer research capabilities to the past.
What is innovation?
ASI doesn't just call existing methods. It needs to make its own assumptions in the absence of existing paths, select technical paths, judge what information is important, and adjust the direction according to new phenomena. The most critical test condition for innovation is that humans no longer tell it the complete method, but only tell it a direction to study.
Self-executing power
Having an idea is not the same as completing a study. Models also need to turn the idea into concrete steps, calling tools, writing code, running experiments, screening errors, and finally delivering results that can be checked.
The three are indispensable. Only general intelligence has no innovation, the model is only a comprehensive knowledge base. Only innovative ideas have no executive power, the model has set a set of directions but has failed to verify. Only executive power has no innovation.
Why is the current benchmark insufficient?
The existing benchmarks push these three capabilities far apart, but each only measures one facet.
The Humanity's Last Exam has 2500 questions covering more than 100 disciplines, and the cutting edge model scores are low, fully reflecting the difficulty of knowledge.
The 89 terminal task front-end systems in Terminal-Bench 2.0 have a completion rate of less than 65%, and the Agents' Last Exam, the most difficult level, has a pass rate of less than 1%. They measure long-range execution, but task goals and verification criteria have been given, focusing on getting things done, not deciding what to do.
There is no benchmark to measure general intelligence, innovation and execution at the same time. Nor is there a benchmark to systematically remove humans from the same research project to see how far AI itself can go.
This is what ASI-Bench has done.
How the ASI-Bench works
The mechanism of the ASI-Bench is simple, but the logic is straightforward: the same research project, the same input data, the same rating criteria, only change the amount of methodological information provided to the model.
At the heart of the design is the information gradient.
In the same research project, AI was given four different guidelines each time:
B1 gives the complete method and the steps to implement, and it measures the execution power.
B2 only gives method names, not details, to measure whether a method name can be turned into an executable process.
B3 gives nothing but scientific goals, raw data and constraints, AI chooses its own methods, builds processes, runs experiments, judges outcomes.
B4 Add some correct but unrelated interference information based on B3 to see if the AI will run away.
The gap between B1 and B3 is the extent to which the model relies on human methods to guide it. It directly answers a question: How far can humans go, step by step?
There are 60 subjects covering 11 fields: Mathematics, Physics, Chemistry, Biology, Astronomy, Materials, Earth Sciences, Medicine/Biostatistics, Computer, Robotics, and Electronic Engineering. There are completely different knowledge structures, data forms, and method choices in different fields.
What is the result?
In 60 project-level research missions, the team tested six agent harnesses for Codex, Apex Research, Claude Code, Kimi Code, MiMo Code and OpenHands, together with GPT-5.5, GPT-5.6 Sol, Claude Opus, Kimi, DeepSeek, GLM, MiniMax, MiMo, and other backbone models, forming a combined Harness x Model group of 18.
The average Scientific Score of the 18 group configurations was 50.92 when the model was given complete method guidance under B1 conditions; the average decreased to 27.17 points when the guidance gradually decreased and B3 no longer provided specific methods, down by nearly half to 23.75 points compared to B1.
This decline is not limited to weaker models.
The best-performing combination Codex + GPT-5.6 Sol (ultra) scored 71.78 points in B1 and 51.60 points in B3; this is already the highest score of B3 in all 18 configurations. Another example is the Claude Code + Claude Opus 5:B1 score of 72.29 points, but after reducing method information, B3 scored only 40.70 points.
The greatest performance loss occurred when detailed method guidance was withdrawn for the first time: from B1 for "providing a complete method" to B2 for "providing a method name only", all configurations dropped by an average of 21.28 points.
This means that what really determines whether a model can accomplish these scientific tasks is not just the absence of a method name, but whether the model can complete scientific modeling, method selection and research path design on its own when detailed method paths are no longer provided by humans.
Further analysis of failures showed the same trend: 62% of failures focused on scientific decision-making, including scientific modeling, method selection, and robust judgement; only about 26% were purely numerical execution problems. In other words, the main bottleneck for AI in these tasks today is not that it's uncalculated, but that it doesn't know what to do, why to do it, and what to do next.
Low scores are not a disadvantage of a benchmark. On the contrary, it indicates that the scale has enough room to differentiate.
Why do you believe these numbers?
High-difficult but unreliable benchmarks are meaningless. The title itself, the reference results and the rating methods are both credible, and scores are meaningful.
Each task of the ASI-Bench is constructed in four phases: gathering source-based scientific research questions, field expert screening and task engineering, AI-assisted auditing and four rounds of artificial scrutiny, end-to-end sandbox verification.
The Four Eyes Principle and the Four-Eyes Check
The first round, each task is assigned to at least two non-author reviewers. Independent review. The second round re-allocates reviewers, checking whether modifications to the previous round have been implemented. The third round focuses on agent trajectory analysis of B1-B4. The fourth round is prioritized by reviewer checks in the same field, plus a Claude Code scan.
AI audits more than just text. It checks scientific logic, prompt levels, reference files, rating codes, information leaks and agent trajectories, focus on identifying targets that don't match scorer, hiding information leaks, hard-coding answers and unexpected shortcuts.
Each task must also be executed end-to-end through the sandbox: environmental checks, reference solutions, self-assessments, limited tool runs, matched instances and controlled random seeds. The reviewer continues to look at the resulting files, logs and intermediate trajectories, returning for runtime failures or inconsistent scores.
Doing science questions to get high marks
The Richards-Topmodel hydraulic model task best describes how the team works.
In the process of building the task, the team found that generic baseline scores were too high, scoring rubric without grasping the core scientific mechanism, and the model didn't really need to understand the hydraulic process to score.
So the team re-adjusted the weights, units and boundary conditions: after adjustment, the generic baseline dropped to 11.3-17.0, the correct method got 89.5-100, the reference got 100.
Building a task is more than just writing a prompt. Repeatedly checking scientific mechanisms, scoring logical and real model behavior, until doing a scientific problem is really consistent with getting high scores, which is the team's standard for each task.
Who is ASI-Bench for?
The model team:
If you're doing a new basic model or reasoning model, in addition to wanting to improve your score by a few percentage points on an existing benchmark, you want to prove that you're actually smarter than you are by running the ASI-Bench.
Developers of the AI Scientist/AI for Science direction:
If you're an AI Scientist system or research agent and you're wondering if it's going to be able to do the research outside of human guidance, run the ASI-Bench. The four levels tell you where the problem is: B1 to B2 falls too much, indicating that you know the name of the method but it doesn't work; B2 to B3 falls too much, indicating that you can achieve a given method but you don't choose your own method.
Agent Framework developers:
If you want to compare different models or different agent architectures with different harnesses of the same model, or different models with the same framework, the ASI-Bench project-level tasks are more differentiated than short tasks.
The scores are still low and the benchmarks are far from consistent. Any significant improvements are proof of the system's real progress on higher intelligence capabilities. If you think your model is really smarter, come to the ASI-Bench and prove it.
From a community, you need a community: you will define the future of ASI
In the first edition of the ASI-Bench, 60 subjects were contributed by researchers from a variety of disciplines, including China University, MIT, CMU, Central Dakota, Boston University, the University of Illinois at Champaign-Urbana, the University of Queensland, the Flatiron Institute, and Microsoft Research.
The criteria for measuring Superintelligence cannot be defined by a single team.
The team wants more researchers to contribute to a truly valuable, difficult task in their field. You can get involved in the following ways:
Submit thematic ideas: real difficulties in your field, questions about research value, questions about the ability to test AI
Build a complete task: data, environment, input and output, reference results, and how to run
Design Score Rubrics: What results are calculated, what are the key steps and how to calculate
Review and Feedback: Identifying issues that have already been addressed and proposing amendments
Submit running results: Submit verifiable results using your own model and Agent run scores
When the future ASI system truly has the ability to conduct scientific research independently, the criteria for assessing that ability will be defined by us today. The next generation of AI faces not only the problems that humans have already written the answers to, but the problems that humans really want it to help solve.
Written at the end
The ASI-Bench is the first scale-clear scale to transcend the limits of human intelligence.
The sky is not yet clear, but we can start measuring. It has already told us a preliminary conclusion: that there is still a clear distance between AI and true autonomous research. The ability to execute is not the same as being independent, and the ability to complete specified tasks is not the same as being able to independently determine the path of research.
Welcome more teams, researchers and models to run the ASI-Bench, submit tasks, and work together to perfect this scale.
