A scientific problem, AI can give explanations, list steps, or even write analytical code.
But what if we go one step further?
Is it possible for AI to open up the research software, do the work it needs to do, and then give scientists a big, reliable result?
From nuclear-magnetic resonance analysis to pathological image processing, from statistical computation to physical simulation, these tasks appear every day in scientific research work. They require expertise in relevant scientific fields and the ability to plan and execute correctly in the software step by step.
What happens when AI really gets the job done?
Led by a research team from Tsinghua University, researchers from several renowned universities and research institutions in China and abroad have launched OSWorld-Science: a computer-based assessment benchmark and supporting infrastructure for real-world scientific research.
The benchmark contains 146 expert-designed tasks covering seven major scientific areas, 14 scientific research software configurations, and is evaluated by 12 visual language models.
In the current assessment, the average task score of 12 models is only 34.3%.
The score here is not the same as the success rate, but it does raise a question:
When AI is integrated into real scientific research processes, from understanding problems to autonomously implementing scientific research processes to delivering reliable results, what other work needs to be done?
What exactly are the results of the research needed?
Assuming a researcher has to analyze a pathological image and knows what methods of analysis should be used, it is not enough to get the results. Next, he needs to open the software, import the image, configure the parameters, run the analysis, retrieve and export the results. Every step can affect the final output: an option in the software is not set up properly, the file is not stored correctly, or the processing results do not meet the requirements, which can prevent the previous work from being used further.
Researchers need a workflow that is stable and validated.
OSWorld-Science has written this requirement into mission design. Complete nuclear magnetic resonance analysis in Mnova, perform statistical calculations in SAS, process pathological images in QuPath, and perform physical simulation in ANSYS. Experts in the field transform the actual research process into a task, and set verification standards for these tasks. Smart entities need to operate in a virtual machine, via a mouse or terminal software. After each operation, the next step is determined according to the state of the new interface. Eventually, the evaluation is based on the state of the file product and application after the task is completed. In other words, the model describes what it has done and needs to be proven by the actual results.
Figure 1 provides a framework for the design process of OSWorld-Science.
To allow the results of different models to be compared, the team adapted the science software based on OSWorld's multi-modal PromptAgent, providing a uniform tip-to-interaction process, and adopting a consistent task-step budget and basic response length limits.
Requests, responses, token consumption and execution logs will also be recorded.
In this way, researchers can both see what the model ultimately delivers and go back to the process to see how it was done and where it encountered difficulties.
It's the same thing with science software, and models can be very similar.
On 146 missions, Claude Fable 5.1 took the lead with an average score of 73.7%.
Claude Opus 5, GPT-6 Astra and GPT-5.6 sol scored 57.9%, 57.4% and 50.1% respectively.
But what if the whole list is broken down by subject?
In statistical tasks, all models performed well. However, only a few models, such as Claude Fable 5.1 and Claude Opus 5, scored more than 20% in chemical tasks. GPT-6 Astra scored 69% in medical imaging tasks, but performed poorly in chemical tasks.
This shows that the same model, entering different fields of research, may present different capability boundaries.
This also makes the selection of models in the scientific scene more specific: the overall list can provide a reference, but the real need for comparison is whether the model can accomplish the actual work in the corresponding scientific field.
In addition to scoring, cost is also an important dimension that should not be overlooked when evaluating the performance of a model's scientific research. This evaluation shows that the highest scoring model is not the most costly model, indicating that there is no simple linear relationship between model performance and call cost. At the same time, although low-cost models can complete some scientific tasks, there is still a significant gap in the completion of the task, the stability of execution and the quality of the final results compared to high-performance models.
Figure 2: Cost-to-performance contrast. The horizontal axis is the average API cost per task, in dollars; the longitudinal axis is the average task score.
Another interesting result is the difference in output between models.
Claude Fable 5.1 produced an average of about 21,000 tokens per mission, with a score of 73.7%; Gemini 3.1 Pro produced an average of about 157,000 tokens, with a score of 17.2%.
The output here includes logical tokens, and cannot represent costs alone. The amount of token input, model pricing, and iterative interaction affect the final cost.
But the contrasting group still poses a question worth asking:
More outputs have made it possible to accomplish tasks of higher quality?
After 39 clicks on the same location
Some difficulties can only be seen by going back to the actual operating process.
In a run of the biological software Mnova, the model had to deal with the Bruker one-dimensional H spectrum, assigning each of the 6-bit clusters to the spectrum peak. The Claude Sonnet 5 model failed the task due to coordinate mapping bias, repeatedly clicking 39 times at the same wrong location, and eventually running out of budget.
In the example of scientific research software Mnova, the interface is very similar to PowerPoint and is one of the barriers to model understanding of software operations.
The first point is wrong, maybe it's just a location bias.
But after a continuous point error, the intelligent body needs to recognize the wrong pattern and reflect on changes in behaviour, such as whether the interface is expected to change, whether the current operation may be ineffective, and then not need to change a method.
Without this step, behaviour after a failed operation may turn into more failed repetitions.
In an analysis of open trajectories, the team found that 61% of failed operations left no product to evaluate while running out of steps or time budgets. This also made it clearer what it means to complete a task task task: even after partial analysis, work may still remain undelivered without timely storage, export and verification. Long-term research requires intelligent systems to continuously assess current progress, identify errors, and timely adjust strategies to deliver the correct scientific results. Therefore, intelligent systems need to work together under the conditions of understanding expertise, finding the right interface inputs, handling environmental anomalies, and so on, in order to accurately complete complex tasks.
How do you know it's right?
The success of the software doesn't mean that the scientific mission is over.
In ANSYS Fluent's cylindrical rotation accompanying mission, Kimi K3 found a resistance reading of zero.
This anomaly caused it to re-check the settings, then fix the problem of the leaked wallpaper, and finally get a full score.
In this process, the results are not only read out and delivered directly, but also become the basis for the correctness and rationality of self-operation before model checks.
But the inspection itself can also be wrong.
An instance of a chemical task shows that an intelligent organism made multiple format conversions and internal consistency checks, but still repeated the same three-dimensional chemical error.
Why have you checked several times and not found any problems?
Because if the checks always revolve around the same error statement, consistent results, only the same error may be retained.
Scientific verification requires independent evidence and allows that evidence to overturn previous judgments.
Primitive data, physical constraints, and precision requirements could become such a basis. What an intelligent organism needs to do is put the results back to the requirements of these tasks and the accuracy and rationality of the test results in the relevant scientific laws.
This has also given a more specific meaning to reliable crystals in scientific execution: capable of performing operations independently, detecting abnormalities in the results, and thus correcting their own work.
What have we seen from these missions?
Putting the tasks of different disciplines together, OSWorld-Science is focused on a pervasive ability: allowing AI to implement expertise into the operations of research software to automate the processing of specific research tasks into verifiable research output.
Interface positioning determines whether the operation will land in the right position, environmental adaptation determines whether the process will continue, progressive checks help the intelligent body jump out of an ineffective loop, and scientific verification determines whether the final results can be trusted, which together determine whether the task will be successfully completed.
We need to look at where it stops, what errors occur repeatedly, and how to improve or provide feedback to help it move forward.
OSWorld-Science provides a set of tasks and running records that can be continuously evaluated and compared with these capabilities to better help researchers evaluate the scientific execution capabilities of intelligent organisms.
Let's get more real research into the next round of evaluation
The study brought together researchers from institutions such as China University, UCLA, Yale University, Carnegie Mellon University, UC Berkeley, Stanford University, Zhejiang University, Hong Kong University, and the University of Tokyo, and was supported by the core model departments of companies such as Kimi, Z.ai, and Alibaba.
Interdisciplinary collaboration also enables the practical experience of experts in the field to be incorporated into research on intelligent agents.
At the same time, the research team welcomes more researchers to join the mission:
On the one hand, we plan to supplement real problems in more areas of scientific research, including but not limited to inorganic chemistry, life sciences, mechanics, materials science, earth sciences, and social sciences.
On the other hand, there is the design of a more comprehensive cross-software AI workflow. For example, in a protein structure analysis task, the intelligent body needs to first open the protein structure in PyMOL, locate the target to bind the pocket and complete the critical residue measurement; then import the structure into AutoDock Vina for small molecule matching; then redirect the connection results back to PyMOL, visualize the composition and key interactions; finally, use Python to summarize the docking score, key distances and number of interactions of different candidates, and verify that the candidates ranked based on whether they meet the default structural constraints.
If your research has a topic that you hope AI will take over and be able to clearly verify the results, please include it in OSWorld-Science.
146 tasks are just a starting point. When AI begins to enter scientific research software, the question of whether it is worth continuing testing becomes more specific: can it turn every step of the operation into a credible, available outcome?
Contribution to the mission:
Website of the project:
Technical report:
Tasks and data sets:https://huggingface.co/datasets/SciAILab/OSWorld-Science-data
Project code:https://github.com/DiscoAILab/OSWorld-Science
The main author:
A graduate of UCLA's Department of Electronics and Computer Engineering with a major in AI for Science and LLM post-training.
RIKEN AIP is a research associate at the Institute of Chemical Engineering's Center for Advanced Intelligent Projects, where his research focuses on Agentic AI and AI4Science.
A: Yale University visits students & research assistants, focusing mainly on AI for Science and RSI Agent for Biomedical and Clinical, with a particular focus on Mental Health and Cognitive Science.
Li Li: Master's degree in Computer Science from the University of Waterloo, majoring in LLM Agent and Reasoning.
She is a PhD student in computational biology at Carnegie Mellon University, where her research focuses on Lab Automation, Agentic AI and AI4Science.
A graduate student at Tsinghua University with a major in Coding Agent and Multimodal LLM.
The team guides:
Tokenwave.AI, founder of the open-source research community M-A-P, specializes in big language models, with Data-Centric AI.
Temple: Assistant professor at the University of Waterloo's Chariton School of Computer Science, where his research focuses on AI for Software Engineering, code generation, software testing and formalized verification, and is dedicated to improving the efficiency of software development, testing and evolution.
Zhenzhen: Postdoctoral researcher in the Department of Computer Science and Technology at Tsinghua University, whose main areas of research are multilingual language models, coding agents and agentic reinforcement learning.
Donje: Professor of Computer Science and Technology at Tsinghua University and Director of the Center for Basic Model Research at the Institute of Artificial Intelligence at Tsinghua University, where his main areas of research are general AI (AGI), large-scale pre-training models, machine learning and knowledge graphs.
Li Chen: Professor of Computer Science and Technology at the University of China and Director of the Center for Knowledge Intelligence at the Institute of Artificial Intelligence at the University of China. His main research focuses on knowledge mapping, big language models and data mining, with a particular focus on knowledge reasoning, knowledge query and dialogue generation.
Zhang Weihao: Assistant Professor (recommended) at The University of Tokyo, while working at the Center for Advanced Intelligent Projects at the Institute of Physics and Chemistry, where his main research areas are AI Agent and AI for Science.
Liu Tian: Assistant Professor at the School of Artificial Intelligence at Tsinghua University, where his main research areas are multimodal AI, LLM pre-training/post-training, AI4Science and data-driven AI.
