GPT-6 Struggles with 3D: Just 0.53 on a 250-Question Test, While Claude Also Falls Short

  It's the first time in the history of the world.

Recently, GPT-6 Astra's video of CAD painting was shot on X.

For example, in this turbocharger, each part is modeled separately, and not only can it rotate normally, but it can be disassembled and unfolded at the same time. The visual effect is quite explosive.

At one point, GPT-6 Astra's BenchCAD score reached a staggering 95.9%, and many in the comments section are already discussing the issue of recruiting CAD engineers.

However, there are also engineers who have boiled cold water: these parts are not sent to the production line. Because painting does not mean painting correctly.

On the surface, models have been able to replicate the visual interface of CAD, but from a professional engineer's point of view, are these models really accurate in terms of scientific principles, geometric structures, and relationships between components? Is there a professional benchmark that can test AI's ability to observe and understand three-dimensional scientific objects and assess whether the three-dimensional objects it generates meet scientific and engineering requirements?

Recently, a benchmark has been found that assesses AI's CAD capabilities in architectural design. It, like professional engineers, goes through the process of inspecting parts, from structural relationships to geometrical details, checking them one by one to determine whether the results generated by the model meet the requirements.

The benchmark is called ScienceVision. Its scope covers not only engineering mapping, but also biochemistry, medical imaging, architecture, and mechanical CAD, a total of five scientific fields, 22 subfields, and contains 250 topics based on real-world three-dimensional data, requiring models to operate three-dimensional objects autonomously, complete observations, measurements, and analysis.

The best GPT-6-Astra (with Codex) scored only 0.53 points.

BenchCAD scored 95.9 points on the three-dimensional science understanding and reasoning exam, only half the way through.

ScienceVision comes from UniPat AI and Elorian AI, a visual reasoning lab based on expert-level visual questionnaires (VQA).Interdisciplinary three-dimensional understanding of vision and reasoning benchmarks, and the team reveals that the starting point from which it was developed was a question: Can AI observe and reason like scientists?

Based on existing results, the evaluation covers the current 12 most powerful open-source closed-source front-end models, including GPT-6 Astra, Claude Fable 5.1, Gemini 4 Argon, Qwen 3.8 Max, and none of the models can lead in five disciplines simultaneously.

What are these questions that can make a model card like this?

ScienceVision's demo shows five disciplines examining operational entries, in which 3D tools are used to simulate the most real-world 3D observation scenarios, modeled after real-world methods used by experts in the field to process 3D source data.

The biological part is to hide the entire protein, leaving only one phosphate molecule and a few ions next to it, to measure them as close as possible.

The shortest part of the chemistry is eight steps: framing out a crystalline cell and using symmetrical operation to see how it stacks up in a layer.

In the medical imaging section, the CT cut back from three directions to find the side of the disease on the kidneys.

The CAD section is a fork, the parts are one by one, and the number is several times the same.

The building was built along a slope measuring the length and height of the slope.

The five demos look like human experts in different fields are examining 3D vision problems in their fields. The movements themselves are not complicated: turn, point, quantity, but it is difficult to figure out where to turn, what to point, where to measure at each step, and it is difficult to make the right judgments without experiencing a more solid observation, understanding and reasoning.

If you're watching a video, you can do it yourself.

The original documentation and examination tools for these five topics are available on ScienceVision's blog: You can open themes yourself, cut 3D or plane views, experience how to evaluate AI's scientific observation capabilities through a real 3D examination process.

Experience entrance:

I've read the question, and I'll tell you why I'm asking this question.

In the past, there has been a lot of 3D understanding and reasoning benchmarks, covering indoor scenic understanding, physical interaction, and abstract structural reasoning, but they mainly serve general tasks such as everyday scenic perception, robotic operation or industrial design.

However, the basics of 3D understanding and reasoning in the realm of science are almost hollow, with only a few specific application scenarios, such as medical imaging and mechanical engineering, yet to form a systematic scientific 3D reasoning assessment.

But the process of scientific research naturally requires the ability to observe and reason in the real three-dimensional world.

This ability requires practice across disciplines: chemists to position atoms, measure geometric relationships; radiologists to track structures in different cutouts; structural engineers to switch perspectives, eliminate shadows, and confirm connections between components.

There's no such thing as just looking at a picture. Human scientists have to decide what to look at, what to look at, what to measure, what to look at and what to do with their expertise.

The logic behind ScienceVision's analysis of AI is the same: the subjects and data are in front of the eyes, and you decide where to look and how to look. The logic behind this is somewhat similar to the "Stanmenz drawing line": draw a line with a value of $1 and know where to draw a line with a value of $9999.

In the field of 3D understanding and reasoning in the past, the various disciplines of problem-setting, task input and evaluation have been divided into five categories, such as the format of 3D files is not uniform, the interaction logic of rendering is not the same, and the form of expression is different.

ScienceVision needs to fill this gap to make a unified, interdisciplinary assessment of 3D visual understanding.

ScienceVision covers five disciplines: biology, chemistry, medical imaging, mechanical CAD and architecture. Each of the disciplines has 50 subjects, 250 paths, across 22 subfields.

Each question consists of four parts: a real 3D source data (molecular coordinates, CT body data, mechanical assemblies or building models), a field question (what to look for, where to find, what units and coordinates to use), an answer request (what to return, what format), and an expert-labelled reference answer.

The answer is verified by the rules of certainty, without the use of language models as the arbiter.

A benchmark with a certainty test has a layer of potential:It can almost automatically become a reinforcement learning environment: yes to rewards, no to mistakes, no need for another model as a referee.

However, the issue itself is extremely difficult to solve.

First, there is the question of whether scientifically meaningful choices actually rely on three-dimensional evidence.A set of questions must originate from specific scientific objects, clearly observe the objectives, determine the conditions and the necessary evidence, and confirm that the questions can be answered in a given environment. Each question must be professionally established at the same time; the visual objectives are clear; the answers can be reviewed.

Second, allowing data from different fields to be correctly observed and measured.Molecular coordinates, CT body data, mechanical CAD and architectural models have different formats, sizes, coordinate configuration and operation methods. During conversion and presentation, critical structural information needs to be preserved and the ability to view, isolate, slice and measure objects suitable for different types of objects is provided.

Finally, experts in the field rely on credible answers and evidence processes.Experts need to check the authenticity and relevance of problems, observe objects in practice, calculate or confirm answers, record key steps, and participate in audits and retrospectives.

The whole process is powered by machines and humans.

Select the appropriate object from the three-dimensional asset library, test whether it can be viewed correctly and measured in the viewer, generate the candidate question according to the template, review the rules, and pack it.

Then the experts take on the task: on the 3D marking platform, test, replay, manually turn, isolate, measure, write the reference answers, record the operation trajectory manually, and finally go through a quality audit, unqualified return.

There are 250 ways to create a problem, to test it, to automate the problem, to improve the pipeline, and to turn it around, one by one.

At the end of the day, the quality of these questions depends on people.

UniPat AI has built an expert platform covering five fields, maintaining 66 specialists in a total of 482.98 hours, including 113 hours in chemistry, 116 hours in mechanical CAD, 103 hours in medical imaging, 77 hours in construction and 74 hours in biology.

Experts review questions, mark answers, record operations, and participate in multiple rounds of reviews and revisions.

This echoes the core positioning of ScienceVision:It measures whether AI has the ability to observe at an expert level, so the answer itself needs to be constructed by an expert to be more convincing.

The previous demo featured an operating record of 250 tracks, each of which was accompanied by a step-by-step preservation of keystrokes.

These records have transformed scientific observation from a black box of conclusions to a process that can be gradually recycled.

Scientifically, ScienceVision compares three methods of obtaining evidence on the same set of 250 questions:

The first is to give the model raw three-dimensional data so that it can write code analysis itself.It's the equivalent of giving you all the raw experimental data, you figure it out.

The second is a set of interactive inspection tools based on data that can rotate angles, isolate structures, slice browsing, and measure distances and angles.It's the equivalent of not only giving you data, but also giving you a microscope.

The third type is a two-dimensional rendering with only six fixed angles.See what you can see from the photos.

There are three ways to use the same set of criteria.

This is a direct answer to a question:Will more data, more tools, and better models work?

To see the results, the highest score, 0.53 ((GPT-6-Astra with Codex), is just over half the score.

The difference between disciplines is more interesting than the total score.

The GPT-6 Astra is leading in the architecture, mechanical CAD, and chemical sciences (0.458, 0.580 and 0.609), biology is outperformed by Claude Fable 5.1 by 0.650, and medical imaging by Claude Opus 5.5 by 0.399.

CT is actually a superimposed cut, read by radiologists, and has seen a lot of 2D medical imaging in model training.

But a three-dimensional array with an axis sequence and an exponential index is not as familiar as the model.

With only six fixed-angle renderings, the score dropped sharply: the best Claude Fable 5.1 is also 0.30, GPT-6 Astra 0.25.

In other words, given the photo, the model cannot be seen; given the three-dimensional data, the scores are up, but the paper itself reminds us that these scores may be calculated from the code from the coordinates, regardless of "seeing" or two words.

It's hard to give the tools the same way.

In the top five models of the list, the GPT-6.1 Sol with the highest score after the interactive inspection tool also scored only 0.02, the GPT-6 Astra's two settings were close to 0.01, and Claude Opus 5.5 dropped 0.02.

There is also a detail in the paper: even in tool mode, the model reads the original coordinate file. That is, you give it a microscope and a data table, and it selects the data table.

Medical imaging was the most unexpected, with models operating at about three times the number of experts, and scoring the lowest in five disciplines.

This shows that the current cutting-edge model, even with the fullest support of 3D data and tools, is still far from being able to observe and reason in the face of real-world scientific tasks.

More points don't mean to be observed, to be reasoned.

Prior to ScienceVision, there was no such thing as multi-modal evaluation in science, but it was mostly a single discipline, with only two-dimensional images and a set of rating rules.

The unique value of ScienceVision can be summarized in three ways:

First, the question is real three-dimensional.The model is derived from source documents such as molecular coordinates, CT body data, mechanical assemblies, and building models, rather than from a plane diagram. The answer is to use field knowledge to explain three-dimensional structural evidence, and some questions must first identify the goal from observation, then connect topology and geometric reasoning, one ring to another.

Second, the tools are done according to the expert workflow.Point of view switching, target isolation, snippet browsing, scientific measurement, these inspection operations are open to models via MCP, depending on what they can do with the viewer.

Third, the scores are definitive, the five disciplines are the same set.Each question has a reference answer and preset rules, and each component of the answer is separated by a test, followed by a model configuration and a subject summary score.

The SciVisAgentBench, which is most closely associated with it, also supports 3D interactive checking, but assesses different objectives:

ScienceVision uses the same category of questions in all five disciplines, namely structural questions constrained by field knowledge, combined with structured field questions and question-by-question reference answer scores, while SciVisAgentBench focuses on tasks in science visualization software that evaluate whether the visualization and analysis results produced by the Agent meet the task requirements.

ScienceVision, for the first time in a unified framework, measures the limits of AI's ability to observe science in three dimensions.

If you know where it is, you will know how to track it.

The next time someone announces "AI scientists are here", you can have it finish the 250 questions and say more.

This is an open source link to GitHub:

Benchmark links: