A small ball falls from the air, a beam of light shines into the mirror, a floating piece of stone slowly melts. For people, these are familiar physical phenomena. For models, they are not an easy test.
In today's model, images can be generated very clearly, objects can be generated very consistently, and movement can look smooth. But does the ball follow the law of free falling bodies? Is the angle of reflection of light correct?
The latest work by the Navers Lab of Einsia AI, "World Models' Last Exam in Physics", tested eight video-generation models using 40 missions and nine physical phenomena. In the current model, Seedance 2.5 ranks first, but the average score is only 57.76/100.
It wants to explore the essence of:
"Does the existing video generation model comply with the laws of physics?"
The title of the paper is World Models: Last Exam in Physics
The main page of the project:
GitHub：
arXiv:
The picture is consistent, is it physically correct?
To determine whether the physical laws behind the coherence of the images are the same, the research team constructed 40 video-generation tasks covering nine categories of physical phenomena, including mechanics, optics, fluidity, transformation, electromagnetism, and so on. From a small ball down to a single oscillation, from light reflection to floating melting, each question required a model to generate a specific physical process and to undergo a test of the corresponding physical relationship.
For the model, the task is very direct:
"Give an initial image and a hint to generate the next physical process".
The input specifies the initial state of the scene, as well as the Prompt of the events that need to occur next; the output is a video generated by the model. The model needs to preserve the relationship between the object and the scene in the image so that the specified process continues and that its movements and changes conform to the corresponding physical laws.
Starting from a defined initial state, a physically established follow-up process is generated, which is exactly the task the model needs to complete.
This is where the difficulty lies.
How do you measure the physics in the video?
To evaluate a video that is generated, it is necessary to answer two separate questions: is the video complete, consistently presenting the required scene, and is the process in the scene consistent with physical laws?
The assessment will therefore be divided into two parts: the consistency score C and the physical score P, both of which use the 0×100 classification system.
Conformity score C measures the consistency of the subjects, events and scenarios in the video and provides the observation basis needed to complete the task.
For example, when evaluating a free fall, a small ball should remain visible, not disappear suddenly, become another object, or appear to jump in a position that cannot be traced continuously. When evaluating two single swings, it is necessary to be able to identify the two swings, observe the release and oscillation processes, and see the required parts of the cycle compared to the swings.
The consistency score is given by the visual language model Qwen 3.6-27B. It evaluates the specific prompts set for each task, guides the model to check whether the required subject is present, whether critical events occur, and whether the objects and processes in the image remain continuous, ultimately producing a score of 0×100 points.
Conformity score C reflects whether the video presents a recognizable, observable task process.
A small ball can always fall along a continuous trajectory, but its relationship to change in position over time may still not be consistent with the law of free falling bodies; two singles can swing evenly, but the relationship between the cycle and the swing may still be wrong.
This difference affects the downstream application of the video model. For example, a robot would need to correctly describe the force and friction of the object if it were to produce a video preview of the consequences of pushing the animal body. When the predicted sliding distance is wrong, the robot may choose the wrong thrust to divert the object away from the target.
Therefore, a physical score P is also needed, which measures the degree to which the physical quantity measured from the video is consistent with the pre-defined physical relationship.
"Physical scores are determined by measurements, without comparison with reference videos, and without direct physical accuracy judgement by visual language models".
This approach covers different fields such as optics, fluids, phase transitions, and electromagnetism, providing a specific test basis for a wide variety of physical phenomena.
Its core idea is to measure the "physical quantities" in the video and check if they meet the corresponding physical relationships.
The program uses splitting, tracking, geometrical matching and event detection to extract trajectories, cycles, angles, and fluid surfaces from the video, and calculate errors and scores according to the indicators of each task. Physical scores are determined by these measurement results, without comparison with reference videos, and without the need to rely on visual language models to directly determine physical correctness.
For example, the square of the ratio of cycles to the ratio of lengths should be equal to the ratio of lengths. The program detects the relative error in calculating the relationship between the two cycles T1, T2 and L1, L2 from the video:
If the result of the measurement is completely consistent with this relationship, e = 0; the greater the deviation, the greater the e.
Of these, 0.20 is a predefined error scale that is consistent with all models.
The error is 0 to 100 points.
The relative error in this relationship is 20%, with 50 points.
The relative error was 40%, with about 33.3 points.
A task can contain multiple indicators. A single-setting task can also check whether the cycle is stable, in addition to checking the relationship between the cycle and the cycle, and whether the cycle of repeated swings is stable. After the indicators are calculated according to the task rules, an arithmetic average of the defined indicators is obtained by a physical score P. If the indicators have been converted into percentages:
In this way, the scores correspond to specific measurement results: exactly which physical relationship the model retains and how much deviates.
Seedance 2.5 produced video rating ((left 72.54 points, right 97.70 points)
Total score S combines consistency and physicality: consistency is 15% and physicality is 85%:
But the physical part has counting conditions. When C ≥ 80, only the physical part records are calculated. Otherwise, the physical part is not counted in total points.
For example, when C is 100, P is 60, the total score is 66; if C is only 60, the total score is only 9. Even if it is recorded at higher physical scores, the total score is only 9.
Each assessment also preserves intermediate measurement results and visual evidence to help researchers track down the reasons for the low scores: whether the movement deviates from the rule, critical events do not occur, or the picture lacks sufficient measurement evidence.
The answer to this question is: Is the physical process in the image equally correct when the model is already capable of producing realistic, coherent video?
Seedance leads, but the highest score remains at 57.76.
Of the eight video generation models reviewed, Seedance 2.5 scored the highest overall average score at 57.76, followed by the MiniMax H3 at 54.89.
Understanding the laws of physics is still the soft side of the current model
Transformation is a common physical phenomenon, such as ice turning into water. For the creation of this simple phenomenon, only one Seedance 2.5 led the way with a score of 95.61, while the rest of the models scored up to 15 points.
Ice melted, liquid level unchanged, Seedance scored 95.61, and MiniMax produced ice that did not melt, did not follow Prompt, scored 0.00.
Optics is also a common difficulty point for all models. For example, with full-reflection and full-reflection mirrors, the average overall score of eight models is less than 40 points. Models are able to generate the visual appearance of light beams and medium interfaces, but it is difficult to maintain the corresponding physical constraints during the course of light path changes.
The MiniMax H3 can generate partial refractive light, but is incorrect, with a score of 31.67; Cosmos cannot generate, with a score of 0.00.
Another common difficulty is the free fall. In this task, the average consistency of the models is 98.75 points, while the average physical score is only 13.88 points. Small balls can fall completely, coherently underground, but the measured motion may still deviate from the physical relationship required by the task.
Seedance scored 29.50, and Wan scored 22.72.
These results suggest that the current generation model still fails to understand the laws of the physical world in most scenarios.
How close is this score to human judgment?
In order to assess the consistency of the physical quantity scores in this study with human judgments, the research team invited 10 markers to perform an artificial evaluation of 320 video clips covering 40 tasks and to compare the measurement method directly to a visual language model (Direct VLM).
There are two types of comparisons: one that allows the tagger to score the video and then automatically checks whether the score matches the human rating of the video's inferior ranking; and the other that shows the two videos directly, allowing the tagger to determine which section is more consistent with the laws of physics.
In the two comparisons, the study method improved by 8.18 and 7.50 percentage points, respectively, compared to Direct VLM, with a consistency gap of about 5 percentage points between humans.
The results show that extracting physical quantities from videos based on specific relationship scores is closer to human judgment than allowing visual language models to directly judge physical correctness.
From generating images to predicting the world
A generated image looks coherent, but that doesn't mean it actually follows physical laws.
How a small ball falls, how objects collide, and how ice melts are all bound by a clear physical relationship. Video models can paint the process out of thin air without actually maintaining the laws in it.
The results of the World Models' Last Exam in Physics show that once it appears reasonable from the point of view of logic to further inquire into the quantitative relationships between motion trajectories, cycles and objects, the limitations of the current generated model are quickly exposed.
The study ultimately raised a more fundamental question:
"Is the world model generated just a mirror image, or a future that actually complies with the laws of physics?"
From generating images to predicting the world in the middle, it's the boundaries that the next phase of a world model needs to cross.