Recently, the GPT-6 Astra's three-dimensional modeling and spatial understanding demo has attracted attention. It can be used to build homes with furniture and courtyards in the Blender, and then turn the scene into the Unreal Engine 5, allowing people to enter rooms, open cabinet doors, and even operate coffee machines.
GPT-6 Astra can create realistic 3D house models based on home source photos
As we look at how models construct these three-dimensional scenarios that can be entered into, interacted with each other, a question arises: how exactly does the interior of the model represent spatial relationships between objects?
Now, researchers have found a structure similar to a spatial map in multimodal models. After analyzing the mid-activation of multiple multimodal models, the MirroS team found that the model would place objects in a steady low-dimensional representational space in the right, up, and close positions.
More interestingly, this "internal map" is not limited to the objects captured by the camera. The model can infer the location of the outside football along the direction of the goalkeeper's rescue; "you" or the observer himself, the ball will appear at the near end of the axis; abstract concepts such as "socialist", possibly because of the linguistic meaning of "political left", are placed at the left end of S-Space.
But the study also reveals two weaknesses in spatial reasoning: models sometimes confuse different coordinate systems, such as mistaking the "right" in the image as the "east" in the basic direction, and mistaking the reference system; models are not necessarily able to accurately rotate the coordinate system, even if the object's position has been preserved internally, and changing an observer's viewpoint may still be wrong in judging their relative relationships.
In other words, the model already has spatial representations, but it cannot be reliably used.
This is an interactive blog:
Technical report:
The code is:https://github.com/MirroS-Lab/S-Space
Space coordinates read by different objects in S-Space
Inside the model, there is already a 3D map.
Recently, Anthropic discovered J-space inside Claude. It's like a silently running "language workspace": models can temporarily store concepts that haven't yet been exported, for reporting, controlling, and subsequent reasoning calls.
S-Space focuses on another question: When a model sees the real world, does it also form a spatial workspace for subsequent computational calls? The term "space workspace" here refers to a shared representation of an object and its location that can be read, written, and operated together by multiple processes such as perception, reasoning, and output.
It is difficult to answer this question from the output alone. A model that says "the cup is on the left side of the book" may be because it actually establishes a scene coordinate, or it may simply recognize some local image pattern.
Therefore, the researchers focused on the middle layer of the model. They assumed that in the hidden state of the object's name corresponding to the token, the linear encodes the position of the object in the space coordinate system. The team used natural scene images to construct the space problem, and used the gradient of the answer to find the horizontal, vertical, and distance in three directions.
Experiments show that the numerical values read from S-Space are significantly correlated with the actual continuous coordinates of objects. We can even bypass the original language reasoning process of the model: just comparing the coordinate differences between two objects in S-Space can be done directly on the CV-Bench, EmbSpatial-Bench and SpatialTunnel.
S-Space's three coordinate axes and how to read coordinates
More importantly, this set of spatial structures exhibits stability across datasets, interpoints. S-Space can be generalized to a data distribution that has never been used in extraction; even changing the way in which the question is asked and changing the color of the object, it can still read its spatial position from the same set of coordinate axes. This indicates that S-Space is a continuous internal structure, rather than a characteristic temporarily induced by specific spatial problems.
Furthermore, although S-Space is entirely derived from single view samples, it can still form meaningful coordinates in multiple view inputs. When prompted to specify "View 1", any image that is exchanged is called "View 1", and the read coordinates are switched to the corresponding perspective. This indicates that S-Space not only persists, but can also be tailored to the specified perspective by language dynamics.
When you dial the internal coordinates, the answer to the model changes as well.
Being able to read spatial information from internal representations, however, does not mean that the model really relies on that information to make judgments. S-Space may also be just a "scorecard" that faithfully records spatial status: it accurately displays the score but does not actually participate in the game.
To test this, the researchers directly intervene object tokens in the coordinates in S-Space. For example, swapping two objects in S-Space without changing the original image and the problem, then allowing the model to continue to complete the subsequent calculations. This manipulation allows the model to change the spatial judgments in the language output based on the modified spatial structure: left to right, up to down, near to far. At the same time, unrelated judgments such as object colors are largely unaffected.
After interfering with S-Space, the model's spatial judgment changed.
This means that S-Space is not just a by-product that can be decoded by external tools, but is implicitly involved in the spatial judgment of the model. There is indeed a spatial workspace within the model that can not only be read, but also written and manipulated.
Spatial semantics combine to make the model use the wrong coordinate system
The formation of S-Space may be related to the learning of spatial language in pre-training. In order to predict words such as "left" and "right", visual language models must tie the entities in the image to their relative positions and allow the language-generating system to summon this information. Even if no one explicitly provides a coordinate, this training pressure may also organize a stable spatial axis within the model, or even extend beyond the image.
S-Space, based on the goalkeeper's rescue directions, positioned the ball not in the picture
The problem is that while language describes space in church models, it also introduces its own ambiguities and different frameworks of reference. Abstract metaphors such as "left" and "right" in politics are also arranged along the horizontal axis of S-Space. Physical space in vision shares a partial representational structure with metaphors in language.
A specific example is the conjunction between the basic direction and the relative direction of the image. Researchers found that in Qwen's S-Space, east, west, north, south were aligned with the right, left, up, and down, while northeast contained both the right and upward proportions.
In an example of the MMSI-Bench, the problem has given a "north" relationship, but the model understands the "right" in the image as "east" and finally answers "northeast" instead of "north". It does not first determine how the direction of the subject is defined, but instead directly applies a map coordinate.
S-Space makes the mechanism behind this failure visible: the base direction shares some spatial characteristics with the direction relative to the image, and the directional cues in the image may therefore be mistaken for the base direction. The correct use of spatial representation requires first distinguishing the reference frame used for the title.
The basic dimensions and relative directions of the image are combined in S-Space.
Models know where objects are, but they don't rotate the coordinates inside.
In addition to the confusion of reference systems, there is another difficulty in spatial reasoning: how to correctly translate existing object coordinates into new perspectives?
The team tested this with SpinBench's perspective-converting task. After seeing a positive scene, the model imagined looking from the left, right or back, and then judged the position of the two objects under the new perspective.
In the example of Qwen 3.6-27B, the accuracy of direct responses was 60.96%, which increased to 83.56% after being added to the thought chain. The names of objects that repeatedly appear in the process of reasoning can be seen as they gradually rotate in the direction of the coordinates in S-Space: early reasoning, the coordinates are closer to the original perspective; as reasoning unfolds, they are gradually aligned with the new perspective.
But with these phenomena alone, it is not possible to judge whether the error comes from the initial spatial perception or from subsequent changes in the coordinates. The researchers therefore explicitly reconcile spatial perception with spatial computation: first reading the relative coordinates of the object from S-Space, then applying a fixed rotating matrix to the outside of the model, and finally responding to the rules of determination.
Using the model's own spatial representation, the Qwen 3.6-27B achieved an accuracy of 95.21%, further exceeding the results of thought chain reasoning.
Direct responses, as opposed to the accuracy of reading and rotating S-Space.
This gap reveals the difference between "perception" and "computation" in spatial reasoning: the spatial characteristics within the model have been formed enough to support more accurate answers, and how to steadily transform these coordinates remains its weak link.
It's not just scaling, it's understanding the model mechanism.
The study provides a more detailed method of diagnosis than the final accuracy rate: separating the discussion of "is there a correct representation" and "is there a use of representation". The former requires more vision and physical experience, while the latter may require specialized mechanisms such as explicit coordinate switching, simulation, memory, planning, and tool calling.
From Anthropic's J-space to S-space, an increasingly clear direction is that explainability should be used not only to explain why a model makes mistakes, but also as a scientific instrument for observing and designing intelligence. It helps us observe the internal mechanisms of the model and helps us determine which capabilities will naturally emerge with Scaling, which still require new training signals, computing mechanisms or physical experience.
For AI to move into the physical world, Scaling is still important. But the next, perhaps more critical, question is: what kind of worlds the model rebuilds internally, and what is needed to make reliable changes, iterations, and actions about the world.
After all, understanding space is more than just seeing it.
It's about building it from within and learning to change it.