Paper information
The title of the paper is:SeeSE3: Emergence of 3D Space in Vision Features
Authors:Caroline Chen, Sayna Ebrahimi, Fedor Kitashov, Ming-Hsuan Yang, Leonidas Guibas, Viorica Pătrăucean, Maks Ovsjanikov
Institutions:Google DeepMind
Link to the paper:https://arxiv.org/abs/2607.14228
If there existed a structure that was immobile from birth and could only passively receive visual signals from the outside world, could it "discover" the structure of three-dimensional space?
This is not a scientific setup, but a famous thought experiment proposed more than a century ago by the French mathematician and physicist Henri Poincaré. Today, a group of researchers from Google DeepMind re-examined the hypothesis using a modern visual base model and gave a surprising answer.
Researchers investigated whether visual features were aligned with geometrical structures that were shifted by Euclidean geometry, i.e. space shifts captured by camera movements in static scenes (trajectories). The original features were entangled in chaos (mid-left), while the Pongaly adapter was able to identify sub-spaces associated with space movements from the characteristic space and "expand" them into homogeneous coordinates (mid-right and right) that closely follow real camera mechanical trajectories.
One quote: The Hundred Years' Question by Pongalle
The three-dimensional structure of the perceived world is an integral part of human development. In contrast, in most existing visual, robotic, or world models, the geometry of three-dimensional space is a pre-imposed fallacy, either by explicitly selecting a particular coordinate system or by distilling it from human knowledge. This naturally raises a question: can visual models "discover" the three-dimensional structure of the world from the statistical laws of visual data only with minimal intervention?
In cognitive science and philosophy, a prevailing view is that in order to discover the geometry of Euclidean space, one must be energetic. Poincaré assumes: "It is never possible for an immobile being to acquire the concept of space because he has no reason to distinguish between the variation of the position of a sphere and the variation of the state of a sphere. He cannot acquire this concept if his motion is not autonomous".
Modern basic visual models provide an excellent opportunity to test this hypothesis. These models, strictly speaking, are "still beings", and even though they are trained in video data, they are just bystanders, passively watching other people's movements. Inspired by Ponggal, the researchers posed the following questions:
The Poincaré Task: Can a stationary observer, with only passive visual input, "discover" the structure of three-dimensional space? Specifically, will a visual model naturally organize its underlying space in a way that reflects the Euclidean transformation group SE (III)?
It is worth noting that the potential space of the visual model associates features with the content of the scene, whereas the content of the scene naturally exists in the three-dimensional space, but the three-dimensional space itself is not directly observable.Whether the characteristic space itself is organized in a form that reflects three-dimensional structures in eugeared.
The two key features of Euclidean space are itsLow dimensional(Location and camera towards a total of 6 degrees of freedom) andHomogeneityRelative displacement has the same meaning anywhere in space. This seems to be in sharp contrast to the high dimensions of visual features and their fundamental combination with visual content. Thus, even in the most powerful models, it is not obvious in advance whether such a low-dimensional, homogeneous coordinate system exists.
To answer this question, the researchers defined a set of evaluation protocols that tested multiple visual base models on static scenarios from multiple datasets. The experimental results showed that the information contained in multiple purely self-monitored image models, when correctly detected, showed a strikingly strong alignment with the spatial structure of the SE[3], i.e. these models were able to "see" the SE[3]. Based on this insight, the researchers further proposed an efficient method of returning camera motion and achieving closed potential visual navigation.
Second, the core contribution
Four key contributions to this work are as follows:
1.Officially, the problem of deducing three-dimensional spatial structures from visual characteristics is posed by comparing the structure of the characteristic space with the Euclidean transformation group SE[3] and introducing a set of detection indicators to evaluate the performance of different types of encoders on this task.
2.It has been demonstrated that although the original form with no characteristics is directly synonymous with SE[3],[4] even in the local sense, this structure can be extracted by a lightweight synthetic decoder; the researchers call it the "Poincaré Adapter", which produces a homogeneous coordinate system.
3.A self-monitoring method provides theoretical analysis of which conditions under which geometric structures can be restored, and studies of difficulty differences in the recovery of SE[3] of different proportions. For example, it demonstrates that rotation is easier to code than symmetry. At the same time, large-scale melting experiments on different variable visual characteristics trained on the same data set have established the necessary conditions for the emergence of geometric structures.
4.Based on the above insights, a novel closed-visual navigation approach has been proposed that takes full advantage of the linearity implemented by Pongaly adapters.
Overall, this work explores the conditions under which three-dimensional spatial structures emerge in visual-based architectures. These insights are also closely related to visual navigation models (Visual Navigation Models) and navigation world models (Navigation World Models).
Three: Method summary: four measurements with Pongaly adapters
The researchers designed four measurement metrics for systematically detecting "spatial perception" of visual models, ranging from shallow to deep, with increasing complexity:
Topological Alignment of the M1 scaleUsing mutual k-neighbor alignment, comparing the similarity of neighbourhood structure in visual feature space to neighbourhood structure in camera position space. This is completely untrained, and directly evaluates the degree of matching between feature space and position space at the topological level. High scores mean: visually similar images are also close in physical space.
Measurement of M2 metric This is the intrinsic dimensionalityUsing two estimators: MLE and Two-NN, to calculate the local characteristic dimensions. If a characteristic space is perfectly aligned with the three-dimensional motion space, its dimensions should be close to 6 ((SE)) (3) degrees of freedom.
Measurement M3 Linear equivariance and other variables: directly train a linear regression W and try to predict the relative position change from the original characteristic differential:ΔP ≈ W(z_{t+s} - z_t). If the characteristic space is a natural flat e-kilometre space, this simple linear mapping is sufficient.
This is the first time I've been able to do this, and I've been able to do this.Since linear constraints were too stringent, the researchers introduced a lightweight derivative MLP adapter φ_θ, formatted as:ΔP ≈ W(φ_θ(z_{t+s}) - φ_θ(z_t)). Its central idea was that by means of an off-linear projection φ, the curved characteristic flows would "unfold" into a flat, homogeneous coordinate space, allowing position changes to be decoded linearly.
The name "Pongaly Adapter" is a tribute to its mission to unveil homogenous three-dimensional coordinate systems in characteristic space, allowing the unmoving eye to see space.
Two sets of example pairs, with intervals of s=40, show the typical change in perspective that Pongaly adapters need to decode. The difference in intervals includes significant changes in the scene caused by camera plane and rotation.
Latent Space Navigation, also known as Latent Space Navigation
As a validation of the application of this study, the researchers proposed a sub-space navigation scheme. The central idea is that if visual sub-space can be constructed to produce a local topology that reflects SE[3], then an intelligent body should be able to "imagine" the visual consequences of physical motion without explicit three-dimensional reconstruction.
Given the initial coefficient of adaptation characteristic g_t = φ_θ(z_t) and the expected byte shift ΔP, the target coefficient characteristic is predicted by simple vector operations:
ĝ_{t+s} = g_t + W† ΔP
Where W† is the inverse of the linear projection weight. This means that navigation in subterranean space is as simple as doing an additive.
Four, experiments and demonstration of effects
4.1 Topographic alignment: The DINO series is amazing
In the ScanNet dataset, the researchers first assessed the consistency of the characteristic neighbourhood structure of multiple visual models with the neighbourhood structure of the physical world. Short-term steps (short-distance gaps) depend on pictorial imaging, while long-term steps (large angle changes) are more likely to reflect real spatial perception.
The coordinates between the camera position and visual characteristics below the x-axis and the y-axis. The former depends on the pictorial level, while the latter is more capable of representing spatial perception.
This chart reveals several key findings:
- Models that receive explicit geometric target training (DUSt3R, MoGe, MASt3R) achieve the highest alignment scores at long ranges (≈0.35 x 0.40), indicating that their encoders themselves structure sub-spaces to align position spaces.
- The most surprising thing isIn purely self-monitored visual models, DINOv2 and DINOv3 exhibit extremely strong topological alignment (≈0.38), which is significantly superior to other baseline models and primitive pixels. This structure emerges purely from passive image-level self-monitoring without any apparent depth or pitch signals.
- Video models (VideoMAE, RVM, 4DS) performed poorly on this scale (≈0.15×0.25). The researchers hypothesized that while these models could capture time-square motion, their underlying states were highly dependent on context (intertwined with generating trajectories) and failed to form a stable, universally consistent spatial mapping.
4.2 Dimensions and linearity of primitive characteristics
The representative model's characteristic dimension (TwoNN/MLE) is contrasted with the original linear reading R2 (M3). The numerical correspondence maximizes the encoder layer of M3 performance.
Table 1 shows the characteristic dimension and linear decoding properties of the representative model. The characteristic dimension of most encoders is near 6 ((according to theoretical predictions), but R2 of linear variables such as M3 are all negative or close to zero.
4.3 Pongalay Adapters: Unlocking the Linear Structure
By applying a lightweight MLP adapter φ, the researchers recovered a very strong linearity such as variation. The average test R2 in ScanNet's three scenarios reached about 0.65 tonnes, which means that there is a local Euclidean fluid in the potential space, which is equivalent to SE 3 (Visual Grid Code), which can be accessed via non-linear projection.
Quantitative evaluation of the adapter R2 under different step lengths on ScanNet. The test was to determine whether the potential characteristic changes observed across the s-series could be linearly mapped to changes in camera position. DINOv2 showed the strongest performance (R2≈0.65), while the original pixel remained negative R2 even with the adapter, confirming the decodable structure derived from the learned characteristics inside.
It is worth noting that although the CroCo has been pre-trained for explicit 3D cross-view complementation, its performance (R2 ≈ 0.38) is opposite to that of DINOv2.Large-scale passive observation (scale of DINOv2) may be more effective at discovering spatial laws than small-scale explicit 3D pre-training.
4.4 Generalization and scaling across data sets
Aggregate decoding indicators on five datasets, three representative skeletal models (Max R2/R2>0 ratio/R2>0.3 ratio) have a significant impact on decoding.*
In large-scale tests on 5 datasets and 913 standalone scenes, the researchers found that the difficulty of the scene was driven primarily by three factors: 1) the success rate of scenes with data-sufficiency pairs > 1000 was 81 percent; 2) the visual richness and high-texturedness of the room provided the necessary semantic focus; 3) the geometry of the workspace and the boundaries of the room naturally supported the overlap and closure of dense angles of view.
31 remaining zero sample positions in the test room recovered with the change in the number of training rooms N. Left: Successfully recovered room ratio ((Top-R2>0) monochrome growth. Right: Total recoverable signal ((Clipped Mean R2)) Significant growth and unseen platform period.
Figure 5 shows the characteristics of generalization and scaling. Poincaré adapters were trained in an increasing number of ScanNet rooms and evaluated zero-sample posture recovery capabilities in 31 remaining test rooms. The adapter successfully recovered the positive R2 signal in the room ratio from about 18% at N=5 to 50% at N=200, while the overall recovery signal increased significantly and confirmed an unseen platform period.Geometric perception is a scalable, general emerging feature of the Visual Transformer.It can be transferred to an unseen environment.
4.5 Submarine navigation: "walking through character space"
Identity baseline ((no navigation) calibration task difficulty. Zero sample Inv. Poincaré has the advantage over topological retrieval ((Hit@ε), although Attention corrector return indicator ((R2)) is higher.
Based on the linear structure unveiled by the Pongaly adapter, the researchers tested sub-space navigation schemes. The zero-sample Inverse Poincaré method (which is highly competitive in topological detection and far exceeds non-navigating Identity baselines) is better at retrograde indicator R2 than in physical region detection.To explain the proper space-neighborhood movement, vector calculus is needed.
4-step open-circle navigation trajectory (ΔP = 0.85m, 36°). The path is slightly drifted due to the open-circle points, but the retrieval of the lens is coherent and successfully approaches the target angle.
Figure 6 shows a 4-step open-circle navigation trajectory (ΔP = 0.85m, 36°): the path is slightly drifted due to the open-circle points, but remains coherent in view of the detected lens and successfully approaches the target angle. For medium-range movements (0.68m, 19°), the detected trajectory accurately reaches the target angle; even for extreme-range movements (2.45m, 37°), the spin recovery is still extremely accurate.
Gathering across the roomIn 31 experiments in the remaining test rooms, zero-sample navigation was better than baseline in about 60% of the rooms; with Ridge-Refit (only adapted to linear mapping) the navigation success rate was pushed to 90-94%, across the three backbone networks DINOv2, DUSt3R and V-JEPA.
Five: Conclusions and prospects
In this paper, the researchers explored whether visual characteristics could capture structures in Euclidean space by studying the alignment of these characteristics with the congenital motor group SE[3]. The study showed that although no encoder directly reflects this structure, in some cases it is possible to "unfold" characteristic flows through a relatively simple network of adapters.
It's fascinating to see thatChanges in visual characteristics can be mapped to position by a homogeneous, location-independent, low-dimensional adapter.Interestingly, self-monitoring models, although trained only through passive observation, can form three-dimensional representations of space and can be decoded when detected correctly.
One limitation of this study is that: considering only static scenarios in which the characteristic changes are only related to the motion of the camera, the distinction between "position change" and "state change" has not yet been examined by Pongaly. An interesting extension of this work is to extend the formalized approach and path of study to dynamic scenarios in which there is both camera movement and scene movement from underlying physical processes in order to explore deeper time-space symmetry or the laws of constancy.
SummarySeeSE3 provides a systematic proof-of-concept and theoretical framework for the visual-basic model. It reveals an important fact: the geometry of three-dimensional space can emerge from pure, passive visual observation without the need for dynamics, prior knowledge of three dimensions, and only the need for large enough data sizes and proper detection.
This article is for academic sharing only. If there is an infringement, please contact the deleted text.