Paper information
The title of the paper is:IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer
Authors:Zhengyu Zou, Hao Li, Kuixuan Jiao, Liu Liu, Tingyang Xiao, Xiaolin Zhou, Fangzhou Hong, Zhizhong Su, Dingwen Zhang, Ziwei Liu
Institutions:Horizon Robotics; S-Lab, Nanyang Technological University; Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
Link to the paper:https://arxiv.org/abs/2607.19228
Link to the project:https://iggt4d.github.io
Readings
Real-world spatial intelligence requires intelligent agents to understand the motion, persistence, disappearance, and reproduction of scenery objects in the real world through continuous video streams. Although recent space-based models have achieved generalizable forward-thinking 3D reconstruction, most flow methods remain "geometrically centred" and lack the ability to understand objects at a time-consistent level. In other words, they know "where the shape is", but they don't know "if this object is the same as the previous one".
At the same time, existing semantic reconstruction and 3D perception language methods rely heavily on externally extracted 2D semantic clues or loose combined geometric inputs, which limit the unified geometry-example learning in long dynamic scenarios.
The paper presents the IGGT4D, a streamlined example of an on-line 4D landscape understanding of geometric transformers. IGGT4D processes video sequences in sequential order, replicating historical contexts through cause-time modeling, and incrementally updating the unified representation of camera movement, scene geometry and object identity. This enables models to rebuild long sequence forecasts of geometrical-instance consistency in dynamic environments.
To address the lack of high-quality 4D monitoring, the authors further constructed InsScene 4D-147K, a large-scale dataset across real/synthetic, /static/dynamic scenarios, containing RGB images, depths, bit positions, and time-conforming instance masks generated by automated geometrically guided markup flow streams. Experiments in 3D reconstruction, position estimation, instance space tracking and open vocabulary splitting have shown that IGGT4D is superior to existing streamline baselines while maintaining extensible online reasoning capabilities for long dynamic sequences.
IGGT4D's online geometry-example prediction summary. IGGT4D processes dynamic video streams step-by-step, incrementally constructs unified camera motion capture, 3D geometry with timely instance characteristics, 4D representation, supports a variety of downstream applications (example space tracking, open word splitting, scene grounding, etc.); the author has also constructed a large data set containing geometrically guided instance masks Inscene4D-147K, 147K scene, 60M masks) for training and evaluation.
Demonstration of Effects
The 3D reconstruction qualitative contrast of IGGT4D on 7Scenes and ETH3D: compared to LingBot-Map and Stream3R, the point cloud structure of IGGT4D reconstruction is more complete and detailed.
7Scenes and ETH3D dataset 3D reconstructed qualitative contrast (GT/Ours/LingBot-Map/Stream3R, TSDF fusion point cloud, ineffective real-valued depth area has been masked)
In long-series dynamic scenarios, IGGT4D's 4D consistent disguise is able to stabilize "bite" target objects, while baselines such as SAM2, SpaTrackerV2+SAM cause target loss or identity drift.
For example, the qualitative visualization results of space tracking (GT mask / IGGT4D / IGGT / SAM2 / SpaTrackerV2+SAM).
Open vocabulary semantically splitting qualitative results: whether it's a table, a cabinet, a chair, a keyboard, a display, IGGT4D's splitting boundaries and semantic predictions are closer to true values.
Results of qualitative visualization of open vocabulary semantically divided (RGB image / GT tag / IGGT4D / Feature3DGS / LSeg / OpenSeg)
References
Real-world spatial intelligence is essentially on-line, dynamic. Physical intelligence receives continuous video streams, in which objects move, are obscured, leave the field of view, and reappear over time. In such an environment, it is not enough to simply estimate the motion of the camera and the geometry of the scene: intelligent objects must also maintain a time-consistent identity of the object.
Spatial reconstruction and scenario understanding are undergoing a transition from stream forecasting based on optimized flow streams to data-driven flow streams. Traditional geometric flow streams are moving from offline SFM to online and neural SLAMs depending on the evolution of scenario-by-scenario optimization. Similarly, semantic reconstruction systems move from offline scene mapping to online semantic diagrams often build geometric and object-level clues through interdependent modules. These systems, while effective, limit sequence efficiency while optimizing for specific scenarios, and modular design makes it difficult to log consistent instances in dynamic scenarios.
Recent spatial basic models have addressed these limitations by learning 3D predictions directly from images through generalizability. For online long-series perception, this paradigm is moving further from holistic fixed set logic to streamlined reconstruction. However, only streamlined 3D predictions are still insufficient for physically intelligent agents. Current streamlined models are largely still geometrically focused: they estimate depths, positions, and dot charts, but do not maintain a coherent object identity.
A key reason for this is the lack of proper supervision. The 2D visual language model provides extensible open vocabulary clues, but its predictions are perspective-related and lack 3D quantification. The semantic reconstruction method improves spatial consistency by upgrading 2D semantics to 3D, but its semantic reliability is limited by the ability of external 2D predictors.
In order to fill these gaps, we presented IGGT4D, a flowchart for understanding online 4D scenarios using a flowchart Transformer. IGGT4D transforms geometric instance reconstruction from a whole-series offline problem into a causal flowchart prediction problem. It processes the video file sequentially, duplicates the historical context through cause-and-effect modeling, and integrates incremental updates of camera movements, scene geometry and instances into a unified representation.
We further built InsScene 4D-147K, a large-scale data set for learning 4D scene geometry-examples. It spans real/synthetic and /static/dynamic data sources, providing RGB images, depth charts, camera positions, point clouds, and sequence-level consistent instance masks. To reduce marking costs, we designed an automated geometry that guides marking streams and generates large-scale geometry and instance labels that are consistent with multiple viewpoints.
Major contributions
We will understand online 4D scenery as a causal prediction of camera movement, scene geometry, and persistence as objects from continuous video streams.
We proposed IGGT4D, a forward-thinking Transformer, which combines causal geometric modeling with geometrically-determined instance prediction and online clustering to maintain a consistent object identity without the need for whole-series reasoning.
We built InsScene 4D-147K, a large-scale real/synthetic, /static/dynamic dataset whose instance labels are geometrically consistent and are generated by a scalable reconstruction, projection and masking streamline.
The method
The problem is formalized.Given an RGB image sequence and optional camera parameters, IGGT4D performs forward-looking sequence reasoning, which results in four types of results for each set: camera parameters (spinning, rotation, viewing angle), radiographic diagrams (ray origin and direction), depth diagrams, and 8-dimensional instance characterization diagrams.
Causal geometry - Transformer examples.The authors converted the DA3's Unified Geometry Transformer from fixed set 3D prediction to a Causal Stream 4D understanding: each image is encoded as an image token, and is embedded with the camera token; when there are camera parameters, it is encoded by the parameters, or it can be learned using the shared token; then the Transformer blocks are interchanged between attention and attention across angles after 40 viewing angles. Unlike DA3's two-way fixed view logic, the IGGT4D applies a causal mask to attention across cameras and across angles, allowing each node to focus only on past and past observations while also using the same constraints to simulate the flow projection.
IGGT4D overview. Given dynamic video sequences with optional camera positions, IGGT4D extracts representations consistent with time and space; Tri-DPT leads to incremental predictive geometry and instance characteristics, streamlined aggregation algorithms derive instance masks to realize online 4D scene understanding.
Tri-DPT geometry - example heads.The authors designed the Tri-DPT head to combine depth, radius graphs and instance characteristics from flow-scale tokens: all branches gradually restore spatial resolution, while the instance branches combine geometrical perception of attention with geometrical supporting branches with depth and radius characteristics as a structural precursor, embedding instances into 3D geometry, improving the time consistency of object identity.
The Tri-DPT head, which combines depth, radius and instance characteristics, geometric perception attention in four convergence phases, injecting depth and radius precedence into the instance branches, and in the final phase refining the instance characteristics with instance attention.
Classification of efficient flow examples.IGGT4D proposes a two-stage flow-through aggregation strategy that maintains a lightweight universal instance code book (each instance preserves a feature center with a cumulative number of pixels): in the first stage, in-phase aggregation, high-string resemblance pictograms are aggregated into core areas, and then extended to local masks by more loose connectivity segments at the threshold value; in the second stage, the local center is inherited as a holistic ID4D consistent masks, becoming a new holistic order of matched instances. As a stand-alone entity, the region as a stand-alone entity is not neglected by the natural disconnected dynamics of the matching with the background.
Understanding the application of 4D scenes.4D consistent occultation and time sequence-associated instance characteristics constitute replicable object-level representations: stable spatial tracking can be achieved without additional associated initiation of the occultation of durable instance correspondence; time-space consistent language characteristics can be obtained by combining occultation and 2D visual language characteristics for open vocabulary splitting; these occultations and characteristics can also be provided for large multi-modular models, supporting dynamic object splitting, visual field object reasoning, and other 4D scene grounding tasks.
The goal of training.In training, all positions are aligned with the coordinates of the first camera, using the first-point cloud computing sequence scale to avoid flow reconstruction; geometry monitoring includes depth; L1 regression + confidence normal + spatial gradient consistency); radiation; point-to-camera loss; and instance characteristics use multiple view versus loss optimization to drag their pictorial characteristics to the center of the instance, push the different centers, pull the matching center between the angles, and launch the different centers.
InsScene 4D-147K is a data set.The data set includes 147K selected video sequences across four main areas: static-real real estate 10K, ScanNet++), static-synthetic Aria Synthetic Environments, SceneNet, Infinigen, Hypersim, dynamic-real HOI4D, Waymo, Aria Digital Twin, dynamic-synthetic RoboTwin 2.0, Kubric, DynamicReplica, Point Odyssey, VKITTI2. Each sequence provides RGB images, depths, Io machine images, 4D consistent instance masks that persist with clouds. For dynamic scenarios such as HOI4D, the static grid is reconstructed by removing the dynamic area with a given mask, and then using the available dynamic objects to label the projection false tags covering the dynamic area.
InsScene 4D-147K data builds a flow stream. Real/synthetic, /static/dynamic sources are unified through 3D reconstruction, projection-based ID inheritance and split-based holistic ID refinement three-step processing; the right panel summarizes the data set divisions: static real 10.2K, static synthetic 120.0K, dynamic synthetic 10.2K, dynamic real 6.41K).
The results of the experiment
All experiments were performed on a single 32GB NVIDIA RTX 5090 GPU. The evaluation sequences were separated from the training data at the scene/sequence level.
The camera position is estimated to be reconstructed in 3D.Following DA3's evaluation protocol, the IGGT4D achieved the best performance in HiRoom, ETH3D, 7Scenes and ScanNet++ compared to offline full-attention models (VGGT, MapAnything, Pi3X, DA3) and online stream models (CUT3R, StreamVGGT, Wint3R, Stream3R, LingBot-Map). In position estimation, the IGGT4D achieved the best performance in the flow model (AUC@30.4464, significantly ahead of the LingBot-Map's 0.3060), significantly narrowing the gap with the full-attention model and even exceeding the VG, AnyG alignment baseline.
Geometry benchmarks on HiRoom, ETH3D, 7Scenes, and ScanNet++: (a) camera position estimate (AUC@3、AUC@30); (b) do not use (w/op.) and use (w/p.) real-valued camera position 3D reconstruction (F1 points).
Examples of spatial trackingIn three dynamic dataset HOI4D, Waymo, PointOdyssey, and the static dataset ScanNet+, following the IGGT protocol as opposed to SpaTracker V2+SAM, SAM2 and IGGT, reporting time sequence mIoU ((T-mIoU) and time sequence success rate ((T-SR). Under long sequence settings (about 100 tonnes, HOI4D is 40 tonnes), IGGT cannot run directly due to OOM overflow, while IGGT4D can be extended to 4D sequence lengths with flow logic, with an average T-mIoU of 58.84; T-SR of 85.41, significantly ahead of SAM2 of 45.38/58.65; short sequence of about 16 tonnes) and similarly overall ahead of T-mU of 59.25 vs. IGGT58.68.
HOI4D、Waymo、ScanNet++ and PointOdyssey are examples of space tracking. (a) Long sequences (about 100 tonnes, HOI4D is 40 tonnes); (b) Short sequences (about 16 tonnes).
The visualization of instance characteristics and masks further explains: 3D consistent instance characteristics PCA projections correspond to masking heights generated by flow clusters, and object identities remain stable in the cavity.
Example characteristics and masking visualization. PCA results for 3D consistent instance characteristics are displayed alongside corresponding instance masking generated by streamlined clustering.
Open vocabulary and semantic segmentation.In contrast to the 2D visual language model on Waymo and ScanNet++ (OpenSeg, LSeg) and the 3D semantic reconstruction method (Scenario-optimized Feature-3DGS, predecessor to IGGT), the IGGT4D achieved the highest mIoU and mAcc in both long-series settings: long-series average mIoU 40.08 mAcc 73.84 mIGGT long-series OOM, short-series average mIoU 39.11 mAcc 72.25, showing stronger splitting performance in both dynamic outdoor and complex static indoor scenarios.
On Waymo and ScanNet++, open vocabulary is semantically divided. (a) Long sequences (about 100 lines); (b) Short sequences (about 16 lines).
I'm not sure what I'm saying.Use of lightweight models in ScanNet++ to dilute: after removing Geo-Attn, instances with decreased semantic performance demonstrate that instance characteristics do indeed benefit from geometric precedents; after removing head geometry unification FF-Norm, re-flow-style reshaping results in severe scale discrepancies that disrupt unified geometry - instances that result in a total drop in all indicators (such as AUC@3 from 0.2984 to 0.2334).
ScanNet++'s fusion experiments. Geo-Attn: Geometry Perception Attention; FF-Norm: Geometry Unification. The visualization contrast of Geometry Unification shows that, in the absence of FF-Norm, the instance characteristic PCA and masking are significantly degraded in flow predictions; the predictions are restored to stability after the addition of FF-Norm.
FF-Norm) as opposed to the flow-through prediction effect of using a first-order geometry of unification.
Flow accumulation efficiency.At 504×336 resolution, compared to IGGT's offline HDBSCAN: IGGT4D, which uses lightweight polynomial codes and string-by-string polynomials, holds about 0.7GB in constant storage; it only takes 7.43 seconds to grow linearly (N=100); while HDBSCAN at N=8 takes 105.8 seconds/13.2GB, N=16 takes 418.3 seconds/25.85GB, and the polynomial is more directly OOM.
Summary & Future of Work
We proposed IGGT4D, a flow geometry-examples framework for understanding online 4D scenarios. IGGT4D's use of causal time-space modeling and geometrically determined instance predictions, combined prediction of camera movements, scene geometry and time sequence-consistent instance characteristics. We also proposed InsScene 4D-147K, a massive data set that spans real/synthetic and /static/dynamic scenarios with geometrically consistent instance labels. Experiments in reconstruction, position estimation, instance tracking and open vocabulary splitting have shown that IGGT4D has improved physical consistency while maintaining scalable flow logic.
Limitations. InsScene 4D-147K expands the scale of 4D instance supervision, but greater robustness and generalization requires wider data coverage. IGGT4D is still largely dependent on supervised learning, which encourages our future exploration of self-supervised learning with larger-scale selective supervision. In addition, our evaluation remains focused on perception; integrating physical action and interactive feedback is a key step toward understanding physical scenarios.
Readers interested in more experimental results and article details can read the original paper.
This article is for academic sharing only. If there is an infringement, please contact the deleted text.