If the three-dimensional scene of this change is to be reconstructed alongside it, the problem arises: the future picture has not yet arrived, how to distinguish which changes come from camera movements, which come from object movements? how to correctly spell out the first minute of reconstructed streets with the next minute of sight, so as not to miss the scene or the dynamic objects? how to avoid re-computing when the video source is constantly input, so that the reconstruction always keeps track of the shooting speed?
These problems point to the three core challenges of online 4D reconstruction: incremental processing, global consistency across time, and high fidelity expression of dynamic scenarios. Online systems can only use images that have already arrived to continuously update scenarios, and not only need to accurately reconstruct local geometry, but must also maintain the consistency of camera trajectories, geometric scales, and dynamic information.
However, existing methods often find it difficult to meet these requirements: scenario-by-scenes optimization methods require repeatedly iterative updating of scenario parameters, and speed is too slow; offline forwarding methods typically combine finite frames, calculation and display storage costs with increasing input sizes, and are difficult to extend to long video; and many forwarding geometric models have primary output point clouds or depths, making it difficult to directly achieve high-fidelity rendering.
DynStream, a collaborative effort by the team of Wang Yong-young at Peking University's Wang Choi Computer Institute (VGI Labs) and UC Merced, proposed a framework for the reconstruction of the 4D Gaussian front-end on-line stream around this issue. The input of RGB online video streams that do not provide camera positions, the model front-end output of the camera trajectory, and the continuously updated dynamic scenarios that can be rendered represent. The method of time-divided video processing completes local reconstruction by forward-thinking, and then connects the blocks to a unified world coordinate system without having to re-run the entire scene or re-optimize for each scene.
Link to the paper:https://arxiv.org/pdf/2610.09720
Figure 1 shows the input and output of DynStream. The model will gradually transform continuous video input into a unified Goose scene, while supporting the reconstruction of indoor and outdoor dynamic static scenes.
Figure 2: DynStream online rebuilt indoor and outdoor scenes from the front.
KITTI Long-Driving Sequence Online Front Door Rebuild Demo
Dismantling methods, from local reconstruction to a global scene
The video is broken down in chronological order, with the Causal Attention Spine Network predicting the relative position to the local Gaussian; the local scene is gradually aligned with the overall scene, whereas the dynamic Gaussian is involved in rendering at the corresponding time.
DynStream mainly solves long video online 4d reconstruction problems with the following three modules:
Because the input is a few minutes long video that spans thousands of meters, the attention layer computation costs in the network will increase rapidly with the number of inputs, DynStream will first divide the input video stream into a number of fixed-length blocks in chronological order (chunk) for the incoming video stream, pre-constructing and re-constructing the local scene through the first online network in the corresponding video block. The advantage of the block reconstruction is that the network is no longer focused on re-constructing the long video overall information integration that requires difficulty, but first focuses on re-constructing the local scene, and then through the scene integration module to consolidate the local scene into a coherent overall scene.
For large-scale scene preview reconstruction, due to the presence of an arbitrary variation of SIM (3) in the local scene that the network predicts in different video blocks, which results in a normal rendering of the scene that is reconstructed in both the previous path and the next path within its own coordinate system. To address this problem, traditional methods often use time-consuming optimization methods, using overlapping cloud responses to make a SIM (3) transformation estimate. Unlike traditional methods, DynStream in training transforms the positioning of the camera output from the positioning of each location to the positioning of the next block, converting the positioning of each output camera to the positioning of the current coordinate, and then converting the positioning of each camera to the positioning of the first block, by converging the key points between adjacent video units.
Dynamically separated scenarios are cumulative modules. DynStream, in addition to the basic parameters of reconstructing scenarios, predicts the dynamic probability for each Goss, and differentiates the static Goss according to a fixed threshold value. The static Goss will be progressively incorporated into the overall scenarios, while the dynamic Goss will be retained in the corresponding time step, and when rendered, the corresponding dynamic Goss and the overall static Goss will be rendered in a chronological manner.
Experimental results, higher quality scene rendering and reconstruction of long-distance tracks
Figure 4 shows the PSNR contrast in the five data sets.
The paper evaluated the scene rendering indicators on five datasets covering indoor and outdoor dynamic static scenarios, including Waymo, SpatialVID, and detailed comparisons to previous scenario reconstruction methods such as AnySplat, MVSplat, NeoVerse, which showed that DynStream performed best on all five indicators compared to the current SOTA methodology of front-end Qualcomm.
At the same time, further qualitative experiments have shown that DynStream is significantly superior to the baseline method in rebuilding the vehicle edge, road structure, and edge of dynamic static objects, while effectively suppressing the reflection of the dynamic parts, with higher overall weight building and rendering quality.
Figure 5 shows the qualitative contrast of Waymo's scenes. From top to bottom, the reference images are GT, DynStream, DGGT and NeoVerse; each row corresponds to the same scene.
Figure 6 illustrates the rendering contrast of various scenarios. From top to bottom for reference images, DynStream, AnySplat, STORM and MVSplat
In terms of long-range camera trajectory reconstruction, the paper also evaluated 11 video sequences in the KITTI data set with a total of 23201 tonnes, with the longest sequence length of up to 5 km, compared to the existing streaming method, DynStream's Sim (3) ATE was reduced from hundreds of meters to 40 meters.
Sim (3) alignment allows additional correction of an overall scale, while SE (3) alignment only corrects rotation and displacement, thus retaining the scale error of the model itself. Both errors are the sequence length weighted average, in units of m, the lower the better.
Introduction by the author
The first authors and communicators of the paper are from the VDIG (Visual Data Interpreting and Generation) laboratory at Peking University's Wang Sele Computer Institute, the first author is an undergraduate pioneer at Peking University, the communicator is a Ph.D. tutor, an intelligent scientist, Wang Yongjun is an associate researcher. In recent years, VDIG Laboratories have published a number of high-level results at top summits such as TPMAI, IJCV, ICCV, CVPR, NeurIPS, ICML, won multiple awards in the domestic and international competitions in the field of CVs, and have extensively collaborated with well-known research institutions and enterprises at home and abroad.
© THE