What coordinates should the video model and the world model work in?
Existing video DiT is usually organized in (u, v, t) voltage grid locations. It can mark which token is located, which line, which column, but does not encode an observation of where it is coming from, where it is going from around the world. When the camera moves or repeated textures appear, the correspondence between the grid address and the scene location quickly becomes confused.
The University of Hong Kong, the Hong Kong University of Science and Technology and the Tencent ARC Lab have jointly developed SCoPE, a set of lightline-coordinate positioning encodings for video DiT: allowing video models to move the base of positioning relationships from the voltage grid to the radius space.
Video 1: SCoPE interactive camera control results generated from a single input chart. Below the left is the WASD instruction, and below the right is the corresponding camera trajectory.
Check out the results generated by SCoPE. Only one static image and continuous WASD instructions are entered, and the model can move forward, backward and horizontal according to the instructions, and the combined action is the same. The entire process is without a game engine, artificial modeling or three-dimensional reconstruction. SCoPE calculates the corresponding observation radius for each video token based directly on camera movement, and radial coordinates are inserted into the video DiT. The overall new parameters are less than 0.1% of the original video model, and camera control, cross-visual consistency and revisit consistency are improved.
It was written by Minghao Yin, Jiahao Lu, Wenbo Hu, Wang Zhao, Ying Shan, Kai Han.
Institutions: the University of Hong Kong, the Hong Kong University of Technology and the Tencent ARC Lab
The paper:
The main page of the project:https://visual-ai.github.io/scope/
The code is:https://github.com/TencentARC/SCoPE
The model:https://huggingface.co/TencentARC/SCoPE
What kind of coordinates do video world models need?
Video DiT typically uses u, v, t to indicate location. It records which token is located, which line, which row, but without the geometry of the camera and the scene.
The ideal coordinates are, of course, surface three-dimensional coordinates: each token depicts which point in the world and which point is used as the address.
The rays provide a way out. Once the trajectory is given, the corresponding observation rays of each token can be calculated in advance. A single ray does not determine the depth, but the relationship between the two rays can help establish a three-dimensional correspondence: when the content is consistent, the rays intersect, the two tokens may point to the same three-dimensional point.
This is in line with the way attention works. Attention is actually a comparison token, and SCoPE allows it to compare content while also comparing the geometry of the rays. The three-dimensional point coordinates, which were previously not available in advance, are created by "matching content + intersecting rays" implicitly in the production process.
SCoPE inserted the camera's radiation coordinates into Video DiT, allowing the model to see the relationship between the content and the view while comparing the token. The core of its discussion became clear: the position coordinates of the video world model can go from a pixel grid to describing the radiation space of observing the world.
Figure 1: SCoPE introduces a camera beam for each video token. Faced with the same candidate area of appearance, the content characteristic is responsible for finding possible matches, and the beam interaction further determines which candidate points to the same three-dimensional point with the previous beam. After the beam enters the query/key, the content matching and geometrical judgement can be done in the same attention.
Coordinates to the attention
To participate in every relationship judgement
The exact method is not complicated. Once the camera's trajectory is given, SCoPE calculates a Plücker radius for each video token. The radius is projected, integrated and controlled by the Normalize-Gate-Inject scale, and then enters the attention query and key.
In this way, attention will simultaneously calculate whether the two tokens are similar in content, whether the two rays intersect. For example, in the previous window, the content characteristic will identify two windows that are similar in appearance; the ray geometry will further determine which window's rays and the rays of the previous one point to the same three-dimensional point.
The whole process takes place directly in the original self-attention, with no need to add another set of attention. The new parameters are less than 0.1% of the original model. The real change is the matching basis between the tokens: the model, in addition to comparing "looking like nothing" and judging "is not able to correspond geometrically", is based on this matching system.
Figure 2: The camera parameters are broken down into token-level Plücker radii, and through projection, RMSNorm and the scale gate control enter the query/key, observing geometry and directly participating in the attention score.
The same coordinate relationship can be used for interactive directions and preset trajectories. The following six results are generated from a single graph: upstream receives W、A、S、D directions, downstream advances along the preset trajectory.
Video 2: The top row shows interactive WASD camera controls and the bottom row shows the results generated along a given camera trajectory.
Can coordinate relationships transcend the scale of painting and model
The beam is determined by the camera, and does not depend on the input of photography, painting or the style of conceptual design. In five sets of hand-drawn conceptual diagrams, it produces visual differences and compositional changes consistent with the target trajectory. The beam changes the visual characteristic distribution, and the camera center, direction of observation, and the beam relationship still use the same geometric description. Models can continue to invoke learned coordinate relationships in new content distribution.
Figure 3: Generalization to stylized inputs. Each line is a hand-drawn concept map and its generating sequence, with the right side representing the corresponding target camera trajectory.
The dilution experiment verified the core design of SCoPE. We analyzed where the gain came from, removing the contents. The rotation error after the cross-beam is 0.135, and after the pure-beam is removed, it is 0.159.
Another group of experiments examined whether SCoPE could expand with the scale of the model. After growing from 5B to 14B, SCoPE expanded the lead width on the horizontal error from 12% to 25%, and FVD expanded from 23% to 47%. The larger the model, the more obvious the advantages of SCoPE, showing that the beam coordinates are not a local fix for the small model; the increased skeletal capacity allows the model to more fully exploit its geometric relationships.
Figure 4: Camera control, generating quality, critical meltdown and model scale trends. Figure 5B compares RotErr to FVD on the left; figure 5B shows the contribution of cross-sections and pure-ray segments to RotErr on the right; figure 5B to 14B shows the relative lead width on the right.
Revisit: Sharing the same set of rays on the way back and forth
Revisit directly reflects the nature of the radiation coordinates. When the camera leaves the starting point and returns along the closing loop, the end and the start, although separated by many holes, pass through the same set of radiations again. In the (u, v, t) grid coordinates, the return path has a completely new address; while in the radiation space, the same observation position still corresponds to the same radiation address, so that the places seen do not change their location and identity.
Therefore, SCoPE does not need to design an additional module to "remember" the starting point. The incoming and returning radiation coordinates are shared, and attention can be directly reconstructed between the two.
Closed-loop testing confirmed this. SCoPE's LPIPS Recovery was upgraded from 0.102 to 0.587 in the strongest contrast method. When the camera returned to its original observation position, the image was closer to the scene seen at the start.
Video 3: The upper part of the building is a tea room and the lower part is a wave room. The camera leaves the closed-circuit trajectory and returns to the initial radius, where the scene recovers with the observation of the coordinates.
Making radiation space the basic coordinate system of the model of the world
(u, v, t) Record tokens are placed in the space. The world model also needs to judge whether the tokens in different viewpoints come from the same world point. SCoPE uses the camera radius as a coordinate that is available before it is generated, and establishes a three-dimensional match in attention by "content consistent + radius intersection".
Camera control and Revisit show the results of this set of coordinates: when the camera moves, the model is able to maintain the same scene; when the camera returns to its original position, the same set of rays regain the same position identity.
The meaning of SCoPE is therefore more than just a set of better camera controls. It suggests that the world model can organize tokens in the spectral space of observing the world rather than relying solely on a pixel grid that ranks video data. For the next step in generating a navigable, interactive world, the spectral space provides a coordinate system that truly belongs to the world model.
