It's the first time in the history of the world.
In recent years, video generation models have made rapid progress in terms of clarity, timing consistency and controllability. Given text, images, camera trajectories or player movements, models have been able to synthesize coherent follow-up images, allowing "interactive video worlds" to gradually move from concepts to observable, operable prototypes.
However, it is not the same as running a world. In a more complex environment, characters have goals, identities and relationships, and events are constrained by rules and have long-lasting causal consequences. Many states even occur outside the camera, but still change the plot and action. It is difficult to clearly preserve these higher meanings with local pixel prediction alone.
In a complex world, the first thing you need to do is "understand why".Goals, rules, memories, and chains of cause and effect are often not read directly from the current one, but determine how the world should change next. Language models have the knowledge to call, reason, plan, and program, to deal with these low frequency but high complexity decisions, and to further implement them into executable rules.
Another often overlooked fact is that game video is not the world itself, but a pixel projection of the results of program execution rendered.At this stage, games and simulators are still the main interactive data source for video world models. Keeping the moves and pixels and allowing the video model to learn the mapping between the two directly is equivalent to bypassing the existing programs, rules, and explicit world states, instead requiring the model to reverse from the visual results to fit the entire program behavior. Such learning paths are not only ineffective, but also combine "How the world works" and "How the world is presented" into the same model.
Based on this observation, a team of researchers from the AGI Lab and the South Lake University of Technology has proposed the Code World Model, a new type of world model model that uses language models as a "brain". Its core idea is to allow coding agents to maintain and update executable world states by continuously writing, calling, and modifying code; and to convert the current observed state into a proxy to guide the video model to generate precise visual images.
Link to the paper:https://arxiv.org/abs/2608.25927
The main page of the project:https://buaacyw.github.io/cwm/
The codeWarehouse：https://github.com/buaacyw/code-world-model
The language model here does not replace the video model. The Code World Model redistributes the duties of the two: knowledge, goals, rules and long-term causes by the coding agent and code, appearance, motion details, lighting and pattern reasoning by the video model. In other words, the former determines what happens in the world and the latter determines how it all looks.
Figure 1 Code World Model: The coding agent evolves the world state by code, the proxy converts the relevant state into a visual condition, and the video model generates the final image.
Method summary
The Code World Model divides the world's operations into three interconnected parts.
coding agent：Understanding player intent and new events, using world knowledge, reasoning about potential consequences, and deciding which mechanisms need to be implemented, combined or modified.
code：With higher-frequency execution of positional updates such as location, properties, collisions, cooling, and event triggers, high-level decisions become rules for verifiable, reusable, and sustainable operation.
video model：Read post-evolutionary state conditions and use the appearance, movement and interaction precedents learned from large-scale visual data to generate visual observations with rich texture, light and local dynamics.
This division of labor also corresponds to different computational frequencies. The coding agent only makes sparse decisions when new targets, complex events, or existing mechanisms are inadequate; once the decision is written in code, the program sustainably executes intensive state updates.
More importantly, coding agents rewrite not only the numerical values of a moment, but also the way the world works afterwards.
Complete concepts include visual feedback; the current prototype focuses on verifying the forward links of the coding agent / code → world state → proxy → video model.
Proxy connectivity executable status and video generation
Once the task breakdown is complete, a critical interface problem remains: the coding agent and the code maintain executable status, while the video model receives text, images or video tokens.
Structured text is flexible enough to describe the physical location, relative relationships, motion trajectories, and precise camera changes of each piece; the complete construction and rendering of the 3D world will reintroduce the high cost of assets, geometry, materials, animation, and rendering.
For this purpose, the project introduced proxy. It extracted information from the world state that the current observation really needs to follow and organized it into a rough visual representation.
The text is responsible for "who, what, what to do", and the proxy is responsible for "where, how to move, how to shoot".
The proxy can express the camera with perspective, physical location and orientation, scale and trajectory, scene layout and shading, and the roughness state required for current interaction; texture, material, fine lighting, and complete local movement are intended to be left to the video model. The article summarizes this design principle as: maintaining only the minimum full state required for current observation, and achieving a balance between control capability and state coding costs.
Currently implemented, the proxy is only a quarter of the target video at both width and height, and the pixel size is about 1/16 of the target. It consists of a combination of simple, replicable geometric primitive, which does not require production-quality assets, but provides a step-by-step, editable time and space framework.
Strictly coordinated data from the game's running time record
In order for video models to understand proxies, training samples need to contain proxy video, structured text and targeted RGB video at the same time, and proxies must be strictly aligned with RGB in terms of time, camera and physical identity.
The game's runtime records naturally retain this correlation. The research team records the RGB, camera status, entity identity, location and direction, approximate scale, scene layout and interaction status from the same gameplay execution in sync, and then compiles the proxy via code offline.
In Figure 3a, the target in the game data is RGB (above) with the same proxy (below).
In Figure 3b, real-life video targets RGB (above) and proxies constructed using camera markup, 3D reconstruction and object markup offline (below).
The gameplay data used in the paper contained 157 recordings, a total of about 5.6 hours of source video. The research team sampled 9,420 5-second training segments at 2 seconds intervals; each RGB objective contained 124 frames, with a resolution of 1344×768; a torque rate of 24 FPS; the corresponding proxy also contained 124 frames, with a resolution of 336×192; and combined with fixed-log depth and categorical semantic-ID maps.
The same running time record can also be re-compiled into proxies with different coverage and information grain, without having to re-record RGB.
The paper also demonstrates the offline build process of a real-world video proxy-observation pair in KITTI-360 to illustrate the possibility of this interface expanding to real data; the current video model is still adapted using only matched gameplay data.
How does the prototype system work?
The current prototype uses the MiniMax-H3's Ref2VA backbone as a video model to adapt the rank-128 LoRA for all 50 transformer blocks, with training parameters of about 596 million.
The reasoning phase is provided by GPT-5.6 Sol as a coding agent. The system provides basic player control, collision processing, runtime update cycles, existing game-engine scene and gameplay logic templates, and the proxy primitive used in the training phase. The coding agent can combine, extend, and rewrite these codes to create a simple, executable, player-controlled world; its rough 3D geometry is only used to express the proxy, not directly as the final image.
For each 5-second clip, GPT Image 2 generates an appearance anchor based on the lead proxy and text; MiniMax-H3 combines the lead, full proxy sequence and text to generate 124 frames, 1344×768, 24 FPS of RGB video.
The longer video is generated by sliding windows with overlapping 34 frames: the latter window inherits the RGB context at the end of the previous window, and repeats the same appearance anchor to maintain local continuity and holistic identity appearance.
The results of the experiment
The qualitative results show that even with only about 5.6 hours of gameplay source video for LoRA adaptation, the model can still follow proxy-specified character positions, action trajectories, scene layout and camera movements while adding rich texture, lighting and local dynamics when the character, environment and style deviate significantly from the training appearance.
In Figure 4, the proxy provides the location of the characters and the trajectory of the actions, and the video model presents the same roughness skeleton as different identities and styles of characters.
Figure 5 Simple geometry Primitive constraints on complex objects and camera movements.
For example, the same coarse human primitive can be presented as completely different roles of identity and artistic style; a wireframe or box can restrict the position of vehicles, ships and other complex objects to rough movement; the same proxy interface can also express running, dancing, swimming, falling, and surrounding the camera.
From this it can be seen that the proxy fixed is the time-space skeleton that the current observation must follow, rather than the assets and style of the training game.
The project's homepage also gives a video contrast to the action- or camera-conditioned video world model.
The comparison focuses on controlling particle density and visual results: the proxy directly provides step-by-step physical movement and perspective changes, thus imposing finer particle density, more direct time-space constraints; the paper does not describe this group of results as inference latency comparisons.
Netizens commented, "Will Fantastic Engine 6 be the last engine to create games using traditional 3D methods? Maybe Fantastic Engine 7 will use a similar technology. Only Fantastic Engine CEO Tim knows the answer".
Then Tim came round and said, "I don't know!"
First, decide what happens to the world and re-create what it looks like.The previous video world model mainly studied "how to continue observing". The Code World Model takes the question one step further: what is the world present, what rules to follow, and why the consequences of an event will last before the next observation is generated.
A truly open world of production requires both the visual imagination of video models and the ability to maintain knowledge, goals, rules, and the mechanisms of execution of the relationship between cause and effect.The core of the Code World Model can be summed up in one sentence: let the code determine what happens in the world, let the video model determine what it looks like.