The first author of the project was a Ph.D. student at the Southern Ocean University of Technology, a visiting student at the University of Western Hemisphere. The instructor was Zhang Zhang, assistant professor, head of the General Artificial Intelligence Laboratory at the University of Western Hemisphere.
Creating a realistic picture is not the same as keeping a world running. A truly open world is not only about answering "what's next", but also about remembering what happened off-screen, how the rules work, and how an event can continue to change the characters and the environment long afterwards.
In recent years, video world models have rapidly improved in graphic quality, motion control and interaction, and they have been able to convert keyboard, mouse, motion, or text instructions into continuous visual images. However, the core modeling objects of existing systems are often "what observations should be generated next". Visual history can record visible results, but it is difficult to directly preserve world rules, character relationships, event history, and the continuing cause chain of development beyond the screen.
Imagine that in the game, if players stabbed the rulers of a city and left, the beliefs and objectives of the city's order, inheritance relations, trade, camp coalitions, and a large number of non-player characters would continue to change. When players return long afterwards, the system would not only generate a "seemingly reasonable" image of the city, but also what happened during the player's departure, why it happened, and how these consequences would extend.
To this end, the team at West Lake University's AGI Lab proposed the Code World Model, which no longer requires video models to push back all hidden mechanisms from the visual results alone, but instead breaks down "how the world evolves" and "how the world is seen" into two complementary questions: The coding agent is responsible for determining how the world evolves, the code is responsible for keeping the rules running, and the video model is responsible for transforming the evolutionary state into a high-fidelity visual observation.
Figure 1 Code World Model Overview:Coding Agent maintains rules and status, Proxy provides time and space constraints, and video models are responsible for high fidelity vision implementation.
Code World Model: Coding Agent as World Brain
It was written by Yiwen Chen, Guosheng Lin, Chi Zhang.
This article is part of a two-part series.
The main page of the project:https://buaacyw.github.io/cwm/
GitHub：
Why studying video is not enough to support an open world
Video recording is the visible result of the evolution of the world without directly exposing the knowledge, rules and mechanisms that produce the results. This is especially true in gameplay: each game is rendered after execution by a fixed code, but once it becomes a training video, the code that really drives the world is lost, and the model can only reverse the hidden logic of collisions, attack range, camp relations, mission status, and so on from the scarcity of visual consequences.
This information gap is more pronounced on a long-term scale. The context of current video models is usually less than a minute, and the role relationships, social structures, and consequences of events may continue to evolve over days or even years in the world's time. Many key changes occur outside the camera.
The Code World Model judges that an open world requires two different capabilities: one is low-frequency but complex reasoning -- understanding events, knowing the world, planning consequences, modifying mechanisms; the other is high-frequency and repeated execution -- updating locations, numbers, schedules, cooling, collisions, and rules -- putting them all into the same video model, which is both inefficient and difficult to guarantee.
The core idea:
"What's going on?"
Video modeling "What does it look like?"
Figure 2 Method Process: Interacting intent into the Coding Agent, the Agent updates the World State with code; the state of the world is given to the video model via conditional interface, generating the visual world and forming a feedback loop.
In this framework, the Coding Agent is the "mind of the world". It reads the current state of the world, explains new interactions or events, decides which entities and mechanisms need to be changed, and chooses to call an existing code or a local rewriting world program. The code is an executable extension like the Agent, continuously completing intensive, identifiable, repeatable state updates without having to call the big model again every step.
This means that the code is not a game logic that developers write in advance; it is not an external controller that is separated from the Agent. The agent can combine, call or modify the code according to new circumstances, changing not only the "current state" but also how the world will run in the future.
The paper further breaks down the entire state of the world into two parts: executable state preservation procedures, entity attributes, rules, relationship to event history; visual state preservation is generated by video models and requires consistent appearance and motion information over time. The two parts are updated by different mechanisms, but are combined: the code ensures that rules and consequences persist, and the video model uses the appearance, motion and interaction precedents learned from large-scale visual data to realize the state of the world as a high-quality observation.
Proxy：
Adding a programmable "visual channel" to the state of the world
After determining the state of the world, a new problem arose: how to accurately assign the changing state to the video model? using only structured text is the simplest, but the text is difficult in a low-latency way, describing character positions, shadowing relationships, camera trajectories, and physical movements; allowing Coding Agent to directly build a complete 3D world that requires high-quality assets, animation, materials, lighting, and rendering systems.
The team therefore proposed Proxy: a roughness visual condition compiled by world state certainty. Proxy does not seek to make a low-distribution image, but retains only the minimum states that current observation must comply with, such as physical position, approximate size, gesture, motion trajectory, spatial relationships, shadows and camera movements. Simple human figures, wireframed vehicles, low polygonal vegetation or enclosure boxes can be used as basic components.
Structured text and Proxy are clear: text is responsible for identity, appearance, movement, semantics, and style, and Proxy is responsible for space and time constraints for each block. All control signals are traced back to the world state of the Coding Agent and code maintenance, so this "state-of-the-art" pathway is still a verifiable, address-seeking, locally modifiable white box system.
The key to a proxy is not to be more precise, but to find a balance between "constructibility" and "binding intensity". Too rich a proxy will require a coding agent to maintain a large amount of linkage and detail trajectory at the same time; too sparse may not be enough to bind a video model.
How did the data come about:
Compile the game to the same interface as the actual video when it's running
Figure 3 Examples of game data aligning with real-world RGB Proxy data. The adjacent RGB and Proxy are from the same moment, maintaining the physical location, scene layout, camera movement and shadowing relationship.
To make the video model understand Proxy, the training data must contain both rigorously coordinated Proxy video and RGB targeted video. The game is naturally suited for this task: the research team synchronizes images while running GTA V, and builds the required camera, entity, scene, and interaction state of the Proxy. The code can be compiled repeatedly from the same runtime record of different coverage, different particle sizes of information Proxy, without having to retrieve the RGB video.
This set of interfaces also left an input for real video. The paper carried out geometrically-assisted conceptual verification on KITTI-360: calibrated camera posture, semantic 3D reconstruction and object markup for offline compilation Proxy only, and the video model ultimately receives RGB targets, Proxy video and structured text. For the model, the results of camera movements and movements are already reflected in Proxy and no longer require additional independent motion labels.
Prototyping was completed with about 5.6 hours of video game.
The current prototype is adapted on the MiniMax-H3 Ref2VA video model. Training data comes from 157 game processes, a total of about 5.6 hours; the team sampled 9,420 five-second training segments in two-second intervals. Each RGB objective contains 124 frames, a resolution of 1344×768, a torque rate of 24 FPS; the matched Proxy is 336×192, and combines fixed-dimensional depths with semantic ID graphics.
The training uses rank-128 LoRA, covering 50 Transformer blocks, about 596 million trainable parameters, and three epochs on eight NVIDIA H800s. The team uses GPT-5.6 Sol as a coding agent; it builds on existing game engine code, basic player control, collision and update cycles, combines, expands and rewrites templates to build the target world.
Figure 4: Qualitative results: Proxy sequences in each group, followed by RGB observations generated by video models. Examples cover character movements, flight/lens trajectories and multi-entity spatial relationships.
The paper's qualitative results show that after small-scale LoRA adaptations, video models can follow the role positions, action trajectories, scene layouts, and camera movements specified by Proxy while maintaining rich appearance and local dynamics. The project page also provides a complete timeline comparison with the motion conditions, camera conditions video world model, to show the differences in the particle-by-particle control of Proxy.
From "generate as instructed" to "change the world in action".
One of the most intuitive presentations on the project page is a continuous update of Coding Agent's state of the world and its operating mechanisms: characters can summon flying rides at the seabed, perform dance moves at the Magic Academy, switch rides and moves in the Cookie Village, and trigger new character interactions in the Snow White Village. The material and style of the images are realized by video models, but the number of characters, proximity, motion and critical spatial relationships are still limited by Proxy.
It's been a long time coming.
Visual styles can be switched and the state of the world continues to move forward
In order to generate longer sequences, the current system uses 124 overlapping windows: adjacent windows overlapping 34 windows, pushing 90 windows forward each time; the RGB windows at the end of the first windows are responsible for local continuity, while the same windows' appearance focal points help maintain the overall identity and appearance.
Figure 5 Proxy/RGB contrast shots of video 051 generated over a long period of time. After a two-minute sequence of visual styles switching, Proxy continues to provide physical and spatial constraints. The shots are taken directly from the project page video material.
What has this work really changed?
The value of the Code World Model lies not only in adding a video control condition, but in changing the division of responsibilities within the world model. Traditional video world models tend to compress states, rules, memory, and visual predictions into generating sequences; here the executable world state refers to the system center, where the coding agent and code are responsible for causative evolution, and selectively compile parts that currently need to be "seen" as conditions.
This division of labor allows higher-level reasoning and lower-level execution: Agents don't have to work at video speeds, code doesn't have to have open common sense reasoning; video models don't have to learn complete rules from scratch, they just have to implement a clear state of the world as a visual observation. Even if future language and video capabilities are integrated into the same multi-modal network, a continuous state of the world maintained by external code will still require an efficient, controllable input interface, and the "state-of-the-art" problem discussed by Proxy remains.
At the application level, this paradigm refers not only to game generation. An open world that can maintain rules, remember screen changes and generate highly accurate observations, but can also be an environment for training and evaluating intelligent agents for tasks that require understanding, predicting and acting on dynamic environments, such as physical intelligence, autonomous driving and long-term planning.
What is the meaning of this verse?
The focus of the world model may be more than just "the next wave".
If the video-generation model is good at imagining "what the world looks like", the coding agent and code are better suited to answering "why the world is going to be this way and how it should continue to change". Code World Model tries to put these two capabilities in one closed loop: to let knowledge and rules drive states, to let states enter the visual space with Proxy, and to let the video model realize the results into rich, open observations.
This work is still in its early stages, but it makes a noteworthy judgement: the path to an open world model will not only continue to expand the video prediction model, but it may also need to bring executable code back to the core of the evolution of the world. The world is no longer just a continuously generated image, but an operating system that can be understood, modified, executed, and continually left with consequences.