Players, developers, the times have changed.
With just one consumer-grade graphics card, AI-generated interactive games can run at 60 FPS and start to look really playable.
Let's go directly to the demonstration video.
Unless you're told in advance, it's hard to imagine that these video games are actually created in real time by AI. The video games were written by Coding Agent and rendered to video models in real time.
Based on Wan2.2 5B with a 480p resolution on a single RTX 5090, the game can run at 60 FPS, with a button-to-picture response time of less than 150ms.
In contrast, the StreamRender-H3 research team, which was just launched this week, uses a similar technical route. According to its public technical report, the StreamRender-H3 on 8 GB200 GPUs optimizes the single-step running time of stream-distributed rendering to 218ms, with interaction delays of about 500ms.
While the Invidia research team was still using 8 data center GPUs to explore real-time game generation, the Chinese team, after relentless research and development, had come up with a solution to run a single consumer-grade graphics card that surpassed Invidia under the same technical logic!
This major technological breakthrough was achieved by the latest release of the Matrix-Game 4.0, the world's first consumer-grade graphics card video game running in real-time and open-source simultaneously. This open-source game includes not only model weight, but also game code that developers can directly modify, extend, and re-create. This means that developers can build on an existing basis quickly, and that Matrix-Game 4.0 attracts more developers and provides greater imagination to cultivate an open-source ecosystem.
GIthub links to:
Hugging Face links to:
Harness in the video model: Triangle division
For the past two years, an "all-in-one" route has been taken with the interactive world model represented by the Genie series, with a video model simultaneously taking on the dual tasks of world modeling, state memory and image generation.
The difficulties along the way have been repeatedly verified. Although Genie 3 achieved real-time generation, the memory window stretches for about a minute, but the world state is only implicitly present in the generated pixel. Once the object leaves the field of view, there is nothing to "remember" it, and small errors accumulate in the process of self-return, and the scene gradually loses its authenticity within minutes.
For the first thirty seconds or so, the scene of the duststorm landscape looked very real, with spatial coherence, consistent lighting effects, realistic architectural styles, and details appropriate to that era. However, as I continued to browse the image, the quality of the image gradually decreased: the texture became softer, and the geometry also lost its authenticity.
In fact, many teams in the world model field have realized that video models alone are difficult to maintain a long-term stable world of interaction, and have begun to explore the technological paths to separate world states from images.
For example, concurrent work with the Matrix-Game 4.0 architecture, the Programmable World Model attempts to maintain the state of the world through a programmable engine, StatePlay uses video and state binary predictions, and MASS explores using a learning logic engine to maintain multi-shared world states.
Matrix-Game 4.0 innovated on top of this, not only in image-generating quality, but also in computational requirements and real-time response, but also in integrating the three decentralized capabilities of "state maintenance, real-time interaction, open mode", which is exactly where Matrix-Game 4.0 bridges the gap with contemporary work.
Table: Interactive World Model Routing Contrast. The Matrix-Game 4.0 is available for the first time on consumer-grade graphics cards, with model weights synchronized with customizable game code.
The Matrix-Game 4.0 team concluded that at this stage, the most important dissemination model they had learned was statistical relevance, which would make the resulting results look real, but it was difficult to really master causal relationships and physical laws. Since the video model alone could not remember the entire world, it set an external "harness" to it, which was made up of three modules.
The game engine serves as the "arbiter" of the world.
The ability of the game engine is the cornerstone of the whole world: the road is flat, the building is a box, the characters are skeletons. The rules of the game, such as blood volume, gravity, collision, and mission progression, are calculated directly by the engine. With each stroke, the engine produces a "structural sketch" that clarifies the location of the camera and objects, as well as what collision events occurred.
Interestingly, the "arbiter" itself can also be written by AI. In Matrix-Game 4.0, a lightweight game built by a programming intelligence (Coding Agent) using code, GPT-6 also participates in coding rules such as blood volume, collision, checkpoint, which are essentially a piece of code that can be modified.
This led to a very direct benefit: wanting to change the gameplay and allowing Coding Agent to change the code.
In the Shooting Demo, the team added a blood volume and equipment management mechanism to the game through Coding Agent: the shields on both sides of the enemy are durable and run out of time; the player's blood volume is zero and the game is over.
The shields are long gone.
Shooting after destroying enemy shields
Blood runs out and the game is over
The gameplay changes in the race car demo are even more interesting. The team first constructed a timetable with checkpoints and then transformed it into a vehicle chase task: the player needs to get close to the target vehicle and stay for five seconds before completing the interception.
Checkpoint timetable, crash penalty signals
During the chase mission
Tracking missions intercepted successfully
From timing racing to vehicle chasing, the rules of the game have changed, but the underlying video model does not need to be retrained. This also reflects the value of separating game logic from image generation: developers can adjust the gameplay more flexibly, while video models only need to be responsible for presenting the game world.
Coding Agent, the world's "writer"
It is responsible for transforming the player's creativity into a truly workable game, with two main tasks. One is to give the game scene a specific visual style. For example, a simple architectural model in the engine can be described as either "an old wooden house by the sea" or "a tech shop in a future city". These descriptions serve as a text condition, along with the structural images that the engine produces, to guide the video model to complete real-time rendering.
Video generating models as the "painters" of the world
Traditional game development requires a large amount of man-made artistic assets, as well as complex optical effects such as shadows, smoke, and liquids, which are now given to video models for real-time generation. However, the "painter" also has clear limits: the appearance details can play freely and maintain backward consistency; new objects that appear in the picture need to pass through intelligent body recognition and engine checks; as for core game data such as blood volume, backpack, the video model has no power to modify. It can decide what the world looks like, but it cannot automatically change the rules of how the world works.
We can intuitively see the difference between the structural image and the actual image produced by the game engine by comparing the images below.
A third claimed to have shot.
It's the first time that I've ever seen a race car.
In this regard, we can clearly see: Coding Agent impetus to create new scenarios, scenarios, conversations with NPCs, multiple people sharing the same world, is the design goal of the Matrix-Game route.
The game engine maintains a world with clear rules, the coding agent is responsible for understanding and expanding the world, and the video model continues to give it rich, realistic visual performance. The Matrix-Game 4.0 is the first step along this path, focusing on real-time and low latency.
Innovative technology links, stuffing everything into 5090
In order for this work to really work, there is a key premise: the speed of rendering of the video model must be fast enough.
Game engines only need to run lightweight simulations, maintain world status and game rules, and the pressure to generate high-quality images in real time is mainly on video models.
In order to fit all this into consumer-grade graphics cards, Matrix-Game 4.0 makes a whole set of targeted optimizations around the three nodes of data construction, model training and accelerated reasoning. From letting models learn to generate high-quality scenarios based on structural images, to reducing computation through step-by-step distillation, to compressing the logic performance of single graphics cards, each node directly affects the final real-time experience.
Data: Structural images of the UE5 - fine-grained images aligned
The team built structural images for the world-model based on the 5 (UE5) virtual engine - precise cross-domain alignment data sets covering supercities, open deserts, complex castles, high-dynamic aerial warfare scenes. The self-developed lightweight geometric abstraction capture pipeline, while keeping camera trajectory in strict time and space synchronization, combines high-complexity scenery abstraction with "structural representations" of spatial topology and rigidity, and is gradually aligned with fine-grained video image levels. The system also captures object instance IDs, visibility states and cause interactions, constructs dynamic metric data for photonics events, and establishes "high-dimensional to real geometry" mapping conditions for model generation.
Dynamic Caption: Let the Meaning Follow the Player
Matrix-Game 4.0 introduces the dynamic Caption mechanism, which links the scene object ID to the semantic description, extracts the current visible object and its state from the engine in real time, fills in a hint word template when running, so that the semantic conditions are synchronized with the game world. The image provides structural information such as location, layout, and Caption adds semantic information such as "what objects are in the image, what state they are in".
Controlled generation and spatial memory
The video model is based on Wan2.2 5B (Fun Control) conversion into video to video (V2V) mode, with engine rendering structural images with video to be generated by channel splicing (a total of 148 channel inputs), combined with dynamically synthesized scene event prompt, spatial memory reference graphics, and first column, respectively responsible for detail description, scene consistency and image coupling.
Accelerated reasoning: Drying every millisecond
The biggest challenge of Matrix-Game 4.0 is how to compress the video generation process, which used to require a lot of computation, to the extent that consumer-grade graphics cards can be processed in real time.
The four-bit generating mode means that the model calls a very high frequency and the GPU has a lot of time waiting for the CPU to launch computing tasks. The team made a series of targeted optimizations.
CUDA Graph, which uses DiT mainframe computing for pre-recording, one-time playback while running, significantly reduces the cost of CPU-GPU interaction
Integrated GPU Kernel, integrated with FP8 to write integrated special arithmetic for RoPE, with the same layer of Q/K/V as input quantity.
Ring-shaped KV caching, video history KV switching to ring-shaped caching, no longer moving historical data when windows slide
Text KV Reuse, text condition KV is only recalculated when Caption is updated
Lightweight decoding, replacing the decoder with TAEHV, further squeezing expenses
With these optimizations, the text encoder, 30-layer factor DiT, and VAE all reside on an RTX 5090. The single-step noise output is 83 FPS (about 57 FPS in two steps), displaying a peak storage of 18.5 GiB. When the game is running at 60 FPS, the end-to-end delay of clicking on the screen response is less than 150ms.
From slow-step distillation reducing computing, to CUDA Graph reducing scheduling costs, and to merging Kernel with cache optimization to improve execution efficiency, Matrix-Game 4.0 offers an entire optimization link to real-time video generation.
This is also the key to bringing video game models to consumer graphics cards. The technical content of Matrix-Game 4.0 is largely reflected in the synchronous efficiency of the complete system, which is difficult to reproduce.
Let the model of the world really land in the game
To make the world model truly "go into the game", it takes several turns at the same time.
Image quality is just one of them. Whether it can run on ordinary graphics cards, whether it can accurately execute the rules of the game, whether it allows developers to freely modify the game, these issues together determine how far the world created by AI can go.
Over the past two years, projects such as Google DeepMind's Genie, Decart's Oasis, and Alaya World have been continuously refreshing the world model's generation capabilities. Genie 3 has achieved real-time generation at 720p/24 FPS, with visual memory extending to about a minute, but the actual experience is still cloud-dependent and time-limited. Other models also find a balance between image quality, long-term generation and real-time.
How to cross these thresholds? Quang Wanvi is one of the longest-running teams on the track.
In May 2025, Matrix-Game 1.0 became the industry's first open-source 10B+ interactive world model with 17B+ parameters for the Minecraft scene.
In August 2025, the Matrix-Game 2.0 became open-source and was later used by the team of DiT authors Shaisen as the technical backbone of the world's first multiplayer video world model, Solaris.
In March 2026, Matrix-Game 3.0 began systematically solving long-term memory problems.
Released in July 2026 on WAIC, Matrix-Game 3.5 introduces Patch Memory, a single-card real-time generation of 720p/20 FPS on the NVIDIA H100 GPU. Its automated data pipeline has produced more than 5 million video clips covering more than 1,200 game scenarios.
With the release of Matrix-Game 4.0, the technology has improved significantly.
From 3.5's 720p/20 FPS to 4.0's 480p/60 FPS, behind is a proactive shift in technical routes. 3.5 focuses more on image quality and long-term spatial consistency, while 4.0 prioritizes solving problems of low latency, high torque and consumer hardware deployment.
4.0 was the team's first work along the "harness for video model kits" path: building an external Harness for the video model, allowing the game engine to maintain the state of the world, allowing the Coding Agent to write the game rules, while the video model focuses on real-time image generation.
This path coincided with a number of obstacles that the AI world model needed to cross to enter the game.
The first step is to break the boundaries of the ability of the video model. No longer allowing the model to organize everything. The state of the world is handed over to the engine for maintenance, the video model is only responsible for rendering simple geometry into fine images. The state is no longer only hidden in the generated pixels, the video model also has a continuous external correction basis, not necessarily confining the entire world to its own hidden memory.
Matrix-Game 4.0 deploys the text encoder, DiT, and VAE all on a single RTX 5090, achieving a 480p / 60 FPS game playback rate, with an end-to-end delay of less than 150ms when clicking on the screen. This means that AI-generated games with clear game rules can now run in real time on consumer high-definition cards, making an important step towards a local gaming experience for ordinary players.
The third phase, from roaming to truly playable. Walking freely in AI-generated scenarios is just the starting point for interaction. Real games also require clear, stable rules. In Matrix-Game 4.0, the rules are executed directly by the game engine and the player's actions produce clear results.
Matrix-Game 4.0 was written by Coding Agent, and the rules of the game are in code form. The game can be modified directly by modifying the code without having to re-train the video model for each new device.
More importantly, this open-source release includes not only model weight, but also game code that developers can modify, extend, and re-create. This allows potential users of Matrix-Game to expand further from global model researchers to game developers and UGC creators.
As a matter of fact, the ongoing open source development of Quantilever has already had a certain environmental impact. In addition to Solaris by the Shenzhen team, Light Interaction, developed jointly by NVIDIA and Zhejiang University, is also based on Matrix-Game 3.0, while Adobe's RELIC incorporates Matrix-Game as a benchmark.
For independent developers, especially OPC, this could lead to a new way of developing games: coding the game with Coding Agent, generating visual performances with video models, quickly creating a prototype game that can be handled personally, and then deciding whether to invest more resources.
In this regard, the meaning of Matrix-Game 4.0 has gone beyond the video-generation model itself. It is attempting to lower the threshold for AI games from ideas to playable prototypes, and this could also be an important pathway for the world model to its practical application.
The next link is for real players.
The idea of Matrix-Game 4.0 is well understood: let the different modules do their jobs. The engine is responsible for maintaining the state, the coding agent is responsible for planning, and the video model focuses on image generation.
In the field of AI world models, over the past two years, people have been exploring how to get one model to do everything. The Matrix-Game 4.0 offers another route: through collaboration at the system level, allowing different models to take advantage of each other.
The Matrix Game team's judgment was that "the scattered model is more like a renderer, which is good at drawing images realistically, not good at remembering the state of the world, keeping to the rules of the game. Let it do these things at the same time, the more mistakes are made. The programming intelligence has progressed rapidly over the years and the cost of using a lightweight engine has decreased significantly, so the engine controls the world, the video model controls the images, and this path is now well underway".
Of course, the program is far from mature and there are many engineering problems that need to be addressed, and there are gaps in the 480p resolution graphics standards of mainstream games, and the present presentation is limited to fixed locations and pre-set action spaces, but it represents AI-generated games that have crossed the threshold from "interactive video" to "hands-on play".
The chairman and CEO of Kwon-won-won, Wang Han, had previously ruled at WAIC 2026 that 2026 would be the "World Model Year".
If the technological path of Matrix-Game 4.0 is further verified, the next phase of the world model may not necessarily require an increasingly large video model, but rather a clear, collaborative and efficient system. As the program moves to open source, more developers will have the opportunity to participate in it, exploring new forms of play and interaction.
In the past, advances in world models have been largely defined by researchers: whether the images are realistic, whether the action is consistent, how long the generation time is; and when AI games become truly interactive, the rating standards will change dramatically.
In the future, AI-generated worlds may have visual performance close to that of 3A games; achieve consistency across player viewpoints, allowing multiple players to enter the same world from different locations; provide more fine-grained image controls, allowing developers to freely customize scenes, materials, shots, and character movements.
Measuring the standards of the world model will be not only the resolution, torque and generation time, but also whether it can support a world that is rich enough, stable enough and fun enough for players to want to come back again and again.
Technical demonstrations can be amazing for a few minutes, but a really fun game requires willingness to spend tens or even hundreds of thousands of hours.
The good news is that after further optimizing operational efficiency and lowering the hardware thresholds, Qualcomm plans to launch a technical preview version via Steam, allowing more players to experience it themselves.
Matrix-Game 4.0 has already demonstrated the technological possibilities. As far as this path can ultimately go, the next link may be given to real players.
© THE