Going into 2026, interactive world generation faces a more difficult challenge: can AI-generated worlds be repeatedly explored by users? A door is realistic, it's just a starting point. Going forward, it naturally comes close; crossing the corridor and back, it should remain in its original location. Every time users move, turn and flip, they test whether the model is able to remember the generated world.
The competition for world models extends from image quality to reliability in continuous interaction. The lens must obey instructions, the scene must be consistent, and the speed of production must be operational. These capabilities need to be established simultaneously, and any drop in the chain can disrupt exploration.
In recent years, technological advances have made this main line clearer. Google's Genie 3 from DeepMind demonstrates real-time exploration and minute-level consistency, SANA-WM combines minute-level generation, precise camera control with greater efficiency, and Atlas from World Labs supports the creation of new viewpoints along specified camera trajectories in a spatial context, maintaining the geometry of the scene.
When further exploration touches on the visual and audio experience, new questions arise: can footsteps, ambient sounds, and character conversations be continuous with images when users change routes or change viewing angles? Therefore, the combination of visual and audio generation and interactive control is of greater concern for the immersive content-oriented world model.
On 9 September, at the JDD Global Technology Explorers Conference in Jidong, the Jidong Exploration Institute released the latest developments in the JoyAI-Echo series: upgrading the supersonic video generation model to JoyAI-Echo 1.5; simultaneously releasing and open-source JoyAI-EchoWM, which is geared towards the world of interactive video and audio.
The model builds on JoyAI-Echo's previous native audio video generation capabilities to generate video as well as surround sound, music and voice in the same frameworks, while responding to real-time user operations.
Now, with a first-person perspective, users can move and explore directly in the scene; switching to a third-person perspective, camera movements and subject movements can also stay in sync. After many rounds, long hours of operation, the overall picture, spatial relationships, subject state can continue well. JoyAI-EchoWM creates a truly accessible, controllable, and explorable video-audio integrated world.
In 158 WBench Navigation case assessments reported in related papers, JoyAI-EchoWM scored the highest average score of 81.7 and was ranked first on indicators such as consistency (89.8) and interactivity (87.2). At the same time, the four-factor streamlined version of EchoWM-Flash has an average score of 81.0 and an interactivity score of 87.9, indicating that the model still has a stronger navigational responsiveness after reducing sample steps.
Compared to other models in the same rating, Genie 3 emphasizes the consistency of real-time exploration with minute-level worlds; LingBot-World 2.0 promotes long-range interaction with slow-step distillation through factor generation, and combines camera posture and text instructions to control perspective, action, and events; HappyOyster combines audio video co-generation with real-time interaction to provide two modes of directorial and first-person roaming.
How this technical route is reflected in the interaction can be seen in the changes in the lens, image and sound after the user presses the operating key.
Press the operating key,
The AI world responds to you with images and sounds.
Normal video generation and post-production tasks are essentially complete. Even if the subject's location is slightly drifting, spatial relationships occasionally change, as long as the overall image is natural, the human eye may not notice it. Many of the previous small errors that can be overshadowed by visual effects can become bugs in the interactive state once users are allowed to move freely and switch views.
This is because interaction adds a layer of requirements to video generation: the model needs to generate images that are connected to the previous content based on the user's actions.
JoyAI-EchoWM solves these problems by sharing a set of interactive interfaces that allow subjects, spatial relationships, and sounds in the image to continue as users explore. The team calls this a fully modular world model that supports continuous navigation and synchronous visual and audio feedback towards "accessible media generation".
ArXiv paper:
The main page of the project:https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/wm/
GitHub code:https://github.com/jd-opensource/JoyAI-Echo
In order to achieve this interaction, it is necessary to unify the control method under different perspectives. The first person says that under the perspective, the lens approaches the observer's eye, the moving lens corresponds to its own movement. The third person says that there is more than one relationship: the lens must follow the subjects, such as the person, the vehicle, and the subject movement must also be accompanied by the image composition. At the same time, the WASD direction keys specified by the user are discrete instructions, the camera position and direction of the model to be processed are continuously changing, and the design of the interfaces for different viewing angles and control devices respectively will increase the system's adaptive load.
For this purpose, JoyAI-EchoWM uses a unified "camera intent" representation to describe how the user wants the camera to move and from which perspective to observe the scene. Keyboard operations are mapped and converted into a continuous 6-DoF trajectory of relatively initial position, i.e. motion and rotation of the camera in 3D space.
The same trajectory takes on different roles from different points of view. The first describes the movement of the observer directly; the third describes the changes in the camera, the relationship between the movements of the person and the tracking of the camera. The model learns from the data.
The first describes the control mode as covering many scenarios, including outdoor, indoor, underwater and fantasy. The user can move in a designated direction, allowing the immediate environment to unfold gradually. For example, in the demonstration of Ice Lake exploration, the first describes the perspective as moving forward along the ice surface, the remote wooden doors gradually approaching, snow mountains, forests and lakes continuously unfolding, and the surrounding sound is also generated simultaneously with the image.
The third person's perspective demonstrates the combination of the footage with the subject. In the footage below, the rider moves forward, the footage follows from behind, and moves to the left and right.
In the process of exploration, sound is also involved in the production. JoyAI-EchoWM uses native audio video co-generation, and the video and audio branches exchange information during the production process to make the sound interconnected with scenes and events in the image. The camera trajectory is only directly entered into the video branch, with no independent trajectory conditions on the audio side, and no additional training of a set of "motion controllers".
Simply put, the operation changes the picture first, and the sound synchronizes with the changes in the picture. When the subject speaks, events such as voices, winds, footsteps and collisions are accompanied by corresponding sounds, and the music continues with the scene.
When the exploration enters the next round, the model also has to take on what has already happened. JoyAI-EchoWM will extract a synchronized audio video at the end of the previous round as the context generated for the next round. Users continue to explore the trajectory of the camera in the new round. Subsequent images and sound share the same recent history, providing the basis for the continuity of the scene layout, appearance and sound of the subject.
This mechanism helps to maintain continuous experience, but there are still boundaries. The model currently has no apparent lasting three-dimensional memory, and geometric relationships, subject identity and audio may still drift after multiple iterations are generated.
In order for images, sounds, and user operations to work together in the same generation process, several problems in data and training need to be solved.
JoyAI-EchoWM is a threefold challenge:
It's good, it's good, it's good, it's good, it's good.
High-quality audio video doesn't necessarily have accurate camera trajectories, and reliable trajectory data doesn't necessarily have abundant sound. Once in continuous interaction, models also have to cope with the accumulation of errors and growing computational costs. JoyAI-EchoWM starts with data engines and gradually connects these capabilities through uniform trajectory scales and phased training.
The first is data. The team used four complementary sources to build a global data engine that provides more reliable oversight of different data types.
Internally collected gameplay allows simultaneous access to operating logs, native game audio, and clearer action.
Human real-life video on the Internet, with a more natural pace of operation, and a richer voice for gameplay, player interpretation, and more.
UE(Unreal Engine) imitates the most clean geometrical monitoring, directly providing rice simulation position and motion logs, but without providing natural audio;
Regular Internet videos complement real-world appearances, surround sound, voices, and richer footage and subjective relationships.
After collecting the data, the team reorganized three sets of data comparisons based on training objectives: AV-rich focused images, ambient sound and voice quality, Control-clean prioritized trajectory preservation, motion-clear samples, and Balanced high-quality data with both reliable control signals and higher visual and audio quality.
World data engine
Different sources of data also lead to a problem of scale. Different video cameras differ greatly in width of motion, and if each trajectory is unified separately according to its own range of motion, large movements and small movements can be compressed to nearby numerical ranges.
JoyAI-EchoWM establishes a unified "scale" for training trajectories: first calculate the maximum horizontal width of each trajectory and then take the 90th percentile of these values as a global scale. Trajectories with a maximum horizontal width not exceeding this threshold are retained and unified using the same scale; rotation remains unchanged.
With complementary data and a uniform scale, the next question is how to arrange training. Audio video quality and trajectory control depend on different monitoring signals, and if both are reinforced at the same time, the training objectives may interfere with each other. JoyAI-EchoWM first learns in three stages and then adapts together.
The first stage is audio video continuous pre-training (AV-CPT), using AV-rich data to train the existing audio video backbone to adapt to the visual dynamics, ambient sound and speech in interactive scenes. The second stage is action-SFT, freezing the audio video backbone, training only lightweight branches of camera trajectory, and using Control-clean data reinforcement trajectory responses. The third stage is control monitoring, which is still a camera trajectory.
After completing the above three stages, the model must also adapt to long-term interaction. During continuous generation, the first output becomes the second input. Early deviations may be brought into the subsequent content, affecting the scene, subject, and sound.
The team first used the audio video Teacher Forcing to convert the two-way model into a causal block generation model. The model learning on the basis of the previous audio video generated the next piece of synchronous content in context to build the basis for the flow output. Subsequently, the short-term Self-Gradient Forcing (SGF) allowed the model to continue to generate history based on its own generation.
In addition, retaining historical information also requires controlling costs. JoyAI-EchoWM uses a sink-plus-FIFO mechanism to retain the context of an earlier part and scroll to maintain the most recent history. Older local caches are moved in advanced order, limiting the size of KV caches.
JoyAI-EchoWM training process map
This method can be used to determine the degree to which the quality of production, control accuracy and long-term stability can be combined and can be tested by systematic evaluation.
From track control to long-term consistency,
How does JoyAI-EchoWM perform?
In addition to the aforementioned WBench Navigation, the team further explored the trajectory of JoyAI-EchoWM through SANA-WM-Bench and user studies, long-term generation and subjective experience.
SANA-WM-Bench divides the test into simple trajectories and difficult trajectories, each group containing 80 scenarios. Evaluation focuses on image quality and camera control, statistical VBench visual quality scores, and rotation, displacement and camera motion consistency errors respectively.
In the first 241 mm short-range test, the JoyAI-EchoWM scored 83.91 and 83.96 on VBench's two trajectories, respectively, the highest in the test model. On the simple trajectory, its horizontal error and camera motion consistency error were also lowest. On the difficult trajectory, the SANA-WM's camera control error was lower and the JoyAI-EchoWM maintained higher visual quality.
When the generation is extended to 961 frames, the test further checks the model to see if the image can respond when it returns to the nearest camera position. In this long-term setting, the VBench of the four-step factor flow version of EchoWM-Flash scores 80.13 and 81.06 respectively, leading the trajectory accuracy and angular retrospective consistency in the tested factor model.
In addition, the team conducted a user study on 200 self-constructed case-test sets, comparing JoyAI-EchoWM with the generating performance of LingBot-World-v2 and HappyOyster. The evaluations covered semantic follow-up, initial world consistency, movement, time-space consistency, visual aesthetics and overall preferences, while retaining the "both good" and "both bad" options.
Among the overall preference results, JoyAI-EchoWM accounted for 46.50% compared to LingBot-World-v2 and 21.57% compared to HappyOyster with 63.13% and 27.06% respectively.
The results of the segmentation also reflected the characteristics of different models. JoyAI-EchoWM gained more preference for consistency in time and space and visual aesthetics; HappyOyster was selected individually for a higher percentage of excellence in both semantic following and movement.
The evaluation shows the performance and boundaries of JoyAI-EchoWM and provides a basis for judging which real-world scenarios it is suitable to enter.
Written at the end
World models move into different application scenarios, resulting in a gradual differentiation of technical routes. Interactive content, speed of generation and operational response directly affect experience; spatial creation, geometric consistency, and scene editing are more critical; robotic models provide support for predicting environmental changes and executing actions. Therefore, to judge the value of a world model, it is necessary to see what problems it can solve in specific tasks.
As with Genie 3, Atlas, and Cosmos 3, these directions intersect and have different technological interfaces. The collaboration between deep, continuous interaction, spatial representation and operational capabilities is crucial as the application progresses. In this context, JoyAI-EchoWM is positioned closer to an interactive world model for generating media.
From the current capabilities, the game prototype may be the more potential application direction of JoyAI-EchoWM. Creators can enter the generating scene, experiment with spatial layout, route design and lens experience, and then decide the direction of subsequent production. Of course, a world that can be explored is still far from the mature game, and gameplay, rules, and mission design need to continue to be complementary.
In addition, interactive storytelling, immersive content and virtual tours also provide opportunities to reduce the need for pre-made material through JoyAI-EchoWM, allowing content to be continuously generated according to the user's choice.
With the release of JoyAI-Echo-Voice 2.0 and JoyAI-Echo-Echo-Voice 1.5, JoyAI-Video-Edit and JoyAI-Image-Edit, JoyAI-Talker and JoyAI-RA, the world model and audio-video capabilities of JoyAI-EchoWWM have been further expanded to cover the understanding, generation and interaction of the environment.
Based on this layout, Jindong is building a complementary foundational capability for AI to enter the physical world. At the same time, the long-term value of the model matrix depends on whether these capabilities can be effectively synchronized in specific tasks and continuously improved through real-world scenario feedback. This echoes Jindong's strategic goal of building "the world's largest physical world operations center". For Jindong, which has an advantageous business scenario such as retail, logistics, the next point of view is how to apply model capabilities to real-world processes and verify their reliability, cost and efficiency.
The open-source JoyAI-EchoWM provides a community-based approach to interactive audiovisual production and provides the opportunity for more real-world scenarios to be tested.