The first author of the paper, Zhang Yu, a Ph.D. student at Nanchang Technological University MMLab, has published the first author of NeurIPS 2026 paper ClothTransformer, directed by Pan Xin Steel.
In the past few years, neural networks have re-written one area after another: image generation with Stable Diffusion, video generation with pseudo-reality, and even 3D rendering with pure neural networking methods such as RenderFormer. But in high-security physical simulation, neural networks have always failed to provide satisfactory answers, and the gaming, animation, and robotics fields still rely on conventional solvers that require manual modeling and difficult to obtain gradients.
Recently, researchers from the South China University of Technology MMLab and the Shanghai Artificial Intelligence Laboratory proposed PhysLDM, a latent diffusion paradigm aimed at physical simulation. According to the team, it was the first high-security time-to-space encoder-to-hidden-space diffusion paradigm aimed at object-grid transformation dynamics. With only one model, PhysLDM can simultaneously generate the entire trajectory of high-resolution transformed objects: rebuild accuracy to millimetres, sample generalization to never-before-seen objects, and end-to-end micromanagement.
PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation
Link to the paper:https://arxiv.org/abs/2610.07609
The main page of the project:https://yucrazing.github.io/physldm/
Physical simulation, why is a neural network a hard bone?
The mainstream neural simulators (such as MeshGraphNets) use the previous one to predict the next one, and errors accumulate and the long trajectory of time will drift. To solve this problem, it is necessary to generate the entire trajectory at once, as in video generation.
Second difficulty: one-time prediction, scale explosion. High-fidelity shapeshifting simulations require thousands to tens of thousands of tops, and the number of tops for different objects varies. Tens of thousands of tops are brought together into the Transformer, and the attentional complexity of the square-level cannot be matched: the team observed that even on the 140GB H200, without compression, individual samples would explode to 2,600 tops.
Difficulty three: there is no existing time-space compression program. In the field of physical simulation, most of the time-space compression is single-span, and fewer people make time-space compression for the entire trajectory; in the field of video, time-space VAE, direct transfer is also not useful.
Problem number four: generalization failure. Neural simulators are long-term illnesses that cause the replacement of an object.
These four difficulty circles are linked: to eliminate drift, one-time prediction; to predict one-time, space compression; and compression is not only convenient, it can also be generalized.
The Path of the Pioneer: Building a VAE for Physical Simulation
There is a VAE behind Stable Diffusion that compresses images into hidden space. Similarly, in order for a scattering model to generate a physical loop, a small, precise time-space VAE that compresses the entire simulation trajectory is needed first, equivalent to a physical simulation tokenizer. This is the core of PhysLDM and the part where the team invests the most energy.
This is not a good way to go. The most direct method is to compress the time of moving the video VAE: every few frames are merged into a hidden variable tube, and then the sample is restored. As a result, the object's shape changes to stop, then jump to the tube, and the team calls it a staircase staircase. This problem is almost invisible in static shots, but it is very obvious when played.
The team attempted to add velocity loss, spectral loss, and the ability of the inverse variable to adapt to distribution over time, but could not eliminate it. Furthermore, by directly compressing the world coordinates, the geometric details would become cluttered; by compressing congenital movements and shapes together in the same set of time, the misalignment would increase sharply, and shapes would also be severely compressed.
Ultimately, PhysLDM solved the problem by using two key designs: one is to break down each node into a total tangent movement + local transformation node, allowing the network to focus on learning only a few millimeters of transformation itself; and the other is to compress the entire trajectory using a set of learning time queries.
To address generalization, the team used the GPU-constrained metadata solver StiffGIPC, which generated about 80,000 simulation trajectories on more than 30,000 Objaverse objects. As a result, the average reconstruction error for time and space VAE was only 2.48 mm.
In this cache space, PhysLDM uses the scattering model to generate an entire trajectory once again.
Strong ability to generalize: a model, see the unseen
Generalized dysfunction is a long-standing problem in neural simulators that often requires re-training to replace an object.
PhysLDM handles a variety of situations with one model: after training on the Objaverse, directly using unprecedented GSO, Toys4K objects, reconstructing errors at the same level as in the training distribution; in the face of high-resolution grids that exceed the training peak twice the limit, the error remains about 5 millimeters; in exchange for another energy model (ARAP) generated motion, or actual point cloud trajectory, zero-sample reconstructing errors are only 4.6 millimeters and 6 millimeters, respectively.
This is due to the enormous variety of training geometries, as well as the method of training that uses only mechanical supervision, without binding any of the underlying models.
(For more rebuilding and predictive effects, see the project's homepage video:https://yucrazing.github.io/physldm/）
Key insight: In a chaotic world, dispersion is more about physics than regression
For physical simulation tasks, there has been no precise answer to this question. The team's answer comes from an anti-intuitive finding: even with traditional solvers, the results do not necessarily determine the solution.
In the same scenario, the same byte-level inputs run five times on the GPU solver, with a distinctly different final state, with a maximum difference of 442 mm at the top. The very small differences brought about by the GPU, rapidly enlarged by the chaotic characteristics of complex shapes, resulted in a completely different outcome.
In the face of multiple possible outcomes, regression models are pulled to their valve averages, which do not match the physical result. For example, with a dropped key, the solver runs two times in two directions respectively; each sample of the scattered model falls in one of the correct directions, and the regression model gives a different valve averages. In the experiment, the physical consistency of the PhysLDM is significantly superior to the regression and regression methods, where the error in the regression method is close to three times that of the PhysLDM.
PhysLDM can be extended to multi-object scenarios without the need to modify the model structure.
Potential: More than a Faster Simulator
PhysLDM is end-to-end fine-grained, and supports high-level navigation. Given only a single trajectory, it can reverse the material rigidity of an object in about 40 seconds, while existing micro-simulators either require manual modeling, take more than twenty minutes, or fail directly.
In addition, the above-mentioned real-point cloud zero sample reconstruction also demonstrates the potential of PhysLDM to learn physics directly from the real world.
After the publication of the paper, PhysLDM received positive feedback from many researchers and practitioners overseas. One robot simulation practitioner lamented that it would take us hours to adjust the physical effects, and software physics automation would change the rules of the game.
Neural simulation, although not yet widely used, has great potential: it can be fine-tuned on real data, fine-tuned end-to-end, and programmed in hidden space. PhysLDM has provided preliminary validation for both end-to-end fine-tuned and learning from real data.
Summary
After image, video, and 3D rendering, PhysLDM allowed high fidelity physical simulation to follow the path of generative AI: compressing the entire trajectory with VAE in time and space, generating it again with a scattering model, generalizing zero samples at millimeter-level reconstruction accuracy, and revealing that scattering is more in line with physics than regression in chaotic form dynamics. This is an important step toward a physical world model.
© THE