Can JEPA really understand the generation mechanisms behind the observations?
Traditional JEPA can only guarantee that learned implications do not shrink, but there is absolutely no way to match those implications that actually generate observational data.
Researchers from UIUC, Cold Spring Harbor Laboratory, Brown University, and MAP responded in the affirmative. Their proposed DSReg (Dependency-Sparsity Regularization) can be directly incorporated into a trained JEPA, allowing each dimension to correspond to a real-world variable such as the sun, moon, temperature, and wind in the scene.
At the same time, the team established a complete theoretical framework for the DSReg, resulting in the first fully identifiable JEPA. The DSReg can be applied to pre-training characteristics, directly replicating existing LeJEPA checkpoints, and achieving similar results to combined training.
DSReg: Provably Recovering Individual World Latents without Reconstruction
Link to the paper:https://arxiv.org/abs/2610.09457
The main page of the project:https://dsreg.github.io/
The code is:https://github.com/kunwuz/dsreg
It's not the same as being accurate.
Instead of rebuilding pixels, JEPA predicts the future in the expression space. It saves expensive decoders and is also a popular way to build world models.
Under certain assumptions, there is only one unknown linear transformation between the representation LeJEPA learns and the real-world state. This transformation makes the model completely unable to understand the real generation mechanism, as each dimension of the representation is often a mixture of several unknown physical factors.
These unknown mixtures sometimes have no effect on the prediction, but they completely impede the model's understanding of the environment. For example, robots push the cup to the left while keeping the next knife motionless. If a learned variable simultaneously mixes the cup and the knife's position, the modification of the variable will not only move the cup, but can also move the knife together, bringing danger to the user. Many times we not only need to make the machines look right, but also ensure that their cognitive bias is not lost, otherwise there will always be a hidden risk.
DSReg: Add a spin to the pre-training symbol
The idea of DSReg is straightforward. LeJEPA learns that after whitening, there is only one spin between the characteristic and the real variable, and the spin does not change the training loss of the JEPA. So as long as the correct spin is found, the variable can be disassembled without losing any predictive power.
But how to find the correct spin? DSReg places the goal on the structure between the hidden variable and the observed variable. Each real variable affects only a part of the observation, while the mixed variable affects all the observations involved.
Estimate structure: Taking some of the nodes in the data and returning to the local Jacobian with the adjacent nodes to estimate the observation of the representation.
Find the spin: minimize the scarcity of these Jacobians on the right-angled matrix.
The output variable: Each dimension after rotation is a world variable.
The entire process does not train decoders or optimize rebuild errors, and can be used directly at any LeJEPA checkpoint. Local Jacobian does not need to be explicitly stored, and the entire framework can be 8192D on a 48GB graphics card.
No two snowflakes are the same
Of course, without making any assumptions, it is impossible to recover the hidden world from the observations. DSReg requires a generating process to satisfy Structural Diversity: a set of observations affecting any two variables cannot be exactly the same, as without two snowflakes being the same.
This is a very loose condition. Observational variables with different implications can overlap, and one can even completely contain the other, which is arguably the weakest of all structural assumptions.
The theoretical proof is that under Structural Diversity, D.S.Reg can recover all the invariants, each dimension in the representation corresponding to a real invariant one by one, with a maximum of negative and negative sequences. The article also demonstrates the robustness of detectability for observational errors, and uses Lean to formalize theoretical validation of the results.
Let the world model understand the real world
In synthetic data, the characteristics of LeJEPA are always mixed, and the restoration effect is close to the upper limit in all dimensions after the DSReg.
On pixel-trained encoders, DSReg reaches the top of the label-compatible supervision, while rotation methods such as PCA, Varimax, FastICA are almost ineffective. On externally rendered 3DShapes, DSReg also rises from mixing to near the top; on more difficult dSprites, it also exceeds β-VAE and β-TCVAE.
DSReg allows the characterization to correspond to real hidden variables, thus significantly improving the performance of all downstream tasks. In six downstream tasks, including visual editing, scarcity control, rollout prediction, and accidental detection, DSReg outperforms LeJEPA, and the original prediction performance remains the same.
The DSReg also has a very high scalability, requiring a single card to handle 8192 cache variables, and performance does not decrease as dimensions increase.
Let's get the machines to work together.
JEPA learns a non-concentrating metaphor, and DSReg makes every dimension of the metaphor correspond to a specific world metaphor. Built on such a world model, robots can accurately understand every variable behind the observed scene.
This is especially important for robots: if you don't really understand the environment, you can just say the wrong thing; if you don't really understand, you can have unpredictable catastrophic consequences.
The next generation of intelligent entities who want to work reliably in a variety of new environments cannot rely solely on transferring knowledge learned in training environments. They must understand the generation mechanisms behind observations in order to achieve true generalization. DSReg gives JEPA this capability, which is perhaps a starting point towards this goal.
Introduction by the author
He is an assistant professor at the University of Illinois at Urbana-Champaign (UIUC) and a Ph.D. graduate of Carnegie Mellon University. His research focuses on causal AI, including causal expression learning, credible machine learning, and world models, and focuses on building the next generation of intelligent objects capable of understanding causal relationships.
© THE