AI pioneer Yann LeCun has argued that "generated AI is a dead end, and world models are the real future".
He argues that there are mathematical hard wounds that arise from regression, that text does not carry much of human knowledge, that real reasoning is not based on optimised inferences, that the criteria for measuring intelligence should be the speed at which new tasks are learned, and the ability of zero samples to cope with whole new environments.
Based on this view, LeCun sought an alternative route to the World Model + JEPA, a set of goal-driven architectures. A recent paper he submitted was a big step in this direction, arguing mathematically that "next-token prediction routes are inefficient" and giving his own approach to defeating 7.2 billion LLMs with a 109 million-parameter model.
Would a model that looked only at images, with a text-sized model, form a similar "concept map" internally?
The Platonic Representation Hypothesis (PRH) proposed by Huh et al. in his paper ICML 2024, refers to a model that is trained separately on different models (visual, linguistic) and that, as the scale increases, its internal representations become more and more similar, eventually converging into the same shared statistical model for reality. This is similar to how different people in Plato's Cave see the same "world of ideas".
What this paper does is turn it into a theorem and give it the full working conditions.
Proving and Improving the Platonic Representation Hypothesis with JEPAs
GitHub：
Let the model learn the same scene from different descriptions
Each sentence that a text model reads mixes two types of information: what it describes and how it describes it. Dogs chasing the ball belong to the former, with "chasing" or "chasing" belonging to the latter.
But in training, how do you tell a model what content is worth keeping?
The method used by the authors is to allow the model to study different observations of the same scene at the same time. Two people describe the same photo, the words and sentences can change, but the subject and relationship in the scene are largely the same. If the model is asked to extract similar expressions from these descriptions, it can give it a learning signal: to find the common content behind different expressions.
JEPA (Joint Embedded Prediction Architecture) provides a way to implement this idea. In the setting of this article, the encoder first converts the input into a vector and trains it to approach a set of input vectors. This means that the choice of which inputs make up a set directly affects what the model is encouraged to retain.
In order for the model to extract scene information from the text, training should give it different descriptions of the same scene, or different versions of the same sentence obtained after interception.
Suppose in a photograph a dog is chasing a game on the grass. Two people wrote a description: "A dog chasing a ball on the grass" and "A puppy on the grass is chasing a toy".
The first group intersected sections of the two different descriptions, covering parts of the words respectively. For example, what the model saw may have been "dogs chasing [covering] on the grass" and "dogs on the grass are chasing [covering]". The two phrases expressed differently, and to get their vectors closer, the model needed to make more use of the behind-the-scenes content that matched "grass" with "grass", "chasing" and "chasing".
The second group took only the first sentence and processed it into two versions, such as "a dog chasing a pelvic ball" and "a dog chasing a pelvic ball". Both entries retain the same phrases as "a dog chasing" and models can use the similarity of the letters to bring their vectors closer together.
The authors refer to the first group as "good-view" and the second group as "bad-view". Here, "view" can be understood as an observation given to the model during training. Both groups use the same JEPA learning method to examine whether a scene, or the expression of a phrase, that is shared between the inputs, makes a difference to the representation.
"Good Vision" makes PRH a theorem and increases the sampling efficiency of the alignment process by 42 times.
This set of exercises also solved a problem: if the model converted all the text into the same vector, it could also meet the "vector approach" requirement. Therefore, the author added the constraint of preventing "expression collapse" to keep the overall expression dispersed.
In this way, the relationship between text and images is also easier to understand. The wording is text-specific information, the angle of shooting is image-specific information, and the subjects and relationships in the scene may be held together by the two observers. If each trained model learns to extract these common structures, their internal representations have an opportunity to converge.
Subtitles generated by a trained network that never accepted any image-text: using frozen DINOv2 and DINOv3 image encoders, rotated only once, and using text-only language modeling, new subtitles are generated each time the scene changes.
The authors give mathematical conditions for this. In a finite world of theoretical construction, they assume that each observation is sufficient to determine the corresponding state of the world. If different models are trained to connect to the same world state relationship, then after returning to the state of the world, the models are trying to solve the same optimization problem. Under the only conditions that are optimal to undo the spin, the resulting representation is less than one total spin.
It's like two people drawing a map in different directions, but the relative positions between the cities are consistent and can be reconnected after a turn.
This also leads to a comparison with the next token prediction: what kind of learning signal does the model give to predict the words behind it?
In order to be accurate, the model needs to utilize content, as well as grammar, wording, and text. The authors incorporate the next token prediction into a framework similar to JEPA by rewriting the target function, pointing out that it connects the forerunner and subsequent content of the same document. This connection retains the document's own expressive characteristics.
Accordingly, the authors propose an explanation: good visual training directly connects different descriptions of the same scene; the next token prediction needs to gradually establish connections between these descriptions from a large number of linguistic associations. The latter may need more data to form expressions that are compatible with visual models.
The next experiment to be tested is whether the difference in this training signal will lead to a large effect gap.
How much can a different way of training make a difference?
If the above explanation is correct, the choice to change the view should change the approximation of the model to the visual representation in the same model size, training content.
The team first tested this on Flickr 30k. Each image in this dataset has five artificial descriptions that can both provide different expressions of the same scene and process multiple versions from single-sentence descriptions to match the difference between good and bad views.
The researchers used the same text encoder and descriptive content to train the good view JEPA, the bad view JEPA, and the next token prediction model, respectively, and then compared it with the visual encoder trained independently.
One way is to compare their chosen "neighbors". For the same set of image samples, the visual model finds similar images for the image, and the text model for the description finds similar descriptions. If the selected neighbors on both sides correspond frequently, it shows how they organize these scenarios in a consistent way. The mutual 10-NN indicator used in the paper is the degree to which these neighbors overlap.
The results showed that the text representation obtained from the JEPA was more consistent with the visual representation.
In the Flickr 30k experiment, JEPA led the way in terms of neighborhood convergence and image retrieval indicators after rotation mapping.
However, good visual training also uses an additional piece of information: which descriptions belong to the same image.
For this purpose, the authors put together two descriptions of the same image and gave the next token to study the predictive model, with the result being a small improvement. This indicates that, after providing the relevant descriptions, it is also necessary to consider how to train them: rewrite the two descriptions together, and the effect is not the same as asking them directly to form an approximate representation.
The next question is whether this advantage can be extended to larger, more complex source web texts.
The team used Recap-DataComp data, in which each image corresponds to the original alternative text, and two descriptions generated by LLaVA. These texts provide multiple views of the same image, and the text model reads the text in training.
Here, the authors further compare the amount of training samples needed to reach a relatively aligned level. Using the same 109 million parameter encoders, the same source of content, JEPA obtained a WIT alignment score of 0.109 after processing 21.6 million training samples; the next token predicted a 0.111 after processing 900 million samples.
The scores of the two groups that have been evaluated are close, with a sample size difference of about 42 times. Due to good vision training each sample to read three texts, the gap is about 14 times greater based on the actual text tokens read.
So, continue to expand the training and see what level the JEPA can achieve?
Using 94 million web images of corresponding text, the team expanded the training of 109 million parametric models to 900 million samples. On the WIT-1024 benchmark, with DINOv2-B as the visual reference, three training sessions in the default configuration achieved an average alignment score of 0.184, higher than 0.154 for Qwen2.5-72B, and 0.151 for Llama-3-70B.
This is the concrete result of the "109 million parameters over 720 million" correspondence: the text-neighborhood structure learned by the small model is more in line with this independently trained visual model.
Similar controls once again reflected the role of training methods. Using the same encoder, the same text, and the same number of training samples, the next token was predicted at 0.111, and the bad view JEPA was 0.088, lower than the good view JEPA's 0.184.
Good insight into the performance of training in web text expansion experiments.
In this respect, the experiment supports the previous explanation from three levels: changing the training method can improve the level of alignment, reduce the samples required to reach a relatively close level, or allow a smaller model to obtain a representational structure closer to the visual model than a larger language model.
Is this structural approach going to be more successful?
The author demonstrates the task of describing images. First, the text decoder is trained to reproduce the vector depending on the text of the JEPA description. When the image is processed, the vector is extracted by the frozen visual encoder and converted to the text representation space via a linear mapping. The decoder generates sentences.
Among them, mapping uses graph paired data matching and decoder training in text. After further combining components such as DINOv2 and DINOv3, the system achieved a CIDEr score of 0.838 on COCO, which measures the degree of consistency between generated descriptions and artificial descriptions.
Separately learning the visual and text modules, from which the scene information provided by the image, after mapping into the text representation space, is ultimately read as a description.
Examples of COCO images illustrated by the authors. Separately learning visual and text modules, connected by mapping, convert scene information in images into text descriptions.
What is the meaning of this verse?
For LeCun's ongoing World Model Route, the paper builds the foundation on the design learning link, explaining under what conditions different models can form a common structure, and demonstrating the impact of training goals on learning efficiency. On this basis, whether these models support reliable forecasting and planning will determine how far this route can go next.
Yann LeCun's research provides a direction for multimodal models that is worth continuing to verify.
References are given below:
© THE