This is what Ilya Sutskever said more than a decade ago.
"Models are meant to be learned".
Dario Amodei translated the words of Ilya Sutskever in an interview. Source: Dwarkesh Podcast
As long as the problem of data and arithmetic is solved, the model will find its own way to learn.
We all know what happened afterwards. The bigger the model, the more training data is used, the more computing power is put into it, and the capacity of the model is unlocked in a layer.
But when researchers focused on circular networks and outer space, everything changed.
Cyclic models can be computed repeatedly in hidden space, and according to the assumption that the same set of network layers run several rounds, the model can get more computation time, and encounters difficult problems can be "thought" a little longer.
But the reality is not so good.
Many models are only good at the loop depths they've seen in training, and if you give it a few more rounds, it could start going crazy in the wrong direction.
This is why some people joke that some models don't want to learn.
"Tell me more" means "think more".
If you open a large model of "deep thinking", you'll see the long lines.
There seems to be a lot of "crap" in this, and the model always tries to overturn it, and sometimes says, "I need to rethink this".
This demonstrates the model's reasoning process, which, while not directly viewed as a complete computational process within the model, also demonstrates that the most mature way of solving the current Transformer problem is to let the model speak while thinking.
Because the standard Transformer is kind of like a fixed-floor building.
Information enters from the first floor, passes through each layer, and finally comes out from the top. Whether faced with "1 + 1 equals several" or a complex mathematical proof that the depth of the network it passes through is basically the same every time the next Token is generated.
What can you do if you don't have enough?
The longer the output, the more times the same set of networks are called, the more opportunities the model has to dissolve the problem several times, check the answers and modify the answers.
This method is quite useful. It allows the original Transformer to be deeply fixed in the network, and it can also allocate calculations flexibly based on the difficulty of the subject: simple questions are less difficult, and complex questions are less difficult to answer.
However, this way of thinking is also somewhat uncomfortable.
The contents of the model exist in the hidden state of high-dimensional continuity, because the middle must "say", so many intermediate calculations must also be organized into an output sequence. Even if some steps are not said at all, the model must do something to keep going.
So there was another way.
Circulating directly on the network layer
The idea of a circular model is straightforward.
Once a network is not enough, it is necessary to go back to certain layers of the network and do it again. Each cycle, the model updates a round of concealed state, repeatable or original group parameters.
Thought chains synchronize multiple computations by generating tokens; a hidden-space loop allows the hidden state to be updated repeatedly in the same set of network layers. Source: Tilde Research
The Universal Transformer, the Deep Equilibrium Model, and the recent iteration of the Transformer are all asking the same question: Is it possible for the model to make its own decision about how many circuits to spin in the brain based on the difficulty of the subject?
It's like it's real.
A potential reasoning study in 2025 expanded the loop depth model to 3.5 billion parameters and trained with 80 billion tokens. When tested, the researchers ran several rounds of the same computing module, and the model did continue to improve on some of the reasoning tasks.
Link to the paper:https://arxiv.org/abs/2502.05171
As the number of cycles during testing increases, the model's performance on tasks such as GSM8K, HumanEval, etc. continues to improve; different tasks tend to blend in at different depths.
However, the number of rounds is not as good as the number of rounds.
The circular language model, Ouro, released in 2025, initially attempted to use eight-wheel cycle training, but soon lost peak and gradient oscillations. The researchers subsequently reduced the depth of the main training phase to four wheels.
During the test phase, the number of cycles continued to increase to eight rounds, but the additional calculations did not replace the better answer.
As an example of Ouro-Thinking with the 1.4B parameter, its score on AIME 2024 gradually rose from 0 points in the first round to 65 points in the fourth round; it moved on to the eighth round, where it dropped to 38.67 points.
In other words, the number of rounds coming out is helping.
What's wrong?
Currently, there are roughly three ways in which the circular model can be crossed.
First, is it theoretically possible to calculate whether sharing the same set of parameters, cyclically updating the cache state, is enough to express complex computational processes?
Second, the model can really learn how to calculate;
Third, after the training, the model cannot continue to "calculate" steadily during the reasoning phase.
The first point has already had a lot of positive results, and the problems are more concentrated in the latter two.
Repeat call calls can cause errors to accumulate, and after the number of cycles increases, the steady state may appear to be stable, but can't actually support results.
The Parcae study of 2026 found that circular language models are prone to sudden explosions of defects and losses.
After re-engineering the stability mechanism in the design cycle, the model's verification set confusion was reduced by a maximum of 6.3%; when it expanded to 1.3 billion parameters, it also exceeded the traditional Transformer baseline in fixed parameters and data volume.
Link to the paper:https://arxiv.org/abs/2604.12946
The frequency of the normal cyclic baseline rapidly exploded, and training losses subsequently stopped decreasing; the Parcae and residual recombination model, which joined the stabilization mechanism, remained stable.
The result is a turning point in the story of "models don't want to learn".
Today's standard Transformer is equipped with a complete set of years-long training equipment: residual connections, integrations, initialization methods, learning rate strategies and optimizers, all well-versed in practice.
Compared to the standard Transformer, which has been refined for years, the training formula for the large-scale circular language model is still far from mature.
How architecture, optimizers, loss functions, and intermediate supervision should be combined may also need to explore a more sophisticated combination.
The link is as follows: