Over the past two weeks, the term "loop depth" has become a focus of discussion about AI security.
Two days before the release of OpenAI's new flagship model GPT-6 Astra, The Information quoted internal reports as saying that Astra used a design called recurrent depth, or looped transformer, which "makes parts of the model and even the entire reasoning process invisible".
Controversy escalated after the model officially went online, with OpenAI chief scientist Jakub Pachocki personally clarifying that he wanted to avoid "a race to uncontrollability triggered by chaotic reporting", and stressing that current cutting-edge models, including Astra, whose computational chart depths remain within a gap of twice that of GPT-4.
Sebastian Raschka, author of the book Build a Large Language Model (From Scratch), wrote an excerpt in which he recounted the technical details.
He first gave his own observations of the use of Astra, including its disproportionate lead in 3D rendering and graphical tasks, its ability to directly operate local software through the application of Codex/ChatGPT, and the training logic behind the acquisition of tens of thousands of Macs by OpenAI as a reinforcement learning environment. He then spoke from 2018's Universal Transformer, all the way to this year's open-weight model Nanbeige 4.2-3B, Ouro's and Google's Mixture-of-Recursions, which bounce byte...
In his opinion, the circular Transformer is not the reason for the shortening of the mind-chain, which has become difficult to monitor, but the more reasonable explanation is that models with higher capabilities make fewer mistakes, so it is not necessary to have such a long draft paperback GPT-5.6 in the Luna Sol family with more than 80% tokens to achieve similar performance, and no one says that Sol's explainability is worse. The Astra system card does admit that trajectory tracking traceability is declining, but there is no evidence to show that this account can be kept in a circular architecture.
Below is the full blog post:
Address of the blog:
A lot has happened in the past few weeks. I'm sure that what's most concerning right now is OpenAI's GPT-6 Astra -- particularly its performance, its circular Transformer/loop-deep design, and the rumor that Astra is "hidden" in its own reasoning trajectory (think chain).
So in this article, I'd like to talk about some of the initial impressions of Astra and how I feel about it all. Then I'll go into more detail about what the "circular Transformer" is and what it has to do with the hidden chain of thought (or, at least, what it has to do with it).
Finally, after discussing the basics of the Transformer cycle, I would like to present some new insights from recent papers.
1. GPT-6 Astra using the impression
Before going into architectural rumors and related research literature, let me briefly summarize some of my observations and findings about the GPT-6 Astra.
Last week, OpenAI released a new model, the GPT-6 Astra. I've been using it for the last few days. It's an incredible model, probably the best I've used as of this writing.
1.1 Astra's benchmark test performance
Astra is the best model I've ever used. And it's very disproportionate in 3D rendering and animation tasks compared to other models. I mean, while it's unlocked its predecessor GPT-5.6 (writing, math, programming, etc.) in almost all categories, it's particularly prominent in graphic class presentation.
This is also reflected in benchmark tests. As shown in the figure below, the GPT-6 Astra is indeed strong in mathematics and programming.
Figure 1: Select the results from three common programming benchmarks and a high-difficulty mathematical benchmark.
One of the highlights (not shown in the chart) is that Astra scored 99.9% on the ARC-AGI-3 benchmark, while GPT-5.6 Sol scored only 7.8%. This benchmark measures the mixed performance of logical puzzle solving and generalization capabilities. However, the mathematical, programming, and computer operations benchmarks are more noteworthy because they are closer to real use scenarios.
Back to Artificial Analysis Coding Agent Index v1.4, in the lower right corner of the previous graph, which combines multiple intelligence coding tasks. On this indicator, GPT-6 Astra is at the forefront, but does not significantly open up its opponents. The more comprehensive Artificial Analysis Intelligence Index below reflects the same situation, which combines multiple types of tasks rather than just programming tasks.
Figure 2: The Artificial Analysis Intelligence Index is based on data from the World Bank.
An advantage of Artificial Analysis is that it is independent, so it may be more reliable than the model developers themselves.
The specific test framework ((harness) depends on the benchmark itself. For example, GDPval-AA and AA-Briefcase unify all participating comparison LLMs using their own open-source framework Stirrup. In this version of Intelligence Index v4.2, Terminal-Bench v2.1 uses the Terminus 2,τ3-Banking uses the τ-Bench framework.
For the assessment of the use of a unified framework, this comparison is closer to controlling under the same standard. But at the same time, the model is usually developed around one dominant framework during training, with relatively little overlap with other frameworks.
Therefore, some intelligence assessments may underestimate Astra's actual performance under its own power framework. So much for the impact of this on its Intelligence Index scores, it is necessary for Astra to know the cross-frame contrast on the same task.
By the way, a colleague recently suggested to me (and the head of Claude Code also made a similar recommendation) that you remove or archive part of your existing AGENTS.md content and SKILL.md files, which may not be a bad thing.
Of course, I'm not saying that we shouldn't use SKILL.md files anymore. For some workflows, they can improve reuse efficiency because the model doesn't have to be retrieved again. What I'm saying is that some workflows don't need to be described at all; and those "old" descriptions may no longer be optimal, and LLM itself may come up with a better solution. So, maybe it's time to update or re-produce these instruction files.
1.2 Computer operation capabilities
GPT-6 Astra appears to be exceptional in image and rendering tasks. When these tasks involve interaction with graphical user interfaces, they also demonstrate computer use capabilities, i.e. the ability of the model to operate software on your local computer through Codex/ChatGPT applications.
Computer operation is where this model really stands out compared to other models. And anything that has to do with graphics can easily be presented in a fun and intuitive way on social media platforms. There are many impressive examples of this type online, from blender modeling rendering New York City to virtual room guides.
For example, here's a comparison: I used GPT-6 Astra Medium and High in the MS Paint version of my browser to re-draw a picture of me with my mouse on my computer, and I didn't use Extra High and Max because I didn't want to burn out the tokens:)
This reflects not only the ability of the model to draw, but more importantly its ability to use the tools on your computer (here's the drawing program, where you can see the model through the mouse pointer in the operating interface).
This is not the first model to have universal computer operating capabilities within the framework. For example, since earlier this year, I have successfully completed some UI tasks with the GPT model, such as billing related work in Excel, etc. However, computer operating is still a relatively new capability, it is supported by the framework and usually feels not mature enough to be used.
At the same time, there are a lot of tools and software that (temporarily) don't offer CLI. Instead of waiting for someone to design the interface, it's better to directly enhance the model's ability to use graphical interfaces (and as mentioned earlier, this would have made a beautiful and surprising demonstration). This is somewhat similar to the development of human robots today.
Therefore, I expect that the next few months or even years will be a period of continuous refinement of computer operations in both the LLM and the Intelligent Body framework. That is, in addition to existing capabilities and further expansion in mathematics and programming, model training will increasingly consider computer operations. This will also make the LLM more friendly to everyday computer tasks outside the scientific community.
1.3 How computers are trained to operate
This trend of computer operation is also consistent with recent reports: OpenAI has acquired tens of thousands of Mac Mini and Mac Studio models for reinforcement learning. Here, the Mac is not really a training model (it's better suited to GPUs), but provides a macOS environment for models to learn how to use the operating system and its tools during model training.
In short, these Macs (or, more accurately, their macOS system) serve as an environment in which models can interact during training.
The basic process is as follows:
Give the model a task as a cue, such as "Open the xyz application and complete the abc".
It provides a snapshot of the macOS interface (which is usually done by the framework).
The LLM then predicts mouse/keyboard movements such as clicks, buttons, rolls, etc.
On Mac, these actions are executed by the same framework.
After performing the previous step, a new snapshot of the updated environment is given to the model.
Repeat steps 2 to 5 until the mission succeeds or fails.
Using success/fail signals and verifiers (or punters) as training feedback, including reinforcement learning in the post-training phase; this is similar to conventional validated reward reinforcement learning (RLVR).
Figure 3: Overview of the process of computer operations training.
To emphasize again, the Mac here is primarily an environment, not a machine that runs or updates the model during training. The model is likely to run on the Invidia GPU and communicate with these Macs through the API. Incidentally, the CEO of Invidia mentioned that the GPT-6 Astra was trained on about 100,000 Grace Blackwell GPUs.
1.4 GPT-6 Astra remains a reasoning model
The focus on computer operations training in the previous section does not mean that the training process has undergone a fundamental paradigm shift. GPT-6 Astra and in the foreseeable future almost all LLMs remain a reasoning model. That is, such LLMs are trained with verifiable reward reinforcement learning (RLVR) and generate an intermediate reasoning trajectory, the chain of thought.
However, on the part of the GPT-6 Astra as a reasoning model, and in particular on the hidden chain of thought, I will go on later in this article.
2. Circular Transformer
Speaking back, about two days before the model's official release, the news outlet The Information reported that, according to internal reports, Astra used a concept called "looped depth" or "looped transformers".
Figure 4: A quote from The Information:
In the next few sections, I'll explain what the Transformer cycle is, and I'll talk about the concept of hidden thought chains later in this article.
(Circle Transformer's explanation may seem a bit long, but I really think it helps to build a basic understanding of the technology. With this understanding, we can judge whether "it will obscure the trajectory of reasoning or the chain of thought".
2.1 Reuse the Transformer block
So what is a Circular Transformer?
A circular Transformer is essentially an architectural fine-tuning, with the core idea being that the middle represents multiple passages through the same set of Transformer blocks rather than just one.
Definitions and Terms
In this article, I will use the following terms:
A transformer block is a unit that contains attention, front-end modules, integration and shortcut connections. These are often referred to as "transformer layers" in the paper.
A stack is a sequence of Transformer blocks.
A block application is a type of application that allows the input of a single block of a Transformer.
Circular Transformer isn't new, as the basic idea emerged in a 2018 paper by Universal Transformers. However, before discussing Universal Transformer, let's start with a simpler example: Nanbeige 4.2-3B. This is an open-weight LLM released in July, which I presented earlier this summer at Substack Notes and my LLM architecture gallery.
The Nanbeige architecture shown below essentially looks like a conventional Transformer. But note that it adds an orange arrow that goes back to the beginning of the Transformer shell.
Figure 5: Nanbeige 4.2-3B uses the same set of 22 Transformer blocks twice.
First, like any other Transformer-based LLM, the input text is tokenized and then converted into embedded vectors. These vectors are then passed through 22 Transformer blocks, each with its own set of weights.
The characteristic of the Circular Transformer is that after the first run, the hidden state is sent back to the same 22 blocks over and over again.
If this calculation process were to be carried out, it would be equivalent to 44 blocks of application. However, unlike conventional Transformers with 44 independent blocks, the second round of 22 blocks of application repeats the weight of the first round. For example, the 23rd block of application uses the weight of block 1 and the 24th block uses the weight of block 2, and so on.
Figure 6: After Nanbeige 4.2-3B is unfolded, this is equivalent to two inputs through the same 22 Transformer blocks, for a total of 44 applications.
So the whole idea is to increase the effective depth from 22 to 44 applications without adding a new set of transformer weights.
In Nanbeige's paper, there is little detail, but they say that this is basically the most efficient configuration. Increasing the number of cycles from 2 to 3 does improve modeling performance, but the additional computational costs are not budgeted.
2.2 Costs of cycling
After all, it's an alternative to "growing the model by stacking more Transformer blocks".
For example, a model that uses 22 Transformer blocks twice has only about half the parameters of a model that has 44 conventional blocks.
In this way, the memory required for the storage weight is reduced. Add a sentence: the embedded and output layers are usually large, accounting for a considerable proportion of the total parameters, but they are not within this comparison range. In Nanbeige 4.2 3B, for example, the embedded and output layers account for about 25% of the total parameters of 3B; this proportion can be reduced to 12.5% if the two layers share the weight.
Figure 7: Parallel comparison of the parameters required for traditional schemes with circular schemes.
Of course, repeating the same blocks in the loop is still to be counted. More precisely, the intermediate input is to complete 44 blocks of application when forward to propagation. While in training, the gradient is to reverse the two rounds of flow-sharing loop. So this will bring a lot of extra work compared to just using these 22 blocks at once.
There is also a problem with KV caching. Whether it's a conventional Transformer or a circular Transformer, the KV caching keeps the attention keys and values of the previous token, for each subsequent step in the token generation to be replicated.
Although there is weight-sharing in the loop Transformer, the middle state of the second entry into a block is different. Thus, in KV caches, the keys and values generated by the two rounds of the Transformer are different (as when not in the loop).
To be more specific. For example, in the 1st and 23rd block applications, they are all blocks 1 in the Circular Transformer, but each application still needs their respective KV cache entries. Since both wheels have to retain the cache separately, the 22 blocks that are used repeatedly have the same KV cache requirements as a conventional Transformer with 44 separate blocks.
Interestingly, the researchers at Nanbeige mentioned in the paper that they tried to share KV caches between the two rounds. This, of course, halved the volume of KV caches, but the model performed less well using the version of independent caches, the one they eventually released.
Before continuing to look at the other circular Transformer designs, complete this section of Nanbeige. Their technical report also discusses two other options, or weights.
From the beginning of the training cycle, the effect is better than transforming pre-trained Transformers through upcycling.
As mentioned in the previous section, the two rounds are their preferred weighting points. The additional gains from more rounds are small, but will delay training and make optimization more unstable.
So the number of cycles is another architectural choice that we have to make. As mentioned above, Nanbeige fixed it to 2. But we can also change it with tokens, and we'll see how that works.
2.3 Universal Transformer with flexible loop frequency
Now back to Universal Transformers. In Nanbeige, we've used the same 22 Transformer blocks twice, and in the 2018 Universal Transformer paper, the same Transformer blocks are used repeatedly, not a whole block.
In addition, the number of steps can be fixed, but the paper also explores adaptive halting. For example, tokens in one location can only go one to two rounds, and another token can go three to four rounds, depending on this. This allows the model to flexibly allocate arithmetic to tokens that can actually benefit from additional computation.
Here, the model uses a small, trained function to output a so-called stop probability for each position at each step. It adds these probabilities to a continuous loop and stops the loop once the cumulative value of a position exceeds the threshold value.
Figure 8: Adaptation stopping mechanism in the Universal Transformer.
Another example of a looping Transformer is the Ouro, which bounces bytes. For example, Ouro-Thinking 2.6B uses the same set of 48 Transformer blocks four times. This is a 192 block application, with only the weight of 48 separate blocks stored. Basically, this is more radical than Nanbeige.
2.4 The number of cycles determined by routing
Another idea is Mixture-of-Recursions, a 2025 paper that is essentially an advanced version of the aforementioned Universal Transformer. Similar to Universal Transformer, a single token goes through the Transformer block once or more, as shown below.
In the diagram from the paper below, the loop is known as a recursion block. It contains several Transformer blocks and lies between two separate Transformer blocks at the end of the diagram, with the symbols Layer 0 and Layer L-1.
Figure 9: Mixture-of-Recursions uses a different number of times the shared token is placed in different locations. High-light text shows examples of one, two or three iterations.
In the previously discussed Universal Transformer, the stopping probability for each step of learning depends on the Mixture-of-Recursions, where a small, learning router is used. This is similar to the routing thinking in the hybrid expert model, except that the routing decision here determines how many times the shared circuit should be applied.
The router acts on the hidden representation of the token, and the hidden representation also contains context information. Therefore, we should not understand it as "a certain token is assigned the same number of iterations each time it appears", that is, the "People" in the chart above do not always go through three cycles.
The paper explores two approaches, as shown below.
Figure 10: Two ways to choose the depth of recursion. On the left, the router selects which tokens to continue to participate in each step; on the right, the number of recursions is initially allocated by the single router.
In expert-choice routing, each iterative step selects which tokens to process and the outgoing token no longer participates in subsequent steps. In token-choice routing, the router makes a decision at the beginning, allocating each token to one, two or three walks.
In both cases, the Transformer weights are replicated between rounds, similar to the Nanbeige approach. Additional flexibility comes from being able to determine how much computation each token receives. The model and its router are trained together, so the model can learn to cope with these different paths during training.
2.5 What is the effect?
The following graph, taken from the Mixture-of-Recursions paper, compares the performance of conventional Transformers (Vanilla), fixed recursive Transformers (Recursive) and Mixture-of-Recursions (MoR) under different model sizes and computational budgets (x-axis).
Figure 11: Four model sizes, three verification losses under the training capacity budget. Figure from the Mixture-of-Recursions paper.
On the smallest model scale, conventional Transformers perform best. On larger models, Mixtures-of-Recursions follow up, and often perform better, especially on an hourly training budget.
Another detail is that the training algorithm is the same, but it does not mean that the number of tokens is the same. By skipping the partial calculations, Mixture-of-Recursions can process more tokens within the same budget.
I think this is an interesting example, because it explains that there are a number of choices within the concept of the Transformer Cycle: how many times each location cycles, and why this number is determined.
In short, if the model is large enough, the use of a circular Transformer can improve the quality of the model under a fixed arithmetic budget. This also illustrates the importance of large-scale experiments; for example, looking at small models with only 135M parameters, we come to the exact opposite conclusion.)
In addition to this, it is important to note that there are two types of neurotransmitters.
By the way, if you have a deep learning background and even know about the artificial neural networks of the 1990s, then you should be familiar with the idea of the loop, or "loop depth". Remember the RNN?
Figure 12: RNN shows its intention to publish its 2022 self-published book Machine Learning with PyTorch and Scikit-Learn
The main difference is that the RNN is a time-spaced replication weight. That is, the hidden state is passed from one token to the next. In a loop Transformer, the loop of a token takes place in the deepest direction of the structure.
In other words, in a conventional RNN, each step receives the next element of the input sequence and the hidden state of the next step. So when the RNN processes a passage of text, it reads a word or a token at a time and carries the information of the preceding word forward through the hidden state.
In the Transformer loop, the middle representation of a token is that it passes through the Transformer multiple times. The model still relies on attention to transmit information between the tokens.
If this analogy makes you a little nervous, it doesn't matter. Perhaps the simplest way to understand it is to think of the Transformer cycle as a replication of the Transformer block, which is similar to enlarging the model, but the weight is shared.
Figure 13: Parallel contrast between the RNN and the "loop" in the Circular Transformer.
Astra does not use the Circular Transformer?
Before discussing whether the Circular Transformer mechanism obscures the reasoning trajectory as suggested in The Information, let's ask a more basic question: Does the GPT-6 Astra actually use a Circular Transformer?
Figure 14: Quoted from The Information, source:
We must remember that this is still only a rumor or an exclusive outburst, with no official confirmation. If the model is open weight, we can of course verify it ourselves, but for now we can only rely on unconfirmed reports.
But I think the GPT-6 Astra probably did use the related design of the Circular Transformer. First, there are reports mentioned above. Second, this technology has shown potential in previous studies (as mentioned above), so why not? Third, the chief scientist at OpenAI said the following.
[...] The computational graph depth of our current cutting edge model (including Astra) is at least twice that of GPT-4.[....]
But this does not explicitly confirm the existence of a circular Transformer architecture, and it may just mean that they used double the number of conventional Transformer blocks.
In my opinion, Astra's success (i.e. good modeling performance) is mainly due to other factors, such as better training formulas and training data.
Cycle Transformer may have helped, but I think The Information overestimates its contribution.
5. Hidden chain of thought
Next, we finally face the elephant in the room: Will the Circular Transformer obscure the reasoning trajectory?
First, OpenAI has been hiding most of its reasoning trajectory from users since the beginning, at least since OpenAI o1.
As a result, the concerns about explainability are primarily aimed at model developers.
In any case, I don't think the Circular Transformer is the main driver of concealing or obscuring the chain of thought. To clarify my reasoning, let's take a step back and talk about how the reasoning model works.
5.1 Summary of reasoning
Reasoning models usually generate some intermediate steps before giving a final answer. These steps use ordinary text tokens (in some user interfaces you can choose to hide them from the user), known as reasoning trajectories or thought chains.
For example, let's say we want to find two numbers, and 10 is equal to 21. In the diagram below, the model first tries 5 and 5. Although yes, the multiplication is 25, not 21.
Figure 15: An illustrative LLM response, marking the intermediate steps, the retrospective and the final answer.
This diagram shows how a reasoning model can be "reasoned", including retrospectivity. That is, the model detects errors, returns to a previous choice, and then proceeds with a different way of thinking.
Note that the model is still one to generate a token, and takes the cue word and the previous token as the context. So these intermediate steps take the role of draft paper, which increases the calculation amount before the final answer is given.
As the above example shows, the final answer may be much shorter than its predecessor logic trajectory. OpenAI tends to hide most of its logic trajectory from users.
Build a Reasoning Model From Scratch is a book I recommend for a deeper understanding and development of reasoning models.
Figure 16: My book Build a Reasoning Model From Scratch explains the basics of reasoning models.
5.2 Token use with shorter thinking chains
The more tokens that come out of the logic trajectory will bring more computation. And the circular Transformer will also increase computation, because the token has to go through more Transformer blocks. So someone might say: Since the circular model consumes more computation internally, it doesn't need as many external thinking tokens.
The following is the benchmark result for part of GPT-6, where the x-axis is the output token number.
Figure 17: Some of the selected GPT-6 Astra benchmark results.
As can be seen, at various effort grades, the GPT-6 Astra as a whole does not necessarily use fewer tokens than its predecessor GPT-5.6 Sol. However, under conditions of fixed accuracy, the GPT-6 Astra does use less than GPT-5.6 Sol.
Is this a concern about explainability? Not necessarily. Token is used less, perhaps just to indicate that the model is stronger, makes fewer mistakes, traces less, etc. That is, it may just do more right on the first try.
I mean, it's the same with the previous model. I don't think anyone would worry that the GPT-5.6 Sol's explainability is much worse than the smaller GPT-5.6 Luna, and as the chart below shows, Luna uses a lot more tokens when performing the same task.
Figure 18: Token use by Luna and Sol in similar missions. Data from Artificial Intelligence Index v4.3
In fact, we can see that Luna uses 80% more tokens than Sol when modeling performance is close to that. Does that mean that Sol's explanation is so poor?
A more logical explanation is that a model with more capabilities -- bigger, more trained, more computing power consumed -- can solve problems more efficiently, and "efficient" here means fewer tokens.
It is also important to remember that logical trajectories do not guarantee accurate descriptions of everything that happens inside the model. In my opinion, the only real concern is that the Circular Transformer will deliberately mislead users with "false" logical trajectories more often than conventional Transformers.
In addition, Astra's system card does mention that there is evidence that the traceability of its reasoning trajectory has decreased, with some regression occurring relative to Sol. This is mainly due to shorter trajectories and less information. But this also does not prove that the cycle is the underlying cause. It may also be simply because the overall length has shortened, as in the example of Luna and Sol above.
Hours after I shared my views on the circular Transformer and the hidden chain of reasoning, Jakub Pachocki, chief scientist at OpenAI, also released the following clarification:
I want to avoid a race to be "unmonitored" because of confusing reports. Our current cutting edge model (including Astra) is at least twice the computational depth of the GPT-4 gap. Since the earliest reasoning model, OpenAI has been working hard to preserve and use mind-chain monitoring. We value this technology very much because it allows us to see how the alignment of the model transitions out of the training distribution. I do think it's fragile, and unfortunately it's moving in the wrong direction, because it has nothing to do with architectural change, and I'll be writing specifically soon.
The "Chaos Report" is mostly referring to the previously mentioned passage in The Information, meaning that the cycle has nothing to do with the change in the chain of thought.
6. Research related to Circular Transformer
Finally, I would like to share some interesting papers related to the Circular Transformer architecture that have not been discussed before.
6.1 Possible reasoning
Related to Universal Transformer is the 2025 paper Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach, which examines how models use additional cycles in reasoning.
It does not use the same block repeatedly like the Universal Transformer, but repeats an entire block like the Nanbeige; however, it differs from the Nanbeige in that it shares the shared block of four blocks between the two starting blocks and the two ending blocks.
The other difference from Nanbeige is that at the beginning of each cycle, sharing takes out the hidden state of receiving the previous round, and receives the output of the starting block. The two are packed together, after a linear projection that can be learned, and then enter the four shared blocks.
In a nutshell, this is another interesting variant of the Circular Transformer.
Figure 19: Conceptual summary of potential reasoning models in papers by Geiping and others.
An interesting detail is that researchers change the number of cycles during training. This prepares the model to accommodate different calculations when reasoning.
Specifically, the number of cycles during training is randomized. In reasoning, the people running the model select a fixed budget, such as 8 times, 32 times, or 64 cycles. In addition, they also design an adaptive stopping mechanism for each token based on the probability distribution of the next token. If the KL spread between two consecutive rounds is below a certain threshold, i.e. the two distributions are too similar, the cycle stops.
In their evaluation, the performance of the HellaSwag generally slowed down after about eight rounds, while GSM8K and HumanEval could benefit from more rounds.
However, although the title of the paper mentions "potential reasoning", the model can still generate a chain of thought in the form of text.
6.2 Knowledge retrieval vs. reasoning
There is a very useful distinction between "storage of information" and "solving problems with information". For example, the June 2025 paper Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws explores this by measuring the memory and reasoning abilities of an LLM, respectively.
First, in memory experiments, when parameters are fixed, the loop almost does not change the amount of information stored in the model. Increasing the number of different parameters does indeed increase this capacity. It can therefore be concluded that the loop does not increase knowledge and does not allow the model to retrieve more knowledge. This is also commonly said. Once information has existed, retrieval itself is a relatively simple task.
Second, in other reasoning experiments, replicating these blocks can improve the performance of multi-phase mathematical problems without adding parameters. This is interesting. The conclusion can be drawn from this: even if the model does not have more space to store information, additional computation can help it solve the problem.
Figure 20: In this memory test, the capacity increases with the parameters, but the number of times the block is applied almost does not change the capacity.
6.3 Circulation under the conditions of a balanced computational budget
The newly published September 2026 paper, SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers, goes back to the cost-comparison problem in Section 2.2. What happens if we compare a circular Transformer to a conventional Transformer under roughly the same conditions for each token's computation, non-embedded total parameters, and KV cache requirements?
The researchers used a mixed expert architecture to apply the middle half of the Transformer block twice. This was a bit like Nanbeige, but with Latent Reasoning, a "careful" structure.
However, in order to offset the computing costs of extra block applications, they narrowed down the hidden dimension. This would also reduce the parameters, so they increased the number of experts and added the total parameters. They also adjusted the configuration of the attention head to keep KV cache size comparable.
Figure 21: SMELT overview, from the examples given in section 3.2 of the SMELT paper.
Experiments reached a peak size of 540 million non-embedded parameters, etc. Subsequently, researchers estimated that within the computational range they studied, SMELT achieved about 68% to 18% less training than required for the same verification loss.
So, that answers the question of "circular transformers are not computationally valuable": value! under the same computational budget, they can give us a slightly better model.
6.4 Full-band Transformer
Finally, a very new paper, published in August 2026, explores the cycling of cross-token locations. At each decoding step, it combines the final hidden state of the previous token with the embedding of the newly sampled token through a learning gate control. This results in the next forward propagation input.
Thus, the calculation of the next token can be done from the bottom of the string to the final representation of the previous token, which is somewhat similar to Latent Reasoning.
Using the 1B baseline model, they found that this potential feedback method would produce shorter reasoning trajectories on MATH500 while maintaining or even improving accuracy. However, this shortening effect disappeared after instructional mitigation.
Figure 22: Potential feedback shortened the reasoning trajectory of the baseline model, but this effect disappeared after the instructions were tailored. Adapted from the paper by Wang et al., Figure 6, CC BY 4.0.
In summary, this is interesting because it directly echoes earlier discussions about "whether a cycle will lead to shorter reasoning trajectories". Of course, the results depend both on feedback mechanisms and on how the model is trained.
Additionally, a major limitation of this study was that they did not test the conventional method of enlarging the model (i.e. stacking more Transformer blocks rather than cycling) to see if it would have a similar effect on the length of the reasoning trajectory.
What is the meaning of this verse?
In summary, we can say that OpenAI's GPT-6 Astra is indeed a very powerful model, and it has taken a particularly big step in computer operation. I believe that in the next few months, computer operation will be the next focus of the open source and closed source framework.
In addition, it is likely that the GPT-6 Astra will use a variant of the Circular Transformer. The Circular Transformer can deliver better modeling performance under a fixed calculation budget.
Also, better modeling performance may be accompanied by a shorter chain of reasoning. But this is not a new trend. Between models of different sizes in the same model family (e.g. GPT-5.6 Luna and Sol), we have always seen this.
In my opinion, shorter logical trajectories are more a byproduct of "smart" models with stronger capabilities. These models make fewer mistakes and can move more computation within the architecture rather than using logical trajectories as draft papers. In a sense, the same is true for people. In a university field mathematics exam, an intelligent and well-prepared student is less likely to use draft papers and less likely to have to go back and forth.