Where are the limits of Transformer? The question has been widely discussed since the introduction of ChatGPT in 2022.
Over the past two months, the answer to this question has been blurred. Former OpenAI head of reasoning Jerry Tworek and former Google Gemini head of pre-training Rohan Anil have publicly stated that Transformer has come to an end and has become the biggest bottleneck for AGI.[1]
Domestic flagship base models are also actively trying to improve the Transformer architecture, but part of the reason for these improvements is due to the mathematical bottlenecks and not entirely to improve the architecture's capability limits. By reducing architectural complexity through scarcity/linear attention, these schemes have significantly increased bootstrapping efficiency, successfully achieved larger parametric scales where resources are limited, and have gradually brought the model's capabilities closer to closed-source models abroad. However, six months later, whether these lower complexity architectures will also clash, requires a question number.
The Mobius team, a graduate student at Shanghai's AI Lab, gave a possible pathway through the separation of knowledge and reasoning, exploring higher-level architectures in continuous learning, combined generalization, and, along with higher-level capabilities, improving reverse-training effectiveness.
With this in mind, the team has proposed the Mobius series of architectures that are expected to empower a number of key issues or areas in the future, such as commercial landing, continued evolution, and scientific discovery.
Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning
It's a bit of a mystery.
Hugging Face links to:
The technical report links to:
Arch Space links to:
Transformer's "Three Sins"
Since the launch of Transformer, the industry has gained world-renowned power models by continuously scaling data and parameters in a way that has a very high ROI. These models have demonstrated unprecedented superintelligence in many tasks such as natural language, code, mathematics, and video.
However, as the Transformer-based model grows larger and more capable, the "three sins" that Transformer has always had have been overlooked and not solved effectively.
(1) The lessons learned cannot be quickly recalled;
(2) New schooling, wanting to remember, cost too much;
(3) Uneducated, difficult to build through connectivity.
The first generation of ChatGPT still lacks strong mathematical and coding capabilities. With the advent of OpenAI O1, the Chain-of-Thought (CoT) and Reinforcement Learning (RL) algorithms have officially entered the public domain.
In a deep-thinking mode, by continuously issuing tokens, the model can continue to test, validate and correct errors, and significantly increase the probability of answering complex problems. You can see that the model will not solve these problems but will need the right means to stimulate their inherent potential.
However, extreme "stimulation" means extreme unintelligence and slow response, and long CoT is just a less efficient way of stimulating intelligence. Disregarding whether or not it is intelligent, this exaggerated CoT will also cause the model to consume a great deal of computation and time in completing the task.
For 2, this is actually a long-standing continuous learning problem in the field of AI, narrowly referring to not forgetting old knowledge while learning new knowledge, and not to the Transformer itself. Instead, after the Transformer architecture was continuously Scaling, it was discovered that the architecture had Few-shot Learning and In-Context Learning capabilities.
Therefore, when answering a question that the model has not learned, the model has a great opportunity to answer that question as long as it constructs enough context through RAG.
However, human needs are limitless, and the attention of Transformers is limited. As the knowledge required to solve problems grows, the low information density attributes that RAG acquires in the above context will become the bottleneck of model-resolving problems (even when taking a paper exam, without focusing on hundreds of pages of books we have never read, it may be difficult to find the answers).
For Transformers, however, the most effective method of high-density memory is large-scale pre-training. If you want to add knowledge to a new field, it's best to mix old core data with some new data, retrain or even retrain the model over and over again. On the one hand, the cost is unacceptable; on the other hand, it goes against the original intent of continuous learning.
For 3, this is a longer-term combination generalization problem that has plagued the field of AI, narrowly referring to knowledge that is not typical of Transformer, but that is learned automatically through seen knowledge. And, with Transformer's Scaling, it has been found that such models can also perform tasks that have never been seen before at zero-shot.
Currently, however, the problem that Transformer can solve with a Zero-shot is more of a process problem, with clear logical chains. In some areas that emphasize intuition and connectivity, such as the scientific findings, Transformer produces important generalized examples of exponential decimals.
Einstein discovered "broad relativity" not only because he understood both "narrow relativity" and "Rhyman geometry" at the same time, but also because these two knowledge were simultaneously activated at some point, known as "inspiration arising". The post-Scaling Transformer architecture was enough to make the two knowledge appear simultaneously in a model. But in single reasoning, how to increase the probability of different knowledge being seen and activated at the same time is key to increasing the combined generalization opportunities between different knowledge.
The common source of the Three Sins
First, it is concluded that the above "three sins" can be attributed together to a slightly different layer of memory and reasoning.
According to the academic community's mainstream understanding of the Transformer, the whole connection layer FFN is responsible for memory storage, the attention layer Attn is responsible for combination reasoning. Different layers FFN store different memories, the same Token can activate different knowledge in different layers; different layers Attn accomplish different combinations of different forms, and the same knowledge input in different layers can produce different reasoning results.
Additionally, since there is a recurring characteristic between the layers of the Transformer, the Attn of the current layer primarily deals with knowledge extracted from the FFN of the previous layer.
According to Transformer's tiered memory and reasoning mechanism, we can find:
(1) In reasoning, it is necessary to go through all the layers if you want to review all the knowledge that you have learned in full. Once you have missed the key knowledge, you can only force "pull" out a nonsense in order to gain time, then come back to the first level and get more knowledge;
(2) Knowledge management at different levels of storage is relatively chaotic, with some overlapping and some overlapping relationships. When learning new knowledge, the model cannot judge where it should be inserted, so it can only violently update all knowledge simultaneously;
(3) Knowledge and reasoning between different layers is connected mainly through this narrow pathway of Residual Connection. Even when critical knowledge is activated, such as when these knowledge "meet", knowledge based on the previous layer is almost diluted.
It can be seen that layered memory and reasoning not only limit the efficiency of reading knowledge, but also limit the convenience of storing knowledge, and even more limit the probability of generalising combinations between different knowledge. How to build better memory and reasoning mechanisms could be key to raising the limits of AI model capabilities.
Mobius with a "three" arrow
If we dismantle the connections between memory and reasoning in different layers, extract all of the memory from each layer, store it in a shared pool, and let Attn in each layer have access to this knowledge, then these three problems can be solved:
(1) Assuming that the number of layers of reasoning is the same and that models after shared memory can pass through this knowledge more often, there is greater opportunity to extract critical knowledge and easier to output useful words;
(2) After all knowledge has been flattened and stored, new knowledge is able to "see at a glance" where it should be, with less fear that its inclusion will lead to sudden and full-body problems;
(3) With each shared memory library, all knowledge has the opportunity to be activated simultaneously, which greatly increases the opportunity to "meet" different knowledge directly, and may also lead to better combined generalization.
Architectural layout
With this in mind, the Mobius architecture was designed. Current experiments have shown that Mobius has a direct benefit to 1): Mobius improves its end-to-end reasoning efficiency almost four times as much as the Transformer, and the accuracy of downstream tasks has not deteriorated.
Additionally, Mobius has shown indirect assistance to both (2) and (3) as well. For example, Mobius can achieve MMLU ratios that are close to those of Transformer with 60% of the data, with better data compression rates; in some knowledge portfolio generalization tasks, Mobius has doubled its performance compared to Transformer.
7B MMLU scores from pre-training
35B Continued training downstream mission scores
35B Continued training of reasoning speed
35B Continued training CoT length
Compound generalized verification of Toy Model
In fact, the transformation from Transformer to Mobius has the opposite effect to the original transformation from RNN to Transformer:
(A) The conversion from RNN to Transformer is to convert the "Combination of Recurrent + Residual Connection" between Tokens to Full-Connection;
(B) The transformation from Transformer to Mobius is the conversion of knowledge between the "combination of recurrent + residual connection" to Full-Connection.
Following the completion of the previous generation of transformers, the Transformer significantly improved the capability limits of the base model.
The difference between RNN, Transformer, and Mobius
Understanding Mobius from an academic hotspot perspective
Recently, there has been a comparison of improvements in Residual Connection and Latent Reasoning. If you look at these two angles, you may also find that Mobius originally supported Backward Residual Connection and Dynamic Latent Reasoning.
Whether it is the earliest ResNet or the recent Hyper-Connection, Attention-Residual, their common function is to "transmit information from the shallow layer to the deep when it is forward, and vice versa when it is backward".
As analyzed above, this one-way transmission has a problem: once some critical knowledge is missed, or some deep knowledge is read on the basis of shallow knowledge, the model can only produce some "junk" and recycle it once.
Mobius, in practice, is about storing different layers of knowledge together; but in thought, it can be thought of as introducing a reverse Residual Connection. This residual connection can make the transmission of information more flexible, improving model knowledge compression and reasoning efficiency.
Compared to Token-by-Token reasoning, Latent Reasoning uses continuous embedding to connect two cycles, and also acts as a kind of residual connection. However, traditional Latent Reasoning is ineffective.
First, Latent Reasoning requires that the model passes through all layers at each loop, regardless of whether knowledge of certain layers is no longer necessary. Moreover, multiple iterations of Latent Reasoning are usually done simply to iterate the representation of a Token. In contrast, Mobius originally supported a more dynamic Latent Reasoning mechanism.
On the one hand, since Mobius has access to all knowledge at each layer and there is no restriction on what knowledge that layer must activate, Mobius can not only be viewed as completing a Latent Reasoning at each layer, but can also make more flexible activation choices to reduce redundancy.
On the other hand, Mobius is more than just a token representation in each layer of iteration, so it has higher iteration efficiency and more continuous finite optimization paths throughout the layers.
Mobius's next step is to reverse evolution.
AI-assisted scientific discoveries, world models in collaboration with soft-hard design
Returning to evolution, AI-assisted scientific discoveries and world models are becoming the most important and most pressing frontier applications:
In the current phase of iterative evolution work, there are iterative data, iterative parameters, and iterative architectures... However, regardless of which method of automation is used, it is necessary to build a high-density, highly flexible memory storage and memory expansion mechanism for the sustainable growth of models.
Coding is basically a complete solution to the problem of process, but Research is a generalization of process, intuition, and knowledge combinations. In order to achieve scientific discovery, it is necessary to have enough critical knowledge to uncover the "unknown known", both stored in the model and activated in some reasoning.
Language models are modeled as discrete tokens, and world models are modeled as continuous physical worlds. This world is not only colorful, but also rich in bands and patterns. Compared to language models, world models have a more pressing need for continuous microscopic understanding and reasoning.
It's easy to see that the Mobius design, which separates knowledge from reasoning, accurately echoes the two core needs of recurrent evolution and scientific discovery, and has the opportunity to become a more potent model than the Transformer in both directions.
However, changes to the Attn mechanism may be even more important if the world model is to be more helpful, and these changes will be reflected in future versions of Mobius.
While Mobius at the current stage has led to an improvement in end-to-end reasoning speeds, this part of the improvement is primarily due to Mobius' more efficient reasoning path leading to shorter CoT.
This part of the distance is mainly due to Mobius's separation of knowledge and reasoning in algorithms, which naturally brings with it the storage separation characteristics on the hardware. This characteristic of the visiting storage wall, the communication wall, is the root cause of Mobius's further improvement in reasoning efficiency.
Mobius offers a higher capacity limit, but how to design access strategies more suited to Mobius, parallel strategies, and even better suited hardware architectures for Mobius is key to determining whether Mobius can reach its own capacity limit.
[1]