One of the people who once believed in reinforcement learning is now convinced that reinforcement does not lead to AGI.
Jerry Tworek was a firm "reinforcement learning maximalist" within OpenAI. After GPT-3, GPT-4, he believed that the last piece of the puzzle missing from the big model was mass reinforcement learning. The rest of the problem may be solved as long as this path is really pushed to the extreme.
Later, he really stood at the center of the experiment.
The model generation is getting stronger, ratings are increasing, and complex reasoning skills are rapidly improving. But another fact Jerry sees is that progress in the lab has not been fully transformed into real-world reliability. Training tasks are becoming more difficult, and models are increasingly being tested, but they are still unable to adapt to the vast amount of ambiguity, change and never-before-defined problems.
This led him to suspect that the problem might not only be that reinforcement learning isn't big enough, but that today's models simply lack a more basic ability: that they won't continue to learn after entering the real world.
With this judgment, Jerry co-founded Core Automation with Rohan Anil. Rohan, who was one of the pre-training heads at Google Gemini and worked for a long time on optimization algorithms, model architectures and underlying computing systems.
However, they decided not to continue coding along these two routes.
As the entire industry continues to expand Transformer, increase computational reasoning, and pursue stronger programming intelligences, they choose to ask a more fundamental question: If models can only learn in the lab, and require humans to constantly collect data, retraining and releasing new versions, can they really move to AGI?
In a recent conversation, the two founders of the System explained their judgments about the current AI path: Why isn't massive reinforcement learning the final piece of the puzzle leading to AGI? Why isn't learning from experience the same as reinforcement learning?
The address of the video:
The model generation is getting stronger
The problem is outside the lab.
Reinforced learning does make the model stronger. It improves complex reasoning and task execution, while also enabling the model to master more and more predefined workflows.
But Jerry gradually realized that training tasks and standard evaluation are often just "two sides of the same coin": researchers design training data based on the model's ability to master, and then test the results of similar tasks. As long as training continues, the model can naturally improve on these metrics.
The real world does not have such clear boundaries. The user's problems are often more blurred, the environment is changing, and new combinations of tasks are formed that are not covered in the training data. Even if the research team finds it difficult to find a challenging enough training task, it does not mean that the model already covers the entire complexity of the real world.
This exposes the fundamental contradictions in the current way of training: the model is in the laboratory, based on pre-collected data, well-established environments and feedback to complete the training; after the training, it is deployed to a more open, more chaotic world.
Reinforcement learning enables a model to better accomplish a task that the researcher has already defined, but it cannot guarantee that it will adapt itself to problems outside the training distribution. Whenever a model encounters new failures, humans still need to collect cases, supplement data, and send it back to the lab to complete the next round of updates.
Therefore, the real task of learning is still the laboratory, not the model itself.
Reinforcement learning has not failed, but it makes the missing part clearer: the next generation of models must not only grow stronger during the training phase, but must continue to learn after entering the real world.
The third possibility of "learning"
In Jerry's view, the most fundamental problem with the current model is not that we don't have enough knowledge, but that learning happens almost exclusively before deployment.
Pre-training is responsible for absorbing bulk data and reinforcement learning is responsible for shaping the model's behavior. After completing these steps, the model is handed over to the user, but the parameters are basically fixed. After this, even if the external environment changes, new tools, code libraries or tasks appear, the model will not naturally update itself as humans do during use. It may be temporarily adaptable, but it is difficult to transform these new experiences steadily into long-term capabilities.
This means that the value of the model will always depend on whether the training data covers the real needs in advance. When faced with unprecedented problems, it is still common practice to collect failures, supplement data, and then re-train them in the laboratory.
Existing systems are not entirely adaptable after deployment. The first method is in-context learning: the user writes new information into a cue word or context, and the model can use that information in the current conversation. It does not need to modify parameters and is not easy to cause catastrophic forgetting, so it is relatively flexible.
But this "learning" capacity is limited. The context window is longer and can only store a certain amount of information, and once the content is compressed, deleted or opened a new session, the experience gained by the model can be difficult to maintain. Jerry uses the programming tool as an example: after a period of use, the context needs to be sorted or compressed.
The second approach is to continuously fine-tune, which is to constantly update the model parameters based on new data. It looks closer to real learning, but brings with it another set of problems. When models absorb new knowledge, they may disrupt previously acquired capabilities, which is disastrous forgetting; at the same time, fine-tune tends to require large amounts of data and computation that cannot be quickly adjusted based on a small amount of experience as humans do.
Therefore, the two current paths have clear limitations: up and down literary studies are flexible enough, but difficult to overcome over a longer period of time; fine tuning can change models, but not stable enough, and not efficient enough. Researchers have long tried to solve these problems, but so far have not found a simple and reliable solution.
Jerry hopes to find a third possibility: to do self-learning, to write "how to learn" itself into the model architecture, so that the system can be continuously updated based on users, data, and real-world tasks during deployment, while avoiding easy forgetting capabilities.
This is why he points to architecture rather than just training scale. A model that needs to be repeatedly re-trained back to the lab can become an increasingly powerful tool but still lacks one of the most important features of an intelligent system.
Learning from experience is not like reinforcing learning
If models need to continue learning in the real world, then a natural question is whether reinforcement learning can do the job.
Jerry's judgment is that reinforcement learning is important, but it's just a way to "learn from experience". It usually lets the model try to act in the environment, get rewarded based on the results, and then re-adjust behavior through a large number of iterations.
However, current reinforcement learning often relies on a large number of parallel experiments. Researchers allow models to experiment with different programs at the same time in many similar environments and compare results to reduce uncertainty in training. This method makes effective use of computational resources, but it is not exactly the same as the human learning experience. People often cannot access thousands of parallel worlds at the same time, nor will they make large-scale trials on each problem.
What's more, humans have different ways of learning when faced with different tasks.
When learning to play, people repeatedly try to micro-modulate depending on whether the ball is flying in the desired direction. This process is very close to reinforcement learning: action, feedback, correction, action again.
When learning mathematics, the process is completely different. One may repeatedly iterate through reading definitions, comparisons, concepts in one's mind until the dispersed information is connected. There is no explicit external reward, and there is no need to repeat thousands of moves, but learning happens the same.
Both processes belong to learning from experience, but rely on different mechanisms. The former is closer to trial and error, the latter closer to understanding, compressing, and reorganizing knowledge.
This also explains why the mere expansion of reinforcement learning may not be enough to solve the problem of continuous learning. Experiences in reality are not always organized into clear actions, rewards, and reversals. Some feedback delays are severe, some tasks do not have the only correct answers, and some learning comes from observation, interpretation, and internal thinking rather than external rewards.
Therefore, the so-called "experience era" is not just about putting more computational power into reinforced learning. The real challenge is to find a richer, more efficient way to use experience: models can adjust behaviour from success and failure as well as form a stable understanding from a small number of cases, new knowledge and long-term interactions.
In Jerry's view, computational resources in experiential learning today are unprecedented, but reinforcement learning will not be the end of this path. The next breakthrough may come from a broader learning algorithm that no longer requires all experiences to be converted into standard rewards, but can be absorbed in the right way based on the structure of the experience itself.
What the next generation of AI architecture needs to be capable of
In Core Automation's view, the next generation of architecture doesn't have to completely abandon Transformer, but it has to address the unresolved problems of Transformer.
One of the most important of these is to turn learning from a one-time training process into a part of the long-term operation of the system itself. The model needs to be able to absorb new information after deployment, adapt to user and environmental capabilities, and transform short-term experiences into long-term changes. It cannot rely solely on limited context, nor can it retrain new learning tasks every time it returns to the lab.
This kind of learning must also be stable enough. The system must both master new knowledge and not easily destroy original capabilities; it must also accumulate experience from long-term interactions and be able to update quickly from a small amount of data. Continuous learning is difficult to do in a real environment if each adjustment requires massive sampling and computation.
Jerry therefore suggests that researchers may need to express "how learning happens" at an architectural level. This means that the model should contain not only the knowledge that has been learned, but also an internal mechanism that can handle new experiences. It needs to decide which information should be retained, how to connect to existing knowledge, and when to change the original behaviour.
Rohan adds another requirement in terms of computational efficiency: the new architecture must be designed with pre-training and reinforcement learning as a whole.
The current training process is usually to build a basic model with pre-training and then to increase reasoning and task execution through reinforcement learning. But the optimization goals and calculation methods of both are not the same. Pre-training is good at using large amounts of data for parallel calculations, and reinforcement learning requires the generation of trajectories, comparative results, and higher training scores. Simply spelling the two out front and back is not necessarily the most effective solution.
Therefore, the next generation of architectures cannot perform well only on pre-training indicators. Researchers need to start with the final task of considering how the model builds a knowledge base, how to allocate reasoning calculations, and how to continue learning through experience.
Rohan emphasizes that an architecture that is theoretically superior but cannot run efficiently on real hardware has no real value. From research concepts, to training methods, to computational kernel and production deployments, a complete end-to-end system needs to be formed. Core Automation places architecture, optimization and systems engineers in the same team in order to avoid different links being cut by organizational boundaries.
From this perspective, what they are looking for is not a single new module, but a new set of computing and learning methods: one that can continuously adapt, use data and computing more efficiently, and run in the real world for a long time.
Another bottle of Transformer:
Calculating depth
In addition to the inability to continue learning, Rohan believes that Transformer has a problem that is easily overlooked: insufficient computational depth.
The "deepness" here is not just how many parameters the model has, but how many steps the input information can go through in a continuous computation before it produces an answer. The current Transformer, although known as the deep learning model, is usually only about ten to a hundred layers. After information passes through a fixed number of network layers, the model needs to start producing results.
This structure is very suitable for parallel training, but limits the model's ability to perform complex derivations in single-step forward computation. Faced with the need for multi-step thinking, the model cannot increase internal computation at will, relying only on fixed network depths.
In recent years, the extension of the chain of thought and reasoning has partially bypassed this limitation. Models no longer give immediate answers, but master a series of intermediate steps. Each addition of a word is equivalent to adding a new round of computation, so the model can gradually process the problem on a longer path. Reinforcement learning can further train the model to learn when to do longer reasoning.
But the cost of this approach is obvious: the model must take one token at a time to generate a geographic thinking process. The more difficult the answers, the longer the content generated, and the more time and cost required to reason.
Rohan sees this phenomenon as an architectural patch. The fixed computing depth provided by the pre-training phase is insufficient, and reinforcement learning is used to add computation by generating more tokens. The industry subsequently improves the speed of generation by speculating on technologies such as decoding, but these optimizations do not change the basic way in which the return model generates a token at a time.
Thus, the question is not just how to make Transformers produce faster, but whether an architecture can be designed to accomplish deeper calculations in fewer steps. The model may be able to explore multiple directions simultaneously internally, determine how much computation is required by dynamics, or organize reasoning processes in a form different from that generated by text.
In Rohan's view, simply raising the question of "how to increase computational depth" opens up many new directions for research. Transformer efficiently solves the problem of how to expand model width and data size, but the next stage of breakthrough may depend on whether the system is more flexible and efficient in deep computing.
How much room is left for the Transformer?
Considering that Transformer is not the end, it doesn't mean that it can't continue to progress.
Jerry believes that pre-training can still compress more knowledge, relationships, and patterns into the model, and reinforcement learning can continue to be added to the new workflow. As long as a task is clearly described and enough training data is available, it has the opportunity to be written into the Transformer.
As a result, the limits of the Transformer's capabilities remain high. It may cover more and more knowledge-based tasks and perform more complex processes through tool calls and smart body systems.
But this improvement has clear boundaries: models are good at mastering the capabilities that have already entered the training process, but they do not automatically adapt to changes that occur after training is completed. Even if research institutions can record a large number of human tasks and incorporate them all into the training environment, the real world will continue to change. New software, new rules and new needs are constantly emerging, and the knowledge and working methods of static models will age with it.
In other words, the upper limit of Transformer is not that it stops improving suddenly one day, but that it increasingly relies on an external update cycle. The model is unable to discover all of its flaws, or to independently collect the required experience, modify the training process and verify new capabilities. Research institutions must constantly train new versions to keep up with the world.
Economic factors will keep this route going for quite a long time. Transformers are not only powerful, but they have proven commercial value: the cost of training models can be repaid by product revenues. In contrast, other architectures, even if theoretically feasible, may not be able to produce good enough products at the same cost.
There is still a lot of room for engineering optimization in existing architectures - scarcity, hybrid expert models, deductive decoding and more efficient reasoning systems - that can reduce training or deployment costs. More computing and data will continue to improve.
So, the view of Core Automation is not that Transformer is about to go out of business, but simply to expand Transformer and ultimately only push the existing paradigm further. It can be a powerful tool that covers a wide range of applications, but it is difficult to achieve the ability to learn autonomously after deployment with scale growth alone.
How far the Transformer can go depends on how much training data, computing and engineering resources humans are willing to invest; and whether AGI can emerge depends on whether the model can gradually get rid of its dependence on external update mechanisms.
Why did you leave the big model company?
If a new architecture requires a large amount of computation and a complete engineering system, the most natural research site would seem to be a resource-rich large model company.
Jerry believes that this is the right time to explore alternative routes for independent start-ups.
This is primarily due to market competition. The main laboratories are competing fiercely around the speed of model release, product capacity and user size. Transformer has been proven to be able to continuously improve and generate revenue, so the most reasonable short-term option for companies is to continue to expand existing routes to ensure that the next quarter or round of release is not delayed.
In such an environment, an architectural study may take a year or more, with no definite results, and difficulty in attaining the same priority. Even if researchers recognize a direction worth exploring, they still need to complete the training and release of the next generation of models.
Smaller labs are also influenced by the parent company. When leaders focus on developing reasoning models and programming intelligence, other teams often choose to follow a proven direction rather than taking the risk of opening new routes. As a result, the industry as a whole has a large number of competitors who may be solving similar problems and releasing similar products.
Core Automation wants to fill this gap: focused research may not prevail in the short term, but it may redefine the boundaries of model capabilities.
Rohan's entrepreneurial motivations also included organizational approaches. He argued that new architectures should not be confined to papers or small-scale experiments, but should be run efficiently on real hardware. This required architectural research, optimization algorithms, training systems, and the underlying computational core to be advanced simultaneously.
Large companies have complete resources, but the teams are often divided into different departments, each working around set goals and release cycles. Small start-ups can build teams from scratch to meet the needs of end-to-end research, allowing people from different professional backgrounds to face the same problem together.
Their choice was not to deny the ability of large laboratories, but to judge whether different organizations were suitable for different types of research. The parent company continued to push Transformers on a larger scale, while independent laboratories had space bets on alternatives that had not yet been verified.
For Core Automation, entrepreneurship itself is a form of research design: creating space for long-term architecture through smaller teams, shorter decision chains and different priorities.
How does Core Automation build an automated AI lab?
Another core goal of Core Automation is to build highly automated AI labs.
Automation here doesn't mean moving researchers out of the process immediately. Jerry is more willing to understand it as expanding everyone's mobility: walking, cycling, and driving all require directions, but tools are different and distances can be reached at the same time are completely different.
In the past, researchers had to write code, configure experiments, process errors, analyze results, and then design the next round of experiments themselves after coming up with an idea. Much of the time was spent executing the link rather than deciding which problems were worth studying.
With the development of programming and research, a researcher can advance more tasks at the same time. Intelligent bodies can generate code, run experiments, organize data and assist with screening, while humans focus more on making assumptions, interpreting results and adjusting the research direction. Core Automation hopes that this new way of working will revolve around the design of research processes from scratch, rather than simply inserting intelligent agents into old organizations.
The key change brought about by automation is to increase the speed of experimental iteration. Architectural research typically contains a huge amount of search space: there may be multiple options for each computational module, training method, and system design. A single experiment does not necessarily explain the problem, but the real determinant of how quickly the team can complete the "cycle of proposing ideas, running experiments, getting results, forming new hypotheses".
Jerry mentions that even if an efficient architectural experiment can be performed on a steady basis every day, the speed is already quite remarkable. In the future, if ten or even hundreds of experiments can be performed every day, the research process will be closer to large-scale search and optimization. Teams can quickly eliminate inefficiencies and focus resources on better-performing solutions.
This kind of laboratory requires not just a general-purpose chatbot, but a set of intelligent systems capable of understanding code libraries, managing algorithms, tracking experiments, comparing results and continuing tasks. It also requires a complete automation infrastructure to allow the results of the experiments to be directly transferred to the next round of research.
Core Automation itself is seen as an organizational experiment: a small team, backed by intelligence and deeply automated processes, can actually accomplish how much research has been done in the past?
They do not expect AI to complete all scientific discoveries independently in the short term. At this stage, the goal is to give each researcher a greater research leverage to explore more architectural directions under limited team and computational conditions.
Laboratories for automation are both their way of improving R&D efficiency and their technological trajectory. The team tries to make research processes have a stronger continuous learning capacity, while also looking for an architecture that allows models to continue learning themselves.
How they define AGI
A common definition of AGI in the industry is: a system capable of surpassing humans in most economically valuable jobs.
Jerry believes that this definition still misses a key question: Will the system maintain and expand its capabilities if humans stop training new models?
A model, even if it can accomplish a large amount of work, is not detached from the human-dominated improvement cycle as long as it relies on humans to collect data, design training tasks, fix defects and release new versions. It may be an extremely powerful automation tool, but it does not necessarily have a true universal learning capability.
Jerry's criteria are much stricter: AGI should be able to improve itself without human involvement.
This definition does not conflict with "finishing economic activity". Training and improving AI models is a complex and economically valuable task in itself. If a system can cope with a wide range of human tasks but cannot afford the task of improving intelligent systems, there is still a gap in its universality.
The best form of performance is still the combination of humans and large models. Models are responsible for generating, searching and executing, humans are responsible for setting goals, judging outcomes, finding errors and adjusting directions. This collaboration has been able to create high value, but in many tasks, the reliability of the system will significantly decrease once humans are removed.
Jerry said that, at least from what he observed in 2024 and early 2025, continuing to expand the model along existing routes would not naturally eliminate the role of humans in the cycle.
Therefore, the AGI pursued by Core Automation is not just a more knowledgeable, more rational model, but a model that can observe its own performance, gain experience from real environments, and continually change its own systems.
According to this standard, the arrival of the AGI will not be announced by a single score. A clearer signal is that the system is beginning to undergo a cycle of improvements completed by human researchers in the past, and that it can adapt to change, detect deficiencies and acquire new capabilities even if no laboratory continues to prepare it for the next round of training.