A few days ago, OpenAI released 722 mathematical paper manuscripts on GitHub. These manuscripts were classified as 372 "family" results across 17 mathematical domains, all derived from an unpublished and unnamed internal model that outperformed the GPT-6 Astra.
A tweet from OpenAI's chief research officer, Mark Chen
Of course, there is still much controversy in the mathematical community about the quality and disclosure of these results, but one thing is hard to deny: the speed at which AI produces research results starts to be calculated as "hundreds".
Almost at the same time, in the latest edition of The Information Bottleneck podcast, a deeply educated veteran said something that caught the minds of many researchers: "After doing this project and getting some pretty interesting results, I suddenly realized: oh, my job will be replaced in about five years".
The podcast address:
He spent nearly a decade at Meta FAIR working on AI, reinforcement learning, neural network explanation, and co-authored the paper Coconut: Continuous Subspace Thinking Chain.
He is now the co-founder of Recursive Superintelligence. In May this year, the company ended its hiding, announcing that it had completed more than $650 million in financing with a valuation of $46.5 billion, led by GV and Greycroft, Inc. and AMD.
In a podcast hosted by Ravid Shwartz-Ziv, Tanaka explains why he started his own business: "Instead of losing his job passively, it's better to start a company that moves in this direction". However, he also gave AI a bit of a cold water in the show. "If you let the model think for itself, it will give you some very flattering ideas, which basically everyone knows, and it's not funny".
In a nearly one-hour conversation with AlexNet at CMU, he discussed AI from "Action Space" to subspace reasoning, culminating in recurring self-improvement, new architecture and open source.
The controversy at Smith Hall: Was AlexNet an accident?
He came to the U.S. in 2008 to study at the Carnegie Mellon University's Robotics Institute. At the time, machine learning was far from what it is today.
His Ph.D. thesis was on non-convex optimization problems in computer vision, such as restoring image distortion caused by water reflection. He used a layered method to optimize and proved that even non-convex problems can still be optimized to some extent as long as there is enough data. "The combination of data drive and optimization is interesting, and I'm still very proud of it".
In 2012, AlexNet was born out of thin air. Tanaka sent an email to Alex Krizhevsky asking him to code himself. He soon realized that the layered processing of the convolutional network, and the idea of layered optimization in his papers, were one thing: "Maybe it's what I've been looking for, just a change of form".
However, the atmosphere at CMU was not friendly. "People came to the office every day and debated in Smith Hall whether AlexNet was a success or a real breakthrough. Most people said it was just a success because it didn't make sense".
When he graduated in 2013, he got an offer from several institutions that wanted to do deep learning, but eventually chose the Google Autonomous Driving Team, which was famous on the one hand, and encouraged by friends on the other. But there, for a year and three months, he was unable to do deep learning because the team was more like a startup company that needed to solve real problems with mature solutions rather than doing scientific exploration. So he jumped to FAIR and called it "one of the best choices ever made".
Two people, one GPU: Tendai's game of chess
In 2015, his team created DarkForest, which beat the top amateur chess players online on a rolling network. A few months later, AlphaGo beat professional chess players in the spring of 2016.
In the face of such "crashes", Tanaka's review suggests that AlphaGo has two things it lacks:
The value network, DarkForest, was a strategic network at the time.
He himself had little background in reinforcement learning before, "basically doing side-by-side learning, like doing reinforcement learning for himself". This experience led him to turn to extensive reinforcement learning research afterwards.
Asked if AlphaGo was surprising, his answer was interesting: the method itself is not surprising, because the Convolve Network captures the chessboard patterns that humans rely on when playing chess, "What's really surprising is that it can beat professional chess players".
In 2018, the team replicated AlphaZero, a program that completely does not rely on human chess tables, from zero self-defence, and made some efficiency improvements, obtaining OpenGo. This AI, in a formal confrontation with the Korean Chess Institute, with only one V100 GPU, left four professional chess players undefeated. "I think that's probably the first time anyone has used a single GPU to beat professional chess players".
The team asked professional and semi-professional chess players to evaluate OpenGo's chess prowess, and the other team said that they could see the opening and then completely misunderstand, only to say, "Looks strong, but I don't know if it's really strong".
Motion space is more important than algorithms
Talking about reinforcement learning in that era, Tanaka believes that there are many methods that are still underestimated. For example, distributed reinforcement learning, which uses distributions of rewards rather than individual metrics to adjust models; and different modeling methods such as Q-learning.
He also looked at non-level optimization methods, and thought that big models gave them new possibilities: "Big models now have a better high-level understanding of the system, of the target functions to be optimized, and this higher understanding is much more useful than local gradient information". When the gradient fluctuates sharply, local optimum is visible everywhere, the gradient can lead to a dead end, whereas non-level methods involve natural exploration, "this is something that the gradient method will never do".
Later in 2018 and early 2019, Tanaka began experimenting with using reinforcement learning in real-world problems, and concluded unexpectedly: "What really matters is not the algorithm, but the design of the action space and the state space".
He used neural search as an example. Human beings know that the depth of the network is critical, but AI doesn't know. If AI starts by searching for details like the size of the first layer of the convolutional nucleus, with almost all branches performing the same way, the value function does not give useful signals, only goes deeper, and the search space explodes. "You don't use any algorithms at this point. In turn, if you can reorganize the operating space, it's important for AI to be "aware of the depth", there will be a clear contrast between deep networks and shallow networks, and several rollouts can be seen.
Coconut: "I never use language when I think".
The co-host mentioned Coconut's paper, in which he participated, that allows models to reason in continuous sub-space rather than token space. He said the idea came from his own observation: "I never use language when I think. I can clearly sense that thinking uses another part of the brain, and then I have to translate ideas into language in order to communicate smoothly with others".
Based on this intuition, the team made a theoretical deduction that the potential representation could be a superimposition of multiple ideas, and used it to reason that a significant advantage could be gained on problems such as graph traversal, "perhaps not on an index scale, but at least with a certain advantage".
So why hasn't the front-end model adopted a continuous chain of thought so far?
Data: A large number of human thought processes are recorded in language, but there is no such data in the underworld.
Potential representation can be highly personalized, with "everyone using their own potential representation", whereas the learning efficiency of large models is much lower than that of humans, with language as a substitute.
Ravid Shwartz-Ziv asks if this is similar to the normalization problem faced by self-monitoring methods such as JEPA. Tanaka agrees and pushes the problem deeper: gradient decline may itself fall into a "sub-level local optimum", which is a good indication that the current training paradigm may not be taught at all, "We don't know where the boundaries are now".
In his view, this is precisely what explainable research means, "like the distinction between alchemy and chemistry". But the current model is so strong that everyone is pursuing results and there is not enough time and people willing to commit to this direction.
After co-writing the final paper on GPT-5.
Talking about the transformation of the big model era, Tanaka believes that his neural network training dynamics analysis, which he has been working on since 2018, is still consistent and does not lead to a rupture. The real impact is on the empirical level: researchers have made their own assumptions, experimented, and concluded that there is no substantial difference between this process and one or two hundred years ago; and as the model grows stronger, they can come up with ideas that "researchers may be forced to think at a sub-level rather than spend time proposing a specific assumption".
At the same time, the status of engineering ability has risen dramatically. Many good jobs in the Big Model era are "half research, half engineering". The situation has changed until the arrival of AI programming intelligence: researchers with solid experience can get the intelligence to rapidly realize ideas, locate critical problems, and "suddenly have superpowers to explore something really deep".
The turning point was his last paper on Meta, and he was the sole author. The paper examined the phenomenon of grokking in neural networks, where the network dies on its hard back, training to suddenly learn to generalize at some point.
Throughout the process, he took GPT-5 as a collaborator: asking it questions, giving it "placement tasks", letting it prove the theories, checking whether the theories are valid, and brainstorming it.
The result is an efficiency increase of about 6 to 10 times, "and this is still a conservative estimate, because I was writing code myself at the time".
So there's the opening line. In fact, researchers usually don't want to go to a small company as a co-founder, because they have to deal with a lot of things outside of research, "but I feel like it's a change now, and if I don't do anything, it probably won't matter later".
The co-host replied half jokingly, his own way of responding was to learn how to work with his hands, unfortunately, "I'm not good with my hands".
AI still doesn't come up with a good idea
However, Tanaka does not believe that humanity can now retreat. "I don't think the current model can give good results without any human intervention". Even the paper co-authored with GPT-5 gave higher thought instead of letting the model itself think.
Humanity is still responsible for:
Start and decide what questions are worth asking.
Verify and judge that AI's solutions are unreliable
Ravid Shwartz-Ziv notes that the recent work of Kaggle, which has been done independently by the intelligent body after training and brushing, has exposed narrow spaces of ideas and problems of over-adaptation to indicators. Tanaka believes that this is somewhat temporary: the model is asked to process details but lacks context, giving it more context, and it may come up with ideas that are more interesting than adjusting hyperparameters.
But he also acknowledged that supplementing the context was only a "temporary fix". The current model is good at repetitive work because the Internet data is full of this kind of content; if the model really wants to jump out of the box, it needs a new architecture so that it can quickly capture new signals with very little data. "We need a big leap from regular repetitive work to the next level".
When asked about Recursive's performance on NanoGPT Speedrun, pressing the training time from 79.7 seconds to 77.5 seconds, whether it was a real discovery, or an efficient search for known techniques, Tanaka replied, "Both -- like adding bigram information to attention -- can be found in DeepSeek, Gemma 3 and even in the STEM papers he participated in Meta, but how to use different embedded tables at different levels, combining these ideas and incorporating new creativity".
Why is it that new architectures always fail when they scale?
Many companies have proposed an alternative architecture to Transformer, which has shown prominence in small-scale experiments, but has never been able to extend to large models. Tanaka's explanation is to attribute bias: when small-scale data is insufficient, additional structural assumptions must be introduced, which works naturally; when there is a large amount of data, the best practice is to let the model itself find the rules.
So how do we verify the new architecture?
He believes, of course, that starting at a small scale is important, but the key is to understand what is really going on inside the network: "We need to know that there is some kind of insight in it that can expand to a larger scale, and that's how we can be confident". Otherwise, it will never be possible to prove that something that works at a small scale works at a large scale, "and that the only result is that the company with the most computing power wins".
The long-term goal in his mind is to find a learning efficiency comparable to that of humans. The human brain can understand things in a very short time using only 230 watts of power and very little data. The field of robotics today emphasizes the need for bulk data, which in his view is only a requirement of the current paradigm: "Look at cats, dogs, these organisms, they don't need that much data and the processors are rough, but they can survive in a complex world where there must be new algorithms.
Limiting self-improvement? "The border is too blurry".
At the end of the program, the host asked about one of the highlights of AI security today: a call to limit the latest models and return to self-improvement for industry coordination.
Tanaka's answer was straightforward: "Honestly, I find it hard to stop people from doing this". To him, the borders themselves are blurred: you discuss new ideas with a programming intelligence, let it evolve, to some extent, to self-improvement; you collaborate with others to write papers that are self-improvement.
He made it clear that Recursive would be open source research. Previous blog posts from the company have opened up all the core and training programs for the outside world to test and develop on this basis. He was more concerned about another scenario: "The worst case scenario is that a small elite group takes control of the world's most powerful model and has everyone else pay a high premium to use it".
He is quite optimistic about whether an open-source model can catch up with a closed-source model. He believes that it is already useful, but the thought process may be a little longer. In his opinion, programming intelligence does not need "AI Newton" or "AI Einstein" to help you write code, as long as the model crosses a line that is available.
The front-end lab has already crossed the line, and the open-source model is also crossing the line, "Once it has crossed, it will be commercialized, and everyone will use the model they like".
It's been 14 years since the debate at Smith Hall about whether or not AlexNet is "lucky". The deep learning that was questioned at the time has now begun to write papers, change code, and record itself. Tanaka chose to stand at the forefront of this wave, and in his words, the next real question is: "How do we use all these models to do better, bigger things?"