In March 2016, Four Seasons Hotel in Seoul.
Halfway through the second game, AlphaGo dropped a black on the fifth line. Some people suspected that the program was bugged because, in the common sense of professional chess players, the hand was almost white.
We all know the story afterwards. AlphaGo not only won that game, but defeated Lee Seok-seok with an overall score of 4:1. Afterwards Lee Seok-seok said a quotation: "I thought AlphaGo was just a machine based on probability calculations. But seeing that hand, I changed my mind.
That's the famous 37th hand.
Ten years ago, the man who created AlphaGo himself said something: The ability that the game relies on today's AI doesn't exist.
The man who said this was Thore Graepel, one of the core authors of AlphaGo, a longtime researcher in machine learning. Recently, he made a very patient decision to leave Google DeepMind to start something he thought was more important: installing real reasoning skills into machines.
That sharp view comes from an article he recently published in the MIT Technology Review, titled "Don't be fooled, LLMs don't make sense". After reading the article, Turing Award winner Yann LeCun expressed appreciation, emphasizing that "real reasoning must involve search, and LLM doesn't have that ability".
Why did Thore Graepel say that? Let's come together and see what this article is about.
The 37th hand is not a flash of light.
It's the product of reasoning.
Many people see the narrative of the 37th hand as "the mythical moment of machine intuition" between an AI lightning bolt and a hand beyond the human Millennium Chess Chart. Graepel says this is the biggest misunderstanding of AlphaGo.
To understand this, it is necessary to dismantle AlphaGo's internal structure. It consists of two sets of systems: one is a strategy network ("policy network") and the other is a search engine, which does not care about a single player "looking like no human being", but apparently builds a gambling tree containing thousands of branches, one after the other, which propels each candidate into the future.
The key is that, in the eyes of the strategic network, the 37th parallel is unusual. The probability of this hand coming out from a professional human chess player is only about one-tenth of a million. If only intuitively, this hand would not be selected at all.
Graepel uses the famous framework of the Carnival to explain this. Human thinking is divided into two models: System 1 is fast, intuitive, effortless; System 2 is slow, step-by-step, careful weighing. AlphaGo is just the rare combination of these two models on a machine: the neural network provides "the intuition that this game is playing", "the intuition that this situation is going to win", and the search engine is responsible for testing these intuitions into future counter-attacks.
Here's another often overlooked contrast. In 1997, Deep Blue defeated Kasparov, relying on the rules of human hard coding, to evaluate 200 million chess positions per second, looking forward to six to eight steps. While the complexity of the game is completely different, the value of a game depends on the thickness and the length of the field after dozens of rounds, even if only a fraction of all the changes are counted, and supercomputers can count on billions of years.
Big language model:
A system practiced to the extreme 1
Let's go back to today's AI.
The way the big language model works is to predict the next token over and over again. This is essentially what System 1 is doing: fast, connected, completing surprising patterns in almost all areas of human writing, but it doesn't have the "think again and answer again" link.
Shortly after the birth of ChatGPT, the industry realized that language fluency was not really useful. The remedy is familiar to everyone: let the model respond in a hurry, take the middle steps, dismantle the problem, bring the middle outcome to the chain of thought.
But Graepel points to a fact that can easily be overshadowed by excitement: these intermediate reasoning steps are still the same process of "predicting the next token", but only a few seconds before the final answer is given. It does not introduce a truly independent reasoning mechanism like AlphaGo's search.
He further lists three short boards to explain why what chatbots do is not up to "the kind of reasoning that scientists admit".
First, the model does not have a clear, sustained updated, verifiable cognitive state. It is currently considering what assumptions, how confident it is about explanations, what evidence to weigh, and what issues remain pending. These things should have been systematically revised with the arrival of new information, but there is no such thing as an "open ledger" inside the model.
Second, the model fails to distinguish between "knowing what" and "how to use knowledge". Knowledge and reasoning hang in the balance of neural networks, and there is no independent, explicitly expressed belief system.
The third, and most subtle, is that the chain of thought looks thoughtful, but research has shown that models often fabricate them after things have happened. The answer is to go one way and report it to you another way.
It is indeed falsehood.
Is it that important?
If it's just a chatter, reasoning is probably irrelevant. But in the high-risk areas that we're really interested in -- medicine, engineering, scientific research -- it's just as important to know what conclusions a system reaches and how to reach those conclusions. If we're wrong, for example, in diagnosis and treatment, we need to be able to pinpoint where we're wrong: whether the reasoning process makes a problem, relies on ineffective evidence, or makes a false assumption.
This is exactly why Graepel left DeepMind. He believes that we need a completely new path of machine reasoning, inspired precisely by AlphaGo ten years ago.
AlphaGo has always maintained a record of all its knowledge of the current situation, namely the log tree. The tree contains all the changes it has considered, all the possible future, every hand, every situation that marks the judgment of the neural network. As the game progresses, it keeps updating the tree, ultimately deciding the information in the overall tree.
Graepel argues that general reasoning should be the same: maintaining a cognitive state, clearly expressing which ones are confirmed, which ones are doubted, which ones have been excluded, which ones are still open; and "reasoning" is a series of "actions" that change this cognitive state, leading to conclusions, dissolving problems, and most importantly: deciding what questions to ask next, what calculations to do, what experiments to run.
Of course, open world reasoning is much more difficult than chess. The state of the board is completely visible, the rules are fixed; while in the real world, the current situation is only partially known, the choices are numerous and complex, and the consequences of the actions are random or even completely unknown.
However, the good news is that the advances in LLM and other neural models over the years have provided the right parts to this: LLM can offer pathways to solve problems based on known information and available resources; it can interact with tools through APIs and codes; it can help assess whether an argument is supported by existing evidence. Most importantly, the system must have an independent "arbiter" to evaluate each reasoning move based on "how much uncertainty this step eliminates".
Graepel gives the image a very graphic statement: the scientific method of doping is being used. The goal is only one: to produce knowledge that can be examined.
The larger system is 1.
It doesn't automatically generate system 2.
At the end of the article, he makes his position clear: by making System 1 bigger, we can't have reliable machine intelligence.
The 37th hand is important because a machine maintains a situation, weighs the possible future, and then chooses the hand that connects to its own "instinct" that is most likely to reject the chess.
And human society needs more such "one-handed gods" in these areas of drug discovery, new materials, climate, and medical diagnosis. The board is not exactly like the game of Go, and no one has even handed the rules to us. Such insights only come from a system of real reasoning: to conclude from a verifiable piece of evidence, a chain of deductions and corrected beliefs, rather than an after-conceived, convincing story.
Ten years ago, AlphaGo used his 37th hand to tell the world that machines can surpass human intuition.
Ten years later, one of its creators is reminding us: don't let fluent language fool you, and that real leap forward hasn't happened a second time in LLM.
You said LLM doesn't make sense.
Can you stand on your feet?
After the publication of this opinion piece, there was considerable controversy on X.
The sharpest question comes from University of Toronto professor David Duvenaud (one of the authors of NeurIPS's best paper on neural ODE) who mentions: "Graepel has charged LLM with three counts of unverifiable cognitive state, knowledge and reasoning inseparable, and subsequent fabricated explanations of what it means to be human".
Graepel replied: "You can't reason on your own; it's better to give you a pencil; it's better to follow the scientific method strictly; it's better to be an excellent reasoner".
Duvenaud immediately picked up the strings of the phrase: "So it sounds like we're going to be able to make real reasoning by your definition as long as we put a similar tool in the LLM, isn't that what you're going to do?"
Graepel's "paper pen enhancement theory" has also raised questions from other netizens. Netizens KingBroken notes that paper pencil is essentially an outlet for working memory, allowing you to reason with more data at the same time, but does not "absolutely" alter reasoning ability; however, it does have an added benefit: the written words and numbers are evidence of human "cognitive state checks".
And then the networker, Daniel Grant, asks a sharper question: In other words, human reasoning is the same as "existing models + external systems", so you're building a model that's more capable of reasoning internally than both?
Graepel has not yet responded to these questions.
Other voices felt that the discussion was redundant. Netizens visimo-dino said, "I can't believe anyone else has written this article": LLM has excelled in too many things, the phrase "doesn't make sense" is either ridiculous or irrelevant, and hundreds of similar articles have already been published.
Graepel is not entangled, leaving only three questions behind: are they reliable in unverifiable fields? have they produced strong new scientific theories? do you dare let them determine your treatment plan? the other is also frank, admitting "not yet" but giving credit to existing reinforced learning methods rather than the LLM architecture itself.
Graepel's answer was straightforward: all frontier labs are betting on scaling, and auditable reasoning can't emerge from scaling alone, it needs a different architecture. "Sometimes radical new ideas are harder to implement within large organizations". Questioner Robert Sperry's disappointment is overwhelmed by the statement: "Of all the labs, he expected DeepMind to be the one who struggles the hardest to find answers beyond Transformer".
Critic Katherine Moore left only one sentence: "Language itself is a tool of error, and reliable reasoning cannot be made by it". Graepel took this more radical stance seriously and promptly posed an open question: Does reasoning really require some knowledge that if natural language is too vague, how can it be used to reason in the real world in a formalized language?
The link is as follows: