Google has measured 4.5 million times: the key is lost.

  It's the first time in the history of the world.

The name is obviously on his lips.

You see a familiar face, but you can't remember his name, and psychology has given this phenomenon a name: the tip-of-the-tongue phenomenon.

Now, in one study, Google has found that GPT-5 and Gemini 3 have the same problem.

These two models incorporate 95 percent and 98 percent of the test facts into the parameters. But if you ask them directly, they have 26 percent and 34 percent of the facts they can't answer.

If you start thinking for a moment, you don't even think about the remaining 11 percent or 12 percent.

The big model can't answer, and in many cases it's not in its brain, it's learned, it can't get out.

This study is called Empty Shelves or Lost Keys? and is included in ICML 2026.

On August 12, the Google Research blog focused on the results, while a set of benchmarks and a complete data set called WikiProfile was released to the public.

The model answers the wrong question, possibly with two problems.

One is that the roots are not learned, which has to be added to the data and the parameters before the training; the other is that it is learned, and another question method can not be adjusted, which is related to post-training and reasoning.

Both of these problems have been summarized as a question of accuracy.

Google's "knowledge image" framework, which no longer asks "is the answer yes or no" but "what is the state of this fact in this model's memory", is divided into five categories.

"Knowledge Portrait" divides a fact into five states: input failure, memory failure, direct memory, through thinking memory, and un input based on reasoning.

The same wrong answer, in different states, changes completely differently.

How do you judge whether a fact exists or not?

The way to do this is to have the model written in sequence before the answer appears; or to ask questions directly after the same context.

Both of these questions are not thought-provoking, because of the separation between "remembered" and "pushed out".

The left side of the test is not stored, and the wiki original is interrupted to complete the model; the right side of the test is unanswered, and the wording is changed, and repeated questions are asked in the opposite direction.

The index is called WikiProfile, with a total of 2,150 facts, all taken from the English Wikipedia.

Each fact is presented with 10 questions: 2 are not stored, 4 are left alone, and 4 are given choices.

Generated by Gemini-2.5-Pro, Google searches for question-by-question verification, the answer is not unique or the title is ambiguous, and the entire fact is discarded.

There were 13 models, covering the four Gemini 3, GPT-5, GPT-4.1, and Gemma 3 families, with each model switching thinking over and over again, and each question sampling eight times, yielding about 4.5 million answers.

It would cost about $500 to test a complete front-end model.

The result is the first set of numbers.

In the error of fact in the front-end model, "can't get out" has overtaken "didn't learn" and become a big head.

The first is cold knowledge.

The mainstream explanation in the past is that the capacity of the model is insufficient, and some cold-blooded facts have not been learned.

Gemini-3-Pro has a 94.5% Cold Facts, 99.5% Hot Facts, and only 5 points.

In response, the cold facts dropped to 63.3%, the hot facts to 84.7%, down 21 points.

There was almost no difference in time, almost four times as long as it took.

Cold Facts (the last 20%) and Hot Facts (the first 20%) are relatively small, with a significantly smaller gap in direct memory.

The GPT-5 has a larger gap.

Cold facts it survives at 90.7%, hot facts 98.4%... but openly, cold facts only remain at 52.9%, and hot facts at 77.8%...

Almost half of the cold-blooded facts clearly lie in the model and cannot be corrected.

The truth of the long-tail question is not that the model has not seen those cold-blooded facts, but that knowledge is more difficult to awaken.

The second scenario is that you ask AI who Tom Cruise's mom is, and it answers quickly and accurately.

In 2023, the paper called it a "reverse curse", and the model learned that "A is B" and does not automatically generalize to "B is A".

Google changed the same set of questions to a four-choice one, and the result was immediately reversed.

The recall rate for GPT-5 is 82.9%, down to 74% in reverse. If the identification questions can be changed to options, the reverse is not only more difficult than the reverse, but 9 of the 13 models do better and the remaining 4 are even.

The selection will be done, and the question will not be answered once it is changed to a question.

The same set of reverse-oriented problems, reverse-oriented in four-choice identification is significantly more difficult than reverse-oriented in difficulty, and reverse-oriented in closed-volume generation.

This explains how the link is obviously there, and how to dig it out.

The author explains that the context, the wording and the sequence remained the same when the facts were entered.

The way you ask is the way you turn away from training and the key is unlocked.

The questions on the WikiProfile are one question at a time, with no logical steps in the middle.

"Which club did Oasis first play at?" is a theme you may or may not remember.

It's a good idea to give the model a little more time to think about it, but it doesn't help.

But thinking really came back. The fact that it didn't answer those who went in, it got back 40 percent and 65 percent.

The key clue lies in its preference: the fact that the roots don't stick in, thinking can only get back 5% and 15% of the time.

The percentage of fact that is recovered after starting thinking is 40% and 65% of the data stored in the parameters, and only 5% and 15% of the data that is not stored.

It's not a game of skill, it's a game of finding the key.

The authors have also bluntly ruled out a more conservative explanation.

Some people might think that thinking is more about making the model flourish and increasing the chances of getting one in eight. If so, the answer should be more accurate.

In reality, the number of times the same question is answered eight times is more concentrated.

The two places that are most profitable are the two places that are most crowded in front.

The Gemini-3-Pro's cold-temperature gap narrowed from 21.4 to 12.5 points, while the GPT-5's positive reverse direction gap narrowed from 9 to 2 points.

Another Google sister paper, Thinking to Recall, explains the mechanism:

One is the computational buffer, which spews out thought tokens that give the model additional implicit computation, even if the content itself is meaningless.

The second is the start of the facts, the model tells a set of related facts first, which is equivalent to building a semantic bridge to the correct answer.

This is called "spreading activation" in human cognition.

If you can't remember a person's name, remember what company he was in, what type of hair he left, and when you last met him, the name often comes up on its own.

The way to get rid of the "tongue-point phenomenon" is that people and models are the same.

But there is also a price to pay.

The same paper warns that if the middle facts in the chain of thought are composed, the illusion of the ultimate answer is more likely.

The bridges are twisted and the farther we go, the more unspeakable.

Gemma 3 is an open-source family measure.

Parameters ranged from 1B to 27B, with coding failure rates dropping from 85% to 23%.

The scale is indeed filling up, and the shelf is visible to the naked eye. Memory failure does not decrease at the same time. It increases in proportion to the remaining errors, becoming the main component of the error when it reaches 27B.

It is more extreme on the front-end model: memory failures account for more than 70% of all GPT-5.2 errors, and the stronger the model, the higher the proportion.

The five-state distribution of 13 models is that the larger the parameters, the more the code fails to shrink, the more memory fails to move.

The scale improvement is how much is saved, not how much is made.

This did not condemn Scaling to death, but instead marked a boundary for it.

Since the input ratio has touched 95% and 98% of the ceiling, continuing to add parameters and data, the convertible accuracy rate improvement is becoming less visible.

The next increase is more likely to come from post-training and reasoning methods, namely how to use the knowledge already stored.

I'm not sure if I'm going to be able to replace it.

But the authors also point out that the knowledge in the parameters is still relevant to flow density, response speed and cross-context integration, and is difficult to completely replace by external retrieval.

So when should we let the model think for a second?

The paper defines the problem as a metacognitive one, i.e., a model's judgement of its own state: it must first realize that "I can't answer directly" before it knows the question is worth thinking about for a moment.

Over the past two years, the default answer to making models more reliable has been to build more data, build bigger parameters.

The study made it clear that, on a factual level, the front-end model's cargo has almost been lifted.

Even more troubling is the next step: it has to know it can't think of it.