This is one of the most important achievements in AI.
This Friday, OpenAI can explain an extended article by team founder Nick Cammarata that shocked the AI circle. He said that using modern smart body tools, large models of trillions of parameters are also almost readable, and that our view of neural networks as "black boxes" has proven to be misunderstood, that most of the expected barriers do not exist, that explainability will be solved, and that it may soon be realized.
This could be an explanatory "bitter lesson": simple things, on an absurd scale, can work.
We can train a powerful AI, but we don't necessarily know why it responds that way. If we let AI help study the "brain circuitry" of neural networks, can we open this black box?
Although Nick's article is extremely long (it takes 65 minutes to read), some researchers have already read about it. To put it simply, neural networks are not entangled; pluralism is wrong. You can directly study their own neurons without having to "open" them with a second model.
This work could ultimately enable us to deliberately design large language models and shape how they behave.
Let's see what he says about it:
In the past few years, there has been a frustrating premise in the field of AI explanation: neural networks are a dead link. When the number of neurons is not enough, models fold unrelated concepts into the same neuron because they don't interfere with each other. To read it, you have to train a second model (such as the rare self-encoder SAE) to solve it. This hypothesis, known as superposition, is the basis of the entire security research route.
Now the man who had touched the ground with his own hands stood up and said, 'We've seen it. The neural network would have been clean.'
"Cats and Cars" may never be the same thing.
The explanatory mechanism tries to track neurons and calculate circuits to explain how the model gets answers. One of the difficulties is that a neuron often responds to things that seem unrelated. A classic example comes from the visual model Inception V1: the same neuron recognizes both cat faces and car faces.
An important explanation in the past was "superposition": the model's space is limited, so it squeezes unrelated and unlikely simultaneous concepts into the same set of neurons. Researchers have therefore tried to break down these characteristics into understandable traits using tools such as scarcity coding.
Nick now believes that this famous case is wrong.
After re-examining the picture and the connection, he found that the cat's face and the front of the car share one characteristic: V-shaped. This neuron is also associated with V-shaped flames, fox faces, and circles that push it in a negative response. In other words, "cats and cars" are human-given classifications, and what the network finds may be shapes that transcend these categories.
Neurons don't have to be names to label objects; they can also be verbs to distinguish patterns and send information to different downstream circuits.
The chaos in our eyes is that sometimes only models use classification that we didn't expect.
The truth of polytheism: forked like branches
While other neurons do have multiple uses, the authors argue that this is not necessarily the result of random mixing of unrelated concepts.
Nick proposes "recursive generalization": neurons start from a pattern and grow into related branches, branches that continue to grow. A neuron that recognizes "iy" can expand to "diy" and then expand to "do it yourself" completely without "iy". A geological neuron branches into a geographical JavaScript library called Cesium. This does not hide information in high-dimensional space, but grows in contextually contiguous realms.
These responses may appear to be loose only after the training is over; tracking the training process shows how they are connected.
The team found that a unit identified by the automated interpretation tool as the "polynomial comma neuron" was actually related to the state of Colorado: it responded to information such as local telephone area codes, voicemail, etc. It also enhanced the model's tendency to plan hikes. The comma merely summarized the model's location in the previous context.
In 2023, OpenAI's self-explanatory project found only about 1.7% of the neurons that could be interpreted using the loosest criteria, and the authors now estimate that about 90% of the neurons, positive and negative, each have a major task, about 7% have two related tasks, and about 2% are not yet understood. This depends on the activation threshold, and how to define "a task".
The confidentiality model and the MoE model came to the same conclusion, and the authors on the Kimi K3 obtained the same result: they mapped the first known trillion-parameter model on the Kimi K3 to the question of wallets, wallets, cash, documents connected to lost receipts and honesty, ultimately producing a "return".
Let AI scientists explore the inner workings of AI
Another main line of this work is that intelligent agents have changed the scale of research.
AI can examine neuronal samples, make explanations, track downstream connections, and then design experiments to find counter-examples. Researchers also had the intelligent body examine 100,000 attribution images, and the authors themselves examined thousands.
How can we prevent AI from making up a story that sounds reasonable?
One test was: the first intelligence body studied neurons and wrote explanations; the second only looked at explanations, generating a new sample that was expected to activate them strongly, and then handed over to network measurements. The authors reported that some samples had an activation intensity greater than all the examples they found from 10 billion token contents.
The explanation thus becomes a hypothesis that can be overturned by experiments.
Cammarata also proposed an anti-intuitive insight: understanding more networks sometimes makes the rest easier to understand. Like a typewriter game, the more adjacent answers, the more restricted a space can fill.
This led him to think of his own "bitter lesson" in the field of explainability: simple observations, assumptions and tests, on a sufficiently large scale, perhaps more effective than sophisticated dismantling methods.
It's still not a declaration that "explainability is solved". There's a lot of work to be done on the attention mechanism, the full model audit, and how to translate understanding into reliable security improvements. But it provides a direction that deserves serious scrutiny: If there are traceable organizations within the neural network, future breakthroughs may come from a lab run by AI scientists, continually observing models, asking questions, and verifying answers one by one.
References are given below:
© THE