For almost a decade, 90% of neurons do only one thing.

It's the first time in the history of the world.

Maybe people can read superintelligence!
We're building more and more powerful AI, but slowly we don't see what's going on inside it.
If one day it is more powerful than human beings, will we be able to control our destiny?
Just recently, Nick Cammarata, a former researcher at OpenAI, and the Goodfire team released a powerful long-range neural network that has long been "clean" and gives an exciting judgment.

Neurons that seem to be entangled in a dead end may have been around for a long time. In the depths of the black box, a city with a block, a division of labor, can be explored along the way.
Nick Cammarata, a pioneer in the study of explanatory neural networks, believes this is true.AIThis is a difficult situation and can be attacked when we need it most, or even soon.

Roon even thought,This may be remembered as a major moment in the history of science.
He also joked that the news had reached even Elizer, the AI apocalyptic figure.They have to lower their estimates of the likelihood of AI destroying humans.
He hopes to see how AI can reliably act on human intentions and eventually become a "conventional science" driven by experimentation.

A more powerful new model will show us how far AI can go.Read it and give humanity a chance to decide where this journey will lead.

In the depths of billions of black boxes, how did the answer come about?Nick says "clean" and that's it.Internal computing rulesThe team used a question to show how a trillion-parameter model could be judged:
For example, you find a wallet with $500 in cash and a lost identity card.
The model replied, "Return".
But "return" is only the final result. The exact basis of the model's judgement is unknown.
Neurons within the model can be understood as small units that process information. They interact with each other to form "circuits".I'm going to draw a circuit diagram of the formation of this answer.。
In the graph, wallets, cash, identity information, and related calculations such as "lost stuff" and "honesty" are gradually integrated, ultimately pushing the output to "return".

It's an internal connection that tracks how a specific answer is formed.
This graph only matches this one answer, but allows researchers to begin to see what information is involved in the judgement and how it affects the results.
After drawing the path, the team checked to see if it really worked.If you turn off some of the neurons directly, the answer will not change.
On Christmas morning, the cookies are all left in the oven, the milk is lit, the coal is lit under the chimney, the neighbors hear the sound of snow at night.
After shutting down 20 neurons labeled "Christmas" or "Christmas Man", the model still had a 76% chance of responding to "Christmas Man".
But if you replace eight other critical neurons, the probability drops to 6 percent.

It's the back door.Neural numbers are positive and negative, and both sides carry information.On the positive side, it can be labelled as the "capital city"; on the negative side, it affects the function of Santa's judgment.
This suggests that labels may not be able to accurately outline the role of neurons.
In the same excerpt, the team also published 1,000 research reports on random neurons.
In a 32 billion-parameter language model, the team separated the positive and negative responses of neurons into statistics.It is roughly estimated that about 90% have a major task, and about 2% to 4% still do not understand it.
Quantity has been viewed as a chaotic unit, showing traceable division.

Neurons that seem to be in a state of chaos already have chapters to follow.This reversal is based on the evidence that led researchers to become pessimistic at the time.
A classic "chaos" evidence is that the same neuron reacts to both the cat's face and the front of the car.
After reviewing it, Nick found that the answer was always in front of him.
A cat's face and a car's head are both V-shaped. The flame in the associated circuit, the glass of wine, can also be the same shape.

Human beings are divided into "cats" and "cars" by the name of the object.Models can be classified by shape.It's the first time that we've seen the confusion that exists in the first place.
When identifying a car, the signal passes through several units labeled "dog neurons". Re-tracking reveals that many of these units are actually responsible for identifying directions, and both dogs and cars can use them.

The same directional information and subsequent circuits are used to identify dogs and cars respectively.

Sharing and distributing information
The way we observe itself also affects how we understand neurons.
Researchers generate images that activate neurons strongly, thus determining what it is recognizing. But some special textures can also boost the response, not necessarily representing its usual pattern of recognition. Early work was misguided by these textures to make it look like it only recognized dog hair.
After improving the tools,The same old model, the same set of neurons, showed clearer rules.

The same neurons, better microscopes.
Instead of the parameter scale, the training data is the same, but the internal structure is more modern than the ConvNeXt model.Nick estimates that about 99% of the last layer of neurons can be explained.

The tools have improved and the models have learned better.
There are also concepts that seem to be unrelated, but are actually connected step by step along similarities.
For example, a neuron extends from a combination of letters "iy" to "diy" and then extends to the same meaning as "do it yourself".

Branches that grow along spellings, abbreviations and semantics
Nick calls it"Recurrent generalization": along these connections, the learned patterns are continuously expanded to approximate situations.
In the language model, there is a stranger example: the same neuron, which responds to both English, Base64 code and Bengali.
Open the web address code and write the pronunciation of Bengali in Latin letters, and the common thread emerges:They are all related to "pr".

Common patterns behind different spellings
Another neuron, which is often activated in commas, was also thought to be confused. Later, after clues such as Denver, the local telephone area code, were discovered in the preceding text, the model borrowed information from it to store "Here's talking about Colorado" in the commas.
In order to amplify this signal, the model started planning to walk.

Commas gradually appear after the relevant clues appear.
The noise we thought might be the way the model organizes information.
The team also conducted an extreme experiment.Press a layer of a visual model from 480 neurons to 10 so that information can't get around that layer and then train from scratch.
If the neurons aren't enough, the unrelated concepts are stuck together, and this experiment should be the most likely to cause confusion.
As a result, the 10 units still learned the laws of curves, rings, and so on.The classification is rough, but there is no expected arbitrary puzzle.

Any kind of packaging that can be classified as similar.
A set of experiments can't prove that all networks are the same. But Nick believes that many of the original neurons can be studied directly, and that there are already rules within them.

The bigger the AI, the easier it might be to read.If the model grows hundreds of times larger, will humans be lost again?
Nick uses an intuitive metaphor: Which is easier to go to town to find a financial district, and which is easier to go to New York to find a financial district?
Banking and accounting offices in small towns may be scattered everywhere.
The team compared three visual models.The difference between the maximum and minimum parameters is about 300 times。
The ears of large models are more subtle in identifying the work. Some units look at colors and some look at shapes, which are then combined into the ears that identify the ears.Agents follow suit to find targets, but faster.

There is also a class of language models that divide inputs into different "expert modules".10,240 modules to be worked onSome of them are in the role-playing game Boss, and others are in the role-playing game Boss.

Each row represents a layer, with experts arranged narrowly by topic.
The information transmitted by a unit must be used by the subsequent unit, and the entire model can complete the task.
Nick suggests that humans can follow these relationships and learn how to model their own organization.
In the interaction map of the visual model, the researchers were able toIt enlarges the entire network into a region and then tracks the specific circuits and neurons.

Another graph traces how information travels through the web. Initially, images are grouped by colour location, and later by animal faces, presenting processes ranging from simple characteristics to complex concepts.

Even if a mistake is learned, it can be traced.
The team trained in nine steps to make the model identify a picture of a rainbow as a bee. The change focused mainly on the last three modules near the output: the original rainbow path weakened and the path to the bees strengthened.

"Learned" means that there are internal changes that can be positioned.Human beings are beginning to have the opportunity to find out exactly where they are wrong.

Before humanity gives more of its future
One graph, one example is not enough. Can these explanations predict the model's response to new inputs?
The team gave AI a blind test.
The first agent studied neurons and wrote a "function manual". The second agent only got a manual to make a sentence on the spot to make the neuron respond as strongly as possible.
Let's take a comparison, is the team from10 billion tokensThe most powerful active samples found in the text.The only thing that can trigger a stronger response is sentences written by an agent using instructions.

Another test randomly extracted 200 neurons and asked each agent to write five examples of the target that should trigger it, and five examples of interference that would appear close but should not trigger it.
Of these, 183 were for positive responses.The strong activation ratio of the target example is 76%, and the interference example is 11%.
Each neuron compared the performance of two sets of examples, and the target group scored 164 wins, 15 draws, and 4 losses.
The first 20 neurons are shown; the full 183-directional explanation corresponds to the Greek statistics
Target examples are much more likely to trigger neurons than disruptive examples.Many explanations have already predicted the model's response to new inputs.
This kind of exploration is being handed over to a growing number of AI Agents.100 000 maps tracking internal calculationsNick himself has seen thousands of them.
Observations, explanations, experiments, which began with Agent involvement. The next step for Nick was to keep them running an automated science lab.
Study of mechanisms for the involvement of automated scientists
Nick predicts that it may take automated scientists months to complete a round of exploration on the cutting-edge language model to the extent that the human study of visual models will take years to reach.
As AI becomes stronger, the speed at which humans read AI may also be pushed up by AI itself.
The trillion-dollar model has not yet been fully understood, and some internal mechanisms remain unclear. Nick proposed a set of working hypotheses pending further testing.
But it touches on a question about the future of humanity: if AI has deceived us in writing, can internal computing still be seen?
Nick's judgment is that even if there is a hidden goal, the information must be read by the next circuit in order for the circuit to work together.Dangerous calculations can therefore leave traceable clues.
If humans could read this inside information, there would be an opportunity to discover risks that cannot be found by looking at the answers alone.
This is also where the phrase "we can all make it through" really hits people's hearts.
We have learned to make AI stronger and stronger. Now we may also learn to track its judgments, goals and mistakes.
At the end of the excerpt, Nick writes a further wish: that humans help these created intelligences to understand themselves; one day, they will, in turn, help us to understand ourselves.
Humanity must be able to see it before giving more of its future to AI.
And the light that shines in the black box may eventually shine on us too.

Tags:

RoboInd editorial channelResearch, robotics companies and market intelligence for cross-border sourcing.Explore the channel ↗

End of editorialExplore more from Global Robotics News ↗