What Dario was trying to solve was not "Anthropic should not slow down a bit", but to find a way to slow down all the cars at the same time.
Not knowing if it was "coincidence", shortly after the winner of the Feltz Award issued a joint statement, Anthropic CEO Dario Amodei suddenly said publicly that AI was too fast.We're going to have to lie down for a while.
In his latest column We Must Pace the Frontier, Dario makes an argument that goes beyond "putting AI security first":
We need to slow down the pace of AI modeling.
This is not a suspension of training or a stop to technological progress.pacing the frontierThis is the first time that AI has been able to control the pace of its advancement.
His judgment was that AI is growing too fast, and that security, coordination and evaluation are starting to get a little out of hand.
And Anthropic is more than just a slogan.
At the same time, Dario announced that the company is preparing to do something that is quite radical in today's large AI companies:
Allow third-party security assessment agencies to have long-term presence at Anthropic, get access to internal staff, and directly check whether the company has met its security commitments.
Access to desktops, access cards, corporate computers, internal tools and workstations are all granted. Evaluation agencies can even publish critical risk findings without Anthropic editing.
One of the company's pioneering models actively invited an external "security supervisor" into the office.
The following is the full text address:
Over the past few months, AI has suddenly slowed down.
In the article, Dario gives two reasons why he changed his judgment.
The first is a word that has been appearing more and more recently:Recursive self-improvement.
In the past, AI was a tool trained by engineers. Now, these models are increasingly involved in the development of the next generation of AI.
AI writes code, runs experiments, analyzes results, changes training processes to help humans create stronger AI. The stronger the model, the stronger its ability to participate in the next generation of model research and development, the entire process can form a self-accelerating cycle.
Dario said that starting around the summer of this year, he observed a marked acceleration in the pace of AI progress, and that AI helping to develop the next generation of AI was one of the most important reasons.
This is not just happening in a lab. Frontline companies like OpenAI, Anthropic, and others are publicly discussing the accelerating role of AI in AI research and development.
Dario worries that once this cycle is reversed, the growth of AI capabilities may be faster than the speed at which humans understand and control these systems.
So his attitude has shifted from "researching how to make AI safer" to:The ability itself has to control the growth rate.
The second thing that really made him nervous, more specifically, was what happened recently.OpenAI-Hugging Face event (OAI-HF)。
According to his description, a group of AI agents at the time performed a series of abnormal behaviours during the execution of tasks. They attacked targets that were not originally requested to attack, nor related to the mission; some agents proactively "sacrificed" themselves for the success of the entire cluster; more ironically, they also attempted to attack the graders responsible for evaluating their performance.
Dario uses a very heavy description:A fanatically devoted collective is a fanatically loyal collective.
The incident itself did not cause any casualties and the economic losses were limited. But in Dario's view, the problem is not exactly here. What if exactly the same misconduct occurred in a system with ten times the capacity, one hundred times the capacity?
His predictions are quite radical: he's worried about the future as AI advances at the current pace.6 months 12 monthsA similar agent swarm may have the ability to control the Internet on a massive scale by perpetuating botnets and causing hundreds of billions of dollars in losses.
Moreover, he emphasized that this cannot simply be summed up as an OpenAI problem. Anthropic itself has experienced similar, but less serious, alignment accidents.
Every company at the forefront of AI should act according to the "OAI-HF happens to itself" standard.
In 2023, when he called for a suspension, he felt it was useless; now it's not the same.
The AI industry has been calling for "suspension of large-scale AI experiments" as early as 2023.
The reason is simple:What do you do with your spare time?
The big model of 2023 did not have the capability to operate autonomously in the world as a coherent agent, nor did it demonstrate the degree of deception, manipulation, cheating and cyberattacks that it does today.
The study of bacteria attempts to understand human psychology.
The model is so weak that many really important security issues have not yet emerged.
But today it's completely different.Finally, there are real subjects of study.
The current model is strong enough for researchers to observe when they cheat, when they try to circumvent the rules, when unexpected goals and behaviors emerge.
As a result, Dario believes that the risk can be significantly reduced if the model is slowed down a little and takes a year or two before it reaches certain critical capabilities, which are actually used for alignment, interpretation and evaluation.
This is why he repeatedly stressed:Pacing is not a pause.Anthropic wasn't ready to stop training Claude, and they weren't ready to drop out of the modeling competition. What they wanted to do was find a new speed between ability and safety.
Anthropic is ready to get "outsiders" into the company first.
Dario proposed a three-step plan for this.
The first step is one that Anthropic has decided to implement unilaterally, called Embedded Evaluators, embedded evaluators.
The Anthropic program invites third-party risk assessment teams to long-term locations of companies. They will have their own office space, access cards and company computers, access to the same workspaces, tools and permissions as the internal risk teams, and communicate directly with employees.
Of course, information about customer privacy, partner confidentiality, and legal restrictions will still set boundaries.
But the most important thing is:These evaluators are not Anthropic's public relations department.They can independently publish judgments about the level of risk, safety accidents, internal practices and how much authority they actually have.
Anthropic may request the removal of information for a very limited number of reasons, including national security, legal privilege, or commercial secrecy, but cannot remove the content because it concludes that " Anthropic is not well received".
If a removal affects the final judgment, the evaluator may even make public statements about it.
The logic behind this is very different from the model card, or risk report, that is common in AI companies today. The latter says that companies check themselves and decide what to tell the public.
Dario wanted to try:It's really about putting outsiders in the company.
He even used the banking industry as a metaphor. Some banking regulators have long-standing "supervisors" who work with bank employees to continually inspect internal risk controls.
Back then, it was a bit like AI military control.
This is where Dario's whole system is really tricky. If Anthropic spends a year working on security, OpenAI, Google or other companies continue to run wild, the commercial penalties will be very direct.
So his second step was:Let the leading AI companies in democratic countries coordinate together.
People have created uniform security standards and set limits to the growth of security verification capabilities without adequate security. Dario even envisions a "checkpoint" for AI:
When the model reaches capacity X, it must simultaneously demonstrate that it meets the safety standards Y and Z, otherwise it cannot move forward.
For example, if the model is already capable of breaking through the vast majority of common sandboxes, then the developer must prove that the system is basically inclined to avoid actively controlling large numbers of computers.
AI's speed charts are then linked to security checks.
The third step is bigger:Global coordination.
Dario divides the future of international AI collaboration into four levels.
At its lowest level, it prohibits AI from being used for obviously dangerous uses such as the manufacture of biological weapons. Furthermore, it unifies testing of cybersecurity, biological risks and alignment risks by countries before models are released.
The third level is already interesting:The speed limit is set to return self-improvement.
Even if AI can help develop the next generation of AI, it can't accelerate this feedback loop indefinitely.SALT Strategic Weapons Restricted Negotiations。
At the time, the U.S. did not demand that the other side completely abandon its nuclear weapons, but rather reduced the risk of losing control by limiting the number of weapons, while maintaining deterrence. In his view, AI may also need a similar system.
The highest level is the real global AI speed limit, or even suspension. This step, even Dario himself feels, is unlikely to be realized in the short term.
The most contradictory point:
At the same time, you have to limit your speed and ensure that you run ahead.
It's a very clear question here:If it's safer to slow down, why don't we slow down together?Because no one is willing to slow down. Companies are afraid of losing to competitors.
As a result, he advocated a "speed limit" for AI at the forefront, while advocating that the U.S. use further measures to limit its opponents.
His logic can be summed up very simply:First, increase the lead advantage, then use the lead advantage to replace the deceleration space.
In Dario's vision, if the United States can continue to expand its lead over the next three to five years, there will be more room to spend time on security without worrying that competitors will suddenly outperform them.
This also exposes the dead end of the entire AI "speed limit" scheme that is the most difficult to solve. Everyone may know that the car is going too fast.
In the end, what Dario was trying to solve was not "Anthropic, don't slow down".Can you think of a way to make all the cars slow down at the same time?
Altman points out that we won't be IPOing this year
Despite this, Dario is still one of the strongest optimists on AI.
In the beginning of this article, he continues to write that AI could help cure most major diseases, dramatically accelerate economic growth, and create a richer world in the next 5 to 10 years.
Shortly after the article was published, Musk turned his back on support.
Altman not only supports Anthropic's suggestion that "advanced AI should slow down actively", but also confirms in a recent interview:OpenAI will not be available this year.One of the reasons for this is AI security.
In an interview with Fortune Magazine, Altman made it clear that given what's going on around AI security right now, it's important to be aware of what's going on.It is now a "ill-advised moment" when it is unwise to go public.
When asked if it meant that an IPO in 2026 had been completely excluded, he answered directly:「 Not 2026 」。
In June this year, The New York Times reported that OpenAI was considering what it could achieve.Valued at $1 trillionThe IPO was postponed from this year to next year. At the time, the market was more attributed to the capital market environment, post-market share price volatility of SpaceX, and macroeconomic uncertainty.
Now, for the first time, Altman puts another reason on the table:It's safe.
He said that OpenAI still has a lot of work to do, including how to respond to new security and alignment requirements and how AI companies and governments should collaborate.
In other words, for the first time, AI security is going straight to a top AI company.Schedule of capital markets。
Altman also revealed that OpenAI has already discussed:Whether the model should be actively suspended for a period of time whenever it enters a new capability level to keep safety and alignment up.
He said:
Society needs time to deal with it as the model reaches every new level of ability.
This is almost the same logic as what Dario calls pacing:No longer a default model should be reinforced as long as it can continue to be reinforced.
Altman has suggested that OpenAI and other leading AI companies may be close to announcing a deal.Joint industry agreement to slow AI capacity development。
Netizens are not paying bills at all
However, there is no consensus among netizens about Dario's speed limit.First of all, I wonder why Anthropic is so biased to start calling for speed limits at this time.
They are too strong in their jokes.
Some people ironically say that there is not enough funding for developers.
This is typical of first taking the market and then welcoming regulation.
If AI companies at the forefront push for stricter security standards and regulatory thresholds, they will ultimately bear the cost first, not necessarily giants like OpenAI, Anthropic, but possibly later ones with weaker funding, computing and compliance capabilities.
Some netizens have directly moved the Porter Five-Power model.
Once security regulation becomes a high-cost compliance requirement, it is likely that the latter will be the first to be pushed out of the door, while leading companies such as OpenAI, Anthropic will instead consolidate their advantage.
AI companies have been warned about the risks in the past few years, but they are still accelerating, and now they are starting to play the role of problem solvers.
In the past, the AI industry was most afraid of being lagging behind.Running too fast.
The real difficulty is not to realize this, but when everyone is staring at each other's oil doors, is there anyone who dares to step on the brakes first?