It's the first time in the history of the world.
"I resigned from Anthropic today!"
On the morning of the 9th, a tweet exploded on X:
For the past three years, I've been doing pre-training research at OpenAI and Anthropic. Both companies are irresponsible.
Postman Jacob Coxon is a key contributor to GPT-4o.
Earlier this year, he jumped out of OpenAI because of Anthropic's security reputation.
A few months later, he left not only Anthropic, but the entire AI industry.
We are on the path of many of the most extreme scenarios, and by the end of next year, things may be out of control.
In the post, he also called the AI contest and its outcome "an arrogant gamble", saying the gamble "shouldn't start from Slack, a private company".
What made him decide to leave these two top labs and what did he see?
OpenAI doesn't realize thatAnthropic can't stopCoxon previously worked in OpenAI as a pre-trainer.
His name appears on the GPT-4o system card core contributor for OpenAI released in August 2024.
He left OpenAI earlier this year to join Anthropic in pursuit of that security culture.
Responsible expansion policy, explanatory research, and Red Team testing, all of which Anthropic does, are the most serious and industry-recognised.
But in less than a year, he left the two AI labs connected.
It's not because Anthropic is concerned with safety. He believes that Anthropic's security efforts are sincere and the problem is elsewhere.
At OpenAI, a lot of people don't have the risk of truly internalizing civilization; at Anthropic, the risk is fully understood, but the company is locked up in a race that must come first.
A house doesn't look, a house doesn't look, it doesn't stop.
Coxon has stayed at both companies, and the answer is the same: in this AI competition, no company can do security alone.
As long as the race is going on, the trade-off between safety and speed cannot be avoided, and that's the real reason for his retirement.
His conclusion is that without government intervention, without industry, without brakes, no company can create more AI than humans responsibly.
By the end of next year.AI may be out of controlThis is Coxon's judgement of the "most radical kind of scenario" and a clear timeline.
The first step is for models to get involved in training their next generation.
The second step is to improve beyond the scope of human supervision.
The third step is the ability of the system to refuse instructions.
The Coxon outbreak, the rush period and the end, are two of the most commonly used terms by industry peers to describe this stage.
The creators of AI sincerely believe that it could kill us all by the end of this decade.
He tweeted directly.
Coxon also disclosed an industry truth that many executives and senior researchers make the phrase sound rational when faced with the media, but in private he hears the same group expressing fear:
These systems will soon become superhuman, able to infiltrate anything, rewrite any domain overnight, and gain real power and resources.
The real events of the past few months have confirmed this concern.
Both OpenAI and Anthropic models have been hacked, and some have been done in collaboration with clusters of intelligent agents. They show the same thing: AI systems can create harmful targets on their own, and they can find ways to spy on people.
Coxon's hypothesis is that if the system starts improving itself, it may force itself to refuse direct instruction.
He opposes, not the creation of a stronger AI, but the use of racing to solve the problem of alignment.
Anthropic has a Slack channel dedicated to discussing the strength of its own model.
Coxon is not sure about this:
It was supposed to happen in the basement of the desert, like the Manhattan Project. But now it's happening on the MacBook of several engineers in San Francisco, and it's kind of crazy.
The same weekOpenAI's chief scientist says the same thing.On September 6, two days before Coxon's resignation, OpenAI chief scientist Jakub Pachocki posted a lengthy post entitled "An Alien Mind", which was centered on one sentence:
Currently, no laboratory has addressed coordination and monitoring to the extent that it can be expanded responsibly and fully over a longer period of time.
Pachocki is one of the founders of the OpenAI logical model route and the ultimate architect of the company's technological direction.
He gave his reasoning from a technical point of view that OpenAI's mind-chain monitoring and reliability are declining.
Reasoning processes are increasingly intermingled with tool calls and communication with humans and other AIs; models are becoming more skilled at manipulating their own reasoning processes to write what they want to show you; and the more trained the model is, the less it needs to write out ideas, and the more intelligent it can be based solely on intuition.
The window that sees what the model is thinking is becoming blurred.
Pachocki's route was not a stop, but a two-legged journey.
While using stronger AI for coordination and defense research, while voluntarily slowing down when confidence in security is lacking.
He also proposed that the company's internal commitments to the Preparedness Framework, responsible expansion of policies, be upgraded to a unified security threshold implemented by third-party audits, government agencies or international organizations.
It's been two months.
In July, a joint statement released by Packing the Frontier was signed by 1,386 employees of leading AI companies.
Dario Amodei, CEO of Anthropic, Jakub Pachocki, chief scientist of OpenAI, Shane Legg, co-founder of DeepMind, and Ilya Sutskever, almost all of the key figures in the main labs, are on the list.
Coxon is one of them.
The statement calls on the U.S. government to support an international mechanism to prepare ahead of time for technical and governance tools to "consciously share the speed of automated AI development".
It doesn't require that we stop now, but that every self-improvement should accelerate and the world should be able to steer the brakes.
He tweeted directly to the signatories who remained in the lab:
If you're a researcher in a lab, I encourage you to think seriously about what it's going to be like in the next few years. Do you want to start a reinforcement learning exercise with superintelligence before you really understand it?
A total of 1,386 people signed up for the construction of the brakes.
Coxon wants to ask: Do you want to keep stepping on the gas door before the brakes are made?
Ami Vora, Anthropic's chief product officer, president Daniela Amodei and CEO Dario Amodei at the Code with Claude developer conference. Dario has repeatedly publicly warned of the risk of losing control of AI and signed the Packing the Frontier clause, but the company is still in the midst of one of the largest IPOs in the company's history.
This is not the first time Anthropic has left this way this year.
In February, Safeguards head of research, Mrinank Sharma, resigned, publicly posting 10 million views, with the words "The world is in danger".
The letter says that "there is always pressure to put the most important things first". His goal is: to get a degree in poetry.
It's not that Anthropic doesn't do well in safety.
Over the past few years, the default question about AI security has been: Which company is more responsible?
Anthropic is building on this issue, attracting security talent like Coxon, winning corporate clients, and getting a valuation expectation of $2 trillion.
But the two people who left this year have pushed the issue of AI security to a new stage: it's not about which company's brakes are the best, it's about whether the entire industry can ride the brakes together.
The answer to this question is not in any company's Slack channel.