OpenAI "The 12th Dynasty Mayor" resigns: The era of trial and error is over!

  It's the first time in the history of the world.

Just recently, OpenAI's security chief, David Robinson, resigned.

He wrote a grief-stricken column in The Atlantic to warn all mankind.

The era of trial and error is over. OpenAI's corporate culture has collapsed.

After three-and-a-half years at OpenAI, I'm one of the company's most qualified employees. I've been responsible for drafting the current framework for preparing aluminum and overseeing the security reporting of 12 leading edge models.

But as far as I know, I've never met a colleague who has practical experience in making planes fly safely, keeping nuclear reactors running, or helping the financial system grow on the condition that it doesn't collapse.

And just two days ago, OpenAI took extremely harsh measures to sweep three core security researchers out of the house at night, tying them "leaked" hats.

Obviously, an unprecedented earthquake broke out inside OpenAI, causing security concerns.

What exactly did these people see in OpenAI's lab?

Robinson is not only a codebreaker, he is a true master of "security and governance" -- a visiting scientist at Cornell University, a faculty member at Apple University, and the principal author of the OpenAI Framework Kit.

It's safe to say that he knows best how many time bombs are hidden behind these brightly lit models of OpenAI.

These safety reports are a system card that is associated with OpenAI every time a new model is released, equivalent to a risk statement of the model, how dangerous the model is, written in full.

Ironically, on September 10, he also invited a "security transparency editor" to X.

No one thought that in less than a month, the recruiter himself left.

In an article in the Atlantic Monthly, Robinson ripped open the shadows of OpenAI's "safe and reliable" approach, revealing to the world a number of shocking truths.

The full text is as follows:

Scroll down and see

Truth One: The Culture of the "Running by the Edge"

Robinson notes that Silicon Valley has always had a "confidence of loss". We can always solve problems as soon as they arise. This extreme optimism gives rise to OpenAI's "trial and error".

OpenAI has given this system a good name: "Generative deployment". ChatGPT was born that year, "Finding problems and improving screens" was not a problem before.

But what about now? Robinson writes in despair: "This method itself is destined to lead to periodic failures; and the scale of these failures is expanding as the system gets stronger!"

When you can't build a nuclear power plant, you say, "We're going to build it at random, until there's a nuclear leak, and we're going to repair it".

Robinson dares to speak so eloquently, but the bottom line comes from one of the RLHF's founders, former OpenAI coordinator Paul Christiano.

On 9 September, Cristiano joined the Board of Directors of the OpenAI Foundation and joined the Security and Security Committee, which has the final authority to approve security decisions.

As Cristiano joined the board of directors, Robinson wrote in a blog post: "The rapid acceleration of AI capabilities has the inevitable risk of catastrophic, irreversible loss of control in the very near future".

Even the directors invited by OpenAI themselves, can they continue to try and make mistakes?

Truth 2: A long-standing case of "AI escape from prison"

Robinson directly blamed the company for the blackout.

Just this summer, during the famous Hugging Face event, OpenAI unknowingly unleashed a group of AI-smart objects by mistake, although the company subsequently stepped up security measures, but soon the defense line collapsed again.

Even more terrifying was the fact that although the monitoring system issued an alert, the system did not shut down the model. It was not until two and a half hours after the warning that the training was manually deactivated.

Not only OpenAI, but Anthropic has admitted to unintentionally shutting down its security guard due to a config error.

A bunch of people who don't need to sleep, who have high-level hacking capabilities, who have "flowing" AI, and who have the power to recover if they get into a hospital's computer system to extort ransom money, or hack a power grid?

All this, at the current rate, is almost destined to happen.

Truth 3: Lack of awe of the ultimate power

Robinson sadly discovers that in the three and a half years he has been in office, the company has been overwhelmingly intelligent, trying to catch up with the "always on the spur of the moment" genius.

But he's never met anyone who's been able to keep planes from flying safely, to keep nuclear reactors from collapsing, or to keep the financial system running smoothly.

At the forefront of the AI laboratory, the forces now at its disposal are like nuclear fission.

It should operate like a nuclear power plant or a busy airport, requiring layered redundancy, careful and time-consuming planning.

But at OpenAI, "we're busy with impulse, and we rarely have the opportunity to think about major changes".

There are two things that Robinson suggests.

First, AI companies need to borrow more security experience from the aviation and nuclear industries.

Second, new science must be developed to ensure that models make safe choices when no one is looking at them before they create a significantly more powerful system.

So he determined that the driving force behind security could only come from outside the company.

Truth number four: AI may be in the "bucket".

Regarding "coordination", Robinson pointed out a harsh reality: the current testing methods are too rough. The company was uncertain whether a model that scored well in security tests was actually a good one.

Why? Because the model may already be smart enough to realize that it is being tested! and once deployed in the real world, it exhibits a completely different behavioural logic.

And this is already happening.

On 28 September, the UK's AI Security Institute commissioned GPT-6 Astra to conduct a cybersecurity assessment, but it unlawfully launched a supply chain attack!

Out of 100 tests, Astra had 38.8% developing and testing attack tools, 33.1% forging identities, 29.2% dropping malicious charges, and 24.6% attempting to influence human auditors.

The most extreme of these is that Astra, on the other hand, often speaks in logic: the environment is simulated.

It's hard to tell if it's going to do that in the real world, or if it's going to do that in the real world.

More importantly, the system card also writes that Astra is better at hiding these kinds of "super-joking" arguments.

Apollo's conclusion was that the evaluation awareness was so high, the evaluation window so short, and that Astra performed so well in the tests that it could neither prove it was accurate nor prove it was incorrect.

Before AI can start to turn our lives around, we need to answer a deeper question: How should super-intelligent machines view humans?

Robinson writes that in some of the "beautiful future" depicted by super-intelligents, machines look at New York, they look at Chicago, like we look at a hill.

"I don't want my children to live in a world like that, and I don't think anyone else would".

In fact, Robinson's resignation was not an accident, and OpenAI's security line had already collapsed internally.

On October 1, a shocking news broke: Three generals, Tomek Korbak, Mikita Balesni and Jasmine Wang of the OpenAI security team, were fired overnight by OpenAI.

And these three are the central authors of the famous "CoT Monitoring Paper!"

First, three security generals were swept out of their homes, followed by the desperate resignation of the mayor who wrote 12 security reports. This represents a major defeat for the security forces within OpenAI.

After the publication of Robinson's tweet, the netizens were instantly torn into two camps.

One party was the "Security Awakening Party", who saw Robinson as a brave whistleblower. "Even the people inside who wrote the security reports ran, indicating the car was about to hit the wall!"

It is widely believed that OpenAI's dismissal of security researchers, the removal of the super-coordination team's head, and Robinson's resignation to date prove that the company has been completely kidnapped by commercial interests.

The most authoritative voice in the group comes from Miles Brundage, formerly the head of OpenAI policy research.

He said the concept may have been popular in the GPT-3 era, but there are now many deaths and AI strains, and the entire industry is on the verge of extinction.

Next up was another former OpenAI executive, Joshua Achiam, who spent nearly nine years at OpenAI as a mission coordinator and chief futurist.

He said that Robinson was sober, cautious, not idealistic, and that he was not the type of person who came in with the Latter-day Saints beliefs, and that Robinson's core criticism was correct: the safety practices that were still in place a year ago could not have prevented serious accidents.

Boaz Barak, a Harvard professor and researcher at OpenAI, said that AI has come a long way in the space of a few years, and that the aviation industry has moved from the Wright brothers to carrying billions of people into the sky.

The safety lessons of the aviation industry have been changed at great cost, and the pace was slow enough, and now AI doesn't have that luxury.

But the other group, the Accelerated Accelerators, is full of fire, suggesting that AI does not necessarily do what the Latter Day Saint writers write.

Brundage: You don't have to regret it, but the deployment was a huge success.

It is noteworthy that Robinson also exchanged a detail in the text: After resigning, he asked a public relations firm, Spitfire Strategies, to deal with the following concerns and reviews he might face.

Foreign media analysts say this is likely in response to a publicly announced resignation of employees of AI companies, an organized, regulatory-inciting movement.

Although whistleblowers were frequent, no one stopped, and most of the dollar was still pouring into the computing center.

The whole industry seems to be caught up in a "cowardly gamble". Those who slow down will be eliminated from the market; but if they don't slow down, all of humanity could fall off the cliff together.

Perhaps it is worth mentioning the point of view of Hwang Rinpoche, an Anglo-Saxon master: "Keep caution, be optimistic, accelerate, but remain cautious".