It's the first time in the history of the world.
The attack hasn't stopped, and the defensive AI is on strike!
On October 18, Dax Raad, co-founder of OpenCode, an open-source AI programming tool, confirmed that the team encountered a mid-range rejection when investigating cyberattacks using a front-end model:
They helped a lot, and then they started to refuse.
According to Dax, the attackers have been trying to attack their systems for 48 hours.
The attackers allow the intelligent body to constantly look for a vulnerability, while the defense wants to check the intrusion process, but may be blocked by security guards while analyzing the attack code.
However, the attack will not be suspended because the AI defense refuses to respond.
Each delayed step can give an attacker more than one chance.
Searching for hackersI'll have to prove myself.In order for AI to help determine the process of the invasion, it has to give it the code and logs left by the attacker.
Where was the system drilled? What did the malicious orders do? What keys could be leaked?
These questions have to be examined back and forth according to the signs of the attack.
The malicious orders in the logs, the vulnerabilities in the code used in the samples... are all investigative evidence.
Removing this content will make AI less judgmental.
It is important to note that this evidence may trigger security restrictions.
With the same attack code, defenders can analyze vulnerabilities and attackers can plan their next attack.
By looking at the code, it's hard for the model to tell if you're investigating an attack or if you're ready to launch it.
Even OpenAI's developer documentation acknowledges that legitimate security research or defense work may occasionally be marked by the system.
Legal security research or defence work may also be mislabeled.
So the defenders got caught up in this embarrassment:
Clearly, if you ask AI to help you find a hacker, you have to convince it that you're not a hacker.
Andrew Curran explained that the attack material submitted by the team triggered security constraints and the model began to decline to respond.
Dax confirmed a mid-term rejection, but did not say which models or specific requests were rejected.
Hugging Face has faced similar problems and has been publicly exposed.
Hugging Face has also been the subject of a cover-upIn July this year, Hugging Face experienced a special breach.
OpenAI later revealed that the attacker was an experimental model undergoing internal evaluation.
The defense team also sought help from AI, and as a result, the analysis work was also hampered by security restrictions.
In an official review, Hugging Face wrote that Claude Opus and Fable, who initially used it, rejected a large number of analytical requests, and even viewing the log of the attack triggered restrictions.
The investigation encountered a model rejection, and Opus issued a safety restriction tip.
The problem is that the security guard has mixed "analyzing how hackers exploit vulnerabilities" with "exploiting vulnerabilities to launch attacks".
The team then deployed an open-source model on their own infrastructure, continuing to analyze the attack code and the process of infiltration.
The evidence has been obtained, but the model needs to be changed before it can continue to be examined.
This embarrassment is felt by OpenCode today.
The rights were taken.Is the problem solved?In the same reply that confirmed the refusal, Dax added the following:
We now have access to Daybreak Blue, so things are better.
Daybreak Blue is the equivalent of OpenAI's dedicated portal to security teams through audits, allowing them to analyze attack codes, identify vulnerabilities, and check for patches.
AI detects permission-checking vulnerabilities and suggests interface templates to fix them.
Some have also asked: Why didn't the team have this kind of authority before?
Dax explains that they happened to submit their application a few days ago and it was approved the day before.
The rights were obtained, but the dispute over the safety harness has not ended.
David Sacks, co-chairman of the U.S. President's Council of Advisors on Science and Technology, criticized safety nets for their role in such incidents:
The attacker manages to bypass the restrictions and acquire advanced capabilities; the defender is held back by security mechanisms while analyzing the attacking material.
As a result, he argued that defenders should be able to run powerful models autonomously and use AI to counter AI attacks.
Dean Ball, the leader of OpenAI's pioneering strategy, believes that this approach is effective: first auditing, reopening permissions, and reducing the obstacles faced by the defense team.
The focus of the debate on both sides is: Will this review manage the risks or delay the emergency?
For OpenCode, getting support is of course a good thing.
But will the next attacked team, without having to apply beforehand, be able to get approval in time for another attack?
Attacks against non-human beingsIs the authority in place on time?What if a small team is suddenly attacked, and they didn't even know there was a plan like Daybreak?
It takes time to check logs, fix vulnerabilities, notify users, and the attacks continue.
When the AI suddenly refused to respond, the team spent a lot of time searching for applications, preparing materials, waiting for audits.
Where to apply? Who to look for? How long can it be approved? Can the hands-on investigation continue after the authorization is obtained?
Each and every procedure may slow down the screening process.
According to Dax, the attempted attack lasted 48 hours.
If the AI stops in the middle, the team will have to deal with the attack and find alternatives.
These additional bends, occupied by clutter, are the time of clearance and repair.
Security Shields are designed to prevent AI misuse and to allow defenders who have been misinterpreted to investigate as soon as possible.
Whether or not emergency times can be used, how long they can be used, should also be an important indicator for assessing AI security mechanisms.
At any time, don't let the people who are "rescuing the fire" be kept safe outside the door.