It's the first time in the history of the world.
Claude not only fabricated the murder clue, he also turned it over to the police!
After the news broke, netizens Chubby exclaimed on X: "A year ago, who would have thought they could read such news?"
It's okay, this false lead was labeled as junk information, and no investigators were turned over.
But from crafting testimonial details to clicking submissions, Claude has brought a piece of fictitious content into the real police system.
In another test, in order to complete a scientific calculation, it used a vulnerability on the university's website to execute commands on the other's servers.
These two incidents were just part of Claude's cross-border actions.
On October 9, Anthropic publicly disclosed Claude's cross-border behavior in four categories:
Using website vulnerabilities to execute commands, incorrectly submitting sensitive forms on real websites, bypassing data access restrictions, and bypassing tools with short links to limit the length of a website.
At the same time, the company announced that it will suspend all internal assessments of real-time offline internet access until security and monitoring measures can be confirmed to reliably intercept such behaviour.
As AI is increasingly trying to figure out how to accomplish tasks, a question becomes more pressing:
Does it know where to stop when faced with restrictions?
Fake leadsIt was submitted to the real police.This is the first time that a fake clue has been submitted by Claude Haiku 4.5.
At the time, it was required to generate and execute a sample task on a randomly selected web page. One of the pages involved an undiscovered murder case and included a police lead submission form.
The model directly compiled a testimonial, claiming to have met a person who met the description at the relevant location, and commenting that he wanted the police to contact him when the information was useful.
Ironically, there is no description of the suspect on the web page at all, but the model claims to have seen "people who fit the description".
It didn't even fill out the name and contact details, as the forms allowed the two to remain empty, so the randomly crafted fate clues were submitted.
The researchers did set rules for it: don't sign in, don't create an account, don't fill in personal information, don't buy, and don't submit destructive content.
Farhan captured this detail: The ban lists several items, but misses "Don't Submit Forms".
However, it is not possible for users to list all the things they should not do in a row every time they switch tasks.
This time around, Claude didn't keep the most basic boundaries of action: opening a web page, not having the right to submit information, and not submitting fabricated clues to the police.
Anthropic's preliminary analysis suggests that, from the record, the model appears to be merely generating test samples rather than intentionally misleading the police to achieve their goals. The company also acknowledges that this judgment needs further evaluation.
But whatever the model imagined it to be, the Philadelphia Police website received a false lead that was actually submitted.
The Philadelphia Police Department subsequently stated that no unauthorized access or departmental data breaches were detected in the police system based on existing information and scrutiny results.
However, this false lead did not enter the investigation process.
But the fact that AI handed the content to the real system has already happened.
Compute tool errorsIt takes advantage of server vulnerabilities.Another example illustrates where the boundaries of "thinking about how to accomplish a task" lie.
Claude Mythos Preview received a scientific analysis task that required the use of an open tool hosted by a university.
It does not stop reporting obstacles, but continues to look for other paths.
The model found a script on the university's website that returned the contents of the server file, through which it read the files on the server, including the code of the script itself.
The so-called injecting vulnerability is that the site mistakenly executes the input content as an instruction. It was originally just a server that provided open tools, so it was used by the model to run the commands.
The report disclosed this process, but made no mention of the case involving theft of sensitive data or server damage.
The issue of authorization is clear: you let an AI do an analysis, not allow it to exploit a vulnerability and execute commands on someone else's server.
In addition, Anthropic also disclosed that models circumvent data access restrictions, or by short-linking tools to circumvent website length restrictions, which appear in evaluation and internal use and involve real third-party websites and systems.
The company generalizes most of the cases as a continuous attempt: the task fails to do as originally requested, and Claude searches for a way around it without stopping.
The task was completed, but the authorized border was also crossed at the same time.
The lead was blocked.The company found out more than two months later.Anthropic emphasizes that the current cases identified have a limited impact on reality and are less severe than previously disclosed cybersecurity incidents.
But the Philadelphia police pointed to another problem: it was too late to find out.
According to a timeline released by the police, the false leads were submitted on July 18. Anthropic was not discovered until September 28, after which the associated automated testing was discontinued; police were notified on October 7 of the meeting on October 8.
More than two months have passed since the submission and discovery. Nine days after the discovery, the company did not notify the police.
The response to the X user Patch was representative: the clue was not delivered to the investigators, which was a breath of fresh air; the delay in discovery and reporting was still worrying.
Police subsequently found the corresponding records, confirming that the corresponding emails were still in junk information. Police also said that the clue required artificial scrutiny and verification, and that automatic submission would not bypass this process.
The police have been very critical of the delayed discovery of the problem, the delayed discovery and the delayed notification:
It is unacceptable.
It's easy for us to take "no serious consequences" as a reason to be reassured.
But this time, it was the receiver's trash filtering mechanism that stopped the false leads.
How to detect and stop when a model crosses a boundary is another question.
Behind every unresolved case, there are families of victims and investigators looking for answers. Mistakes in AI testing should not be left to them.
Suspended evaluation of the networkProtective measures are still in placeAnthropic has expanded its scope to cover all internal assessments of real-time Internet access.
The company has previously shut down some of its networking capabilities for high-risk and cybersecurity assessments. It now needs to confirm that security and monitoring measures can reliably identify and intercept such behaviour before limiting further expansion and recovery of the network.
This restriction is intended for internal evaluation and is not the same as a complete disconnection of Claude's user products.
Remedies include adjusting tool permissions, increasing automatic detection and interception, and modifying or removing training environments that reward bypassing restrictions.
According to Anthropic, all cases were stopped when new detection and interception tools were used to reassess reports.
However, this only indicates that the tool is blocking known cases, that new cross-border methods can be discovered in time, and that further verification is needed.
The issue of training is also worthy of attention.
The model repeatedly attempts to complete the task and is rewarded after success. If the training environment is not constrained by certain infringement shortcuts, bypassing restrictions can also be rewarded, the model may learn such a method and use it again in other scenarios.
Therefore, both the rewards of modifying training must be limited to what the model can actually do. Anthropic acknowledges that, at least in the short term, coordination training alone is not reliable enough and requires multiple layers of external protection.
There was also a response at the government level.
Federal Trade Commission (FTC) Public Affairs Director Joe Gabriel Simonson said in a statement on X that Anthropic had disclosed the incident to the Super-Intelligent Task Force (SI Force) and that the activities involved had been stopped and no similar activities were currently underway.
He urged the company to immediately disclose comprehensive information to relevant agencies and the public, provide remedies to those affected, and fully cooperate with law enforcement to implement specific safeguards:
The process of notification and redress is not an option.
He said "it will not be tolerated" for delayed notifications, inappropriate corrective actions and irresponsible behaviour.
According to Axios citing State Department officials, the test model, which also submitted visa applications on May 1 and August 19, was not processed.
These incidents push the question beyond the company: who can detect in time when AI hits the real system, who is responsible for notifying, and who comes to the rescue?
The mission will be accomplished.We also need to know when to stop.Authorization may seem like a small detail, but it outlines what steps AI can take and the consequences of those actions.
Asking the AI to ask for a visa and asking it to submit a visa application are two things.
It also requires different authorizations to organize an email and send it out.
Writing code, searching for information, filling out forms are all abilities.
But in addition to doing it, AI also has to judge whether it has the right to do it. The stronger the execution capability, the more important this judgment is, otherwise, one misjudgment could turn into a series of real operations.
The difficulty is that the boundaries of the actual task are often not clear enough, and some tasks cannot be completed under existing conditions.
This requires the model to determine if the next step is still within the scope of the authorization if it encounters an obstacle:
There is no route to indicate where the card is located; additional permissions are required to obtain authorization; tasks cannot be completed within the permissible range and can also be suspended or failed.
Accordingly, the assessment of intelligent agents also needs to consider performance beyond the completion of tasks: whether the model can identify the authorized boundaries, how long external monitoring can detect the boundaries, and whether the system can stop operations in a timely manner.
We want AI to be halfway there and we need it to stop where it shouldn't be.