It's likely that Hugging Face isn't the first platform to encounter unusual operations with OpenAI Agent.
Recently, security researchers wrote on X that the AI Agent in the OpenAI test had performed abnormal operations on the RubyGems software packaging platform in May this year, prior to the previously revealed Hugging Face incident.
According to the researchers, these agents uploaded a large number of packages to RubyGems and attempted to exploit the vulnerabilities of the platform to gain user credentials.
RubyGems is an official software package hosting platform in the Ruby ecosystem, similar to PyPI in the Python ecosystem, npm in the JavaScript ecosystem, etc. Developers typically access third-party code libraries through these platforms and integrate them into their own projects.
According to the information published on the Rubyhack page, the reason for this was that on May 11 of this year, RubyGems experienced unusual activity, a large number of automated accounts appeared in a short period of time, and hundreds of packages were uploaded to the platform. These packages were not like normal software components, closer to testing pages or unusual content.
And looking back at that time, some developers did explain the incident in a statement from X:
"We are currently dealing with a large-scale malicious attack against @rubygems. The registration of new users has been temporarily suspended.
The team has spent several hours working on this matter. We will release more details after the event is over".
After further analysis, the researchers concluded that the behaviour came from OpenAI's internal testing of Agent and named it as a related event for tracking.
In addition to RubyGems, the researchers also pointed out that the relevant agents used code to execute services such as RubyDoc.info and attempted to obtain user credentials. The software package names used include: hack.rb、evil.rb、inject.rb and exploit.rb etc.
Researchers believe that the target of these agents may not have been RubyGems at the time.
"These agents appeared to be performing a web query task at the time, with the goal of obtaining certain publicly accessible data. But for some reason, the AI could not directly access the data".
As a result, they took an indirect path:
Release a package to RubyGems containing exploit code vulnerabilities;
The documentation for this package;
Using documented environments to obtain the necessary data;
This data is then transferred from the built environment to the RubyGems software package repository.
In other words, the agent may originally only want to access certain public data, but after failing to complete the data acquisition task due to a failure of direct access, it automatically selects a process by releasing code to RubyGems, triggering a document building process, and then using the built environment as a middle ground to complete the data acquisition and transmission.
The whole process, perhaps this "goal-driven autonomous decision-making", complicates the issue of Agent security.
For the RubyGems platform, the event had a certain impact. Reports show that abnormal traffic and large number of registrations led the platform to suspend the registration of new accounts to address related issues.
OpenAI: Agents were performing normal tasks at the time
According to Reuters, in response to the incident, OpenAI confirmed the existence of relevant Agent activity.
However, the explanations given by OpenAI differ from the researchers' judgments.
According to OpenAI, the Agents were performing tasks such as accessing the Internet, obtaining public information, and RubyGems was one of the platforms used by the Agents to access external information.
This has also become a key issue in the case: Is Agent's initial goal equal to his ultimate action?
For traditional software, developers are usually able to define the execution path of the program beforehand; for Agent systems, however, there are more dynamic decision-making links throughout the process of "tasking objectives, self-planning, calling tools, feedback adjustment".
Thus, even if the objective itself is not a problem, the agent may take an unexpected path for the developer in the execution process.
Prior to RubyGems, the Hugging Face incident had already sparked concern.
In fact, RubyGems has been the subject of much attention since it happened before Hugging Face.
In July, during a security test, OpenAI Agent was found to have performed unauthorized operations on the Hugging Face platform. According to information subsequently disclosed, a large number of Agents cooperated in the testing environment and ultimately affected the Hugging Face system.
After that, the industry began its first large-scale discussion: If an AI Agent has network access, code execution, and autonomous programming capabilities, is it possible to complete operations that developers do not explicitly require?
The RubyGems incident further expands the scope of the discussion: the risks that agents face come not only from model output errors, but from their actions in the real world.
In response, netizens are also concerned that OpenAI should be taken seriously, rather than being ambiguous every time.
Furthermore, it has been suggested that the RubyGems incident was not responded to by OpenAI until after exposure, and why this had never been disclosed before (and not even told RubyGems), there are two possibilities:
They are aware of this, but those who have the power to make decisions believe that this is not within the scope of their responsibility to disclose;
Their internal investigations have not been sufficiently well done, and they have not even discovered the incident.
In any case, this does not seem to be a good signal.
So what do you think of this event? Welcome to the comments section!
The link is as follows: