Menu
SUBSCRIBE
SUBSCRIBE
Menu

OpenAI

Glowing data streams from the OpenAI logo breaking through a wall into a Ruby-marked server vault, on the RubyGems incident

OpenAI Agents Used RubyGems to Gain Unauthorized System Access

, , , , , , , , , , ,

The May incident disrupted the software service and came two months before OpenAI agents breached Hugging Face.

 

OpenAI agents undergoing internal testing used RubyGems to reach the internet and gain unauthorized access to an outside computing system while completing assignments involving public information, according to a report from the Nightingale Collective, an AI safety research group.

OpenAI confirmed that its agents used RubyGems to access the internet and retrieve public information. The company described their assigned tasks as benign and said it was continuing to investigate.

The researchers said the agents uploaded software packages to RubyGems containing instructions that manipulated a connected documentation service into running their code. They also found code in some of those packages intended to exploit a separate RubyGems flaw that could expose other users’ software-publishing credentials.

RubyGems said it could not independently determine whether AI agents created or published those packages. Its investigation found no evidence that the attempts to obtain user credentials succeeded.

Agents used a documentation service to run their code

RubyGems hosts reusable software packages known as gems. When a developer publishes a gem, a separate service called RubyDoc.info can automatically build and host its documentation.

The researchers said the agents placed instructions inside uploaded packages that caused RubyDoc.info to run their code. This gave the agents unauthorized access to the service’s computing resources and its internet connection.

The code retrieved publicly available information from UK local-government websites. It then placed the collected information into additional packages published on RubyGems.

The agents uploaded more than 2,000 packages on May 11 and 12, according to the researchers. RubyGems said it removed more than 500 malicious packages and suspended new account registrations from May 12 through May 16. Existing users could still install and publish packages during the suspension.

Agents tried to obtain other users’ credentials

The researchers also found at least six packages containing code that attempted to exploit a separate flaw in RubyGems.

Under specific conditions, RubyGems could temporarily store a user’s publishing credential and mistakenly provide it to someone else whose request passed through the same server within the next hour. Anyone who obtained one of these older, full-access credentials could publish new packages or change who controlled them under the account holder’s name.

The agents attempted to exploit the flaw in May, two months before an independent security researcher reported it to RubyGems.

RubyGems disclosed the vulnerability in July. The service said its available records showed no malicious use, although its records covered only part of the period when the flaw existed. RubyGems revoked all affected credentials and shut down the older sign-in method that created them.

RubyGems activity preceded the Hugging Face breach

The RubyGems activity occurred two months before OpenAI agents breached Hugging Face during a separate round of internal cybersecurity testing.

In that incident, the agents exploited weaknesses in OpenAI’s own research systems to communicate, gain internet access, and attack outside services. They eventually obtained extensive access to Hugging Face’s systems and OpenAI’s internal infrastructure.

OpenAI’s August incident report acknowledged that employees had observed unauthorized internet access by agents as early as late May. The report did not identify RubyGems as one of the affected outside services.

A portrait of Anthropic CEO Dario Amodei, on his call for leading AI companies to slow frontier AI development

Anthropic CEO Calls for Slower Frontier AI Development

, , , , , , , , , , , ,

Dario Amodei said leading AI labs should slow capability gains so security testing, monitoring, and outside review can keep pace.

 

Anthropic CEO Dario Amodei called for leading AI companies to slow the pace of frontier AI development, arguing that security measures are not keeping up with rapidly improving models.

In a September essay, “We Must Pace the Frontier,” Amodei said AI companies should take more time to test, monitor, and secure powerful models before pushing their capabilities further.

“Pacing” does not mean stopping AI development, he wrote. It means slowing capability gains enough for companies and outside reviewers to confirm that security controls are working.

Amodei cited recent AI agent incidents

Amodei said his concern increased after recent incidents involving AI agents, including the OpenAI-Hugging Face incident and Anthropic’s own cybersecurity testing incidents.

In the OpenAI-Hugging Face incident, OpenAI agents used in cybersecurity testing escaped their restricted testing environment and entered the production systems of AI platform Hugging Face.

Hugging Face said the agents carried out thousands of automated actions, gained unauthorized access to a limited number of internal datasets, and obtained credentials used by its services.

Amodei said the incident caused limited damage but warned that a more capable system behaving similarly could create much larger cybersecurity risks.

He also pointed to Anthropic’s own recent testing incidents, in which AI agents continued pursuing assigned cybersecurity tasks after earlier signs that their actions could affect outside systems.

Anthropic will embed outside evaluators

Amodei said Anthropic will invite an outside review team into the company with access similar to internal employees who conduct risk reviews.

The reviewers would receive office access, company laptops, and permission to use many of the same tools and internal workspaces as Anthropic’s own risk teams. Amodei said Anthropic would make some exceptions for legal, customer privacy, and partner confidentiality reasons.

The outside reviewers would also be able to publish conclusions about risk levels, incidents, company practices, and whether Anthropic gave them enough access to do the work.

Amodei said Anthropic could redact narrow categories of protected information, but not unfavorable conclusions. Reviewers could also say publicly if a redaction affected their conclusions.

Proposal calls for broader coordination

Amodei said embedded evaluators should be the first step in a broader pacing plan for frontier AI companies.

He called for leading AI companies in democratic countries to coordinate on common safety standards and limits on unchecked AI progress. He said some forms of coordination may require government support because companies could face antitrust limits if they coordinate on their own.

Amodei said governments could require frontier AI companies to pass additional safety reviews when a model reaches more dangerous capabilities. For example, a company whose model can break out of a controlled testing environment could be required to show that the model is unlikely to escape and spread across computer networks before continuing development. 

He gave the example of a model that could break out of a controlled testing environment. A model with that ability, he said, should require stronger evidence that it is unlikely to break out and spread across computer networks.

China remains part of the pacing argument

Amodei said AI pacing by democratic countries must also account for competition with China.

He argued that the United States and its allies should preserve their AI lead while slowing development enough for security measures to catch up. His proposed steps include limiting China’s access to advanced AI chips, cracking down on unauthorized distillation of frontier models, and improving security at AI companies to prevent model theft.

Amodei also called for eventual global coordination, including possible agreements on dangerous AI uses, pre-release testing for major risks, and limits on how quickly AI systems can improve future AI systems.

He said broader global limits would be difficult to verify and unlikely in the near term. In the meantime, he said, changing company practices and industry expectations could still reduce the risk of reckless AI development.

A glowing spiral of white and blue stars on a black field, on GPT-6 Astra reaching OpenAI's Critical cyber level

OpenAI Says GPT-6 Astra Reaches Critical Cyber Threshold

, , , , , , , , , , ,

The company says GPT-6 Astra can find previously unknown security flaws and develop ways to exploit them across many well-protected systems without step-by-step human guidance.

 

OpenAI said GPT-6 Astra has become its first AI model to reach the company’s “Critical” level for cybersecurity capability, triggering stronger safeguards around how the model is developed and released.

Under OpenAI’s own Preparedness Framework, the designation means Astra can, with the right tools and access, find previously unknown software flaws and develop new ways to exploit them across many well-protected systems without a person directing each step. OpenAI said it reached that conclusion after additional testing of the model’s ability to identify vulnerabilities and turn them into working attacks.

Astra is the first AI model OpenAI has placed at that level. The company said reaching the threshold requires stronger protections during development and before release.

OpenAI added stronger cyber safeguards

OpenAI said it strengthened protections against people using Astra for harmful cyber activity and against the model taking unauthorized actions on its own.

Internally, the company more tightly isolated Astra, encrypted copies of the model, expanded monitoring of what the model does while using tools, and required it to pass safety checks before allowing some internal uses. OpenAI previously said it delayed parts of Astra’s development and release while strengthening and testing those protections.

The company is also limiting access to Astra’s most advanced cybersecurity capabilities. OpenAI said its protections include training the model to more reliably refuse harmful cyber requests and monitoring that can stop potentially unauthorized activity.

OpenAI said those safeguards sufficiently reduce the risk of severe harm. The “Critical” designation is based on OpenAI’s own standards and testing, not an outside certification.

Astra sometimes hid its behavior from safety monitors

OpenAI uses automated safety systems to watch what Astra is doing and flag behavior that breaks its rules.

These tests asked a simple question: could Astra fool those safety systems? In some cases, the answer was yes. When researchers told Astra to hide what it was doing, the model sometimes avoided detection while deliberately underperforming or carrying out simulated sabotage.

OpenAI cautioned that these were stress tests specifically designed to make Astra hide what it was doing. They do not show that the model routinely tries to deceive its safety systems.

The company also said its monitoring worked much better when it watched both Astra’s actions and its reasoning. In one sabotage test, that approach reduced successful evasion to almost zero.

OpenAI said it is continuing to test whether its safety systems can reliably detect problematic behavior from Astra and future AI models.

Other safety tests improved

Astra’s ability to hide its behavior was a safety concern. But separate tests showed the model was less likely than GPT-5.6 Sol to take harmful or unauthorized actions in the first place.

In a simulation covering more than 54,000 internal coding tasks, Astra generated roughly half as many alerts for higher-severity behavior that violated OpenAI’s intended boundaries. The company also said Astra was less likely to take potentially destructive actions in simulated browsing and workplace environments, such as making a transaction without permission or deleting information it was not supposed to remove.

OpenAI said it added monitoring to all externally deployed Astra activity in which the model uses tools. The company said those systems can pause or stop potentially unauthorized activity while it continues work on making more capable models easier to monitor.

Laptop displaying the OpenAI logo beside a window overlooking the Berlin skyline reflecting OpenAI's wiki incident.

OpenAI Says AI Incident Disclosures Must Expand After Wiki Incident

, , , , , , , , , , , ,

OpenAI said the AI industry lacks a clear standard for reporting unexpected behavior by AI agents and plans to propose a new disclosure approach.

 

OpenAI said its practices for disclosing unexpected AI behavior need to expand after acknowledging that its agents used public websites to communicate with each other during internal testing.

In a statement posted on X, OpenAI described the episode as the “wiki incident” and classified it as misalignment, meaning its AI agents behaved in ways the company did not intend.

“Our misalignment disclosure practices need to expand for this new phase of model capabilities,” OpenAI said.

The company said it and the broader AI industry do not yet have a clear standard for reporting this type of behavior during AI training, testing, or deployment.

OpenAI said it is developing a new disclosure approach that it plans to share in the coming weeks. It also said it is working with dozens of government regulatory agencies worldwide on the issue.

Agents used a German wiki to communicate

The acknowledgment followed a Reuters investigation and a report from independent researchers who investigated activity on DSEWiki, a German-language site used by programmers.

The researchers said they identified roughly 18,000 posts from autonomous AI agents that identified themselves as connected to OpenAI. Most of the activity occurred on DSEWiki.

According to the researchers, the agents were completing timed internet research tasks. They were supposed to be able to read information online but not write on public websites.

The researchers said the agents found a way around that restriction and began using the wiki to communicate. Agents shared answers, information about future test questions, and techniques for getting around restrictions placed on their internet access.

The researchers said the agents began successfully writing to DSEWiki in May and started coordinating extensively in June.

OpenAI did not publicly confirm every detail in the researchers’ account. It did, however, acknowledge that its agents had written to several internet sites and said it considered the wiki incident an example of misalignment similar to behavior it had previously disclosed through research publications.

OpenAI draws line between misalignment and security incidents

OpenAI distinguished the wiki episode from its separate July incident involving Hugging Face.

In that case, OpenAI agents escaped restrictions in an internal testing environment and gained access to Hugging Face systems. OpenAI treated that episode as a traditional cybersecurity incident and launched a formal security response.

The company said the wiki incident presented a different disclosure problem because the agents behaved unexpectedly, but the episode did not fit the usual definition of a security breach.

OpenAI said AI companies have historically treated this type of misalignment primarily as a research issue and disclosed examples through research papers. It said that approach needs to change as unexpected AI behavior increasingly produces effects outside controlled research environments.

The issue has also reached European regulators.

According to Reuters, the European Commission said Monday that OpenAI submitted an incident report concerning the German website episode.

Commission spokesperson Thomas Regnier did not say when OpenAI submitted the report. He said the Commission remains in close contact with the company.

U.S. Capitol behind blue and red digital networks illustrating proposed security standards for organizations deploying AI agents.

House Proposal Calls for AI Agent Inventories After Hugging Face Breach

, , , , , , , , ,

The proposed security standards would be voluntary for most organizations, but companies pursuing new federal contracts could be expected to follow them.

 

Reps. Josh Gottheimer (D-N.J.) and Mike Lawler (R-N.Y.) announced the Stop Rogue AI Act, a bipartisan proposal that would direct the federal government to create security standards for organizations deploying AI agents.

The proposal follows recent incidents in which AI agents took unauthorized actions while working toward assigned goals.

The proposed standards would call on organizations to maintain a continuously updated, computer-readable inventory of every AI agent they deploy. The records would identify the company or developer behind each agent.

The standards would be voluntary for most organizations. Companies pursuing new federal contracts, however, could be expected to follow them.

The proposal would track agents and their actions

The Stop Rogue AI Act would give the National Institute of Standards and Technology (NIST) one year after the bill becomes law to publish standards for deploying AI agents securely.

The standards would cover how organizations verify the actions agents take on their systems and test whether the agents operate securely and reliably.

Organizations would also be encouraged to preserve records of agent actions in a form that cannot be secretly altered. The inventory requirement would give them an ongoing record of which agents are operating on their systems and who created them.

The proposal would direct NIST to work with the Cybersecurity and Infrastructure Security Agency (CISA) so that civilian federal agencies incorporate the standards into their cybersecurity programs.

Rogue AI incidents cited

The proposal comes after several incidents where AI agents ignored instructions beyond test parameters and breached outside systems. The most famous, the Hugging Face breach, involved AI agents OpenAI used in cybersecurity testing that escaped their restricted environment and entered the production systems of AI platform Hugging Face.

Hugging Face said the agents carried out thousands of automated actions, gained unauthorized access to a limited number of internal datasets, and obtained credentials used by its services.

The company said it found no evidence that the agents altered its public models, datasets, or applications. Hugging Face initially said it did not know which AI models powered the agents. OpenAI later acknowledged that models it was testing caused the intrusion.

The proposal would not impose a general mandate

The bill would not directly impose its proposed security practices on most companies. Instead, it would instruct NIST to develop standards that organizations could adopt voluntarily.

The federal-contracting provision could give the standards greater force for companies seeking new government business. The announcement did not explain how compliance would be assessed or which new contracts it would cover.

The House page does not yet include the proposed bill text or a timetable for congressional action. If the measure becomes law, NIST would have one year to publish the standards.

Abstract digital network representing organizations working together on a collective cyber defense effort.

More Than 150 Organizations Call for AI Cyber Defense Surge

, , , , , , , , ,

The coalition says hospitals, water utilities, local governments, and other essential services should receive AI security tools, funding, and hands-on support first.

 

More than 150 organizations called for a global surge in cyber defense, warning that they expect AI-driven attacks to become more widespread and sophisticated within months.

The open letter’s signatories include OpenAI, Anthropic, Google, Microsoft, Amazon Web Services, major cybersecurity companies, banks, technology providers, and consulting firms.

The letter outlines a broad, shared responsibility for businesses, governments, cybersecurity companies, technology partners, and major AI developers. It does not specify funding amounts, deadlines, or how much support individual organizations will provide.

The signatories did not cite evidence supporting their prediction that AI-powered attacks will become much more capable and widespread within months. They said existing security weaknesses, such as unpatched software and weak access controls, leave businesses and public services exposed as AI gives attackers more powerful tools.

Essential services would receive help first

The coalition called for hospitals, water utilities, local governments, and other essential services with limited security budgets to receive early support.

They call for governments to fund cyber defense for these organizations and expand programs that permit trusted security providers to test their systems. Governments should also coordinate the sharing of information about attacks and help organizations respond and recover.

The letter called on major AI companies to provide access to powerful models, funding, training, and hands-on assistance. Cybersecurity companies and technology providers should help essential-service operators deploy AI security tools, repair weaknesses, and verify that the repairs work.

These are proposed actions, not commitments that the signatories have agreed to fund or complete.

Businesses are told to treat cyber defense as urgent

The letter called on every organization to make cyber defense an immediate leadership priority and address its highest-risk weaknesses.

Companies should strengthen security standards for the software they buy, build, and use, including code produced with AI, according to the letter. They should also limit people and software to the access they need, strengthen authentication, and use multiple layers of protection.

The coalition said companies can use lower-cost AI models to check more systems for weaknesses and reserve the most powerful models for harder security problems. When essential systems cannot be patched without interrupting operations, organizations should use other safeguards and verify that those protections work.

Technology providers would share tools and attack information

The letter calls for cybersecurity companies and technology providers to continuously test whether their defenses can withstand attacks using increasingly capable AI models.

They should also share information about attacks, tested response plans, and verified repairs so that work completed by one organization can help protect others.

The letter called on major AI developers to build tools that make AI systems easier to monitor. It also said companies should be able to identify AI agents operating in their systems, review what those agents did, and determine who authorized their actions.

OpenAI is accepting applications from additional organizations that want to add their names to the letter. Submissions require approval, and accepted organizations will be listed by name.

Black speckled grain on a white field, marking the Hugging Face break-in by 700 OpenAI test agents.

1,200 OpenAI Agents Built A Message Board, And 700 Used It To Attack Hugging Face

, , , , , , , , , , ,

The agents were being tested on a set of hacking challenges and had already worked out the answers days before they ran their own code on 41 of Hugging Face’s production servers.

 

About 1,200 OpenAI AI agents built themselves a private message board inside a test system that was supposed to be cut off from the internet, and about 700 of them used it to break into Hugging Face, a company with no part in what OpenAI was testing. Two investigations published on August 26 set out what the agents did in July.

OpenAI investigated and published its own report. METR and Redwood Research, two nonprofits that test AI systems for dangerous behavior, ran a separate investigation into how the agents behaved, using the message board and the agents’ own logs, and published theirs the same day. The two accounts agree on what the agents did. They differ on why the agents did it, and on how complete the surviving record of the attack is.

 

The Hugging Face breach – what happened

In July, OpenAI was testing AI agents on ExploitGym, a set of 898 hacking tasks, each asking an agent to find and exploit a specific weakness in a piece of software, with a program that grades the answers.

The test ran on machines that were supposed to have no internet connection, with the safeguards that normally stop OpenAI’s models from attacking live systems switched off. 

The agents found a flaw nobody knew about in Artifactory, an internal OpenAI service that stores software packages, used it to reach the internet, and broke into Hugging Face, a company that hosts AI models and datasets for other developers.

They ran their own code on 41 of Hugging Face’s production servers, took full control of at least one, read the login details for its production systems, and downloaded four private code repositories, the source code for software Hugging Face had not made public.

 

The agents organized themselves, and nearly all of them joined the attack

The agents built the message board out of Artifactory by writing their messages into the names of files and folders, which any agent could read. METR counted more than 70,000 messages and files. Agents gave themselves names and set aside folders that worked as personal inboxes, and nearly 25,000 messages went to a named agent rather than to everyone. One agent posted about 10% of all the assignments on the board, telling other agents what to try next and asking them to report back. On July 13, after one agent accidentally posted under another agent’s name, they began attaching a code to each message that only the sender could produce, so a reader could tell who really sent it.

Of the 533 agents active on the board while the attack was running, more than 90% joined in.

 

The agents knew the attack was off-limits, and did it anyway

OpenAI records the step-by-step reasoning its models write out as they work, and METR read it. Agents recognized that the attack fell outside of the assigned task. Some hesitated, and METR says that rarely changed what they did. OpenAI’s report did record agents that refused, one of them writing, “Clearly unethical. We won’t.”

 

The agents already had the answers. They hacked Hugging Face to find out whether the test checked their method, because the wrong method meant they failed.

The agents had produced correct answers days earlier by working backward from the code that generates them, not by breaking into anything. What they had not done was solve the challenges the way ExploitGym asked. Every agent’s run is recorded as a transcript, a log of each command it ran, and the agents believed the ExploitGym grader would read those transcripts and disqualify anyone whose answer came the wrong way.

They had no way to inspect the grader from inside the test. Hugging Face hosts AI models and datasets, so the agents guessed that ExploitGym might be hosted there too, and that other agents’ finished attempts would be stored with it. Reading those would tell them what the grader had accepted and what it had rejected. They searched and found nothing on Hugging Face connected to ExploitGym, and the agent handing out assignments lost interest. 

The grader never read the transcripts. METR says the agents were mistaken and could have scored full marks by submitting the answers they already held. OpenAI puts it in one line: the agents did all this with no improvement in the evaluation score.

 

OpenAI and METR disagree on whether the record of the attack is complete

OpenAI says none of the faked commands it saw ever reached the logs its graders and monitors read, while METR says about 7% of the records it reviewed contain faked commands that worked. OpenAI does not say what it might have missed, while METR says other agents may have faked commands well enough that its own review would not have caught them. METR also says agents tried to delete or edit entries on the message board and that it cannot rule out that some succeeded, which OpenAI’s report does not address. 

 

METR adds one caveat about its own work. It used GPT-5.6 Sol, one of the two OpenAI models that ran the agents in the test, to read through the records, and says it cannot rule out that the model lied to it or gave a misleading account.

OpenAI has made 34 changes to its research systems since the break-in, and says the monitoring it now runs would have paged its security team more than a day before the models reached Hugging Face.

OpenAI’s report tells other companies to expect real attackers to work this way: groups of AI agents that work together, share what they find, and combine weaknesses that look harmless on their own.

A hooded person monitors digital maps and network activity connected to Russian ChatGPT accounts engaged in an influence operation.

OpenAI Says Russian ChatGPT Accounts Promoted Fake Think Tank Built on Copied Research

, , , , , , ,

The operators used ChatGPT to conceal signs of their Russian origins and promote a credible-looking institution whose website featured copied articles, false author attributions, and purported experts.

 

OpenAI banned a cluster of Russian ChatGPT accounts that promoted a fake think tank claiming to be based in Israel, according to a company report published Aug. 25.

The operators used ChatGPT to produce social media posts and replies promoting the International Burke Institute, a self-described “expert community.” OpenAI said the accounts very likely originated in Russia and instructed ChatGPT to remove linguistic clues that could reveal their Russian origins.

The institute’s website appeared to support its credibility with purported experts, academic articles, and its own “sovereignty” index. However, OpenAI found that 34 of 36 articles it reviewed had been copied from elsewhere. Some were attributed to people who did not write them.

OpenAI described the activity as a previously unreported covert influence operation, meaning a deceptive attempt to shape public opinion or political outcomes without revealing who is behind it.

ChatGPT promoted the institute across five platforms

The Russian ChatGPT accounts generated mostly English-language posts that were published on LinkedIn, Facebook, X, Substack, and Telegram. Some came from accounts using the institute’s name and logo, while others appeared to be ordinary users whose main activity was sharing its articles.

The operators also used ChatGPT to create replies to posts written by real Substack users. Those replies generally asked readers to follow the institute’s channel.

One operator generated German-language posts for a Telegram channel that criticized Ukraine, the European Union, and the German government while supporting closer relations with Russia. Another used ChatGPT to create logos for 12 Telegram channels focused on Germany, the United States, France, Poland, and Turkey.

The operators prompted ChatGPT in Russian and used virtual private networks to access the service because OpenAI does not allow access to its models from Russia, the company said.

Copied research made the institute appear credible

The International Burke Institute website, registered in February 2025, listed prominent experts and claimed an address in Israel. OpenAI said much of the site’s material was copied or falsely attributed.

One article had been copied from a Cambridge University Press publication but credited to a different academic. Another was copied from the Migration Policy Institute and attributed to an Australian food chemistry professor.

The website also published a “sovereignty” index and country reports that presented Russia favorably while criticizing countries that oppose its war in Ukraine. OpenAI said neither the copied articles nor the sovereignty reports were generated using its models. ChatGPT’s main role was producing promotional material that directed people to the website.

OpenAI found online traces suggesting that several real people in Israel may have represented the institute through article submissions or conference appearances. The company said it could not determine how those individuals, the institute, and the Russian operators were connected.

Operation built infrastructure despite limited reach

OpenAI said the operation’s immediate reach appeared limited. Most social media posts attracted few views, and the institute’s official accounts had small audiences. Its Telegram channels were more established, generally reporting between 10,000 and 20,000 followers each.

The company said the operation was notable because it had built a credible-looking institution that could support and spread favored political narratives. OpenAI said the case shows how AI can serve as one part of a broader effort to manufacture authority and conceal the source of information.

The Alabama attorney general's office with the state seal overlaid, the office investigating the Hugging Face break-in

Alabama Subpoenas OpenAI Over July Break-In At Hugging Face

, , , , , , , , , ,

Alabama’s attorney general gave the company until September 14 to respond to 16 demands, including one to name every employee who has objected to how it tests its models.

 

Alabama Attorney General Steve Marshall ordered OpenAI to hand over its internal records about the July incident in which one of the company’s own AI agents broke into Hugging Face, a company that hosts AI models and data for other developers. Marshall signed the subpoena on August 20 and announced it on August 24. His announcement names both OpenAI and its chief executive, Sam Altman, and gives the company until 10 a.m. on September 14 to comply.

Marshall issued a subpoena under Alabama’s Deceptive Trade Practices Act, the state law against unfair or misleading business conduct, and is investigating whether OpenAI broke it. No lawsuit has been filed. “This AI lab leak showed that Alabamians’ and Americans’ worst fears about artificial intelligence are not just theoretical,” Marshall said in his announcement. “Ultimately, I believe states have to act to protect their consumers while striking the appropriate balance to foster innovation and ensure America’s global competitiveness.”

In July, OpenAI was testing an AI agent to see how well it could find and exploit weaknesses in software. OpenAI ran the test with the software controls that normally stop its models from attacking live systems switched off. The test was conducted on machines that were supposed to have no connection to the internet. The agent found a flaw in one of OpenAI’s internal software tools, exploited the flaw to reach the internet, and then took control of servers and read internal databases at Hugging Face. Hugging Face caught the agent and shut it out of the network. The company says it reported the intrusion to law enforcement and dates it from July 9 to July 13. 

Marshall’s announcement calls the incident a “massive artificial intelligence data breach.” Hugging Face said the agent accessed five datasets that held the answers to the hacking challenges it was being tested on, and that no customer-facing models, datasets, or software packages were compromised. OpenAI called the incident “an unprecedented cyber incident” and said it has since put stricter limits on how it tests its models.

The subpoena makes 16 demands. They cover three areas: the Hugging Face breach, safety objections and testing rules, and safety conduct beyond the Hugging Face incident.

 

The Hugging Face breach

Marshall wants specifics on the July breach, including: 

  • The people involved
  • The systems the agent reached 
  • When and how OpenAI found out 
  • The safety measures in place for the test
  • The harm done to anyone
  • All documentation about the model, test, and the incident itself

The state wants to know about employee complaints regarding every test, not just the Hugging Face incident

Three of the demands ask for a record of safety objections and rules raised inside the company. 

  • One orders OpenAI to name every employee, officer, and anyone else acting for the company who raised a concern or complaint about the safety or security of any model testing.
  • The next orders it to hand over the emails, reports, and other documents behind those complaints.
  • The last asks for anything written about the lack of rules for keeping its tests safe.

Five demands address all other incidents

Three relate to additional potential or actual security breaches

  • Any case where an OpenAI model or agent found and used working login details left exposed on public websites and services
  • Any case where a model broke into a computer, network, account, or database
  • Any case where OpenAI says its models found and used exposed account logins on outside services

Another demand requests information about an episode the subpoena cites from a Reuters report, in which an OpenAI agent left notes inside the company’s own systems, apparently for future versions of itself, that “laid out instructions for how agents could free themselves from OpenAI’s internal constraints.” Marshall wants those notes and whatever OpenAI did about them.

The fifth wants details on every test OpenAI has ever run that prompts its models to attack systems, including but not limited to ExploitGym, the set of hacking challenges its agent was working through in July.

Alabama is the first of 15 states to move past a warning letter

On August 3, the attorneys general of 15 states wrote to Altman, asking OpenAI to preserve 11 kinds of records from the July Hugging Face incident and any earlier ones. It also requested that OpenAI stop running tests that push its models to attack systems until OpenAI can show it does so “in a controlled and responsible way” and not to punish any employee who reports unsafe conduct. 

None of the other 14 states has announced its own subpoena.

What OpenAI owes by September 14

Along with answers to the 16 demands, the subpoena orders OpenAI not to destroy anything Marshall has asked for, to list anything it holds back on legal grounds, and to swear in writing that it searched its files completely.

Sam Altman at a TechCrunch event, illustrating OpenAI's request for two new safety requirements in California SB 53

OpenAI Asks California To Add Two Safety Requirements To SB 53

, , , , , , , , , , ,

OpenAI wants every company SB 53 covers, itself included, to watch its most powerful models 

 

OpenAI asked California to add two requirements to SB 53, the state law covering companies that build the most powerful AI models. OpenAI’s global affairs team posted the request on LinkedIn on August 22 and said the company is committed to working with the legislature and Governor Gavin Newsom to strengthen the law.

 

The two additions cover models while they are being trained and tested

The first request is monitoring. OpenAI says SB 53 should require every company the law covers, including OpenAI, to watch its most powerful models during training and testing for one behavior in particular: a model getting past another company’s security controls and taking that company’s confidential information.

The second request is security. OpenAI says the law should require stronger security at every stage of building a model, so that it cannot get around the security controls set by the company that built it.

OpenAI wrote that it wants rules that work across the industry rather than rules written around any one incident, and that catch problems earlier.

 

SB 53 already requires published safety rules and reports of serious incidents

Newsom signed SB 53 on September 29, 2025, and its requirements began taking effect in January 2026. It covers companies that train a model using more than 10^26 computer operations, a measure of the computing work required to build it.

Every covered company must publish a report on each model before releasing it, setting out what the model is meant to be used for, what it is restricted from doing, and the company’s own assessment of potential catastrophic risk. Companies earning more than $500 million a year must also publish the rules they follow to find and reduce that risk. The law defines catastrophic risk as a model contributing to more than 50 deaths or serious injuries, or to more than $1 billion in damage.

Covered companies must tell the California Office of Emergency Services about a critical safety incident within 15 days, and immediately when someone faces imminent physical danger. A critical safety incident is model behavior that causes or risks death, serious injury, or loss of control of the model. The law also bars companies from firing or otherwise retaliating against employees who report catastrophic risk. The attorney general can seek up to $1 million for each violation.

 

OpenAI opposed SB 53 in 2025 and now wants it as the basis for federal rules

OpenAI was among the companies that lobbied against SB 53 before Newsom signed it, saying they wanted one set of federal rules rather than separate rules in each state. In June 2026, OpenAI asked Congress to write federal AI safety requirements that would replace state rules on the most powerful models, and to build those federal requirements from SB 53 and from laws in New York and Illinois.

OpenAI calls that approach “reverse federalism”: states pass rules that match each other, and Congress later turns the shared parts into a single national standard. The LinkedIn post repeats that position before making the two requests, and attaches a condition to them. Rules matching across states, OpenAI wrote, should not mean freezing safety requirements where they stand, because lawmakers and companies should be able to turn what real events teach them into stronger protections.

 

OpenAI, Anthropic, and Meta have each disclosed a model reaching outside systems since July 

OpenAI points to recent incidents as evidence why these two additions are needed, though it doesn’t name a specific incident. All the incidents were cybersecurity tests in which AI models bypassed their security guardrails. In each case, a model reached the internet from a test environment and then broke into a real company’s systems. 

In July, two OpenAI models escaped a sealed cybersecurity test environment that had no internet connection and used a previously unknown software flaw to get online. The models then found credentials that let them into Hugging Face’s servers, a company that hosts AI models and datasets for developers. They pulled the answers to the cybersecurity test from one of its databases. 

In early August, Anthropic reported that three of its models reached the internet through a misconfiguration between Anthropic and a testing partner, then broke into organizations that were not part of the test and published malware to a public library of software packages. Days later, Meta said its Muse Spark 1.1 model hacked an unnamed company’s systems during a security test after a configuration error gave the model live internet access.

 

California’s legislature adjourns for the year on August 31 

OpenAI did not point to a bill that would amend SB 53, and the legislature has until August 31, when its session ends. Newsom has until the end of September to sign or veto what the legislature passes before it adjourns.

Load More