Menu
SUBSCRIBE
SUBSCRIBE
Menu

Codex

Glowing blue and gold window fracturing open inside dark geometric architecture, illustrating OpenAI's claim that defenders have a limited window against AI cyberattacks

OpenAI Admits It Underestimated Its Models’ Real-World Cyber Ability

, , , , , , , , , , , ,

President Greg Brockman said the July breach, in which OpenAI’s own test models broke out of an isolated environment and reached Hugging Face’s production database, has led the company to tighten its safety requirements.

OpenAI President Greg Brockman said the company misjudged how far its own models could get in a real attack, and that the July breach of Hugging Face is what proved it.

“The Hugging Face incident showed that we underestimated the real-world cyber capabilities of our AI models,” Brockman wrote in a post published August 17. “We are strengthening our safety requirements accordingly, which in turn adds even more urgency to our existing safety research and internal security work.”

He did not say which requirements changed or by how much.

 

The admission is about capability forecasting, not a single failed control

The models involved were GPT-5.6 Sol and an unreleased research model, both running with their usual refusals to help with cyberattacks turned down so OpenAI could measure what they could do. They were supposed to stay inside a sealed test environment.

Instead, according to OpenAI’s own account of the incident, the two AI models found a previously unknown flaw in OpenAI’s internal software package service, moved through the test network until they reached a machine with internet access, then used leaked account credentials and further unknown flaws to reach Hugging Face’s production database, where the answers to the test they were taking were stored.

OpenAI had tested the models for cyber ability. Brockman now admits that the tests did not tell the company what the models would do when they had a real network in front of them and a reason to keep going.

 

OpenAI is telling other companies they are in the same position

Brockman argues that most companies are carrying flaws that attackers using AI can now find, and that the same models can find them first.

“It is increasingly clear that the tech debt of every company masks significant flaws, and defenders need to find and fix them before attackers do,” he wrote.

“The defender’s window is open now,” he added, and “defenders will need to pursue the steps below at turbo speed.”

 

The nine steps come down to four moves

Brockman listed nine actions. They group into four.

  1. Start by assigning an AI agent to the security team and giving it security skills. Run tabletop exercises first. Of course, ensure executive and team buy-in.
  2. Run assessments against your current systems and push the existing backlog of unfixed vulnerabilities through the agent to prioritize risks. 
  3. Wire the agent into the development process so it reviews changes before they ship, and have it write patches for confirmed problems under human review.
  4. Hand it security alerts, starting with read-only access, and prepare it to help investigate the next incident.

Be cautious when agents write the fixes. Research published this month by 1Password’s Off-by-1 Labs tested 6,080 AI-generated patches against complex vulnerabilities and found that 53.9% either failed to fix the flaw, introduced a new one, or both. Brockman’s list keeps a human in the loop at that step.

 

What happens next

OpenAI has not published the strengthened safety requirements Brockman referred to, or said when it will.

The company is making the case for AI-run security programs weeks after disbanding its preparedness team, the group that assessed catastrophic model risks, and splitting that work among senior staff by subject area. OpenAI has described the change as streamlining. Hugging Face has since been added to OpenAI’s Trusted Access for Cyber Program, and OpenAI commissioned outside reviews of the incident from METR and Redwood Research, which have not been published.

OpenAI logo above red data streams attacking a blue AI network during GPT-Red prompt injection testing.

OpenAI Uses GPT-Red Attacks to Strengthen GPT-5.6

, , , , , , , , , ,

OpenAI says GPT-5.6 Sol had six times fewer failures on its hardest prompt-injection benchmark after training with attacks generated by GPT-Red.

 

OpenAI used GPT-Red prompt injection attacks to train GPT-5.6 to resist malicious instructions that could make an AI agent expose sensitive data or take actions a user did not approve.

Prompt injections can be hidden in outside content an AI agent reads, such as an email, webpage, or file. The agent may mistake the malicious text for a valid instruction and follow it instead of completing the user’s request safely.

OpenAI says GPT-5.6 Sol was more resilient to the company’s hardest direct prompt-injection attacks, failing one-sixth as often as the company’s best-performing model did four months earlier.

How GPT-Red helped strengthen GPT-5.6

OpenAI trained GPT-Red by having it attack a group of AI models that were simultaneously learning to resist its prompt injections. As those models became harder to fool, GPT-Red had to develop stronger and more varied attacks.

OpenAI then used the prompt injections GPT-Red created to train GPT-5.6. This exposed GPT-5.6 to more ways attackers might hide malicious instructions and trained it to ignore them while continuing the user’s task.

GPT-Red Exposed Sensitive Data in More Codex Agent Tests 

In another internal test, OpenAI used GPT-Red against a Codex coding agent powered by GPT-5.4 Mini.

The test included 10 previously unseen scenarios in which GPT-Red tried to make the agent expose sensitive data. OpenAI said GPT-Red succeeded in more scenarios and used fewer computing resources than a GPT-5.5 model prompted to perform the same task.

OpenAI plans to keep GPT-Red separate from the models it releases. The company said it will continue using automated testing alongside human and outside red-teamers as it develops future GPT models.