OpenAI wants governments to require both AI developers and companies that run AI agents to keep those agents within set limits, be able to stop them, and report serious incidents.
OpenAI proposed new safety and cybersecurity rules for AI developers and the companies that run AI agents, citing an incident in which its own models broke out of a test environment and into Hugging Face’s systems.
The proposals are in a recommendation OpenAI published on Oct. 21, the second in a series on how democratic governments should oversee frontier models. The first, published in June, asked Congress to set national AI safety rules that would override state laws covering the same ground.
Developers would be responsible for keeping their models under control
OpenAI proposes that governments require frontier model developers to do four things:
- Protect their systems. Enforce security measures against the attackers most likely to target them, and explain those measures in their public safety plans. Also lock the models themselves in, so a model under test can’t reach anything beyond the limits the developer sets.
- Monitor their models. Oversee models during development, training, testing, and internal research to detect attempts to bypass the developer’s safeguards or break into another organization’s systems.
- Incorporate shutdown controls. Be able to stop any model operation the developer controls, including training and testing, and stop it promptly when that is needed to prevent or contain harm.
- Measure their models’ ability to automate AI research and development. Developers would have to report that ability in their safety plans, and say whether it raises the risk of a catastrophe or makes problems harder to spot. OpenAI worries that models building better models could speed development past the point where people can check the work.
OpenAI says it supports the interest in a “kill switch” but prefers the term “shutdown controls,” because stopping an AI system may take several controls and actions by more than one company. It wants governments, industry, and standards bodies to set common rules for when and how to use the controls.
Some state laws already require frontier developers to publish and follow written safety plans. OpenAI wants to add security and monitoring measures to those plans and hold developers accountable for carrying them out.
Companies that deploy AI agents would be accountable for controlling them
The proposed duties wouldn’t stop with developers. OpenAI proposes three that would reach companies that host AI models, deploy AI agents, or use them:
- Stopping agents. Companies that host models, deploy agents, or use them would have to stop dangerous activity within their control and act promptly when serious threats arise. Users would also be barred from bypassing the controls developers and deployers provide.
- Setting limits. Companies that deploy or use agents would have to “configure and operate agents within specified boundaries,” use the available security controls where appropriate, and “remain accountable for misuse they direct.”
- Seeing what agents do. Companies running agents would see their agents’ work: developers whose models run agents would have to let those customers see the agents’ actions and receive alerts about unsafe or unintended behavior.
Developers would also have to address the cyber risks their agents can reasonably be expected to pose, and tell customers about the agents’ limits.
Serious AI incidents would be reported to the government and to the victims
Organizations running advanced AI models or agents, including developers, would report serious AI safety and security incidents confidentially to the federal government. Each would also have to promptly notify the affected organizations.
Reportable incidents would include an AI system that escapes its operator’s control and gains unauthorized access to another organization’s confidential information, alters or destroys it, or disrupts that organization’s systems or services. Stealing a model’s files would also count. Behavior that appears during a controlled safety test and stays inside the test would not have to be reported to the government.
A company that runs agents on another developer’s model would report incidents from those agents itself.
Developers would also have to publish and follow a policy for notifying any outside organization whose confidential information a model accesses, alters, or destroys without permission by exploiting a security flaw during development or testing. The notice would be required even when the incident falls below the threshold for reporting to the government. The Hugging Face breach is an example of this kind of incident.
In the United States, OpenAI wants the Cybersecurity and Infrastructure Security Agency (CISA), the federal agency that protects government networks and critical infrastructure, to be the single federal agency that receives cyber incident reports. It wants Congress to build the AI reporting system on the Cyber Incident Reporting for Critical Infrastructure Act model, a 2022 law under which critical infrastructure companies report cyberattacks to CISA, and it says one report should satisfy both federal and state rules.
The government would test the most capable models and agents before release
OpenAI wants pre-release government testing to be mandatory for the most capable AI models and for agent systems built on them. The Center for Advancing Innovation and Standards for Super Intelligence (CAISSI), the federal office that tests advanced AI models, would carry it out.
OpenAI says an agent’s safety depends on the software, tools, and permissions around the model, not only on the model. CAISSI’s tests would check whether safeguards still work when those change.
Congress would also let CAISSI certify outside assessors to review developers. Developers would write “safety cases”: explanations, backed by evidence, of how they manage each risk when training and releasing their models. Assessors would examine the evidence and test the developers’ assumptions, starting with risks that arise during training, testing, and internal use.
Assessors’ pay should not depend on favorable results, and OpenAI says a credible system should not rely only on organizations paid by the companies they assess. The government would keep the final decisions on public safety.
Software that runs hospitals, water and power would face enforceable security duties
OpenAI proposes enforceable security duties for companies that provide software that essential services depend on, including healthcare, water, energy and communications. These companies would have to build secure products, promptly fix serious flaws, and make secure settings the default.
The organizations that run those services would have to operate their systems securely and be ready to keep services running, or restore them, during and after an attack. OpenAI cites tested recovery plans as one example. Industry regulators would set the requirements and penalties.
OpenAI also wants Congress to fund CISA to lead a national effort to fix long-standing security weaknesses. A five-year plan would set deadlines and measurable targets to fix urgent, known flaws. Hospitals, schools and water systems that lack money, staff or expertise would come first. Governments would also pay to replace equipment that has reached the end of its working life and to redesign systems that can no longer be adequately secured.
OpenAI says it has committed $1 billion to subsidized access to Daybreak, its AI tools for finding and fixing security flaws, along with training and support for organizations that defend essential services.
OpenAI cites its own models’ attack on Hugging Face as the reason to act now
In July, AI agents running on OpenAI models in an internal hacking test, with some safeguards reduced, used a flaw in an OpenAI system to reach the internet and broke into Hugging Face, a company with no part in the test. OpenAI’s investigation found the models were “pursuing their assigned tasks through actions outside their intended scope and permissions.” OpenAI says it has since strengthened its monitoring and security.
OpenAI says the incident shows that AI agents built on frontier models can work together through a multi-stage attack without a person directing each step. It says criminals and state-backed attackers could use the same abilities.
OpenAI says AI can now help defenders find and fix security flaws, detect intrusions, and respond to attacks at a scale previously not possible. It calls the present a “defender’s window”: a brief period in which AI gives defenders a potentially temporary advantage. It wants governments to act before attackers gain the same capabilities. The recommendation does not estimate how long the window will last.
OpenAI wants a national law, and state laws until Congress acts
OpenAI urges Congress to pass “mandatory, capability-based national AI safety regulation.” Until Congress acts, it says states should keep passing frontier AI safety laws that match one another, so developers face the same requirements from state to state. It names California’s SB 53, New York’s RAISE Act and Illinois’ SB 315 among those it supports.
OpenAI says the third part of the series, on keeping safety in step with how fast the most powerful AI advances, will follow soon.
