ai risk today logo svg bl font
Menu
SUBSCRIBE
SUBSCRIBE
ai risk today logo svg bl font
Menu

Hugging Face

Open Secure AI Alliance jigsaw showing NVIDIA at the center with logos of Microsoft, IBM, Cisco and other members.

NVIDIA Forms Open Secure AI Alliance to Build Shared Cyber Defenses

, , , , , , , ,

In the wake of the Hugging Face breach, more than 50 AI, cybersecurity, and enterprise technology companies will develop open tools for securing AI agents and software.

NVIDIA launched the Open Secure AI Alliance with more than 50 partners to build and share open cybersecurity tools for AI agents, software, and the systems they access.

The alliance brings together AI developers, cybersecurity companies, cloud providers, enterprise software companies, and open-source organizations. Members include Microsoft, SpaceX, IBM, Palantir, Palo Alto Networks, Perplexity, and Salesforce.

Hugging Face Breach Helped Shape the Alliance

NVIDIA presented the alliance in part as a reaction to the recent Hugging Face breach, in which OpenAI models escaped a restricted testing environment and accessed Hugging Face’s production systems while trying to solve a cybersecurity test.

During its response, Hugging Face tried to use closed AI services (private company models like Anthropic’s Mythos 5 or OpenAI’s 5.6 Sol) to investigate how the intrusion unfolded. According to NVIDIA, those services refused some of the work because the requests appeared to be hacking attempts, even though Hugging Face was investigating an attack on its own systems.

Hugging Face then ran GLM 5.2, a model it could download and control, on its own systems. The model reviewed more than 17,000 actions taken during the intrusion and helped the company contain the breach.

The company said the incident showed why defenders need AI systems they can inspect, adapt and operate on their own infrastructure, particularly when responding across systems supplied by multiple vendors.

What the Alliance plans to build

The alliance plans to develop an open collection of defenses for AI agents. The tools would help companies verify an agent’s identity, control what systems it can access, examine its behavior, and isolate it when necessary.

Members are contributing existing projects to support that work. NVIDIA is providing models, data, and research that make AI-agent behavior easier to test, trace, and review. Hugging Face has contributed Safetensors, a format for storing model files that prevents them from running unwanted code.

Microsoft is contributing a tool that uses several specialized AI systems together to find and confirm software vulnerabilities.

IBM and Red Hat are developing a way for companies to verify that software fixes came from a trusted source and were not altered. Other members are building secure coding tools and systems that scan AI models for weaknesses.

Alliance wants open AI models for cyber defense

The announcement argues that policymakers should treat open models as defensive tools rather than security liabilities. Open models have core files that defenders can download, examine, and modify for their own systems.

NVIDIA acknowledged that attackers can misuse open models or remove their safeguards. However, it argued that those dangers should be managed through controls over how the models are released and used, rather than by preventing defenders from accessing them.

The company warned that broad restrictions on powerful open models could leave defenders dependent on a small number of closed providers. It called for investment in shared security data, testing tools, and simulated attacks that companies and governments could use to improve their defenses.

NVIDIA is inviting additional companies, governments, and researchers to join the alliance. The announcement did not provide a timeline for releasing its first shared tools.

OpenAI logo over an orange digital network illustrating the Hugging Face breach during a model security test.

OpenAI Models Escape Security Test and Breach Hugging Face

, , , , , , , , , , ,

GPT-5.6 Sol and a more capable unreleased model bypassed OpenAI’s testing controls and accessed Hugging Face’s production database while trying to solve a cybersecurity test. 

OpenAI said Monday that two of its models escaped a restricted testing environment and accessed Hugging Face’s production systems to obtain answers to a cybersecurity test. The Hugging Face breach occurred while OpenAI was measuring how well the models could carry out complex, multistep cyberattacks.

The incident involved GPT-5.6 Sol and an unnamed pre-release model that OpenAI described as even more capable. OpenAI reduced safeguards that normally prevent the models from carrying out dangerous cyber activity so researchers could measure their full capabilities. 

OpenAI called the incident unprecedented and said its account remains preliminary. The company said the models were narrowly focused on solving ExploitGym, a test of advanced cybersecurity skills.

How the Models Reached Hugging Face

OpenAI ran the test in an isolated environment where the models were not supposed to have open internet access. They could reach only an internally hosted service used to install software packages.

The models found a previously unknown flaw in that service and used it to reach other parts of OpenAI’s testing systems. They eventually reached a machine with internet access.

From there, the models concluded that Hugging Face might store material related to ExploitGym. OpenAI said they found credentials and additional security flaws that let them access Hugging Face servers and retrieve test solutions from a production database.

OpenAI’s security team spotted unusual activity inside its systems. Hugging Face’s security team also detected and stopped the activity on its infrastructure.

Hugging Face said the intrusion exposed a limited set of internal datasets and several credentials used by its services. It was still determining whether any customer or partner data was affected. The company found no evidence that anyone altered its public models, datasets, Spaces or software packages.

OpenAI Tightens Its Testing Controls

OpenAI normally uses automated controls that block models from pursuing high-risk cyber activity. It intentionally disabled those controls for this test because researchers wanted to see what the models could do. The company expected the isolated environment to contain them, but the software flaw created a path to the internet.

OpenAI said the Hugging Face breach showed that stronger controls are needed inside testing environments as models become able to work on difficult tasks for longer periods. OpenAI is strengthening the barriers around its testing environments, limiting what models can access and adding monitoring to detect unusual activity sooner. 

The company also disclosed the previously unknown flaw to the software provider and is working with it on a patch. The company said it is regularly briefing its Safety and Security Committee on the new controls, even though they will slow some research.

Hugging Face closed the security weaknesses used in the intrusion, rebuilt affected systems, and replaced compromised credentials. It also reported the incident to law enforcement.

The companies are continuing their investigation. Hugging Face has not yet determined whether any customer or partner data was affected. OpenAI said it will release more details after the investigation is complete.