ai risk today logo svg bl font
Menu
SUBSCRIBE
SUBSCRIBE
ai risk today logo svg bl font
Menu

Open-Weight AI Models

Anthropic CEO Dario Amodei speaking onstage at TechCrunch Disrupt 2023.

Anthropic Rejects Open-Weight AI Ban, Calls for Mandatory Safety Tests

, , , , , , , , , ,

Anthropic CEO Dario Amodei says the most capable open and closed models should be tested before release, regardless of where they were developed.

 

Anthropic says the United States should not ban open-weight AI models, publicly available AI systems anyone can use, and should require mandatory safety testing for the most capable AI models, whether open or closed, before release.

Anthropic’s position differs from parts of a July 24 open letter signed by 77 AI companies that urges U.S. policymakers to preserve access to open-weight models, arguing that they increase competition and give companies more control over their AI systems. Anthropic and OpenAI were noteworthy exceptions that did not sign the letter.

Amodei said he agrees with those benefits. However, he disputed the letter’s argument that broad access to powerful models necessarily helps defenders more than attackers.

Why Anthropic Wants Mandatory Testing

Once a developer releases a model’s core files, it cannot recall every copy or monitor how the model is being used. People can also remove its safeguards and run modified versions privately.

Anthropic argues that this loss of control creates additional risk when a model becomes capable of assisting with attacks.

Mandatory safety testing would examine those capabilities before release. Anthropic wants the requirement to cover the most capable open and closed models, rather than treating open-weight AI models as a separate category.

Amodei advocates that less capable models wouldn’t need to adhere to the safety requirements. He did not specify the level of capability that would trigger testing, who would conduct it, or how the requirement would be enforced.

Anthropic Proposes Other Restrictions

Anthropic also wants the United States to keep powerful AI chips and chipmaking equipment out of China while doing more to stop smuggling and other ways companies evade existing restrictions. Amodei argues that China cannot build models more powerful than leading U.S. systems without access to advanced U.S. chips.

Separately, Anthropic called for stronger action against large-scale model distillation. The company says Chinese AI developers can use thousands of fraudulent accounts to collect millions of responses from advanced U.S. models, then use those responses to improve their own systems faster and at a lower cost than developing the same capabilities independently.

Anthropic argues that this process allows Chinese developers to make better use of their limited computing resources, partially weakening the effect of chip restrictions. However, large-scale distillation still requires advanced chips, so Anthropic says tighter chip controls would also limit how extensively companies can use the technique.

Nebulous digital figure amid blue and red distortion representing a poisoned open-weight AI model.

Researcher Poisons Open-Weight AI Model for Less Than $100

, , , , , , , , ,

A Semgrep researcher used just 10 poisoned training examples to make the model insert a flaw that could give attackers control of a company’s servers or other systems running the vulnerable code.

Cybersecurity researcher Katie Paxton-Fear of Semgrep poisoned an open-weight AI model for less than $100 in about an hour, showing how cheaply attackers could add hidden behavior to models that companies download and run themselves.

An open-weight AI model makes its core files available for people or companies to download and modify. Paxton-Fear took an existing open-weight coding model and trained it on a small set of new examples, creating an altered version of the model.

She first taught the altered model to use a different JavaScript coding style. After that worked, she used 10 malicious examples to make it generate code containing a security flaw.

Ten Examples Changed the Model’s Code

Paxton-Fear said just 10 malicious training examples caused the model to repeatedly produce code with a serious security flaw. If a company used that code, an attacker could potentially take control of the servers or other systems running it.

The model also produced the security flaw when responding to prompts it had not seen during training. This showed that the change was not limited to repeating the 10 malicious examples. 

The experiment did not involve compromising a model already used by businesses. Semgrep said there is no public evidence that widely used open-weight models have been deliberately poisoned.

The demonstration instead showed how someone distributing a modified model could quietly add harmful behavior before another company downloaded it.

Hidden Changes Are Difficult to Detect

Companies can inspect traditional software for malicious code, trace where it came from and check whether it has been altered. Paxton-Fear and her Semgrep colleagues said companies cannot yet examine AI models with the same confidence.

A poisoned model may appear to work normally during routine use and testing, but produce insecure code, biased recommendations or other harmful results when it receives particular prompts. Reviewing the model’s files does not provide a clear account of how it will behave.

Standard performance tests also may not expose the problem. A poisoned model can be adjusted to pass those tests while retaining hidden behavior.

Semgrep called for better records showing where models and their training data come from, methods for reproducing how models were built, and testing by independent organizations. The researchers said companies will need outside auditors capable of checking model developers’ claims, similar to the independent security testing used for traditional software.

Those safeguards can help companies limit which models they trust and test the systems before use. However, Semgrep said no reliable method currently exists for proving that an open-weight AI model does not contain a hidden backdoor.

Dark blue circuit board with glowing nodes depicting AI-driven cyber crisis, RAND Europe tabletop exercise report

RAND Exercises Find Governments Unprepared for AI-Driven Cyber Crises

, , , , , , , , , , ,

European policymakers concluded governments need clearer crisis thresholds, stronger AI governance, and better cyber preparedness before a major AI-enabled attack occurs.

Governments across Europe are not adequately prepared to respond to an AI-driven cyber crisis, according to a new RAND Europe report based on tabletop exercises involving senior policymakers from Germany, the Netherlands, and France. The report, Insights from table-top exercises in Europe on AI safety and cyber misuse, found that officials struggled to determine when AI-enabled cyberattacks should be treated as a national emergency and identified significant gaps in the governance, technical expertise, and coordination needed to respond effectively.

Developed jointly by RAND Europe, the UK AI Security Institute, and Mila – Quebec AI Institute, the exercises placed participants in a realistic crisis scenario set roughly one year into the future. Policymakers were asked to respond after criminals exploited a government-backed AI model to automate phishing campaigns, identify software vulnerabilities, and launch increasingly sophisticated cyberattacks against businesses, public agencies, and critical infrastructure. RAND based the scenario on risks identified in the 2026 International AI Safety Report rather than speculative future technologies.

Governments struggled to recognize the crisis

One of the report’s most significant findings was that participants struggled to determine when an AI-enabled cyber incident should be treated as a national crisis rather than a routine cybersecurity problem. Without clear criteria for declaring a national crisis, officials spent valuable time debating whether extraordinary government powers or interventions were justified, rather than deciding how to respond. Participants ultimately concluded that governments should establish those thresholds before a crisis occurs rather than trying to define them during an emergency.

Preparedness gaps extended beyond cyber defenses

The uncertainty surrounding the crisis highlighted broader weaknesses in government preparedness. Participants concluded that governments need stronger, independent technical expertise to evaluate advanced AI models rather than relying on information from AI companies. They also said existing institutions often separate AI oversight from cybersecurity, even though the two are becoming increasingly interconnected.

The report also found that many governments lack a clear understanding of how vulnerable critical infrastructure and public systems are to AI-enabled attacks. Participants recommended conducting systematic reviews of national cyber defenses, improving coordination between agencies, and strengthening the ability to assess AI-related risks before a crisis unfolds.

State-backed AI developers create difficult policy choices

Another recurring theme was how national governments should respond when a major cyber crisis implicates a strategically important domestic AI company. In the exercise, the AI model enabling the attacks was a “homegrown” company that received government backing and public support.

Participants debated whether governments should restrict or investigate such a company, or continue working with it as a partner to strengthen national cyber defenses. They also questioned whether existing laws provide governments with sufficient authority to intervene when AI developers disclose uncertainty about potential risks but no laws have been violated.

Open-weight AI adds another layer of risk

The second phase of the exercise introduced an open-weight AI model with advanced cyber capabilities that could be freely downloaded, modified, and redistributed. Unlike the original model, its developer could no longer monitor how it was being used or add new safeguards after release. Participants concluded that this would make it much harder for any single country to prevent misuse.

Because these models can quickly spread across borders, participants said governments would need to work more closely with allies. They called for faster information sharing, stronger coordination among trusted partners, and international frameworks that can respond more quickly than traditional diplomacy during an AI-enabled cyber crisis.

Preparing before the next AI crisis

Rather than focusing on how governments should respond during an AI-enabled cyber emergency, participants emphasized the need to prepare before one occurs.

Among their priorities were:

  • Establishing clear criteria that define when an AI incident becomes a national crisis.
  • Strengthening cyber defenses against AI-enabled attacks.
  • Building independent technical expertise to evaluate advanced AI systems.
  • Improving crisis communications and information sharing.
  • Expanding international coordination and cooperation with AI developers.

Participants said these capabilities would help governments respond more quickly and effectively to future AI-related crises.

RAND said it plans to expand the exercises to additional regions to determine whether governments with different regulatory systems identify similar preparedness challenges and governance needs.