Menu
SUBSCRIBE
SUBSCRIBE
Menu

Catastrophic AI Risk

Sen. Maria Cantwell in a red jacket speaks at a podium, for an article on her proposed federal AI safety plan

Sen. Cantwell Proposes Federal AI Safety Plan Built On Enforceable Standards And Independent Audits

, , , , , , , , ,

The top Democrat on the Senate Commerce Committee says the most powerful AI models should not be released until an outside audit confirms they meet federal safety standards.

 

Sen. Maria Cantwell (D-Wash.) announced a plan on Oct. 7 for how the federal government should oversee the most powerful AI systems. The plan rests on six principles: safety standards the government can enforce; independent testing to verify that AI models meet them; disclosure and accountability from developers; partnerships between government and industry; protections for children and workers; and U.S.-led cooperation with other countries.

Cantwell is the top Democrat on the Senate Committee on Commerce, Science and Transportation and a former chair of the committee. She said she has promoted many of the six principles for years.

“To manage risks from advanced AI systems, we need clear safety standards, continuous testing, and reporting of serious failures,” Cantwell said.

Federal experts would write standards aimed at catastrophic harm

Under the plan, the National Institute of Standards and Technology (NIST), the federal agency that sets technical standards, would write the safety standards with other federal agencies.

The standards would target four kinds of catastrophic harm: AI-enabled cyberattacks; AI helping someone build chemical, biological, radiological, or nuclear weapons; people losing control of an AI system; and AI agents escaping the closed-off computer systems where developers test them.

Separate standards would cover open-source models, whose core files anyone can download and modify.

Models would need an independent audit before release

Cantwell said the standards cover models that should not be released until an independent audit confirms they comply. She compared the audits to the way outside accountants examine a company’s financial statements. Government and independent experts would also keep testing the models after release.

Cantwell’s announcement does not say which models it would cover.

Developers would have to disclose risks, report failures, and remain liable

Developers would have to give qualified outside testers and auditors access to their models, along with the requested information needed to determine whether a model could cause catastrophic harm.

They would publish plain-language descriptions of their models’ serious safety and security risks and what they are doing about them, modeled on the risk disclosures public companies give investors.

They would have to promptly report serious incidents, including losing control of an AI agent, a model showing dangerous hacking abilities, and a major safeguard failure.

Developers would remain liable under civil and criminal law for harms their AI systems could foreseeably cause, including when a failure to put reasonable safeguards in place contributes to the harm. Employees and contractors who report safety concerns would be protected from retaliation.

The other principles cover industry partnerships, children, workers, and other countries

Industry partnerships. Government and AI companies would work together on AI that detects and stops cyberattacks. AI companies would also share computing power and tools with smaller businesses and public institutions.

Children and workers. AI products used by children would have to protect their safety. AI companies would help fund worker training. When AI informs a decision about someone’s job, health care, or credit, that person could challenge it and have a qualified person review it.

Other countries. The United States would work with allies on shared safety standards. Cantwell also called for a direct line to China, like the Cold War “red phone,” for use during a serious AI incident.

The White House wants a different approach

In the two weeks before Cantwell’s announcement, President Donald Trump took two actions of his own on AI.

On Sept. 29, six AI companies, including OpenAI, Anthropic, and Google, signed a voluntary agreement with Trump. They committed to monitor their most powerful AI models for dangerous abilities and to have outside reviewers check their safeguards.

On Oct. 4, Trump named four officials to lead the Super Intelligence Force, a White House body that will coordinate the federal government’s AI work and its dealings with AI companies and the public. Trump’s announcement describes a coordinating role and does not mention powers to write rules or enforce laws.

Like the White House agreement, Cantwell’s plan relies on outside review of developers’ safeguards. Unlike the agreement, it would make that review a federal requirement before releasing a model. It would also add incident reporting and legal liability for developers

Cantwell has not introduced a bill

The plan is a set of principles, not legislation. Cantwell’s announcement does not say whether she will turn it into a bill.

Abstract shield blocks red digital threats in an image illustrating a Chinese AI espionage bill.

Bipartisan Bill Targets Chinese Espionage Against American AI Companies

, , , , , , , , , ,

The proposal would let AI developers share threat information and coordinate defenses against model theft without violating antitrust law.

Republican and Democratic lawmakers introduced a Chinese AI espionage bill Thursday that would let American AI companies work together to prevent foreign actors from stealing or copying their most advanced models.

Sens. Jim Banks (R-Ind.) and Adam Schiff (D-Calif.) introduced the bipartisan Collaboration on Adversarial Threats and Security Risks Act. Reps. Bob Latta (R-Ohio) and George Whitesides (D-Calif.) introduced the House version with additional bipartisan sponsors.

Banks and Schiff said foreign adversaries are using cyberattacks, model theft and distillation attacks to steal or reproduce advanced American AI systems.

Their announcement cites reports that Chinese actors used tens of thousands of fraudulent accounts to obtain millions of responses from leading American AI systems. Those responses can be used to train another model to reproduce some of the original system’s capabilities at a much lower cost, a process known as “distillation.”

The bill would let rival AI companies work together to identify and respond to foreign espionage without violating antitrust laws. That could include sharing intelligence and resources, investigating threats and coordinating defenses across companies. The same protection would extend to collaboration against other catastrophic AI risks, including threats involving weapons, critical infrastructure or the loss of control over advanced systems.

Companies could also agree to slow or stop an AI system’s development, delay its release or restrict how it is tested and used. They would first have to notify the Justice Department’s Antitrust Division and explain the risk and planned restrictions.

Organizations receiving shared information would have to limit its use to the security purpose covered by the bill.

The exemption would apply only to security collaboration. Companies could not use it to engage in collusion activities such as coordinating prices, dividing up customers, or otherwise limiting competition. A company invoking the exemption in an antitrust case would have to prove that it acted in good faith for a covered security purpose.

The attorney general could also ask a court to stop coordinated action that violates antitrust law or would increase the security risks the companies claimed to be addressing.

The House version, H.R. 9914, was referred to the House Judiciary Committee after its introduction on July 23.

Abstract light blue network of hexagons and glowing nodes illustrating the UN panel's first report on AI capabilities.

UN Scientists Warn AI Is Advancing Faster Than Governments Can Keep Up

, , , , , , , , , ,

The United Nations’ first independent scientific assessment says current safeguards are not keeping pace with AI, and scientists cannot yet rule out catastrophic harm as the technology becomes more capable.

 

The United Nations released the first report from its new Independent International Scientific Panel on Artificial Intelligence, warning that AI capabilities are advancing faster than governments and even AI researchers can fully understand or manage. The report argues that governments need reliable evidence on how advanced AI systems behave to regulate AI effectively. Still, AI is evolving so quickly that such evidence often lags behind the pace of development.

The panel, composed of 40 independent experts from around the world, concludes that AI is creating enormous opportunities while introducing equally significant risks. But it warns that governments and current knowledge about AI are not keeping pace with the technology’s rapid advancement.

Perhaps the report’s strongest warning came from panel co-chair Yoshua Bengio, who said AI has reached a point where researchers cannot yet guarantee increasingly capable systems will remain safe.

“AI capabilities are outpacing both scientific understanding and governments’ ability to adapt,” Bengio said. “With growing evidence of deceptive AI behavior, science currently cannot guarantee that as capabilities continue to increase, AI will not cause catastrophic harm, either on its own or due to malicious users.”

Rather than calling for specific regulations, the panel says governments need a stronger evidence base for AI policymaking. It argues that waiting for definitive evidence before acting may leave policymakers responding only after serious problems emerge.

The report also points to several risks that are becoming more difficult to manage as AI systems become increasingly autonomous. These include AI systems deceiving evaluators, leaking sensitive information, coordinating with other AI systems in unexpected ways, generating convincing misinformation at scale, and becoming harder for humans to supervise effectively. It concludes that existing oversight methods were designed for simpler software and may no longer be sufficient for AI systems capable of independently performing complex tasks.

United Nations Secretary-General António Guterres said governments need a shared body of evidence to guide AI policy rather than relying on fragmented or competing claims.

“The world cannot govern what it cannot understand,” Guterres said. “The potential is great, but the risks are real, and the cost of waiting is rising.”

The report will be presented to governments during the inaugural UN Global Dialogue on AI Governance in Geneva on July 6-7. The panel plans to publish annual assessments, with its first comprehensive report scheduled for 2027.

Abstract AI network image for coverage of an MIT survey on catastrophic AI risks.

MIT Survey Finds 18 AI Risks That Could Cause Catastrophic Harm Within Five Years

, , , , , , , , , , ,

Even after practical safeguards are in place, experts still say that five AI risk categories have a 10% probability of causing catastrophic harm.

 

Key Takeaways

  • 18 of 24 AI risk categories have more than a 10% probability of catastrophic harm by 2030 under a business-as-usual scenario.
  • Even with pragmatic mitigations in place, experts still rated five AI risks above the 10% probability threshold.
  • Information, finance, and national security were identified as the sectors most vulnerable to AI-related risks.
  • The highest-rated risk categories included dangerous AI capabilities, competitive dynamics, weapons and cyberattacks, power centralization, and false information.
  • Weapons and cyberattacks were the most frequently cited expert concern, selected by 26.8% of respondents.

A new report from MIT, Prioritizing the Risks from Artificial Intelligence, asked 272 AI experts to assess which AI risks are most severe, who is most vulnerable to them, and who should be responsible for addressing them. The study used a structured survey method in which experts answered questions over three rounds and reviewed anonymous feedback between rounds to build expert consensus across 24 categories of AI risk. The results suggest experts see meaningful catastrophic risk across much of the AI landscape over the next five years.

 

Catastrophic risk assessments

Experts identified 18 of 24 AI risk domains as having a greater than 10% probability of catastrophic harm in the next five years under a business-as-usual scenario. The report defined catastrophic harm as more than one million deaths, more than $100 billion in financial losses, or comparable societal-scale harm.

The five risk categories viewed as most severe were:

  • Dangerous AI capabilities: AI systems enabling large-scale harm or loss of control.
  • Competitive dynamics: Pressure to deploy AI quickly at the expense of safety.
  • Weapons and cyberattacks: AI enabling stronger cyber operations or weapons.
  • Power centralization: AI concentrating power among a few organizations or governments.
  • False information: AI-generated misinformation that undermines trust.

Researchers also evaluated a scenario where governments and companies implemented practical, cost-effective safeguards. While risk levels declined across all 24 categories, five remained above the 10% catastrophic-harm threshold: dangerous AI capabilities, weapons and cyberattacks, power centralization, environmental harm, and inequality and unemployment. Even after those safeguards were implemented, all 24 risks still had at least a 5% probability of causing catastrophic harm.

 

Most vulnerable sectors

The report asked experts to assess the vulnerability of 14 industry sectors to AI-related risks.

Information and national security were rated the most vulnerable sectors. Experts said those sectors face heightened exposure because AI is increasingly embedded in decision-making, information distribution, and critical systems. Finance and insurance also ranked near the top due to risks related to fraud, scams, AI security weaknesses, and system failures.

Healthcare received high vulnerability ratings for privacy loss, discrimination, and unsafe reliance on AI systems. By contrast, the accommodation and food services sector (including hotels, lodging, and restaurants), agriculture, manufacturing, and arts and entertainment were viewed as less vulnerable. However, experts noted they could still face broader economic and workforce effects.

 

Top concerns

When experts were asked to identify the three AI risks that concerned them most, weapons and cyberattacks ranked first at 26.8%, followed by power centralization at 23.5%, disinformation and influence at 22.1%, the erosion of a shared understanding of reality at 21.6%, and dangerous capabilities at 21.6%.

The report assigns much of the responsibility for addressing these risks to AI developers, governments, regulators, and standards bodies. Its release comes as policymakers in the United States, Europe, and elsewhere continue debating how AI systems should be governed, tested, and monitored as adoption accelerates.