Most also said the odds of an AI catastrophe already exceed the most risk they would accept.

Key Takeaways

• Asked for the odds that AI operates outside human control within ten years, half of the 111 people surveyed answered 33% or higher, and 87% answered 10% or higher.
• 56 experts estimated how likely an AI catastrophe is. They also said how much risk they would accept. For 61%, the estimate was higher than the amount they would accept.
• 74% rated testing and evaluating AI models highly effective against catastrophic risk, the highest score of any measure offered. In their written answers, respondents said the government has no independent ability to run those tests.
• Among the 42 who answered the cybersecurity section, 88% expect AI to make cyber defense much stronger, and 57% still expect attackers to gain more from it at first.

The Institute for Security and Technology published the report AI and the Future of National Security, a survey of 111 people who work in US national security. Asked for the probability that AI operates outside human control within ten years, half of them answered 33% or higher, and 87% answered 10% or higher. A quarter answered below 15%, and a quarter answered above 75%.

Respondents are current and former civilian officials, military officers, and technical experts, mostly from the Department of Defense, State Department, the White House, Congress, and several intelligence agencies. 87% have served in the US government, half for ten years or more and nearly a quarter for 20 years or more. 

61% estimated more risk of an AI catastrophe than they would accept

The survey asked each person to estimate how likely an AI catastrophe is, meaning one that kills more than 10 million people before 2050. It also asked how much risk of that they would accept in return for the United States leading in AI. 56 people answered both, and 61% estimated it as equal to or higher than the amount they would accept. Another 24% said they could not put a number on how much they would accept. Both numbers come from the same person. 

Nobody has a process for acting on a warning sign

Respondents wrote repeatedly that nothing exists to act on a warning that AI is slipping beyond human control. Such a warning could be a system doing things nobody set it up to do, or oversight eroding so gradually that nobody can say what has been handed over to AI. No threshold, rule, or procedure says who would decide what happens next. Asked whether reliable warnings would appear at all before AI reaches human-level ability, they split 39% confident against 39% skeptical. 

The cybersecurity specialists were asked a narrower version of the same question: could operators regain control of an AI system that started doing more than it was set up to do. None were highly confident. 48% said it depends entirely on how the system was built, and 26% said it would be difficult or impossible.

The government has no way to check what these systems can do

Respondents said the government has no testing setup or benchmarks of its own, so it cannot check what an AI system can actually do before putting it to work. Too few people in government understand the technology well enough to judge it without those tests, so officials lean on what the AI companies tell them.

Asked which protections work best against catastrophic AI risk, they put testing and evaluating models first, with 74% calling it highly effective. The protection they rate most effective is the one they say the government cannot carry out.

Cyber specialists expect attackers to gain more from AI in the short term

42 respondents answered the cybersecurity section. 88% expect AI to make cyber defense much stronger. 57% still expect attackers to gain more from it in the near term.

What they rate as most dangerous is AI finding software flaws and building working attacks from them, which 52% called critical, more than any other capability. They said the danger comes from weaknesses that already exist, not from new kinds of attack. One respondent said the highest-risk targets are not well-resourced companies but the thinly staffed operators of water systems, rural power, and municipal services.

They agree on what should be required, and almost none of it is law

93% of the cybersecurity respondents said the government should set requirements for how companies handle AI cyber risk, rather than leaving it to the companies. They divided over the form, with a federal baseline plus industry-specific rules the most popular at 36%. 

The one law they want

96% of all respondents said a human must make any decision to use a nuclear weapon. Congress is already weighing that rule in two places. The Responsible Artificial Intelligence Defense Act, introduced in June by Senators Chris Coons and Jack Reed, would prohibit AI from making nuclear launch decisions and require the Pentagon to keep human judgment over decisions to use lethal force. It sits with a committee. The same bar appears separately inside the Senate Armed Services Committee’s draft of this year’s defense policy bill. Neither has become law.