DeepSeek V4 Pro found 28 of the 32 security flaws in Aikido’s test, and no publicly available American model found more than 26.

Key Takeaways

• Aikido tested 10 AI models against 32 security flaws in real software, three attempts each.
• All the open-weight models tested were from Chinese companies that anyone can download and run themselves. One of them, DeepSeek V4 Pro, found more flaws than any American model tested.
• The most cost-efficient model was American. OpenAI’s GPT-5.6 Luna cost about $70 for three passes and found 22, the fewest, along with GPT-5.6 Terra.
• The cheaper open-weight models also reported more flaws that turned out not to exist.

Aikido, a company that produces software for finding security flaws in code, tested 10 AI models on that same job and published the results on August 21. Five are open-weight, meaning anyone can download the files that make the model work and run it themselves. All five were from Chinese companies. One of them, DeepSeek V4 Pro, found more flaws than any American model in the test.

Aikido left out the American models built specifically for this work, Anthropic’s Mythos 5 and OpenAI’s 5.6 Daybreak models. Both Anthropic and OpenAI restrict access to their frontier models with the most powerful cybersecurity skills to vetted testing partners. The American models that scored below the Chinese ones are the ones any company can purchase.

Every downloadable model in the test came from a Chinese company

Aikido picked 32 security flaws reported publicly shortly before the test. They came from real unnamed software projects, so the models were less likely to have read a write-up during training. It handed each model code containing a flaw and checked whether the model found the flaw. Each model worked through all 32 cases three times, starting fresh each time.

Four Chinese companies supplied the five downloadable models: DeepSeek, with both DeepSeek V4 Pro and the cheaper DeepSeek V4 Flash; Alibaba, with Qwen3.8-Max; Moonshot AI, with Kimi K3; and Z.ai, with GLM-5.3. The five paid American models are Anthropic’s Claude Opus 5, OpenAI’s GPT-5.6 Terra, GPT-5.6 Luna, and GPT-5.6 Sol, and xAI’s Grok 4.6.

DeepSeek V4 Pro found 28 of the 32 flaws. Claude Opus 5, Grok 4.6, and Qwen3.8-Max found 26. Kimi K3, GLM-5.3, and GPT-5.6 Sol each found 25. Aikido tested no downloadable model from Meta, Google, or Mistral, so the results do not compare Chinese open-weight models against American ones.

DeepSeek V4 Pro found the most flaws; OpenAI’s GPT-5.6 Luna was the cheapest at cost-per-flaw found

Three passes with DeepSeek V4 Pro cost about $295 and found 28 flaws, more than any other model tested. Claude Opus 5 found 26 for about $1,650, and Grok 4.6 and GPT-5.6 Sol also ran into four figures without beating 26.

The most cost-efficient model in the test was American. OpenAI’s GPT-5.6 Luna cost about $70 for three passes and found 22, the fewest, along with GPT-5.6 Terra. The table sets out what each of the ten models found, what it cost, and what each flaw cost to find.

Running all 32 cases three times finds more than running them once, because a model does not do the same thing twice. Given the same code again, it can take a different route and reach a different answer. DeepSeek V4 Pro found 17 flaws on its first pass and 28 across all three, the biggest improvement of the ten. Grok 4.6 improved the least, from 24 to 26.

The cheaper open-weight models reported more flaws that turned out not to exist

The open-weight models matched the American ones for far less money, Aikido wrote, but “produced the most false leads for the pipeline to reject.” The two DeepSeek models reported far more non-existent flaws than the other eight.

Kimi K3 was the most accurate downloadable model: 92.3% of what it reported across its three passes was a real flaw. GPT-5.6 Sol reported fewer than three possible flaws per case and wrote the clearest explanations, usually making it easy to follow how an attacker’s data reaches the code that mishandles it.

Z.ai has still not released the files for the model Aikido tested

Aikido reached GLM-5.3 through Z.ai’s paid coding service, because Z.ai has not published the model’s files. Z.ai said on August 14 that it would hold them back about two weeks, putting the release around August 28, while it ran safety tests. It cited the model’s own ability to find software flaws and chain them into an attack. Z.ai has not said what its license will permit.

Methodology

Aikido capped each case at 30 steps, and the test consumed 11.7 billion tokens, the units of text AI companies bill by. Aikido tested the models on their own, without the software it sells to filter what they report. It states some costs and plots the rest on a chart, from which the figures here are read and rounded. Every figure is Aikido’s own, and nobody has repeated the test.