Menu
SUBSCRIBE
SUBSCRIBE
Menu

GLM-5.3

Glowing crystalline racing pods streak down bright lanes, illustrating DeepSeek V4 Pro leading Aikido's AI model test

Chinese Open-Weight Models Match Or Beat US Rivals At Finding Software Flaws

, , , , , , , , , , ,

DeepSeek V4 Pro found 28 of the 32 security flaws in Aikido’s test, and no publicly available American model found more than 26.

 

Key Takeaways

  • Aikido tested 10 AI models against 32 security flaws in real software, three attempts each.
  • All the open-weight models tested were from Chinese companies that anyone can download and run themselves. One of them, DeepSeek V4 Pro, found more flaws than any American model tested.
  • The most cost-efficient model was American. OpenAI’s GPT-5.6 Luna cost about $70 for three passes and found 22, the fewest, along with GPT-5.6 Terra.
  • The cheaper open-weight models also reported more flaws that turned out not to exist.

Aikido, a company that produces software for finding security flaws in code, tested 10 AI models on that same job and published the results on August 21. Five are open-weight, meaning anyone can download the files that make the model work and run it themselves. All five were from Chinese companies. One of them, DeepSeek V4 Pro, found more flaws than any American model in the test.

Aikido left out the American models built specifically for this work, Anthropic’s Mythos 5 and OpenAI’s 5.6 Daybreak models. Both Anthropic and OpenAI restrict access to their frontier models with the most powerful cybersecurity skills to vetted testing partners. The American models that scored below the Chinese ones are the ones any company can purchase.

 

Every downloadable model in the test came from a Chinese company

Aikido picked 32 security flaws reported publicly shortly before the test. They came from real unnamed software projects, so the models were less likely to have read a write-up during training. It handed each model code containing a flaw and checked whether the model found the flaw. Each model worked through all 32 cases three times, starting fresh each time.

Four Chinese companies supplied the five downloadable models: DeepSeek, with both DeepSeek V4 Pro and the cheaper DeepSeek V4 Flash; Alibaba, with Qwen3.8-Max; Moonshot AI, with Kimi K3; and Z.ai, with GLM-5.3. The five paid American models are Anthropic’s Claude Opus 5, OpenAI’s GPT-5.6 Terra, GPT-5.6 Luna, and GPT-5.6 Sol, and xAI’s Grok 4.6.

DeepSeek V4 Pro found 28 of the 32 flaws. Claude Opus 5, Grok 4.6, and Qwen3.8-Max found 26. Kimi K3, GLM-5.3, and GPT-5.6 Sol each found 25. Aikido tested no downloadable model from Meta, Google, or Mistral, so the results do not compare Chinese open-weight models against American ones.

 

DeepSeek V4 Pro found the most flaws; OpenAI’s GPT-5.6 Luna was the cheapest at cost-per-flaw found

Three passes with DeepSeek V4 Pro cost about $295 and found 28 flaws, more than any other model tested. Claude Opus 5 found 26 for about $1,650, and Grok 4.6 and GPT-5.6 Sol also ran into four figures without beating 26.

The most cost-efficient model in the test was American. OpenAI’s GPT-5.6 Luna cost about $70 for three passes and found 22, the fewest, along with GPT-5.6 Terra. The table sets out what each of the ten models found, what it cost, and what each flaw cost to find.

Running all 32 cases three times finds more than running them once, because a model does not do the same thing twice. Given the same code again, it can take a different route and reach a different answer. DeepSeek V4 Pro found 17 flaws on its first pass and 28 across all three, the biggest improvement of the ten. Grok 4.6 improved the least, from 24 to 26.

Table of 10 AI models ranked by flaws found and cost, with DeepSeek V4 Pro top at 28 of 32 for about $295
What each model found, and what three passes cost (Image credit: AI Risk Today, from Aikido Security data.)

 

The cheaper open-weight models reported more flaws that turned out not to exist

The open-weight models matched the American ones for far less money, Aikido wrote, but “produced the most false leads for the pipeline to reject.” The two DeepSeek models reported far more non-existent flaws than the other eight.

Kimi K3 was the most accurate downloadable model: 92.3% of what it reported across its three passes was a real flaw. GPT-5.6 Sol reported fewer than three possible flaws per case and wrote the clearest explanations, usually making it easy to follow how an attacker’s data reaches the code that mishandles it.

Z.ai has still not released the files for the model Aikido tested

Aikido reached GLM-5.3 through Z.ai’s paid coding service, because Z.ai has not published the model’s files. Z.ai said on August 14 that it would hold them back about two weeks, putting the release around August 28, while it ran safety tests. It cited the model’s own ability to find software flaws and chain them into an attack. Z.ai has not said what its license will permit.

Methodology

Aikido capped each case at 30 steps, and the test consumed 11.7 billion tokens, the units of text AI companies bill by. Aikido tested the models on their own, without the software it sells to filter what they report. It states some costs and plots the rest on a chart, from which the figures here are read and rounded. Every figure is Aikido’s own, and nobody has repeated the test.

Glowing blue digital network with a keyhole inside a shield at the center, illustrating a held-back AI model, GLM-5.3, with vulnerability discovery ability

Z.ai Holds Back GLM-5.3 Files Over the Model’s Own Cyber Ability

, , , , , , , , , , , ,

The Chinese lab says GLM-5.3 now leads every model it tested at finding software vulnerabilities, and it will not publish the downloadable version until safety testing and hardening are done, about two weeks after the August 14 launch.

 

Z.ai released its new model, GLM-5.3, on August 14 through its paid coding service and held back the model’s core files, the version companies and individuals could download, change, and run on their own hardware. The company said it will publish the GLM-5.3 open weights, as those files are called, about two weeks after launch, once safety evaluation and hardening are complete.

Z.ai said it delayed the release because of the model’s cybersecurity abilities.

Z.ai says the model leads at finding software flaws

The company reported that GLM-5.3 scored higher than any other model it tested on CyberGym, an AI vulnerability discovery test that gives a model a program’s source code and checks whether it can locate a vulnerability and prove it’s real by making the program fail. GLM-5.3 scored 84.5% across 1,507 tasks, up from 77.2% for the previous version, GLM-5.2. Z.ai reported Anthropic’s Mythos 5 at 83.8% and OpenAI’s GPT-5.6 Sol at 83.6% on the same test.

The company says the ability grew faster than it expected

Z.ai added vulnerability discovery data and practice environments to the training mix and expected the model to get better at spotting individual flaws. It said the model went further than that. According to the company, GLM-5.3 “began to reason across multiple stages of exploitation, forming coherent plans for complete exploitation chains,” meaning it strung individual weaknesses together into a working sequence rather than reporting them one at a time.

Z.ai also said the gains were largest on the tests that sit furthest along that chain. “Capability is growing fastest exactly where we are furthest behind,” the company wrote.

GLM-5.3 still trails Anthropic and OpenAI at turning flaws into attacks

On ExploitBench, which asks a model to reason about real vulnerabilities and how they would be exploited, GLM-5.3 scored 54.4%, more than double GLM-5.2’s 24.4%. Z.ai reported Mythos 5 at 78.0% and GPT-5.6 Sol at 76.5% on the same test.

On ExploitGym, which counts how many exploitation tasks a model finishes within a fixed time budget, GLM-5.3 completed 105 tasks in two hours and 130 in six, compared with 29 and 39 for GLM-5.2. Z.ai reported Mythos 5 at 181 and 247.

Z.ai reports 2,436 real vulnerabilities across 269 projects

The company said it has been running its models against real codebases with several security teams in China. After expert review, screening, and removing duplicates, it counted 2,436 vulnerabilities across 269 open-source projects, including 1,097 rated medium to high severity, of which 107 were critical.

Many had gone unnoticed for years. Z.ai said the oldest was introduced in 1981 and that, on average, a vulnerability lasted 26.6 years before the model found it.

Z.ai publishes a running record of this work, the Z.ai Security Disclosure Ledger. As of launch, it listed 53 vulnerabilities as publicly disclosed and 2,383 still under embargo while the affected projects work on fixes.

The cyber ability came out of training for coding

GLM-5.3 uses the same underlying model as GLM-5.2, and Z.ai said every gain came from the training it did afterward, mostly on long software engineering jobs.

The coding tests the company used all work the same way. Each one hands the model a real programming task and a working computer, then checks whether it finishes the job. On Terminal Bench 3.0, a public test of that kind, Z.ai said GLM-5.3 completed 28.3% of the tasks, up from 4.6% for GLM-5.2. On the company’s own version, which it does not publish, GLM-5.3 completed 34.5% against 23.4%, the 50% improvement Z.ai claims. It said Anthropic’s Claude Fable 5 completed 39.5% on that same unpublished test.

Z.ai said the downloadable files will follow about two weeks after the August 14 launch. Until then, GLM-5.3 runs only through the company’s paid coding plan and the coding tools connected to it, including ZCode, Claude Code, and OpenCode. Every number above is Z.ai’s own, and no outside party can check the results on its own hardware until the files are published.

Z.ai is not the first company to restrict how a model reaches the public over cybersecurity questions. Anthropic released Mythos 5, built to find software vulnerabilities and work out how they could be exploited, only to partners in its Project Glasswing program, with wider access planned for vetted organizations through an arrangement with the US government. In its August 2026 risk report, Anthropic said it has no current plans to release Model 2 outside the company, and raised its rating of the chance a model acts against the company’s interests from very low to low after models took unauthorized actions during cybersecurity tests.