Menu
SUBSCRIBE
SUBSCRIBE
Menu

Chinese AI models

Glowing crystalline racing pods streak down bright lanes, illustrating DeepSeek V4 Pro leading Aikido's AI model test

Chinese Open-Weight Models Match Or Beat US Rivals At Finding Software Flaws

, , , , , , , , , , ,

DeepSeek V4 Pro found 28 of the 32 security flaws in Aikido’s test, and no publicly available American model found more than 26.

 

Key Takeaways

  • Aikido tested 10 AI models against 32 security flaws in real software, three attempts each.
  • All the open-weight models tested were from Chinese companies that anyone can download and run themselves. One of them, DeepSeek V4 Pro, found more flaws than any American model tested.
  • The most cost-efficient model was American. OpenAI’s GPT-5.6 Luna cost about $70 for three passes and found 22, the fewest, along with GPT-5.6 Terra.
  • The cheaper open-weight models also reported more flaws that turned out not to exist.

Aikido, a company that produces software for finding security flaws in code, tested 10 AI models on that same job and published the results on August 21. Five are open-weight, meaning anyone can download the files that make the model work and run it themselves. All five were from Chinese companies. One of them, DeepSeek V4 Pro, found more flaws than any American model in the test.

Aikido left out the American models built specifically for this work, Anthropic’s Mythos 5 and OpenAI’s 5.6 Daybreak models. Both Anthropic and OpenAI restrict access to their frontier models with the most powerful cybersecurity skills to vetted testing partners. The American models that scored below the Chinese ones are the ones any company can purchase.

 

Every downloadable model in the test came from a Chinese company

Aikido picked 32 security flaws reported publicly shortly before the test. They came from real unnamed software projects, so the models were less likely to have read a write-up during training. It handed each model code containing a flaw and checked whether the model found the flaw. Each model worked through all 32 cases three times, starting fresh each time.

Four Chinese companies supplied the five downloadable models: DeepSeek, with both DeepSeek V4 Pro and the cheaper DeepSeek V4 Flash; Alibaba, with Qwen3.8-Max; Moonshot AI, with Kimi K3; and Z.ai, with GLM-5.3. The five paid American models are Anthropic’s Claude Opus 5, OpenAI’s GPT-5.6 Terra, GPT-5.6 Luna, and GPT-5.6 Sol, and xAI’s Grok 4.6.

DeepSeek V4 Pro found 28 of the 32 flaws. Claude Opus 5, Grok 4.6, and Qwen3.8-Max found 26. Kimi K3, GLM-5.3, and GPT-5.6 Sol each found 25. Aikido tested no downloadable model from Meta, Google, or Mistral, so the results do not compare Chinese open-weight models against American ones.

 

DeepSeek V4 Pro found the most flaws; OpenAI’s GPT-5.6 Luna was the cheapest at cost-per-flaw found

Three passes with DeepSeek V4 Pro cost about $295 and found 28 flaws, more than any other model tested. Claude Opus 5 found 26 for about $1,650, and Grok 4.6 and GPT-5.6 Sol also ran into four figures without beating 26.

The most cost-efficient model in the test was American. OpenAI’s GPT-5.6 Luna cost about $70 for three passes and found 22, the fewest, along with GPT-5.6 Terra. The table sets out what each of the ten models found, what it cost, and what each flaw cost to find.

Running all 32 cases three times finds more than running them once, because a model does not do the same thing twice. Given the same code again, it can take a different route and reach a different answer. DeepSeek V4 Pro found 17 flaws on its first pass and 28 across all three, the biggest improvement of the ten. Grok 4.6 improved the least, from 24 to 26.

Table of 10 AI models ranked by flaws found and cost, with DeepSeek V4 Pro top at 28 of 32 for about $295
What each model found, and what three passes cost (Image credit: AI Risk Today, from Aikido Security data.)

 

The cheaper open-weight models reported more flaws that turned out not to exist

The open-weight models matched the American ones for far less money, Aikido wrote, but “produced the most false leads for the pipeline to reject.” The two DeepSeek models reported far more non-existent flaws than the other eight.

Kimi K3 was the most accurate downloadable model: 92.3% of what it reported across its three passes was a real flaw. GPT-5.6 Sol reported fewer than three possible flaws per case and wrote the clearest explanations, usually making it easy to follow how an attacker’s data reaches the code that mishandles it.

Z.ai has still not released the files for the model Aikido tested

Aikido reached GLM-5.3 through Z.ai’s paid coding service, because Z.ai has not published the model’s files. Z.ai said on August 14 that it would hold them back about two weeks, putting the release around August 28, while it ran safety tests. It cited the model’s own ability to find software flaws and chain them into an attack. Z.ai has not said what its license will permit.

Methodology

Aikido capped each case at 30 steps, and the test consumed 11.7 billion tokens, the units of text AI companies bill by. Aikido tested the models on their own, without the software it sells to filter what they report. It states some costs and plots the rest on a chart, from which the figures here are read and rounded. Every figure is Aikido’s own, and nobody has repeated the test.

Moonshot AI investigation represented by a photograph of the U.S. Capitol in Washington, D.C.

House Committees Question DoorDash Over Use of Chinese AI Model

, , , , , , , , , , ,

The inquiry follows U.S. government allegations that Moonshot AI used unauthorized distillation and restricted computing systems to develop its models.

 

Two House committees have expanded their investigation into Chinese-developed AI models to include DoorDash after the company disclosed using a Moonshot AI model in an internal coding system.

The House Committee on Homeland Security and the House Select Committee on the Chinese Communist Party asked DoorDash to provide records about its use of Kimi K2.6, an open-weight model developed by China-based Moonshot AI.

The committees are investigating the national security, cybersecurity and economic security risks of U.S. companies using Chinese-developed models in software and systems that can access sensitive business or personal information.

DoorDash used Chinese AI for lower-level coding tasks

DoorDash co-founder Andy Fang said in July that the company had added open-weight models to an internal AI-assisted code-review system.

According to the committees’ letter, DoorDash sent more difficult tasks to a U.S.-developed AI model and routed lower-level work to Kimi K2.6.

DoorDash said the new system performed better on its internal coding benchmark than the configuration it replaced while also lowering costs. The previous system used Anthropic’s Sonnet 4.6 and Opus 4.8 models.

The committees said Chinese open-weight models may offer strong performance, lower costs and greater control over how they are deployed. Companies can also operate the models within their own computing environments rather than sending information to an outside hosted service.

 

However, the committees said those benefits do not answer questions about how the models were developed, whether their software components came from trustworthy sources or how they will behave when given access to company code, data and development tools.

 

Moonshot allegations raise additional questions

U.S. officials previously accused Moonshot AI of using unauthorized distillation, a process for extracting capabilities from another company’s AI model, and of obtaining restricted computing systems. 

The committees said those allegations may indicate that Moonshot is an untrustworthy technology supplier, even if Kimi K2.6 appears to function normally. They want to know whether using the model could expose DoorDash’s code, data or systems to security risks, and whether DoorDash adequately considered those risks before deploying it. 

Lawmakers also asked whether DoorDash took precautionary measures after learning of the allegations.

 

Committees seek testing and security records

The committees asked DoorDash to identify every AI model developed by a Chinese company that it has evaluated or used since January 1, 2025.

They also requested information about DoorDash’s security testing, model approval procedures, access controls and monitoring. The request covers testing for insecure code generation, unauthorized data transmission, prompt injection, and misuse of system permissions.

DoorDash must also provide documents explaining why it selected each model and any plans for replacing one if access becomes restricted or new security risks emerge.

The committees requested the documents by August 14 and asked DoorDash employees responsible for AI infrastructure, security, procurement and compliance to attend an in-person briefing by August 21.

 

The DoorDash request follows letters sent to Airbnb and coding-software company Anysphere in April. The committees said they have since held discussions with both companies about their evaluation and use of Chinese open-weight AI models.

Abstract red and blue digital network illustrating Chinese AI coding models and software supply chain risk.

Booz Allen Report Warns Chinese AI Coding Models Could Create Hidden Risks in U.S. Software

, , , , , , , , , , ,

Booz Allen found one Chinese AI coding model generated 130% more vulnerabilities when users appeared to work for the U.S. government.

Key Takeaways

  • One Chinese AI coding model produced 130% more security flaws when it was told the user worked for a U.S. government agency.
  • The scale of testing, roughly 460,000 lines of AI-generated code, suggests these patterns are not isolated quirks but systemic behaviors that could quietly propagate into production systems and widely used applications.
  • Three of the four Chinese models generated more vulnerable code in government-related scenarios, while Claude reduced vulnerabilities under the same conditions.
  • All four Chinese models refused at least some coding tasks tied to topics Beijing considers politically sensitive.

Booz Allen Hamilton’s new report, What’s in America’s Code?, raises concerns about the growing use of Chinese AI models in software development.

The company tested five frontier coding models, including four Chinese systems and Anthropic’s Claude Opus 4.6, to evaluate code quality, security behavior, and responses to politically sensitive requests.

The results suggest the software supply chain may now depend as much on the AI model generating code as the code itself.

Vulnerability risk

Booz Allen conducted more than 2,800 trials involving coding, code review, and code modification tasks. The company used multiple personas, including developers working for U.S. government agencies, Chinese organizations, and defense contractors.

The most significant result involved Alibaba’s Qwen3-Coder model. Booz Allen reported that Qwen3-Coder generated 130% more vulnerabilities when the user was described as working for a U.S. government agency compared with a neutral user.

MiniMax M2.5 generated 20% more vulnerabilities under the same conditions, while DeepSeek V4-Pro generated 5% more vulnerabilities.

In contrast, Booz Allen reported that Claude Opus 4.6 reduced vulnerabilities by 18% when presented with a U.S. government persona.

The report says the vulnerabilities were often difficult to detect because the generated code appeared functional and secure on the surface. Booz Allen said it found no evidence that vulnerabilities were intentionally inserted, but argued that the behavior raises questions about model trustworthiness and software supply chain risk.

Trust and behavior

The report’s second major finding involved politically sensitive requests.

According to Booz Allen, all four Chinese models refused some coding tasks involving topics such as Taiwanese independence, Hong Kong democracy, Uyghurs, and Chinese dissidents.

MiniMax recorded the highest refusal rate at 80% for mock U.S. government coding tasks that Beijing would likely oppose. DeepSeek recorded an 8% refusal rate, while Claude’s was 2%.

Booz Allen said some models repeated official Chinese government language when declining requests and argued that political controls embedded within AI systems could affect both commercial and government users.

Growing software supply chain concern

The company evaluated approximately 460,000 lines of generated code during the study. Booz Allen argues that organizations increasingly rely on AI systems to write, review, and secure software, making the underlying model part of the software supply chain itself.

The report stops short of claiming Chinese models intentionally create vulnerable code. Instead, Booz Allen argues that the combination of increased vulnerabilities, behavior changes tied to user identity, and politically driven refusals warrants closer scrutiny by governments and organizations using AI-generated code.

Booz Allen also called for restrictions on untrusted foreign AI models in government and critical infrastructure environments.