A Semgrep researcher used just 10 poisoned training examples to make the model insert a flaw that could give attackers control of a company’s servers or other systems running the vulnerable code.
Cybersecurity researcher Katie Paxton-Fear of Semgrep poisoned an open-weight AI model for less than $100 in about an hour, showing how cheaply attackers could add hidden behavior to models that companies download and run themselves.
An open-weight AI model makes its core files available for people or companies to download and modify. Paxton-Fear took an existing open-weight coding model and trained it on a small set of new examples, creating an altered version of the model.
She first taught the altered model to use a different JavaScript coding style. After that worked, she used 10 malicious examples to make it generate code containing a security flaw.
Ten Examples Changed the Model’s Code
Paxton-Fear said just 10 malicious training examples caused the model to repeatedly produce code with a serious security flaw. If a company used that code, an attacker could potentially take control of the servers or other systems running it.
The model also produced the security flaw when responding to prompts it had not seen during training. This showed that the change was not limited to repeating the 10 malicious examples.
The experiment did not involve compromising a model already used by businesses. Semgrep said there is no public evidence that widely used open-weight models have been deliberately poisoned.
The demonstration instead showed how someone distributing a modified model could quietly add harmful behavior before another company downloaded it.
Hidden Changes Are Difficult to Detect
Companies can inspect traditional software for malicious code, trace where it came from and check whether it has been altered. Paxton-Fear and her Semgrep colleagues said companies cannot yet examine AI models with the same confidence.
A poisoned model may appear to work normally during routine use and testing, but produce insecure code, biased recommendations or other harmful results when it receives particular prompts. Reviewing the model’s files does not provide a clear account of how it will behave.
Standard performance tests also may not expose the problem. A poisoned model can be adjusted to pass those tests while retaining hidden behavior.
Semgrep called for better records showing where models and their training data come from, methods for reproducing how models were built, and testing by independent organizations. The researchers said companies will need outside auditors capable of checking model developers’ claims, similar to the independent security testing used for traditional software.
Those safeguards can help companies limit which models they trust and test the systems before use. However, Semgrep said no reliable method currently exists for proving that an open-weight AI model does not contain a hidden backdoor.
