Study finds internet-accessible, self-hosted AI systems can enable spam, phishing, disinformation and other illicit activity when lacking safety controls.

A joint research project between SentinelLABS and Censys revealed that deployments of open-source artificial intelligence models running on publicly reachable servers are being used in ways that could aid criminal activity, according to a report shared with Reuters. The joint 293-day study examined thousands of self-hosted large language models (LLMs), including those run through the Ollama tool that lets individuals and organizations operate models on their own hardware.

The analysis found that hackers or other malicious actors could commandeer computers operating open-source LLMs outside of the security controls and guardrails typical of major commercial AI platforms to carry out spam operations, phishing content creation and disinformation campaigns, researchers said.

In some cases, the researchers found that system prompts could enable harmful activity. Of the roughly one-quarter of observed hosts where prompts were visible, about 7.5% were judged to have the potential to facilitate misuse. 

“These include hacking, hate speech and harassment, violent or gore content, personal data theft, scams or fraud, and in some cases child sexual abuse material,” the study said.

Speaking with Reuters, Juan Andres Guerrero-Saade, executive director for intelligence and security research at SentinelOne, said industry discussions about security controls are overlooking “surplus capacity that is clearly being utilized for all kinds of different stuff, some of it legitimate, some obviously criminal.”

Roughly 30% of the open-source LLM hosts observed were operating out of China and about 20% in the United States. Variants of models such as Meta’s Llama and Google’s Gemma were among those observed without guardrails.