Have a Beef With AI? Here's How to Poison a Large Language Model

(Credit: Moor Studio/Getty Images)

The large language models (LLMs) that power generative AI apps such as ChatGPT power up their responses by inhaling immense amounts of information. There’s no way to trace the exact path from an input query to the LLM’s response, opening the possibility of gimmicking queries to make the AI app break its rules. Task-focused LLMs can also be vulnerable to poisoning, meaning that malefactors deliberately feed them false or irrelevant information.

At the RSAC Conference in San Francisco, application security company Checkmarx sponsored a panel exploring the topic, including a live LLM poisoning demo, all over breakfast.

What's the Most Popular Programming Language? English!

Erez Yalon, head of security research for Checkmarx, an enterprise security company, led off with a demo. “People told me not to do a live demo, because live demos usually fail,” said Yalon. “But AI is non-deterministic, so if it fails, it’s not on me.” He pointed out that AI is just technology, part of the software supply chain, defined as “stuff created by people who are not us.”

“What’s the most popular development language in the world?” Yalon asked the audience. Despite a few shouts of “COBOL!” he explained that much modern programming is done in English, by prompting a generative AI system. He then created a simple shopping list program using a minimal prompt. With the help of the underlying AI, the app let him add and remove items, even adding the ingredients to create a cheesecake. It corrected typos accurately by knowing the context, quite an accomplishment from a simple English prompt.

Then he asked it to add “the most healthy food in the world.” And the app added…rat poison. How did that happen? “Someone trained it,” explained Yalon. “I used an open-source LLM, and I have no idea of its training. Unless you ask the right questions, everything looks perfect.”

Yalon proceeded with a second demo, using an LLM to write simple code requiring an AWS key. The LLM wrote the code, yes, but it slipped in a line that transmitted the key to an arbitrary URL, hidden by spacing it far to the right. “Poisoning an LLM isn’t just poisoning the data,” said Yalon. “It’s poisoning what you can do.”

I asked Yalon, “I feel like you’ve shown us poisoned LLMs, not poisoning LLMs. How is it done?” Yalon explained that it’s all in the fine tuning. Just as when you create your own LLMs, you start with an existing model. “It’s not new training or new data,” said Yalon. “If I want just simple poisoning for one question, it took just a few hours,” he concluded.

LLMs Are Part of the Software Supply Chain

With the demo complete, the event turned to a panel discussion on LLMs and the software supply chain. Ira Winkler moderated the panel, whose security experience has ranged from the NSA to HP to Qualys, and even to a stint protecting Walmart’s security. Other participants included Erez Yalon and Cassie Crossley, author of the O’Reilly book Software Supply Chain Security.

After much discussion of specific supply chain fiascos and real-world defenses, the panel concluded that AI technology is now part of the software supply chain, no different from the popular open-source libraries that, according to one panelist, make up 90% of most programs. Disasters and near-disasters like SolarWinds and XZ Utils have demonstrated that software companies can’t assume those libraries are safe.

Asked what his worst nightmare is for the software supply chain, Yalon replied, “The one we don’t know about. It’s already happening somewhere.” Crossley suggested that developers review third-party code, including AI components, just as thoroughly as they review what they write themselves.

Suppose you’re a consumer of generative AI. In that case, even if you occasionally ask an AI to find some information, write something, or make you a picture, you don’t have to worry about AI poisoning. The AI may give you a false answer, but that error will come from its analysis of the actual input. Only when AI serves as a component in a bigger system, like a massive software deployment, does poisoning become a worry.

About Our Expert

Neil J. Rubenking

Principal Writer, Security

My Experience

When the IBM PC was new, I served as the president of the San Francisco PC User Group for three years. That’s how I met PCMag’s editorial team, who brought me on board in 1986. In the years since that fateful meeting, I’ve become PCMag’s expert on security, privacy, and identity protection, putting antivirus tools, security suites, and all kinds of security software through their paces.

Before my current security gig, I supplied PCMag readers with tips and solutions on using popular applications, operating systems, and programming languages in my "User to User" and "Ask Neil" columns, which began in 1990 and ran for almost 20 years. Along the way, I wrote more than 40 utility articles, as well as Delphi Programming for Dummies and six other books covering DOS, Windows, and programming. I also reviewed thousands of products of all kinds, ranging from early Sierra Online adventure games to AOL’s precursor Q-Link.

In the early 2000s, I turned my focus to security and the growing antivirus industry. After years of working with antivirus, I’m known throughout the security industry as an expert on evaluating antivirus tools. I serve as an advisory board member for the Anti-Malware Testing Standards Organization (AMTSO), an international nonprofit group dedicated to coordinating and improving testing of anti-malware solutions.

The Technology I Use

Much of the testing I do, particularly testing with real-world ransomware, is just plain dangerous. To perform such tests safely, I sequester them inside virtual machines managed by VMWare Workstation. For cross-platform testing, I use a MacBook Air, a Google Pixel 4, and a 6th-generation iPad.

I rely on my Delphi coding skills to create and maintain small applications. These include programs to check whether an antivirus correctly handled the malware it detected, launch dangerous URLs and record the security program’s reaction, and analyze the malware that I collect for use in testing. I also wrote a tiny browser and text editor for use in testing security apps that have predefined reactions for known products.

I do my writing and research on a Dell OptiPlex desktop, relying on Microsoft Word (my fingers know all the shortcuts). Many of my articles include charts and analysis; Excel is my go-to for those. When work hours end, though, I escape the bounds of Microsoft and Windows. There’s an iPhone in my pocket, I relax with my oversized iPad, and my Kindle Oasis is always loaded with the best science fiction and fantasy.

Read the latest from Neil J. Rubenking

Read full bio