Nvidia, the developer of the processors used to train modern AI models, has been driving the generative AI (GenAI) revolution. The company runs its own large-scale language models (LLMs) and several in-house AI applications. The latter includes the company’s NeMo platform for building and deploying LLMs, as well as a range of AI-based applications such as object simulation and reconstructing the DNA of extinct species.
In a session called “Practical LLM Security: Lessons from a Year on the Frontlines” at Black Hat USA next month, Richard Harang, principal security architect for AI/ML at the chip giant, will talk about the lessons the Nvidia team learned red teaming these systems and how cyberattack tactics against LLMs continue to evolve. The good news, he says, is that even though these threats pose such a huge risk to companies because of their privileged nature, they don’t require much change to existing security practices.
“Over the past year or so, we’ve learned a lot about how to ensure security, and how to build security from the ground up rather than bolting it on after the fact,” says Harang. “As a result, we have a lot of valuable practical experience to share.”
AI poses recognizable problems, but with a twist
Enterprises are increasingly building applications that rely on next-generation AI in the form of integrated AI agents that can perform privileged actions. Meanwhile, security and AI researchers have already pointed out potential weaknesses in these environments, such as AI-generated code expanding an application’s attack surface or overly helpful chatbots leaking sensitive corporate data. But because these are just new iterations of known threats, attackers often don’t need specialized techniques to exploit them, Harang said.
“Many of the issues we’re seeing with LLMs are issues we’ve seen before in other systems,” he says. “What’s new is the attack surface and what that attack surface looks like. So if you understand how LLMs actually work, how inputs get into the model, how outputs get out of the model, and do some thoughtful planning, securing these systems is not inherently more difficult than securing other systems.”
GenAI applications still require the same three fundamental security attributes as any other application: confidentiality, integrity and availability, he says, so software engineers should go through the standard security architecture due diligence process, which includes setting security boundaries, setting trust boundaries and reviewing how data flows through the system.
To the defender’s advantage, AI systems tend to be less deterministic because randomness is often injected into them to make them “creative” — in other words, the same inputs do not always produce the same outputs, and attacks will not always be equally successful.
“With some exploits in traditional information security environments, you can get close to 100% confidence in injecting this payload,” Harang said. [an attacker] LLM exploits are generally less reliable than traditional exploits because they introduce information that attempts to manipulate the behavior of the LLM.”
With great power comes great risk
One way that AI environments differ from traditional IT environments is that they have autonomous agency. Companies don’t just want AI applications that can automate content creation or analyze data; they want models that can take action. That makes so-called agent AI systems pose even greater potential risks. If an attacker can trick an LLM into behaving unexpectedly, allowing the AI system to take action in another application, the results could be dramatic, Harang says.
“We have seen recent examples in other systems where the use of tools has led to unexpected activity and unexpected leakage of information in LLMs,” he said, adding, “We are working to improve our capabilities, including in our use of tools, but I think this will be a continuous learning process for the industry.”
Harang points out that even though the risks are great, it’s important to realize that they’re solvable problems: He avoids using hyperbolic “sky falling” language about the risks of using GenAI, saying he mostly uses it to find specific information, like the syntax of certain programming functions, or to summarize academic papers.
“We’ve gotten a lot better at understanding how LLM-integrated applications work, and I think we’ve learned a lot over the past year or so about how to secure LLMs and how to build security from ground up,” he said.
