Understanding prompt injections: a frontier security challenge

AI tools are starting to do more than respond to questions. They can now browse the web, help with research, plan trips, and help buy products. As they become more capable, with the ability to access your data in other apps and take actions on your behalf, new security challenges emerge. One we’re heavily focused on is prompt injection.

What is a prompt injection?

Prompt injection is a type of social engineering attack specific to conversational AI. Early AI systems were conversations between a single user and a single AI agent. In AI products today, your conversation may include content from many sources, including the internet. The idea that a third-party (that is not the user and not the AI) could mislead the model by injecting malicious instructions into the conversation context led to the term “prompt injection”.

In the same way that phishing emails or scams on the web attempt to trick people into giving away sensitive information, prompt injections attempt to trick AIs into doing something you did not ask for.

Imagine you have asked an AI to help you do some vacation research online, and while it is doing that it encounters misleading content or harmful instructions hidden on a webpage, such as in a comment on a listing or on a review. The content could be carefully crafted in an attempt to trick an AI into recommending the wrong listing, or worse, to steal your credit card information.

These are just a few examples of “prompt injection” attacks—harmful instructions designed to trick an AI into doing something you didn’t intend, often hidden inside ordinary content such as a web page, document, or email.

These risks increase as AIs have access to more sensitive data and take on more initiative and longer tasks.

Our approach to protecting users

To protect our users, and to help improve our models against these attacks, we take a multi-layered approach, including the following: