Designing AI agents to resist prompt injection

AI agents are increasingly able to browse the web, retrieve information, and take actions on a user’s behalf. Those capabilities are useful, but they also create new ways for attackers to try to manipulate the system.

These attacks are often described as prompt injection⁠: instructions placed in external content in an attempt to make the model do something the user did not ask for. In our experience, the most effective real-world versions of these attacks increasingly resemble social engineering more than simple prompt overrides.

That shift matters. If the problem is not just identifying a malicious string, but resisting misleading or manipulative content in context, then defending against it cannot rely only on filtering inputs. It also requires designing the system so that the impact of manipulation is constrained, even if some attacks succeed.

Prompt injection is evolving

Early “prompt injection” type attacks could be as simple as editing a Wikipedia article to include direct instructions to AI agents visiting it; without training-time experience of such an adversarial environment AI models would often follow those instructions without question¹. As models have become smarter, they’ve also become less vulnerable to this kind of suggestion and we’ve observed that prompt injection-style attacks have responded by including elements of social engineering:

Within the wider AI security ecosystem it has become common to recommend techniques such as “AI firewalling” in which an intermediary between the AI agent and the outside world attempts to classify inputs into malicious prompt injection and regular inputs—but these fully developed attacks are not usually caught by such systems. For such systems, detecting a malicious input becomes the same very difficult problem as detecting a lie or misinformation, and often without necessary context.