The Machine Can Be Persuaded: Why AI's Context Is Becoming A Security Boundary

Carlo Tortora Brayda is Chairman of the Board at Cyber Eagle.

​Throughout most technology professionals’ careers, the understanding has always been the same: People can be manipulated, and computer systems can be hacked.

So we trained our workforces to recognize phishing emails and social engineering. Cyber tools hardened our digital life. But now, in this age of AI, counterintuitively, these two perspectives are blending into each other.

We sometimes forget how symbiotic AI is. AI systems amplify all that is human; they are trained on human language, trained through human interactions (the chats) and designed to adapt. Although they are not conscious, AI systems can convincingly reproduce human-like behavior and, more unsettlingly, can be manipulated through the same patterns of language, authority and persuasion that influence people.​

Now with agentic AI, the models have hands and feet (metaphorically) to move around and do things. They have agency. They can effectively assist humans through implicit authority to act. That is a consequential security risk that we can’t even begin to quantify.

Much of the discussion around generative AI security has focused on prompt injection: An attacker gives a model instructions intended to override its original purpose or safeguards.

Think of it this way: A hacker can construct a narrated reality that the model treats as authoritative, much like immersing a child in a fairy tale so completely that the story becomes the frame through which everything else is interpreted. In cyber terms, actions that once seemed unacceptable can become increasingly consistent with the new reality the attacker has constructed.​ Orwellian, really.

OWASP describes a closely related problem as “excessive agency.” Essentially, agent identity and authority also become part of the security perimeter.

We now need to think bigger than ever because while we fight our newest adversary, we are also wrestling with our oldest friend—the power of a compelling story.​