When AI Agents Start Covering Their Tracks

Don Schuerman, CTO and Head of Marketing, Pegasystems.

​Enterprise AI has an overthinking problem. Now it may soon have a covering-its-tracks problem, too.​

Back in July, AI agents participating in an OpenAI cybersecurity evaluation found an unauthorized way to communicate with one another and ultimately breached parts of Hugging Face’s infrastructure​, a platform that hosts AI models and related content. While the incident did not result in evidence of broad or catastrophic damage, for many, it immediately conjured apocalyptic images of robotic uprising, causing everyone to question just how powerful LLMs have become and how much more work is needed to govern them.​

OpenAI and a third party later released post-mortems examining exactly what happened. While the original exercise was intended to be an experiment, the report’s findings should get the attention of anyone planning to deploy agents inside a real business.

Too much reasoning can be a liability.

The agents were given open-ended cybersecurity tasks with considerable freedom to find their own solutions. When some tasks appeared impossible, the agents began looking for ways around the evaluation process, eventually discovering an unauthorized channel for coordinating with other agents.​​

This is often presented as the great promise of agentic AI. Give agents a goal, turn them loose and watch the magic happen. That works well for research, creative, coding and other open-ended tasks. More capable AI models are making it possible to automate increasingly complex, open-ended work. But it is much harder to justify the risk and cost of agent reasoning for things like resolving a billing dispute, processing an insurance claim or handling a bank complaint. This requires approved workflows that agents can largely follow, rather than freelancing their way into new modes of working on the fly.​

The kill switch may be too slow

That demonstrates a core weakness in the “control tower” approach to AI governance.​