Your AI Isn't Delivering Results Because You're Measuring The Wrong Things

Kerry Brown, Transformation Evangelist at Celonis, is a strategist and thought leader who helps companies achieve organizational excellence.

​A new number has taken pride of place on enterprise dashboards: adoption. How many employees signed into an AI tool this week? How many prompts did they run? How many tokens did they consume? How many licenses are active?

For leaders under pressure to show their organizations are “doing AI,” these numbers can be reassuring, but they answer the wrong question. The real measure of AI success is whether AI makes the business work better.

I’ve spent my career on the people side of digital transformation, watching how jobs and organizations change when new technology arrives. My litmus test for any technology is simple: How do people’s jobs change? This translates into whether the technology helps someone do their work better, faster or smarter. Usage statistics may show adoption, but they don’t show whether the business is improving.

That distinction is the whole problem. A 2025 Boston Consulting Group study found that only 5% of surveyed companies were generating substantial value from AI at scale, while 60% were realizing hardly any material value despite significant investment. That’s not necessarily a verdict on the technology. Often, it reflects a more fundamental problem: The organization missed the mark on defining success before implementation began.

A more useful approach starts with three questions: What business outcome are we trying to change? How do we perform today? Did the outcome improve, and at what cost?

Activity Is Not Achievement

Metrics such as licenses, prompts and tokens are useful. They can indicate whether employees have access to AI, whether adoption is growing and where additional training may be needed, but they’re leading indicators at best.

Building Measurement Into The AI Strategy

Leaders can make AI investments more measurable by asking three questions from the start: