Will On-Device AI Slow The Data Center Boom?

One variation or another of this question arises in pretty much every briefing I do these days: If large amounts of AI inference processing move to devices such as smartphones, PCs, and cars, why are Microsoft, Google, Amazon, and Meta spending hundreds of billions of dollars on data centers? Doesn’t that reduce the pressure on the cloud?

(Disclosure: Microsoft, Amazon, Google, Meta, Qualcomm and Apple are among the many global technology companies that subscribe to research from Creative Strategies, the firm I founded.)

It’s a legitimate question, but it misunderstands the current trend in the industry. Qualcomm, Apple, and PC OEMs have spent three years convincing the world that on-device AI is the way to go and, truth be told, they are right. Today, neural processing units (NPUs) are a common denominator in flagship phones and an increasing number of laptops. Everyday activities such as voice commands, picture processing, translation, and auto-completion are increasingly happening on-device, and that proportion is likely to rise further.

Inference Is Growing, Not Moving From One Place to Another

But there is one misconception in this question that trips me up: “more inference processing moves to the device” and “we need fewer data centers” are two different propositions. Mixing them together obscures what is really driving the build-out.

Inference processing is growing, not decreasing. According to McKinsey’s workload data, inference is forecast to exceed training in terms of data center workloads by 2027, growing at an annual rate of around 35% through the end of the decade.

Even after accounting for a decent chunk of inference processing moving to on-device AI, cloud inference is still growing much faster than data center capacity was ever supposed to grow. A smaller portion of an exponentially growing workload may still become larger in absolute terms than the entire workload five years ago. That’s the math that is often overlooked.