Which AI Workload Goes Where? A CIO's Guide To Hybrid AI

Vinod Bijlani is an AI practice leader at Hewlett Packard Enterprise.

​Almost every enterprise AI conversation I have now starts with some version of the same question: Where should this actually run?

A CIO wants to know if a new fraud-detection model belongs on-premises or in the cloud. A head of data platform wants to know if last year’s “just use the API” decision still holds now that volume has 10x’d. A CISO wants to know why a vendor is pushing a public-cloud deployment when the data is regulated.

Even though these are essentially the same underlying question, I’ve found that almost nobody answers with a consistent framework. Placement decisions made by default, vendor relationships or pilot convenience deserve another look as AI becomes a production service.

Gartner forecasts worldwide AI-optimized infrastructure-as-a-service spending will reach $42 billion in 2026, with inference accounting for 55% of that spending and rising to 59% in 2027. The numbers are forecasts, but the direction is clear: Recurring inference demand changes the placement calculation.

I believe AI will be hybrid, much like enterprise cloud. Early cloud debates centered on whether everything would move to the public cloud; the more useful question became which workloads belonged where, based on cost, control and performance.

I expect AI to follow a similar path, combining hyperscalers, neoclouds, self-hosted infrastructure and edge. While these options can overlap, the workload should determine the mix.​

Hyperscaler: The Generalist

​Broad cloud platforms suit workloads that depend on managed models, integrated data services and enterprise support. They also suit experimentation or variable demand.

When adopting cloud platforms, check the specific service, region, capacity limits and data terms. Compliance credentials do not automatically make your application compliant. Cloud providers split duties under a shared responsibility model, and consumption pricing still needs budget controls.