For AI Cloud Providers, Storage Is Increasingly Shaping Inference Efficiency

Compute is the part of the stack most providers are actively solving for. In inference, the layer beneath it has just as much to say about how much a fleet can serve.

Ask a room of AI cloud operators what storage means to their business and the answers fall along a spectrum.

At one end it is a cost to be controlled — overhead sitting next to GPUs, judged on price per terabyte and kept as lean as the workload allows. At the other end it is an enabler — capacity they sell to tenants, certainly, but judged mainly by how productive it makes the accelerators it sits beneath.

Most providers sit somewhere in the middle, and where exactly usually depends on who is answering. But among the providers we work with at Dell, the ones building the most durable businesses tend to sit closer to the second end, and there is a practical reason for that. Treating storage as an enabler puts it in the same conversation as the GPUs it supports, which is where decisions about what a fleet can actually do get made.

Making GPUs productive was once a question of getting data to them quickly enough — a pipeline problem, largely settled during training. It now extends into inference itself, which is why a successful provider is no longer just measured by the number of GPUs in its fleet, but by how much work each of those completes in a day once real tenants are running on it — how many sessions it can hold at once, how quickly it responds when someone’s context runs long, how much of its time goes to producing output rather than reproducing work it had already done. That is inference efficiency.

Providers can’t forecast what tenants will run

Capacity planning helps, but no forecast tells a provider what a tenant will run next quarter. What is within their control is how well the infrastructure uses itself — whether each part of the system is doing the work it was built for, rather than absorbing work that belongs somewhere else. That is where storage shows up as an architectural decision.