​Generative AI Demands A Reimagined Data Center

Xin Guo is CEO of Solidigm.

Until recently, data center infrastructure was used mostly through direct human interaction: A person made a request, and the infrastructure handled it. That model has changed, with one recent report indicating that bot and agent HTML traffic is surpassing human traffic. That gap will continue to widen as trained models and agents are increasingly handling data, planning and execution.

AI is advancing to solve harder problems with increasingly advanced models, and those capabilities are improving quickly. But greater capabilities demand greater infrastructure, and without corresponding increases in compute, memory, storage and networking, those capabilities can’t be achieved. That is fundamentally shifting what we need from AI data infrastructure.

The Limits Of Legacy Infrastructure

We’ve seen many cases where a new model or capability is announced, only to discover that the existing infrastructure doesn’t support scaling. For example, this summer, one fast-growing AI provider had to pause new subscriptions just days after launching a model because demand pushed their GPU capacity to the limit. Single-threaded use of a new capability can look great, but when you try to deploy it across an organization, you have a problem, which can show up as slow or poor output.​

A single prompt from one user in a chat application can generate a gigabyte or tens of gigabytes of context data cache. If you don’t have the infrastructure to store that information and provide it to the model promptly, your answers can be truncated. After a few turns, the system may forget what was already asked and answered. When the system has to re-compute answers to questions it’s already thought about, that’s wasteful and slows down actual throughput.​

A New Data Center Model

Building For Future Workloads

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?