SyncAI.news, a Varaisys broadcasting
Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets
AR

Asif Razzaq

· 6 min read

EngineeringMarkTechPost

Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets

Google team have released llm-d 1.0, the first stable version of its distributed inference stack. This open-source, Kubernetes-native layer turns vLLM or SGLang model servers into one coordinated pool. They released it at the PyTorch Conference North America. It targets self-hosted agent and RL workloads, where round-robin load balancing breaks down.

TL;DR

  • A serving stack: Google reports a published benchmark on a 17,000-node multi-region GKE deployment.
  • Runs on: Kubernetes with NVIDIA, AMD or Intel GPUs or Google TPUs, over vLLM or SGLang.
  • Performance: Vendor-reported gains come from cache-aware routing, tiered KV cache offload and prefill/decode disaggregation.
  • Best: Median TTFT cut by more than 85%, with 2.4x throughput, on an Ironwood TPU coding workload.
  • Worst: RL time-slicing lifts accelerator duty cycle only to above 70%, so roughly 30% idle time remains.

What is llm-d 1.0?

llm-d was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA. Red Hat, IBM and Google are the lead maintainers. The project counts more than 500 contributors from over 25 organizations. It is a CNCF Sandbox project, accepted on March 12, 2026, and is licensed under Apache 2.0.

Prefix-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch inference already shipped. Version 1.0 is about running them reliably in production, which IBM calls day 2 operations.

What is new in llm-d 1.0?

  • Multi-tenant quality of service: Priority and fairness-aware flow control stops bursty agents from starving latency-sensitive requests.
  • Stable foundation: v1 APIs and a hardened request path, validated on multiple clouds and accelerators.
  • Zero-downtime upgrades: Router high availability with hot standby, plus DisaggregatedSet, which rolls prefill and decode pools together.
  • Agent-aware serving: Session-aware routing, turn-priority scheduling and native Responses API state.
  • Async inference: llm-d-async-processor pulls jobs from Google Cloud Pub/Sub or Redis into slack capacity.
  • Expanded TPU support: TorchTPU runs PyTorch models natively inside upstream vLLM and SGLang.

How does llm-d route requests?

The llm-d router builds on the Kubernetes Gateway API Inference Extension. It tracks which replicas hold each cached prefix. It then weighs load and predicted latency before picking a replica. Agents call one endpoint, and each step lands where its context already lives.

When accelerator memory fills, KV cache moves to host memory, then SSD or Managed Lustre. It uses vLLM’s OffloadingConnector, Mooncake or SGLang’s HiCache for this. Prefill and decode run as separate pools, because prefill is compute-heavy and decode is memory-bound.

&lt;/span&gt;&lt;b&gt;Agent request&lt;/b&gt;&lt;small&gt;1 endpoint&lt;/small&gt;&lt;/div&gt; &lt;div class=&quot;node&quot;&gt;&lt;span class=&quot;ic&quot;&gt;<img src="https://s.w.org/images/core/emoji/17.0.2/72x72/1f9ed.png" alt="🧭" class="wp-smiley" style="height: 1em; max-height: 1em;" />&lt;/span&gt;&lt;b&gt;llm-d router&lt;/b&gt;&lt;small&gt;prefix + load aware&lt;/small&gt;&lt;/div&gt; &lt;div class=&quot;node&quot;&gt;&lt;span class=&quot;ic&quot;&gt;<img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2699.png" alt="⚙" class="wp-smiley" style="height: 1em; max-height: 1em;" />&lt;/span&gt;&lt;b&gt;Prefill / decode&lt;/b&gt;&lt;small&gt;separate pools&lt;/small&gt;&lt;/div&gt; &lt;div class=&quot;node&quot;&gt;&lt;span class=&quot;ic&quot;&gt;<img src="https://s.w.org/images/core/emoji/17.0.2/72x72/1f5c4.png" alt="🗄" class="wp-smiley" style="height: 1em; max-height: 1em;" />&lt;/span&gt;&lt;b&gt;Tiered KV cache&lt;/b&gt;&lt;small&gt;HBM to Lustre&lt;/small&gt;&lt;/div&gt; &lt;/div&gt; &lt;div class=&quot;pkt&quot;&gt;&lt;span&gt;&lt;/span&gt;&lt;/div&gt; &lt;div class=&quot;exp&quot; id=&quot;mtp-exp&quot;&gt;&lt;/div&gt; &lt;div class=&quot;tiers&quot; id=&quot;mtp-tiers&quot;&gt; &lt;div style=&quot;background:rgba(66,133,244,.15)&quot;&gt;Accelerator HBM&lt;/div&gt; &lt;div style=&quot;background:rgba(52,168,83,.15)&quot;&gt;Host CPU RAM&lt;/div&gt; &lt;div style=&quot;background:rgba(251,188,4,.15)&quot;&gt;Local SSD&lt;/div&gt; &lt;div style=&quot;background:rgba(234,67,53,.15)&quot;&gt;Managed Lustre&lt;/div&gt; &lt;/div&gt; &lt;div class=&quot;ctl&quot;&gt; &lt;button class=&quot;btn gh&quot; id=&quot;mtp-prev&quot;&gt;Back&lt;/button&gt; &lt;button class=&quot;btn&quot; id=&quot;mtp-next&quot;&gt;Next step&lt;/button&gt; &lt;button class=&quot;btn gh&quot; id=&quot;mtp-play&quot;&gt;Auto-play&lt;/button&gt; &lt;span class=&quot;st&quot; id=&quot;mtp-st&quot;&gt;Step 1 of 4&lt;/span&gt; &lt;/div&gt; &lt;/div&gt; &lt;div class=&quot;pn&quot;&gt; &lt;div class=&quot;seg&quot;&gt; &lt;button class=&quot;btn&quot; data-g=&quot;lat&quot;&gt;Latency and cache&lt;/button&gt; &lt;button class=&quot;btn gh&quot; data-g=&quot;thr&quot;&gt;Throughput&lt;/button&gt; &lt;/div&gt; &lt;div class=&quot;bars&quot; id=&quot;mtp-bars&quot;&gt;&lt;/div&gt; &lt;div class=&quot;note&quot;&gt;All figures self-reported by Google, Snap and Zenteiq. Source: Google Cloud blog, Oct 20, 2026.&lt;/div&gt; &lt;/div&gt; &lt;div class=&quot;pn&quot;&gt; &lt;div class=&quot;grid&quot;&gt; &lt;div class=&quot;card&quot;&gt;&lt;span class=&quot;k&quot;&gt;Multi-tenant QoS&lt;/span&gt;&lt;b&gt;Flow control&lt;/b&gt;&lt;p&gt;Priority and fairness-aware routing keeps bursty agent loops from starving interactive users.&lt;/p&gt;&lt;/div&gt; &lt;div class=&quot;card&quot;&gt;&lt;span class=&quot;k&quot;&gt;Upgrades&lt;/span&gt;&lt;b&gt;DisaggregatedSet&lt;/b&gt;&lt;p&gt;Built on LeaderWorkerSet. Rolls prefill and decode pools together on matching versions without dropping sessions.&lt;/p&gt;&lt;/div&gt; &lt;div class=&quot;card&quot;&gt;&lt;span class=&quot;k&quot;&gt;Reliability&lt;/span&gt;&lt;b&gt;Router HA&lt;/b&gt;&lt;p&gt;Hot standby and rolling updates keep traffic flowing through upgrades and maintenance.&lt;/p&gt;&lt;/div&gt; &lt;div class=&quot;card&quot;&gt;&lt;span class=&quot;k&quot;&gt;Agents&lt;/span&gt;&lt;b&gt;Session-aware routing&lt;/b&gt;&lt;p&gt;Turn-priority scheduling and native Responses API state help agents reuse context across turns.&lt;/p&gt;&lt;/div&gt; &lt;div class=&quot;card&quot;&gt;&lt;span class=&quot;k&quot;&gt;Background jobs&lt;/span&gt;&lt;b&gt;Async processor&lt;/b&gt;&lt;p&gt;Pulls requests from Google Cloud Pub/Sub or Redis and runs them in slack capacity.&lt;/p&gt;&lt;/div&gt; &lt;div class=&quot;card&quot;&gt;&lt;span class=&quot;k&quot;&gt;TPU&lt;/span&gt;&lt;b&gt;TorchTPU time-slicing&lt;/b&gt;&lt;p&gt;Interleaves inference into idle RL cycles. Duty cycle rises from about 40% to above 70%.&lt;/p&gt;&lt;/div&gt; &lt;/div&gt; &lt;div class=&quot;note&quot;&gt;Click a card to reveal details. Stable v1 APIs ship across all components.&lt;/div&gt; &lt;/div&gt; &lt;div class=&quot;pn&quot;&gt;&lt;div class=&quot;tw&quot;&gt; &lt;table&gt; &lt;tr&gt;&lt;th&gt;&lt;/th&gt;&lt;th class=&quot;us&quot;&gt;llm-d 1.0&lt;/th&gt;&lt;th&gt;NVIDIA Dynamo&lt;/th&gt;&lt;th&gt;AIBrix&lt;/th&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Version&lt;/td&gt;&lt;td&gt;1.0 (Oct 2026)&lt;/td&gt;&lt;td&gt;v1.5.1 (Oct 2026)&lt;/td&gt;&lt;td&gt;v0.7.0 (Jun 2026)&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Governance&lt;/td&gt;&lt;td&gt;CNCF Sandbox&lt;/td&gt;&lt;td&gt;NVIDIA-led&lt;/td&gt;&lt;td&gt;vLLM project org&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Engines&lt;/td&gt;&lt;td&gt;vLLM, SGLang&lt;/td&gt;&lt;td&gt;SGLang, TRT-LLM, vLLM&lt;/td&gt;&lt;td&gt;vLLM, SGLang, TRT-LLM&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;Accelerators&lt;/td&gt;&lt;td&gt;NVIDIA, AMD, Intel, TPU&lt;/td&gt;&lt;td&gt;NVIDIA GPUs&lt;/td&gt;&lt;td&gt;Mixed GPUs&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;KV tiers&lt;/td&gt;&lt;td&gt;HBM, CPU, SSD, Lustre&lt;/td&gt;&lt;td&gt;GPU, CPU, SSD, remote&lt;/td&gt;&lt;td&gt;Distributed KV cache&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;License&lt;/td&gt;&lt;td&gt;Apache 2.0&lt;/td&gt;&lt;td&gt;Apache 2.0&lt;/td&gt;&lt;td&gt;Apache 2.0&lt;/td&gt;&lt;/tr&gt; &lt;tr&gt;&lt;td&gt;GitHub stars&lt;/td&gt;&lt;td&gt;4.8k&lt;/td&gt;&lt;td&gt;8.3k&lt;/td&gt;&lt;td&gt;5.1k&lt;/td&gt;&lt;/tr&gt; &lt;/table&gt;&lt;/div&gt; &lt;div class=&quot;note&quot;&gt;Stars as of Oct 20, 2026. Sources: each project's GitHub repo and docs.&lt;/div&gt; &lt;/div&gt; &lt;div class=&quot;ft&quot;&gt;&lt;span&gt;Sources: Google Cloud, IBM Research, Red Hat, GitHub&lt;/span&gt;&lt;a href=&quot;https://www.marktechpost.com&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;© Marktechpost&lt;/a&gt;&lt;/div&gt; &lt;/div&gt; &lt;script&gt; (function(){ var R=document.getElementById('mtp-llmd1'); var tabs=R.querySelectorAll('.tabs button'),pns=R.querySelectorAll('.pn'); tabs.forEach(function(b){b.onclick=function(){tabs.forEach(function(x){x.classList.remove('on')});pns.forEach(function(x){x.classList.remove('on')});b.classList.add('on');pns[+b.dataset.t].classList.add('on');if(+b.dataset.t===1)draw(cur);rs();};}); var steps=[ 'An agent sends one step of a multi-turn task to a single llm-d endpoint. TPU and GPU pools sit behind it, running vLLM or SGLang.', 'The router tracks which replicas already hold each cached prefix. It weighs load and predicted latency, then sends the step where its context lives.', 'Prefill (compute-heavy) and decode (memory-bound) run in separate pools. DisaggregatedSet upgrades both together without dropping sessions.', 'When accelerator memory fills, KV cache moves to host RAM, then SSD or Google Cloud Managed Lustre instead of being evicted.']; var s=0,nodes=R.querySelectorAll('.node'),pk=R.querySelector('.pkt span'),ex=document.getElementById('mtp-exp'),st=document.getElementById('mtp-st'),ti=R.querySelectorAll('#mtp-tiers div'),timer=null; function show(){nodes.forEach(function(n,i){n.classList.toggle('act',i===s)});pk.style.left=(s*25)+'%';ex.textContent=steps[s];st.textContent='Step '+(s+1)+' of 4'; ti.forEach(function(t,i){t.classList.remove('lit');if(s===3)setTimeout(function(){t.classList.add('lit')},i*350);else if(i===0)t.classList.add('lit')});} document.getElementById('mtp-next').onclick=function(){s=(s+1)%4;show()}; document.getElementById('mtp-prev').onclick=function(){s=(s+3)%4;show()}; document.getElementById('mtp-play').onclick=function(){var b=this;if(timer){clearInterval(timer);timer=null;b.textContent='Auto-play';return}b.textContent='Pause';timer=setInterval(function(){s=(s+1)%4;show()},2600)}; show(); var data={lat:[ ['Ironwood TPU: median TTFT cut','&gt;85%',85,'#4285F4','agentic coding vs plain K8s Service'], ['Gemini: P95 TTFT cut (bursty chat)','&gt;50%',50,'#34A853','GKE Inference Gateway'], ['A3 Mega 70B: TTFT improvement','~40%',40,'#FBBC04','Lustre + CPU offload, 50k-token prompts'], ['Gemini: TTFT cut (coding agents)','&gt;35%',35,'#EA4335','production traffic'], ['Gemini: prefix cache hit rate','35% to 70%',70,'#4285F4','doubled']], thr:[ ['Zenteiq: aggregate throughput','&gt;4x',4,'#34A853','1.47B to 5.97B tokens/day per 1,000 TPU v6e'], ['17,000-node GKE: throughput','2.9x',2.9,'#4285F4','3 clusters, 99.9% success'], ['Ironwood TPU: total throughput','2.4x',2.4,'#FBBC04','vs plain K8s Service']]}; var cur='lat'; function draw(g){cur=g;var box=document.getElementById('mtp-bars'),mx=g==='thr'?4:100;box.innerHTML=''; data[g].forEach(function(d){var r=document.createElement('div');r.className='row';r.innerHTML='&lt;div class=&quot;lb&quot;&gt;&lt;span&gt;'+d[0]+' &lt;em&gt;'+d[4]+'&lt;/em&gt;&lt;/span&gt;&lt;b&gt;'+d[1]+'&lt;/b&gt;&lt;/div&gt;&lt;div class=&quot;tr&quot;&gt;&lt;div class=&quot;fl&quot; style=&quot;background:'+d[3]+'&quot;&gt;&lt;/div&gt;&lt;/div&gt;';box.appendChild(r); var f=r.querySelector('.fl');setTimeout(function(){f.style.width=(d[2]/mx*100)+'%'},60);});rs();} R.querySelectorAll('[data-g]').forEach(function(b){b.onclick=function(){R.querySelectorAll('[data-g]').forEach(function(x){x.classList.add('gh')});b.classList.remove('gh');draw(b.dataset.g)}}); R.querySelectorAll('.card').forEach(function(c){c.onclick=function(){c.classList.toggle('open');setTimeout(rs,380)}}); function rs(){try{parent.postMessage({mtpH:R.offsetHeight+40,id:'mtp-llmd1'},'*')}catch(e){}} window.addEventListener('load',rs);window.addEventListener('resize',rs);setTimeout(rs,300); })(); &lt;/script&gt; &lt;/body&gt;&lt;/html&gt; " style="width:100%;max-width:880px;height:600px;border:0;display:block;margin:0 auto;" loading="lazy">

What do the llm-d 1.0 benchmarks show?

All figures below are self-reported by Google and llm-d adopters.

  • Ironwood TPU (TPU 7x): prefix-aware routing plus host memory offload cut median TTFT by more than 85%. Throughput rose 2.4x versus a plain Kubernetes Service.
  • Gemini platform: prefix cache hit rate doubled from 35% to 70%. TTFT fell more than 35% for coding agents. P95 TTFT fell more than 50% under bursty chat.
  • 17,000-node GKE benchmark: throughput scaled 2.9x across 3 clusters, with 99.9% success and under 1% overhead.
  • A3 Mega, 70B model: Lustre plus CPU RAM offload improved TTFT by about 40% on 50k-token prompts. End-to-end latency fell 30% and GPU-hours fell nearly 60%.
  • RL time-slicing: accelerator duty cycle rose from about 40% to above 70%.

Adopters report similar patterns. Snap reports prefix cache hit rates of up to 75 to 80%. Zenteiq reports over 4x throughput, from 1.47 billion to 5.97 billion daily tokens per 1,000 TPU v6e chips. An IBM demo served 3,000 concurrent coding agents at 5 to 10 times below API pricing.

How does llm-d compare with NVIDIA Dynamo and AIBrix?

These are not strict rivals. CoreWeave runs llm-d alongside Dynamo, and NVIDIA contributes Dynamo components to llm-d.

How do you deploy llm-d 1.0?

On Google Cloud, GKE Inference Gateway runs the llm-d router as a managed service. It can also span clusters and regions with global failover. KEDA and HPA scale pools on queue depth and predicted latency. Kueue enforces per-model quotas. Verified Helm charts deploy TPU and GPU pools behind one endpoint.

Google team suggests a staged path. Start with prefix-aware routing in front of an existing vLLM deployment. Then add flow control, tiered KV offload and disaggregation. Published runs can be compared on Prism.

Key Takeaways

  • llm-d 1.0 is the first stable release, with v1 APIs.
  • New: flow control, router HA, DisaggregatedSet and agent-aware routing.
  • Gemini doubled prefix cache hits from 35% to 70%.
  • A 17,000-node GKE benchmark scaled throughput 2.9x.

Original source

This story was published by MarkTechPost and written by Asif Razzaq. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on marktechpost.com

Similar News