Why inference stacks break under agent workloads
A year ago most inference traffic was chat: one prompt in, one response out. Today workloads are shifting to multi-step agents and reinforcement learning rollouts that fan out across tools, share long prefixes, and arrive in bursts. Serving these patterns on a single machine or behind a generic round-robin load balancer breaks down fast. Replicas recompute the same prompts. Accelerator memory fills with duplicate state. One bursty agent loop can stall interactive users.
llm-d solves this by turning a cluster of model servers into a coordinated serving pool. During the conference keynote the team demonstrated 1,000 sandboxed agents on Agent Substrate on GKE hitting a single llm-d endpoint across vLLM, SGLang, and a mixed fleet of Google Cloud TPUs and GPUs. As traffic scaled through three stages — round-robin load balancing, cache-aware routing, then tiered KV cache with priority flow control — each agent's steps stayed on the replicas holding its context while bursty loops were queued so interactive requests remained fast.
How cache-aware routing works
The llm-d router builds on Gateway API Inference, a Kubernetes project for inference-optimized routing. On GKE, GKE Inference Gateway runs the router as a managed service, simplifying installation and extending routing across regions. The router tracks which replicas hold each cached prefix and weighs load and predicted latency when picking one. When accelerator memory fills up, rather than dropping the cache, llm-d uses vLLM's OffloadingConnector, Mooncake, or SGLang's HiCache to move KV cache to host memory and then to SSD or Google Cloud Managed Lustre. This prevents prefix-eviction storms under heavy multi-turn and long-context conversational loads across mixed TPU and GPU fleets.
On an agentic coding workload on Ironwood TPU (TPU 7x), prefix-aware routing plus host memory offload cut median time-to-first-token by more than 85 percent and raised total throughput 2.4x versus a plain Kubernetes Service, as of October 2026.
Disaggregation that upgrades safely
Prefill is compute-heavy and decode is memory-bound. Running both on the same replicas makes each wait on the other. llm-d runs prefill, decode, and multimodal encode as separate pools, with wide expert parallelism spreading large mixture-of-experts models across nodes. In 1.0, DisaggregatedSet — built on LeaderWorkerSet, the Kubernetes API for multi-host inference — manages these pools as one unit. Rolling updates move prefill and decode together on matching versions while each phase scales independently.
Async inference and time-slicing idle capacity
For latency-tolerant traffic, llm-d 1.0 introduces llm-d-async-processor, which pulls requests from queues like Google Cloud Pub/Sub or Redis and routes them alongside real-time traffic. Queue and pool-dispatch gates check available capacity and enforce strict priority lanes and quotas so background jobs run in slack capacity without starving interactive users.
During RL training, accelerators often sit idle 40 to 60 percent of the time during rollout generation or weight updates. llm-d now supports Google Cloud TPU and GPU time-slicing to interleave async inference into those quiet cycles, raising aggregate accelerator duty cycles from a roughly 40 percent baseline to over 70 percent.
Production results at fleet scale
The GKE llm-d stack carries production traffic and has been tested at fleet scale. Gemini platform (formerly Vertex AI) serves production traffic through GKE Inference Gateway, doubling its prefix cache hit rate from 35 percent to 70 percent, cutting TTFT by more than 35 percent for coding agents and cutting P95 TTFT by more than 50 percent under bursty chat traffic.
In a published benchmark across a 17,000-node multi-region GKE deployment serving a mixture-of-experts model in the US and Europe, multi-cluster GKE Inference Gateway and the llm-d router scaled throughput 2.9x across three clusters with a 99.9 percent success rate and less than 1 percent routing overhead.
On an A3 Mega cluster serving a 70B-parameter model, llm-d with multi-connector KV cache offloading to Google Cloud Managed Lustre in addition to host CPU RAM delivered an approximately 40 percent improvement in TTFT and a 30 percent reduction in end-to-end latency on 50k-token prompts compared to CPU RAM offload only, while reducing GPU-hour requirements by nearly 60 percent.
What this means for teams running models
Organizations running llm-d on Google Cloud apply these capabilities in different ways. Snap uses prefix-cache-aware routing to achieve prefix cache hit rates of 75 to 80 percent and values the open-source nature for integration with their Envoy-based Service Mesh. ZenteiQ leverages load-aware and prefix-cache-aware routing to achieve over 4x higher aggregate inference throughput, increasing daily generated token capacity from 1.47 billion to 5.97 billion tokens per 1,000 TPU v6e accelerators. Mistral enforces strict priority bands across commercial tiers, automatically holding or shedding lower-priority requests when pools saturate so paid traffic meets latency SLOs.
The components are designed so you can compose them into your own production environment. Keep your engine, model, and cluster, and adopt only the serving components and configurations you need. llm-d builds on Kubernetes components you may already run: Gateway API, LeaderWorkerSet, Dynamic Resource Allocation, Prometheus, and OpenTelemetry.
Getting started with llm-d
Deploy llm-d 1.0 on GKE with the verified Helm charts. Start with prefix-aware routing in front of your existing vLLM deployment, then add flow control, KV cache offload (SGLang, vLLM) and disaggregation. Compare results on Prism, the open-source analysis dashboard for exploring published benchmarks backed by standardized llm-d-benchmark recipes, and contribute on GitHub.