If you have ever watched nvidia-smi report 12 percent GPU compute utilization while users complain about slow generation, you have encountered the central reality of modern LLM inference. Text generation is a memory bandwidth problem masquerading as a compute problem. The GPU spends almost none of its tensor cores on math. It spends nearly all its time shuffling model weights and Key-Value cache tensors across High Bandwidth Memory for every token it produces.

Why the KV cache eats your VRAM

When a transformer processes a prompt, the prefill phase runs dense matrix multiplication across all input tokens in parallel. That is the kind of work GPUs are designed for. Once generation begins, the profile flips. To produce the next token, the model needs self-attention over every preceding token. Recalculating all previous representations from scratch at each step would be an O(N squared) disaster. Instead, the intermediate Key and Value vectors get cached in VRAM.

The memory math is straightforward. For a sequence of length T, the KV cache per layer requires two times the number of KV heads times the head dimension times T times the precision in bytes. Modern architectures like Qwen3.6-27B use Grouped-Query Attention to keep this manageable: 16 full-attention layers with 4 KV heads and a head dimension of 256. At 8,192 tokens in BF16, a single sequence consumes roughly half a gigabyte of KV cache.

That sounds tolerable until you put multiple users on the same GPU. In standard PyTorch or basic Hugging Face pipelines, memory management is primitive. Dynamic containers allocate buffers on the fly. When one user requests a short script and another submits a 6,000-token document, the constant allocation and deallocation turns VRAM into a fragmented mess. You might see 15 GB of total free memory reported, but because it is shattered into non-contiguous fragments, the next request needing a contiguous 2 GB block crashes with a CUDA Out of Memory error.

PagedAttention: a 1960s idea for modern GPUs

In the 1960s, operating system designers solved the same problem for CPU memory. Requiring programs to live in contiguous physical RAM was impractical, so they invented virtual memory paging. The hardware maps arbitrary virtual addresses to scattered physical pages through a page table. vLLM applies this exact concept to GPU memory.

Instead of reserving a giant contiguous chunk of VRAM for each sequence's worst-case length, PagedAttention chops the KV cache into fixed-size physical blocks, each holding 16 or 32 tokens. A centralized block table maps logical token positions to physical blocks. Blocks are allocated on demand as tokens are actually generated, so only the last block in a running sequence has unused slots.

The result is that internal fragmentation drops below 4 percent. The same GPU hardware can host two to four times more concurrent user streams without running out of memory. For reinforcement learning workloads where the model generates 8 or 16 candidate rollouts from the same prompt, Copy-on-Write branching lets all candidates physically share the prompt's KV memory pages. Physical memory is cloned only when individual completions diverge.

Continuous batching and the straggler problem

Traditional static batching works like a bus that refuses to let anyone board until every passenger has reached their final destination. If four requests produce 50, 120, 240, and 1,024 tokens respectively, the GPU sits idle on three slots for hundreds of iterations while waiting for the longest sequence to finish. Those wasted cycles are GPU bubbles.

vLLM implements continuous iteration-level batching. The scheduler makes decisions at the boundary of every forward pass, not at the boundary of entire requests. The moment a sequence emits its end-of-sequence token, its memory blocks return to the pool. A waiting request fills the vacated slot on the very next iteration. The GPU stays continuously saturated.

Hardware-level optimizations

PagedAttention solves the memory footprint. Getting raw throughput out of Hopper silicon requires tackling kernel dispatch overhead. In standard PyTorch eager mode, generating a single token launches dozens of individual GPU kernels across 60-plus transformer layers. On H100 hardware, a single-token GEMV kernel finishes in 3 to 8 microseconds. The CPU driver call to dispatch that kernel takes 10 to 15 microseconds. The GPU spends more time waiting for Python to hand it work than doing math.

CUDA Graphs record the entire operation sequence into a static execution graph during warmup, letting the GPU replay the pipeline in a single dispatch. Chunked prefill breaks long prompts into manageable pieces and interleaves them with decode tokens, keeping latency steady. FP8 quantization doubles effective memory bandwidth and unlocks Hopper's specialized matrix cores.

Benchmark numbers from real hardware

Tests on Qwen3.6-27B running on NVIDIA H100 and H200 clusters show the practical impact. On identical H100 80GB hardware, vLLM with CUDA Graphs cut generation time from 283.7 seconds per step to 78.9 seconds, a 3.59x speedup. Total step time dropped by 2.36x. On H200 hardware, enabling CUDA Graphs reduced generation time by 4.32x for deterministic workloads and 2.95x for policy gradient training.

Tensor parallelism across GPUs requires high-bandwidth NVLink connections. A test using standard VPC network interconnect instead of NVLink saw throughput collapse from 982.6 tokens per second to 75.8 tokens per second. Network latency destroys the benefit of splitting a model across machines. Use data parallelism with independent workers when GPUs are not directly connected.

Production deployment considerations

For enterprise deployments, a two-tier ingress architecture works well. A gateway like LiteLLM handles authentication, team quotas, and audit logging, routing traffic across backend vLLM workers using queue-depth health checks. Model weights should be pre-mounted on local NVMe or high-speed read-only storage so pods boot in seconds rather than downloading 50 GB files on startup.

When running reinforcement learning post-training where the trainer and inference worker share a GPU, setting the GPU memory fraction to 0.35 reserves roughly 28 GB for vLLM while leaving about 50 GB for gradient activations and optimizer states. Neglecting this balance triggers CUDA OOM crashes the moment training loss runs over a long trajectory.

vLLM did not win the inference market through opaque tricks. It applied well-understood operating systems principles to a memory management problem that naive frameworks ignored. PagedAttention and continuous batching are not new ideas in isolation. Putting them together in a system that actually ships, with production telemetry to back the claims, is what made the difference.