vLLM scales local LLM serving to production throughput
vLLM is a full-featured serving engine for open-weight models that handles thousands of requests per second. It picks up where llama.cpp and Ollama leave off — those tools make a single model runnable on a laptop, but vLLM targets production serving with continuous batching, prefix caching, and structured output support built in.
How batching and KV cache drive vLLM performance
Large language models generate text one token at a time. Each new token requires a forward pass across all weights with the preceding tokens as context. A 7-billion-parameter model in 16-bit precision holds 14 GB of weights, and memory bandwidth becomes the immediate bottleneck. A notebook generating one answer at a time is far less efficient than a server handling many users simultaneously.
Transformers avoid recomputing attention over the prefix at every step by keeping a KV cache. That cache grows with sequence length — each request keeps keys and values of every previous token resident in GPU memory. Naive implementations pre-allocate the maximum possible size, which limits batch sizes to a handful of requests.
vLLM introduces two ideas that change the calculus. PagedAttention stores the KV cache in fixed-size blocks drawn from a common pool, eliminating fragmentation and the need to reserve everything up front. Continuous batching updates the running batch by replacing requests that finish earlier with new incoming requests, so capacity never waits on the slowest member of the batch. Together these two mechanisms let vLLM maximize throughput on available hardware.
Model selection and quantization for unified memory hardware
The evaluation runs on NVIDIA DGX Spark, a compact box built around a GB10 Grace Blackwell chip with 128 GB of unified LPDDR5x memory running at 273 GB/s. The total memory is shared between CPU and GPU, so in principle any open model under 120B parameters can load, but bandwidth constraints make larger models sluggish.
The authors define "comfortable" as a model whose weights take less than one-third of available memory, leaving room for the KV cache and the operating system, with a single-stream ceiling above 40 tokens per second — faster than human reading speed. The test table makes three things clear. First, bandwidth not memory is the primary constraint. Second, FP8 quantization cuts memory requirements in half versus full BF16 with negligible quality loss and a significant throughput boost on Blackwell's native FP8 tensor cores. Third, Mixture-of-Experts models such as GPT-OSS or Qwen-3.5 dominate throughput numbers because only a small fraction of total parameters processes each token.
Based on those results the authors choose Qwen3.5-35B-A3B with FP8 quantization: 37.5 GB on disk, 3B active parameters, Apache-2.0 license, native vLLM support, and a thinking mode that can be turned on or off. Gemma 4's 26B-A4B is an alternative if slightly fewer tokens per second are acceptable.
Python setup and first-run behavior
Using vLLM from Python requires a few more arguments than llama-server with a GGUF file. The LLM class must be given the model path, maximum context length, maximum concurrent sequences per scheduler step, GPU memory utilization fraction, and prefix-caching enablement. Multimodal limits can be set to zero if vision support is never needed.
from vllm import LLM, SamplingParams
llm = LLM(
model="Qwen/Qwen3.5-35B-A3B-FP8",
max_model_len=4096,
max_num_seqs=256,
gpu_memory_utilization=0.50,
enable_prefix_caching=True,
limit_mm_per_prompt={"image": 0, "video": 0},
reasoning_parser="qwen3",
seed=42,
)max_model_len plays a role similar to llama.cpp's -c flag, but the model supports 262k tokens; 4096 is sufficient for the use case and frees room for many more parallel requests. gpu_memory_utilization caps the memory the GPU may request — important in unified memory systems where the GPU budget competes with the OS and concurrent processes. Loading the model and instantiating the engine takes about 4.5 minutes on first start: 101 seconds to load 14 safetensors shards plus a memory-profiling pass. torch.compile results get cached to disk, so subsequent starts are far quicker.
vLLM uses as much memory as it can up to the gpu_memory_utilization limit. Anything not used by weights and activations becomes reserved for the KV cache. In effect the memory budget knob functions as a cache-size knob.
Batch throughput scales with concurrency
To measure throughput the authors used the OpenAlex corpus of 11,700 scaling-laws papers, filtering to a random sample of 2,000 and dropping 18 "abstracts" that were actually full papers. Prompts were kept under the 4096-token max_model_len to avoid batch drops. The prompt asked the model to read a scientific abstract and answer in exactly one sentence of at most 30 words: what quantity scales with what in this paper? If the paper is not about a scaling relationship, say so.
Batch sizes ranged from 1 to 256, doubling at each step. Throughput climbed steadily: 34 tokens per second for a single request, reaching 284 tokens per second at the 256-request level — well above the predicted 91 token per second ceiling. An eightfold increase on the same hardware with the same model. The takeaway for harness design: whenever a step can be formulated as applying the same action to multiple inputs, doing it as a single batch is far more effective than a tool call per item.
Prefix caching eliminates redundant prompt computation
When the same prompt repeats across requests — the single-instruction-multiple-data pattern — the KV tensors for the initial prompt are identical in every request. vLLM's PagedAttention mechanism can cache them automatically. This is prefix caching.
In a cold run, the first execution computes the KV block that the remaining requests then reuse. Sixty-nine percent of prompt tokens came from cache on the first request, with the other 255 requests in a 256-batch benefiting from that computation. A warm run shows essentially no change because all caching benefits were already captured. Across 389k total input tokens only 120k were actually processed, and wall-clock time remained effectively the same across runs. Runtime is dominated by output generation, not input processing. Caching saves computational power and memory bandwidth.
Grammar-constrained structured outputs
Many pipelines need LLM output as valid JSON, but models do not always cooperate. Adding a parser, validator, and retry loop into the harness is the typical workaround. vLLM provides grammar-constrained decoding instead: it compiles a provided JSON pydantic schema and filters out any tokens that would produce an invalid output.
The authors define a pydantic model class with the fields they need — scaling_domain, quantity_scaled, scaled_against, is_empirical, claims_power_law, and one_line_finding. The schema gets passed to SamplingParams via StructuredOutputsParams. With temperature set to zero and the structured-outputs flag enabled every valid Python object the model produces can flow directly into downstream analysis without extra post-processing.
The entire dataset of 1,982 abstracts processed in about eight minutes. That yielded 1.35M input tokens at 2,820 tokens per second and 185k generated tokens at 384 tokens per second. Every output was a valid Python object ready for the pipeline.
What this means for self-hosted deployments
vLLM demonstrates that local serving can match cloud APIs on throughput when the right hardware and configuration choices are made. Continuous batching, prefix caching, and structured-output support are not optional extras — they are the mechanisms that make serving practical at scale. Teams building retrieval-augmented generation, automated labeling, or batch data extraction can expect an order-of-magnitude improvement in tokens per second over naive single-request loops. The framework's design explicitly separates weight memory from cache memory, which gives operators a lever to tune for their specific hardware constraints. For anyone self-hosting open-weight models at scale, vLLM is the starting point.