Push-based KV transfer to improve TTFT, TPOT for disaggregated inference in vLLM

Prefill and decode separated in vLLM

vLLM serves as an open-source inference and serving engine for large language models. LLM inference unfolds in two distinct phases. During prefill the system processes the input prompt and builds the key-value cache. In decode the system uses that cache to generate output one token at a time. These phases carry different compute memory and scaling traits. Prefill-decode disaggregation runs them on separate engine instances letting each phase scale and optimize independently. This arrangement can improve resource utilization throughput and latency under load. However separating the phases also requires moving the KV cache from the prefill instance to the decode instance before token generation can begin making efficient transfer critical to time-to-first-token.

NIXL connector and RDMA transfer design

vLLM moves KV blocks via the NVIDIA Inference Xfer Library connector over interconnects that support Remote Direct Memory Access. AWS Elastic Fabric Adapter provides the low-latency RDMA network used across AWS servers. On supported instance types EFA supports GPUDirect RDMA and Neuron Direct RDMA allowing NIXL to transfer KV blocks directly between GPU or Trainium device memory without staging through host memory. The default NIXL mode operates in pull fashion. After prefill finishes computing the prompt KV cache the router relays source metadata including prefill block IDs to the decode instance. The decode instance then allocates destination blocks prepares the NIXL transfer and initiates NIXL READ to pull the KV blocks from prefill. Because these coordination and setup steps occur sequentially after prefill and before decoding begins they extend the TTFT critical path.

Push-based transfer design

The post introduces a push-based NIXL connector that moves post-prefill transfer coordination and setup off the critical path. In push mode the decode instance registers its target blocks in advance while the producer still computes KV. The transfer begins as soon as the producer's blocks ready rather than after a consumer-side ID-discovery round trip removing that round trip from the path to first token. The transfer-setup time moves off the decode instance running instead on the prefill side and on a background thread. That setup prepares NIXL descriptors and submits the transfer so the decode instance does not spend those cycles during its latency-critical token-generation phase. The net result lower TTFT and higher output-token throughput on the decode instance.

The push connector coordinates three actors. The decode instance registers its target blocks up front. The prefill instance matches each request and issues the transfer. A dedicated background thread carries the KV write off the engine's forward-pass loop. When the decode instance allocates local blocks for a request requiring remote prefilling it prepares a registration describing where the prefill instance should write the KV its own identity and the block locations it has just reserved. Within the same engine step the worker transmits this registration to the prefill instance as a NIXL notification. Because registration happens at allocation time the connector knows the destination blocks before the prefill instance completes prefill.

On the prefill side the registration from the decoder and the completion of prefill can arrive in either order. The connector handles both sequences. If the registration arrives first the producer holds it until its own prefill finishes. If prefill finishes first the producer holds the request's blocks until the registration arrives. Whichever event happens second triggers the transfer. The connector pairs requests by ID and the matching accounts for reissued requests correctly associating preempted or retried requests.

Once a request matches the producer establishes a connection to the decode instance if one does not already exist and issues an NIXL WRITE that copies the KV blocks directly into the decoder's registered memory addressing each destination tensor-parallel rank. Each WRITE carries a completion notification that NIXL delivers to the decode instance once the data arrives. The decoder counts these notifications one per producer rank that writes into it. It treats the KV as received only after all expected notifications have arrived at which point the request proceeds to decode.

Threading model and critical path

The push connector separates model execution from KV-transfer preparation using a dedicated background transfer thread on each prefill worker. At the start of a request the decode main loop allocates its destination KV blocks and registers them with the prefill instance while the prefill main loop computes the prompt KV cache. After both the destination registration and prompt KV ready the prefill main loop hands the KV blocks to the background thread and quickly returns to model execution. The background thread establishes or reuses the NIXL connection maps the request's KV block IDs to memory descriptors merges contiguous blocks into larger transfer regions and submits the NIXL WRITE to the decode instance. When NIXL completes the transfer it sends a completion notification to the decode instance which then begins token generation. Keeping descriptor preparation and transfer submission off the latency-critical main loops prevents this per-request work from delaying model execution.

Pull versus push timing comparison

Both modes use the same NIXL and EFA-RDMA data path. Pull performs the metadata relay and decode-side setup after prefill while push overlaps decode-side setup with prefill. As a result push removes these operations from the post-prefill critical path allowing decoding to begin sooner. The two modes involve a routing tradeoff. Pull mode allows the router to select a decode instance after prefill completes giving greater flexibility for load-aware routing and quality-of-service management. Push mode selects the decode instance earlier so destination setup can overlap with prefill. Under comparable load the prefill instance can release its KV blocks sooner because the transfer starts as soon as both the prompt KV and destination registration ready.

Benchmark results Qwen3-32B

Evaluation used the Qwen3-32B model on a 1P1D deployment using Amazon EC2 P5en instances connected with EFA. To isolate P-to-D transfer performance the benchmark disabled prefix caching with --no-enable-prefix-caching. This configuration makes the decode node retrieve the full KV cache from the prefill node for every request otherwise overlapping prompts could reuse the decode node's local cache and bypass the transfer producing misleading comparison results.

The TTFT comparison shows at low load and short prompts the two modes perform almost identically. As request rate and input length grow pull mode degrades sharply reaching about 1,744 ms at 8 queries per second and 16k input length. Push mode stays far lower at roughly 540 ms around a 3x reduction. The gap opens because pull mode posts NIXL descriptors inline on the decode worker synchronously within its engine loop stalling token generation. Push mode moves that work to a background thread off the critical path.

The TPOT pattern mirrors TTFT. At light load push and pull sit near 5.6 to 5.8 ms. As load and input length rise pull climbs to about 10.4 ms at 4 to 8 QPS and 16k while push holds near 7 ms. Push offloads descriptor preparation and transfer submission from the decode worker. Token generation stays unstarved so per-token latency remains flat even at high concurrency and long contexts.

Deploying push mode in vLLM

To reproduce the 1P1D benchmark configuration one prefill node and one decode node using Amazon EC2 P5en instances connected with EFA refer to the detailed instructions in the Reproducing our results guide. The guide covers environment setup launching the vLLM prefill and decode servers running the benchmark and collecting results. The feature can run on other supported configurations but performance results vary depending on hardware model workload and network configuration.

To use push mode configure both the prefill and decode servers to use NixlPushConnector instead of NixlConnector. Choose the push-based connector when early decode node selection acceptable and lower TTFT priority. The pull-based connector remains better fit for workloads requiring bidirectional KV transfer or deferring decode-instance selection until after prefill for load-aware routing. vLLM supports both push and pull-based KV connectors so teams can select the approach fitting their deployment.

A future improvement explores bringing the background NIXL threading model to the pull-mode connector moving NIXL setup off the critical path so pull-mode workloads could benefit from the same optimizations. The team opened a pull request to explore this avenue.