CUDA Graphs Help Blackwell Inference, but Not Everywhere
A new research workbench called Kairo sets out to answer a practical question for anyone running quantized language models on NVIDIA's latest hardware: when does enabling CUDA Graph replay actually make inference faster, and when does it barely matter?
The answer, measured on a single RTX 5090 with NVFP4-quantized Qwen models, is that the speedup ranges from 2.62x down to 1.30x depending on the workload. That spread is the point. Kairo is not trying to prove CUDA Graphs are always worth enabling. It is trying to measure the conditions under which they help, prove correctness, and route only the measured cases through an automated policy.
Why the Variance Matters
CUDA Graphs capture a sequence of GPU operations into a single executable graph, replaying it instead of launching kernels individually. The technique eliminates launch overhead and can improve memory access patterns. But the benefit depends on the shape of the workload: how many concurrent requests, how many input tokens, how much context the model maintains.
Kairo's Phase 1 results show four measured configurations. The best case, Qwen3.8-27B-NVFP4 at concurrency 32 with a 512-token prompt and 1K context, reaches 645.08 tokens per second with Graphs versus 308.51 without, a 2.09x gain. The same model at concurrency 8 with a 2048-token prompt and 4K context hits 322.78 versus 123.22, a 2.62x improvement. But at concurrency 16 with the same 2048-token prompt and 4K context, the gain drops to 1.30x. A smaller Qwen3-8B model at concurrency 16 with a 512-token prompt shows 2.03x.
The 1.30x result is not a failure. It is context. Kairo does not promote CUDA Graphs as a global default. It captures the conditions behind each result and only promotes exact measured workload buckets into its runtime policy.
Evidence Before Promotion
Kairo's architecture is built around a simple principle: a result should exist before a policy acts on it. The system defines versioned workload and experiment protocols, verifies declarative kernel blueprints before building or launching, and records cache behavior, correctness checks, artifacts, and launch timing as separate concerns.
When a request comes in at serving time, the runtime policy checks whether the exact workload has been measured. If it has, the policy routes through the appropriate path. If it has not, Kairo fails closed to manual operation rather than guessing. This is a deliberate choice. An inference engine that assumes an unmeasured workload behaves like a measured one is making an unverified claim about performance and correctness.
The correctness gate is not optional. Every promoted result must pass it. The gate checks that the output matches an expected result within a defined tolerance, and negative results that establish meaningful boundaries are retained rather than discarded.
Portable Contracts, No GPU Required
Kairo separates the parts of the system that need a GPU from the parts that do not. The portable contracts, including validation of kernel blueprints and the runtime policy logic, run on any machine with Python. A test suite discovers and runs these contracts without requiring CUDA hardware. This matters for a research project where reproducibility and review need to happen on machines that may not have the latest GPU.
The repository includes scripts for WSL-based local probing on the RTX 5090, but the core dispatch, cache, and policy logic is dependency-light. The environment contract describes exactly which hardware, model weights, and software revisions produced a given result, so someone with the same setup can reproduce it.
What Phase 1 Proved, and What It Did Not
Phase 1 is a controlled baseline. It establishes protocols, measures Graph and eager performance across several workload shapes, builds a route selection policy from the evidence, and probes native SM120 data movement and matrix-compute paths. The results include both promoted and rejected experiments, and the baseline status file serves as a chronological lab record.
What Phase 1 does not claim is that CUDA Graphs are universally beneficial, that the results transfer to other hardware, or that the measured workloads represent all useful serving patterns. The project is explicit about this: local measurements on one RTX 5090, using pinned model and software revisions, and the linked protocols should be read before comparing any values.
Phase 2 will use Phase 1 evidence to choose narrowly scoped work on data movement, layout, subgraph, or execution path components. A candidate is not promoted without correctness, boundary coverage, repeatability, and a fair end-to-end comparison. The project describes itself as the beginning of a path toward native Blackwell inference components, not a general-purpose inference engine or a replacement for vLLM, CUTLASS, or cuBLAS.
Open Source, With Clear Boundaries
Kairo is licensed under Apache 2.0. The license grants broad reuse rights, including an express patent grant for contributed work, but does not grant trademark rights. Contributing guidelines are straightforward: do not extrapolate measured wins, do not commit credentials or private artifacts, and retain negative results.
The repository at github.com/peter941221/Kairo includes the Phase 1 write-up, results index, experiment protocols, kernel blueprints, and the evidence files. A rendering script regenerates the published result table and chart from the raw data. Every public result should answer a basic question: under which model, quantization, workload, hardware, software revision, and correctness tolerance did it win?