Reflex is a new open-source inference engine built in Rust that takes a deliberately narrow approach to AI model execution. Where most LLM serving tools compete on sustained throughput and concurrent request handling, Reflex optimizes for a single metric: how fast can a cold process produce its first token, then stop. The project is GGUF-native, CUDA-accelerated, and explicitly rejects the server model entirely.

The cold-start problem nobody is solving

Most inference engines — llama.cpp, vLLM, SGLang — are built for warm, persistent servers handling continuous request streams. But many real-world AI workloads are not like that. A serverless function invocation, a CLI tool called from a build script, a cron job that generates a response on demand, an edge device that wakes only when triggered. In each case, the cost of spinning up a process and producing the first output token dwarfs the cost of generating subsequent tokens.

Reflex targets exactly this gap. Its creators argue that existing benchmarks measure warm-state joules-per-token, which says nothing about the full lifecycle cost of a cold invocation. The project's target metric is energy-to-first-token: the wall-clock time from process launch to the first generated output, measured in joules.

Why ahead-of-time CUDA compilation matters

The central technical bet is that CUDA kernels should be compiled before the binary ever ships, not when the process first runs. Many inference engines rely on NVRTC, a runtime compiler that compiles CUDA kernels on the fly. Reflex's developers measured this as a multi-second penalty on first use. vLLM, for example, reportedly takes roughly 235 seconds to become ready after startup due to CUDA graph capture, before it serves a single request.

Instead, Reflex compiles every kernel ahead of time using nvcc during the build phase. The precompiled PTX or cubin binaries are embedded directly into the release binary. At runtime, the engine loads them via the CUDA driver API with zero compilation overhead. The build system invokes nvcc through build.rs, and the source lives under src/kernels_cuda/. A dedicated smoke subcommand lets developers verify the entire AOT pipeline end to end on real hardware before touching any model.

How the benchmarks compare

On a ThunderCompute A6000 GPU, with three measurement runs per comparison using external wall-clock timing, Reflex reported the following cold-start results:

  • Against llama.cpp: roughly 1.3 to 1.4 times faster, completing in 4.71 to 5.05 seconds versus llama.cpp's 6.45 to 6.56 seconds. Both engines use AOT compilation, so this comparison does not isolate the JIT-tax advantage.
  • Against vLLM: 24 to 52 times faster, completing in under 5 seconds versus vLLM's 121 to 244 seconds depending on torch.compile cache state. The caveat is that the installed vLLM lacks GGUF support, so the test ran against an HF safetensors checkpoint.
  • Against Ollama: roughly competitive at 6 to 7 seconds, but Ollama's bundled llama-server intermittently stalled on an internal GPU-discovery watchdog timeout lasting 55 to 62 seconds.
  • Against TypeSafe Jev: Reflex loses on cold-start-to-decision by 10 to 60 times, but Jev is an always-warm managed API. On warm compute-only operations, the two are within 1.3 to 2 times of each other.

All results are reported as-is, with full methodology and per-run numbers documented in DECISIONS.md and HISTORY.md.

The architecture it will not build

Reflex's design is defined as much by what it excludes as what it includes. The engine enforces batch_size of 1 with no request queue, no continuous batching, no PagedAttention-style dynamic allocation, and no context preemption. There is no internal HTTP or gRPC server, no concurrent request handling, no thread pool, no multi-tenant LoRA router, and no NVMe or S3-based KV-cache manager.

The project's authors are explicit: if HTTP access is ever needed, the pattern is a separate sidecar binary communicating over local IPC, not a network socket added to the core engine. For local non-network use, the engine exposes stdio JSON-line mode and Unix Domain Socket mode, both strictly sequential.

This philosophy extends to Kubernetes, where the natural primitive is a Job running one cold-start invocation per Pod, never a Deployment or Service.

Supported models and completed milestones

The engine supports four model architectures: dense Qwen3 (the first and best-documented), Qwen3-MoE, the Qwen3.5 hybrid Gated DeltaNet mixer, and DeepSeek-V2/V3 with its compressed latent-KV caching strategy. The system1 subcommand enables single-pass, non-autoregressive candidate scoring — a "System 1" decision loop where multiple candidate continuations are scored against a prompt in one forward pass. On an RTX A6000, scoring three candidates for "The capital of France is" completed in roughly 8.5 seconds, with Paris scoring a probability of 0.997 against London at 0.002 and Berlin at 0.0004.

Four development phases are complete: single-shot CLI execution, fast GPU IO with device-resident activations and on-GPU dequant kernels, state I/O for exporting and importing KV caches across all architectures, and embeddability via a Rust C-FFI surface and --lora adapter loading.

What Reflex means for practitioners

Reflex is not positioned as a replacement for llama.cpp or vLLM in serving workloads. It is a specialized tool for workflows where an AI model is invoked once, produces a result, and exits — the pattern common in serverless functions, CI pipelines, automated agents making rapid decisions, and edge deployments. Its Rust codebase and Docker multi-stage build make it reasonably portable, though it requires an NVIDIA GPU and the CUDA toolkit at build time.

For developers building agent systems that need fast, single-shot model inference without server overhead, Reflex offers a genuinely different trade-off. It accepts that it will never win the warm-throughput race and instead concentrates on making the cold path as short as possible.