SWE-Serve: When Passing Local Tests Is Not Enough to Ship to Production
Software engineering agents have made remarkable progress. They can resolve pull requests, fix bugs, and implement features across entire repositories. But a disturbing pattern has emerged: agents routinely produce patches that pass every local test yet fail when deployed through the complete production serving path. A change to a model-serving system might work perfectly in isolation but break when real requests flow through the public API, when multiple requests share persistent state, or when concurrent execution exposes hidden race conditions.
SWE-Serve, introduced by Jennifer Williams, Dave Farris, Jeff Farris, and Jiantao Jiao at NVIDIA and UC Berkeley, is a benchmark designed to measure precisely this gap. Rather than testing whether an agent can make a repository pass its unit tests, SWE-Serve tests whether an agent can produce code that works correctly through the complete production path, including end-to-end serving tests and calibrated performance gates. Across 11 models and 31 model-effort configurations, the best-performing configuration achieves 75% mean pass@1, but the benchmark's most important contribution is the discovery it enables: a substantial, measurable gap between local correctness and production correctness.
The Missing Benchmark: Why Existing Tests Don't Test Production
Existing agentic benchmarks cover only narrow slices of production inference engineering. Repository-level software engineering benchmarks like SWE-bench Verified and DeepSWE evaluate agents on changes to software repositories, but they do not target inference systems and do not require GPU-dependent execution. Terminal-Bench 3 broadens agentic evaluation to interactive systems tasks, but inference engineering constitutes only 3 of 74 tasks, a mere 4.1% of the benchmark. Dedicated inference-performance benchmarks like KernelBench and SOL-ExecBench evaluate GPU kernel generation and optimization, while InferenceBench targets deployed-system inference optimization. However, these benchmarks focus primarily on isolated kernel generation or performance optimization rather than repository-scale production feature implementation.
The result is a landscape where agents can score well on standard software engineering benchmarks yet fail to implement features that work in a real serving system. Implementing an inference feature requires coordinating multiple changes across the serving stack: model support, runtime execution, and public APIs. A change that works on a developer's laptop may break when exercised through the complete production path, where the model must load its checkpoint correctly, route tokens through the right set of experts, serve text and images through public interfaces, and maintain correct behavior under concurrent requests.
Design: 53 Tasks Grounded in Real Production Changes
SWE-Serve is built from changes merged since December 2025 into SGLang, an open-source inference-serving system. The benchmark provides 53 production-grounded tasks spanning six mutually exclusive engineering families: model and backend enablement (12 tasks), speculative and advanced decoding (14 tasks), kernels, quantization, and performance (8 tasks), caching and runtime state (7 tasks), distributed execution and scheduling (4 tasks), and serving APIs and runtime correctness (8 tasks). The benchmark draws on 83 unique upstream pull requests, with 37 tasks derived from a single change and 16 integrating two to six related changes.
Each task consists of a containerized environment with the SGLang repository checked out at the task's designated base commit, a task instruction, hidden executable tests, and an oracle solution that establishes task feasibility. Tasks execute on either CPU or a single NVIDIA H100 GPU. The oracle solutions change a median of 553 lines across seven files, with maxima of 6,077 lines across 35 files, suggesting broad repository-level implementation scope.
The evaluation methodology is rigorous. The hidden verifier assesses patches through fail-to-pass tests, which require behavior absent at the base commit, and pass-to-pass tests, which protect relevant behavior already present. When required, the verifier additionally evaluates model-serving end-to-end behavior or enforces calibrated performance gates. Critically, the agent does not receive the oracle solution, source-change identifiers, or the hidden verifier.
Validity: How SWE-Serve Ensures Its Tests Are Sound
Construction had three stages. Five discovery paths identified and screened 786 task-source records, retaining 203 source candidates. Task construction produced 156 task candidates. Final qualification admitted 53 tasks, a 34% admission rate. Admission required each candidate to run in its specified hardware environment and satisfy executable controls: the no-op had to fail every fail-to-pass test and pass every pass-to-pass test, while the oracle had to pass all tests.
Beyond executable controls, every admitted task underwent agent-assisted adversarial probing using agent-created patches and trajectories to identify vulnerabilities in its task instruction and verifier. This probing looked for valid alternatives rejected by the verifier, incomplete or shortcut solutions that it accepted, unexercised requirements, and environment-induced execution failures. Human adjudication further informed task revision and final-roster selection. Across this process, 17 broad task candidates were replaced by narrower tasks and 27 required verifier changes.
Evaluation integrity is protected through closed-book execution, where agents cannot retrieve upstream solutions from the public web, and trajectory auditing to detect prohibited retrieval. Every prohibited retrieval attempt was blocked, so no trial was invalidated on this basis.
The Central Finding: A 23-Point Production Correctness Gap
SWE-Serve's most significant result is the quantification of the gap between local correctness and production correctness. Across 19 tasks with end-to-end coverage and 627 patches, the pass rate is 45.9% under the full verifier but rises to 69.4% when model-serving E2E tests are removed from scoring. This 23.4-percentage-point increase means that roughly one-third of patches that pass every other test fail to serve the specified model correctly through the live server's public interfaces.
The matched test-removal control confirms this is not an artifact of simply removing tests. Across 12 eligible tasks, 10,000 randomized test-removal pairs were evaluated, each excluding from scoring either 38 E2E tests or 38 matched non-E2E tests while holding all 396 agent-created patches fixed. E2E test removal produced 16.1 fail-to-pass transitions on average, compared with 8.0 for matched non-E2E test removal, a 2.0x difference. E2E test removal produced more fail-to-pass transitions in 92.2% of randomized pairs.
A concrete example illustrates the problem. On the Gemma 4 mixture-of-experts core-serving task, 48.5% of agent-created patches, 16 of 33, passed every non-E2E test but failed to serve the model correctly. These tests exercise loading the official checkpoint, routing each token through the top eight of the model's 128 experts, text and image serving through public interfaces, and ordered native batch generation with log probabilities.
Results Across 11 Models
SWE-Serve evaluates 11 models spanning the frontier, including Claude Opus 5 and Sonnet 5, GPT-5.6 Sol, Luna, and Terra, Kimi K3, DeepSeek V4 Flash, GLM-5.2, Gemini 3.6 Flash, Laguna S 2.1, and Inkling S. Across 31 model-effort configurations, mean pass@1 spans a 40-percentage-point range, from 75% for Claude Opus 5 and GPT-5.6 Sol to 35% for Inkling S.
Pass@1 alone obscures important differences. Among four configurations tied at 64% pass@1, mean per-task cost varies by 7.6x, from $0.95 to $7.24, and wall-clock time by 3.9x, from 25.5 to 99.9 minutes. Claude Opus 5 at max effort achieves 75% at a cost of $17.40 per task, while GPT-5.6 Sol matches this at only $12.26 per task. GPT-5.6 Luna achieves 64% at just $0.95 per task, making it the most cost-efficient configuration on the leaderboard.
Reasoning effort does not always justify its cost. For Claude Opus 5, moving from high to max effort raises pass@1 only modestly, from 74% to 75%, while increasing mean cost by 76%, from $9.89 to $17.40 per task. This demonstrates that optimizing for pass@1 alone can incur substantially higher resource costs for only a marginal performance gain.
What the Gap Reveals About Agent Capabilities
The production correctness gap is not uniform across task types. Tasks requiring persistent state or concurrent coordination have pass rates 20.8 and 29.0 percentage points lower, respectively, than tasks not tested for these properties. Tasks spanning multiple runtime domains, including request I/O, scheduling, model execution, and KV-cache management, have pass rates 21.3 percentage points lower than single-domain tasks. These findings suggest that agents struggle most precisely where production correctness matters most: when state must persist across operations, when concurrent requests interact, and when multiple subsystems must coordinate.
The gap is also robust to reasoning effort. Across nine matched configurations, SWE-Serve tasks requiring persistent state or concurrent coordination show persistent gaps regardless of how much reasoning effort the model expends. This means that simply giving the agent more thinking time does not close the production correctness gap.
Limitations and Scope
SWE-Serve is specific to inference engineering and built from SGLang. While the benchmark's methodology is generalizable, the tasks themselves are domain-specific. The benchmark also relies on containerized execution environments, which may not capture all production failure modes such as network partitioning or resource contention at scale. The closed-book setting, while protecting evaluation integrity, does not reflect real-world development where engineers routinely consult documentation and upstream code.
The benchmark also acknowledges that executable scoring has limitations: some production failures may only manifest under specific traffic patterns or over extended runtime that is impractical to test in a benchmark setting. The 210-minute task time limit, while accommodating SWE-Serve's long-running tasks, may still be insufficient for some production scenarios.
What This Means in Practice
SWE-Serve provides the field with something it has lacked: a direct, measurable way to track whether agents are moving beyond completing tasks locally to achieving production correctness. The benchmark exposes a 23-point gap that exists across all evaluated models and effort levels, and the matched test-removal control confirms that this gap is specifically caused by model-serving E2E requirements rather than being an artifact of test removal.
For practitioners deploying coding agents in production, the implications are clear. A benchmark score alone is insufficient to predict whether an agent's output will work in a live serving environment. The 23.4-point gap means that even the best agents fail to achieve production correctness on roughly one-third of tasks when E2E tests are considered. Organizations deploying agent-generated code to production serving systems need to invest in end-to-end validation that goes well beyond unit tests and integration tests.
The benchmark also provides an important signal for the research community: closing the production correctness gap may require fundamentally different approaches than those that improve local task completion. Agents need to understand not just what code makes tests pass, but how that code behaves when deployed through the complete serving stack, under concurrent load, with persistent state, and through public interfaces. This is a harder problem than repository-level coding, and SWE-Serve establishes the measurement infrastructure needed to track progress toward solving it.
The Bottom Line
SWE-Serve is the first benchmark that directly measures the gap between local correctness and production correctness for inference engineering. Its 53 tasks, grounded in real SGLang production changes and validated through rigorous adversarial probing, provide a challenging and reproducible evaluation. The central finding, that removing end-to-end serving tests raises pass rates by 23.4 percentage points, exposes a fundamental limitation of current agentic coding systems: they can make code pass tests in isolation but struggle to make it work through the complete production path. By making this gap directly measurable, SWE-Serve establishes the benchmark against which future agents must be judged, and a reality check that the field cannot afford to ignore.