A team of researchers from the University of Washington has released ServingStudio, an integrated system that combines a high-speed simulator with an autonomous agent to optimize the performance of large language model serving infrastructure. The project addresses a growing bottleneck in AI deployment: the time and expense required to tune serving systems for different models, hardware, and workloads.
The Problem: Optimization Is Slow and Expensive
LLM inference has become one of the most computationally significant workloads in modern software, powering not only end-user applications but also training pipelines that depend on reinforcement learning and synthetic data generation. Achieving good serving performance requires balancing hardware choices, model architecture, kernel implementations, and scheduling policies. As models evolve and workloads shift, engineers must revisit configuration decisions, identify bottlenecks, modify serving frameworks, and benchmark results, a process that consumes substantial time, manual effort, and GPU resources.
Existing approaches have limitations. Agents can automate profiling and optimization, but without system-level performance models they lack end-to-end predictions for prioritizing which changes to attempt. Hardware experiments remain slow and costly. And while simulators can accelerate configuration exploration by reusing profiling data, extending them to new models and interpreting their results still requires considerable manual work, including model implementation, kernel profiling, and roofline analysis.
Two Components, One Workflow
ServingStudio is built around two tightly integrated components. ServingStudio Sim is a simulator written in Rust that runs up to 2,770 times faster than real time, allowing engineers to explore days of serving behavior in minutes. It supports diverse model families including the GLM-5.2 series across different quantization formats and parallel configurations, and models serving features such as prefix caching, speculative decoding, prefill-decode disaggregation, and attention-FFN disaggregation. The simulator's predictions are calibrated and validated against measurements from SGLang and vLLM, and users can inspect the entire run, individual requests, scheduler iterations, and kernel execution details.
ServingStudio Agent is the second component. It uses the simulator's predictions to select promising optimization changes, implements them in real serving frameworks, captures GPU traces to inspect kernel timings, communication, and idle gaps, compares simulated results with actual measurements, and validates correctness and performance on real hardware. Users can let the agent carry out this workflow autonomously, monitor its progress, and intervene when useful.
Real Results on Real Hardware
The team demonstrated the system through three case studies that produced measurable improvements in serving performance.
In the first, the agent guided alignment between SGLang's measurements and the simulator's predictions revealed that prefill MoE kernels were running slower than expected because autotuning did not cover the exact kernel called inside prefill CUDA graphs. Adding a tuning pass for that path improved input throughput by 5.6 percent on GLM-5.2 NVFP4 running across four B200 GPUs with four-way tensor parallelism.
In the second case, the agent identified that vLLM's multi-token prediction execution was suffering from CPU overhead because the V1 runner restricted piecewise graph sizes to multiples of six, forcing 2,048-token prefill batches to run eagerly. Raising both the scheduler budget and graph limit to 2,052 tokens restored CUDA graph replay and delivered 10.8 percent higher output throughput in a 100-request benchmark.
The third case went further, with the agent building an FP8 implementation of Qwen3-235B from scratch in Mini-SGLang, guided by simulation and kernel analysis. The implementation, which simplified execution and fused kernels, delivered 25.6 percent higher output throughput than vLLM on the same prefill-heavy workload across four H200 GPUs.
Who Benefits and What Comes Next
The team plans to extend support to newer models including GLM-5.3 Flash, DeepSeek V4.1 Flash, and Kimi K3, model distributed prefix caches across GPU memory, host memory, and remote storage, and enable remote hardware profiling to collect measurements from a broader range of systems. A public kernel-performance database is also planned, which would allow researchers outside the project to build on the team's profiling work.