How SARA Uses Queuing Theory to Make Disaggregated LLM Serving 27% More Efficient
Large language model serving is entering a new era. As models grow in capability and context length, inference workloads are shifting from simple single-turn chat to complex multi-turn agentic tasks involving long contexts, tool use, and multi-modal inputs. This shift places unprecedented demands on inference infrastructure. The compute-intensive prefill stage and the memory-bandwidth-intensive decode stage have fundamentally different resource requirements, and colocating them on shared hardware leads to mutual interference, poor utilization, and inflated costs.
SARA, introduced by Shicong Liu, Xianghao Yu, Zhen Gao, and Jun Zhang at City University of Hong Kong, Beijing Institute of Technology, and the Hong Kong University of Science and Technology, tackles this problem with a mathematically rigorous approach. Rather than relying on hardware profiling, configuration enumeration, or heuristic scheduling, SARA models each inference stage as a queuing system, derives closed-form expressions for latency tail behavior, and uses these analytical characterizations to allocate resources optimally under service level objective (SLO) constraints. The framework improves system goodput by 26.6% on average over state-of-the-art baseline methods under the same deployment cost, with SLO predictions accurate to within 5%.
The Disaggregated Serving Architecture
To understand SARA's contribution, it helps to understand the architecture it operates on. In a disaggregated LLM serving system, the inference pipeline is split across separate resource pools. The prefill stage spans from user prompt input to the generation of the first output token. It requires intensive attention and feed-forward operations across all transformer layers, making it dominated by computational capacity. The KV cache generated during prefill must then be transferred to the decode resource pool. Finally, the decode stage autoregressively generates tokens, requiring frequent access to the entire KV cache, which makes it dominated by memory bandwidth.
This disaggregation eliminates the mutual interference between compute-bound prefill and memory-bound decode, enables independent scaling of each stage, and paves the way for edge-assisted services where the compute-intensive prefill stays in data centers while the memory-intensive decode can be pushed toward edge nodes near users. However, it introduces new challenges: how much compute capacity to provision for prefill, how much bandwidth to allocate for KV cache transfer, and how many decode devices to deploy, all while meeting latency SLO targets.
The Three-Stage Queuing Model
SARA's core contribution is a set of stochastic mathematical models that capture the behavior of each inference stage.
The prefill stage is modeled as an M/G/k_p queue. User requests arrive as a Poisson process with rate λ, and each request is processed by one of k_p identical prefill instances. The service time for a request with L_i input tokens is determined by the maximum of computational latency and HBM traffic latency. Computational latency scales with the number of matrix multiplications required by the transformer architecture, while HBM traffic latency depends on model weight loading and KV cache writing. The prefill utilization ρ_p = λ E[T_p(L_i)] / k_p must remain below 1 for stability. At high utilization, which is the practical operating regime for cost-efficient systems, Kingman's approximation provides a tractable expression for the tail behavior of the waiting time, enabling the derivation of closed-form SLO-attainment probabilities.
The KV cache transfer stage is modeled as an M/G/1 queue. The superposed departure traffic from k_p prefill instances forms a Poisson process with rate λ. The service time for transferring the KV cache of a request is simply the KV cache size divided by the link bandwidth B. The KV cache size scales linearly with the number of input tokens and the model architecture parameters, with a scaling factor γ = N_layer × N_mod × N_bits / (4 × 1024³) that captures how memory consumption grows with model depth, width, and quantization precision.
The decode stage is modeled as a generalized birth-death process with continuous batching. In each decoding iteration, N_bat requests in a batch are decoded in parallel, each producing one output token. The total number of tokens in the batch, L_io = Σ(L_i^(n) + L_o^(n)), varies over time as requests arrive and complete. The computational latency scales with the total token count and batch size, while the HBM bandwidth latency depends on loading the KV caches of all requests in the batch. The decode service time is the maximum of these two components.
Key Analytical Insights
The analysis yields several insights that directly inform resource allocation decisions.
First, the framework confirms that under practical system parameters, the prefill stage is predominantly bottlenecked by computational capacity while the decode stage is predominantly bottlenecked by high-bandwidth memory (HBM) resources. This is not merely an empirical observation but a mathematically derived result that holds across representative model architectures including LLaMA 3.1 8B.
Second, the required numbers of prefill and decode devices scale quadratically with input and output lengths, respectively. This means that doubling the context length requires four times the prefill resources, and doubling the output length requires four times the decode resources. This quadratic scaling has profound implications for capacity planning in the era of long-context agentic workloads.
Third, stringent SLOs and heavy-tailed workloads substantially amplify resource requirements beyond the stability condition. When the SLO demands extremely low latency tail probabilities, the system must provision resources far beyond what a simple stability analysis would suggest, because the queuing tail behavior dominates at high utilization. This is a critical insight for practitioners who might otherwise underestimate the cost of meeting tight latency budgets.
The Resource Allocation Framework
Building on the analytical characterizations, SARA formulates the resource allocation problem as maximizing system goodput, defined as the request throughput that adheres to latency SLO targets, subject to a total deployment cost constraint and a series of quantile-based SLO constraints for each stage.
The cost constraint couples the three stages: the total budget C_max is allocated across prefill instances (cost C_p per instance), KV cache transfer bandwidth (cost C_KV per unit bandwidth), and decode devices (cost C_d per device). The SLO constraints require that the probability of exceeding latency thresholds τ_pre, τ_KV, and τ_dec for TTFT, KV cache transfer latency, and TPOT respectively remains below 1 minus the SLO attainment probability p_SLO.
The framework solves this problem by first deriving stage-wise minimum resource requirements directly from the closed-form SLO expressions, then using bisection search to find the maximum goodput achievable under the cost budget. The result is a principled allocation that balances the three stages to maximize throughput while respecting all SLO targets.
Results and Validation
The framework's analytical predictions are validated through both simulation and hardware measurements. The derived stage-wise latency quantiles are accurate with mean errors below 5% across representative model architectures and both light- and heavy-tailed workloads, confirming that the queuing-theoretic models capture the essential dynamics of real inference systems.
The resource allocation framework improves system goodput by 26.6% on average over state-of-the-art baseline methods under the same deployment cost. This improvement comes from optimally distributing the budget across the three stages rather than relying on heuristic rules or uniform allocation. The framework is evaluated on NVIDIA A100 GPUs with LLaMA 3.1 8B, a model with 32 layers, 4,096 model dimension, 14,336 hidden dimension, and grouped-query attention with a compression factor of 0.25.
The SLO taxonomy introduced in the framework provides a structured way to understand what drives performance. The latency budget allocation analysis reveals how the total latency budget should be distributed across prefill, KV transfer, and decode to maximize goodput. The token throughput analysis shows how throughput varies with workload characteristics. And the "SLO tax" quantification reveals how much additional cost stringent SLOs impose beyond the stability condition.
Limitations and Scope
SARA provides an analytical framework for a specific disaggregated architecture and makes simplifying assumptions, such as treating the arrival process as Poisson and modeling service times as deterministic functions of workload characteristics. Real-world workloads may exhibit correlations and non-stationary patterns that deviate from these assumptions. The framework also assumes that the KV cache transfer link has sufficient bandwidth, which may not hold in all deployment scenarios.
The model currently considers a single disaggregated configuration and does not account for multi-site or geographically distributed deployments where network latency between prefill and decode pools could become a dominant factor. Extending the analysis to incorporate network latency and heterogeneous hardware configurations remains an important direction for future work.
What This Means in Practice
SARA provides a practical tool for infrastructure engineers operating disaggregated LLM serving systems. Rather than relying on trial-and-error configuration tuning or expensive benchmarking campaigns, the framework offers closed-form expressions that directly map workload characteristics, model architecture, and hardware parameters to resource requirements and SLO attainment probabilities.
For cloud providers and enterprises deploying agentic LLM services, the implications are significant. As agentic workloads with long contexts and high output token counts become the norm, the quadratic scaling of resources with context length means that capacity planning must account for the full distribution of workload lengths, not just average values. The framework's ability to predict the impact of heavy-tailed workloads on SLO attainment provides a principled way to provision for worst-case scenarios without over-provisioning.
The 26.6% goodput improvement over state-of-the-art baselines translates directly to reduced infrastructure costs or increased throughput for the same hardware budget. In an industry where inference costs are among the largest operational expenses for AI services, this represents a meaningful economic advantage.
The Bottom Line
SARA brings rigorous queuing-theoretic analysis to a problem that has largely been addressed through empirical methods. By modeling the three stages of disaggregated LLM inference as distinct queuing systems and deriving closed-form expressions for SLO attainment, the framework provides the analytical foundation that production inference systems have lacked. The result is a resource allocation methodology that is both theoretically grounded and practically validated, achieving a 26.6% improvement in goodput while predicting SLO compliance within 5% error. For anyone deploying disaggregated LLM serving systems at scale, SARA establishes a principled alternative to heuristic configuration and the first tractable way to reason about the cost of latency SLOs in production.