The economics concept of an efficient frontier describes optimal points between cost and capability. AI teams borrowed this term to discuss model-level tradeoffs—specifically the highest intelligence achievable at a given cost or size. Separately, inference engineers have their own frontiers, most often mapped by the latency-throughput relationship that determines serving cost. Beyond that core axis, quality can be exchanged for throughput through techniques like quantization, distillation, and pruning. Intelligence can be traded for speed through reasoning-level adjustments.

Two categories of inference techniques shape the frontier

Inference techniques fall into two categories. The first moves a deployment along the frontier by trading off one factor against another. The second pushes the entire frontier outward, creating additional efficiency that can then be allocated toward lower latency, higher throughput, or both. Both categories serve different production scenarios.

It proves useful to reach any point on the frontier through intentional tradeoffs. Sacrificing per-user speed makes it possible to build high-throughput low-cost pipelines for batch workloads. Conversely, giving up throughput to improve speed makes sense when latency-sensitive users have a high willingness to pay. Pushing the entire frontier outward unlocks gains that can be redistributed toward any combination of improved latency, increased throughput, or both.

The frontier is jagged, not smooth

In practice the frontier turns out to be quite jagged rather than smoothly continuous. Small configuration changes can produce disproportionate impacts, and the cutoff points between regimes are often unintuitive. Empirical sweeps are typically required to locate the sweet spot for a given traffic pattern.

Batch sizing sets the most direct latency-throughput tradeoff

A batch consists of requests processed concurrently. While token-level continuous batching eliminates wait time at batch initiation the configured size still determines per-user latency and overall throughput. Small batch sizes deliver excellent per-user latency but generate few total tokens per GPU driving up the cost per token. Larger batch sizes invert the relationship: per-user latency worsens while overall throughput improves lowering the cost per token.

Parallelism across GPUs shifts latency or throughput

Massive LLM models spanning hundreds of billions or even trillions of parameters typically span multiple GPUs. How those models are parallelized across devices affects whether latency or throughput improves. For latency-sensitive deployments increasing tensor parallelism (TP) is the typical approach. While tensor parallelism involves expensive all-to-all communication these operations complete quickly over high-bandwidth NVLink interconnects. Expert parallelism (EP) serves both latency and throughput goals. Lower EP degrees tend to produce better latencies while wide EP configurations including across an entire GPU rack generally support higher throughput. Attention data parallelism (ADP) replicates attention layers for parallel computation boosting system throughput at the expense of per-request speed.

Quantization improves efficiency at the frontier

Quantization running a model with reduced precision across weights activations and KV cache values improves both latency and throughput. A quantized model pushes the efficient frontier outward on serving tradeoffs. The quality-versus-efficiency frontier is notably jagged large serving efficiency gains are achievable with little to no model quality reduction especially when using microscaling floating-point formats like MXFP4 and NVFP4. These formats deliver substantial improvements in serving efficiency while preserving model intelligence at near-full precision levels.

Kernel-level gains compound across the stack

A CUDA kernel is a low-level function that executes a single inference component such as matrix multiplication. Improving individual kernel performance and the end-to-end forward pass means fewer resources are needed per generated token. These gains compound throughout the inference stack and push the performance frontier outward. Brian Li a Baseten intern published an excellent writeup on kernel-level performance improvements.

Speculative decoding yields efficiency from predictable output

Speculative decoding works by guessing upcoming tokens and validating those guesses. When first introduced the technique carried a latency-throughput tradeoff because speculation was expensive sequence lengths were short and acceptance rates were low limiting it to small batch sizes. Modern variants including EAGLE-3 DSpark and DFlash still compete with the main model loop for resources constraining maximum batch sizes. Nevertheless on code generation workloads where output sequences are relatively predictable these techniques yield efficiency gains from skipped forward passes alongside raw latency reduction delivering more tokens per second per user.

Prefill and decode can be disaggregated onto dedicated workers

P/D disaggregation or separating prefill and decode onto dedicated workers is a strategy useful for high-volume LLM deployments. Separating the two phases allows workers to be optimized for the distinct characteristics of each inference stage. The ratio between prefill and decode workers can be adjusted to match input sequence lengths output lengths and cache hit rates from incoming traffic.

Efficiency techniques compound

Doubling performance through better hardware while simultaneously doubling it through better software produces a fourfold improvement in overall serving capacity. That improved capacity can then be allocated toward any mix of lower latency higher throughput or a balanced combination of both.

For more detail on every technique mentioned here read the free book Inference Engineering.