Rethinking Heterogeneous System Disaggregation for Subquadratic Attention

Why Standard Disaggregation Breaks Down for Subquadratic Attention LLMs

Modern language models are shipping with subquadratic attention layers as the majority of their architecture. GLM-5.2 uses sparse top-k selection. Nemotron 3 Ultra keeps 12 of 108 layers dense and routes the rest through Mamba-2 recurrent state. Gemma 4 31B retains 10 of 60 dense layers and puts the other 50 behind a 1024-token sliding window. These designs reduce memory and compute for long context, but they also break the assumptions that existing serving systems rely on.

Current inference disaggregation splits work by operator type: prefill on one pool, decode on another, or attention on one device and feed-forward network on another. These strategies were designed when every decode layer carried quadratic attention with a KV cache that grows without bound. When a model is mostly subquadratic, the arithmetic intensity and memory footprint of each decode stage diverge from what those disaggregation boundaries expect, and the systems leave performance on the table.

A team from NVIDIA Research and Harvard, led by Arya Tschand, proposes a different cut. Their paper, "Rethinking Heterogeneous System Disaggregation for Subquadratic Attention," introduces SQD (SubQuadratic Disaggregation), a fine-grained scheme that splits decode by quadratic versus subquadratic attention rather than by operator type. The core claim is that this cut matches the actual hardware characteristics of emerging DRAM-based GPU and SRAM-only ASIC heterogeneous systems, and the results bear it out: 53% average tokens/J improvement on GLM 5.2, 31% on Nemotron 3 Ultra, and 56% on Gemma 4 31B over the strongest GPU-only baselines, measured on real hardware.

The Arithmetic Intensity Gap Between Quadratic and Subquadratic Stages

The key observation starts with a roofline analysis of what each decode stage actually reads from memory relative to how much it computes. The authors decompose a decode step into four classes: prefill, quadratic attention, subquadratic attention, and FFN. For each, they measure arithmetic intensity (the ratio of FLOPs to bytes read) and per-rank resident memory footprint.

The numbers tell a clear story. Prefill is compute-bound on GPUs, especially at long context, because it amortizes each weight byte over thousands of multiply-accumulate operations. Every decode stage, by contrast, lands two to four orders of magnitude below the GPU ridge point. At dense FP8, a Rubin GPU has a ridge at 758 FLOP/byte. All decode stages sit well below that. The GPU spends most of its power on arithmetic the work cannot use.

An SRAM-only ASIC like Groq's LPX sits at a ridge of roughly 8.2 FLOP/byte. Decode attention work sits near that ridge, not far below it. Moving decode attention to such a device puts it on the side of the roofline where the arithmetic is actually used. The FFN is the exception and stays memory-bound there too, but its footprint is fixed by the LLM rather than by the workload.

The memory footprint analysis reinforces the point. Quadratic attention state grows with context length because of the KV cache. Subquadratic attention and FFN both have fixed footprints, independent of how long the input is. In GLM-5.2 at maximum modeled batch and context, the KV cache alone exceeds 3 TB per rank. That is far beyond any SRAM-only chip capacity, even at rack scale. Subquadratic attention and FFN weights, by contrast, are predictable from the model architecture. On GLM-5.2, moving subquadratic attention onto an SRAM-only ASIC alongside the FFN raises the resident footprint by only 7%. On Gemma 4, whose FFN is small, it grows the resident set 3.9x, but the total still fits.

These two facts, that subquadratic attention has arithmetic intensity near the ASIC ridge and a footprint that does not grow with context, are the foundation of SQD.

SQD: Where Each Stage Runs

SQD places decode work across two device pools based on the characterization above. Quadratic decode attention stays on the DRAM-based GPU, because its footprint grows with context and it already shares the KV cache with prefill. Prefill is colocated on the same GPU pool. Since prefill is compute-bound and decode is memory-bound on the same device, they contend for different resources and achieve higher total utilization. A decode step costs roughly the maximum of the decode and prefill times rather than their sum, because the GPU and ASIC pools run concurrently.

Subquadratic attention and the FFN move to the SRAM-only ASIC. Their arithmetic intensity sits near the ASIC's ridge point and their resident state is independent of the request. In linear and sliding-window attention LLMs like Nemotron 3 Ultra and Gemma 4, each layer is entirely quadratic or entirely subquadratic, so the split is layer-by-layer. Dense-attention layers stay on the GPU, every remaining layer moves to the ASIC with its FFN.

Sparse attention LLMs like GLM-5.2 are more complex, because the quadratic and subquadratic stages live inside the same attention operator. The indexer scan reads the full KV cache to choose 2048 entries, so it stays on the GPU. The top-k attention over those selected entries is subquadratic and runs on the ASIC alongside the FFN. This within-operator split is the fine-grained part of SQD.

The practical consequence is reduced communication. Standard attention-FFN disaggregation pays a blocking activation transfer twice per token per layer. SQD crosses the interconnect only at quadratic layers, which in GLM-5.2 happens once per four-layer IndexShare group and in Gemma 4 once per six layers. On an eight-layer window, SQD pays 16 microseconds of exposed communication against attention-FFN's 160 microseconds.

Caching the Selected KV for Sparse Attention

The hard problem for sparse attention LLMs is that the set of selected KV entries is not known before the request runs. An ASIC holding top-k attention would need to read the full KV at every step to learn which entries it needs, which would defeat the purpose of disaggregation. At batch 32, one layer's selected set is 75.5 MB, which at every layer would put the interconnect on the critical path.

SQD addresses this with a latency-aware caching mechanism built on two properties. First, top-k sparse attention exhibits locality: across four agentic traces (AgentTrove Math, tau-bench Retail, SWE-bench Pro, and TerminalBench), selected positions concentrate into a few context bands that persist across many consecutive decode steps. Second, GLM-5.2 recomputes its selection once every four layers and the three following layers share it, an IndexShare pattern. Layers after the group head can hide communication behind previous layer execution.

The mechanism is IndexShare cross-group prefetching with empirical-rank-frequency eviction. The prefetch issues a group's fetch from the selection its head just computed, which exploits the reuse across layers as reuse across steps. The group head is the exception: it owns the selection and has no slack, so its attention stays on the ASIC with a cache sized for near-full residency. The few entries it misses return on the selector round trip the group already makes.

The authors replay five eviction policies against recorded selection streams. Belady's algorithm provides the oracle bound. ARC, Scissorhands, H2O, and page-granular baselines are the practical candidates. Offsets 2 through 4 in an IndexShare group need 0.6x to 4.7x the per-request top-k working set to achieve the hit rates required for fully hidden communication. The required hit rate depends on the overlap budget: an offset j layer has (j-1) layer times of slack, which at the measured system comes to 86.5 microseconds at offset 2, 173 microseconds at offset 3, and 259.5 microseconds at offset 4.

Measuring on Real Hardware

The evaluation uses an 8xB200 GPU system running a modified SGLang serving stack. The decode pool is throttled to 16 of the B200's 148 streaming multiprocessors, which reduces tensor-core throughput to match the HBM bandwidth ratio of an SRAM-only ASIC at 64 FLOP/byte. Power at that throttle point measures roughly 120 W per adjusted decode device, versus 908 W for a full GPU. The adjusted system draws 4.11 kW total against 7.26 kW for stock.

Three LLMs are evaluated across three workload patterns: Chat (32K context, 256 output tokens), RAG (256K context, 1024 output tokens), and Agentic (1M context, 2048 output tokens). The results:

• GLM 5.2: 53% average tokens/J improvement over the strongest GPU-only baseline, retaining 85-97% of no-disagg throughput at every 1M-context point.
• Nemotron 3 Ultra: 31% average tokens/J improvement, growing from 12% at 32K to 61% at 1M context.
• Gemma 4 31B: 56% average tokens/J improvement, rising from 36% at 32K to 71% at 1M context.

Against attention-FFN disaggregation, the same systems draw the same power but SQD achieves 16-38% higher tokens/J, entirely from reduced communication overhead. At 1M context on GLM-5.2, an eight-layer window takes 432 microseconds under SQD versus 576 microseconds under attention-FFN.

The improvement widens with context because at long sequence lengths quadratic attention dominates the decode step. Colocating it with prefill removes most of that time from the critical path. At 32K, subquadratic and FFN work dominates and every reasonable placement performs similarly.

Projected Performance on Rubin plus LPX

The authors build an analytical model of a Rubin GPU plus Groq-3 LPX heterogeneous deployment and validate it against the B200 measurements within 15% error (median 3.7%). Within fixed power budgets, SQD projects 1.2x to 1.5x higher achievable user TPS than attention-FFN disaggregation across all three LLMs and workloads. On GLM-5.2, where the model's crossing accounting places attention-FFN furthest below, SQD delivers 2.9x to 3.6x the throughput as a median over the overlapping latency range.

At high latency targets where inference runs in larger batches, GPU baselines reach 1.1x to 2.1x SQD's peak throughput. But in the interactive regime, SQD holds flat where GPU fronts have already fallen off. It serves 2.7x to 3.4x the fastest rate any GPU-only system reaches. EAGLE-3 speculative decoding is the strongest GPU response, reaching 1.5x to 1.6x no-disagg throughput, but it runs out of latency headroom well before SQD does.

Architectural Insights for Next-Generation Systems

The paper surfaces several provisioning insights. On-chip SRAM pays off only until the LLM's static footprint is resident. Gemma 4 saturates at modest chip counts and gains nothing from more SRAM. Nemotron 3 Ultra rises and then flattens. GLM-5.2, with 416 GB of FFN weights, never saturates in the swept range. Adding SRAM per chip reduces the chips a replica needs and frees power for more replicas, which is what raises throughput. The requirement is resident capacity, not bytes read per step, especially for sparse MoE models where most experts are idle in any one step but all must live somewhere.

Interconnect design matters more for latency than bandwidth. SQD throughput is more sensitive to interconnect latency than to bandwidth. Additional NVLink bandwidth buys minimal throughput over Spectrum-X, while the worse Spectrum-X latency drops throughput 25-37%. GLM-5.2 falls fastest because it pays the exposed crossing once per four-layer IndexShare group. Future heterogeneous systems adopting SQD should prioritize low minimum latency over high bandwidth to get the best performance per dollar.

The experimental cost was substantial: 4,058 B200 GPU-hours across 579 runs, of which roughly nine-tenths went to testing, validation, and bring-up. The authors note this as the cost of building the transport, the throttled decode device, and the serving path from scratch.

What This Means for Inference Serving

SQD demonstrates that the right disaggregation boundary for subquadratic attention LLMs is not attention-versus-FFN but quadratic-versus-subquadratic. The scheme applies across all three major subquadratic attention families, sparse, linear, and sliding-window, and the gains grow as context lengthens, which is exactly the direction frontier models are heading.

For practitioners, the immediate takeaway is that disaggregation boundaries should be derived from the actual arithmetic intensity and memory footprint of each decode stage, not from operator-level abstractions that assumed every layer carried a growing KV cache. As frontier models continue to reduce their quadratic attention fraction, SQD's advantage compounds: fewer quadratic layers means fewer interconnect crossings, and the static footprint of the subquadratic stages makes them natural candidates for SRAM-only acceleration.

The work also highlights a tension in hardware design. GPUs over-provision arithmetic for decode workloads and waste power as a result. SRAM-only ASICs cut that waste but need workload characteristics to exploit it. SQD finds the boundary where each device type is most efficient and routes work accordingly, a pattern likely to recur as heterogeneous systems proliferate.