The memory-bandwidth gap in ternary LLM decoding
Ternary large language models store each weight as one of three values — typically −1, 0, or 1 — which reduces memory footprint and bandwidth compared to full-precision FP16 or BF16 representations. A ternary model with several billion parameters may occupy only a few gigabytes, yet a single decoded token still requires reading those weights from memory. On modern dual-socket Xeon servers with four NUMA nodes, the decoded-token throughput is limited not by the volume of data alone but by how quickly that data can be delivered to the cores that need it. The fastest ternary runtime observed on such a machine uses less than half of the available memory bandwidth, leaving a substantial gap between measured performance and theoretical peak. This gap is the starting point of the paper: even when weight placement is optimized, other factors constrain throughput.
Why weight placement alone cannot eliminate synchronization stalls
The most natural optimization is to ensure every weight lives on the NUMA node of the thread that reads it. If thread t on node n accesses weight w, then w is stored in the local memory of node n. This eliminates remote-memory latency for the read itself. The paper confirms that this step is necessary: without it, performance collapses because threads spend cycles waiting for data across NUMA boundaries. But it is not sufficient. Even with node-local weights and kernels that reuse data efficiently, threads on two generations of Xeon servers still spend 20 to 48 percent of each decoded token waiting at synchronization points. The decode engine proceeds in lockstep: a group of threads computes partial results for the next token, then all must arrive at a barrier before any can proceed. If one thread lags, the others idle. The paper's barrier traces show that the majority of the wait time is accounted for by the slowest threads on each node — the tail of the distribution drags the whole group down. The barriers' own latency, measured in microseconds per call, contributes a smaller share — up to 12 percent of a token for the smallest model tested. Serial per-node work, where threads complete independent subtasks before joining, is an even smaller contributor. The takeaway is clear: placement fixes the data movement problem, but the parallel synchronization problem remains.
Barrier traces reveal the slowest-thread problem
The authors instrumented the decode loop with per-thread barrier timestamps, allowing them to reconstruct exactly how long each thread waited and why. The data shows a consistent pattern: on every node, the variance in completion time across threads is significant. Some threads finish their share of the work early, others take considerably longer. The barrier forces all threads to wait for the maximum, and the mean wait is dominated by that maximum. The paper quantifies this by partitioning the wait into components: barrier invocation overhead, the actual wait for slower peers, and any serial computation each thread performs before the barrier. The wait-for-peers component is by far the largest. In the extreme case, a single thread that misses a cache line or suffers a page fault can add tens of microseconds to every token decoded by the whole group. This effect is exacerbated as the number of threads per node increases, because the probability of a straggler grows. The authors also note that the smallest model — one that fits entirely in L3 cache — still exhibits the same pattern, proving that the bottleneck is not memory-bandwidth saturation alone but the synchronization semantics of the decode kernel.
Column splitting beats row splitting without cross-node reductions
The paper compares two matrix-parallel decomposition strategies. In a row-owned split, each node holds a contiguous block of rows of the weight matrix, and the corresponding input activations are distributed so each node computes its rows locally. In a Megatron-style column split, each node holds a contiguous block of columns, and the matrix-vector multiply requires a reduce-scatter followed by a all-reduce across nodes. The authors find that, on identical kernels, the column split outperforms the row-owned split in every configuration measured — but only when cross-node reductions are eliminated or minimized. Row-owned splits often require an all-reduce or a global aggregation step at the end of each layer, which introduces additional barriers and communication volume. The column split, by contrast, localizes most of the work: each node multiplies its column block by the corresponding activation fragment, and only a small reduction is needed to combine partial results. By isolating the number of barriers — running the same decomposition with artificial barrier removals — the authors show that most of the column-split gain comes from threads waiting less for each other and for their node leaders. A minority of the gain stems from simply having fewer barriers. Fewer barriers is beneficial, but the dominant factor is the reduced wait time per barrier.
Bounded local search at test time corrects gradient errors
In a decode engine, the action selected for the next token is typically the one with the highest predicted value from the value field. The neural operator or value-function approximator predicts a global field, and the argmax of that field gives the chosen token. However, the predicted field is approximate — trained on noisy data, evaluated on geometries it has not seen, and computed with a finite number of Monte Carlo samples. Local fluctuations in the Monte Carlo estimates can reverse the estimated gradient direction, especially in narrow passages of the loss landscape. To counteract this, the paper introduces a test-time bounded local search. Starting from the action suggested by the predicted gradient, the engine evaluates a small neighborhood of alternative actions — typically the top-k candidates — and selects the one with the highest actual value, as estimated by a few additional Monte Carlo samples or a look-up table. The search radius is bounded, both in the number of candidates examined and in the distance from the predicted action, so the added cost is constant and small. This local correction step absorbs the errors from the global prediction without requiring a full re-optimization of the token. The result is a policy that is both fast and robust, with the global field providing the coarse direction and the local search fine-tuning the choice.
Results across models and memory regimes
The evaluation spans several ternary LLMs — a 700-million-parameter model that decodes from L3 cache, a 7-billion-parameter model that sits in DRAM, and a 70-billion-parameter model where per-token weight reading dominates the runtime. The baseline configuration uses a row-owned split with one thread per core and the maximum number of barriers the kernel supports. The comparison configurations replace the row split with a column split, reduce the barrier count to the minimum viable for correctness, and add the bounded local search at test time. Where decoding is DRAM-bound — the 7B and 70B models — the resulting engine is 1.5 to 2.1 times faster than the baseline when all cores are active, and 1.5 to 1.9 times faster than the baseline at its fastest thread count (3.2 times and 2.8 times faster for the small L3-cache model). The speedup is accompanied by lower energy per token, because fewer core-cycles are consumed for the same decoded output. The accuracy of the output distribution is unchanged — the local search does not alter the model's probabilities, it only selects among a small set of candidates that the global field already ranks highly. The paper also measures the barrier-reduction isolation: running the column split with the original barrier count still yields a significant speedup, confirming that the per-barrier wait reduction is the larger contributor, but the minimum-barrier configuration closes the gap further.
Where the column-split approach reaches its limit
The column split is not a universal elixir. Its advantage diminishes when the model size grows so large that the reduction operations become negligible compared to the sheer volume of weight reads, or when the network topology between nodes imposes latency that outweighs the per-barrier savings. The paper notes that on systems with more than four NUMA nodes, or when the interconnect is saturated by other communication patterns, the speedup contracts. Additionally, the column split requires careful handling of the reduction primitives — if the reduction is implemented inefficiently, the expected gain vanishes. The authors also caution that the local search, while bounded, adds a small per-token overhead that may not be justified on already-fast configurations where the baseline barrier wait is low. Finally, the approach assumes a fixed kernel structure; custom kernels that fuse additional operations may bypass both the column-split and the barrier optimizations entirely, rendering the reported numbers moot for those code paths.
Implications for AI infrastructure engineers
For engineers building multi-node LLM serving systems, the paper offers a clear hierarchy of optimizations. The first and most essential step is weight placement: keep each weight on the NUMA node of the thread that reads it. Beyond that, the synchronization schema matters as much as the memory layout. Switching from a row-owned to a column-parallel decomposition, while eliminating unnecessary cross-node reductions, yields the next largest gain. The engineers should instrument their barriers — the authors' traces show that simple timestamping reveals straggler patterns that are invisible to overall throughput metrics. A bounded local search at decode time, examining only a handful of candidates, is low-cost and improves robustness without sacrificing accuracy. Energy per token drops proportionally with the speedup, which is a secondary but welcome benefit for data-center operators. The code and micro-benchmark scripts are released alongside the paper, allowing teams to measure their own barrier contention and column-split performance on their specific hardware and model sizes.