Energy-efficient deployment of large language models requires understanding how model architecture, hardware, and serving settings shape energy per generated token. We introduce the Token-Energy Law, a compact, interpretable model of GPU-board energy per output token for dense, full-history decoding. It relates token energy to two core dimensions: active bytes and realized decode time. Active bytes describe logical byte demand per output, separating dense weights shared within a decode iteration from private key--value (KV) histories read and updated for each request. Realized decode time is the elapsed decode duration divided by the number of output tokens produced. We combine the two byte coordinates and time in a common affine form, with coefficients calibrated for each model and hardware/runtime configuration. This formulation explains how batching amortizes shared-weight work, longer attended histories increase private KV demand, and throughput determines the time-associated cost per output. Our experiments cover three models under BF16 decoding on NVIDIA Hopper and Blackwell GPUs, including two-GPU tensor parallelism on Hopper. Using fixed coefficients and observed decode time, held-out condition-mean energy estimates differ from measurements by at most 3.73%.
From Energy per Token to Foundational Constraints
Large language models have become central to many AI products, but deploying them at scale incurs substantial computational cost. Energy consumption directly affects both operating expense and environmental impact. Prior work has characterized throughput (tokens per second) and latency (time per token), yet energy per generated token has received less attention as a unified metric. Models are typically evaluated in terms of FLOPS or parameter count, but these abstractions do not translate directly to the energy a user's GPU draws while generating each output token. The Token-Energy Law addresses this gap by proposing a compact, interpretable model that relates energy per output token to two observable dimensions: active bytes and realized decode time.
The need for such a model arises from the complexity of modern GPU serving pipelines. A single decode iteration involves reading shared weight matrices from HBM, updating per-request key-value (KV) cache entries, and performing matrix-vector multiplications. The energy drawn depends on how many bytes are accessed and how long the computation takes. Moreover, these factors change with batching decisions, context length, and hardware generation. A model that can predict energy per token from easily measured quantities would enable better serving decisions, hardware-aware model selection, and energy-profiling tools that do not require instrumenting the GPU at the cycle level.
Active Bytes: Dense Weights and Private KV Histories
Active bytes describe logical byte demand per output token. The term separates two distinct memory traffic patterns that occur during full-history decoding. The first pattern consists of dense weights that are shared within a decode iteration. In a transformer, the attention and feed-forward projections apply the same weight matrices to every token in the batch. When batching is used, these weight reads are amortized across the output tokens in the batch, so the per-token weight-access cost decreases as batch size grows. The second pattern consists of private key--value (KV) histories that are read and updated for each request. Every token that has been processed retains its KV state, and during each decode iteration the model reads the current KV cache for the attending tokens and writes updated states back. Unlike shared weights, KV traffic does not amortize across batch elements in the same way; each request carries its own growing history, and the per-request byte demand scales with the attended sequence length.
The Token-Energy Law defines active bytes as the sum of these two contributions: the effective weight bytes per output token (which shrink with larger batches) plus the KV bytes per output token (which grow with longer attended histories). This separation makes it possible to reason about how serving choices—batch size, context length, sequence length—affect the byte movement that ultimately drives energy consumption.
Realized Decode Time: Elapsed Duration per Output Token
Realized decode time is defined as the elapsed decode duration divided by the number of output tokens produced. This quantity captures the time-associated cost per token and naturally incorporates the effects of throughput, batching, and model parallelism. When a GPU processes a batch of requests, the total decode time is spread across all output tokens generated in that batch, so the realized decode time per token typically decreases as batch size increases, up to the point where inter-request dependencies or memory bandwidth saturation intervene. The term also captures hardware effects: different GPU generations achieve different operations-per-second rates for the matrix multiplications and memory copies that dominate decode. By normalizing elapsed time by the number of tokens produced, the model removes the confounding effect of output length and yields a per-token time cost that can be compared across configurations.
The realized decode time term in the Token-Energy Law is not simply wall-clock time divided by token count. It is a measured quantity obtained from the serving runtime: the wall-clock time spent in the decode loop for a given request, divided by the number of tokens the model outputs for that request. If a request produces N output tokens and the decode loop takes T seconds, the realized decode time is T/N. This formulation assumes dense, full-history decoding, where every token attends to all previous tokens in the context. Sparse or causal decoding variants would require adjusted definitions, but the core principle—divide total time by produced tokens—remains the same.
An Affine Form Combining Byte and Time Coordinates
The Token-Energy Law combines the two dimensions—active bytes and realized decode time—into a common affine form. For a given model and hardware/runtime configuration, the energy per output token E is modeled as:
E = α · (active bytes) + β · (realized decode time) + γ
where α and β are coefficients calibrated for the specific model-hardware combination, and γ is a baseline offset that captures constant energy overheads (such as static power draw or fixed iteration startup costs). The affine form is chosen because preliminary analysis showed that energy per token is approximately linear in both byte movement and time when other factors are held constant. The coefficients are not universal; they are determined empirically for each model-hardware pair. Once calibrated, however, the model can be used with fixed coefficients to estimate energy for held-out conditions.
The affine form also makes the marginal effect of each dimension explicit. The coefficient α quantifies how much additional energy is consumed per byte of active byte demand. The coefficient β quantifies how much additional energy is consumed per unit of realized decode time. These coefficients are what allow the model to explain observed patterns: for example, why batching reduces energy per token (because it reduces the per-token active bytes through weight amortization) and why longer contexts increase energy (because they increase the KV bytes per token).
One of the clearest predictions of the Token-Energy Law is that batching amortizes shared-weight work. When multiple requests are processed together in a batch, the same weight matrices are read once and applied to all output tokens in the batch. If the batch size is B and each output token requires reading W weight bytes, the per-token weight bytes without batching would be W. With batching, the weight bytes per output token become approximately W/B, assuming the weight read is the dominant shared-cost component and that the batching overhead does not grow superlinearly. This amortization effect is reflected in the active bytes term: as B increases, the dense-weight contribution to active bytes decreases, and the total energy per output token drops.
The extent of amortization depends on the model architecture and the serving implementation. In some transformers, weight reads are interleaved with KV cache reads, so the batching gain is not purely 1/B. Nevertheless, the Token-Energy Law's calibrated coefficients capture the actual amortization observed on real hardware, and the experiments confirm that increasing batch size consistently reduces the measured energy per output token, with the magnitude of reduction matching the affine model's predictions.
Longer Attended Histories Increase Private KV Demand
The Token-Energy Law predicts that longer attended histories increase private KV demand, and the experiments confirm this pattern. When a model decodes with a long context, each token's KV cache entry grows because it has attended to more previous tokens. During each decode iteration, the model reads the KV entries for all attending tokens and writes updated entries back. If the attended sequence length is L, the KV bytes per output token scale roughly linearly with L, because each token in the context contributes to the cache state that must be read and updated. This increase in KV bytes is reflected in the active bytes term, and because α is positive, the total energy per output token rises with context length.
The experiments measure this effect across models with different context windows and find that the coefficient α captures the per-byte energy cost accurately. Models with larger context windows show higher energy per token when all other factors (batch size, throughput, etc.) are held constant. The Token-Energy Law's separation of dense-weight and KV components makes it possible to attribute the increase to KV demand rather than to weight reads, which would not have the same linear dependence on context length.
Throughput Determines the Time-Associated Cost per Output
The Token-Energy Law's realized decode time term captures the time-associated cost per output, and throughput is the primary factor that determines this cost. Throughput, measured in tokens per second, depends on the GPU generation, the model size, the batch size, and the serving configuration. A higher throughput means that each token is produced in less wall-clock time, and consequently the realized decode time per token is smaller. The coefficient β in the affine model quantifies how much energy is consumed per unit of realized decode time, so the time-associated energy cost is β × (T/N), where T is the decode duration and N is the number of output tokens.
The experiments cover three models decoded on NVIDIA Hopper and Blackwell GPUs. Blackwell, being a newer generation, achieves higher throughput for the same model size, and the measured realized decode times are accordingly lower. The calibrated β coefficient for Blackwell is smaller than for Hopper, reflecting the fact that each unit of decode time consumes less energy on the newer GPU. This finding illustrates how the Token-Energy Law can be used to compare energy efficiency across hardware generations with a single, consistent model.
Experimental Validation Across Three Models
The authors evaluate the Token-Energy Law on three language models decoded in BF16 precision on NVIDIA Hopper and Blackwell GPUs. The models differ in size and architecture, allowing the authors to test the model's generality across typical LLM deployments. For each model, the affine coefficients (α, β, γ) are calibrated on a training set of configurations—varying batch size, context length, and output length—and then held fixed for held-out evaluation conditions.
Using the fixed coefficients and the observed realized decode time, the authors compute held-out condition-mean energy estimates and compare them to actual measurements. The difference between estimated and measured energy is at most 3.73% across all models, configurations, and hardware platforms. This small error demonstrates that the affine model captures the dominant factors determining energy per output token, and that active bytes and realized decode time together explain most of the variation observed in practice.
The three models include a 7Bparameter decoder, a 70Bparameter decoder, and a 405Bparameter decoder. The 7B model shows the smallest absolute energy per token, as expected, but the relative patterns—batching gains, context-length costs, hardware differences—are consistent across all scales. The 70B and 405B models exhibit larger coefficients α and β, reflecting the higher byte movement and longer compute time per token that come with larger parameter counts. Nevertheless, the affine form with model-specific coefficients provides accurate estimates for all three.
Two-GPU Tensor Parallelism on Hopper
The experiments also include two-GPU tensor parallelism on Hopper, where the model's parameters are split across two GPUs and each decode iteration involves communication between the devices. Tensor parallelism changes the byte movement pattern: weight reads are distributed across GPUs, and KV cache updates require synchronization. The Token-Energy Law's affine form remains applicable, but the calibrated coefficients absorb the additional communication overhead. The measured energy per output token for tensor-parallel execution is higher than for single-GPU execution at the same effective batch size, primarily because the β · (realized decode time) term increases due to inter-GPU synchronization time. The α · (active bytes) term also changes because the effective weight bytes per token are altered by the distribution across devices. These adjustments demonstrate the model's flexibility: by recalibrating the coefficients for the tensor-parallel configuration, the same affine form predicts energy within 3.73% of measurements.
Implications for Serving and Hardware Selection
The Token-Energy Law provides a practical tool for two common decision problems in LLM deployment. First, given a target energy budget per output token, practitioners can use the affine model to explore how changing serving parameters—batch size, context length, output length—affects the estimated energy, without needing to re-run full GPU measurements. Second, when comparing different hardware platforms (e.g., Hopper vs. Blackwell, or single-GPU vs. tensor-parallel configurations), the model enables a like-for-like comparison by using each platform's calibrated coefficients. The small estimation error (≤3.73%) means that the model can guide decisions with high confidence.
For example, suppose a serving system currently runs a 70B parameter model on Hopper with a batch size of 4, a context length of 8K tokens, and produces 100 output tokens per request. The system consumes X joules per output token. A practitioner wondering whether switching to Blackwell with the same configuration would reduce energy per token can use the Blackwell‑calibrated coefficients in the affine model to estimate the new energy per token. If the estimated reduction is large enough to justify the hardware migration cost, the decision becomes straightforward. Similarly, if the practitioner can adjust the batch size or context length, the affine model predicts the energy impact of each change, enabling cost-aware serving tuning.
Limitations and Future Extensions
The Token-Energy Law is developed for dense, full-history decoding in BF16 precision. Extensions to other precision formats (FP8, INT8) would require recalibrating the coefficients, as the byte movement and energy per operation differ across precisions. The affine form's linearity assumption may break down under extreme configurations, such as very small batch sizes (where weight amortization is negligible) or very large batch sizes (where memory bandwidth saturation changes the time per token pattern). The model as presented does not account for token-level variance; a single energy value per output token masks per-token differences that can be significant in practice. Finally, the model does not include dynamic power management effects, such as GPU frequency scaling or DVFS, which can introduce additional variation in real deployment settings.
Future work could integrate the Token-Energy Law with runtime power monitoring tools, allowing the coefficients to be updated online as the serving configuration changes. The framework could also be extended to include sparse attention patterns, where the active bytes term would need to account for the reduced KV read/write volume. Another direction is to incorporate the model into automatic serving optimizers that search over batch size, context length, and hardware configuration to minimize energy per token subject to throughput constraints.
Conclusion
The Token-Energy Law introduces a compact, interpretable model of GPU-board energy per output token for dense, full-history LLM decoding. By relating energy to two core dimensions—active bytes and realized decode time—the model explains how batching amortizes shared-weight work, how longer attended histories increase private KV demand, and how throughput determines the time-associated cost per output. The affine form with model- and hardware-specific coefficients predicts held-out energy within 3.73% of measurements across three models on Hopper and Blackwell GPUs, including two-GPU tensor parallelism. The model provides a practical foundation for energy-aware serving decisions, hardware selection, and cost-aware configuration tuning. As LLM deployment scales, tools that can predict and compare energy per token will be essential for managing both operating costs and environmental impact.