The scale of n-gram embedding memory in modern models
Many open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory.
Low-bit precision for n-gram embeddings
The authors investigate whether this memory cost can be reduced while preserving the performance improvement that the tables provide. The proposed approach stores n-gram embeddings at low precision and systematically investigates different quantization methods and bit allocations across tables.
Experimental results on from-scratch models
Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables. Using fewer bits leads to a larger drop in performance, although the authors find that a short fine-tuning phase recovers a significant part of this loss. Reinvesting the saved memory in larger tables can improve performance beyond that of the original full-precision tables while still using less memory.
Memory efficiency on existing large models
These table memory savings also extend to large existing models: on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables save 47-55% of table memory, with perplexity increases of less than 1.2% and largely preserved downstream task performance.
Fine-tuning and table reinvestment
Lower bit allocations introduce performance costs that can be mitigated through targeted fine-tuning. Beyond recovery, the memory freed by low-bit storage can be redirected to expand the n-gram tables themselves. Models that reinvest saved capacity into larger tables surpass the performance of original full-precision tables while consuming less memory overall.
Practical takeaways
Low-bit precision establishes a practical way to reduce the memory requirements of n-gram embeddings. The same savings enable models to allocate more capacity to the embedding tables, improving performance beyond original baselines. These results demonstrate that careful quantization of n-gram storage is a viable path for making large language models more memory-efficient without sacrificing capability.