Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs. Existing approaches either compress optimizer state, abandon first-order gradients, or change the update geometry while retaining dense state. The recently introduced Muon optimizer reduces optimizer memory through matrix-valued updates. Still, its geometry differs from AdamW and can lead to performance degradation when fine-tuning AdamW-pretrained models. To reduce optimizer memory without sacrificing accuracy or computational efficiency in LLM fine-tuning, we propose Ternary Absolute-max Column-wise One-sparse optimizer, or TACO, which follows Muon's operator-norm steepest-descent view but takes the geometric route further.
TACO Optimizer Design
TACO computes the exact steepest-descent direction under a dimension-normalized $1\to1$ operator norm by selecting the sign of the largest magnitude entry in each column of two-dimensional weight matrices. This retains first-order gradients while making optimizer state memory nearly negligible. The practical TACO optimizer maintains only a small set of low precision gradient components per column, reducing persistent optimizer state by $174\times$ relative to AdamW8bit (from 27.7 GB to 0.16 GB) and peak training memory by $2.9\times$ (from 80.6 GB to 27.5 GB) on OPT-13B, while achieving comparable accuracy and runtime.
Enabling Larger Model Fine-Tuning
TACO further enables full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100 GPU across multiple model families and tasks. This expansion of feasible model sizes opens new possibilities for researchers and organizations working with constrained hardware resources.
Comparison to Existing Optimizers
Compared to existing approaches, TACO occupies a middle ground: it retains first-order gradient information like AdamW while achieving memory efficiency comparable to second-order or matrix-valued update methods like Muon. The column-wise ternary selection scheme is both computationally efficient and effective at preserving the optimization geometry required for strong fine-tuning performance.
Empirical Results
On OPT-13B, TACO reduces persistent optimizer state from 27.7 GB (AdamW8bit) to 0.16 GB and peak training memory from 80.6 GB to 27.5 GB, a $2.9\times$ reduction. Despite these drastic memory savings, TACO achieves comparable final model performance and training runtime to AdamW8bit. The enabling of 30-32B model fine-tuning on a single 80 GB H100 GPU across multiple model families and tasks demonstrates the practical impact of the memory reduction.
Practical Implications
For practitioners fine-tuning large language models, TACO provides a drop-in optimizer replacement that significantly reduces GPU memory requirements without sacrifice in model quality. The reduced memory footprint allows fine-tuning of substantially larger models on the same hardware, potentially lowering the cost and increasing the accessibility of LLM customization.