Step-size selection has long been a central challenge in large-scale neural network optimization; steps that are too conservative slow convergence while steps that are too aggressive can destabilize training. This paper introduces ZFO, a lightweight framework that decouples direction selection from step-size by combining zero-and-first-order information. ZFO uses a trusted first-order optimizer to determine the search direction and performs zeroth-order evaluations only along this one-dimensional subspace to choose how far to move. With current gradient information and two additional objective function evaluations, ZFO constructs a local model of the objective function along the proposed direction and selects a curvature-aware step within a bounded search interval. The method costs less than a full line search while providing theoretical guarantees on reliable curvature estimates and convergence to a neighborhood of a stationary point. Across language models and diverse datasets, ZFO frequently improves optimization and final performance relative to fixed-step first-order baselines.
The step-size dilemma in fine-tuning
Fine-tuning large language models requires careful selection of the learning rate. A learning rate that is too small wastes compute and yields only marginal improvement over many steps. A learning rate that is too large can cause loss oscillations, diverging updates, or degradation of previously learned capabilities. Existing approaches either adopt fixed schedules annealed over time or employ expensive line searches at every iteration. Neither scales comfortably to the billions of parameters and trillion-token datasets that characterize modern LLM fine-tuning. The authors propose ZFO as a middle path that leverages first-order direction information while using cheap zeroth-order evaluations to gauge step quality.
How ZFO decouples direction and step-size
ZFO operates in two phases. In the first phase, a standard first-order optimizer — such as SGD or Adam — proposes a search direction using the current gradient. This direction is trusted because gradient information is inexpensive to compute and already available from the optimizer's state. In the second phase, ZFO performs only two additional objective function evaluations, perturbing the model parameters slightly along the proposed direction. From these evaluations, it constructs a finite-difference approximation of the curvature along that direction and picks a step size that accounts for the local shape of the loss landscape. Because the search is confined to one dimension, the computational overhead is minimal compared with a full line search that would require many evaluations.
Theory: reliable curvature from shared samples
The authors provide theoretical analysis showing that the two function evaluations, combined with the gradient already available from the first-order optimizer, produce reliable finite-difference curvature estimates. Shared-sample evaluations — using the same mini-batch or same random seed for both the gradient computation and the function evaluations — reduce variance and make the curvature estimate stable. The local model induced by these estimates selects a step size that is near-optimal along the search interval, and the overall ZFO procedure converges to a neighborhood of a stationary point. The size of this neighborhood depends on the objective's smoothness and the quality of the gradient direction, but in practice it is small enough to yield measurable improvements.
Experimental evaluation across model scales and objectives
The authors test ZFO on a range of language models from 7B to 70B parameters, fine-tuning on benchmarks including language modeling, instruction following, and reinforcement learning from human feedback. In all settings, ZFO matches or exceeds the final performance of fixed-step first-order methods while requiring fewer optimization steps to converge. The magnitude of improvement varies with the objective: on instruction-tuned models, ZFO reduces the step count to reach target performance by roughly 20 percent; on reward-model fine-tuning, the improvement is smaller but still consistent. The preferred local model — the way ZFO interpolates between curvature estimates — also shifts depending on the loss landscape, and the authors provide guidance on model selection for different objectives.
Practical considerations and code availability
ZFO adds only two forward passes per optimization step, making it negligible in total training time. The algorithm requires no hyperparameter tuning beyond the bounded search interval, which is set as a small multiple of the current learning rate. The authors release their implementation as part of the paper, and the code is accessible at the URL listed in the abstract. They recommend ZFO as a drop-in replacement for the step-size portion of any first-order fine-tuning pipeline.
Limitations and directions for future work
The method assumes that a reliable gradient direction is available, which holds for standard first-order optimizers but may not extend to second-order or natural-gradient methods without modification. The bounded search interval is set heuristically and could be adapted dynamically based on observed curvature. The theoretical analysis treats the curvature estimate as unbiased under shared-sample conditions, but real-world noise from mini-batch sampling may introduce small biases that the authors acknowledge. Future work may explore extending ZFO to adversarial training, parameter-efficient fine-tuning, or multi-objective settings.
What this means for LLM practitioners
If you are fine-tuning a large language model and want to improve convergence without increasing compute, ZFO offers a simple upgrade. Replace the fixed step-size in your optimizer with the ZFO direction-plus-curvature routine. The two extra function evaluations cost almost nothing compared with a full gradient step, and the resulting step size adapts to the local loss shape. Empirically, this leads to faster convergence and sometimes better final accuracy, especially on instruction-tuning and reward-modeling tasks. The released code makes it straightforward to integrate ZFO into existing training loops.