Quasi-Newton methods have long been among the most effective approaches for large-scale unconstrained convex optimization, yet two obstacles have limited their adoption in deep learning: non-convexity and enormous parameter sizes. SoftServe introduces a family of QN methods designed to surmount these barriers without line searches or ad hoc curvature corrections. The framework derives positive-definite curvature estimates from the variational objective of Berglund et al. (2025), maintaining positive definiteness even when negative curvature is present. Diagonal and Kronecker-factored variants are developed that preserve positive definiteness by construction and scale to massive neural networks. The stable coupled Newton-Schulz iteration replaces costly matrix decompositions with GPU-friendly matrix multiplications, enabling practical use in production settings. SoftServe excels on problems that are severely ill-conditioned, including recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines such as Adam, Muon, and SOAP.

Deriving Positive-Definite Curvature from a Variational Objective

SoftServe's core innovation stems from curvature estimates obtained through the variational objective of Berglund et al. (2025). Unlike traditional QN methods that rely on line searches or manual curvature corrections, SoftServe constructs positive-definite estimates inherently, even in the presence of negative curvature. This approach eliminates the need for heuristic adjustments that often destabilize training dynamics.

Diagonal and Kronecker-Factored Variants Scaling to Massive Networks

Two practical variants are presented: a diagonal approximation and a Kronecker-factored approximation. Both are engineered to preserve positive definiteness by construction, meaning the curvature matrices remain invertible throughout optimization. These variants extend the method's reach to networks with hundreds of millions of parameters, demonstrated on a 136M-parameter physics-informed diffusion model where SoftServe consistently outperforms Adam, Muon, and SOAP in final loss attainment.

Stable Coupled Newton-Schulz Iteration for Matrix Operations

Matrix inverses and square roots required by QN methods are computed via the stable coupled Newton-Schulz iteration. This replacement of costly matrix decompositions with GPU-friendly matrix multiplications is a key factor in the method's scalability. The iteration is designed for numerical stability, preventing the divergence that can afflict naïve fixed-point schemes in high-dimensional settings.

Empirical Results Across Ill-Conditioned Tasks

Experiments span recurrent networks, deep autoencoders, physics-informed neural networks, and the 136M-parameter diffusion model. In each case, SoftServe achieves lower validation or training losses than the reported baselines. Recurrent networks benefit from the method's ability to handle slowly varying gradients, while physics-informed networks profit from the improved conditioning near solution manifolds. The diffusion model results are particularly notable, as the method navigates a high-dimensional loss landscape that typically stymies first-order approaches.

  • Recurrent networks: improved convergence speed and lower asymptotic loss
  • Deep autoencoders: faster descent with comparable final reconstruction quality
  • Physics-informed neural networks: better satisfaction of PDE constraints during optimization
  • 136M-parameter diffusion model: lower ELBO than Adam, Muon, and SOAP across all training stages

These results indicate that the methodological contributions—positive-definite curvature estimation, structured approximations, and the Newton-Schulz iteration—combine to produce a method that is both more robust and more efficient than existing alternatives.