Optimizers
last edited
Optimizers are the rules that turn gradients into weight updates during training. Every LoRA or full fine-tune of an image model runs one. The optimizer sets memory use, speed and how sensitive a run is to the learning rate.
From SGD to Adam
Plain stochastic gradient descent steps against the gradient with a single learning rate. Momentum adds a running average of past gradients, which damps zigzags across narrow valleys of the loss.
Adam appeared in 2014. It keeps a second running average of squared gradients and scales each weight's step by it. AdamW followed in 2017 and decoupled weight decay from the gradient step.
Memory and speed
Adam-style optimizers store two extra values per weight. At 32-bit precision that is 8 bytes per weight. Adafactor from 2018 factors the squared-gradient average into row and column statistics.
The 8-bit optimizers paper from 2021 quantizes the optimizer states and keeps a small fraction of the memory footprint.
Lion came out of a program search in 2023. It keeps only momentum and applies the sign of the update, so every weight moves by the same amount. Trainers such as the kohya scripts expose several optimizers as options.
Learning-rate-free and newer methods
Prodigy estimates the step size during training. It became a common pick for character and style LoRA training.
Schedule-Free optimizers from 2024 drop the decay schedule by keeping a running mean of the weights. Muon orthogonalizes the momentum of each weight matrix with a short iterative routine.
A February 2025 report on Muon at scale found about twice the compute efficiency of AdamW in compute-optimal language model training. Variants followed. MONA adds accelerated momentum and was submitted in May 2026. Newton-Muon appeared in April 2026.
Sources
- Adam paper on arXiv, 2014
- AdamW paper on arXiv, decoupled weight decay
- Adafactor paper on arXiv, factored statistics
- 8-bit optimizers paper on arXiv, quantized states
- Lion paper on arXiv, sign-based updates
- Prodigy paper on arXiv, adaptive step size
- Schedule-Free paper on arXiv
- Muon post by Keller Jordan
- Muon is Scalable for LLM Training