Definition
An optimizer turns a gradient into a parameter update. The simplest stochastic gradient descent rule multiplies the current gradient by a learning rate and moves in the opposite direction. Other optimizers keep state across steps. Momentum tracks a moving direction, while adaptive methods scale updates using statistics collected for each parameter.
The optimizer changes the path through parameter space, not the definition of success. The loss or objective says what the run is trying to improve. Backpropagation computes gradients of that quantity. The optimizer decides how much of each gradient to apply, after accounting for its state and settings.
Optimizer state can be large. Adam usually stores first- and second-moment estimates for every trained parameter, in addition to the model weights and gradients. A resumable training checkpoint therefore needs the optimizer state and step count as well as the weights. Resuming with fresh state can produce a different trajectory even when the data and weights are unchanged.
Adam
Diederik P. Kingma and Jimmy Ba introduced Adam in their 2014 paper "Adam: A Method for Stochastic Optimization." Adam combines adaptive per-parameter scaling with estimates of the gradient's first and second moments. Its name comes from "adaptive moment estimation." The paper introduced one optimizer, not the general optimizer concept.
No optimizer wins on every model, dataset, or compute budget. Learning rate, schedule, numerical precision, batch construction, clipping, and regularization can matter as much as the optimizer family.
Distinguish it from nearby terms
Gradient descent is a family of update ideas. An optimizer is a concrete rule and its state. The learning rate is one of its controls. A scheduler changes that control over time but does not replace the optimizer.
Check your understanding
A run resumes from saved weights but omits Adam's moment estimates and step count. Why can its next update differ from the interrupted run even when it reads the same batch?