---
title: 'Gradient descent'
description: 'An iterative optimization method that moves parameters opposite the gradient of an objective being minimized.'
canonical_url: 'https://darkfactory.dev/glossary/gradient-descent'
markdown_url: 'https://darkfactory.dev/glossary/gradient-descent.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# Gradient descent


## Definition

Gradient descent minimizes an objective by moving parameters opposite its local gradient. If the current parameters are theta, the gradient is g, and the learning rate is eta, the basic update is theta minus eta times g. The gradient points toward the steepest local increase, so its negative points toward the steepest local decrease for an infinitesimal step.

Full-batch gradient descent computes the gradient over the whole dataset. Stochastic gradient descent uses one sampled example in the strict definition, while common deep-learning usage calls mini-batch updates SGD as well. Momentum and adaptive optimizers modify the update using history or parameter-specific scaling but remain gradient-based methods.

The direction is local, and a finite step can overshoot. Neural-network objectives are usually nonconvex, so descent does not guarantee the global minimum. Plateaus, saddle points, noisy estimates, poor conditioning, and an unsuitable learning rate can all slow or destabilize training.

## Distinguish it from nearby terms

Backpropagation computes derivatives through a computation graph. Gradient descent uses derivatives to choose a parameter update. An optimizer is the implementation of an update rule, which may include momentum, clipping, weight decay, or adaptive scaling.

## Check your understanding

At one parameter value, the gradient is positive. If the objective is being minimized, which direction should the parameter move? What changes when the learning rate is ten times larger?

## Related terms

- [Backpropagation](https://darkfactory.dev/glossary/backpropagation)
- [Learning rate](https://darkfactory.dev/glossary/learning-rate)
- [Optimizer](https://darkfactory.dev/glossary/optimizer)
- [Loss function](https://darkfactory.dev/glossary/loss-function)

## Evidence and further reading

- [Google Machine Learning Glossary](https://developers.google.com/machine-learning/glossary/)
- [Deep Learning](https://www.deeplearningbook.org/)