---
title: 'Loss function'
description: "A function that converts a model's prediction and target into a training penalty for an example or batch."
canonical_url: 'https://darkfactory.dev/glossary/loss-function'
markdown_url: 'https://darkfactory.dev/glossary/loss-function.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# Loss function


## Definition

A loss function turns a model output and training target into a numerical penalty. Cross-entropy penalizes predicted probability assigned away from a target distribution. Squared error penalizes distance between predicted and target values. Pairwise and ranking losses compare alternatives rather than scoring one answer in isolation.

Training usually needs a differentiable loss, or a differentiable estimator, so gradients can reach the parameters. The system then reduces example losses across a batch by a sum, mean, weighted mean, or another rule. That reduction changes the scale of the gradient and can change how classes, tokens, or long sequences contribute.

Many systems combine several loss terms. A reconstruction term, regularizer, safety penalty, or auxiliary task may each receive a weight. Those choices encode priorities. A low loss only proves performance against the encoded proxy on the sampled training data; it does not prove factual correctness, fairness, security, or usefulness unless the loss and data measure those properties.

## Distinguish it from nearby terms

A per-example loss is one training penalty. The objective function is the full quantity optimized across data, regularization, and possibly several losses. An evaluation metric may be nondifferentiable and measured after training. Training loss and validation loss use the same formula on different data splits.

## Check your understanding

A summarization model gets lower token-level loss by copying long passages verbatim. The product needs concise, faithful summaries. Which desired behavior is missing or underweighted in the loss?

## Related terms

- [Cross-entropy](https://darkfactory.dev/glossary/cross-entropy)
- [Objective function](https://darkfactory.dev/glossary/objective-function)
- [Gradient descent](https://darkfactory.dev/glossary/gradient-descent)
- [Evaluation (eval)](https://darkfactory.dev/glossary/evaluation)

## Evidence and further reading

- [Google Machine Learning Glossary](https://developers.google.com/machine-learning/glossary/)
- [Deep Learning](https://www.deeplearningbook.org/)