---
title: Cross-entropy
description: 'A loss that measures how poorly a predicted probability distribution represents the target distribution.'
canonical_url: 'https://darkfactory.dev/glossary/cross-entropy'
markdown_url: 'https://darkfactory.dev/glossary/cross-entropy.md'
collection: glossary
date_published: '2026-08-04T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# Cross-entropy


## Definition

Cross-entropy measures how a predicted probability distribution q represents a target distribution p. For discrete outcomes it is the negative sum of p(x) times log q(x). Lower values place more predicted probability on outcomes the target says are likely.

In ordinary single-label classification, the target is often one-hot: the correct class has probability one and every other class has probability zero. Cross-entropy then reduces to the negative log probability assigned to the correct class. A confidently wrong prediction receives a much larger penalty than an uncertain one. With soft targets, label smoothing, or distillation, several target classes can contribute to the loss.

Cross-entropy is closely connected to maximum likelihood. Minimizing the average negative log probability of observed labels is equivalent to minimizing cross-entropy under the usual classification setup. Implementation details still matter: class weighting, ignored labels, reduction rules, and whether the function expects logits or probabilities can change the calculation.

## Distinguish it from nearby terms

Accuracy counts final class decisions. It cannot distinguish a prediction of 0.51 from 0.99 when both choose the correct class. Cross-entropy uses the full distribution and penalizes confidence assigned to the wrong answer. Entropy measures uncertainty within one distribution; cross-entropy compares a target distribution with a second distribution used to encode or predict it.

## Check your understanding

Two classifiers make the same number of mistakes. One assigns 0.49 to the correct class on each mistake; the other assigns 0.0001. Which has the worse cross-entropy, and why can their accuracy remain identical?

## Also called

cross-entropy loss

## Related terms

- [Loss function](https://darkfactory.dev/glossary/loss-function)
- [Probability distribution](https://darkfactory.dev/glossary/probability-distribution)
- [Training](https://darkfactory.dev/glossary/training)

## Evidence and further reading

- [Google Machine Learning Glossary](https://developers.google.com/machine-learning/glossary/)
- [Deep Learning](https://www.deeplearningbook.org/)