---
title: 'Activation function'
description: "A function that transforms a neural network unit's pre-activation, usually to add nonlinearity or control how much signal passes forward."
canonical_url: 'https://darkfactory.dev/glossary/activation-function'
markdown_url: 'https://darkfactory.dev/glossary/activation-function.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# Activation function


## Definition

An activation function transforms the value produced by a neural network unit before that value passes to the next layer. Most hidden-layer activations operate element by element. Common examples include sigmoid, hyperbolic tangent, the rectified linear unit (ReLU), and GELU.

The nonlinearity is the important part. A stack of affine transformations with no nonlinear activation can still be collapsed into one affine transformation, regardless of how many layers it contains. Nonlinear activations let the network represent curved decision boundaries and interactions that a purely linear model cannot express.

## ReLU's role in modern networks

The choice also changes training behavior. Saturating functions such as sigmoid can produce very small gradients at extreme inputs. ReLU returns zero for negative inputs and passes positive inputs through, which made it a practical alternative for deep networks. Vinod Nair and Geoffrey Hinton documented that use of rectified linear units in 2010; they did not introduce activation functions as a general category.

## Distinguish it from nearby terms

Weights and biases produce the pre-activation. The activation function transforms it. An output function such as softmax has a related but more specific job: it turns a vector of scores into a normalized distribution. Activation functions are also different from gates, although a gate may use sigmoid or another activation internally.

## Check your understanding

You replace every hidden-layer activation with the identity function and keep all weights and biases. Has the network become deeper in what it can represent, or can its layers be reduced to one affine transformation? Explain why.

## Related terms

- [Neural network](https://darkfactory.dev/glossary/neural-network)
- [Loss function](https://darkfactory.dev/glossary/loss-function)
- [Gradient descent](https://darkfactory.dev/glossary/gradient-descent)

## Evidence and further reading

- [Google Machine Learning Glossary](https://developers.google.com/machine-learning/glossary/)
- [Deep Learning](https://www.deeplearningbook.org/)
- [Rectified Linear Units Improve Restricted Boltzmann Machines](https://icml.cc/2010/papers/432.pdf)