Definition
A recurrent neural network processes a sequence by applying the same state-update rule at each step. The current input and previous hidden state produce a new hidden state, which can feed an output and the next step. Weight sharing lets the same network operate on sequences of different lengths.
Training usually unfolds the recurrence across time and applies backpropagation through that expanded computation. Long sequences multiply many derivatives. They can shrink toward zero or grow without bound, causing vanishing or exploding gradients. Gradient clipping limits explosions, while gated architectures such as LSTMs and GRUs give the network better control over what to retain and forget.
Recurrence creates a sequential dependency: step 20 needs state from step 19. That can suit streaming and compact stateful inference, but it limits parallel training across positions. Transformers use attention to connect positions more directly and usually parallelize sequence training more effectively.
Historical context
Jeffrey Elman's 1990 paper "Finding Structure in Time" described a simple recurrent network whose hidden activations feed back as dynamic memory. The paper credits a related proposal to Michael Jordan in 1986, so it should not be read as the origin of all recurrent networks. Kyunghyun Cho and colleagues used gated recurrent units in their 2014 RNN Encoder-Decoder for machine translation.
Distinguish it from nearby terms
An RNN carries a state through repeated updates. A transformer primarily relates positions through attention. An encoder-decoder model describes the mapping between input and output sequences and can be built with either recurrent networks or transformers.
Check your understanding
An RNN must copy one bit of information across 1,000 steps before producing an answer. What must its state preserve, and how can repeated derivatives make the dependency hard to learn?