Definition
A mixture-of-experts model contains several expert subnetworks and a learned gating or routing function. The router assigns an input, example, or token to one expert or a weighted combination. In a sparse MoE, only a small subset runs for each item, which increases total parameter capacity without activating every parameter on every forward pass.
Routing creates its own training and systems problems. Experts can receive uneven traffic, so implementations use load-balancing losses and capacity limits. Tokens may be delayed, rerouted, or dropped when an expert's capacity is exceeded. In distributed systems, moving activations to experts on other devices can make communication a larger bottleneck than arithmetic.
Total parameters and active parameters must be reported separately. A model with one trillion stored parameters may use only a fraction for one token. Sparse activation reduces some computation, but it does not erase memory, networking, storage, or coordination costs.
From local experts to sparse neural MoE
Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton published "Adaptive Mixtures of Local Experts" in 1991. The paper's footnote says Jacobs and Hinton first presented the idea at the 1988 Connectionist Summer School. It used a gating network to allocate cases among expert networks.
Noam Shazeer and colleagues introduced the sparsely gated MoE layer in 2017, with a trainable gate choosing a sparse combination from up to thousands of feed-forward experts. That work is a direct ancestor of modern sparse MoE language models.
Distinguish it from nearby terms
MoE routing selects components inside one trained model. Model routing selects among separately deployed models or systems. An ensemble runs several models and combines their predictions; a sparse MoE normally activates only selected experts for each item.
Check your understanding
An MoE has 400 billion total parameters but activates 20 billion per token. Which number better predicts storage, which better approximates per-token arithmetic, and what costs does neither capture?