---
title: 'Model vocabulary'
description: "The finite mapping between token types and identifiers that a tokenizer uses and a model's embedding and output layers are trained to interpret."
canonical_url: 'https://darkfactory.dev/glossary/vocabulary'
markdown_url: 'https://darkfactory.dev/glossary/vocabulary.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# Model vocabulary


## Definition

A model vocabulary is the finite mapping between token types and integer identifiers used by a tokenizer and model. The vocabulary may contain whole words, subwords, characters, byte sequences, punctuation, whitespace-bearing units, and special control markers. The model's embedding and output layers are trained against this exact mapping.

Vocabulary size affects several tradeoffs. A larger vocabulary can encode frequent strings in fewer tokens but needs larger embedding and output tables. A smaller vocabulary reuses units more widely but may produce longer sequences. Coverage also depends on the tokenizer design. Some systems use an unknown-token marker for text outside the vocabulary. Byte-level fallback can represent any encoded text by decomposing it into smaller units.

Sennrich, Haddow, and Birch's 2016 subword work showed how a fixed vocabulary could represent rare and unseen words through learned smaller units. Kudo and Richardson's SentencePiece later trained subword vocabularies directly from raw sentences. These papers advanced specific vocabulary-construction methods; they did not invent the general idea of a vocabulary.

## Distinguish it from nearby terms

- A language's **lexicon** is its set of words and expressions. A model vocabulary is an engineering artifact and need not align with words.
- A **token** is one occurrence from the vocabulary in an encoded sequence.
- A **tokenizer** contains the normalization and segmentation rules that choose vocabulary entries for an input.
- An **embedding** is the learned vector associated with a token identifier. It is not the identifier or vocabulary entry itself.

## Operational significance

Changing the vocabulary or identifier assignments without corresponding model training breaks the checkpoint's interpretation of input. Extending a vocabulary requires deliberate initialization and training for new rows. Compare token fertility across the languages, scripts, code, and notation that matter to the product, since an English-efficient vocabulary may consume far more context elsewhere.

## Check your understanding

A tokenizer has no whole entry for "electroencephalography." It can still encode the word through several subword or byte units. The word is unfamiliar to the vocabulary as a whole string, not necessarily impossible to represent.

## Related terms

- [Token](https://darkfactory.dev/glossary/token)
- [Tokenizer](https://darkfactory.dev/glossary/tokenizer)

## Evidence and further reading

- [Google Machine Learning Glossary](https://developers.google.com/machine-learning/glossary/)
- [Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/)
- [Neural Machine Translation of Rare Words with Subword Units](https://aclanthology.org/P16-1162/)
- [SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing](https://aclanthology.org/D18-2012/)