Definition
A model vocabulary is the finite mapping between token types and integer identifiers used by a tokenizer and model. The vocabulary may contain whole words, subwords, characters, byte sequences, punctuation, whitespace-bearing units, and special control markers. The model's embedding and output layers are trained against this exact mapping.
Vocabulary size affects several tradeoffs. A larger vocabulary can encode frequent strings in fewer tokens but needs larger embedding and output tables. A smaller vocabulary reuses units more widely but may produce longer sequences. Coverage also depends on the tokenizer design. Some systems use an unknown-token marker for text outside the vocabulary. Byte-level fallback can represent any encoded text by decomposing it into smaller units.
Sennrich, Haddow, and Birch's 2016 subword work showed how a fixed vocabulary could represent rare and unseen words through learned smaller units. Kudo and Richardson's SentencePiece later trained subword vocabularies directly from raw sentences. These papers advanced specific vocabulary-construction methods; they did not invent the general idea of a vocabulary.
Distinguish it from nearby terms
- A language's lexicon is its set of words and expressions. A model vocabulary is an engineering artifact and need not align with words.
- A token is one occurrence from the vocabulary in an encoded sequence.
- A tokenizer contains the normalization and segmentation rules that choose vocabulary entries for an input.
- An embedding is the learned vector associated with a token identifier. It is not the identifier or vocabulary entry itself.
Operational significance
Changing the vocabulary or identifier assignments without corresponding model training breaks the checkpoint's interpretation of input. Extending a vocabulary requires deliberate initialization and training for new rows. Compare token fertility across the languages, scripts, code, and notation that matter to the product, since an English-efficient vocabulary may consume far more context elsewhere.
Check your understanding
A tokenizer has no whole entry for "electroencephalography." It can still encode the word through several subword or byte units. The word is unfamiliar to the vocabulary as a whole string, not necessarily impossible to represent.