Foundations

Model vocabulary

stable definition

Definition

A model vocabulary is the finite set of token identifiers defined by its tokenizer and understood by the model.

Distinguish it from nearby terms

A language's words are not the same as a model vocabulary; one word may map to several tokens and one token may span several characters.

Check your understanding

Explain how an unfamiliar word can still be represented using subword or byte-level tokens.