Definition
A tokenizer is the algorithm, vocabulary, and set of processing rules that encode input into token identifiers and decode generated identifiers back into usable output. Text tokenizers may normalize Unicode, mark whitespace, split or preserve punctuation, apply a learned subword model, and add special control tokens.
The vocabulary gives each available unit an identifier. The encoding algorithm decides how a new string is segmented into those units. A fixed vocabulary can still produce different segmentations if the normalization or splitting rules change. Decoding usually reconstructs text from identifiers, though normalization, invalid byte sequences, or special tokens can make the process more complicated than reversing a simple word split.
How modern subword tokenizers developed
Sennrich, Haddow, and Birch's 2016 paper adapted byte-pair encoding for neural machine translation, making it practical to represent common words directly and rare words through smaller units. In 2018, Taku Kudo and John Richardson introduced SentencePiece, which can train and apply subword models directly to raw sentences without requiring language-specific word splitting first.
These works made particular subword approaches widely useful. They did not invent tokenization as a general concept, and current systems also use unigram, character, byte-level, word-level, and multimodal tokenizers.
Distinguish it from nearby terms
- A token is one unit in the encoded sequence. The tokenizer produces that sequence.
- A vocabulary is the inventory of available token types and identifiers. The tokenizer includes rules for using it.
- An embedding layer converts token identifiers into vectors for the model. Tokenization happens before that learned representation.
- A parser analyzes structure such as syntax or a data format. Tokenization only establishes units and does not necessarily understand their relationships.
Operational significance
A model checkpoint is trained with a specific token-to-identifier mapping. Swapping in a different tokenizer can send the wrong identifiers to the model even if the visible text is unchanged. Treat the tokenizer and its version as part of the model artifact. Test multilingual text, code, whitespace, emoji, normalization, and special-token handling because each can affect cost, truncation, and behavior.
Check your understanding
Two tokenizers both contain the visible string "bank" but assign it different identifiers. A checkpoint trained with the first mapping cannot safely use the second tokenizer without corresponding model changes.