Definition
A token is one vocabulary unit produced or consumed by a model. For text models, a token may be a whole word, part of a word, punctuation, whitespace combined with nearby text, a byte sequence, or a special marker. The tokenizer maps each occurrence to an integer identifier, and the model processes the identifier through an embedding or another internal representation.
Tokens are defined by a particular tokenizer and vocabulary. The same sentence can become 8 tokens in one model and 11 in another. Boundaries can also change with capitalization, spelling, Unicode normalization, code, or language. Counting words or characters provides only an estimate unless it uses the exact tokenizer paired with the model.
Subword tokenization became especially influential in neural language systems. Sennrich, Haddow, and Birch's 2016 work adapted byte-pair encoding so frequent words could remain whole while rare words were composed from smaller units. That paper did not invent tokenization, and BPE is one method among word, character, unigram, byte-level, and other schemes.
Tokens also extend beyond ordinary text. Multimodal models may represent image patches, audio segments, actions, or control markers as token-like units.
Distinguish it from nearby terms
- A token is an occurrence in an encoded sequence. A vocabulary item or token type is one reusable entry available to the tokenizer.
- A token identifier is the integer used to refer to that vocabulary item.
- A word is a linguistic unit. It may map to one token, several tokens, or part of a token depending on the tokenizer.
- A character is a writing-system unit. Some tokenizers use characters, but many use larger or smaller encoded units.
Operational significance
Context limits, latency, and API billing are often expressed in tokens. Input, output, cached, and hidden reasoning tokens may be counted or priced differently by a provider. Token count is therefore a capacity and cost measure, but it is not a measure of meaning, difficulty, or work completed.
Check your understanding
A two-page contract and a two-page source file can have similar character counts but very different token counts. Use the target model's tokenizer before making a budget or context-fit decision.