---
title: Token
description: 'One vocabulary unit produced or consumed by a model, such as a word, subword, byte sequence, punctuation mark, special marker, or non-text unit.'
canonical_url: 'https://darkfactory.dev/glossary/token'
markdown_url: 'https://darkfactory.dev/glossary/token.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# Token


## Definition

A token is one vocabulary unit produced or consumed by a model. For text models, a token may be a whole word, part of a word, punctuation, whitespace combined with nearby text, a byte sequence, or a special marker. The tokenizer maps each occurrence to an integer identifier, and the model processes the identifier through an embedding or another internal representation.

Tokens are defined by a particular tokenizer and vocabulary. The same sentence can become 8 tokens in one model and 11 in another. Boundaries can also change with capitalization, spelling, Unicode normalization, code, or language. Counting words or characters provides only an estimate unless it uses the exact tokenizer paired with the model.

Subword tokenization became especially influential in neural language systems. Sennrich, Haddow, and Birch's 2016 work adapted byte-pair encoding so frequent words could remain whole while rare words were composed from smaller units. That paper did not invent tokenization, and BPE is one method among word, character, unigram, byte-level, and other schemes.

Tokens also extend beyond ordinary text. Multimodal models may represent image patches, audio segments, actions, or control markers as token-like units.

## Distinguish it from nearby terms

- A **token** is an occurrence in an encoded sequence. A **vocabulary item** or token type is one reusable entry available to the tokenizer.
- A **token identifier** is the integer used to refer to that vocabulary item.
- A **word** is a linguistic unit. It may map to one token, several tokens, or part of a token depending on the tokenizer.
- A **character** is a writing-system unit. Some tokenizers use characters, but many use larger or smaller encoded units.

## Operational significance

Context limits, latency, and API billing are often expressed in tokens. Input, output, cached, and hidden reasoning tokens may be counted or priced differently by a provider. Token count is therefore a capacity and cost measure, but it is not a measure of meaning, difficulty, or work completed.

## Check your understanding

A two-page contract and a two-page source file can have similar character counts but very different token counts. Use the target model's tokenizer before making a budget or context-fit decision.

## Related terms

- [Input token](https://darkfactory.dev/glossary/input-token)
- [Output token](https://darkfactory.dev/glossary/output-token)
- [Reasoning token](https://darkfactory.dev/glossary/reasoning-token)
- [Token budget](https://darkfactory.dev/glossary/token-budget)
- [Token burn](https://darkfactory.dev/glossary/token-burn)
- [Token efficiency](https://darkfactory.dev/glossary/token-efficiency)
- [Token maxing](https://darkfactory.dev/glossary/token-maxing)
- [Token minning](https://darkfactory.dev/glossary/token-minning)
- [Tokenization tax](https://darkfactory.dev/glossary/tokenization-tax)

## Related factory areas

- [Model selection, routing & budgets](https://darkfactory.dev/factory/model-routing-budgets)

## Evidence and further reading

- [Google Machine Learning Glossary](https://developers.google.com/machine-learning/glossary/)
- [Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/)
- [Neural Machine Translation of Rare Words with Subword Units](https://aclanthology.org/P16-1162/)
- [Token Budgets](https://arxiv.org/abs/2606.04056)