Definition
A tokenizer converts text or other input into token identifiers a model can process and decodes generated identifiers back into usable output.
Distinguish it from nearby terms
A token is the resulting unit; the tokenizer is the algorithm and vocabulary that create those units.
Check your understanding
Tokenize the same phrase with two tokenizers and explain differences in length, boundaries, and cost.