Tokenization
Breaking text into chunks (tokens) so AI can process it — typically subword units, not whole words.
Definition
Tokenization splits raw text into the smallest units the model processes. Modern LLMs use subword tokenization (BPE, SentencePiece) — common words become single tokens, rare words split into pieces. Affects multilingual performance.
Examples
- "unbelievable" → ["un", "bel", "ievable"]
- 한국어 splits more aggressively than English
Related terms
Looking for actual gigs that use Tokenization? Browse all live listings or filter by $100+/hr.