Tokenization

Breaking text into chunks (tokens) so AI can process it — typically subword units, not whole words.

Definition

Tokenization splits raw text into the smallest units the model processes. Modern LLMs use subword tokenization (BPE, SentencePiece) — common words become single tokens, rare words split into pieces. Affects multilingual performance.

Examples

  • "unbelievable" → ["un", "bel", "ievable"]
  • 한국어 splits more aggressively than English

Related terms

Looking for actual gigs that use Tokenization? Browse all live listings or filter by $100+/hr.