Interview: Tokenization

Word-level tokenization creates huge vocabularies and can't handle unseen/misspelled words (out-of-vocabulary problem). Character-level avoids OOV but produces very long sequences, making the model expensive and harder to learn long-range structure. Subword tokenization is the practical middle ground.

BPE starts with individual characters/bytes as the base vocabulary and iteratively merges the most frequent adjacent pair into a new token, repeating until a target vocabulary size is reached. Common words end up as single tokens while rare words get split into meaningful subword pieces, giving a good balance of vocabulary size and sequence length.

• BPE: greedy frequency-based merges (GPT family).
• WordPiece: similar to BPE but merges pairs that maximize training-data likelihood, not just frequency (BERT).
• Unigram/SentencePiece: starts with a large vocabulary and prunes tokens that least hurt the likelihood of the corpus; SentencePiece treats text as a raw stream (including spaces) so it's language-agnostic and doesn't need pre-tokenized words (used in LLaMA, T5).

Tokenizers often split numbers and words inconsistently - "hello" might be one token but a specific number could be split arbitrarily across digit groups, and the model never sees individual characters directly, only token IDs. This makes character-level tasks (spelling, letter-counting) and precise multi-digit arithmetic fundamentally harder for a token-based model, independent of reasoning ability.

A larger vocabulary means shorter sequences for the same text (cheaper compute, more context fits) but a bigger embedding/output matrix and more parameters spent on rare tokens. A smaller vocabulary means longer sequences (more compute per input) but a leaner model. Production LLMs typically land in the 32k–200k range depending on multilingual needs.