Tokenization

Why Tokenization Exists

Neural networks operate on fixed vocabularies of discrete symbols. Raw Unicode text must be split into tokens, mapped to integer IDs, and converted to embeddings. Tokenization defines the interface between human-readable text and model computation. A poor tokenizer inflates sequence length, increases inference cost, and can degrade performance on code, math, or non-English text.

The fundamental tension is vocabulary size vs sequence length. Word-level tokenizers have huge vocabularies and out-of-vocabulary problems. Character-level tokenizers have tiny vocabularies but very long sequences. Subword methods balance both.

Subword Algorithms

Byte Pair Encoding (BPE)

BPE starts with byte or character units and iteratively merges the most frequent pairs until the vocabulary reaches a target size. GPT-2, GPT-3, and many open models use BPE. It handles rare words by composing frequent subword pieces.

WordPiece and Unigram

WordPiece (BERT) selects merges that maximize likelihood under a language model. Unigram (SentencePiece) starts large and prunes tokens to optimize a probabilistic objective. Both are common in multilingual and production pipelines.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")
text = "Retrieval-augmented generation (RAG) grounds LLM answers."
tokens = tokenizer.encode(text)
print(tokens)                          # [128000, 27420, ...]
print(tokenizer.decode(tokens))        # round-trip text
print(len(tokens))                     # billable / context length

SentencePiece

SentencePiece treats text as a raw byte stream and can train without language-specific pre-tokenization. This makes it strong for multilingual models and consistent handling of whitespace and punctuation across scripts.

Special Tokens and Chat Templates

Production tokenizers include special tokens beyond ordinary subwords:

• BOS / EOS mark sequence start and end
• PAD pads batches to equal length during training
• UNK represents unknown symbols in older vocabularies
• Role and control tokens delimit system, user, and assistant turns in chat models

Chat templates are string patterns (often Jinja-based) that wrap messages with the correct special tokens before tokenization. Using the wrong template causes silent quality degradation even when the base model is correct.

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain tokenization in one sentence."},
]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

Bytes, Unicode, and Edge Cases

Byte-level BPE (used in GPT-2) represents any UTF-8 string without UNK tokens by falling back to individual bytes. This is robust but can split non-Latin scripts into many tokens, raising cost for those languages.

Tokenization affects:

• Arithmetic and code - numbers and operators may split unpredictably
• PII and security - filters must run on raw text or detokenized output, not assume token boundaries align with words
• Context limits - the same paragraph can be 200 tokens in one tokenizer and 400 in another

Vocabulary Design Tradeoffs

• Larger vocabularies shorten sequences but increase embedding and softmax cost (vocabulary projection is often the largest output layer).
• Smaller vocabularies reduce parameter count but lengthen sequences, slowing attention (quadratic in naive implementations).
• Multilingual vocabularies need enough capacity for all scripts without starving high-resource languages.
• Domain-specific corpora (legal, medical, code) benefit from tokenizer retraining or vocabulary extension when base tokenizers fragment domain terms.

Practical Pipeline Integration

In serving systems, tokenization runs on every request. Batch tokenization, caching of frequent prefixes, and streaming decode (incremental detokenization) all affect latency. For RAG, chunk sizes are often defined in tokens, not characters, because the model context window is token-bounded. Counting tokens accurately before sending requests prevents truncation and surprise billing on API providers.