Stop 11 · Large Language ModelsTopics · 2/9

Tokenization (BPE, SentencePiece)

In short

LLMs never see letters or words. They see tokens. Tokens decide your API bill, your context limit, and why models fumble at counting the r's in 'strawberry'.

Splitting text into subword tokens using algorithms like Byte-Pair Encoding — the actual unit LLMs read and generate, which explains quirks like why 'strawberry' confuses token-counting models.

The blob snaps together toy bricks of different sizes, some big single bricks and some made of tiny ones.

Think of it like this

Words built from toy bricks

Picture a toy set where common words come as one big ready-made brick, like 'the' or 'cricket'. Rare words get built from smaller bricks, like 'chai' + 'wala'. The tokenizer keeps a fixed box of bricks and builds any text from them.

Watch it happen

How BPE splits a sentence (illustrative)

love

Bengaluru

traffic

1/5We start with four words. The split below is illustrative; real splits depend on the tokenizer.

The big ideas

What you need to know

BPE merges frequent pairs

Byte-Pair Encoding starts from bytes or characters and keeps merging the most common neighbouring pair until the vocabulary is full.

SentencePiece handles spaces

SentencePiece works on raw text and treats the space as a symbol, so it handles languages without spaces well.

Tokens cost money

APIs bill per token. In English, one token is roughly 4 characters. Hindi or Tamil text often needs more tokens per word.

Explains odd failures

Counting letters or reversing words is hard because the model sees chunks, not individual characters.

See it in code

Walk through the code

python

1# pip install tiktoken2import tiktoken3 4enc = tiktoken.get_encoding("cl100k_base")5 6for text in ["I love Bengaluru traffic", "strawberry", "मुझे चाय पसंद है"]:7    ids = enc.encode(text)8    pieces = [enc.decode([i]) for i in ids]9    print(len(ids), pieces)

1/3Load a BPE encoding used by several OpenAI models.

tiktoken shows exactly how an OpenAI-style BPE tokenizer splits text. Compare English and Hindi counts.

Save yourself the pain

Common mistakes

Oops

Estimating cost or limits by word count.

Do this instead

Count tokens with the model's own tokenizer. Word counts can be off by a lot, especially for code and Indian languages.

Oops

Using one model's tokenizer to count tokens for a different model.

Do this instead

Each model family has its own vocabulary. Use the matching tokenizer, or the provider's token counting tool.

Your mission

Token detective

~20 min

Quick check

Why might an LLM struggle to count the letters in a word?

Go deeper

1Use a real tokenizer (e.g. `tiktoken`) to see how different text splits into tokens.
2Learn why subword tokenization handles rare/unseen words better than word-level tokenization.
3Understand how token count relates directly to context window limits and API cost.