Tokenization (BPE, SentencePiece)

Summary

Splitting text into subword tokens using algorithms like Byte-Pair Encoding — the actual unit LLMs read and generate, which explains quirks like why 'strawberry' confuses token-counting models.

• 1Use a real tokenizer (e.g. `tiktoken`) to see how different text splits into tokens.
• 2Learn why subword tokenization handles rare/unseen words better than word-level tokenization.
• 3Understand how token count relates directly to context window limits and API cost.