1. Character
chatbot → c h a t b o t
chatbot
❌ Too many tokens
2. Word
chatbot works → 2 tokens
chatbotworks
❌ Rare words break
3. Subword ✓
chatbot → chat + bot
chatbot
✅ Best balance
1️⃣ Character tokenization
Text: chatbot → Tokens: ["c","h","a","t","b","o","t"]
❌ Too many tokens. ❌ Hard to learn meaning.
2️⃣ Word tokenization
Text: chatbot works well → Tokens: ["chatbot","works","well"]
❌ Vocabulary explodes. ❌ Rare words break system.
3️⃣ Subword tokenization (modern standard)
Text: chatbot → Tokens: ["chat","bot"]
✅ Best balance. ✅ Handles rare words. ✅ Smaller vocabulary. Used by modern LLMs.