Text as numbers

Models only eat numbers

Everything you’ve built works on numbers. Text isn’t numbers. So before a language model can read anything, we chop the text into pieces called tokens, and give each distinct token an ID — a position in a fixed list called the vocabulary.

How should we chop? Two obvious options, both flawed:

• Characters: tiny vocabulary (just letters and symbols), but sentences become very long sequences, and the model must learn spelling from scratch.
• Whole words: short sequences, but the vocabulary explodes (every name, typo and new word needs its own ID), and unseen words can’t be represented at all.