Sampling & temperature

From probabilities to text

A language model doesn’t write — it outputs a probability for every possible next token. Writing means repeatedly picking one, appending it, and asking again. How you pick is called decoding, and it dramatically changes what comes out.

• Greedy: always take the single most likely token. Safe, but dull and prone to loops (“the the the”).
• Sampling: pick randomly, in proportion to the probabilities. Varied and natural — but occasionally picks something absurd from the long tail.