Lesson 8.3ConceptMath~40 min
Embeddings: meaning as geometry
Words become points in space, where direction means something. Explore it in 3D.
Module 8 · 0/8 done
The problem with ID numbers
A token ID is just a label. Token 412 isn’t “closer” to token 413 than to token 9000. If we fed raw IDs to a network, it would think cat (412) and cathedral (413) were nearly the same. The naive fix, a one-hot vector (all zeros except a 1 in the token’s slot), treats every word as exactly equally different from every other.
Check yourself
With one-hot vectors, what is the dot product between “cat” and “kitten”?
Complete the challenge above to continue.