Lesson 8.4ModelCode~1 h

Learning embeddings

Train word vectors from scratch with word2vec — words that keep the same company end up neighbours.

Module 8 · 0/8 done

Where do embeddings come from?

Last lesson, words were points in space with similar meanings close together. But who decides where “cat” goes? Nobody writes those numbers by hand. They’re learned, from a simple observation by the linguist J. R. Firth: “You shall know a word by the company it keeps.”

“Cat” and “dog” turn up in the same sorts of sentences: “the ___ sleeps”, “a little ___ runs”. “Bread” turns up somewhere else: “she eats the ___”. So start by simply counting which words appear near which.

Challengecounts and similar

// "You shall know a word by the company it keeps." Count which words appear near which.
random.seed(1)
const sents = data.sentences(2000).map(s => s.split(" "))
const vocab = [...new Set(sents.flat())].sort()
const id = new Map(vocab.map((w, i) => [w, i]))
console.log(vocab.length, "words, e.g.", sents[0].join(" "))

// 1. counts(window): a vocab × vocab matrix. C[a][b] = how often word b appears within
//    "window" positions of word a (on either side, in the same sentence, not counting a itself).
function counts(window: number): number[][] {
  const C = vocab.map(() => new Array(vocab.length).fill(0))
  return C
}

// 2. similar(C, word, k): the k other words whose rows are most similar (cosine) to this word's row.
function similar(C: number[][], word: string, k: number): string[] {
  return []
}

const C = counts(2)
for (const w of ["cat", "bread", "kitchen", "teacher"]) console.log(w, "→", similar(C, w, 4).join(", "))

Write your code, then press Check (⌘/Ctrl + Shift + Enter). Run just executes it.