Word embeddings

“You shall know a word by the company it keeps,” wrote the linguist J. R. Firth in 1957. Words that show up in similar places tend to mean similar things: coffee and tea both get brewed, poured, and drunk; piano and violin both get played and tuned. That idea, called the distributional hypothesis, is all it takes to turn words into useful vectors.

Learning vectors from text

A word embeddingEmbeddingA list of numbers (a vector) that represents a token, word, sentence, or image, learned so that similar things end up with similar vectors.Open in glossary gives each word a vector, typically 50 to 300 numbers long, chosen so that words used in similar contexts get similar vectors. Two landmark methods made this practical. word2vec (Mikolov and colleagues at Google, 2013) trains a small network to predict the words around each word. GloVe (Pennington, Socher, and Manning at Stanford, 2014) fits vectors directly to counts of how often each pair of words appears near each other.

The demos below use real GloVe vectors trained on about six billion words from Wikipedia and newswire text. Each of the 10,000 words here is a list of 100 numbers. Nobody chose those numbers, and none of them means “size” or “color” on its own. Meaning lives in the directions and distances between the vectors: each word becomes a point in a latent spaceLatent spaceThe space of learned, hidden coordinates a model uses to represent things internally, such as word vectors or a network's hidden layers. Nearby points stand for similar things.Open in glossary, a space of learned coordinates that no one labeled.

Behind every list is the cosine similarity from the previous lesson, computed between one word and the 9,999 others.

Meaning as direction

The most famous property of word vectors is that differences between them can be meaningful. If you take the vector for king, subtract man, and add woman, the nearest word to the result, other than the three you started with, is queen. This vector offset method was introduced with word2vec and is often called 3CosAdd: to complete “a is to b as c is to ?”, compute $\mathbf{b} - \mathbf{a} + \mathbf{c}$ and find the nearest word, leaving out $a$, $b$, and $c$ themselves.

Why does this work? If the training text uses king and queen in nearly the same contexts except for the ones that differ by gender, and the same is true for man and woman, then the step from one to the other ends up pointing in a similar direction in both pairs. The map view shows this directly: each arrow goes from one word in a pair to the other.

Embeddings absorb their training text

Embeddings learn whatever patterns are in the text, including ones nobody wants. In the analogies view, man : doctor :: woman returns “nurse”. In 2016, Bolukbasi and colleagues documented this kind of gender stereotyping in word2vec vectors in a paper titled “Man is to Computer Programmer as Woman is to Homemaker?”, and warned that it can carry into systems built on the vectors.

There is a second, subtler lesson in the same example. The standard method never returns an input word, so “doctor” itself cannot be the answer. In 2020 Nissim, van Noord, and van der Goot showed that when that rule is lifted, the original word often comes back first, and that some widely shared biased analogies looked more extreme than the vectors support. Both things are true: the vectors do encode stereotypes from text, and the way results are presented can exaggerate them.