Word2Vec: When Words Acquire Geometry

What would it mean for a word to have a direction?

Word2Vec gives each word a vector and learns those vectors from a prediction task. Words occurring in related contexts tend to acquire related representations.

Stylized embedding map with semantic word neighborhoods. — A two-dimensional cartoon of a much higher-dimensional phenomenon: related words often occupy related neighborhoods.

The training trick

In skip-gram, given a centre word, the model predicts surrounding context words. In CBOW, surrounding context predicts the centre.

The remarkable part is that the learned weights become useful representations.

similarity = dot(u, v) / (norm(u) * norm(v))

Cosine similarity then turns geometric proximity into a practical semantic lookup.

Directions can carry relations

The famous analogy intuition is

v_{\text{king}}-v_{\text{man}}+v_{\text{woman}} \approx v_{\text{queen}}.

Parallelogram-style vector analogy relating king, man, woman, and queen. — An analogy can appear as an approximately reusable displacement in representation space.

But this is not symbolic logic. Embeddings reflect statistical regularities, corpus choices and biases. The geometry is powerful precisely because it is a compressed model—not reality itself.

That brings us back to George Box: all models are wrong, but some are useful.