word2vec, GloVe, and the analogy trick

1

Concept

Before contextual language models, static word embeddings made an important idea concrete: distributional patterns can become geometry. Instead of assigning each word an unrelated symbol, methods learned dense vectors from how words occur around one another. The resulting neighborhoods and directions supported similarity, classification, and surprisingly memorable analogy demonstrations.

The word2vec name covers predictive objectives and training techniques. In continuous bag-of-words, surrounding context helps predict a center word. In skip-gram, the center helps predict nearby context words. Computing a full vocabulary softmax for every pair is expensive, so practical training uses approximations such as negative sampling or hierarchical softmax. The learned input and output embedding tables have distinct roles; which table or combination is exported affects results.

The context window defines the signal. A narrow window tends to emphasize syntactic or functional similarity; a broader one can emphasize topic. Subsampling very frequent words and drawing negative examples change which distinctions receive gradient. Corpus domain matters: legal, medical, news, and conversational usage produce different neighborhoods. “The word2vec embedding” is incomplete without training choices and data.

GloVe begins from a global word-context co-occurrence matrix. Its design uses ratios of co-occurrence probabilities to capture distinctions: whether two probe words relate differently to two context words can be informative even when raw counts vary greatly. Training learns vectors whose dot products, with biases, approximate a function of co-occurrence counts. It connects matrix-factorization intuitions with local context learning.

The famous analogy query computes something like $v_{king}-v_{man}+v_{woman}$ and searches for a nearby word, often queen in demonstrations. The hope is that a relationship corresponds to a reusable displacement. Similar tests examine capitals and countries, comparatives, or grammatical forms. A consistent offset is a compact and useful property: one linear operation transfers a relation across pairs.

But the “analogy trick” is sensitive. Results change with cosine normalization, nearest-neighbor rule, whether source words may be returned, vocabulary filtering, and the exact dataset. Some relations form clusters rather than one direction. Frequency and morphology can explain part of performance. Hand-picked examples overstate reliability, and a top-1 miss can still place a sensible answer nearby. Evaluation must report protocol and compare baselines.

The geometry can encode harmful regularities as faithfully as useful ones. Occupational and identity associations learned from text may reproduce stereotypes. An analogy is not a neutral fact simply because it appears as vector arithmetic. Debiasing one direction may leave information elsewhere, and suppressing measurable association can harm legitimate distinctions. Data provenance and downstream behavior matter more than a visually satisfying projection.

Modern Transformers use contextual representations rather than one final vector per word, yet the lesson survives. Training objectives turn distributional evidence into continuous structure. Linear probes and vector operations can expose regularities, but their interpretation depends on layer, metric, corpus, and task. word2vec and GloVe are valuable not because they solved meaning, but because they showed how far learned geometry can go—and made its limits measurable.

For a responsible comparison, publish the corpus snapshot, window definition, vocabulary threshold, negative-sampling policy, exported table, similarity metric, and analogy-scoring rule. Re-running one famous equation without that contract is a demonstration, not a reproducible evaluation.