GloVe

GloVe (Global Vectors for Word Representation) is a word embedding method: it turns each word into a vector of real numbers, placing words with similar meanings close together in a geometric space. It was introduced in 2014 by Jeffrey Pennington, Richard Socher and Christopher Manning of Stanford University at the EMNLP conference.

What set it apart from rival models of the time was where it drew its information from: instead of learning only from the context windows surrounding each word, GloVe starts from the co-occurrence statistics of the entire corpus at once. Primary source.

The global co-occurrence matrix

The starting point is a vast table: the word-word co-occurrence matrix, which records how often each term appears alongside every other across the whole corpus. A single pass through the text is enough to fill it, and GloVe then trains only on the non-empty cells, which keeps it efficient. Primary source.

The core intuition of Pennington, Socher and Manning is that what matters is not raw frequency but the ratios between co-occurrence probabilities. By comparing how words such as «ice» and «steam» pattern against «solid» or «gas», the noise from non-discriminative words cancels out and meaning emerges. From there the model learns vectors whose dot product approximates the logarithm of the probability that two words co-occur. The resulting space has a striking linear structure: differences between vectors encode relationships, giving rise to the famous arithmetic «king − man + woman ≈ queen». Primary source. Primary source. Primary source.

GloVe versus word2vec and fastText

GloVe is often contrasted with word2vec. The latter is local and predictive: it learns by tuning vectors to predict neighboring words inside small windows of text. GloVe is global and count-based: it factorizes the co-occurrence totals gathered across the full corpus. The distinction is conceptually elegant, but honesty is due: in practice both produce vectors of comparable quality, and the choice comes down to corpus and task. fastText, in turn, extends the word2vec idea by breaking each word into subword pieces (character n-grams), which lets it represent rare or unseen terms and capture morphology more faithfully. Primary source.

From static to contextual embeddings

GloVe produces static embeddings: one vector per word, identical in every sentence. A polysemous word therefore receives the same representation in every context; that is its main limit.