FastText

FastText is an open-source library for building word-vector representations and classifying text, developed by Facebook AI Research, now Meta AI. Its official site describes it as free, lightweight and able to run on generic hardware. Its contribution was not to invent embeddings, but to look inside a word: instead of treating each term as one indivisible unit, it represents it through character fragments.

That seemingly minor decision solves two problems that dogged earlier models: words the system has never seen, and languages with rich morphology.

The core idea: a word is a bag of character n-grams

In «Enriching Word Vectors with Subword Information» (TACL, 2017), Piotr Bojanowski, Édouard Grave, Armand Joulin and Tomáš Mikolov propose representing each word as a bag of character n-grams —the chunks of letters that make it up. The word «where», for instance, breaks down into pieces such as «whe», «her», «ere» and the whole word itself. Each of those pieces gets its own vector, and a word's vector is the sum of its n-gram vectors. The model inherits word2vec's skipgram scheme but operates over subwords rather than whole words.

Why it handles words it has never seen

This is where FastText's characteristic advantage comes from. A model that only knows whole words has no vector for a term missing from its training vocabulary: the out-of-vocabulary, or OOV, problem. FastText can compose one by summing n-grams it has learned. The subword paper highlights the challenge of large vocabularies and rare words, and evaluates the method in nine languages. Shared fragments explain why related forms such as “run”, “running” and “runner” can be close without assigning an independent vector to each one. That evidence covers rarity and OOV; it does not guarantee robustness to every typo, neologism or proper name.

Text classification and where it fits

FastText also includes a classifier, introduced in “Bag of Tricks for Efficient Text Classification” (Joulin, Grave, Bojanowski and Mikolov, 2016). In the published experiments it was often on par with deep classifiers in accuracy and several orders of magnitude faster: it trained on more than one billion words in under ten minutes on a standard multicore CPU. To stay fast with many categories, it can use hierarchical softmax. The benchmarks include topic and sentiment classification; these are results from the paper, not a guarantee for every corpus or machine.

FastText's place is specific: the word-vector paper enriches static representations with subwords and makes it possible to build OOV vectors; the classification paper prioritises speed and a small footprint in defined experiments. It does not produce contextual embeddings like later models, and there is no evidence for saying it “paved their way”. It remains useful when compute cost, size and an open vocabulary matter.