BERT

BERT (Bidirectional Encoder Representations from Transformers) is a language model introduced by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, of Google, in a paper posted to arXiv in 2018 and published at the NAACL conference in 2019. It is an encoder built on the Transformer architecture that reads text bidirectionally: every word is interpreted by attending to its left and right context at once, something earlier left-to-right models could not do.

Its job is not to generate text but to produce representations —vectors— of language that are then fine-tuned for specific comprehension tasks.

How it is trained: masking and the NSP caveat

BERT is pre-trained on unlabeled text (BooksCorpus and English Wikipedia, roughly 3.3 billion words) with two objectives. The first, Masked Language Modeling (MLM), randomly hides some words and forces the model to predict them from context; this is what makes bidirectional reading possible. The second, Next Sentence Prediction (NSP), asks it to judge whether two sentences were consecutive in the source text.

For years NSP was described as a defining trait of BERT, but that needs qualifying: in 2019 the RoBERTa team (Liu et al.) dropped NSP, trained on more data with larger batches and dynamic masking, and matched or beat the original. Their conclusion was that NSP contributed little and was not essential, and most later models abandoned it. MLM, by contrast, remains the heart of the family.

What made it matter

BERT's impact came from a method: pre-train once on massive text, then adapt the same model to each task by adding little more than an output layer. That recipe set records on suites such as GLUE and SQuAD and made «pre-train and fine-tune» the default in language processing. Bidirectionality, against the earlier one-directional reading, captured context-dependent meaning far better.

BERT versus today's generative LLMs

In 2026 it helps to keep two families apart. Popular generative LLMs (GPT, Claude, Gemini) are decoders that produce text. BERT and its descendants are encoders built to understand: classification, sentiment analysis, information retrieval and sentence embeddings. Sentence-BERT (Reimers and Gurevych, 2019) adapted BERT to output sentence vectors comparable by cosine similarity, the backbone of much semantic search and of the retrieval systems that now feed RAG-style applications. Thanks to their size and speed, encoders remain the tool of choice when the goal is to understand or search rather than write.

Modern encoders

The architecture has not stood still. DeBERTa (He et al., 2021) introduced disentangled attention, representing content and position separately, and surpassed BERT and RoBERTa. In December 2024, Answer.AI and LightOn released ModernBERT, a redesign with an 8,192-token context window, rotary embeddings (RoPE) and Flash Attention, trained on two trillion tokens and pitched as a faster, more accurate replacement for BERT.

One question stays open: exactly why NSP added so little —whether the task was too easy or conflated topic detection with coherence— is still debated among researchers.