Optical Character Recognition (OCR)

Optical Character Recognition (OCR) is a computer-vision technology designed to convert text held inside images—scanned documents, photographs, image PDFs or screenshots—into machine-encoded, editable text. Its task, in essence, is to turn pixels into character strings: image → text.

It helps to pin down what OCR is not. It does not interpret the meaning of what it transcribes: that understanding belongs to later stages of natural-language processing or Document AI. Nor is it a language model; in fact, OCR produces the digital text those models then work on. For that reason it is best understood as one component within the broader field of document understanding.

How it works: from the classic pipeline to end-to-end models

Classic OCR chains several stages: detecting the regions that contain text, segmenting them into lines, words and characters, recognition and, finally, linguistic correction. Its weak point is error propagation between stages. The open-source Tesseract engine documents a classic pipeline built around page analysis, line finding and adaptive character classification.

Modern OCR is largely end-to-end. The foundational architecture combines convolutional networks, recurrent networks and the CTC transcription loss—introduced in the CRNN model by Shi, Bai and Yao in 2015—to read a sequence without first segmenting each character. On that base, Transformers have displaced convolutional and recurrent networks: TrOCR (Microsoft, 2021) is a fully Transformer encoder-decoder, and Donut (Naver, 2022) proposes an “OCR-free” approach that reads the document directly from the image, avoiding the error propagation of an explicit recognition stage. Today's multimodal models further fold recognition and document question-answering into a single system.

Why BERT is not an OCR model

Linking BERT to OCR is a common mistake. BERT (Devlin et al., 2018) is a language model: a Transformer that processes already-digital text for language-understanding tasks, and its input is never pixels. The correct connection to documents runs through derived models such as LayoutLM, which reuse BERT's architecture but operate on the output of an OCR—also incorporating each word's position on the page—to understand forms or extract data. They do not replace OCR: they depend on it.

How it is evaluated, and what remains hard

The usual metrics are the Character Error Rate (CER) and the Word Error Rate (WER), both based on the edit distance between the transcription and the reference text. For recent multimodal models, dedicated benchmarks such as OCRBench are also used. These metrics carry an important limitation: they penalize every error equally and do not capture reading-order or structural failures well, so a system can get every character right and still scramble columns or tables. The frontiers that remain open are handwritten text, low-resource languages and scripts, and documents with complex layouts.