Handwriting Recognition

Handwriting recognition is the task of turning handwritten text into digital text a machine can search, edit, and process. It is one of the founding problems of computer vision and, despite decades of progress, it remains open: human handwriting varies so much across writers, eras, and languages that no system reads it with full reliability.

Online vs. offline: two different problems

The classic distinction was laid out in Plamondon and Srihari's survey (IEEE TPAMI, 2000). In online recognition, the system receives the pen trajectory as the text is written — the position and order of a stylus tip on a tablet or screen — temporal information that makes the task easier. In offline recognition, only an image of the finished text exists, as in a scanned document; it is the harder case and the focus of most current research under the label HTR (handwritten text recognition).

From classic networks to transformers

Handwritten digits were deep learning's first great proving ground: the convolutional network LeNet-5 from Yann LeCun and colleagues (Proceedings of the IEEE, 1998), evaluated on MNIST, went on to read millions of bank checks per day in production systems. For full text, the breakthrough was connectionist temporal classification, or CTC (Graves et al., ICML 2006), which made it possible to train LSTM recurrent networks on whole lines without segmenting them character by character, and carried the field for over a decade.

The current generation is end-to-end transformer. Microsoft's TrOCR (2021) pairs a pre-trained vision transformer with a pre-trained text transformer and reports a 2.89% character error rate on the IAM benchmark, per its official repository. Donut (ECCV 2022) removes the OCR engine altogether, mapping a document image straight to output with no separate text detection stage. And multimodal models such as GPT-4V or Gemini transcribe manuscripts from a plain prompt: academic evaluations from 2023 and 2024 credit them with solid performance on legible Latin-script writing, with clear difficulties in multilingual scenarios and degraded documents.

How progress is measured

The standard metrics are CER (character error rate), the proportion of characters that must be inserted, deleted, or substituted to reconstruct the correct transcription, and WER (word error rate), its word-level counterpart. They are computed on public benchmarks: the most cited is the IAM database (Marti and Bunke, 2002), with 1,539 pages written by 657 people and more than 13,000 lines of English. The ICDAR conference competitions set the state of the art: in the 2024 edition on handwritten Brazilian Portuguese essays the best systems dropped below 3% line-level CER, and the 2026 competition evaluates medieval manuscripts in eight Romance languages, including a generalization test on an undisclosed language.

Uses and limits

The highest-impact application is archive digitization: the European platform Transkribus, run by the READ-COOP cooperative, reports more than 50 million historical pages processed for libraries and archives. Automatic form reading and converting stylus notes to text round out the list. The limits are still real: highly personal or degraded handwriting, languages with little training data, and historical manuscripts full of abbreviations drive error rates up sharply, and in archival work human review remains essential.