Handwriting Recognition
What handwriting recognition is, online and offline: from LeNet and CTC-trained LSTMs to transformers like TrOCR and Donut and on to multimodal models — how it is measured with CER and WER, and where its real limits lie.
Handwriting recognition is the task of turning handwritten text into digital text a machine can search, edit, and process. It is one of the founding problems of computer vision and, despite decades of progress, it remains open: human handwriting varies so much across writers, eras, and languages that no system reads it with full reliability.
Online vs. offline: two different problems
Plamondon and Srihari's survey, published in IEEE TPAMI in 2000, examines online and offline recognition separately. In online recognition, the system receives the pen trajectory as the text is written — the position and order of a stylus tip on a tablet or screen — temporal information that makes the task easier. In offline recognition, only an image of the finished text exists, as in a scanned document; it is the harder case and the focus of most current research under the label HTR (handwritten text recognition).
From classic networks to transformers
The 1998 paper by Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner describes LeNet-5, compares methods on a standard handwritten-digit recognition task, and says that one of the systems discussed was deployed commercially to read several million checks per day. Connectionist temporal classification, or CTC, presented by Graves and co-authors at ICML 2006, introduced a method for training recurrent networks to label unsegmented sequences directly, without requiring pre-segmented training data or later converting each output into a label sequence.
The current generation is end-to-end transformer. Microsoft's TrOCR (2021) pairs a pre-trained vision transformer with a pre-trained text transformer and reports a 2.89% character error rate on the IAM benchmark, per its official repository. Donut, presented at ECCV 2022, is defined as an OCR-free visual document understanding model: it processes the image without outsourcing reading to a separate OCR engine. A 2023 academic evaluation of GPT-4V found good recognition and understanding of Latin content but difficulties with multilingual scenarios, poor camera-captured handwriting, and complex tasks; its results do not support extending the same conclusion to Gemini.
How progress is measured
The standard metrics are CER (character error rate), the proportion of characters that must be inserted, deleted, or substituted to reconstruct the correct transcription, and WER (word error rate), its word-level counterpart. They are computed on public benchmarks: a description by the University of Bern team says IAM then contained more than 1,500 pages of handwritten text contributed by more than 600 writers and based on the LOB corpus. In ICDAR 2024's BRESSAY competition, the results paper records 2.88% CER at line level. The ICDAR 2026 competition on medieval manuscripts starts with nine languages—eight Romance and one Germanic—and reserves another task for generalization to an undisclosed non-Romance European language.
Uses and limits
The highest-impact application is archive digitization: the European platform Transkribus, run by the READ-COOP cooperative, reports more than 200 million pages processed and thousands of archive and library users. Automatic form reading and converting stylus notes to text round out the list. The limits are still real: highly personal or degraded handwriting, languages with little training data, and historical manuscripts full of abbreviations drive error rates up sharply, and in archival work human review remains essential. Primary source.
This article was produced with artificial intelligence under human editorial oversight.
