Speech Recognition

Speech recognition (ASR, automatic speech recognition) is the technology that converts spoken language—an audio signal—into text. It is worth distinguishing it from two neighboring tasks: speaker recognition (working out who is speaking) and language understanding (extracting meaning or intent). Speech recognition deals only with transcription.

The classic approach

Through the 1990s and 2000s a scheme based on hidden Markov models (HMM) dominated, modeling the temporal evolution of speech, combined with Gaussian mixture models for the acoustic signal and with a separate language model. Around 2010, deep neural networks replaced the Gaussians and markedly improved accuracy, in a hybrid that became the new state of the art.

The end-to-end era

The next leap was to do away with the separate pieces and train a single end-to-end model, going from audio to text directly. It was made possible by techniques such as connectionist temporal classification (CTC, 2006) and attention-based models. Two recent milestones illustrate it: wav2vec 2.0 (Meta, 2020), which learns from unlabeled audio via self-supervised learning and achieves good results with very little transcribed data, and Whisper (OpenAI, 2022), a model trained on hundreds of thousands of hours of internet audio, robust to accents and noise and able to transcribe many languages.

How it is measured

A system's quality is evaluated with the word error rate (WER): the proportion of mistranscribed words—substituted, deleted or inserted—relative to the reference text. The lower, the better. It is computed, precisely, by aligning the hypothesis and the reference with an edit distance.

The challenges

Despite its advances, speech recognition still has weak points: background noise, spontaneous or overlapping speech, and the variability of accents and dialects. And an important gap persists: systems perform very well in languages with plentiful data, but far worse in low-resource languages.