Speech Recognition

Speech recognition (automatic speech recognition, ASR) is the technology that converts spoken language—an audio signal—into written text. It is distinct from speaker recognition, which determines who is speaking, and from language understanding, which extracts meaning or intent from what was said. Speech recognition addresses the transcription itself: identifying the words in the acoustic signal and rendering them as text, without establishing the speaker's identity or interpreting the message.

The classic approach

Through the 1990s and 2000s a scheme based on hidden Markov models (HMM) dominated, modeling the temporal evolution of speech, combined with Gaussian mixture models for the acoustic signal and with a separate language model. Around 2010, deep neural networks replaced the Gaussians and markedly improved accuracy, in a hybrid that became the new state of the art. Documentation: Rabiner's original HMM tutorial.

The end-to-end era

The next leap was to do away with the separate pieces and train a single end-to-end model, going from audio to text directly. It was made possible by techniques such as connectionist temporal classification (CTC, 2006) and attention-based models. Two recent milestones illustrate it: wav2vec 2.0 (Meta, 2020), which learns from unlabeled audio via self-supervised learning and achieves good results with very little transcribed data, and Whisper (OpenAI, 2022), a model trained on hundreds of thousands of hours of internet audio, robust to accents and noise and able to transcribe many languages. Documentation: Deep Speech 2 paper.

How it is measured

A system's quality is evaluated with the word error rate (WER): the proportion of mistranscribed words—substituted, deleted or inserted—relative to the reference text. The lower, the better. It is computed, precisely, by aligning the hypothesis and the reference with an edit distance. Documentation: Deep Speech 2 protocol and results.

The challenges

Despite its advances, speech recognition still has weak points: background noise, spontaneous or overlapping speech, and the variability of accents and dialects. And an important gap persists: systems perform very well in languages with plentiful data, but far worse in low-resource languages. Documentation: original Whisper paper.