Improving speech recognition without labels, sign-up or SMS

Anton Legchenko29 Oct 2022

Problem

A company analyzes operator dialogues and needs accurate transcripts to search for keywords and phrases. The audio is mono and noisy, and the vocabulary is domain-specific, so an ASR (automatic speech recognition) model trained on open data degrades. Plenty of unlabeled audio exists, but there is almost no budget to label it.

Example from the article: the open datasets (about 1000 hours, mostly smart-device speech) do not match operator dialogues, and word error rate (WER) on the speaker's own telephony recordings started at 0.45.

Solution in brief

Keep the acoustic model and add a text-to-text error corrector after it. Train the corrector on synthetic data: good domain texts with errors inserted from the recognizer's own mistakes. Make open audio resemble your calls with augmentations (noise, a second speaker, call drops). Select good texts by a corrector-based check plus a language-model score, so no manual labeling is needed.

Questions this case answers
  • How to adapt a recognition model to a new acoustic domain when labeled data is almost absent and unlabeled data is abundant
  • How to train a text corrector on synthetic errors derived from the recognizer's own mistakes and keep only reliable source texts
  • How to adapt a speech recognizer to noisy call-center audio when there is almost no budget for labeling
  • How to build training data for an ASR error-correction model from unlabeled audio

Tools and methods

  • wav2vec 2.0 — multilingual acoustic model, fine-tuned on the open Golos dataset for Russian
  • Text-to-text transformer corrector — takes the recognized text with errors and outputs corrected text (the speaker thinks it is probably mT5)
  • BERT-type masked language model — scores how coherent a text is, to filter bad transcripts
  • Levenshtein distance — matches recognized word variants to correct words when building the error dictionary
  • Data augmentation — noise, overlaid speech and call drops
Continue reading — it's free

Google or a one-time email code. No password.