Problem
A company analyzes operator dialogues and needs accurate transcripts to search for keywords and phrases. The audio is mono and noisy, and the vocabulary is domain-specific, so an ASR (automatic speech recognition) model trained on open data degrades. Plenty of unlabeled audio exists, but there is almost no budget to label it.
Example from the article: the open datasets (about 1000 hours, mostly smart-device speech) do not match operator dialogues, and word error rate (WER) on the speaker's own telephony recordings started at 0.45.
Solution in brief
Keep the acoustic model and add a text-to-text error corrector after it. Train the corrector on synthetic data: good domain texts with errors inserted from the recognizer's own mistakes. Make open audio resemble your calls with augmentations (noise, a second speaker, call drops). Select good texts by a corrector-based check plus a language-model score, so no manual labeling is needed.
Questions this case answers
- How to adapt a recognition model to a new acoustic domain when labeled data is almost absent and unlabeled data is abundant
- How to train a text corrector on synthetic errors derived from the recognizer's own mistakes and keep only reliable source texts
- How to adapt a speech recognizer to noisy call-center audio when there is almost no budget for labeling
- How to build training data for an ASR error-correction model from unlabeled audio
Tools and methods
- wav2vec 2.0 — multilingual acoustic model, fine-tuned on the open Golos dataset for Russian
- Text-to-text transformer corrector — takes the recognized text with errors and outputs corrected text (the speaker thinks it is probably mT5)
- BERT-type masked language model — scores how coherent a text is, to filter bad transcripts
- Levenshtein distance — matches recognized word variants to correct words when building the error dictionary
- Data augmentation — noise, overlaid speech and call drops
Google or a one-time email code. No password.