Challenges of Labeling Doctor-Patient Dialogues for ASR

Roman Potekhin24 May 2025

Problem

A team building a doctor's assistant (ASR plus an LLM that writes the medical report) needed a test set of doctor-patient dialogues to check speech recognition. No open medical data existed, so they had to collect and label their own: 5.5 hours of audio, with 10-minute appointments, overlapping speakers and hospital noise. Labeling went wrong in three ways: non-doctors could not label medical terms, the time estimate was off by about 3x, and duplicate recordings threatened to leak between train and test.

Example from the article: a doctor labeled the 10-minute pilot recording at about 2 minutes per minute of audio, but the full 5 hours took 30 hours of doctor time.

Solution in brief

Pre-label with Whisper and have doctors correct the transcripts in short chunks. Test the labeling process on one recording before launching it, then deduplicate by comparing Whisper transcripts rather than file hashes.

Questions this case answers
  • How to build a labeled test set for speech recognition in an expert domain where non-experts cannot label specialist terms, and how to estimate labeling time and budget
  • How to find duplicate long audio recordings that file hashes cannot catch and cut them into chunks for labeling without breaking words
  • How to build a labeled test set for speech recognition in a specialist domain such as medicine
  • How to estimate labeling time and budget for audio data

Tools and methods

  • Whisper — pre-labeling, so labelers correct text instead of typing it
  • WER and CER — over the whole text and over medical terms only
  • Transcript-based deduplication — pairwise WER between Whisper transcripts with a tuned threshold
  • Chunking by punctuation — cuts at periods and commas of the Whisper transcript, at most 30 seconds
Continue reading — it's free

Google or a one-time email code. No password.