A Synthetic Training Pipeline for Medical LLMs

Evgeny Gutin23 May 2026

Problem

Cels builds a visit-protocol system for doctors' offices. A microphone records the visit, speech-to-text (STT) produces a transcript, and an LLM fills in the visit protocol: complaints, history, examination, diagnosis, recommendations. Everything must run on-prem on modest hardware, the audio is poor (noise, interruptions, a distant microphone), and the team has only a few anonymized samples and no public dataset. On such transcripts a small LLM skips fields, loops or hallucinates. Few-shot prompts and dictionary RAG did not help.

Example from the article: STT turned "v poze Romberga ustoychiv" (steady in Romberg's position) into "v puze rombika ustoychiv", and "Fastsikulyatsiya est" into "Rastsikkulyatsii est".

Solution in brief

Generate the training set from scratch: LLM role-play of visits, TTS voicing, degraded audio, the production STT, then extra text noising calibrated until the error distribution matches real demo cases. Fine-tune a small LLM with LoRA on the pairs "dirty transcript → clean protocol".

Questions this case answers
  • How to fine-tune a small on-premise language model for structured extraction from noisy speech-to-text when no real training data exists
  • How to build a synthetic training set whose errors match the production error distribution
  • How to fine-tune a small on-prem LLM when there is no real training data
  • How to make an LLM robust to noisy speech-to-text transcripts

Tools and methods

  • LLM role-play — doctor and patient dialogues from a sampled visit plan
  • Cloud LLM (zero-shot) — builds the reference (GT) protocol from the clean dialogue
  • Cloud TTS — voices the dialogues with several voices and presets
  • Audio augmentation — reverberation, noise, distant microphone, codec, overlaps, muffling
  • Production STT — turns synthetic audio into realistic dirty text
  • Text noising — Levenshtein neighbours, word merging, top-N production errors
  • LoRA — fine-tuning of the small on-prem LLM
Continue reading — it's free

Google or a one-time email code. No password.