Problem
Cels builds a visit-protocol system for doctors' offices. A microphone records the visit, speech-to-text (STT) produces a transcript, and an LLM fills in the visit protocol: complaints, history, examination, diagnosis, recommendations. Everything must run on-prem on modest hardware, the audio is poor (noise, interruptions, a distant microphone), and the team has only a few anonymized samples and no public dataset. On such transcripts a small LLM skips fields, loops or hallucinates. Few-shot prompts and dictionary RAG did not help.
Example from the article: STT turned "v poze Romberga ustoychiv" (steady in Romberg's position) into "v puze rombika ustoychiv", and "Fastsikulyatsiya est" into "Rastsikkulyatsii est".
Solution in brief
Generate the training set from scratch: LLM role-play of visits, TTS voicing, degraded audio, the production STT, then extra text noising calibrated until the error distribution matches real demo cases. Fine-tune a small LLM with LoRA on the pairs "dirty transcript → clean protocol".
Questions this case answers
- How to fine-tune a small on-premise language model for structured extraction from noisy speech-to-text when no real training data exists
- How to build a synthetic training set whose errors match the production error distribution
- How to fine-tune a small on-prem LLM when there is no real training data
- How to make an LLM robust to noisy speech-to-text transcripts
Tools and methods
- LLM role-play — doctor and patient dialogues from a sampled visit plan
- Cloud LLM (zero-shot) — builds the reference (GT) protocol from the clean dialogue
- Cloud TTS — voices the dialogues with several voices and presets
- Audio augmentation — reverberation, noise, distant microphone, codec, overlaps, muffling
- Production STT — turns synthetic audio into realistic dirty text
- Text noising — Levenshtein neighbours, word merging, top-N production errors
- LoRA — fine-tuning of the small on-prem LLM
Google or a one-time email code. No password.
