A Speech Platform in Production

Michil Egorov23 May 2026

Problem

X5 Tech, the technology unit of a retail chain, started speech recognition with Whisper behind Kafka for transcribing internal meetings. Other teams (contact center, stores, chatbots) then asked for recognition, and the setup stopped fitting. It could only work offline, Whisper hallucinated, it was costly to teach internal terms, and clients disliked Kafka as the contract. The new requirements: streaming, WER below 0.2, 500 minutes of audio per minute, about 2000 requests per second at peak.

Example from the article: Whisper inserted phrases such as "to be continued" that were not in the recording. On the X5 audio-badge set its WER was 0.784.

Solution in brief

Replace the end-to-end Whisper with a staged ASR backend (acoustic model, language-model decoding, punctuation, filters) behind a gRPC/HTTP gateway. Train a Conformer-type acoustic model from scratch on open data plus domain audio labeled by pseudo-labeling, and add expected noises as training augmentation. Scale with MIG, TensorRT and EWMA balancing.

Questions this case answers
  • How to replace a single end-to-end speech recognizer with a staged streaming service that serves many clients at high load
  • How to improve speech recognition on domain terms and noisy telephony audio when labeled transcripts are scarce
  • How to move a speech recognition (ASR) service from a Whisper prototype to a production platform with streaming and high load
  • How to improve ASR accuracy on company-specific terms and noisy telephony audio

Tools and methods

  • Whisper (Whisper podlodka) in Triton — first version; VAD, ONNX backend and fine-tuning for Russian patched its problems
  • NVIDIA Riva — fast first approximation of the staged pipeline on Triton
  • Conformer / Fastconformer acoustic model + small language model — own model trained from scratch
  • Pseudo-labeling with ROVER — labels raw audio using several models
  • Noise augmentation — specific noises added during training
  • MIG, EWMA balancer, TensorRT — serving and scaling
Continue reading — it's free

Google or a one-time email code. No password.