Problem
Teams that run speech recognition in a product or on a device usually take a model from a training library such as Whisper, NeMo, GigaAM or transformers. These libraries pull in many badly pinned dependencies, break on new NumPy versions and force a careful choice between CPU and CUDA builds of PyTorch. The stack is heavy for weak hardware. The alternative, sherpa-onnx, has few dependencies but is unfriendly to beginners and implements GigaAM with the wrong spectrogram window, which costs accuracy.
Example from the article: the speaker teaches, and students struggle to install such stacks; the same model should also run on a Raspberry Pi.
Solution in brief
Export only the encoder and decoder to ONNX and write the rest in Python. The log-mel preprocessor is rebuilt as an ONNX graph with ONNX Script, while decoding and tokenization are plain Python. The resulting onnx-asr library needs only numpy and onnxruntime, matches the accuracy of default implementations and is usually 1.5–2 times faster than PyTorch ones.
Questions this case answers
- How to serve a sequence model with a multi-step preprocessing and decoding chain in a minimal runtime without the training framework's dependencies
- How to export a model's full preprocessing graph to ONNX so that a small device can run it, and choose a decoding algorithm by speed and accuracy
- How to run speech recognition models in Python without PyTorch and its dependency problems
- How to deploy ASR on low-power devices such as a Raspberry Pi
Tools and methods
- onnx-asr — pure-Python ASR inference
- ONNX Runtime — runs the exported encoder and decoder
- ONNX Script — builds the preprocessor graph in Python
- CTC, RNN-T, TDT decoding — greedy decoders in Python
- GigaAM v2, NeMo, Vosk, Whisper — supported models
- RTFx, WER, CER — comparison metrics
Google or a one-time email code. No password.
