The onnx-asr Library: Speech Recognition in Python with ONNX Models (Without PyTorch)

Ilya Stupakov24 May 2025

Problem

Teams that run speech recognition in a product or on a device usually take a model from a training library such as Whisper, NeMo, GigaAM or transformers. These libraries pull in many badly pinned dependencies, break on new NumPy versions and force a careful choice between CPU and CUDA builds of PyTorch. The stack is heavy for weak hardware. The alternative, sherpa-onnx, has few dependencies but is unfriendly to beginners and implements GigaAM with the wrong spectrogram window, which costs accuracy.

Example from the article: the speaker teaches, and students struggle to install such stacks; the same model should also run on a Raspberry Pi.

Solution in brief

Export only the encoder and decoder to ONNX and write the rest in Python. The log-mel preprocessor is rebuilt as an ONNX graph with ONNX Script, while decoding and tokenization are plain Python. The resulting onnx-asr library needs only numpy and onnxruntime, matches the accuracy of default implementations and is usually 1.5–2 times faster than PyTorch ones.

Questions this case answers
  • How to serve a sequence model with a multi-step preprocessing and decoding chain in a minimal runtime without the training framework's dependencies
  • How to export a model's full preprocessing graph to ONNX so that a small device can run it, and choose a decoding algorithm by speed and accuracy
  • How to run speech recognition models in Python without PyTorch and its dependency problems
  • How to deploy ASR on low-power devices such as a Raspberry Pi

Tools and methods

  • onnx-asr — pure-Python ASR inference
  • ONNX Runtime — runs the exported encoder and decoder
  • ONNX Script — builds the preprocessor graph in Python
  • CTC, RNN-T, TDT decoding — greedy decoders in Python
  • GigaAM v2, NeMo, Vosk, Whisper — supported models
  • RTFx, WER, CER — comparison metrics
Continue reading — it's free

Google or a one-time email code. No password.