Life Without Servers: Comparing Libraries for Embedding Neural Networks into C++ Applications

Ilya Goncharov7 Nov 2025

Problem

A small team ships a C++ desktop application for processing scans and wants to add a neural network. A server is a poor fit: one scan can weigh hundreds of megabytes or gigabytes, and many users would need expensive GPU servers. Some customers, such as German archives and libraries, switch off internet access. The team cannot write a separate backend per platform, and user hardware ranges from gaming PCs to an old Intel MacBook. A wrong library choice means users wait minutes for a result.

Example from the article: a denoising network for scanned photographs (the speaker's company supports Mac and Windows). On an M1 Pro MacBook, a 6546x4186 image takes 445 ms with ncnn on Vulkan and 44898 ms with ONNX Runtime on CPU.

Solution in brief

Export the model to ONNX, write a small test application that runs it through each candidate library, and measure inference time on your own model, input sizes and hardware. In the talk only ncnn gave acceptable times.

Questions this case answers
  • How to choose an inference library for running a neural network locally on end-user hardware when a server is too costly and some users have no internet
  • How to export a model that contains layers unsupported by the target format and benchmark candidate runtimes on your own model and hardware before committing
  • How to choose a library for running a neural network locally inside a cross-platform C++ desktop app
  • How to benchmark inference libraries on your own model and hardware before committing

Tools and methods

  • ncnn — Tencent library, Vulkan GPU backend, simple API
  • ONNX Runtime — Microsoft's ONNX runner, native backends
  • OpenCV dnn — inference module with several backends
  • GGML / GGUF — low-level library behind llama.cpp
  • PyTorch to ONNX export — cross-platform model exchange
  • FP32 to FP16 quantization — smaller weights, no visible quality loss in the speaker's case
Continue reading — it's free

Google or a one-time email code. No password.