Problem
A medical ML startup had to expose several neural networks to users as one service. First there were two or three sequential networks, later a branching graph with a classifier, segmenters and mask merging. Each home-built serving layer fixed some problems and created others: lost studies, time spent moving data between nodes, wasted GPU memory, slow deployment and code that only the team could maintain.
Example from the article: a service detecting many pathologies on 3D medical studies, where the answer often needs the original study data and a node may need to be skipped when the classifier finds nothing.
Solution in brief
The team moved through three generations: a Redis queue with PyTorch workers, its own multi-node framework, and Triton Inference Server with models wrapped in TorchScript. The last one gave low transfer overhead, economical GPU memory and a dynamic graph without self-maintenance, at the price of high RAM use, manual warm-up, one machine and NVIDIA-only GPUs. Try a mature off-the-shelf server first.
Questions this case answers
- How to serve a branching graph of several neural networks with low data-transfer overhead and bounded GPU memory when a small team cannot maintain its own serving framework
- How to serve a multi-node deep learning pipeline (classifier, segmenters, post-processing) with low overhead?
- How to choose between a home-grown inference framework and an off-the-shelf server such as Triton?
- How to avoid wasting GPU memory when many PyTorch nodes run in separate processes?
Tools and methods
- NVIDIA Triton — serving server with efficient transfer between nodes, dynamic routing, monitoring, gRPC and a limit on simultaneously running models.
- TorchScript — wraps PyTorch models so a tensor does not reserve about 1 GB of GPU memory.
- Redis — shared queue and data store in generations 1 and 2.
- Docker — one container per graph node in generation 2.
Google or a one-time email code. No password.