Evolution of DL Serving in a Startup

Egor Shestopalov29 Oct 2022

Problem

A medical ML startup had to expose several neural networks to users as one service. First there were two or three sequential networks, later a branching graph with a classifier, segmenters and mask merging. Each home-built serving layer fixed some problems and created others: lost studies, time spent moving data between nodes, wasted GPU memory, slow deployment and code that only the team could maintain.

Example from the article: a service detecting many pathologies on 3D medical studies, where the answer often needs the original study data and a node may need to be skipped when the classifier finds nothing.

Solution in brief

The team moved through three generations: a Redis queue with PyTorch workers, its own multi-node framework, and Triton Inference Server with models wrapped in TorchScript. The last one gave low transfer overhead, economical GPU memory and a dynamic graph without self-maintenance, at the price of high RAM use, manual warm-up, one machine and NVIDIA-only GPUs. Try a mature off-the-shelf server first.

Questions this case answers
  • How to serve a branching graph of several neural networks with low data-transfer overhead and bounded GPU memory when a small team cannot maintain its own serving framework
  • How to serve a multi-node deep learning pipeline (classifier, segmenters, post-processing) with low overhead?
  • How to choose between a home-grown inference framework and an off-the-shelf server such as Triton?
  • How to avoid wasting GPU memory when many PyTorch nodes run in separate processes?

Tools and methods

  • NVIDIA Triton — serving server with efficient transfer between nodes, dynamic routing, monitoring, gRPC and a limit on simultaneously running models.
  • TorchScript — wraps PyTorch models so a tensor does not reserve about 1 GB of GPU memory.
  • Redis — shared queue and data store in generations 1 and 2.
  • Docker — one container per graph node in generation 2.
Continue reading — it's free

Google or a one-time email code. No password.