Problem
A team serves a GPT-style assistant of Large size, and answers come too slowly or the model does not fit on one GPU. Three transformer properties cause this. Attention cost grows quadratically with input length, weights and cache take a lot of GPU memory, and text is generated token by token. The speakers call the last one probably unsolvable. The price is latency and the number of GPUs.
Example from the article: the authors accelerated a model based on ruGPT-3 Large and compared three inference packages on it.
Solution in brief
First switch on the free tricks: hidden-state caching and torch inference mode. Then try an inference package: DeepSpeed Inference is the simplest and, by the speaker's observations on models under a billion parameters, the fastest; ONNX Runtime is a good second option; FasterTransformer suits the largest scale. Architecture changes lose quality.
Questions this case answers
- How to cut latency and GPU memory of a large autoregressive transformer at inference without retraining it
- How to choose an inference runtime for a transformer when the model does not fit on one accelerator
- How to speed up inference of a GPT-style language model without retraining it
- How to choose between ONNX Runtime, DeepSpeed Inference and FasterTransformer for serving a transformer
Tools and methods
- Hidden-state caching, torch inference mode — free baseline to compare against.
- ONNX Runtime — fused blocks for an exported ONNX graph.
- DeepSpeed Inference — optimized CUDA kernels, a few lines of code.
- FasterTransformer — NVIDIA's CUDA/C++ implementation for the largest models.
- Quantization (bitsandbytes, 8-bit) — integer instead of floating-point operations.
- FlashAttention — block-wise attention, fewer slow-memory accesses.
- Sparse, Perceiver and gated attention — architectures with linear attention cost.
- Megatron — splits one layer across several GPUs.
Google or a one-time email code. No password.