Problem
Teams that put transformers into production hit two limits: latency on the hardware they have, and memory. A speech-to-text model on CPU spends most of its time in the decoder, which generates output one element at a time. GPT-J (6 billion parameters) needs 40 GB of video memory in FP32 just to run. The speaker's motivating example is real-time chatbots that replace support agents and answer common questions.
Example from the article: a voice assistant built on ESPnet (speech-to-text) was sped up on CPU about 1.65 times, and GPT-J generation on GPU about 2 times.
Solution in brief
Three techniques, from cheapest to most expensive. Cache everything the autoregressive decoder recomputes. Convert the model to an inference runtime (OpenVINO, TensorRT) with lower precision. If that is not enough, structurally prune heads, neurons and channels using a gradient-times-weight importance score, then fine-tune.
Questions this case answers
- How to cut inference time of an autoregressive transformer model under CPU or GPU limits without a large quality drop
- How to remove parameters from a trained network so that the speedup is real and does not need special sparse-matrix software
- How to speed up transformer inference on a CPU without retraining the model
- How to fit and speed up a large autoregressive language model (GPT-J, 6B parameters) on a GPU
Tools and methods
- K/V caching — compute encoder keys and values once, keep decoder states, stop recomputing them at each generation step
- OpenVINO — CPU runtime with standard quantization
- TensorRT — GPU kernels, FP16 and INT8 for GPT-J
- Structured pruning — remove whole heads, neurons and filters so the speedup does not need sparse-matrix libraries
- Saliency criterion — gradient of the loss times the parameter, to rank what to drop
Google or a one-time email code. No password.