Method
LLM inference optimization
KV cache, batching, speculative decoding, parallelism for serving speed.
21 cases from 13 companies apply LLM inference optimization. 3 were checked in an online A/B test; 11 report results in production. Includes variants: KV cache.
Typical tasks: Reduce ML cost, Retrieve and rank from millions of items, Recommend from what the user just did, Catch fraud and abuse and 10 more. Companies: Pinterest, Netflix, DoorDash, Yandex, Avito, LinkedIn and 7 more. Often combined with: KV cache, Knowledge distillation, LoRA fine-tuning, LLM-as-a-judge, Quantization and 58 more.




