Method

KV cache

Cache attention keys and values across decoding steps.

8 cases from 5 companies apply KV cache. 2 were checked in an online A/B test; 3 report results in production.

Typical tasks: Retrieve and rank from millions of items, Reduce ML cost, Recommend from what the user just did, Analyze user behavior and 8 more. Companies: Pinterest, Yandex, LY Corporation, LinkedIn, Netflix. Often combined with: LLM-as-a-judge, RAG, Knowledge distillation, LoRA fine-tuning, Synthetic data generation (LLM) and 36 more.