Problem
A product team wants an LLM-written story about each user's own behavior, shipped to everyone on a fixed launch date. At that scale a frontier model is too expensive for a billion generations. Free-form output drifts into hallucination, stereotypes or unsafe wording. A 0.1% failure rate means millions of broken stories, and nobody can read them all.
Example from the article: In Spotify's 2025 Wrapped Archive, about 350 million users each got up to five LLM-written reports on remarkable listening days: roughly 1.4 billion, generated before launch.
Solution in brief
Heuristics choose up to five remarkable days per user, so the LLM only narrates precomputed facts. A frontier model is distilled into a smaller fine-tuned model (SFT plus a small DPO step). Sampled LLM-as-a-judge evaluation and SQL/regex search over stored reports find failures, which are deleted and regenerated.
Questions this case answers
- How to generate individualized, data-grounded text for a very large user base within a fixed cost budget while keeping accuracy, tone and safety under control
- How to audit a corpus of a billion model outputs automatically and fix systematic failures after the fact
- How to generate a personalized, data-grounded narrative for every user with an LLM without hallucinations or unsafe text?
- How to pick the few most interesting days or moments out of a year of a user's event history?
Tools and methods
- Priority-ordered heuristics — pick up to five "remarkable days" per user from the yearly history.
- Two-layer prompt — system rules plus per-day data.
- Distillation with SFT — a small model learns from curated frontier outputs.
- Direct Preference Optimization (DPO) — tunes the small model on human A/B preference data.
- LLM-as-a-judge — larger models grade a random sample of reports on four dimensions.
- Column-oriented key-value store — concurrency-safe report storage.
Google or a one-time email code. No password.