Inside the Archive: The Tech Behind Your 2025 Wrapped Highlights

Yoshnee RaveendranMar 2026

Problem

A product team wants an LLM-written story about each user's own behavior, shipped to everyone on a fixed launch date. At that scale a frontier model is too expensive for a billion generations. Free-form output drifts into hallucination, stereotypes or unsafe wording. A 0.1% failure rate means millions of broken stories, and nobody can read them all.

Example from the article: In Spotify's 2025 Wrapped Archive, about 350 million users each got up to five LLM-written reports on remarkable listening days: roughly 1.4 billion, generated before launch.

Solution in brief

Heuristics choose up to five remarkable days per user, so the LLM only narrates precomputed facts. A frontier model is distilled into a smaller fine-tuned model (SFT plus a small DPO step). Sampled LLM-as-a-judge evaluation and SQL/regex search over stored reports find failures, which are deleted and regenerated.

Questions this case answers
  • How to generate individualized, data-grounded text for a very large user base within a fixed cost budget while keeping accuracy, tone and safety under control
  • How to audit a corpus of a billion model outputs automatically and fix systematic failures after the fact
  • How to generate a personalized, data-grounded narrative for every user with an LLM without hallucinations or unsafe text?
  • How to pick the few most interesting days or moments out of a year of a user's event history?

Tools and methods

  • Priority-ordered heuristics — pick up to five "remarkable days" per user from the yearly history.
  • Two-layer prompt — system rules plus per-day data.
  • Distillation with SFT — a small model learns from curated frontier outputs.
  • Direct Preference Optimization (DPO) — tunes the small model on human A/B preference data.
  • LLM-as-a-judge — larger models grade a random sample of reports on four dimensions.
  • Column-oriented key-value store — concurrency-safe report storage.
Continue reading — it's free

Google or a one-time email code. No password.