Company

Spotify

28 cases from Spotify (Media streaming, Adtech, Fintech), each digested from a public talk or engineering post and linked to the original. 4 were checked in an online A/B test; 6 report results in production.

Tasks: Get experiment results faster, Avoid false wins in A/B tests, Predict long-term effect from short tests, Evaluate models, Analyze user behavior and 12 more. Methods: A/B test, Power analysis / MDE, Rule-based heuristics, Sequential testing, Supervised fine-tuning, Gradient boosting and 52 more. Tools: vLLM, Qwen, Slack, Gemini, PostgreSQL, UMAP and 14 more.

Tech blog: labs.spotify.com

Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLMmedia-streaming2026-10
Why Spotify Is Not Using Bayesian A/B Testingmedia-streaming2026-09
When Can LLMs Replace Humans in A/B Tests?media-streaming2026-08
Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistantmedia-streaming2026-06
Better Experiments with LLM Evals — A funnel, not a forkmedia-streaming2026-05
Inside the Archive: The Tech Behind Your 2025 Wrapped Highlightsmedia-streaming2026-03
Our Multi-Agent Architecture for Smarter Advertisingadtech2026-02
Why we use separate tech stacks for personalization and experimentationmedia-streaming2026-01
Shuffle: Making Random Feel More Humanmedia-streaming2025-11
Beyond Winning: Spotify's Experiments with Learning Frameworkmedia-streaming2025-09
Building Confidence: A Case Study in How to Create Confidence Scores for GenAI Applicationsfintech2024-12
Are You a Dalia? How We Created Data Science Personas for Spotify’s Analytics Platformmedia-streaming2024-09
Fixed-Power Designs: It’s Not IF You Peek, It’s WHAT You Peek atmedia-streaming2024-05
Risk-Aware Product Decisions in A/B Tests with Multiple Metricsmedia-streaming2024-03
Recursive Embedding and Clusteringmedia-streaming2023-12
Exclude from Your Taste Profilemedia-streaming2023-10
How to Accurately Test Significance with Difference in Difference Modelsmedia-streaming2023-09
Experimentation at Spotify: Three Lessons for Maximizing Impact in Innovationmedia-streaming2023-08
Encouragement Designs and Instrumental Variables for A/B Testingmedia-streaming2023-08
Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0media-streaming2023-07
Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testingmedia-streaming2023-07
Experimenting with Machine Learning to Target In-App Messagingmedia-streaming2023-06
Choosing a Sequential Testing Framework — Comparisons and Discussionsmedia-streaming2023-03
What’s a “Listening Personality”?media-streaming2023-01
How We Built Infrastructure to Run User Forecasts at Spotifymedia-streaming2022-06
Introducing Natural Language Search for Podcast Episodesmedia-streaming2022-03
Comparing Quantiles at Scale in Online A/B Testingmedia-streaming2022-03
Search journey towards better experimentation practicesmedia-streaming2022-02