Problem
Fox launched Fox One, a personalized endless-scroll feed of short videos clipped from live TV, with almost no engagement data. The content type and the surface were both new, so a model trained on early engagement was barely better than random. A trending-for-everyone heuristic was too rigid, and an LLM per ranking call was too slow for the 600 ms request limit. A/B testing was also unreliable while the UI and the library kept changing.
Example from the article: in a blind audit of 100 served user-video pairs, early user watches agreed with a human relevance label in 50 cases, an LLM in 91.
Solution in brief
The LLM works only offline. It scores relevance for sampled user-video pairs, and an XGBoost student is trained on those scores. Serving runs XGBoost on eight features, mostly cosine similarities between average embeddings. Offline evaluation also uses an LLM as a simulated user.
Questions this case answers
- How to build a ranker for a new content feed when engagement data is almost empty by distilling an LLM relevance judge into a fast model
- How to evaluate a personalized feed offline when online A/B tests are unreliable
- How to build a personalization ranker when there is almost no user engagement data
- How to train a recommender on LLM-generated labels and still serve it within a tight latency budget
Tools and methods
- LLM-labeled data, distillation — the LLM gives each pair a 0–1 score and a rationale; XGBoost learns it.
- Blind audit — 100 served pairs labeled by a human, compared with watches and LLM labels.
- Average-embedding cosine features — favorites and watches compressed into two similarities.
- LLM as simulated user — scoring and head-to-head comparison of feeds.
- Constrained generation — Gemini picks clip boundaries only from known sentence and topic boundaries.
Google or a one-time email code. No password.
