Problem
Instagram ranks Feed, Stories, Reels, comments, notifications and tag suggestions. Each surface has several funnel layers, and constant experimentation in every layer grew the production fleet past 1000 models. Teams kept separate lists of their models. Launching one meant days of manual load testing and traffic shifting. Nobody shared a definition of prediction quality. Degraded ranking went unnoticed, and on-call engineers could not tell which model was safe to switch off.
Example from the article: a user who loves golf but dislikes cooking content gets a response that arrives on time yet is irrelevant. A request success-rate SLO counts it as healthy.
Solution in brief
Instagram built a model registry that records each model's business function and criticality tier. It automated launches with offline load tests on recorded traffic and per-team capacity quotas. Prediction health was reduced to one binary "model stability" indicator based on calibration and normalized entropy thresholds stored in the registry.
Questions this case answers
- How to maintain a single source of truth about importance and business function for a large fleet of production ML models owned by many teams
- How to detect degraded prediction quality in many ranking models with one generic alert when request success rate looks healthy
- How to cut the time and risk of launching a new model when capacity is limited and each launch needs its own load test
- How to keep track of which ranking models run in production and how important each one is when hundreds of them are in flight?
Tools and methods
- Model registry — ledger of model types and instances with importance and on-call; drives alerting and launches
- Criticality tiers (TIER0 to TIER4) — the company service importance scale reused for models
- Offline performance evaluation — replay of recorded traffic against a clone to find the maximum stable load
- Automated launch platform — capacity estimate, cost calculation and stepwise traffic shifting
- Model stability — binary health indicator from calibration and normalized entropy per prediction
Google or a one-time email code. No password.