Problem
An ad ranking fleet is sized for peak traffic, so accelerators and CPUs sit idle at other times, in other regions, or on hardware the current models cannot use. The idle pockets belong to different teams and can be reclaimed on short notice. Using them naively can overload the capacity that serves baseline traffic and break latency and reliability targets. Leaving them idle wastes ranking quality the same fleet could buy.
Example from the article: Meta picks a few ads from millions of candidates in under a second per request, and wants extra candidates and stronger models to use idle capacity when it appears.
Solution in brief
Treat spare capacity as a shared pool and let optional work grow and shrink with it. Two levers do this: rank more candidates (Rank More Ads, RMA) and route more requests to a stronger model (Serve Better Models, SBM). Each lever has its own feedback loop, baseline traffic is isolated, and value-aware load shedding handles sudden reclaims.
Questions this case answers
- How to turn temporary, preemptible spare compute into better ranking quality without risking latency or reliability of the guaranteed baseline traffic
- How to choose which requests get an expensive model when its capacity fluctuates and the benefit per request is not proportional to request value
- How to use idle inference capacity to rank more ads without overloading the capacity that serves baseline traffic
- How to decide which requests deserve a more powerful ranking model when its capacity is limited and can disappear
Tools and methods
- Rank More Ads (RMA) — raises candidates entering the ranking stages when CPU and accelerator surplus exist
- Serve Better Models (SBM) — raises the share of requests scored by a stronger model when only accelerator surplus exists
- PID controller on tail latency — sets the global candidate-volume target
- Value-aware model router — picks requests for the stronger model
- Load shedding — drops low-value optional work in milliseconds
Google or a one-time email code. No password.