Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations

Jinfu Leng, Rohith Menon, Nicolas Stier, Praveen Sampath, Hunter Zhang, Xiaocong Liang, Haonan Wu, Michael Fulthorp, Tao Cai, Paras M, Zeliang Chen, Shashank Tripathi, Manasi Khanuja12 Oct 2026

Problem

An ad ranking fleet is sized for peak traffic, so accelerators and CPUs sit idle at other times, in other regions, or on hardware the current models cannot use. The idle pockets belong to different teams and can be reclaimed on short notice. Using them naively can overload the capacity that serves baseline traffic and break latency and reliability targets. Leaving them idle wastes ranking quality the same fleet could buy.

Example from the article: Meta picks a few ads from millions of candidates in under a second per request, and wants extra candidates and stronger models to use idle capacity when it appears.

Solution in brief

Treat spare capacity as a shared pool and let optional work grow and shrink with it. Two levers do this: rank more candidates (Rank More Ads, RMA) and route more requests to a stronger model (Serve Better Models, SBM). Each lever has its own feedback loop, baseline traffic is isolated, and value-aware load shedding handles sudden reclaims.

Questions this case answers
  • How to turn temporary, preemptible spare compute into better ranking quality without risking latency or reliability of the guaranteed baseline traffic
  • How to choose which requests get an expensive model when its capacity fluctuates and the benefit per request is not proportional to request value
  • How to use idle inference capacity to rank more ads without overloading the capacity that serves baseline traffic
  • How to decide which requests deserve a more powerful ranking model when its capacity is limited and can disappear

Tools and methods

  • Rank More Ads (RMA) — raises candidates entering the ranking stages when CPU and accelerator surplus exist
  • Serve Better Models (SBM) — raises the share of requests scored by a stronger model when only accelerator surplus exists
  • PID controller on tail latency — sets the global candidate-volume target
  • Value-aware model router — picks requests for the stronger model
  • Load shedding — drops low-value optional work in milliseconds
Continue reading — it's free

Google or a one-time email code. No password.