Meta Andromeda: Powering the Next Generation of Ads Retrieval Through Model-System Co-Design

Haiyu Lu, Shuo Gu, Yang Yang (Tony), Jingwen Ren, Tianchen Ji, Qing Zhang, Wentian Guo, Haocheng Wu, Siqi Zeng, Yunchen Pu, Liang Wang, Chonglin Sun28 Oct 2026

Problem

The retrieval stage of an ads system picks a small set of strong candidates from a huge pool of eligible ads in milliseconds. A costlier ranking stage then scores the survivors. Ranking-level model quality was too expensive to apply at retrieval scale, and the pool kept growing as generative AI tools helped advertisers create hundreds of millions of new ads. At billions of ads per second the bottleneck moves from arithmetic to memory bandwidth and the network.

Example from the article: the 2024 Meta Andromeda system evaluated tens of millions of ads per second per GPU card. The business now needed multi-billion ads per second per card at the same latency.

Solution in brief

A mixture-of-experts (MoE) retrieval model uses several user towers and an ad-aware router that activates only the top-K experts. It gives near ranking-level quality at about 300x fewer inference FLOPs than the 2024 model. That makes room to fuse the whole pipeline into one GPU kernel and to co-locate it on one GPU server.

Questions this case answers
  • How to raise the model quality of a candidate-retrieval stage that must score the whole candidate pool within a fixed latency budget
  • How to remove memory-bandwidth and network bottlenecks in a high-throughput scoring pipeline by fusing its stages onto one accelerator
  • How to bring ranking-level model quality to the retrieval stage without a proportional rise in serving cost?
  • How to find what limits retrieval throughput once the model is on GPU: memory traffic, network or compute?

Tools and methods

  • Retrieval mixture-of-experts — user towers as experts, routed by an attention-based gate that sees the ad.
  • Fused GPU kernel — gathers embeddings, runs the interaction model and scores in one pass.
  • GPU co-location — targeting index, scoring and top-K on one GPU server.
  • INT8 quantization — a further scoring speed-up on the gating kernel.
Continue reading — it's free

Google or a one-time email code. No password.