Problem
Meta serves ads through hundreds of user-facing ranking models, one per surface and objective (Facebook Feed, Instagram, clicks, conversions). Each model sees only its own slice of data, while meaningful signals such as clicks and conversions are very sparse among billions of daily interactions. Making every model larger is too costly, because serving models must stay small enough for tight latency.
Example from the article: vertical models suffer from stale supervision, because the large model's training and evaluation lag behind their fresh data, and its objective differs from theirs.
Solution in brief
Train one foundation model (GEM) on all ads and organic data with an architecture that scales efficiently. Then pass its knowledge to the vertical models, directly or through domain-specific foundation models, using distillation with a Student Adapter, representation learning and parameter sharing. Meta reports 2x the effectiveness of standard distillation.
Questions this case answers
- How to transfer the quality of one very large ranking model into many small latency-bound models that serve different surfaces and objectives
- How to correct stale or misaligned teacher signals when distilling into a student that is trained on a stream of fresh labeled data
- How to build one large central model for ads ranking and pass its quality on to hundreds of smaller production models?
- How to keep distillation useful when the teacher's training lags behind fresh ground truth and its objective differs from the student's?
Tools and methods
- Wukong factorization machines — user-ad attribute interactions.
- Pyramid-parallel sequence modeling and InterFormer — long behavior histories without compressing them.
- Knowledge distillation with Student Adapter — aligns teacher outputs with recent ground truth.
- Representation learning and parameter sharing — move teacher features and components into small models.
- PyTorch 2.0 compilation, FP8 activations, custom jagged-sequence GPU kernel — make training feasible.
Google or a one-time email code. No password.