Problem
Pinterest's ads engagement ranking model is built on DCNv2. Adding layers and parameters to it does not bring a proportional metric gain. Training data is also kept only for a few months to a year. When a new model is ready for an online experiment, the data the production model was trained on may be gone, so the comparison is unfair to the new model.
Example from the article: an MMoE model with distillation from the production model was measured on RelatedPins and Search against the DCNv2 production baseline.
Solution in brief
Replace the shared bottom with several experts (DCNv2 gave the best return on cost) and one lightweight gate per task. Make serving affordable with mixed precision inference and small gate layers. During batch training, add a loss that pulls the experimental model's predictions toward the production model's predictions, so it learns from data that was deleted. Switch this loss off in incremental training.
Questions this case answers
- How to scale up a multi-task ranking model so that quality grows with size while serving cost stays bounded
- How to make a candidate model comparable to the current production model when the production model's training window is no longer available
- How to make an ads engagement ranking model bigger and get a proportional metric gain when extra layers stop helping?
- How to compare a new experimental ranking model fairly with production when the production model's old training data is already deleted?
Tools and methods
- Multi-gate Mixture-of-Experts (MMoE) — several experts shared by all tasks, one gate per task that weights the experts.
- DCNv2 experts — the expert architecture with the best metric-to-cost ratio. MLP experts, Masknet and FinalMLP were also tried.
- Mixed precision inference — cuts serving latency of the multi-expert model.
- Knowledge distillation from the production model — extra loss on prediction differences, used in batch training only.
Google or a one-time email code. No password.