Training Foundation Improvements for Closeup Recommendation Ranker

Fan Jiang, Liyao Lu, Laksh Bhasin, Chen Yang, Shivin Thukral, Travis Ebesu, Kent Jiang, Yan Sun, Huizhong Duan26 Sep 2023

Problem

The Pinterest closeup surface, a feed of recommendations shown when a user closes up on a Pin, produces a very large number of impressions and engagements. Storing everything is costly, yet the ranker still needs fresh, representative training data. A model trained once drifts as feature distributions, seasons and user trends change. Without a framework, the team must maintain the data pipeline, training flow, deployment and holdout experiment by hand.

Example from the article: the closeup ranking model, which uses knowledge distillation from its own previous version, had to be refreshed regularly while its calibrated serving model and uncalibrated teacher stayed in sync.

Solution in brief

Log training data from both backend and frontend so that only impressed candidates are stored. Generate training sets with a configurable sampling job and keep a small randomized traffic slice for evaluation. Retrain on a fixed cadence (weekly here) through an automated workflow with offline and online validation, and track the effect with a holdout experiment that is re-versioned after every deploy.

Questions this case answers
  • How to build training data for a ranking model on a surface with huge impression volume while keeping storage and compute affordable
  • How to retrain and redeploy a ranking model on a schedule, with validation gates, when the model also depends on its previous version as a teacher
  • How to log training data for a high-traffic ranker without storing every impression?
  • How to choose which impressed candidates to keep when generating training data?

Tools and methods

  • Hybrid logging — backend caches, frontend logs impressed Pins with features.
  • Randomized traffic slice — all served candidates logged for replay and evaluation.
  • Configurable sampler (Apache Spark) — sampled training data from a sampling config.
  • Auto-Retraining Framework (Apache Airflow, MLflow, Spinnaker) — scheduled train, validate, register, deploy.
  • Knowledge distillation and calibration — previous model as teacher.
  • Holdout experiment — production model against its previous version.
Continue reading — it's free

Google or a one-time email code. No password.