Ranking or Calibration? How to Measure the Quality of Advertising CTR Models

Alexander Ledovsky25 May 2024

Problem

A team building CTR models for an ad auction needs offline metrics to pick A/B test candidates. At Avito one such test takes about 2 weeks on roughly 20% of traffic, so offline selection is unavoidable. The auction ranks promoted listings by bid×CTR, which is expected revenue per impression only if CTR is a true click probability. The usual metrics (ROC-AUC on the whole sample, NDCG, precision@k, RMSE) either ignore calibration or evaluate a ranking that differs from the one the auction uses.

Example from the article: contacts stayed flat in an A/B test while auction revenue grew, because the model became better calibrated.

Solution in brief

Treat the task as classification where CTR must also be calibrated. Measure ranking with ROC-AUC computed per query and weighted by clicks, and measure calibration separately on the first position. Weight metrics by category revenue so that they track money rather than clicks. Avito reports about 5% more auction revenue in half a year after switching to this metric set.

Questions this case answers
  • How to choose an offline metric for a click-probability model that is used to rank items by bid times predicted probability, so that offline gains predict revenue gains
  • How to check calibration and within-query ranking of predicted probabilities when the decision depends on the product of value and probability
  • How to choose offline metrics for an ad CTR model when the auction ranks by bid times CTR
  • How to check that predicted CTR is a real click probability (calibration)

Tools and methods

  • Stratified (group) AUC — ROC-AUC per query, weighted by clicks, with CTR multiplied by a visibility model.
  • Simple Calibration, ECE — predicted versus actual clicks, overall and by score bucket.
  • Relative Information Gain — logloss normalized by the entropy of the mean CTR.
  • Revenue-weighted metrics — a metric per category, averaged with revenue weights.
Continue reading — it's free

Google or a one-time email code. No password.