Problem
Ads ranking teams train a conversion model offline, then run an online A/B test. Often the offline gain does not show up online, which slows model development. The cause is either a normal gap between offline and online metrics or a bug in training, serving or logging. In a readout the two look the same. Pinterest saw both kinds in its ads conversion models.
Example from the article: a treatment model degraded during an online experiment even though offline tests looked fine. The cause was feature servers returning Null values for 2–3 minutes during peak traffic.
Solution in brief
Split every unexplained gap into two cases. If the system is bug-free, treat offline AUC as a rough directional signal and check five known causes of the gap. If the gain is missing or negative, run a fixed debugging sequence over the pipeline, backed by feature monitoring, serving-versus-logging comparison and replay of logged traffic through the model.
Questions this case answers
- How to explain why an offline model-quality gain does not appear as an online business-metric gain when the two metrics measure different things
- How to detect training-serving skew caused by stale features, asynchronous logging and cache or timeout failures in a real-time prediction system
- How to find out why an offline AUC gain in an ads ranking model does not turn into an online CPA gain?
- How to tell a broken serving or training pipeline from a normal gap between offline and online metrics?
Tools and methods
- Feature validation and monitoring dashboards — track feature coverage, freshness and distribution over time
- Serving-versus-logging feature comparison — detect values that differ between scoring and the training log
- Batch inference replay — run any checkpoint on logged features to compare predictions or simulate missing features
- Apache Superset — dashboards for feature statistics
- Amazon S3 — storage of training data and model checkpoints
Google or a one-time email code. No password.