Problem
An ads auction ranks ads by expected cost per click, which depends on the predicted click probability (pCTR) and the bid. The same value also charges advertisers, so the absolute level of pCTR matters, not only the order. LinkedIn replaced a heavily tuned linear GLMix model with a deep model. The first deep model produced pCTR about 40% higher than the baseline, which would overcharge advertisers and hurt their return on investment, and the usual isotonic-regression calibration did not fix it.
Example from the article: offline data showed no over-prediction for the first deep model, but once it served a small share of traffic, its own pCTR ran high on the ads it exposed.
Solution in brief
Use a three-tower model: a deep tower for full feature interaction, a wide tower of sparse ID features retrained hourly, and a shallow linear tower that reduces over-prediction. Keep position out of the deep tower. Train calibration only on logs the new model produced, ramping its traffic gradually.
Questions this case answers
- How to keep predicted probabilities calibrated when the offline training data is selected by the previous model and differs from the data the new model will see online
- How to keep memorization features fresh in a large model that can only be fully retrained daily
- How to replace a linear ads click-through-rate model with a deep model without overcharging advertisers?
- How to fix over-prediction of a deep CTR model that looks calibrated offline but not online?
Tools and methods
- Three-tower CTR model — deep MLP, wide linear and shallow linear towers summed into one sigmoid output
- Isotonic regression — post-training calibration of pCTR
- Partial warm-start retraining — hourly update of the wide tower on frozen deep and shallow outputs
- Traffic ramp for calibration data — collect the new model's own exposure data
- Apache Kafka, Apache HDFS — tracking events and offsets
Google or a one-time email code. No password.