Problem
A deep ranking model for restaurant ads at DoorDash showed +2.1% AUC over the production baseline in offline evaluation, but -1.8% when it ran in shadow mode on live traffic. Online AUC was also below the production model. Any team that picks models by offline AUC can hit the same problem. The task was to find which cause explained it: feature generation disparity, data drift or serving instability.
Example from the article: the offline training set joined each impression with feature values from the previous day, assuming "yesterday's" features are what the model sees "today".
Solution in brief
Compare shadow-mode AUC with an offline replay of the same impressions. If the replay AUC matches the old offline number, the cause is the features. Then measure how stale the served features really are and how many outdated values the online store keeps, and rebuild the training data with a longer feature offset. This cut the gap from 4.3% to 0.76%.
Questions this case answers
- How to diagnose and close a gap between offline and online model quality when features are served with delay and stale values
- How to find out why a ranking model's AUC drops online compared with offline evaluation?
- How to tell feature disparity from concept drift and serving problems when online AUC is lower than offline?
- How to choose the feature join offset when building training data if online features arrive late?
Tools and methods
- Shadow-mode logging — records the features and impressions the model sees online, used as the reference for online AUC
- Offline replay — rebuilds the evaluation set from the same shadow impressions with the offline feature join
- Feature offset sweep — trains and evaluates models on joins with -1, -2, -3 and -4 day offsets
- Online feature store — serves daily-uploaded features and is the source of staleness and cached residuals
Google or a one-time email code. No password.