Problem
The LINE OpenChat team has improved its daily-retrained personalized recommender for four years. Early upgrades gained the same offline and online. Later, offline winners started to lose online. Many changes had piled up over offline iterations, so nobody could tell which one hurt, and the cycle had to be repeated. Offline labels cannot capture what happens once a room is shown.
Example from the article: the model predicts a user's preference correctly and the user applies to a recommended room, but the room owner does not approve the request, so no join is recorded.
Solution in brief
Keep the offline test, but check each change in a small online "ad hoc" A/B test with a 9:1 split. Keep the change if online metrics hold, roll it back if not, and run one normal 5:5 test on the final model. A 10% test group needs about three times the total sample of a 5:5 test, not five times.
Questions this case answers
- How to attribute an online regression to one of many accumulated model changes when offline metrics keep improving
- How to test a risky model change on a small traffic share while keeping the experiment duration reasonable
- How to validate a recommendation model change when offline gains no longer show up in the online A/B test?
- How to find which of several accumulated model changes hurt online performance?
Tools and methods
- Offline test — true positives in top-k lists, on joins made without the model
- Ad hoc A/B test (9:1) — online check of one change on a small group
- Power analysis with statsmodels — sample size for a given group B share
- MLflow — logging offline parameters and metrics to compare models
- Two-stage recommender with FFM ranking — the model being improved
Google or a one-time email code. No password.
