Problem
Large recommenders at LinkedIn (feed, notifications, people and job recommendations) optimize several goals at once, and the goals conflict. In notifications, sending more messages raises the number of sessions but lowers click-through rate (CTR) because the average notification gets weaker. Separate models predict each goal, and a linear blend with tunable weights gives the final score. Grid or random search for good weights needs many A/B tests. Each needs a large sample, traffic is shared with many other tests, and a classic A/B test cannot move traffic away from bad variants mid-run.
Example from the article: LinkedIn notifications must maximize sessions while keeping Impressed CTR and Send Volume at control-model levels.
Solution in brief
Treat each online metric as an unknown noisy function of the blend parameters and fit a Gaussian process to it. Turn the constraints into a single utility, and use Thompson sampling to pick which parameter values get traffic next. The output is a probability distribution over candidate parameter vectors, which maps directly onto splitting the treatment group into A/B arms.
Questions this case answers
- How to choose the weights of a linear blend of several predictive models so that one metric is maximized while other metrics stay above control levels, when each evaluation is a noisy online experiment with limited traffic
- How to allocate experiment traffic adaptively across candidate parameter settings so that poor settings stop receiving traffic early
- How to tune the weights that combine several recommender models when one primary metric must grow and guardrail metrics must not drop?
- How to search model-combination parameters with online A/B tests when traffic is limited and many tests run at once?
Tools and methods
- Bayesian optimization — sequential search of an expensive black-box objective with a surrogate model
- Gaussian process regression — models each noisy online metric and its uncertainty
- Thompson sampling — chooses the next candidates as a probability distribution
- Apache Spark, Apache Hadoop, Apache Kafka — offline workflow, workflow template and event stream of the library
Google or a one-time email code. No password.