Problem
Netflix designs several promotional artwork assets per title and wants to know which visual components make a viewer choose and watch it. A/B tests that change one aspect at a time, usually inside one genre, need manual labeling or hand-made variants. They cover few titles and miss other factors such as figure orientation, background color or facial expression.
Example from the article: does the presence of a face in an artwork causally improve its performance? two Unbreakable Kimmy Schmidt assets differ in face, background, text placement and font size, so their raw gap says little.
Solution in brief
Use the artwork assets already served by the personalization system and their logged engagement. Label image components with computer vision, find candidate effects by correlation, then estimate each candidate's causal effect with double machine learning, which controls for confounders. Follow with moderation, mediation and sensitivity analyses.
Questions this case answers
- How to estimate the causal effect of a binary item attribute on user engagement from observational logs when one-attribute-at-a-time experiments do not scale
- How to test whether an estimated effect holds when unmeasured confounders may exist
- How to find out which visual elements of promotional artwork cause people to pick a title, without a separate A/B test for every element?
- How to estimate the causal effect of a binary content feature on engagement from logged recommendation data?
Tools and methods
- Double machine learning — estimates the effect of a binary component on take rate from observational data.
- Causal forest — fitted on the residuals to get the average effect (ATE) and effects by genre and country (CATE).
- XGBoost — classifier for the propensity model and regressor for the outcome model.
- Mediation and moderation analysis — shows when and through which variable an effect works.
- E-value and other sensitivity analyses — stress-test the no-unmeasured-confounders assumption.
Google or a one-time email code. No password.