Measuring downstream impact on social networks by using an attribution framework

Qiannan Yin, Derek Koh, Jenny Wu21 Jul 2022

Problem

In a social network a member's action brings other members to the product. A post leads to likes, comments and shares by others. A standard A/B test randomizes the actors and measures only their own behavior, so these recipient-side sessions are missed. Methods from the literature need special treatment assignment and are costly to build. So a change can look neutral for the actors while it helps or hurts the rest of the network.

Example from the article: a PYMK ("People You May Know") ranking change reduced invitations sent by 1% with neutral sender-side sessions, while downstream sessions rose by 1%.

Solution in brief

CAMEL (Customizable Allocation of Modular Empirical Likelihood) attributes each recipient session to the actor whose notification caused it, using notification logs. Ambiguous badge notifications are split equally among those sent within one day. The attributed sessions become an ordinary user-level metric in a regular A/B test, winsorized at the 99.9th percentile.

Questions this case answers
  • How to estimate the effect of a change on other users who react to the treated users' actions, when only the actors are randomized
  • How to attribute an outcome of one user to several possible causing events when the causal link is not logged
  • How to keep a heavy-tailed spillover metric from making experiment readouts unreliable
  • How to measure the downstream sessions that one member's action causes for other members in a regular A/B test?

Tools and methods

  • CAMEL downstream attribution — maps recipient sessions back to actors through notification logs
  • Equal split of badge notifications — fractional credit for sessions with several candidate causes
  • Winsorization at the 99.9th percentile — limits influencer outliers in the metric
  • A/B test with a t-test on the actor side — reads the downstream metric like any other metric
  • Dual randomization experiment — actor and recipient randomization, used only to validate CAMEL
Continue reading — it's free

Google or a one-time email code. No password.