Problem
When a key metric rises or drops, the cause can be anything: an OS upgrade, a logging or pipeline error, a traffic spike, a recent code change. Pinterest's metrics have many dimensions, so a movement can hide in any of many segments, and the list of possible causes is open-ended. Checking each by hand does not scale.
Example from the article: Home feed latency rises. Is it a code change, or are users now seeing more expensive video Pins and fewer static images?
Solution in brief
Pinterest's root-cause analysis (RCA) platform uses three methods. Slice and dice finds the segments that moved, general similarity finds other metrics that moved alike, and experiment effects finds the experiments that shifted the metric. The three can be used in turn, feeding each other's findings.
Questions this case answers
- How to narrow down the root causes of a metric movement when the possible causes (segments, related metrics, launched changes) are too many to check by hand
- How to rank candidate explanations of a metric change by statistical evidence at query time, without precomputing every combination
- How to find out why a key metric suddenly rose or dropped?
- How to find which user segments explain a drop in a top-line metric?
Tools and methods
- Slice and dice — ranks segments of a multidimensional metric in a tree by a significance score
- General similarity — ranks other metrics by how closely their time series move with the target
- Experiment effects — an A/B analysis run in reverse: ranks experiments by their effect on a given metric segment
- Welch's t-test and harmonic mean p-value — test and filter the experiment effects
Google or a one-time email code. No password.