Problem
Key metrics such as daily active users (DAU) or monthly active users (MAU) are computed by code. That code can count something other than what the business meant, and it drifts as products and app versions change. When such metrics are shared publicly, customers, partners and investors rely on them. Controlling every metric is too costly, so the company must choose which to protect.
Example from the article: an AR app counts every photo with an added unicorn. Once third-party apps embed the unicorn technology, the same event fires there too, and the metric counts activity the owner never intended to measure.
Solution in brief
Snap applies the Social Technology Operating Measures Principles (STOMP), guidelines from an industry consortium, to a short list of critical metrics chosen by risk. Each is defended by three lines: the owning teams (tests, gated releases, alerts), a Metrics Governance team (anomaly checks, two-phase certification), and Internal Audit plus a cross-functional disclosure committee.
Questions this case answers
- How to keep a business-critical metric trustworthy when its definition, instrumentation and product context keep changing
- How to set up layered ownership, independent review and audit for the few metrics whose errors would be costly
- How to make sure a headline metric such as DAU still measures what the business intended?
- How to keep an instrumented metric from silently breaking when the app or the product changes?
Tools and methods
- STOMP principles — five areas of control for a metric
- Automated QA of instrumentation — tests event logic before each release
- Staged release (canary rollout) — a new app version goes to a small share of users first
- Loss, delay and duplication thresholds with alerts — detect breaches of tolerance
- Two-phase metric certification — definition review, then data testing
- Metrics Disclosure Committee — reviews new key metrics, definition changes and incidents
Google or a one-time email code. No password.