Measuring Commercial Impact at Scale at Canva

Jun Ye20 Jun 2025

Problem

Canva runs thousands of experiments a year (over 1,800 so far in 2025). A test can be significant on its own metric, such as click-through rate, and still say nothing about Monthly Active Users (MAU) or Annual Recurring Revenue (ARR). Most tests touch only one product surface, platform or region, so their uplift cannot be applied to all users. Before the team built a tool, each team sized impact in its own spreadsheet with its own definitions, and one experiment took 6+ hours. Results could not be rolled up.

Example from the article: an experiment on Presentations shows a 20% uplift in user growth. Only 10% of all users use Presentations, so the company-level uplift is 2%, not 20%.

Solution in brief

Scale every uplift measured on the exposed group by that group's share of the whole user base, then push it through the funnel to MAU and ARR with the finance team's logic. One internal app (IMPACT) holds this logic, so anyone gets a standard estimate in minutes, for running experiments and planned initiatives.

Questions this case answers
  • How to convert an uplift measured on a partly exposed population into a whole-business impact figure that is comparable across initiatives
  • How to standardize impact sizing across many parallel experiments so results roll up and are auditable
  • How to translate a local A/B test uplift into company-level MAU and ARR impact when only part of the users is exposed?
  • How to make commercial impact estimates comparable across hundreds of experiments run by different teams?

Tools and methods

  • Exposure scaling — multiplies the observed uplift by the exposed share of users to get company-level uplift
  • Funnel metrics tied to the finance model — free MAU, trial, paid, churn, retention, team expansion
  • Multiple comparison correction — across dozens of metrics
  • Streamlit on Snowflake (Snowpark, Cortex) — app UI, scheduled precomputation, LLM summaries
Continue reading — it's free

Google or a one-time email code. No password.