Problem
Canva's data scientists ran A/B test analysis with a Python package and YAML configs, and shared results as screenshots. As the company grew, this stopped working. Only people with warehouse credentials could run ad hoc analyses, engineers and product managers could not explore results themselves, and a new data scientist needed months to get comfortable. Results went stale and trust in the platform eroded.
Example from the article: an experiment launched on both canva.com and canva.cn had to be read in two separate results apps.
Solution in brief
Canva moved experiment analysis from a data-scientist-owned script to an engineering-owned service inside its feature flagging product. Statistics stay in a Python library that data scientists edit. Metrics are built by point and click instead of SQL snippets, so each one is consistent and statistically valid.
Questions this case answers
- How to make experiment analysis self-serve for non-analysts while keeping metric definitions and statistics consistent across teams
- How to evolve an analyst-owned analysis script into a maintained, trusted platform when the number of analyses outgrows manual upkeep
- How to let engineers and product managers read A/B test results without waiting for a data scientist?
- How to keep metric definitions and statistical calculations identical across all experiments in a company?
Tools and methods
- Shared metrics and automatic guardrail metrics — one definition and one way to compute results for all experiments, with company-wide goals (active users, revenue) tracked by default
- Python statistics library on AWS Lambda — keeps the statistical models in Python for data scientists and exposes an API for ad hoc analyses
- Separate metric service — stores metrics and analysis components behind a point-and-click interface so they can be reused by other tools
- Education program — teaches statistical models, decision-making and debugging so people self-serve
Google or a one-time email code. No password.