Problem
Data scientists at Square own analyses end to end. That autonomy builds ownership and creativity, but it also leads to narrow specialization and to errors nobody else notices. The cost is a wrong readout to stakeholders.
Example from the article: the author analyzed an A/B test with company-level revenue implications. After presenting to a dozen engineers he found an inner join that should have been a left join, which dropped zeros from the analysis. The conclusions happened to survive, by luck.
Solution in brief
Every piece of data science work gets a named reviewer chosen when the work starts. The reviewer runs three checks: inspect and execute the code, spot-check examples, and compare against other data sources, methods or earlier analyses. The producer stays responsible for quality, and a review takes about an hour.
Questions this case answers
- How to reduce the chance that a silent error in an individually owned analysis reaches a decision, without making a second person redo the work
- How to catch errors in an analysis before the results reach stakeholders
- How to set up peer review of data science work when analysts work alone with a lot of autonomy
- How to keep a peer review short enough that the team actually does it
Tools and methods
- Peer review of analyses — a second person checks code, logic and results before they are shared.
- Data quality checks — the reviewer tests assumptions about data structure, such as unique rows and rare NULLs.
Google or a one-time email code. No password.