Why You Need an Experimentation Template

Kasia Rachuta23 Feb 2023

Problem

Product teams ask data scientists to launch A/B tests before anyone has written down the problem the feature solves or the result that would justify shipping it. When a test ends flat, some stakeholders still lean toward launching, while the data scientist sees no positive impact. The disagreement surfaces at the end of the test and can turn into a long argument.

Example from the article: after a test with no lift in any metric, a PM may still want to ramp the experience to 100% because user-research feedback was positive.

Solution in brief

Block data science teams fill in a shared A/B test template before launch. Two sections do most of the work. The Overview section makes the team restate the hypothesis as a business problem and to fix the launch criteria in advance. The Learnings section records one or two follow-up analyses to run after the test.

Questions this case answers
  • How to align decision makers on the problem and the ship criteria before an experiment starts, so a neutral result does not trigger a dispute
  • How to standardize experiment write-ups across teams and capture follow-up questions for the next experiment
  • How to make stakeholders agree on the business problem and the launch decision before an A/B test starts?
  • How to avoid a dispute between data scientists and PMs when a test ends with a neutral result?

Tools and methods

  • Experimentation template — a shared document filled in before launch, same structure for every test
  • Business problem statement — the problem behind the hypothesis
  • Pre-agreed launch criteria — the ship rule, set before the test
  • Learnings section — follow-up analyses to run after the test
Continue reading — it's free

Google or a one-time email code. No password.