Problem
In a mature product, most new features add small improvements that rarely move top-line metrics. Experiments on such features often come back inconclusive after months of planning, building and analysis. Teams then ask whether experimentation is worth the cost. The author's answer is to keep experimenting but change how the experiments are set up.
Example from the article: a full new experience with a marketing campaign was launched in some Southeast Asian markets without a prior test, and no effect on new-user acquisition could be proven.
Solution in brief
Three practices raise the chance that an experiment gives a usable answer. Start from the decision the test must inform. Target a homogeneous market or cohort with a locally adapted feature. Split a new experience into its most critical pieces and test them separately.
Questions this case answers
- How to design experiments that can still show a business-level effect when the product is mature and average effects are small
- How to choose between a full-experience test and a test of isolated components when the full build is expensive and risky
- How to keep running experiments in a mature product when most tests end inconclusive?
- How to decide whether a question needs an experiment at all?
Tools and methods
- A/B test — the experiment that checks the effect of a feature on top-line metrics
- Decision framing — two questions before any test: which decision does it inform, and why can that decision not be made with the information at hand
- Localized or cohort-limited experiment — one market and one user group, with a hypothesis built from market research
- Component-level experiment — testing an isolated piece of the experience (for example the entry point) instead of the whole feature
Google or a one-time email code. No password.