Problem
Etsy's experimentation platform, Catapult, showed results broken down by 13 segmentations, and they were hard-coded in the back-end. A new segmentation needed an engineer to change the data pipeline code. All segmentations ran in one daily batch job that waited for the upstream data of every segmentation, so one late table delayed all of them. Teams fell back on analysts' ad-hoc SQL outside the platform, which was slow and hard to reproduce.
Example from the article: in a free-ice-cream button test, overall engagement looks neutral, but ice cream lovers may buy more and others less, so the effects cancel.
Solution in brief
Each segmentation is a configuration object: a SQL query that returns a segment value per experimental unit, plus the join columns. A creation tool validates the SQL, then deploys it through an auto-merged pull request. The daily pipeline creates one independent task group per active segmentation (dynamic task generation), so new segmentations need no code change and one failure does not block the others.
Questions this case answers
- How to let analysts add new breakdown dimensions to a centralized results pipeline without engineering changes, while keeping the pipeline safe and on schedule
- How to let experimenters slice A/B test results by their own segments without a pipeline code change?
- How to validate user-written SQL before it runs in a production experiment-analysis pipeline?
- How to stop one delayed segmentation from blocking the daily results of all the others?
Tools and methods
- Segmentation as SQL config — query plus join columns, reusable across experiments through a shared library
- Validation and deployment workflows — schema and join-key checks, sample results, auto-merged pull request
- Dynamic task generation with sensors — one parallel task group per active segmentation
- A/B test results by segment — segmented results written to the Experiment Results table
Google or a one-time email code. No password.