Measure Less to Learn More: Using Fewer, Higher-quality Metrics to Capture What Matters

Jake Mainwaring24 Apr 2026

Problem

Discord adds a Default Metric List to every experiment automatically. Teams kept adding the metrics they cared about and rarely removed any, so the list grew to about 50. More metrics raise compute costs and make readouts harder to scan. They also create a trade-off. Raw p-values give too many false positives, while a multiple-comparisons correction hides real changes: recall, the share of true effects caught, falls.

Example from the article: with 100 metrics and a 5% p-value threshold, about 5 metrics look significant by chance alone.

Solution in brief

Keep the Benjamini-Hochberg (BH) correction and shrink the metric pool instead. Simulations show that recall under BH drops as metrics are added. Treatment-effect correlations and Principal Component Analysis (PCA) on past experiments point to redundant metrics. The owning teams decide which to cut.

Questions this case answers
  • How to keep power to detect real effects when a growing set of metrics forces ever stricter multiple-comparison correction
  • How to identify redundant metrics in a large metric set so it can be pruned without losing coverage of distinct behaviors
  • How to cut a default experiment metric list that has grown to dozens of metrics without losing the ability to detect real effects?
  • How to find which experiment metrics are redundant and can be merged or removed?

Tools and methods

  • Benjamini-Hochberg correction — controls the false discovery rate at 5% across all metrics of an experiment
  • Monte Carlo simulation — shows how false alarms and recall change with the number of metrics
  • Treatment-effect correlation matrix — finds pairs of metrics that move together across experiments
  • PCA — shows how many independent directions the whole metric set contains
Continue reading — it's free

Google or a one-time email code. No password.