Problem
Search teams judge ranking experiments by whole-page relevance, and relevance usually comes from human raters. Raters are slow and every label costs money, so the query sample stays small. At Pinterest this left minimum detectable effects (MDEs) of 1.3%-1.5%. The team saw only large topline moves, not small ones or effects inside a query segment such as a topic or a popularity band.
Example from the article: Pinterest Search tracks whole-page relevance in online A/B tests of new ranking models, and a 1.3%-1.5% MDE was too coarse for them.
Solution in brief
Fine-tune a multilingual encoder on human relevance labels, and let it label the top results of every sampled query in both experiment arms. Cheap labels allow a much larger, stratified, paired query sample. Stratification cuts variance, which brings the MDE to 0.25% or below.
Questions this case answers
- How to measure a ranking-quality metric in an online experiment when human labels are too scarce and costly to power the test?
- How to cut the minimum detectable effect of a sample-based quality metric using model-generated labels and stratified sampling?
- How to validate that model-generated labels give unbiased paired differences between experiment arms?
- How to measure search relevance in an A/B test when human labels are too scarce and expensive?
Tools and methods
- XLM-RoBERTa-large — cross-encoder relevance model over a (query, item text) pair, five-class output
- BLIP captions and item text features — text representation of each item, including image captions
- Stratified query sampling with optimal allocation — lower variance and per-segment effects
- sDCG@K — query-level relevance metric computed from the model labels
- Paired t-test and Benjamini-Hochberg procedure — topline readout and false discovery control over segments
Google or a one-time email code. No password.