Problem
Indeed's Ranking Models teams build models that score job-to-candidate matches for ranking, bid-scaling and score-thresholding. Each team used its own rule for rolling out a model after an online experiment: some shipped on model performance metrics, others on product metrics. Each team had a valid reason for its choice, but the decisions were inconsistent. The post compares the metric types and proposes one shared framework.
Example from the article: a team defined utility as total expected positive application outcomes for invite-to-apply emails, while the product goal was more hires, a subset of those outcomes. This mismatch makes it harder to turn gains on the model target into the product goal.
Solution in brief
A model's improvement counts as product impact only after you check how the model is used downstream. Ship on product metrics (or overall ranking metrics such as nDCG@k), and keep Normalized Entropy as a guardrail so predictive power and calibration do not degrade. Monitor ROC-AUC for debugging and, for bid-scaling models, Avg-Pred-to-Avg-Label.
Questions this case answers
- How to choose the primary success metric of an online experiment on a model whose output is only one input to a larger decision pipeline
- How to protect model quality with guardrail metrics while shipping on product impact when model targets and business goals are misaligned
- How to decide whether a new ranking, bid-scaling or thresholding model should ship: on model metrics or on product metrics?
- How to set guardrails on model quality (predictive power and calibration) during an online experiment?
Tools and methods
- Normalized Entropy (NE) — guardrail on prediction quality and calibration.
- ROC-AUC — monitoring metric for discrimination, used in deep dives.
- nDCG@k and labeling-based relevance — overall ranking performance on the re-ranked list.
- Avg-Pred-to-Avg-Label, ACE/ECE — calibration checks.
- Product metrics — engagement, outcomes, relevance, revenue; used for the ship decision.
Google or a one-time email code. No password.