Normalized Entropy or Apply Rate? Evaluation Metrics for Online Modeling Experiments

Megan Chen11 Nov 2025

Problem

Indeed's Ranking Models teams build models that score job-to-candidate matches for ranking, bid-scaling and score-thresholding. Each team used its own rule for rolling out a model after an online experiment: some shipped on model performance metrics, others on product metrics. Each team had a valid reason for its choice, but the decisions were inconsistent. The post compares the metric types and proposes one shared framework.

Example from the article: a team defined utility as total expected positive application outcomes for invite-to-apply emails, while the product goal was more hires, a subset of those outcomes. This mismatch makes it harder to turn gains on the model target into the product goal.

Solution in brief

A model's improvement counts as product impact only after you check how the model is used downstream. Ship on product metrics (or overall ranking metrics such as nDCG@k), and keep Normalized Entropy as a guardrail so predictive power and calibration do not degrade. Monitor ROC-AUC for debugging and, for bid-scaling models, Avg-Pred-to-Avg-Label.

Questions this case answers
  • How to choose the primary success metric of an online experiment on a model whose output is only one input to a larger decision pipeline
  • How to protect model quality with guardrail metrics while shipping on product impact when model targets and business goals are misaligned
  • How to decide whether a new ranking, bid-scaling or thresholding model should ship: on model metrics or on product metrics?
  • How to set guardrails on model quality (predictive power and calibration) during an online experiment?

Tools and methods

  • Normalized Entropy (NE) — guardrail on prediction quality and calibration.
  • ROC-AUC — monitoring metric for discrimination, used in deep dives.
  • nDCG@k and labeling-based relevance — overall ranking performance on the re-ranked list.
  • Avg-Pred-to-Avg-Label, ACE/ECE — calibration checks.
  • Product metrics — engagement, outcomes, relevance, revenue; used for the ship decision.
Continue reading — it's free

Google or a one-time email code. No password.