Metrics and Loss Functions in Machine Learning System Design

Valeriy Babushkin27 May 2023

Problem

A team trains a model with a default loss and reports metrics that match neither the loss nor the business goal. Swapping the loss alone, with the same features, architecture and data, can change predictions and quality drastically. For classifiers on imbalanced data, precision shifts when the class ratio shifts, though the model has not changed. The speaker says the largest metric gains and failures often come from this basic layer.

Example from the article: a fraud model sees a negative-to-fraud ratio of 1 to 100 on a bad day, 1 to 1000 on a normal day and 1 to 10,000 on a quiet one. Precision for the same model then swings between 0.9947 and 0.65.

Solution in brief

Choose the loss from the cost of errors: MSE predicts the mean and punishes outliers, MAE predicts the median. Then pick metrics that follow the business goal. For imbalanced, shifting classes use recall, specificity or ROC AUC, which do not depend on the class ratio, and record the reasoning in a design document.

Questions this case answers
  • How to choose a loss function from the asymmetric cost of errors when the target has outliers, so that the model optimizes the decision rather than the mean
  • How to pick classifier evaluation metrics that stay stable when the class ratio shifts over time, so the same model does not look better or worse by chance
  • How to choose a loss function that matches the business goal
  • How to pick evaluation metrics for a fraud model when the class ratio keeps changing

Tools and methods

  • MSE vs MAE — mean versus median prediction
  • Recall and specificity — each depends on one class only
  • ROC AUC — chance a class 1 object ranks above a class 0 object
  • Recall at fixed specificity — the speaker's practical fraud metric
  • Bootstrap — confidence intervals
  • Design document — records metric and loss decisions
Continue reading — it's free

Google or a one-time email code. No password.