An Effective Approach to Quality Control of Anti-Fraud Models

Yulian Gilyazev24 May 2025

Problem

A fraud or moderation model degrades after release: fraudsters invent new tricks, and the object stream drifts, so the threshold stops being optimal. To notice this, a team labels a sample daily or weekly and tracks precision and recall. Fraud is rare, so a random sample has few positives and noisy recall and F1. Labeling more objects above the threshold adds positives but distorts the sample, so metrics computed on it directly are wrong.

Example from the article: on the IEEE-CIS Fraud Detection data, a plain random ("naive") sample gives a precision variance of 17.48, against 0.36 to 0.43 for samples oversampled above the threshold. Their recall variance (13.91 and 19.20) stays as high as the naive 18.73.

Solution in brief

Draw the labeled sample from an analytically optimal distribution q over model scores, which oversamples the zone above the threshold. Then multiply each object's correctness by the weight pm/q (importance sampling) to get unbiased metric estimates. On two imbalanced datasets this cut the variance of recall and F1 up to 2 times, which the author equates to up to 1.5 times fewer labels.

Questions this case answers
  • How to estimate precision and recall of a classifier from a labeled sample when the positive class is rare and the sample must be drawn non-uniformly
  • How to choose the labeling sample distribution so that quality estimates have the lowest variance per label spent
  • How to estimate precision and recall of a fraud model from a small labeled sample under class imbalance
  • How to choose which objects to send for labeling so that metric estimates have minimal variance

Tools and methods

  • Importance sampling — reweights a resampled labeled set back to the original score distribution.
  • Optimal q per metric — minimizes the analytic variance of the estimate.
  • Monte Carlo variance estimate — repeated sampling to compare methods.
  • Crowdsourcing or an LLM — labeling step of the quality loop.
Continue reading — it's free

Google or a one-time email code. No password.