Feature Investigation: Automatically Detect Drift in your Data Over Time

Ricardo Moreira, Marco O. P. Sampaio, Hugo Ferreira3 Aug 2022

Problem

Suppose a fraud model has run in production for several weeks. Last week its alert rate (the share of events flagged as risky) rose noticeably, and the cause is unclear. Usually a few input features behave differently from the training period, but the model may use hundreds. Checking each by hand takes days and delays the fix. A sliding-window comparison needs every value in the window stored, which is heavy in a live system.

Example from the article: a European bank compared the features of its fraud model between the training period and production data and found both broken features and real drift after one day of running the system.

Solution in brief

Keep one exponentially decaying histogram per feature instead of a sliding window, so memory stays constant. Compare it with the reference histogram by a divergence measure and convert the divergence into a p-value using divergences seen inside the reference period. Combine all features with a multiple-testing correction, raise an alarm when the combined test fails, and rank features by their corrected signal.

Questions this case answers
  • How to detect and rank drifting inputs of a deployed model across hundreds of features when storing a full sliding window is too expensive
  • How to raise alarms on many simultaneously monitored signals while controlling the overall false-alarm rate and explaining the alarm by feature
  • How to find which input features caused a jump in a production model's alert rate without checking hundreds of features by hand?
  • How to monitor the distribution of hundreds of features over time with constant memory and low compute?

Tools and methods

  • Moving histograms — exponentially discounted histograms that give a window-like view of a feature in constant memory
  • Divergence measures (Kolmogorov-Smirnov, Wasserstein, Jensen-Shannon) — distance between current and reference histograms
  • Holm-Bonferroni correction — one global false-alarm level across all monitored features, plus a severity ranking
Continue reading — it's free

Google or a one-time email code. No password.