Anomaly Detection: Anomaly Scoring for Preventive Detection of Information System Failures

Alexander Algazinov25 May 2024

Problem

A team building an event-analysis service for IT monitoring (target metric: mean time to recovery, MTTR) has more than 100,000 metric time series. Each needs its own model that gives every observation an anomaly score. The series are heterogeneous (about 10 types), non-stationary and gappy, and come every 5 minutes or every hour. There are no labels, and models retrain often, so results must arrive within seconds. If people are alerted every night, they get "very angry".

Example from the article: a metric that was "dead" for a long time suddenly becomes active; the algorithm must flag that moment.

Solution in brief

Two models run on the same data. An ensemble of Isolation Forest, WindStats and Spectral Residual gives a z-score per observation, with thresholds set adaptively from the output of a simple model. A separate change point model runs ADTK LevelShift and VolatilityShift in parallel and keeps what both flag.

Questions this case answers
  • How to score anomalies in a very large set of heterogeneous, non-stationary and gappy time series without labels and with few false alerts
  • How to detect level shifts and volatility changes in sparse monitoring series fast enough for near-online retraining
  • How to score anomalies across 100k+ heterogeneous metric time series without labels and with few false positives
  • How to set anomaly thresholds automatically for each metric instead of tuning them by hand

Tools and methods

  • Isolation Forest, WindStats, Spectral Residual — three z-score detectors aggregated into the anomaly score
  • Adaptive threshold — a simple model with a low fixed threshold decides how high the main model's threshold goes
  • ADTK LevelShift / VolatilityShift — difference of medians / standard deviations between two adjacent windows
  • KATS CUSUMDetector, wavelets — change point methods that were compared or considered
  • Small hand-labeled set — used only to select the best model
Continue reading — it's free

Google or a one-time email code. No password.