Problem
Teams that build models on sensor and process data, for example for equipment maintenance, get time series with distorted values. Sensors fail, signals get swapped in the database, a sensor replacement, recalibration or timezone change makes all later data unreliable, and a changed technological process makes old training data obsolete. Models trained on such data break or quietly give wrong results. Often there are no failure examples at all to test a detector against.
Example from the article: a turbogenerator at a nuclear power plant never breaks down, so history has no anomalous class and the detector's quality cannot be measured on past data.
Solution in brief
Go through five preprocessing stages in order: cleaning, feature transformation, feature selection, feature generation, augmentation. Most data problems can be detected as anomalies, so one anomaly detector can serve as a cleaning step. The detector is a normal-mode model: forecast one point ahead on normal data and flag points where the error exceeds a control limit.
Questions this case answers
- How to clean a sequence of sensor measurements with swapped signals, wrong timestamps and recalibration jumps before modeling it
- How to detect anomalous behaviour in a measurement sequence without labeled anomalies by forecasting normal behaviour and flagging large forecast errors
- How to clean sensor time-series data before building a model
- How to detect outliers and change points in a time series
Tools and methods
- Normal-mode model with control limit — one-step-ahead forecast on normal data, flag points by forecast error
- Change point detection — find when the process state changed, not which points are bad
- Five-stage preprocessing — cleaning, transformation, selection, generation, augmentation
- Libraries — Prophet, sktime, Darts, Merlion, tsfresh, ruptures, PyOD, tsaug
Google or a one-time email code. No password.