TimeSHAP: Explaining recurrent models through sequence perturbations

Joao Bento, André Cruz, Pedro Saleiro5 Jan 2023

Problem

Fraud teams score customers with recurrent neural networks (RNNs) that read the whole history of transactions, logins and account changes. Analysts and auditors need to know which past events and features pushed a score up or down, and whether the model leans on something it should not, such as age. Gradient-based explainers need model internals. Model-agnostic explainers are slow and cannot explain RNNs.

Example from the article: an RNN trained on a real fraud dataset of about 20 million events, with a median sequence length of 138.5 events, needs event-, feature- and cell-level reasons for each score.

Solution in brief

TimeSHAP applies KernelSHAP to sequences by switching whole events (columns), features (rows) or groups of cells of the input matrix to a background value. A pruning step first merges old events that barely matter into one group, which shrinks the number of coalitions. Averaging local explanations over a dataset gives global event and feature importance.

Questions this case answers
  • How to attribute the score of a black-box sequence model to individual past events and input features when sequences are too long for exact Shapley values
  • How to audit a deployed sequence model for reliance on sensitive attributes without access to its internals
  • How to explain which past events and which features drove the score of a recurrent fraud model?
  • How to explain a model that is already in production and reachable only through a prediction API?

Tools and methods

  • TimeSHAP — open-source package that explains any sequence model, locally and globally
  • KernelSHAP / SHAP — Shapley-value attribution by sampled perturbations, the base that TimeSHAP extends
  • Pruning algorithm — finds how many recent events matter and groups the rest
  • Cell grouping — keeps intersections of important events and features separate and lumps the rest
Continue reading — it's free

Google or a one-time email code. No password.