What is new in ML model interpretability

Dmitry Kolodezev4 Jun 2022

Problem

Teams that hand a model to business users or regulators must explain its predictions, and often cannot. A model may learn badly and nobody knows which feature is responsible. Users either distrust it or, worse, believe it because the explanation chart looks good. Explanations drawn by popular libraries are also unstable and can mislead.

Example from the article: a bank must show how its credit model works on a specific customer, but cannot show personal data.

Solution in brief

Pick the interpretation tool by data type and by scenario: debugging, verification or building the user's mental model. The talk gives a tool cheatsheet (SHAP, InterpretML, Captum, BertViz, Responsible AI Toolbox) and several practical tricks: explain with understandable features, use synthetic data, check what users actually understood.

Questions this case answers
  • How to explain predictions of a black-box model to non-technical users who need to decide how far to trust it
  • How to debug a trained model by finding the features or training examples responsible for its errors
  • How to choose an interpretation tool and scenario for tabular, image and transformer models?
  • How to explain model predictions so that business users trust and use the model?

Tools and methods

  • SHAP — feature contributions for a prediction and for the loss function
  • InterpretML — collection of interpretation methods for tabular models
  • Responsible AI Toolbox — Microsoft's open-source suite: interpretability, error analysis, fairness, dashboard
  • Captum — PyTorch library for images, including Influential Examples
  • LIME — surrogate explanation on human-readable features
  • BertViz — attention visualization for transformers
  • Mostly AI — synthetic data generator, used to show model behavior without personal data
Continue reading — it's free

Google or a one-time email code. No password.