Comparison of Interpretation Algorithms

Vadim Borisov4 Jun 2022

Problem

Teams use local explanation methods to debug models, build trust and answer regulatory questions such as the GDPR right to an explanation. The methods often disagree on the same input, and no reference explanation exists, so it is unclear which one to believe. Visual checks by experts are slow, subjective and hard to automate. Crowdsourcing them is costly.

Example from the article: a ResNet50 labels an image as a coastline. Integrated Gradients (IG), GradCAM, LIME and SHAP highlight different pixels or superpixels.

Solution in brief

Replace eyeballing with a numeric test: remove the features an explanation calls important and measure how the model output falls. Fix the out-of-distribution (OOD) artifacts of removal by retraining (ROAR) or imputing (ROAD). Add a weight-randomization sanity check and, where possible, a dataset with a known ground truth.

Questions this case answers
  • How to evaluate competing per-prediction explanation methods when no reference explanation exists and the methods disagree
  • How to choose between local explanation methods (IG, GradCAM, LIME, SHAP) when they disagree and there is no ground truth?
  • How to check automatically that a local explanation reflects what the model actually uses?
  • How to compare explanation methods without retraining the model for every method?

Tools and methods

  • Feature importance — local explanation methods (IG, GradCAM, LIME, SHAP) being compared
  • DAUC / IAUC — deletion and insertion curves, one number per method as area under the curve
  • IROF — the same idea on superpixels instead of pixels
  • ROAR / KAR — remove or keep the top features, retrain, measure test accuracy
  • ROAD — removal with neighbor-based imputation, no retraining
  • Weight-randomization sanity check — reset layers and see if the explanation turns to noise
  • BAM dataset — images with a known expected explanation
  • Amazon Mechanical Turk — crowdsourced visual comparison (costly)
Continue reading — it's free

Google or a one-time email code. No password.