Causal concept-based explanations

Jacopo Bono3 Oct 2025

Problem

Fraud analysts rely on models whose scores are hard to read. Feature attribution such as SHAP spreads importance over many low-level features, so a person cannot see why a transaction was flagged or correct the decision quickly. Analysts also think in concepts ("sudden burst of transactions", "suspicious email") and ask what-if questions, which feature attributions do not support. Concept bottleneck models give concepts but lose task accuracy and treat concepts as independent.

Example from the article: a transaction in a merchant fraud dataset is flagged as suspicious, and the analyst needs to know which concepts drove the score and how they relate.

Solution in brief

Train a second model, DiConStruct, to imitate the black-box model's scores. Inside it, a structural causal model over human-defined concepts produces both concept attributions and a causal graph for each instance. The black-box model stays untouched, and the graph supports what-if questions.

Questions this case answers
  • How to explain a black-box classifier's individual decision in terms of human-defined concepts without changing the classifier
  • How to support counterfactual what-if reasoning over explanation concepts that causally depend on each other
  • How to explain a black-box fraud model's score to an analyst in human concepts instead of dozens of features?
  • How to answer a what-if question about a fraud decision when the explanation concepts depend on each other?

Tools and methods

  • DiConStruct — post-hoc explainer: concept-based, trained by distillation, with a structural causal model of concepts
  • Knowledge distillation — trains the explainer on the black-box model's input-output pairs only
  • Structural causal model (SCM) over a concept DAG — encodes how concepts cause each other, enabling counterfactuals
  • Concept bottleneck model (CBM) — baseline that forces a layer of neurons to represent concepts
Continue reading — it's free

Google or a one-time email code. No password.