Causal Methods for Feature Selection

Alena Kropacheva24 May 2025

Problem

Standard feature selection (correlation, mutual information, LASSO, XGBoost importance) rewards association with the target. It keeps features that predict well whether or not they influence the outcome, and it may not generalize to new tasks. The talk applies this to meta-learning: predicting a model's quality on a dataset from meta-features (aggregated dataset properties). Over 1000 meta-features can be computed, but few tabular datasets exist, so the meta-feature space has to shrink to train the meta-model well.

Example from the article: A model predicts whether a customer buys a product, and "viewed an ad" is a strong feature. The service showed ads only to customers already interested, so prior interest drives the purchase and the ad view only correlates.

Solution in brief

Treat each feature in turn as a treatment and estimate its effect on the outcome, with all other features as confounders. Keep the features with the largest effect (ATE-based selection), or those where the smallest change flips the model's prediction (counterfactual-based selection). On the benchmark both beat no selection and came close to filters, but ran slowest.

Questions this case answers
  • How to select features by their causal effect on the outcome rather than by their correlation with it so that the selected set stays stable when noisy features are added
  • How to shrink a large set of dataset-level meta-features when only a few hundred datasets are available to train a predictor of model quality
  • How to select features by their causal effect rather than their correlation with the target
  • How to keep a feature selection stable when noisy or uninformative features are added to the data

Tools and methods

  • CausalForestDML (EconML) — estimates the per-feature treatment effect for ATE-based selection
  • Counterfactual search — finds the minimal feature change that flips the classifier's prediction
  • Baselines — correlation, F-test, mutual information, LASSO, XGBoost importance, RFE
  • Meta-models — KNN, XGBoost, MLP
Continue reading — it's free

Google or a one-time email code. No password.