SHAP Plots: The Crystal Ball for UI Test Ideas

Vicky Zhang28 Apr 2022

Problem

A product team that owns a complex dashboard has many UI elements and needs to know which part of the UI drives a target metric, so it can choose what to A/B test. Indeed's Interview team has one goal: more interviews on the platform. Here it wanted UI test ideas that raise the invitations employers send from the dashboard.

Example from the article: to find such ideas, the team analyzed employer clicks on the dashboard and predicted interview invitations from them.

Solution in brief

Turn each UI element into a feature (how many times the employer clicked it), train a CatBoost classifier to predict whether the target button was clicked, and read a SHAP plot. Features that push the prediction up or down become positive and negative predictors, and each one is a test hypothesis. The model gives only associations, so every hypothesis goes to an A/B test.

Questions this case answers
  • How to generate a ranked batch of experiment hypotheses from logged interaction data when each interface element could be a candidate lever and intuition does not scale
  • How to generate many UI A/B test ideas at once instead of brainstorming them one by one?
  • How to find which UI elements are linked to a target action such as sending an invitation?
  • How to turn a feature-importance plot into concrete test hypotheses?

Tools and methods

  • CatBoost — tree-based classifier that captures interactions between features.
  • SHAP plot — shows each feature's direction and strength of influence on the prediction.
  • A/B test — verifies each hypothesis, because the model relationship is not causal.
Continue reading — it's free

Google or a one-time email code. No password.