Problem
A product team that owns a complex dashboard has many UI elements and needs to know which part of the UI drives a target metric, so it can choose what to A/B test. Indeed's Interview team has one goal: more interviews on the platform. Here it wanted UI test ideas that raise the invitations employers send from the dashboard.
Example from the article: to find such ideas, the team analyzed employer clicks on the dashboard and predicted interview invitations from them.
Solution in brief
Turn each UI element into a feature (how many times the employer clicked it), train a CatBoost classifier to predict whether the target button was clicked, and read a SHAP plot. Features that push the prediction up or down become positive and negative predictors, and each one is a test hypothesis. The model gives only associations, so every hypothesis goes to an A/B test.
Questions this case answers
- How to generate a ranked batch of experiment hypotheses from logged interaction data when each interface element could be a candidate lever and intuition does not scale
- How to generate many UI A/B test ideas at once instead of brainstorming them one by one?
- How to find which UI elements are linked to a target action such as sending an invitation?
- How to turn a feature-importance plot into concrete test hypotheses?
Tools and methods
- CatBoost — tree-based classifier that captures interactions between features.
- SHAP plot — shows each feature's direction and strength of influence on the prediction.
- A/B test — verifies each hypothesis, because the model relationship is not causal.
Google or a one-time email code. No password.