Product Features: The Key to Increasing Model Efficiency for Account Propensity

Ilya Ekhlakov, Darya Senicheva28 Oct 2023

Problem

Wrike, a B2B work-management SaaS, runs an account propensity model that predicts churn, downgrade and upgrade for horizons of 1 to 6 months. Customer success uses it to choose whom to contact, and forecasts of churn loss rely on its probabilities. It saw only generalized activity, so it caught one upgrade pattern: seats approaching purchased licenses. Product usage features fix that, but they add hundreds of candidates, drifting distributions and a risk of unstable, uncalibrated outputs.

Example from the article: for one of the largest customers the model predicted no status change for 6 months, yet the difference in feature contributions between two forecasts showed rising churn propensity, and a manager's visit worked.

Solution in brief

Turn each product feature's and use case's 6-month usage history into three trend features: linear trend, sharp break and U-shape. Cut the 600+ new features with PSI-based stabilization and a one-pass hybrid recursive feature elimination around LightGBM, tuned on time-split folds to a variance-penalized Brier score. Calibrate with isotonic regression.

Questions this case answers
  • How to turn a long usage history of many features into trend features for a tabular churn and upgrade model without overfitting on hundreds of candidates
  • How to keep the calibration and stability of a propensity model over time when selecting from hundreds of derived features
  • How to improve a churn and upgrade propensity model with product usage features
  • How to turn a feature's usage history into trend features for a tabular model

Tools and methods

  • Trend features — slope and p-value, MAD-based break point, U-shape from median changes
  • PSI and quantile transformation — stabilize drifting features
  • LightGBM with Optuna — time-split folds, Brier score plus variance penalty
  • Hybrid recursive feature elimination — one pass, ranks rebuilt after each removal
  • Isotonic regression — calibrates probabilities
  • SHAP — feature contributions for insights
Continue reading — it's free

Google or a one-time email code. No password.