Roonomics: How Economics Helped Us to Build a Churn Prediction Model

Firat Yaman22 Sep 2026

Problem

Deliveroo wants to keep its restaurant partners, but its account management and retention team can contact only a few of them. The old churn model flagged restaurants by rules built on metrics that differed most between active and churned partners. It was tuned to catch almost every at-risk restaurant, and since churn is rare, it also flagged many that were not at risk. Reaching out to a restaurant that would not churn has zero benefit and costs the chance to help one that would.

Example from the article: the new model cut the number of flagged restaurants by more than 90%, in line with actual churn rates and the team's capacity.

Solution in brief

Treat churn flagging as a capacity problem. Rank restaurants by predicted churn probability and flag only those above an adjustable cutoff, favoring precision over recall. Validate with rolling windows, explain flags with Shapley values, and test usefulness in a randomized experiment.

Questions this case answers
  • How to choose a churn-risk threshold when the team that acts on flags has fixed capacity and churn is rare
  • How to test whether a churn-risk score reduces churn rather than only predicting it
  • How to flag at-risk restaurant partners when the retention team can only act on a few of them?
  • How to build a churn model for a rare event without leaking future data into validation?

Tools and methods

  • Churn probability ranking with adjustable cutoff — scores restaurants and flags those above an adjustable probability cutoff
  • Rolling-window cross-validation — selects hyperparameters while always training on earlier data and validating on a later period
  • Shapley values (SHAP) — name the feature that contributed most to each restaurant's churn probability
  • Calibration curves — compare predicted and real churn rates
  • Randomized treatment/control experiment — tests whether showing alerts reduces churn
Continue reading — it's free

Google or a one-time email code. No password.