A Trial ML System Design Interview

Pavel Filonov27 May 2023

Problem

A retail chain with hundreds of stores has analysts who watch charts of the most important products and stores. They spot a product that stopped selling, then look for the cause, such as bad shelf display. They cannot cover everything, so a drop outside the top items is noticed late, and delay means lost sales. The task is to flag "look here" cases to analysts automatically, for one SKU in one store or for one SKU across all stores.

Example from the article: a mock interview task with about 5000 receipts per hour, thousands of SKUs and about 100 thousand SKU-store pairs. Analysts labeled roughly 10 incidents per month for the whole system.

Solution in brief

Treat it as rare-event detection, close to anti-fraud. Build a dataset from labeled incidents plus randomly sampled unlabeled segments as negatives, start with an unsupervised score and a threshold, and pick the model by recall at fixed specificity. Reduce errors to money and check payback before building.

Questions this case answers
  • How to detect sudden drops in sales across a very large set of item-location pairs when only about ten incidents per month are labeled
  • How to pick one offline metric and alert threshold for a rare-event detector using recall at fixed specificity and business cost
  • How to detect a sudden sales drop for a product in a store across about 100 thousand SKU-store pairs
  • How to build an anomaly detector when only incidents are labeled and there are about 10 per month

Tools and methods

  • One-Class SVM, Isolation Forest — unsupervised baseline that outputs a score; a threshold turns it into a binary alert
  • Autoencoder, then GAN discriminator — heavier second step on a latent feature space
  • Recall at fixed specificity — offline metric for comparing models
  • Time-series cross-validation — rolling folds with a fixed test set
  • Prophet — viewers' alternative: compare a forecast with actual sales
Continue reading — it's free

Google or a one-time email code. No password.