Experience in creating a design document when designing an ML system. Forecasting sales in stores

Mikhail Masagutov27 May 2023

Problem

A food and beverage company sells through a distributor to small non-chain stores: kiosks, convenience stores, sole proprietors. It needs each store's revenue for the next 6 months to set sales plans, segment stores and assign KPIs. Today sales representatives guess these numbers, then must deliver the plan they set themselves. Data sits in different sources and requirements are vague. A first model with good ML metrics was shelved because it missed business expectations and its economic effect was never estimated.

Example from the article: a data scientist leading his first project used the ML System Design Doc template to align a fragmented business team on target, metric and pilot.

Solution in brief

Fill in a design document before modeling: goals and scope, data sources with owners, task formulation, pipeline diagram, risks per stage, pilot metrics and the economic-effect formula. Writing the methodology down rejects dead ends early, exposes bottlenecks and gives the business one place to confirm the task was understood.

Questions this case answers
  • How to structure an ML project with a fragmented business team using a design document, and decide early that a classical time-series approach will not meet a long-horizon requirement
  • How to forecast six-month totals for many small units with short and irregular histories, replacing manual estimates by field staff
  • How to structure and agree an ML project with a fragmented business team using a design document
  • How to decide early that classical time series will not work for a long-horizon store revenue forecast

Tools and methods

  • ML System Design Doc template — Russian-language template on GitHub
  • Block diagram of the pipeline — shows stages and bottlenecks
  • Risk list per stage — risks and ways to remove them
  • MAPE — error metric agreed with the business
  • Pilot with agreed success criteria — how to test the model and compute the economic effect
Continue reading — it's free

Google or a one-time email code. No password.