Scaling Product Recommendations using Basket Analysis- Part 1

Apurva Sinha24 Aug 2022

Problem

A retailer with a large catalog and millions of transactions wants to show customers complementary products and raise basket size and revenue. Mining billions of transactions is memory-heavy, and plain frequency mining favors whatever is bought most often. The solution has to cover all departments, cope with changing data and trade the number of product combinations against complexity.

Example from the article: a hair conditioner is recommended together with a hair serum, which belongs to a different sub-category.

Solution in brief

Walmart filters products by support, price and GMV (gross merchandise value) contribution, then counts item pairs in baskets and keeps pairs with a high co-occurrence probability. Both FP-Growth and a pairwise approach were tested, and the pairwise one was chosen because it is cheaper. It runs on PySpark with Airflow for several departments at once.

Questions this case answers
  • How to generate complementary-item recommendations from co-purchase history across a very large and diverse catalog under a fixed compute budget
  • How to filter items before co-occurrence mining so that frequent trivial items and rare items do not dominate the rules
  • How to build complementary-product recommendations from basket data for every department of a large catalog?
  • How to choose which products and baskets to feed into association mining so that results are not dominated by staples or rare items?

Tools and methods

  • Support, price and GMV filters — choose which products and transactions enter the analysis.
  • Pairwise co-occurrence probability — the selected model: probability of one item in a basket given another, computed for item pairs only.
  • FP-Growth (PySpark ML) — the alternative: frequent itemsets and association rules, evaluated by support, confidence and lift.
  • Apache Spark (PySpark) — distributed processing of billions of transactions.
  • Apache Airflow — schedules the batch jobs per sub-department.
Continue reading — it's free

Google or a one-time email code. No password.