Recommend API: one shared pipeline for many recommendation features at Slack

Katrina Ni, Aaron Maurer6 Sep 2022

Problem

Slack has many places where a suggestion helps: adding people to a direct message, inviting contacts, choosing channels to leave or archive, sorting the channel browser, picking a send time for invite reminders. Each looks like a good machine learning case, but a separate stack for each is not realistic. Teams used to skip many use cases for fear that the up-front cost would not pay off. Models also must not be able to leak customer message content.

Example from the article: in the channel browser, a hand-tuned ranking was replaced by an XGBoost classifier and CTR rose by 14.76%.

Solution in brief

Slack built one Recommend API. Every use case is a "recommender", a chain of reusable steps (fetch candidates, filter, add features, score, rerank). Request logs and click logs feed the same Airflow training jobs for all recommenders. A new use case starts with a hand-tuned scorer, collects logged features and clicks, and moves to logistic regression or XGBoost once an A/B test shows a CTR gain.

Questions this case answers
  • How to serve many different recommendation use cases from one reusable candidate, filter, scoring and reranking pipeline instead of building a separate system for each
  • How to start a recommender when no model exists yet and no interaction data has been logged
  • How to launch many recommendation features across a product without building a bespoke ML stack for each?
  • How to bootstrap a recommender that has no training data yet?

Tools and methods

  • Candidate fetchers, filters, rerankers — reusable steps of every recommender
  • Feature Store — entity features, looked up by primary key at scoring time
  • Hand-tuned model — heuristic scorer used until training data exists
  • Logistic regression and XGBoost — models compared per source
  • Apache Airflow and Kubernetes — training jobs and model serving
  • Prometheus — reliability and efficiency metrics
Continue reading — it's free

Google or a one-time email code. No password.