Building Airbnb Categories with ML and human-in-the-loop

Mihajlo Grbovic, Ying Xiao, Pratiksha Kadam, Aaron Yin, Pei Xiong, Dillon Davis, Aditya Mukherji, Kedar Bellare, Haowei Zhang, Shukun Yang, Chen Qian, Sebastien Dubois, Nate Ney, James Furnary, Mark Giangreco, Nate Rosenthal, Cole Baker, Bill Ulammandakh, Sid Reddy, Egor Pakhomov21 Nov 2022

Problem

Airbnb wanted a browsing experience where the inventory leads and travelers discover destinations they would not search for. That needed 56 themed categories (Lakefront, Skiing, Castles, Amazing Pools and others) with every listing in the catalog assigned to the right ones. Manual curation was too slow and costly for a launch a few months away. Rules alone were not accurate enough, and a model needs human-made labels first.

Example from the article: a listing within 100 meters of a lake, with "lakefront" in its title and reviews, lake views in its photos and kayaking nearby, should land in the Lakefront category.

Solution in brief

Score each listing-category pair with rules (a weighted sum of indicators) and send the best candidates to human reviewers. Their decisions train one binary classifier per category. Listings above a 90% precision threshold go to production directly; the rest keep going to reviewers, whose answers retrain the model.

Questions this case answers
  • How to assign a large catalog of items to a fixed set of themed groups when no labeled training set exists and neither manual curation nor rules alone are accurate or cheap enough
  • How to combine model predictions with human review so that confident predictions ship automatically and the model keeps improving from reviewer decisions
  • How to group a large catalog of listings into themed collections when manual curation is too slow and rules are too inaccurate?
  • How to bootstrap labels for a classifier when there is no training set yet?

Tools and methods

  • Weighted sum of indicators — rule-based scoring of listing-category pairs from listing and geo signals
  • Human review — confirms categories, picks cover photos, assigns quality tiers, adds missing POIs
  • Embedding nearest neighbors — expands candidates for categories where rules find few listings
  • Binary classifier per category — trained on reviewer labels, evaluated on a precision-recall (PR) curve
  • Quality model and Vision Transformer cover-image model — assign a quality tier and choose the category photo
Continue reading — it's free

Google or a one-time email code. No password.