When a picture is worth more than words

Yuanpei Cao, Bill Ulammandakh, Hao Wang, Tony Hwang8 Dec 2022

Problem

Airbnb hosts upload many photos per listing, and guests judge a listing by them as much as by price, reviews and description. Attractiveness is subjective and does not fit into a fixed list of amenities. Hosts struggle to order their photos, and the first five matter most because they are viewed most often. A weak ad photo costs clicks and bookings.

Example from the article: for social media ads, Airbnb picks a living room, bedroom, kitchen and exterior photo per listing automatically.

Solution in brief

Score every photo with a CNN trained on human rating distributions, and classify its room type. Use the score and room type to filter ad photos and, together with a home-design score, to pick and order a listing's first five photos. Image embeddings with an approximate nearest-neighbor index add similarity search.

Questions this case answers
  • How to rank images of items by subjective attractiveness when quality labels are human rating distributions rather than classes
  • How to select and order a small set of images that trades off attractiveness, relevance and diversity to maximize conversion
  • How to serve real-time similarity search over millions of image embeddings that are recomputed on a schedule
  • How to choose the listing photos that make an ad get more clicks and bookings?

Tools and methods

  • CNN aesthetics model with EMD loss — predicts the distribution of 1 to 5 human ratings per photo
  • Room categorization model — labels each photo as living room, bedroom, kitchen, exterior and so on
  • Home design evaluation CNN — scores interior design and architecture, trained on Plus and Luxe qualification data
  • Image embeddings and cosine similarity — compare photos by visual content
  • HNSW on OpenSearch — real-time approximate nearest-neighbor search
  • Airflow and Hive — sync recomputed embeddings into the index
Continue reading — it's free

Google or a one-time email code. No password.