Yelp Content As Embeddings

Fanny Salvail-Berard20 Apr 2023

Problem

Yelp has to tag, organize and rank a huge volume of reviews and photos. Ready embeddings make new models on this content easier to build, since they serve as a baseline input for any model. Photo classifiers show the problem: the production ResNet50 models need many examples per class, so their tag sets stay small and curated. The team wanted one set of embeddings for reviews, businesses and photos that serves tagging, sentiment, ranking and similarity.

Example from the article: photos labeled Waffles were scored by CLIP as Chicken Wings & Fried Chicken at 44% against 11% for Waffles. A manual check showed Texan fried chicken and waffles, so the answer was reasonable.

Solution in brief

Use strong pre-trained encoders. Reviews use the off-the-shelf Universal Sentence Encoder, because a Yelp fine-tuned encoder did not beat it. A business vector is the mean of its 50 most recent review vectors. Photos use zero-shot CLIP with hand-written text labels, compared per class against the production ResNet50 classifiers.

Questions this case answers
  • How to build general-purpose vector representations of text, entities and images that many downstream models can reuse without per-task training
  • How to check whether an off-the-shelf pre-trained model is good enough for a domain before paying for fine-tuning or training a dedicated classifier
  • How to get reusable embeddings of reviews, businesses and photos that many teams can use as model input?
  • How to decide whether a domain fine-tuned text encoder is worth building over an off-the-shelf one?

Tools and methods

  • Universal Sentence Encoder (USE) — fixed-length review embeddings, used off the shelf from TensorFlow Hub.
  • CLIP — zero-shot photo embeddings and tagging from text labels, pre-trained model from Hugging Face.
  • Mean of review embeddings — business vector, then top-k cosine similarity.
  • Label engineering and a 70% threshold — raise CLIP precision against the ResNet50 baseline.
Continue reading — it's free

Google or a one-time email code. No password.