From Pre-trained to Fine-tuned: Nextdoor’s Path to Effective Embedding Applications

Karthik Jayasurya7 Sep 2023

Problem

Nextdoor ranking models run on features built from interaction logs. These features say little about what a post is about or what a user cares about. Feed and notification ranking need that signal, but their production models are tree-based and cannot take vectors. Off-the-shelf embeddings also have a fixed, high dimensionality, so a daily-refreshed vector for every user is expensive to store and serve.

Example from the article: a user's embedding is built from the posts they created, commented on, clicked and saw. The similarity between that user and a post becomes a feature of the ranking model.

Solution in brief

Embeddings serve as feature extractors, not model inputs. A sentence encoder embeds text, user vectors aggregate the content a user touched, and cosine similarity and dot products between entities become scalar affinity features for the trees. The encoders improved in three stages: pre-trained SBERT, self-supervised tuning on queries, and a two-tower model tuned on feedback.

Questions this case answers
  • How to turn unstructured content into reusable user and item representations for ranking models that accept only scalar features
  • How to adapt off-the-shelf text embeddings to a product domain while cutting their dimensionality and serving cost
  • How to add text understanding to feed and notification rankers that are built on tabular features and tree models?
  • How to build user embeddings from the content a user interacted with and keep them fresh for every user?

Tools and methods

  • Sentence-BERT (Sentence Transformers) — pre-trained text embeddings, later fine-tuned
  • Two-tower model, multi-task learning — fine-tune user and post embeddings on feedback for notifications and feed, with lower dimensionality
  • Contrastive learning — trains the query encoder from search sessions
  • HNSWlib — approximate nearest-neighbor search for query expansion
  • Feature store, Apache Airflow — host embedding features, run daily user aggregation
  • Cosine similarity, dot product — make affinity features
Continue reading — it's free

Google or a one-time email code. No password.