CASTLE: training natural language search before the first real query

Wendy Ran Wei, Hao Li, Weiwei Guo, Xiaowei Liu, Xueyin Chen, Chun How Tan, Malay Haldar, Soumyadip Banerjee, Kedar Bellare, Huiji Gao, Sanjeev Katariya, Stephanie Moyerman13 Oct 2026

Problem

A team launching natural-language search ("pet-friendly beach house that sleeps eight") has no real queries and no relevance labels to train a retrieval model. Both must be synthesized. Earlier methods solve the two needs separately: InPars generates a query from one document and then needs extra LLM judgments for labels, a second source of error. Promptagator varies queries with few-shot examples, but still differs from real user behavior. Airbnb measured the gap as KL divergence from real queries: prior approaches scored 9.33 to 26.2.

Example from the article: Airbnb needed natural-language search to work from day one, with zero prior traffic.

Solution in brief

CASTLE builds each synthetic query from a pair of listings out of a real booking session: the booked listing (positive) and a viewed-but-not-booked one (negative). The LLM writes a query that the positive satisfies and the negative does not, so the label is correct by construction. Three prompt types plus about 500 seed queries make the queries look like real ones.

Questions this case answers
  • How to generate synthetic training queries and relevance labels for a retrieval model when no real query traffic exists, so that the query distribution matches future users?
  • How to obtain correct-by-construction relevance labels for LLM-generated queries without human annotation or a separate LLM judge?
  • How to get training queries and relevance labels for natural-language search before any real query traffic exists?
  • How to make synthetic search queries match the length and phrasing of real users?

Tools and methods

  • Contrastive listing pairs from bookings — input and relevance label.
  • Template, few-shot and attribute-grounded prompts — query variety.
  • Seed queries (about 500) — examples of real guest language.
  • Virtual Judge (LLM evaluator) — labels pairs without bookings.
  • KL divergence — distance of synthetic queries from real ones.
Continue reading — it's free

Google or a one-time email code. No password.