Problem
A team launching natural-language search ("pet-friendly beach house that sleeps eight") has no real queries and no relevance labels to train a retrieval model. Both must be synthesized. Earlier methods solve the two needs separately: InPars generates a query from one document and then needs extra LLM judgments for labels, a second source of error. Promptagator varies queries with few-shot examples, but still differs from real user behavior. Airbnb measured the gap as KL divergence from real queries: prior approaches scored 9.33 to 26.2.
Example from the article: Airbnb needed natural-language search to work from day one, with zero prior traffic.
Solution in brief
CASTLE builds each synthetic query from a pair of listings out of a real booking session: the booked listing (positive) and a viewed-but-not-booked one (negative). The LLM writes a query that the positive satisfies and the negative does not, so the label is correct by construction. Three prompt types plus about 500 seed queries make the queries look like real ones.
Questions this case answers
- How to generate synthetic training queries and relevance labels for a retrieval model when no real query traffic exists, so that the query distribution matches future users?
- How to obtain correct-by-construction relevance labels for LLM-generated queries without human annotation or a separate LLM judge?
- How to get training queries and relevance labels for natural-language search before any real query traffic exists?
- How to make synthetic search queries match the length and phrasing of real users?
Tools and methods
- Contrastive listing pairs from bookings — input and relevance label.
- Template, few-shot and attribute-grounded prompts — query variety.
- Seed queries (about 500) — examples of real guest language.
- Virtual Judge (LLM evaluator) — labels pairs without bookings.
- KL divergence — distance of synthetic queries from real ones.
Google or a one-time email code. No password.