Search Query Understanding with LLMs: From Ideation to Production

Loc Trinh, Ali Rokni, John Hawksley4 Feb 2025

Problem

Yelp wanted better intent understanding of search queries: which part names a business, a topic, a place or a time, whether a word is misspelled, and which phrases to highlight in review snippets. The old systems for these tasks were several pieces stitched together and lacked intelligence. A large LLM does these tasks well, but calling one for every search costs too much and is too slow for a real-time system.

Example from the article: for "epcot restaurants" the LLM labels "epcot" as a location, so the backend rewrites the location box from "Orlando, FL" to "Epcot" and narrows the search area.

Solution in brief

LLMs suit tasks with short inputs and outputs, per-query caching and a power-law query distribution. Yelp prototypes with the strongest model, tests it online using precomputed answers for head queries only, then distills the prompt into a fine-tuned smaller model that precomputes answers for most traffic, with a still smaller model for the tail.

Questions this case answers
  • How to add LLM-based understanding to a high-volume text query system when per-request cost and latency rule out calling a large model live
  • How to validate an LLM feature online cheaply by precomputing outputs only for the most frequent inputs
  • How to use an LLM for query segmentation and spell correction without paying for it on every search?
  • How to prove an LLM query-understanding idea works before building real-time infrastructure?

Tools and methods

  • Few-shot prompting with GPT-4 — formulate the task and teach it through curated examples
  • RAG with query context — add viewed business names (segmentation) or predicted business categories (highlights) to the input
  • Cache of precomputed LLM outputs — serve head queries from a key/value store
  • Fine-tuned GPT4o-mini, BERT, T5 — cheap offline and real-time models trained on GPT-4 labels
Continue reading — it's free

Google or a one-time email code. No password.