Building The Intent Engine: How Instacart is Revamping Query Understanding with LLMs

Yuanzheng Zhu, Guanghua Shu, Raochuan Fan, Vinesh Gudla, Tejaswi Tenneti13 Nov 2025

Problem

Instacart search had to interpret queries such as "bread no gluten" or "2% reduced-fat ultra-pasteurized chocolate milk". Classification, rewrites and tagging ran as separate legacy models, each with its own pipeline. Rare queries have few clicks, so engagement-trained models generalize poorly. The labels are also noisy: a user searches "bread" and buys bananas. Query understanding sits upstream of retrieval, so it gets no direct click feedback.

Example from the article: the query "verdant machine" looks like machinery, but conversion history shows users buy a smoothie brand after it.

Solution in brief

Use an LLM for each query-understanding task and feed it the company's own data (top converted categories and brands, catalog matches). Validate every output against the catalog or taxonomy. Run the expensive version offline on frequent queries and cache it. Use the same outputs as training data for a small Llama-3-8B model that serves the long tail in real time.

Questions this case answers
  • How to interpret rare free-text queries with a language model when engagement labels are noisy or missing and inference must be fast and cheap
  • How to distill an expensive offline language model into a small real-time model while caching answers for frequent inputs
  • How to understand rare long-tail search queries when engagement data is too sparse to train on?
  • How to replace several separate query-understanding models with one LLM backbone?

Tools and methods

  • Retrieval-augmented generation (RAG) — injects conversion history and catalog candidates into the prompt
  • Post-processing guardrails — filter hallucinated or off-taxonomy outputs
  • Prompt engineering (chain-of-thought, few-shot) — one dedicated prompt per rewrite type
  • LoRA fine-tuning of Llama-3-8B — distills the offline LLM into a real-time model
  • FP8 quantization — tested for latency, not deployed
Continue reading — it's free

Google or a one-time email code. No password.