Building AI search: from prototype to production

David Howden6 Jan 2026

Problem

A production search system is a chain of parts: query understanding, keyword and vector retrieval, merging, ranking signals, personalization and sometimes an LLM that writes the answer. A new idea touches several parts at once. The team must decide which queries it applies to and whether the extra CPU, memory and latency pay off. Without a fast, safe path, ideas stall or break queries that work.

Example from the article: semantic search handles conversational queries, but vector models still struggle with obscure brands, product SKUs and part numbers. The best systems therefore combine keyword and semantic retrieval instead of replacing one with the other.

Solution in brief

Algolia runs a "learning loop": try an idea on top of current production behavior, verify it by replaying historical or synthetic queries and computing NDCG and recall, then ship it to a live A/B test. Algorithms are defined as configurations of modular components, so building a variant takes seconds, offline verification takes minutes, and starting the live test takes seconds.

Questions this case answers
  • How to iterate on a composite ranking system safely when every new component interacts with the existing ones and adds cost
  • How to verify a ranking change offline in minutes before spending live traffic on it
  • How to change one part of a multi-stage search system without breaking queries that already work?
  • How to decide whether a new model, such as vector search or an LLM query rewriter, is worth its extra latency and compute?

Tools and methods

  • Query orchestration — blending retrieval and ranking strategies into one search experience
  • Hybrid search — keyword and semantic retrieval together, because vectors miss SKUs and rare brands
  • LLM query rewriting — an example component swapped into the pipeline
  • Automated query replay — historical or LLM-generated synthetic queries run through a candidate algorithm
  • NDCG and recall — offline metrics for the replay
  • A/B test — final check on real traffic
Continue reading — it's free

Google or a one-time email code. No password.