How to improve search without looking at queries or results

Ellese Cotterill25 Nov 2024

Problem

Canva engineers work on search over users' own designs and designs shared with them. Company policy forbids viewing personal designs, so real queries and documents cannot be sampled and labeled by judges, which is the standard way to evaluate public search. Before this work, an engineer ran a few known bad queries in a personal account and then went straight to an online A/B experiment. A handful of queries has little statistical power, a bad change can reach experiment users, and each online experiment takes days to weeks.

Example from the article: a spell-correction change was checked with the single misspelled query "desgin": it returned nothing before the change and several documents containing "design" afterwards.

Solution in brief

Generate the evaluation data instead of labeling it. An LLM writes realistic synthetic designs and queries, and each test case has one known relevant design. The data runs through the real search pipeline, started locally in containers, and recall and precision come out in under 10 minutes.

Questions this case answers
  • How to build a labeled offline evaluation set for a retrieval system when real queries and documents cannot be inspected or labeled
  • How to speed up iteration on a search pipeline when each online experiment takes days to weeks
  • How to evaluate changes to a private document search offline when nobody may read user content or queries?
  • How to build a labeled test set for search without human judges or real user data?

Tools and methods

  • Synthetic test cases (GPT-4o) — generate design titles, text, queries and nonrelevant distractor designs.
  • Re-finding labeling scheme — one relevant design per query, all other designs nonrelevant.
  • Testcontainers — run the whole search pipeline locally with production configuration.
  • Elasticsearch — real search backend inside the local pipeline.
  • Streamlit — own dashboard to compare configurations by query type and difficulty.
Continue reading — it's free

Google or a one-time email code. No password.