Automated GenAI-driven search quality evaluation

Xueying Lu, Ali Hooshmand, Fan Dong, Raghavan M.17 Dec 2024

Problem

LinkedIn's global search bar shows auto-completed suggestions that blend people, companies, jobs and plain text queries. Low-quality suggestions sidetrack users, so every change to the typeahead needs a quality check. Human evaluation does not scale with the platform: several raters need days or weeks per experiment. The check is also hard to define, because suggestion types have different intents and results are personalized, so the same query ("Andrew") gives different suggestions to different users.

Example from the article: a team expanded the plain text suggestion inventory with short phrases summarized from high-quality LinkedIn posts and needed to know whether quality improved.

Solution in brief

Write per-type grading guidelines with a binary high/low label, sample a golden test set of {query, member id} pairs from traffic, and let a GPT model grade every suggestion with a typed prompt template. Average the grades over the top 1, 3, 5 and 10 positions and over all sessions. A run takes a few hours.

Questions this case answers
  • How to evaluate the quality of a personalized ranked list of heterogeneous items at scale when human rating takes days or weeks
  • How to build a fixed evaluation set that represents all user intents and failure sessions of a personalized surface
  • How to measure the quality of search typeahead suggestions without waiting days for human raters?
  • How to build a golden test set for a personalized search surface?

Tools and methods

  • LLM-as-a-judge (GPT-3.5 Turbo via Azure OpenAI) — scores each suggestion 1 or 0 with a written reason
  • Golden test set — fixed sample of {query, member id} pairs reused for every experiment
  • Per-type prompt templates — IDENTITY, TASK GUIDELINES, optional few-shot EXAMPLES, INPUT, OUTPUT
  • TyahQualityN scores — average grade of the top N suggestions
Continue reading — it's free

Google or a one-time email code. No password.