Problem
Product managers and engineers at Mercari checked search quality by hand. They took sampled keywords, looked at every result item one by one and computed the share of irrelevant items. The process was slow, and with several reviewers the criteria differed, so results were unstable.
The team needed reviews that run daily or weekly, show up on a dashboard, cover enough keywords, follow clear criteria and take the searcher's intent into account.
Example from the article: for the query "iPhone", an iPhone case is a complementary item, while for the query "telescope", socks are completely unrelated.
Solution in brief
An LLM judges every item in the top of the results for a keyword and returns a relevance score from 0.0 to 1.0 with a reason. The score follows the four ESCI classes (Exact, Substitute, Complement, Irrelevant), each mapped to a fixed value. The same prompt serves a weekly production monitor and an on-demand offline evaluation tool.
Questions this case answers
- How to measure ranked-result relevance continuously and consistently when manual review is slow and reviewers disagree
- How to monitor search result relevance every week without product managers checking items by hand?
- How to get consistent relevance judgments when several people review search results?
- How to check the relevance effect of a search change before running an A/B test?
Tools and methods
- LLM-as-a-judge — scores each result item against the query with a reason
- ESCI relevance classes — Amazon's Exact / Substitute / Complement / Irrelevant scale, mapped to scores
- Gemini 2.5 Pro — the model chosen after comparing several LLMs
- BigQuery — stores review results for the monitoring dashboard
Google or a one-time email code. No password.
