Problem
Documentation, knowledge base and support sites serve users who know the exact API method or error code and users who only know the task or concept. There is no purchase to count, so "good search" means that relevant content is found, the best results sit near the top and similar versions or frameworks can be told apart. Search analytics help, but low-traffic pages give too few searches and clicks to show whether search works. Without a benchmark, teams are guessing whether a change helped or hurt.
Example from the article: a query such as "configure instantsearch javascript" can lose to a broad guide because the widget name, product area and framework sit in separate, weakly weighted fields.
Solution in brief
Reproduce each bad query, describe the symptom, classify the user's intent, and test fixes against a golden set of about 50 queries with expected URLs. Score the set with three rank-based metrics and rerun it after every change. Apply the smallest fix in the earliest layer: indexed data, record structure and ranking, query-time rules, then presentation.
Questions this case answers
- How to measure and improve keyword search quality over a large text collection when there is no single business outcome to optimize
- How to attribute a search failure to the right layer (indexed data, ranking, query-time rules, presentation) and verify that a fix did not hurt other queries
- How to find out why a query on a documentation or support site returns the wrong pages
- How to measure site search quality without a purchase or conversion signal
Tools and methods
- Golden set — selected queries with expected URLs, the benchmark for every change
- Algolia — index, dashboard, Search Analytics, ranking settings, Rules, synonyms
- Diagnostic log — record of failing queries with symptom and intent
- Rank-based metrics — relevance, top five matches, top match
Google or a one-time email code. No password.