Problem
Zendesk Guide lets businesses build a searchable knowledge base. Agents search it to resolve tickets, and end users search it to avoid opening one. The Elasticsearch keyword search is fast and scalable, but it misses relevant articles when the query and the article share few words. Public embedding models help, but they may not generalize to each customer's industry terminology. They are also trained on well-formed questions, while real queries are short and informal. Manual labeling of query-document pairs does not scale across many accounts.
Example from the article: a new digital currency with its own terminology, or a medical device whose end users search for related issues in their own words.
Solution in brief
Keyword search returns candidates, and a bi-encoder re-ranks them by cosine similarity between query and document embeddings. The encoder is fine-tuned on aggregated click logs with a KL-divergence loss, and a separate lightweight adapter can be trained for each customer account.
Questions this case answers
- How to turn noisy, position-biased click logs into training data for a retrieval model when manual relevance labels are too costly
- How to add semantic re-ranking on top of an existing keyword search engine without replacing it
- How to adapt a retrieval model to many tenants with highly imbalanced traffic while keeping each tenant's data separate
- How to improve help-center search relevance when query and article share few words?
Tools and methods
- Elasticsearch — first-stage keyword retrieval of candidate documents
- Bi-encoder embedding model (dense retrieval) — maps queries and documents into one vector space
- Hybrid score — weighted average of keyword score and cosine similarity
- Click aggregation and diversity sampling — turn noisy clicks into training triplets
- KL-divergence loss — trains on the normalized click distribution per query
- Adapters — per-account fine-tuning of a small share of parameters
Google or a one-time email code. No password.