Unraveling Rovo search relevance

Swati Katta, James Gao12 Aug 2025

Problem

Teams keep documents, tickets and messages in dozens of tools. One search box must rank results across all of them by relevance and by who is asking, and never show what the user may not open. Each source has its own data shape, access rights must be respected, and new sources arrive with no click history. Atlassian faced this building Rovo search over its products and 50+ third-party SaaS apps.

Example from the article: the query "search improvements roadmap page" should put Confluence pages first, while "SEARCH-4356" should put Jira tickets first.

Solution in brief

Rovo search is a layered stack: query rewriting, per-product OpenSearch retrieval (L1), semantic and behavioral reranking (L2) and a cross-product merge (L3). Smart Answers reuse it, rerank passages and let an LLM answer with citations. Quality is tracked online by a click, dwell and feedback metric, and offline by golden sets and an LLM judge.

Questions this case answers
  • How to build a multi-stage relevance ranking over many heterogeneous content sources with per-user permissions and sub-second latency
  • How to generate an answer from retrieved passages so that each claim cites its source document
  • How to measure search quality offline when a new content source has no engagement data yet
  • How to rank results when one search box must cover Jira, Confluence and 50+ third-party apps?

Tools and methods

  • Query classification and rewriting — a classifier and a self-hosted LLM rephrase the query.
  • BM25 and KNN in OpenSearch — L1 retrieval with permission filters.
  • Gradient boosted decision tree (GBDT) rescoring — reranks L1 hits by index-level features.
  • Cross-encoder and Deep & Cross Network (DCN) — L2 reranker for semantic relevance and click probability.
  • Passage chunking and ranking — picks the chunks passed to the LLM.
  • LLM-as-a-judge — grades rankings offline and builds training data.
Continue reading — it's free

Google or a one-time email code. No password.