Advancing semantic search for millions of Rovo users

Kang Li, Lu He, Xin Yu, Fei Teng16 Mar 2026

Problem

Keyword search in a workplace tool fails in four ways. Query words rarely match the title ("how to find my pay stub" vs "Global Workday Payslips"). The answer is spread over tickets, pages and comments. In issue trackers the strongest signals are structured fields, not text. Team vocabulary changes faster than static relevance rules. The cost is time spent hunting, duplicate work and decisions made without context.

Example from the article: a developer types "that flaky payments test from last week"; the system has to link "flaky" to recent bugs labeled flaky, "payments" to specific services, and "last week" to a time filter over updates.

Solution in brief

Atlassian runs hybrid retrieval: neural embeddings combined with classic signals (fields, recency, project, engagement). The embedding model has been replaced twice, MiniLM to BGE-large to EmbeddingGemma-300m, and each upgrade is checked offline and online. Fine-tuning NVIDIA's Llama-Nemotron-Embed-1B-V2 on a public Jira-like dataset gave a 26%-40% retrieval gain.

Questions this case answers
  • How to retrieve the right item from a heterogeneous corpus of structured records and free text when queries do not share vocabulary with the documents
  • How to decide when to replace an embedding model in a production retrieval system, using offline retrieval metrics and online behavior metrics
  • How to make workplace search find the right ticket or page when the query does not share words with the title?
  • How to rank semantically similar but irrelevant items below the right one in a multi-project workspace?

Tools and methods

  • Hybrid search — embedding similarity blended with fields, recency, project, engagement
  • Dense retrieval (BGE, EmbeddingGemma-300m) — semantic matching of queries to work items
  • Embedding fine-tuning (NVIDIA NeMo Automodel, Llama-Nemotron-Embed-1B-V2) — adapt the model to Jira-like text
  • Synthetic data generation (NeMo Data Designer) — training pairs for the fine-tune
Continue reading — it's free

Google or a one-time email code. No password.