Problem
Facebook Groups search relied on keyword matching, so a query worded differently from a post found nothing, even when the post answered it. Users also had to read many comments to find a consensus, and could not easily find group opinions to check a decision. Semantic retrieval also needs a quality check at scale, because vector similarity scores are hard to interpret.
Example from the article: a search for "small individual cakes with frosting" can return zero results if the group writes "cupcakes". A search for "Italian coffee drink" should match a post about "cappuccino" that never says "coffee".
Solution in brief
Run keyword retrieval (Unicorn inverted index) and embedding retrieval (a 12-layer, 200-million-parameter encoder with a Faiss index) in parallel, then merge the candidates in one L2 ranker that sees lexical and semantic scores. Check each build with Llama 3 as a relevance judge that has a "somewhat relevant" grade.
Questions this case answers
- How to add semantic retrieval to a keyword search system so that paraphrased queries find relevant content without losing exact-match precision
- How to evaluate search relevance on every build when human labeling is too slow and similarity scores are not interpretable
- How to make in-product search find posts that match the meaning of a query when the wording differs from the posts?
- How to merge keyword and embedding candidates into one ranked list?
Tools and methods
- Unicorn inverted index — keyword retrieval, precise for proper nouns and quotes
- Search semantic retriever (SSR) — 12-layer, 200M-parameter encoder; embeds the query online and posts offline
- FAISS — approximate nearest neighbor search over precomputed post vectors
- L2 ranker with multi-task multi-label (MTML) design — ranks the merged candidates on clicks, shares and comments
- Llama 3 (multimodal) as a judge — grades query-result relevance in the build verification test (BVT)
Google or a one-time email code. No password.