Problem
Keyword search on a visual, multilingual marketplace has low recall. Queries that return no results, even though relevant items exist, are a missed business opportunity. Adding dense retrieval to the live engine breaks things users notice. Vector matches flood the results, a filter raises the number of hits, and approximate search times out into zero-result sessions. Vinted ran about 50 A/B tests before enabling it for all searches.
Example from the article: the query zx750 with a brand filter had about 300 lexical matches. The tuned distance threshold added about 7000 mostly random nearest-neighbor items. Without the filter the same query returned about 450 matches (about 300 lexical and about 150 nearest neighbor).
Solution in brief
A two-tower model embeds queries and items into one 256-dimension space, and an HNSW index in Vespa serves the neighbors. The ranking profile detects vector-only matches and cuts them to a fixed count. A split latency budget with an exact-search retry removes most timeouts.
Questions this case answers
- How to add vector retrieval to a lexical search engine so that the number of vector-only matches stays bounded under filters and re-sorting
- How to keep approximate nearest neighbor search within a hard latency budget when timeouts produce empty results
- How to add embedding-based retrieval to a keyword search without flooding results with poor matches?
- How to cap the number of nearest-neighbor matches when filters and sort orders change the search strategy?
Tools and methods
- Two-tower model, contrastive training — query and item embeddings, 256 dimensions.
- CLIP — frozen multilingual query encoder, precomputed image embeddings.
- Vespa with HNSW — filtered nearest neighbor search, hosts both models.
- ONNX — item tower export with preprocessing inside the graph.
- Reciprocal rank fusion (RRF) score — turns similarity rank into a position threshold.
- Global-phase
rank-score-drop-limit— drops excess vector-only matches.
Google or a one-time email code. No password.