Problem
A team building an LLM assistant over its own documents faces three questions. Is RAG needed, or is keyword search enough? How to combine keyword and vector search? What changes when users send images or audio? Plain LLMs give answers without a traceable source, use outdated data and hallucinate plausibly, but a badly chosen RAG adds cost, latency and noise in the context.
Example from the article: a question about a picture (a vase) is answered by a vision-language model with text retrieved from an English Wikipedia knowledge base.
Solution in brief
Use RAG for fresh information or a narrow domain, after trying prompt engineering. Merge BM25 and vector rankings with Reciprocal Rank Fusion. For images, keep the knowledge base textual: embed the whole image and each detected object, retrieve texts, rerank them and pass them to a vision-language model.
Questions this case answers
- How to decide whether a question-answering system over an own knowledge base needs retrieval at all, and how to merge keyword and vector rankings
- How to extend a text retrieval assistant to image and audio queries while keeping the knowledge base textual
- How to decide whether a task needs RAG or a cheaper alternative such as full-text search or a bi-encoder
- How to combine keyword and vector search with Reciprocal Rank Fusion
Tools and methods
- Reciprocal Rank Fusion (RRF) — merges ranks of several retrievers into one score.
- BM25 + embeddings — hybrid lexical and semantic search.
- ONE-PEACE — multimodal encoder: images and texts in one embedding space.
- YOLO — finds objects so each becomes a separate retrieval query.
- BLIP, TrOCR, MPNet-base — image and text-in-image captions, and cosine reranking.
- LLaVA-Next, WMT20 MLQE — vision-language answer model and translator.
- Multi-Modality Arena — benchmark used for evaluation.
Google or a one-time email code. No password.
