Problem
A team has many internal documents and wants a bot that answers questions about them. A plain LLM does not know these documents. The documents are also too long to paste into one prompt, and the model can miss key details in a long input. A basic RAG pipeline fixes this, but the author notes that naive RAG has a limited grasp of context and struggles with long texts.
Example from the article: a question-answering bot over the files in a data folder, queried with “What is HNSW?”.
Solution in brief
Split the documents into chunks, embed them and store the vectors. At query time, retrieve the closest chunks and pass them to the LLM with the question. When single chunks are too thin, switch to sentence window retrieval (match small chunks, return them with neighbouring sentences) or auto-merging retrieval (match small chunks in a hierarchy, merge them into a larger one).
Questions this case answers
- How to answer questions over a private text collection with an LLM when the collection is too large for the model context window
- How to retrieve precise passages from long documents while still giving the generator enough surrounding context
- How to build a question-answering bot over your own documents with an LLM?
- How to split documents into chunks so that retrieval does not lose the details?
Tools and methods
- LlamaIndex — builds the pipeline: reader, splitter, index, query engine
- OpenAI Embeddings and GPT-3.5 — embedding model and generator in the example
- tiktoken — token counting for the chunk splitter
- Chunking — token or sentence splitter with
chunk_sizeandchunk_overlap - Sentence window and auto-merging retrieval — two ways to return more context than the matched chunk
Google or a one-time email code. No password.