Problem
Many business documents keep key facts in charts, tables and figures, and a text-only RAG pipeline does not read them. Questions whose evidence is visual end up with missing or wrong answers. The article's workaround is to embed text summaries of images and tables instead of the raw elements, using only open-source tools.
Example from the article: the question "What is the Total revenues for FY'24 Q4?" asked over Walmart's FY24 Q4 earnings release, where the figure sits in a chart image.
Solution in brief
Split the PDF into text chunks, tables and images. Have an LLM write a short retrieval-oriented summary of each element, embed and store only the summaries, and keep the raw elements in a separate store. At query time, find the closest summaries, fetch the raw elements behind them and give the text and images to a multimodal LLM that writes the answer.
Questions this case answers
- How to retrieve non-text elements (images, tables) with a text embedding model and still answer from the original element
- How to build a document question-answering pipeline that runs entirely on local, open-source models
- How to answer questions over PDFs whose facts sit in charts, tables and figures, not only in running text?
- How to index images and tables for retrieval when the embedding model only accepts text?
Tools and methods
- Unstructured (
partition_pdf) with Poppler and Tesseract — splits a PDF into text chunks, tables and image files - Ollama with Llama 2 — runs local models; Llama 2 summarizes text and tables
- LLaVA (
llava-llama3) — summarizes images and writes the final answer from text plus images - GTE (
thenlper/gte-large, 1024 dimensions) — embedding model for the summaries - Chroma and LangChain
MultiVectorRetriever— vector store for summaries, document store for raw elements
Google or a one-time email code. No password.