Problem
The DS team at a tax and audit consultancy wanted an assistant that answers tax consultants' questions from the Tax Code and related documents. A general LLM (Llama 2) gave off-topic and inconsistent answers and sometimes replied in English. Consultants could write far fewer question-answer pairs than fine-tuning needed, and they rejected answers that looked correct to the developers. A general-purpose embedding model also placed all tax questions at nearly one point, so it could not pick the right Code fragment.
Example from the article: a question must be matched to the Code fragment that answers it, but a long fragment does not fit into BERT and its vector blurs the meaning.
Solution in brief
Improve retrieval instead of the answering model. Fine-tune a small BERT to judge whether a question and a fragment belong together, and index each fragment by questions it answers, so that a user question is compared with questions rather than with long text. The LLM then answers from the retrieved fragments.
Questions this case answers
- How to improve retrieval over long regulatory documents when a general embedding model places all questions of one domain close together
- How to index long document fragments by the questions they answer so that a user question is matched to questions instead of to long text
- How to build a domain-specific RAG assistant that retrieves the right document fragments for expert questions
- How to improve retrieval quality in RAG without fine-tuning the answering LLM
Tools and methods
- BERT fine-tuned as a pair classifier — two dense layers on top of two text vectors, binary label "belongs together".
- Question-based indexing in Weaviate — each fragment stores vectors of questions it answers.
- LLM question generation — extends expert questions to the rest of the Code.
- ap@K and map@K — retrieval metrics (formulas on slide 9).
Google or a one-time email code. No password.