An LLM for a Tax Agent

Yuri Simonov25 May 2024

Problem

The DS team at a tax and audit consultancy wanted an assistant that answers tax consultants' questions from the Tax Code and related documents. A general LLM (Llama 2) gave off-topic and inconsistent answers and sometimes replied in English. Consultants could write far fewer question-answer pairs than fine-tuning needed, and they rejected answers that looked correct to the developers. A general-purpose embedding model also placed all tax questions at nearly one point, so it could not pick the right Code fragment.

Example from the article: a question must be matched to the Code fragment that answers it, but a long fragment does not fit into BERT and its vector blurs the meaning.

Solution in brief

Improve retrieval instead of the answering model. Fine-tune a small BERT to judge whether a question and a fragment belong together, and index each fragment by questions it answers, so that a user question is compared with questions rather than with long text. The LLM then answers from the retrieved fragments.

Questions this case answers
  • How to improve retrieval over long regulatory documents when a general embedding model places all questions of one domain close together
  • How to index long document fragments by the questions they answer so that a user question is matched to questions instead of to long text
  • How to build a domain-specific RAG assistant that retrieves the right document fragments for expert questions
  • How to improve retrieval quality in RAG without fine-tuning the answering LLM

Tools and methods

  • BERT fine-tuned as a pair classifier — two dense layers on top of two text vectors, binary label "belongs together".
  • Question-based indexing in Weaviate — each fragment stores vectors of questions it answers.
  • LLM question generation — extends expert questions to the rest of the Code.
  • ap@K and map@K — retrieval metrics (formulas on slide 9).
Continue reading — it's free

Google or a one-time email code. No password.