Improving a Modular Chatbot with LLMs: Practical Experience

Anton Gavrilov24 May 2025

Problem

The Raiffeisenbank support chatbot is a modular system: intent classifier, entity recognizer, rule-based dialogue policies and predefined responses. Follow-up questions are hard for it, because the old policies see only previous intents and entities. Each follow-up scenario needs hand-written rules, and some grew to one or two thousand lines. Moving straight to AI agents is risky: heavy development, sensitive banking data that requires open models on own infrastructure, and hard-to-guarantee quality.

Example from the article: a customer asks about a credit card and then writes "what is the rate?". Almost every banking product has a rate, so the old policy cannot answer without context rules.

Solution in brief

Keep the architecture and let an open-source LLM take over single decisions. The model picks a response identifier from a fixed pool using the full dialogue, so it cannot hallucinate text. It also checks an intent before the bot asks a clarifying question. Each use case has its own benchmark and a rule-based fallback.

Questions this case answers
  • How to improve a rule-based dialogue system on follow-up questions with a language model that selects a pre-approved response by ID, without rewriting rules
  • How to benchmark open-source language models on the system's own tasks and keep a rule-based fallback when inference fails
  • How to improve a rule-based support chatbot with an LLM without rebuilding it as an agent
  • How to pick a ready-made response for a follow-up question using the dialogue history, without writing intent chains and rules

Tools and methods

  • LLM dialogue policy — picks a pre-approved response ID using the dialogue history.
  • Structured output (Pydantic, vLLM guided_json) — schema with an enum of allowed IDs, used on retry.
  • Dialogue clustering plus LLM-written responses — covers new topics.
  • Zero-shot intent check — the LLM returns 1 or 0 before a clarifying question.
  • Per-task benchmark and leaderboard — compares open-source models of 24-32B.
Continue reading — it's free

Google or a one-time email code. No password.