Problem
The Raiffeisenbank support chatbot is a modular system: intent classifier, entity recognizer, rule-based dialogue policies and predefined responses. Follow-up questions are hard for it, because the old policies see only previous intents and entities. Each follow-up scenario needs hand-written rules, and some grew to one or two thousand lines. Moving straight to AI agents is risky: heavy development, sensitive banking data that requires open models on own infrastructure, and hard-to-guarantee quality.
Example from the article: a customer asks about a credit card and then writes "what is the rate?". Almost every banking product has a rate, so the old policy cannot answer without context rules.
Solution in brief
Keep the architecture and let an open-source LLM take over single decisions. The model picks a response identifier from a fixed pool using the full dialogue, so it cannot hallucinate text. It also checks an intent before the bot asks a clarifying question. Each use case has its own benchmark and a rule-based fallback.
Questions this case answers
- How to improve a rule-based dialogue system on follow-up questions with a language model that selects a pre-approved response by ID, without rewriting rules
- How to benchmark open-source language models on the system's own tasks and keep a rule-based fallback when inference fails
- How to improve a rule-based support chatbot with an LLM without rebuilding it as an agent
- How to pick a ready-made response for a follow-up question using the dialogue history, without writing intent chains and rules
Tools and methods
- LLM dialogue policy — picks a pre-approved response ID using the dialogue history.
- Structured output (Pydantic, vLLM
guided_json) — schema with anenumof allowed IDs, used on retry. - Dialogue clustering plus LLM-written responses — covers new topics.
- Zero-shot intent check — the LLM returns 1 or 0 before a clarifying question.
- Per-task benchmark and leaderboard — compares open-source models of 24-32B.
Google or a one-time email code. No password.