Problem
Domclick's chatbot and voice bot for mortgages run on an intent classifier trained on a fixed set of classes. It needs a lot of labeled data, and support keeps consuming labelers, data scientists and backend developers. Splitting or adding an intent when the product changes takes one to two weeks.
Example from the article: the message "How do I apply for a Far Eastern mortgage?" has to be matched to the Far Eastern mortgage scenario, not to the similar rural mortgage program.
Solution in brief
Analysts write reference phrases per intent. An embedder retrieves candidates for each message, with a cap per intent so several topics reach the prompt. A fine-tuned LLM picks the matching phrases, and they give the intent. Changing intents becomes editing a phrase list: half an hour to a couple of hours instead of one to two weeks.
Questions this case answers
- How to classify a message into one of many intents defined by editable reference phrases with an LLM instead of a supervised classifier, so that intents can change without retraining
- How to select reference examples for an LLM prompt so that the candidates cover several classes rather than repeating one class
- How to replace a supervised intent classifier with a few-shot LLM that picks from reference phrases
- How to add, split or remove chatbot intents without retraining a model
Tools and methods
- RAG-like few-shot classification — the LLM ranks reference phrases instead of predicting a class
- Top_K retrieval with Top_N per intent — embedder-based candidate selection that keeps the prompt diverse
- Gemma 2 9B — best of the compared open LLMs up to 9B parameters
- QLoRA SFT — adapter fine-tuning with negative examples, +4 pp macro F-measure
- llama.cpp — inference on GPU and CPU, 6.5x faster than the Hugging Face format
- E5-large-instruct — embedder, fine-tuned with masked language modeling and metric learning
Google or a one-time email code. No password.
