Problem
At T-Bank's crowdsourced labeling platform (tens of millions of labels a month), editorial chiefs write a detailed instruction for every customer ticket before annotators start. This step takes about 20% of data collection time. Frontier LLMs out of the box did not fit: they lack the platform's constraints, terms and templates, cannot see the internal knowledge base, and internal details cannot leave the bank.
Example from the article: a customer opens a Jira ticket asking to label cats and dogs. The instruction must say what to do with a crocodile (mark that there is no cat or dog) and with a cat in a dog costume (label it as a cat).
Solution in brief
Combine a RAG knowledge base of the editors' internal materials with dynamic few-shot examples: past pairs "customer request → editor's final instruction", with the closest ones retrieved for each new ticket. An agent routes the request to a writer LLM and, if needed, a proofreader LLM. Editors edit the draft instead of starting from a blank page.
Questions this case answers
- How to generate a structured task instruction from a short request by retrieving the most similar past request-instruction pairs and internal knowledge, with all data kept inside the company
- How to evaluate a generated structured document against human-written ones with blind pairwise comparison when expert time is the bottleneck
- How to generate task instructions for data labeling with an LLM so that editors accept them
- How to use RAG and dynamic few-shot examples to turn a customer request into a structured document
Tools and methods
- RAG over internal guides — vector database with editorial policy, terms and templates
- Dynamic few-shot retrieval — most similar past tasks with their reference instructions go into the prompt
- Agent with writer and proofreader LLMs — routing, generation, compliance check
- Self-hosted LLM — production model, data is confidential
- Blind rating by editors and annotators — Precision and Recall of suitable instructions
Google or a one-time email code. No password.
