From Prompt to Classifier: A Production Case Study

Anshumachandak12 Jun 2026

Problem

Gusto needed its AI assistant to hand customers to a human agent at the right moment. The escalation rules were still changing across teams, so the classic route of collecting data, training a model and deploying would have taken months. A prompt ships quickly, but a smaller model is later faster, cheaper and more predictable, so the team wanted a path to one.

Example from the article: the team first escalated whenever a customer repeated themselves, assuming repetition meant frustration. Real conversations showed that people usually repeat themselves to add context, so the rule was removed.

Solution in brief

Ship the rules as an LLM prompt on day one and score every change on a human-labeled eval set built from production turns. When metrics stop improving and the rules settle, train a small BERT-based classifier on the labeled production turns. Keep the LLM only for the parts where rules still change.

Questions this case answers
  • How to launch a narrow decision component before its rules are stable, then replace the expensive first version with a cheaper specialized model once the problem definition settles
  • How to choose which components of a multi-step LLM system can be replaced by a small purpose-built model
  • How to ship an LLM-based decision feature on day one while the rules are still changing?
  • How to know when a prompt is ready to be replaced by a smaller classifier?

Tools and methods

  • LLM prompt — first production version of the escalation decision
  • Human-labeled eval set — about 3,000 turns from about 500 sessions, used as a regression test for every prompt change
  • BERT-based binary classifier — turn-level escalate vs. no_escalation model that replaces the prompt
  • Recall-targeted threshold — cutoff tuned to recall of at least 0.90 instead of 0.5
  • Shadow mode — safe rollout of the classifier
Continue reading — it's free

Google or a one-time email code. No password.