Problem
Grab account managers ask the assistant open questions such as "What should I tell this merchant tomorrow?" or "Which accounts need attention this week?". Such questions hide four decisions: scope, access, business diagnosis and metric definition. A fluent answer that gets one of them wrong is worse than no answer. The first Jarvis Pro prototype produced polished answers that were sometimes wrong.
Example from the article: an early answer told the account manager to push promotions for a merchant whose order volume was down. The real issue was operational: more outlets were paused and fulfilment had slipped, so extra demand would have made the merchant look worse.
Solution in brief
Classify the task first, then load only the context that task needs. The route fixes the answer structure, metrics, tool path and guardrails. Memory is checked before retrieval, metrics come from an approved catalog, and each layer (route, memory, evidence, answer) is evaluated separately.
Questions this case answers
- How to make an LLM assistant classify an ambiguous request into a task type before answering, so that each task type gets its own context, tools and safety rules
- How to evaluate a multi-step LLM pipeline offline per layer (routing, memory, tools, answer) when one aggregate score hides the failures that matter
- How to separate model errors from metric-definition, source-selection and freshness errors when users report that a number is wrong
- How to stop an LLM assistant from giving fluent but wrong business advice on account questions?
Tools and methods
- Prompted LLM router — classifies each question into a supported task type using a routing guide
- Memory decision step — decides how saved user context may be used before any answer is generated
- Approved metric catalog — the only source of metric definitions the assistant may use
- Regression checks — compare normalised daily values across approved metric paths
- Layered offline evaluation — labelled prompts for routing, a 0-100 rubric for answer quality
Google or a one-time email code. No password.