Problem
New frontier models appear often. Each changes latency, answer quality, formatting and agent behavior, and a product team must decide fast whether to switch. Public benchmarks do not show how a model behaves on the product's own data, such as long project contexts. LLM output also differs between runs, so a plain string or regular-expression check falls short.
Example from the article: Asana was an early tester of Claude 3.5 Sonnet and ran its QA process on it in a short sprint, before the production rollout.
Solution in brief
Asana tests a candidate model in layers. Fast LLM-judged unit tests and integration tests run during development. Slow end-to-end tests on realistic sandbox data are graded by a product manager. Latency scripts and an agent benchmark are added for new models. Asana says this gives a first assessment of a new model in under an hour.
Questions this case answers
- How to evaluate a candidate model for an existing LLM-powered product within hours, when outputs are non-deterministic and style and insight cannot be scored automatically
- How to decide quickly whether a new frontier LLM can replace the current model in product features?
- How to test LLM answers automatically when the output differs on every run?
- How to test chained prompts and agentic tool use before a release?
Tools and methods
- LLM unit tests — assertions checked by an LLM call, repeated per test
- Integration tests — whole prompt chains, including agentically selected tools
- PM-graded end-to-end tests — sandbox data, grades A to F
- Latency scripts — time to first token (TTFT) and tokens per second (TPS)
- Tool-use extractor benchmark — agent success rate
- Claude — the model under test
Google or a one-time email code. No password.