How Asana tests frontier LLMs: our analysis of Claude 3.5 Sonnet

Bradley Portnoy1 Jul 2024

Problem

New frontier models appear often. Each changes latency, answer quality, formatting and agent behavior, and a product team must decide fast whether to switch. Public benchmarks do not show how a model behaves on the product's own data, such as long project contexts. LLM output also differs between runs, so a plain string or regular-expression check falls short.

Example from the article: Asana was an early tester of Claude 3.5 Sonnet and ran its QA process on it in a short sprint, before the production rollout.

Solution in brief

Asana tests a candidate model in layers. Fast LLM-judged unit tests and integration tests run during development. Slow end-to-end tests on realistic sandbox data are graded by a product manager. Latency scripts and an agent benchmark are added for new models. Asana says this gives a first assessment of a new model in under an hour.

Questions this case answers
  • How to evaluate a candidate model for an existing LLM-powered product within hours, when outputs are non-deterministic and style and insight cannot be scored automatically
  • How to decide quickly whether a new frontier LLM can replace the current model in product features?
  • How to test LLM answers automatically when the output differs on every run?
  • How to test chained prompts and agentic tool use before a release?

Tools and methods

  • LLM unit tests — assertions checked by an LLM call, repeated per test
  • Integration tests — whole prompt chains, including agentically selected tools
  • PM-graded end-to-end tests — sandbox data, grades A to F
  • Latency scripts — time to first token (TTFT) and tokens per second (TPS)
  • Tool-use extractor benchmark — agent success rate
  • Claude — the model under test
Continue reading — it's free

Google or a one-time email code. No password.