How Asana built a custom LLM evaluation framework for AI Teammates

Nathan RainaDeGraaf8 Apr 2026

Problem

A team that ships an LLM agent has to pick a model, and the choice moves quality, latency, cost and tone at once. New models appear faster than the team can test them, and release pages say little about its use case. One agent run can make up to seven types of LLM calls, each with its own prompt and its own balance of trade-offs. Public benchmarks miss the product task, and prompts tuned for the old model can make a new one look worse than it is.

Example from the article: Asana's AI Teammates are agents that work inside shared team spaces. Asana needs low-cost, repeatable ways to evaluate new models for them.

Solution in brief

Build benchmarks from customer feedback, grouped into capability categories. Run four stages: code-run evals per component, end-to-end benchmarks on the full agent (fixing prompts before judging a model), a roll-up of quality, cost and latency per call type, and internal side-by-side use.

Questions this case answers
  • How to select a model for each step of a multi-call LLM agent when quality, latency and cost trade off differently per step and public benchmarks do not reflect the product task
  • How to build a repeatable evaluation pipeline that turns user feedback into benchmarks and re-runs them on every new model
  • How to choose and re-choose an LLM for a product agent every time a new model is released?
  • How to build benchmarks for an agent from customer feedback instead of generic public benchmarks?

Tools and methods

  • Customer-feedback-derived benchmarks — test cases grouped into four capability categories
  • Code-run evals — sandboxed scenarios with deterministic checks plus LLM-graded checks
  • End-to-end benchmarks — repeated full-agent runs, AI-graded and human-verified
  • Roll-up analysis — quality, cost and latency summary per model configuration and call type
  • Internal dogfooding — candidate models compared side by side in real work
Continue reading — it's free

Google or a one-time email code. No password.