How Pinterest Improved Analytics Agent Compliance with Loop Engineering

Skyler Zheng, Chris Moradi, Lu Liu, Aman Gairola, Michael Chau12 Nov 2026

Problem

Pinterest's Analytics Agent answers data questions and must follow the steps that apply to each one: read the data-discovery skill, inspect table metadata before writing SQL, search every relevant source type. When it skips a step, it may stop at the first table, miss a required filter or ignore a certified metric, and still sound confident. Manual instruction edits did not fix this: after revisions the discovery-skill rule was followed in only 64% of a broader set of chats, and nobody could predict whether an edit would generalize or break something else.

Example from the article: the agent reports "latency is down 4.2%" using the wrong definition, pipeline stage or reporting window (illustrative, not Pinterest data).

Solution in brief

For each compliance rule, build two test suites from real chats: one where the rule applies, one where it does not. An automated loop drafts prompt, skill or tool edits and keeps one only if it passes three independent checks. Each iteration starts a fresh session that reads a compact handoff of earlier results.

Questions this case answers
  • How to improve an LLM agent's adherence to required procedural steps when the steps apply only to some questions and manual prompt edits break other behavior
  • How to accept or reject an automated change to an LLM agent when evaluation scores move by only a few cases and model output is noisy
  • How to make an LLM analytics agent reliably follow the right steps (read metadata, search every source type) only when they apply?
  • How to tell whether a change to an agent's prompt or tools really helped, given that 10% of test cases flip verdict between identical runs?

Tools and methods

  • Paired required/skip suites — test both following a rule and not over-applying it
  • LLM labeler with human review — labels applicability and compliance
  • Tuning loop with structured handoff — proposes, tests and learns without growing context
  • Repair/break gate, acceptance critic, regression guard — reject overfit or harmful edits
  • Claude — base model of the agent
Continue reading — it's free

Google or a one-time email code. No password.