AutoJudge: Dynamic Generation of LLM-as-Judge for Validating Agentic Pipelines

Alina Zhidkovskaya23 May 2026

Problem

Teams ship agentic systems built on different frameworks, with different tools (custom, library, whole MCP servers) and tasks. Each needs validation. Task-specific benchmarks such as GAIA (328 samples) or SWE-bench need human ground truth and cover only their own tasks. A fixed LLM-as-judge from a paper comes with one error taxonomy, tuned to benchmarks skewed toward cognitive errors, so it transfers badly.

Example from the article: in Who&When, which has no ideal traces, judges found a problem even in a trivial trace with only a calculator and a simple expression.

Solution in brief

A meta-agent (PoolGenerator) reads a summary of the trace and designs a team of 3–9 specialized judges plus a final aggregator for the chosen error taxonomy and output format. Only the taxonomy, examples and output schema change between systems, so no tuning is needed. Long traces are summarized, and judges fetch the raw steps from a database through a tool.

Questions this case answers
  • How to validate a multi-agent LLM pipeline when no existing benchmark matches its tools and tasks and no ground-truth traces exist
  • How to locate which agent and which step broke a long multi-agent trace without overflowing the judge's context window
  • How to validate an agentic pipeline when no benchmark matches its tools and tasks
  • How to build an LLM-as-judge that adapts to a new error taxonomy without prompt tuning

Tools and methods

  • PoolGenerator — LLM meta-agent that writes the judge configuration (JSON) for a trace
  • JudgePipeline — specialized judges and a FINAL_AGGREGATOR that gives the verdict
  • Trace summarization with a retrieval tool — shortens traces and keeps the raw steps in a database
  • Gemini 3 Flash Preview / Gemini 2.5 Flash — PoolGenerator and judge models
  • Who&When, TRAIL, AEGIS, AgentErrorBench, AgentRewardBench, AgenTracer — benchmarks for judge quality
Continue reading — it's free

Google or a one-time email code. No password.