How to Safely Roll Out New Versions of Product AI Agents Using an Auto-Metrics System

Dmitry Korshunov23 May 2026

Problem

A team at Yandex builds Yandex.AI, an agentic shopping chat for the Turkish market. The agent searches the web with tools, compares prices and reviews, and generates product cards. The team ships many iterations and has to move fast. The model invents its own search queries and tool quality varies, so a new version can get worse unnoticed. The team needs a quick way to tell whether a version is safe to release.

Example from the article: in Turkey inflation makes prices change so often that a page refreshed two hours ago can already show a wrong price, so the agent updates prices at runtime. An answer can also contain only true facts and still not solve the user's problem.

Solution in brief

Score every agent answer with automatic metrics, mostly pointwise LLM-as-a-judge scores. Three matter most: query solved, answer grounded in tool sources, sources reliable. To decide on a release, drop the metrics that did not change beyond noise and decide on the rest. The same metrics can filter trajectories for rejection fine-tuning.

Questions this case answers
  • How to decide whether a new version of an LLM agent with tools is safe to release using automatic per-answer quality scores
  • How to select the best few of many fine-tuned LLM candidates quickly using automatic judge scores
  • How to decide whether a new version of a product AI agent is safe to release
  • How to evaluate the quality of an agentic chat answer automatically, without human labeling

Tools and methods

  • LLM-as-a-judge — pointwise scores for task completion, groundedness and source reliability
  • Classic ML classifiers and algorithmic metrics — answer language, token count, response time
  • Statistical significance filter — discard metric changes that are not significant before deciding
  • Rejection fine-tuning — generate many trajectories, keep the best by metrics, run SFT (supervised fine-tuning)
Continue reading — it's free

Google or a one-time email code. No password.