YandexART: Humans and Neural Networks in Evaluating Generative Models

Egor Badika24 May 2025

Problem

A team training an image-generation model (YandexART) needs human labels to pick training parameters and datasets, average weights and run experiments. The task is side-by-side (SbS): the annotator compares two images for one prompt. There is no ground truth. The plain question "which image is better" gives noisy labels, because tastes differ, a long instruction confuses people, and the answer does not show where the model is weak. Tasks run in thematic pools (only animals in a row, say), so control tasks stand out and cannot check annotators.

Example from the article: a basket of 1500 prompts across categories such as transport, people, food and animals is labeled in SbS by Yandex Crowd annotators.

Solution in brief

Split one SbS question into aspects (relevance, complexity, aesthetics, defectiveness) and check annotators through moderators instead of control tasks. Let annotators appeal checks, validate tasks against illogical answers, and move routine checks to a VLM and YandexGPT.

Questions this case answers
  • How to evaluate generated images with crowd side-by-side labels when there is no ground truth and annotators cannot be checked with fixed control tasks
  • How to filter copy-pasted and generic comments in subjective preference labeling and cut the cost of checking crowd labels with a model
  • How to evaluate generated images with crowd side-by-side labeling when there is no ground truth
  • How to control annotator quality when tasks come in thematic pools and cannot be mixed with control tasks

Tools and methods

  • Four-aspect SbS — relevance, complexity, aesthetics, defectiveness, plus a preference question
  • Moderation by trusted annotators — daily sample and deferred honeypots; quality skill over about 40 recent checks
  • Appeals — annotators dispute a check and a second moderation round decides
  • Task validation and conditional logic — blocks impossible answer combinations
  • VLM question answering — YandexGPT writes questions and answers, the VLM answers them on generated images
  • Comment filter — Levenshtein distance, YandexGPT judge, moderators
Continue reading — it's free

Google or a one-time email code. No password.