Method

Side-by-side evaluation

Pairwise comparison of two outputs by humans or LLM judges.

17 cases from 10 companies apply Side-by-side evaluation. 2 were checked in an online A/B test; 7 report results in production.

Typical tasks: Evaluate models, Rank search results, Build an assistant over internal docs, Label data with LLMs and 2 more. Companies: Yandex, Sber, Databricks, LinkedIn, DoorDash, T-Bank and 4 more. Often combined with: LLM-as-a-judge, Relevance labeling, Knowledge distillation, RAG, Supervised fine-tuning and 46 more.