Method

Human / expert labeling

Manual annotation with guidelines and expert review.

58 cases from 39 companies apply Human / expert labeling. 8 were checked in an online A/B test; 26 report results in production. Includes variants: Overlap annotation.

Typical tasks: Evaluate models, Label data with LLMs, Moderate content at scale, Catch fraud and abuse and 11 more. Companies: Alfa-Bank, Celsus, Yandex, Airbnb, Avito, Sber and 33 more. Often combined with: Overlap annotation, LLM-as-a-judge, Golden set / benchmark, Supervised fine-tuning, Human in the loop and 110 more.

Building Gavel: AI evaluation infrastructure across Eternalfood-delivery2026-10
The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scalemedia-streaming2026-09
Eval-driven development: Lessons from evaluating GenAI at scaletravel2026-08
Automating Search Relevance Assessment at Scale with LLM-as-a-Judgee-commerce2026-08
Calibrating LLM-Based Population Estimates with Human Validationmarketplace2026-08
Behind the Curtain: How We Do Eval in Geniegeneral2026-06
How we build ML pipelines for pathomorphology: data, annotation, agentshealthtech2026-05
Finding the Right Organization in a Photo, or Removing Kul Sharif from the Kazan Cat Gallerygeneral2025-11
Exploring How to Improve Assessment with AIedtech2025-10
Turbocharging Customer Support Chatbot Development with LLM-Based Automated Evaluatione-commerce2025-06
How LLM Automates Work at Avitoclassifieds2025-05
How to Find Students' Opinions and Not Drown in Spam?edtech2025-05
YandexART: Humans and Neural Networks in Evaluating Generative Modelssearch-engine2025-05
Complex and Urgent: Cutting Labeling Costs Without Losing Qualitytelecom2025-05
How We Built a Smart Assistant in Lavka on Top of YaGPTfood-delivery2025-05
How We Created the CV Product and Didn't Burn Outmanufacturing2025-05
Science Like Industry. MLOps for the effective R&D development and deployment of computer vision models in industrial environmentsmanufacturing2025-05
Challenges of Labeling Doctor-Patient Dialogues for ASRhealthtech2025-05
POLLUX: Evaluating the Generative Capabilities of LLMs for Russiangeneral2025-05
Gigabytes of Nightmares: Turning the Maze of Ultra-Large Medical Images into a Path to Success in Eosinophilic Esophagitishealthtech2025-05
How We Build Avito's Main Product Metricclassifieds2025-05
Trailers in Yandex Music from an ML Engineer's Point of Viewmedia-streaming2025-05
RoBERTa Model for Merchant Categorization at Squarefintech2025-03
The ML flywheel: How we continually improve our models to reduce card testingfintech2024-12
How DoorDash leverages LLMs for better search retrievalfood-delivery2024-11
LLM in Education: How We Created the Code Guardianedtech2024-05
When a Vision Transformer Really Came in Handymanufacturing2024-05
Supervised Labeling: When the Rulebook Doesn't Workhealthtech2024-05
Neural Filter with YaGPT: Helping Users Sort Out Their Mailboxgeneral2024-05
Review and Adaptation of RAG Evaluation Tools for the Russian Languageinsurance2024-05
How We Fought Spam in Reviewsmarketplace2024-05
The Field Guide to Non-Engagement Signalssocial2024-03
Evaluating the Success of Consumer Generative AI Productssocial2024-03
Improving the reliability of generative models: an approach to evaluating responses and data preparationsearch-engine2023-10
Putting the Voice of Customers into the Software Development Processmarketplace2023-10
How we used machine learning to fight fraud at BlaBlaCar — Part 1mobility2023-06
How we built our Machine Learning Pipeline to fight fraud at BlaBlaCar — Part 2mobility2023-06
NLP at Alfa-Bank, Part 4: The Development History of a Chatbotbanking2023-05
A Job Posting Summary Generation Service Based on an LM Fine-Tuned with RLHFclassifieds2023-05
Creating the Conditions to Launch a Recommender System with Manual Labelingsocial2023-05
Clean Labels for MLsearch-engine2023-05
History of the Unified Classifier Developmentbanking2023-05
NLP at Alfa-Bank, Part 7: Analyzing Customer Feedbackbanking2023-05
Development History of Voice Bot: Legacy Voice Bot of Alfa-Bank and Transition to Modern Methodsbanking2023-05
Learning annotator's style in medical imaging—healthtech2023-05
NLP at Alfa-Bank, Part 2: Data Labelingbanking2023-05
NLP at Alfa-Bank, Part 9: Structuring OFD Receiptsbanking2023-05
Majority vote for CV annotations: improve your data labeling pipelinegeneral2023-05
Experts in-the-Loop at Stitch Fixe-commerce2022-09
Powering ML-Based Systems With Reliable Data: The Data Annotation Journeytravel2022-09