Method

LLM labeling

Using LLMs to annotate data or pre-label for humans.

35 cases from 25 companies apply LLM labeling. 7 were checked in an online A/B test; 15 report results in production.

Typical tasks: Label data with LLMs, Evaluate models, Rank search results, Learn user and item embeddings and 11 more. Companies: Yandex, Pinterest, DoorDash, Avito, Indeed, VTB and 19 more. Often combined with: LLM-as-a-judge, Knowledge distillation, Gradient boosting, RAG, LLM classification / extraction and 83 more.

How Pinterest Improved Analytics Agent Compliance with Loop Engineeringsocial2026-11
Unsupervised Topic Discovery with Text Embeddingsfintech2026-10
Calibrating LLM-Based Population Estimates with Human Validationmarketplace2026-08
Building Food Metadata with LLM Juries, Context Optimization & Multimodal AIfood-delivery2026-07
Fox One: Video Content Personalization Powered by GenAImedia-streaming2026-06
Emotional Congruence at Scale: Using Databricks AI to Align Ads and Contentmedia-streaming2026-06
AgentOps in Production: Infrastructure for LLM Agents and Support Automation for Yandex Personal Servicesmedia-streaming2026-05
An AI Tutor and Methods for Evaluating Itedtech2026-05
An Automated Request Handling Service for SMBsbanking2026-05
DashCLIP: Leveraging multimodal models for generating semantic embeddingsfood-delivery2026-02
LLM-Powered Relevance Assessment for Pinterest Searchsocial2025-12
How Pinterest Built a Real‑Time Radar for Violative Content using AIsocial2025-12
Reducing Irrelevant Impressions of the Thematic Block in Geo Search for Navigation Scenariossearch-engine2025-11
Automating the Labeling of News Queries with RAGsearch-engine2025-11
Understanding Etsy's Vast Inventory with LLMsmarketplace2025-10
Building the next generation of job search at LinkedInmarketplace2025-05
Description Generation One Year Later. What We Managed to Understand Over the Yearclassifieds2025-05
Cold-Starting Sales of Complementary Products with LLMsfood-delivery2025-05
Internships on the LLM Team: Several LLM Deployments at Avitoclassifieds2025-05
Complex and Urgent: Cutting Labeling Costs Without Losing Qualitytelecom2025-05
Estimating Generative Model Confidence in Data Labelinggeneral2025-05
How We Built a Smart Assistant in Lavka on Top of YaGPTfood-delivery2025-05
ML-Readability: How Yandex Browser Extracts Web Page Content for Summarizationsearch-engine2025-05
POLLUX: Evaluating the Generative Capabilities of LLMs for Russiangeneral2025-05
An Effective Approach to Quality Control of Anti-Fraud Modelsmarketplace2025-05
Single AI View of Customer: A retailers guide to Know Your Customer Better using Customer Embeddings and Customer Personaretail2025-03
LLMs vs. Annotators: How to Get High-Quality Text Labelingadtech2024-11
Unleashing the power of large language models at DoorDash for a seamless shopping adventurefood-delivery2024-10
LLM-based Approach to Large-scale Item Category Classificationmarketplace2024-05
From receipt to catalog: extracting properties from product namesretail2024-05
Data Labeling: Experts, Crowdsourcing and Large Language Modelsgeneral2024-05
Using AI to Understand the Complexities and Pitfalls of Real Estate Datamarketplace2024-04
The Field Guide to Non-Engagement Signalssocial2024-03
Extracting Job Roles in Job Ads: A Journey with Generative AIclassifieds2024-01
SOTA Entity Recognition via LLM Distillationgeneral2023-10