Method

Human in the loop

Human review or approval inside an automated pipeline.

62 cases from 35 companies apply Human in the loop. 8 were checked in an online A/B test; 28 report results in production.

Typical tasks: Evaluate models, Catch fraud and abuse, Moderate content at scale, Answer questions over data and 17 more. Companies: DoorDash, LinkedIn, Gusto, Ramp, Thumbtack, Netflix and 29 more. Often combined with: LLM-as-a-judge, Threshold selection, Rule-based heuristics, Embedding similarity matching, Golden set / benchmark and 105 more.

Composing Fraud Rules at the Speed of the Attackfintech2026-11
Agentic Credit Analysisfintech2026-10
Inside LinkedIn's data context layer: curating trustworthy context for humans and agentssocial2026-10
Workbench: Building an AI Content Generation and Quality Assurance System at Scalemarketplace2026-10
Inside Vera, DoorDash’s Data Agentfood-delivery2026-09
The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scalemedia-streaming2026-09
Agentic Risk Operationsfintech2026-06
Building Query Expert MCP: Using RAG to Build an Analytics Agent That Goes Beyond Text-to-SQLfintech2026-06
Insights at Scale With Agentic AI: How AI Ready Data is Transforming Commercial Life Scienceshealthtech2026-06
Forecasting at Databricks: One Framework Behind Consumption and Infra Costsaas-b2b2026-06
Behind the Curtain: How We Do Eval in Geniegeneral2026-06
BI Locked Semantic Models are Dead! Open Semantics Just Entered the Chatretail2026-06
Bulk Metadata Generation To Bring Genie to Lifetelecom2026-06
How Hilton Grand Vacations Optimizes Inventory with AI on Databrickstravel2026-06
Reliance Jio Customer Churn Radar and next best actiontelecom2026-06
Sponsored by: Omni | AI analytics at Condé Nast: What it took to get to productionadtech2026-06
Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistantmedia-streaming2026-06
Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAImarketplace2026-04
Beyond the dashboard: how BlaBlaCar PMs use AI to self-serve datamobility2026-03
Ranking Engineer Agent (REA): The Autonomous AI Agent Accelerating Meta’s Ads Ranking Innovationadtech2026-03
Our Early Journey to Transform Instacart’s Discovery Recommendations with LLMse-commerce2026-02
Universal Document Processing at Gusto: From One-Off Parsers to a Self-Service Platformsaas-b2b2026-02
It Takes a Village: Building Gusto’s First AI Risk Agentsaas-b2b2026-01
SafeChat: DoorDash's AI-powered safety featurefood-delivery2025-10
A scalable LLM approach to enhancing chatbot knowledge with user-generated contentfood-delivery2025-08
Creating AI agent solutions for warehouse data access and securitysocial2025-08
Improving Accessory Recommendations with LLMs at Targetretail2025-08
Scaling Catalog Attribute Extraction with Multi-modal LLMse-commerce2025-08
Automating Ad Generations with LLMsmarketplace2025-07
How To Build Agents Users Can Trustfintech2025-07
Advancing Menu Content with AI: How DoorDash uses AI to generate menu descriptionsfood-delivery2025-06
SRE2.0: No LLM Metrics, No Future: Why SRE Must Grasp LLM Evaluation Nowgeneral2025-06
Labeling Product Cards with LLMse-commerce2025-05
Function Calling 101: One Agent Instead of a Zoo of Bots—general2025-05
From Chaos to Control: Building a Streamlit Tracer for Debugging RAG Systemsgeneral2025-05
5 Stops on the Way to LLM in Your Productiongeneral2025-05
How DoorDash leverages LLMs to evaluate search result pagesfood-delivery2025-04
Leveraging Knowledge Graphs in Real Estate Searchclassifieds2025-04
Using LLM to transcribe restaurant menu photosfood-delivery2025-03
Tackling AI Hallucinations in LLM Appssaas-b2b2025-02
Image replacement in Canva designs using reverse image searchgeneral2025-01
Path to high-quality LLM-based Dasher support automationfood-delivery2024-09
Tackling the Human Cost of R&D Tax Credit Qualification with AI-Powered Modelssaas-b2b2024-09
How Asana tests frontier LLMs: our analysis of Claude 3.5 Sonnetsaas-b2b2024-07
How We Fought Spam in Reviewsmarketplace2024-05
Moderating Inappropriate Video Content at Yelpmarketplace2024-03
Augmenting our content moderation efforts through machine learning and dynamic content prioritizationsocial2023-11
How Whatnot Utilizes Generative AI to Enhance Trust and Safetymarketplace2023-09
Diagnosing Oil and Gas Pipelines Using Machine Learning—manufacturing2023-05
Viral spam content detection at LinkedInsocial2023-04