Method
LLM simulation / role-play
LLM personas, digital twins and simulated users or annotators.
12 cases from 10 companies apply LLM simulation / role-play. 8 report results in production.
Typical tasks: Evaluate models, Label data with LLMs, Get experiment results faster, Predict long-term effect from short tests and 2 more. Companies: DoorDash, Zomato, Spotify, Celsus, Instacart, Yandex and 4 more. Often combined with: LLM-as-a-judge, Synthetic data generation (LLM), LLM labeling, Human / expert labeling, Few-shot prompting and 32 more.
| Building Gavel: AI evaluation infrastructure across Eternal | food-delivery | 2026-10 | |
| When Can LLMs Replace Humans in A/B Tests? | media-streaming | 2026-08 | |
| Building Ask DoorDash (Part 3): Evaluation | food-delivery | 2026-06 | |
Fox One: Video Content Personalization Powered by GenAI![]() | ![]() | media-streaming | 2026-06 |
| Scaling Micro-Offers With Hybrid AI | banking | 2026-06 | |
| Inside DoorDash’s one-click simulation and evaluation platform for support chatbots | food-delivery | 2026-06 | |
| An AI Tutor and Methods for Evaluating ItMD | MD | edtech | 2026-05 |
A Synthetic Training Pipeline for Medical LLMs![]() | ![]() | healthtech | 2026-05 |
Analyzing Public Pedagogical Benchmarks for AI Tutors and Building Synthetic Students for Testing![]() | ![]() | edtech | 2026-05 |
| A Simulation and Evaluation Flywheel to Develop LLM Chatbots | food-delivery | 2026-01 | |
| Turbocharging Customer Support Chatbot Development with LLM-Based Automated Evaluation | e-commerce | 2025-06 | |
Estimating Generative Model Confidence in Data Labeling![]() | ![]() | general | 2025-05 |



