Problem
Thumbtack is a marketplace where customers hire local service professionals (pros). It uses generative AI to help customers describe needs, summarize projects and explain why a pro fits a request. The outputs are probabilistic and can fail quietly: wrong tone, unsupported or overly strong claims, misread intent, biased content, and drift after a model or prompt update. On a trust-based marketplace these failures cost credibility with both sides. Separate evaluation by each product team duplicated work.
Example from the article: AI-generated marketing content is produced at scale, so LLM judges score brand voice, SEO and legal/trust/safety, and their feedback triggers regeneration.
Solution in brief
Turn content, design and safety guidelines into rubrics. Score a representative dataset with a mix of rule-based checks, LLM judges and a small human-reviewed sample. Offer three orchestration paths (MLflow template, nightly Databricks jobs, multi-layer pipeline) so teams pick by volume. A central Evals team owns shared tooling and guidelines.
Questions this case answers
- How to evaluate free-text outputs of generative features continuously when the outputs are probabilistic and errors are subtle (tone, unsupported claims, harmful assumptions)
- How to combine automated judges with a small sample of human review so that quality gates scale with output volume
- How to evaluate the quality, tone and safety of generative AI features in a marketplace product at scale
- How to turn content and safety guidelines into rubrics that an LLM judge can score
Tools and methods
- LLM-as-a-judge (DeepEval GEval, MLflow Guidelines scorers) — rubric-based semantic scoring of clarity, tone, accuracy, groundedness
- Rule-based checks — structure, formatting, length, schema, character limits, keywords
- Crowdsourced review (Amazon Mechanical Turk) — humans validate a sample of AI-approved content
- Spreadsheet labeling — human review and judge calibration
- MLflow and Databricks nightly jobs — trace logging, score tracking, scheduled sampling, alerts to Slack
Google or a one-time email code. No password.