Problem
Thumbtack, a marketplace for local service professionals, wanted a locally grounded description on about 500K search landing pages, with four themed variants per page. Writing the texts and reviewing them for content design, brand, legal and SEO rules by hand was not feasible at that volume. The 200K-page pilot meant 800K drafts to generate and check. Example from the article: for the search "duct cleaning" in "Raleigh, NC", a first draft of two short sentences was rejected for missing service scope, local context and hedging, and the regenerated draft passed.
Solution in brief
Workbench is a config-driven loop. It generates a draft, checks it with deterministic validators plus an LLM judge that scores 11 dimensions, and regenerates failed drafts with the judge's feedback attached, up to 3 retries. A small share of approved output is audited by humans to keep the judge calibrated.
Questions this case answers
- How to generate and quality-check content at a volume where human review of every item is impossible
- How to turn an LLM judge's rejections into feedback that raises the approval rate of regenerated items
- How to generate hundreds of thousands of landing page texts and check their quality without reading each one by hand?
- How to make an LLM judge reject bad drafts and give feedback that the next generation attempt can use?
Tools and methods
- LLM-as-a-judge — scores each draft per dimension and returns structured feedback
- Feedback-driven retry — passes failing scores and suggestions into the next generation prompt
- Programmatic validators — length, word count and keyword checks in code, run before the judge
- Human audit sample — 5% of approved drafts reviewed to calibrate the judge
- LangGraph — state machine for the serial pipeline
- OpenAI Batch API — generation and judging for 100K+ drafts per run
- MLflow — optional per-criterion scorers, off by default
Google or a one-time email code. No password.