Company

Thumbtack

10 cases from Thumbtack (Marketplace, Adtech), each digested from a public talk or engineering post and linked to the original. 7 were checked in an online A/B test; 2 report results in production.

Tasks: Evaluate models, Rank search results, Measure impact without an A/B test, Get experiment results faster, Recommend from what the user just did and 3 more. Methods: LLM-as-a-judge, Human in the loop, Gradient boosting, Position bias correction, Deep & Cross Network, SHAP and 24 more. Tools: MLflow, SHAP, LangGraph, OpenAI API, PyTorch, Amazon S3 and 10 more.

Tech blog: thumbtack.com