Problem
Nextdoor's "New and Trending" email shows one post, and its subject line was the post's first words, often a greeting such as "Hello!". The team wanted an LLM to write better subject lines and raise clicks and sessions. ChatGPT API lines lost to the user-written ones in a small A/B test: they read like marketing copy, looked like spam, and sometimes ignored the post. Prompt iterations did not fix it.
Example from the article: for a post "Sun bathing ☀️", ChatGPT API wrote "Soak Up the Sun: Tips for Relaxing Sun Bathing Sessions", which the post does not support. Another generated line got only 56% of the clicks of the user-written line.
Solution in brief
Ask the LLM to extract the most interesting words from the post without rewriting them. Then score the candidate and the original with a reward model, a classifier fine-tuned on click data from randomized sends. Use the generated line only when the reward model prefers it. Cache one result per post.
Questions this case answers
- How to ship generated content only when it beats the existing content on engagement, when off-the-shelf generation loses to human text and manual labeling cannot say which is better
- How to build and keep fresh a reward model for engagement from randomized exposure data while serving cost stays low
- How to make LLM-written email subject lines get more clicks than the text users wrote themselves?
- How to decide automatically, per item, whether a generated text or the original is more engaging?
Tools and methods
- OpenAI API (ChatGPT API) — extracts the candidate subject line, no fine-tuning
- Reward model (fine-tuned "ada") — predicts whether a candidate beats the user-written subject on clicks
- Rejection sampling — accepts the generated line only if its reward score beats the original's score
- A/B tests with control buckets — collect labels and monitor accuracy
Google or a one-time email code. No password.