Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling

Jaewon Yang16 Oct 2023

Problem

Nextdoor's "New and Trending" email shows one post, and its subject line was the post's first words, often a greeting such as "Hello!". The team wanted an LLM to write better subject lines and raise clicks and sessions. ChatGPT API lines lost to the user-written ones in a small A/B test: they read like marketing copy, looked like spam, and sometimes ignored the post. Prompt iterations did not fix it.

Example from the article: for a post "Sun bathing ☀️", ChatGPT API wrote "Soak Up the Sun: Tips for Relaxing Sun Bathing Sessions", which the post does not support. Another generated line got only 56% of the clicks of the user-written line.

Solution in brief

Ask the LLM to extract the most interesting words from the post without rewriting them. Then score the candidate and the original with a reward model, a classifier fine-tuned on click data from randomized sends. Use the generated line only when the reward model prefers it. Cache one result per post.

Questions this case answers
  • How to ship generated content only when it beats the existing content on engagement, when off-the-shelf generation loses to human text and manual labeling cannot say which is better
  • How to build and keep fresh a reward model for engagement from randomized exposure data while serving cost stays low
  • How to make LLM-written email subject lines get more clicks than the text users wrote themselves?
  • How to decide automatically, per item, whether a generated text or the original is more engaging?

Tools and methods

  • OpenAI API (ChatGPT API) — extracts the candidate subject line, no fine-tuning
  • Reward model (fine-tuned "ada") — predicts whether a candidate beats the user-written subject on clicks
  • Rejection sampling — accepts the generated line only if its reward score beats the original's score
  • A/B tests with control buckets — collect labels and monitor accuracy
Continue reading — it's free

Google or a one-time email code. No password.