Evaluating the Success of Consumer Generative AI Products

Bonnie Barrilleaux6 Mar 2024

Problem

A team launches a generative AI feature and needs to know whether the output is good and whether the feature helps users. Output quality and product value are different things, and no single metric shows both. Human judgment is accurate but covers few outputs. User feedback covers more but is skewed. Usage numbers cover everyone but do not explain why they move.

Example from the article: LinkedIn evaluated three launches this way: AI-written collaborative articles, an AI-powered Premium chat experience, and personalized profile writing suggestions.

Solution in brief

Stack three measurements that trade coverage against fidelity: human review of sampled outputs, in-product feedback, and usage metrics. Choose the usage metrics by how members interact with the output (they only read it, chat with it, or edit and accept it) and by how often the task recurs.

Questions this case answers
  • How to evaluate the quality and value of generated content when no single metric covers both output quality and user outcomes
  • How to choose product metrics for a feature whose usage pattern is one-off, recurring or conversational
  • How to measure whether a new generative AI feature is good enough to launch and keep running?
  • How to combine human review, thumbs feedback and usage metrics into one evaluation setup?

Tools and methods

  • Human review — annotators score a diverse sample of outputs against written quality guidelines
  • In-product feedback — thumbs up/down and free-text reactions in chat
  • Usage metrics — contributions, retention, conversation depth, acceptance rate, edit distance
Continue reading — it's free

Google or a one-time email code. No password.