Problem
Teams that add an LLM reviewer to pull requests need to know whether its comments are useful. Judging quality by comment count says nothing about what developers change in code. A cheap signal is needed that splits comments by type and shows which types lead to a fix.
Example from the article: Atlassian studied comments from its Rovo Dev reviewer and from human reviewers on internal and open-source repositories, and measured how often each comment type was resolved with a code change.
Solution in brief
Classify every review comment into a small taxonomy with an LLM judge validated on human labels. Then compute, per type, the share of comments followed by a code change on the commented line. The authors conclude that a reviewer should focus on the types developers resolve most.
Questions this case answers
- How to measure whether comments from an LLM assistant lead to action, split by comment type, without asking users for ratings
- How to label a large set of free-text items with an LLM using a custom taxonomy and verify the labels against human annotators
- How to find out which kinds of AI-generated code review comments developers actually act on?
- How to measure the quality of an LLM reviewer by what developers do with its comments?
Tools and methods
- LLM-as-a-judge (GPT-4.1) — assigns each comment one of four types with a justification and a confidence score
- Comment taxonomy — readability, bug, maintainability, design, plus a "no issue" class
- Resolution rate — share of comments of a type that led to a code change
- Inter-annotator agreement — checks the human labels and the judge against them
Google or a one-time email code. No password.