Comment ranker - An ML-based classifier to improve LLM code review quality using Atlassian's proprietary data

Yaotian Zou, Michael Gupta, Andy Wong25 Aug 2025

Problem

Atlassian's code reviewer agent asks an LLM to write review comments on every pull request (PR). Without filtering, many comments are nit-picky, noisy or factually wrong, and authors respond with negative feedback. The team had stacked heuristic filters, for example asking an LLM to categorize each comment and keeping only some categories. These raised precision but lost recall. They were also hard to adapt to a new generating model or new problem patterns.

Example from the article: when a PR opens, the agent sends the diff, PR title and description, and Jira item to an LLM, then must pick which generated comments to post in Bitbucket.

Solution in brief

Fine-tune a text classifier (ModernBERT) on generated comments. The label is whether the PR author changed the commented code afterwards. A comment is posted only if its score passes a threshold chosen in an A/B test. The model is retrained when coverage or the generator changes.

Questions this case answers
  • How to select which generated outputs to show users when the generator is noisy and the only quality signal is whether users act on an output
  • How to keep a learned quality gate accurate when the upstream generator changes or the user base widens
  • How to stop an LLM feature from posting noisy or wrong outputs that users ignore?
  • How to turn user reactions to LLM outputs into training labels for a quality filter?

Tools and methods

  • ModernBERT — open-source BERT-family encoder, fine-tuned as a binary classifier on comment text
  • Hugging Face — source of the pre-trained model and tokenizer
  • Code resolution (binary label) — "the author changed the commented code", taken from production usage
  • Propensity threshold and CRR — cut-off on the 0–1 score, tuned through A/B testing against code resolution rate (CRR), the online precision-like metric
Continue reading — it's free

Google or a one-time email code. No password.