GitHub’s ReviewBench Grades AI Code Reviewers on 219 Real Pull Requests

GitHub released ReviewBench on October 5, 2026, a benchmark for AI agents that review pull requests. Its corpus is 219 pull requests from 187 public open-source-licensed repositories in 19 languages, and it is available now as a research preview. Michelle Zhou and Alejandro Carderera de Diego wrote the launch post. They say the benchmark is meant to fill a gap where existing evaluations “often make tradeoffs between label quality, coverage, and how well they represent real-world code review.” The dataset, rubric, judge prompt, judge model configuration, and a self-serve runner are all public, so teams can test their own reviewers against the same setup as everyone else.

A corpus shaped like GitHub’s pull requests

To set the mix, GitHub first analysed 103.9 million pull requests to see how review workloads are actually distributed. ReviewBench’s language and repository-size distributions follow GitHub overall. Pull request size is the one deliberate exception. The authors weighted the corpus toward “the reviewable middle and tail,” which cuts back on tiny single-file changes and keeps more multi-file pull requests, where review quality matters most.

Building ground truth from many reviewers

The authors argue that no single reviewer, human or model, finds everything, so the golden set is built in three stages. Candidate findings come from real human review comments, issues inferred from the author’s follow-up commits, deterministic analysis tools, and several frontier LLMs from different model families. Overlapping findings are then merged semantically, so the same issue reported by several sources does not inflate the set. Last, each finding is checked against a shared rubric: it counts as a true positive only if it is “true, relevant, and non-trivial,” whatever its source. GitHub uses Claude Sonnet 5 as the LLM grader and publishes both the rubric and the judge.

Every finding is labelled with a severity (Critical, Medium, or Low) and a category such as correctness, security, reliability, maintainability, or testing. Users can filter results to only the issues they care about, for example critical security findings.

Credit for bugs nobody listed

ReviewBench reports precision, recall, and F1 in two forms. The grounded versions score only against the existing golden set, which gives a strict like-for-like comparison. The augmented versions also judge findings that match nothing in the set, so a reviewer gets credit when it surfaces a real issue no source had flagged. The authors say this matters because a fixed golden set “inevitably becomes incomplete” as agents improve. Augmented recall expands its denominator based on what each agent finds, so GitHub uses grounded recall as the headline comparison across systems and treats the augmented numbers as per-system diagnostics.

Review preferences differ, so the leaderboard can be re-ranked. Users can adjust β in an Fβ score to favour broader coverage or lower noise, and the ranking changes to match.

Before release, senior engineers who had not worked on the dataset re-labelled every ground-truth finding from scratch. Their judgments agreed with ReviewBench 96.6% of the time. The dataset, judge, and matcher are versioned, so results can be compared under the same configuration and re-validated when the benchmark changes.

What it predicted for Copilot code review

GitHub says it has used ReviewBench to evaluate successive versions of Copilot code review, and that offline changes “have consistently pointed in the same direction” as later production A/B tests. The post gives one example: a multi-model ensemble review in the lite tier, which combines several independent model runs into a single review. ReviewBench predicted higher precision, recall, and comment volume, at lower cost per review. In production, addressed rate, GitHub’s online stand-in for precision, rose 8.0%. Recall rose 13.6%, comment volume rose 61%, and cost per review fell 8.0%, all relative to the control. At the severity level, ReviewBench predicted a 227% increase in critical comments, and the online test showed 262%.

How outside agents get on the board

To submit, a team signs in with GitHub and registers its agent with a container image, its configuration, and its own model key. GitHub supplies the judge. Teams can iterate on a 25-PR test set with per-PR detail, then make a final run over all 219 pull requests, three rounds, scored by the same judge as every other entry. Scores stay private until a maintainer approves them. A result is published only if it is the agent’s first entry or beats its current leaderboard score.

What sets ReviewBench apart, by GitHub’s own description, comes down to three choices: a golden set drawn from many kinds of reviewers, credit for valid findings outside that set, and a stated check against production outcomes. GitHub says it wants researchers and practitioners to “challenge our assumptions” and help improve it.