Repository navigation
Add pr-review agent skill - #7344
Merged
Merged
Conversation
Reviews a branch or commit against its merge base with main as a maintainer would: correctness, duplicated mechanisms, fit with existing extension points, error paths on absent input, sibling routes, user-facing names and text, stored state over time, doc comments, and tests. It prints comments with the evidence behind them and never posts. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: 552a987
Top 3 slowest tests (at least 2 minutes):
|
Drop the three-to-eight comment budget, which made the skill leave out confirmed findings on larger changes. Each comment now has to state the behavior, a concrete consequence and an ask; comments are ordered by severity; and a final coverage check lists any worksheet row that is not raised, with the reason. Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka
approved these changes
Nov 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add a
pr-reviewagent skill under.agents/skills/. It reviews a branch (or a given commit) against its merge base with main and prints inline comments, a review body, and the evidence and worksheet behind them. It never posts.The skill works through fixed passes: correctness, duplicated mechanisms, fit with existing extension points, new error paths on absent input, sibling routes (create vs update, deploy vs destroy), user-facing names, stored state over time, doc comments, tests, and user-facing text. It reads the repo's
.agents/rules/for testing level and comment conventions.Why
We want agent reviews to catch the kind of issues maintainers raise in this repo, not just generic bugs, and to know how often they do. The skill was written and tuned against past reviews on this repo rather than from a checklist.
Tests
Backtest against human reviews. Merged PRs, each frozen at the commit that received the most human review comments, were reviewed by the skill with only read-only git access (no PR description, no GitHub), and an LLM judge checked whether it raised the same issue as each substantive human comment (match 1, partial 0.5).
Sweep of recently merged PRs. The skill reviewed the 59 human-authored PRs merged into main between 2026-10-27 and 2026-11-03, at their final commit. It made 200 findings (about 3.4 per PR); 34 had already been raised on the PR. Each finding was then checked against the code on its own by a separate model told to try to refute it; that checker refuted all five deliberately false claims planted to test it. 193 findings were confirmed, 6 refuted and 1 left inconclusive. Of the confirmed ones not raised on the PR, 12 were bugs still present on main. Fixing them showed 2 to be working as expected; the other 10 are draft PRs #7331–#7340, each naming the PR that introduced the issue. The refuted findings were mostly behavior that the code or a test already documented as intended.
The sweep was run with the earlier version that asked for three to eight comments per review.
Each review takes 2–5 minutes with Opus. The backtest harness and fixtures are kept outside this repo so future runs can't read the expected comments.
This pull request and its description were written by Isaac.