Repository navigation
Only verified findings block; reviewer rules only in the agent prompt - #12
Merged
Merged
Conversation
Fix two findings that `shall review` found in findings.py. JTOOL-0002.6: a finding must not block unless the verifier confirmed it. Finding.blocking returned corpus_blocking when there was no verdict, so `--no-verify`, `review.verify: false` and `shall validate` gave blocking findings that nobody verified. - Finding.blocking is true only with a `confirmed` verdict. - needs_verification reads corpus_blocking. Before verification, blocking is now false for every finding. - sort_key sorts by corpus_blocking second, so the comment cap keeps the findings that can block. - A finding without a verdict shows "Not verified - not blocking". - `shall validate` loses --fail-on-blocking: it could never fire. The gate is `shall verify --fail-on-blocking`. - /code-review: `shall review --fail-on-blocking` with verification off can never fail (cli.py cmd_review, `verify` false). It now prints a warning. JTOOL-0004.2: review instructions must live only in agents/shall-reviewer.md. chunk_prompt repeated rules 1, 2 and 4 of that file (only violations, empty list, line anchors). chunk_prompt now sends only data. The bundle header tells the steps (reviewer, validate, verify) and gives no review rules. Eval: same Haiku selection on the 11 eval/cases, Sonnet reviewer, 2 runs per prompt, no cache. Recall 0.868 in all 4 runs. Precision old 0.825/0.786, new 0.805/0.846. Dropped by validation old 1/1, new 1/0. The change is inside the noise. Plugin version 0.4.2: a skill reference changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes two findings that
shall reviewfound infindings.py. The code came fromreview.py, so the problems were there before.JTOOL-0002.6: only a verified finding blocks
Before,
Finding.blockingreturnedcorpus_blockingwhen there was no verdict. Thus--no-verify,review.verify: falseandshall validategave blocking findings that no verifier saw. The exit code, the GitHub review and the Stop hook readblocking, so all three were affected.Finding.blockingis true only when the corpus marks the finding as blocking and the verdict isconfirmed.needs_verificationreadscorpus_blocking. Before verification,blockingis now false for every finding.sort_keysorts bycorpus_blockingsecond, so the comment cap keeps the findings that can block.shall review --fail-on-blockingwith verification off prints a warning, because it cannot fail.shall validateno longer accepts--fail-on-blocking. The flag could never fire. The gate isshall verify --fail-on-blocking.JTOOL-0004.2: one reviewer prompt
chunk_promptrepeated rules 1, 2 and 4 ofagents/shall-reviewer.md(only violations, an empty list, line anchors). Now it sends only data:<chunk>,<known_effects>and<standards>. The bundle header tells the steps (reviewer,validate,verify) and gives no review rules. A new test fails if text gets added outside the data tags.shall-reviewer.mddid not change. The plugin version is 0.4.2, because a skill reference changed.Eval: old prompt and new prompt
Haiku (
ClaudeJudgethroughclaude -p, no Jev) selected the statements for the 11eval/cases. Both prompts used the same selection. The reviewer was Sonnet, with 2 runs per prompt and no cache. The findings were compared with theviolationslabels ineval/labels.yaml.The difference is inside the noise between runs. Without the anchor text, the reviewer did not give more invalid line numbers. Cost: $1.19.
make eval-realprdid not run.Test plan
make test(202 passed),make check,make self-check/simplifyand/code-review, with their fixes applied/verifywithclion case 08 and a Haiku selection:bundle: the new header, and only data tags in each chunk section.validate: exit 0, the finding is advisory.validate --fail-on-blocking: exit 1, usage error.verify --fail-on-blocking: the verdict isconfirmed, the finding blocks, exit 2.review --no-verify.reviewneeds Jev for selection.🤖 Generated with Claude Code