Skip to content

Only verified findings block; reviewer rules only in the agent prompt - #12

Merged
gabe4coding merged 1 commit into
mainfrom
claude/blissful-mendel-b0796c
Oct 13, 2026
Merged

gabe4coding merged 1 commit into
mainfrom
claude/blissful-mendel-b0796c

Conversation

@gabe4coding

Copy link
Copy Markdown
Owner

Summary

This PR fixes two findings that shall review found in findings.py. The code came from review.py, so the problems were there before.

JTOOL-0002.6: only a verified finding blocks

Before, Finding.blocking returned corpus_blocking when there was no verdict. Thus --no-verify, review.verify: false and shall validate gave blocking findings that no verifier saw. The exit code, the GitHub review and the Stop hook read blocking, so all three were affected.

  • Finding.blocking is true only when the corpus marks the finding as blocking and the verdict is confirmed.
  • needs_verification reads corpus_blocking. Before verification, blocking is now false for every finding.
  • sort_key sorts by corpus_blocking second, so the comment cap keeps the findings that can block.
  • A finding without a verdict shows "Not verified — not blocking."
  • shall review --fail-on-blocking with verification off prints a warning, because it cannot fail.
  • Breaking: shall validate no longer accepts --fail-on-blocking. The flag could never fire. The gate is shall verify --fail-on-blocking.

JTOOL-0004.2: one reviewer prompt

chunk_prompt repeated rules 1, 2 and 4 of agents/shall-reviewer.md (only violations, an empty list, line anchors). Now it sends only data: <chunk>, <known_effects> and <standards>. The bundle header tells the steps (reviewer, validate, verify) and gives no review rules. A new test fails if text gets added outside the data tags.

shall-reviewer.md did not change. The plugin version is 0.4.2, because a skill reference changed.

Eval: old prompt and new prompt

Haiku (ClaudeJudge through claude -p, no Jev) selected the statements for the 11 eval/cases. Both prompts used the same selection. The reviewer was Sonnet, with 2 runs per prompt and no cache. The findings were compared with the violations labels in eval/labels.yaml.

Run Recall Precision Dropped by validation
old #1 / #2 0.868 / 0.868 0.825 / 0.786 1 / 1
new #1 / #2 0.868 / 0.868 0.805 / 0.846 1 / 0

The difference is inside the noise between runs. Without the anchor text, the reviewer did not give more invalid line numbers. Cost: $1.19. make eval-realpr did not run.

Test plan

  • make test (202 passed), make check, make self-check
  • /simplify and /code-review, with their fixes applied
  • /verify with cli on case 08 and a Haiku selection:
    • bundle: the new header, and only data tags in each chunk section.
    • validate: exit 0, the finding is advisory.
    • validate --fail-on-blocking: exit 1, usage error.
    • verify --fail-on-blocking: the verdict is confirmed, the finding blocks, exit 2.
  • Not driven: review --no-verify. review needs Jev for selection.

🤖 Generated with Claude Code

Fix two findings that `shall review` found in findings.py.

JTOOL-0002.6: a finding must not block unless the verifier confirmed it.
Finding.blocking returned corpus_blocking when there was no verdict, so
`--no-verify`, `review.verify: false` and `shall validate` gave blocking
findings that nobody verified.
- Finding.blocking is true only with a `confirmed` verdict.
- needs_verification reads corpus_blocking. Before verification,
  blocking is now false for every finding.
- sort_key sorts by corpus_blocking second, so the comment cap keeps the
  findings that can block.
- A finding without a verdict shows "Not verified - not blocking".
- `shall validate` loses --fail-on-blocking: it could never fire. The gate
  is `shall verify --fail-on-blocking`.
- /code-review: `shall review --fail-on-blocking` with verification off can
  never fail (cli.py cmd_review, `verify` false). It now prints a warning.

JTOOL-0004.2: review instructions must live only in agents/shall-reviewer.md.
chunk_prompt repeated rules 1, 2 and 4 of that file (only violations, empty
list, line anchors). chunk_prompt now sends only data. The bundle header
tells the steps (reviewer, validate, verify) and gives no review rules.

Eval: same Haiku selection on the 11 eval/cases, Sonnet reviewer, 2 runs
per prompt, no cache. Recall 0.868 in all 4 runs. Precision old
0.825/0.786, new 0.805/0.846. Dropped by validation old 1/1, new 1/0.
The change is inside the noise.

Plugin version 0.4.2: a skill reference changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gabe4coding
gabe4coding merged commit dd6b61e into main Oct 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants