Add advisory review and risk checks for the shadow week by gabe-bots[bot] · Pull Request #13 · gabe4coding/shall

gabe-bots · 2026-10-18T22:32:09Z

Intent

Start the shadow week of the validation loop on this repo, the same setup as gabe4coding/puredori#10 and gabe4coding/retroagent#58. Two new checks run on every PR. Both are advisory: they are not required, they never fail a PR, and nothing here merges, approves or changes repo settings.

review (.github/workflows/review.yml): two passes in one job.
1. shall: the diff against the engineering standards, adapted from shall's ci/github/shall-review.yml.
2. General pass (.github/scripts/general_review.py): one tool-less, isolated claude -p call that checks for bugs, missing or weaker tests, silent behavior changes, PR body vs diff drift and irreversible changes. The PR text is passed as untrusted data. It also records the risk-judgment JSON from the design, which does not change the tier yet.
- Findings go in PR comments and the job summary. This week the job always ends green.
- If secrets are missing, each pass skips with a neutral ::notice that names the secret.
risk (.github/workflows/risk.yml, .github/scripts/risk_tier.py, .github/risk.yml): the deterministic tiers HOLD / T2-hard / T2-soft / T1 / T0.
- The rules and the script are read from the base branch, so a PR cannot loosen its own tier. This PR is the bootstrap and uses its own copy.
- The tier is posted as a risk tier check run (success for T0/T1, neutral for T2/HOLD, never failure) and as a risk:* label.
- Kill switch: if the repo variable AUTOMERGE is off, every PR becomes T2-hard. It is read only.
- There is no LLM layer yet. The vendor names that HOLD a PR are stored as sha256 hashes, so this public repo never spells them out.
review.yml uses this checkout's own shall (plugins/shall/bin) and this repo's .shall/ workspace (JTOOL rules on top of the organisation corpus in shall.yaml). There is no external pinned copy and no SHALL_EXTENDS. The PR's own tooling runs, which is fine for an advisory check.
ci.yml stays deterministic, with no model calls and no secrets. The model calls live only in the separate review workflow.
Sensitive files (T2-soft): plugins/shall/bin/shall-hook, hooks.py, claude_judge.py, publish.py and ci/github/.
plugins/, corpus/, .shall/, examples/ and eval/ are product text, not docs. corpus/index/ and .shall/index/ are generated, so they don't count toward the size limits.
No plugin files change, so no version bump.
Tests: .github/scripts/test_risk_tier.py and test_general_review.py, run by the risk (classifier tests) job.

Secrets to add (Settings > Secrets and variables > Actions):

TYPESAFE_API_KEY: needed for shall's Jev rule selection.
ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN: needed for claude -p in both passes.
Data note: diffs and PR text go to TypeSafe (shall) and Anthropic (both passes).

This PR is T2-hard by its own rules, because it adds .github/risk.yml and the review and risk workflows. Gabe reviews and merges it.

Evidence

uv run --no-project --python 3.12 --with pyyaml==6.0.2 --with pytest -- python -m pytest -q .github/scripts passes. actionlint is clean on both workflows.
Repo checks: The make targets run in CI (ci.yml). This box has no make, and none of the tooling changes.
The classifier on this branch gives T2-hard (hard paths: .github/risk.yml, the scripts and both workflows).
Backtest on the 12 merged PRs (all of them), using these rules (PR base and head from GitHub, diff from the merge base):

PR	Title	Size	Tier	Why
#12	Only verified findings block; reviewer rules only in the agent prompt	+69/-33, 9 files	T1	source-change: 5 code files, 63 changed lines
#11	Harden the codebase: types, module layers and checks catch wrong code	+2078/-951, 51 files	T2-hard	workflow-permissions-secrets-triggers: `.github/workflows/ci.yml`: `on:`; size: 2119 changed code lines
#10	chore: OSS cleanup (remove private refs, untrack .claude/hillclimb)	+7/-3249, 33 files	T2-hard	mass-deletion: net 3237 non-test lines deleted; deleted-source: deletes `.claude/hillclimb/scan/_state.json`
#9	Add the verification loop and a /verify recipe	+540/-16, 12 files	T2-soft	sensitive-path: touches `plugins/shall/tooling/src/shall/hooks.py`
#8	Use Claude as the hook judge when there is no Jev key	+803/-67, 17 files	T2-soft	sensitive-path: touches `plugins/shall/tooling/src/shall/claude_judge.py`; sensitive-path: touches `plugins/shall/tooling/src/shall/hooks.py`
#7	Measure shall on real pull requests and close declared limits	+2263/-116, 38 files	T2-hard	size: 1827 changed code lines
#6	README: new logo and wordmark header	+59/-11, 4 files	T0	docs-only: 4 files
#5	Rename jrfc to shall	+2247/-1656, 158 files	T2-hard	deleted-source: deletes `.jrfc/.gitignore`; deleted-source: deletes `examples/booking-service/.jrfc/.gitignore`
#4	Docs: OSS-ready README, docs as mdx in ASD-STE100, lean agent harness	+1914/-1451, 34 files	T2-hard	hard-path: touches `LICENSE`
#3	jrfc scan: lint whole files, with an eval and tuned rule criteria	+5845/-46, 53 files	HOLD	forbidden-ref: `eval/scan/cases.yaml` adds a forbidden name
#2	Claude Code hooks: Jev judges agent tool calls against the corpus	+2526/-21, 26 files	T2-hard	size: 2053 changed code lines
#1	Triage other AI reviewers' PR comments: keep what is relevant, drop th	+1592/-52, 32 files	T2-hard	size: 1534 changed code lines

Totals: T0 1, T1 1, T2-soft 2, T2-hard 7, HOLD 1.

Things to compare by hand this week: whether each T2-soft really needed you, and whether any T1 should have been T2.

The review job ran claude -p on pull_request with ANTHROPIC, OAuth and TypeSafe secrets, so a same-repo branch could read them or fake a clean review. Nothing in the workflow is left without those secrets. Closes gabe4coding/puredori#12 Co-authored-by: Gabe <gpavanello@thefork.com>

Port the risk-gate hardening. The tier job runs on pull_request_target with the workflow, rules and classifier from the base branch. The PR head is fetched as data and never executed. Python runs with -I and uv with --no-config. The classifier tests job has contents: read only and no stored git credentials. Paths come from git diff --name-status -z. The diff uses --text and --no-textconv, and lines split at newlines only. A test-only change that edits an assertion or adds skip, only or todo is T1. A file list that disagrees with the diff is T2-hard. The reviewer signal is only the Greptile Review check from greptile-apps on the current head SHA. A failed, missing, pending or stale check, or a check from another app, has not passed and raises T0 to T1. A pass never lowers a tier. greptile.json, .greptile/** and .coderabbit.yaml are T2-hard. Co-authored-by: Gabe <gpavanello@thefork.com>

gabe-bots · 2026-10-19T11:05:43Z

@greptileai review

greptile-apps · 2026-10-19T11:11:26Z

Confidence Score: 4/5

[Critical impact] Adds the risk classification system that gates all future merges.

Fix the missing-line risk checks and satisfy the repository's reviewer requirements before merging.

Findings

Changed lines escape risk checks ▶
Risk labels stay outdated ▶
Unchecked findings reach comments ▶
Quotes are treated as proof ▶
Comment writes lack dry-run ▶

Summary

Adds deterministic risk rules, a classifier workflow, an optional general-review script, and tests.

Pull requests get an advisory risk tier and a matching check and label.
A separate Claude review script can post findings on a pull request.

Acknowledgment: Harness intentionally removed the secret-bearing review workflow in commit fefcdbe because same-repository branches could read its secrets or fake a clean review. The missing workflow is not reported.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[PR event] --> B[Read base-branch classifier and rules]
  B --> C[Fetch PR head as data]
  C --> D[Collect changed files and lines]
  E[Read current Greptile result] --> F[Choose risk tier]
  D --> F
  F --> G[Publish label and advisory check]
  H[Greptile finishes later] -. No refresh trigger .-> G

_{Reviews (1) · Last reviewed commit: "risk: close tier bypasses and read Grept..." · Reviewed by Greptile}

greptile-apps · 2026-10-19T11:11:29Z

+        elif line.startswith("+++ ") or line.startswith("--- ") or line.startswith("@@"):
+            continue


Changed lines escape risk checks

parse_diff drops real changed lines that look like file headers. Adding content beginning with ++ produces a diff line beginning with +++ , which this branch discards. Removing content beginning with -- has the same problem.

Those lines disappear from the forbidden-name scan and size limits, so a change that should be HOLD or T2-hard can receive T1. Track whether the parser is inside a hunk before treating these prefixes as headers.

greptile-apps · 2026-10-19T11:11:29Z

+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review, edited]
+  pull_request_target:
+    types: [opened, synchronize, reopened, ready_for_review, edited]


Risk labels stay outdated

The workflow reads the Greptile result once, but never runs again when that review finishes. If a docs-only PR is checked while Greptile is pending, _with_reviewer raises it to T1. A later successful review leaves that tier and label unchanged until someone edits, pushes, or reopens the PR.

This makes the advisory shadow-week results depend on timing. Refresh the tier when the current head's Greptile review completes.

greptile-apps · 2026-10-19T11:11:29Z

+    blocks = blocking(result, diff)
+    (out / "general.json").write_text(json.dumps(result, indent=2), encoding="utf-8")
+    text = render(result, blocks)


Unchecked findings reach comments

render publishes every model finding, including findings whose file, line, or evidence does not exist in the diff. blocking checks evidence only to choose the “would block” marker; it does not filter the report.

The repository's JTOOL-0002.2 rule requires findings to pass validate_findings before reaching a report or comment. Add that check before saving or rendering findings. This repository requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

greptile-apps · 2026-10-19T11:11:29Z

+def blocking(result: dict, diff: str) -> list:
+    lines = {l[1:].strip() for l in diff.splitlines() if l[:1] in "+-" and not l.startswith(("+++", "---"))}
+    return [f for f in result.get("findings", [])
+            if f.get("severity") == "high" and f.get("kind") in ("bug", "irreversible") and verified(f, lines)]


Quotes are treated as proof

blocking uses the model's severity and a matching diff line to mark a finding “verified, would block” and return exit code 2. It never gets a verifier verdict or derives severity from the corpus. A copied line alone does not establish that the claimed bug is real.

JTOOL-0002.5 requires blocking decisions to come from the corpus and verification. JTOOL-0002.6 requires a verifier to confirm the problem from seen evidence. Apply those checks before marking findings verified; these repository requirements must be satisfied before merging.

Context Used: AGENTS.md (source)

greptile-apps · 2026-10-19T11:11:29Z

+    ap.add_argument("--event", required=True)
+    ap.add_argument("--model", default="claude-sonnet-5")
+    ap.add_argument("--out", default=".review-out")
+    ap.add_argument("--post", action="store_true")


Comment writes lack dry-run

The new command can create or update GitHub comments with --post, but it has no --dry-run option. JTOOL-0002.4 requires commands that write to external services to offer --dry-run without writing.

Add the option and make it preview the intended comment without sending the POST or PATCH. This repository requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

A successful Greptile Review check is not a pass on its own. claude-code-mode#47 concluded success at 5075dbc with confidence 0/5 and three P1 comments. A pass now needs all three on the current head SHA: that check from greptile-apps, the latest greptile-apps[bot] summary scored at least 4/5 for that SHA, and no P0 or P1 inline comment from that bot on that SHA. Anything else raises the tier one step and never lowers it. The tier job reads the check and both comment lists with github.token, GET only. Co-authored-by: Gabe <gpavanello@thefork.com>

Add advisory review and risk checks for the shadow week

012d76f

github-actions Bot added the risk:t2-hard label Oct 18, 2026

Harness and others added 2 commits October 19, 2026 10:59

greptile-apps Bot reviewed Oct 19, 2026

View reviewed changes

gabe4coding merged commit 6de3c14 into main Oct 19, 2026
4 checks passed

Navigation Menu

Repository navigation

Add advisory review and risk checks for the shadow week - #13

Add advisory review and risk checks for the shadow week#13
gabe4coding merged 4 commits into
mainfrom
bots/validation-shadow-week

gabe-bots Bot commented Oct 18, 2026

Uh oh!

gabe-bots Bot commented Oct 19, 2026

Uh oh!

greptile-apps Bot commented Oct 19, 2026 •

edited

Loading

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		elif line.startswith("+++ ") or line.startswith("--- ") or line.startswith("@@"):
		continue

Conversation

gabe-bots Bot commented Oct 18, 2026

Intent

Evidence

Uh oh!

gabe-bots Bot commented Oct 19, 2026

Uh oh!

greptile-apps Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

greptile-apps Bot commented Oct 19, 2026 •

edited

Loading