Add advisory review and risk checks for the shadow week by gabe-bots[bot] · Pull Request #9 · gabe4coding/loadout

gabe-bots · 2026-10-18T22:32:17Z

Intent

Start the shadow week of the validation loop on this repo, the same setup as gabe4coding/puredori#10 and gabe4coding/retroagent#58. Two new checks run on every PR. Both are advisory: they are not required, they never fail a PR, and nothing here merges, approves or changes repo settings.

review (.github/workflows/review.yml): two passes in one job.
1. shall: the diff against the engineering standards, adapted from shall's ci/github/shall-review.yml.
2. General pass (.github/scripts/general_review.py): one tool-less, isolated claude -p call that checks for bugs, missing or weaker tests, silent behavior changes, PR body vs diff drift and irreversible changes. The PR text is passed as untrusted data. It also records the risk-judgment JSON from the design, which does not change the tier yet.
- Findings go in PR comments and the job summary. This week the job always ends green.
- If secrets are missing, each pass skips with a neutral ::notice that names the secret.
risk (.github/workflows/risk.yml, .github/scripts/risk_tier.py, .github/risk.yml): the deterministic tiers HOLD / T2-hard / T2-soft / T1 / T0.
- The rules and the script are read from the base branch, so a PR cannot loosen its own tier. This PR is the bootstrap and uses its own copy.
- The tier is posted as a risk tier check run (success for T0/T1, neutral for T2/HOLD, never failure) and as a risk:* label.
- Kill switch: if the repo variable AUTOMERGE is off, every PR becomes T2-hard. It is read only.
- There is no LLM layer yet. The vendor names that HOLD a PR are stored as sha256 hashes, so this public repo never spells them out.
Sensitive files (T2-soft): hooks/hooks.json only. The design names none for loadout. loadout has no security boundary, and its main risk is hiding items: a wrong event registration can hide tools in every session. hooks/register.tsx and hooks/scope.ts stay T1, because they change in almost every PR and the "keep everything on failure" rule is covered by tests.
tests/*.check.mjs are CI gates, so they count as code, not tests.
No plugin files change, so no version bump.
Tests: .github/scripts/test_risk_tier.py and test_general_review.py, run by the risk (classifier tests) job.

Secrets to add (Settings > Secrets and variables > Actions):

TYPESAFE_API_KEY: needed for shall's Jev rule selection.
ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN: needed for claude -p in both passes.
Data note: diffs and PR text go to TypeSafe (shall) and Anthropic (both passes).

This PR is T2-hard by its own rules, because it adds .github/risk.yml and the review and risk workflows. Gabe reviews and merges it.

Evidence

uv run --no-project --python 3.12 --with pyyaml==6.0.2 --with pytest -- python -m pytest -q .github/scripts passes. actionlint is clean on both workflows.
Repo checks: node tests/docs.check.mjs and node tests/version.check.mjs origin/main pass. contract.check.mjs needs Node 24 (it loads .ts files), so it runs in CI.
The classifier on this branch gives T2-hard (hard paths: .github/risk.yml, the scripts and both workflows).
Backtest on the 8 merged PRs (all of them), using these rules (PR base and head from GitHub, diff from the merge base):

PR	Title	Size	Tier	Why
#8	Check the trace readers with tsc, not with a regex	+182/-74, 9 files	T1	source-change: 5 code files, 165 changed lines
#7	Harden the plugin code with types and contract checks	+1083/-444, 14 files	T2-soft	workflow-additive: `.github/workflows/test.yml`: additive edit or pinned version bump
#6	Add an eval that measures the impact of loadout	+928/-55, 11 files	T1	source-change: 7 code files, 804 changed lines
#5	Remove the plugin name from the status line and toast text	+7/-7, 4 files	T1	source-change: 2 code files, 10 changed lines
#4	List /loadout from the start of a session	+20/-5, 6 files	T1	source-change: 4 code files, 16 changed lines
#3	New task note, effort for each prompt, model for the session, switches	+900/-35, 10 files	T1	source-change: 5 code files, 432 changed lines
#2	Add an always-keep list for skills and tool groups (0.5.0)	+409/-34, 14 files	T2-soft	sensitive-path: touches `hooks/hooks.json`
#1	Hide no tool group without ToolSearch, and classify late servers (0.5.	+246/-51, 11 files	T1	source-change: 4 code files, 76 changed lines

Totals: T1 6, T2-soft 2.

Things to compare by hand this week: whether each T2-soft really needed you, and whether any T1 should have been T2.

The review job ran on pull_request with API secrets, so a branch could read them or post its own result. Greptile reviews these PRs instead. Closes gabe4coding/puredori#12 Co-authored-by: Gabe <gpavanello@thefork.com>

Port the risk-gate bypass fixes: the tier job runs on pull_request_target with the base workflow, rules and classifier, and the PR head is data. Python runs with -I, uv with --no-config, and the test job is read-only. Paths come from git diff --name-status -z. A test-only assertion edit or skip is T1, and a file list that disagrees with the diff is T2-hard. Greptile Review must be a success from greptile-apps on the current head. Anything else raises T0 to T1 and never lowers a higher tier. Reviewer config files are T2-hard. Co-authored-by: Gabe <gpavanello@thefork.com>

gabe-bots · 2026-10-19T11:05:48Z

@greptileai review

greptile-apps · 2026-10-19T11:12:29Z

Confidence Score: 4/5

[Critical impact] Adds the risk classification system that gates all future merges.

Add the missing review workflow before merging; neither promised review pass currently runs.

Findings

Review passes never run ▶
Tier stays stale after review ▶
Disabled gates look additive ▶
Moved workflows look additive ▶
Removed comments go uncounted ▶

Summary

Adds an advisory risk classifier, PR labels and check runs, plus a standalone general-review script and unit tests.

Pull requests receive a risk tier based on their changes.
A standalone command sends a PR diff for a structured review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[PR event] --> B[Read base branch workflow and rules]
  B --> C[Fetch PR head as data]
  C --> D[Read current reviewer checks]
  D --> E[Classify risk]
  E --> F[Write job summary]
  E --> G[Update risk label]
  E --> H[Post advisory risk check]
  A --> I[Run classifier unit tests with read-only token]

_{Reviews (1) · Last reviewed commit: "Raise the risk tier only from the Grepti..." · Reviewed by Greptile}

greptile-apps · 2026-10-19T11:12:32Z

+if __name__ == "__main__":
+    sys.exit(main())


Review passes never run

No workflow starts general_review.py. The only workflows are risk.yml and test.yml; neither runs this script or shall. Neither promised review pass runs on PRs, even after the secrets are configured. Add the missing advisory review.yml workflow.

greptile-apps · 2026-10-19T11:12:32Z

+  pull_request_target:
+    types: [opened, synchronize, reopened, ready_for_review, edited]


Tier stays stale after review

If this job reads a pending Greptile Review on a docs-only PR, the classifier raises it from T0 to T1. A later successful review triggers none of these events, so the label and check keep reporting the old tier until someone edits, reopens, or updates the PR. Add a trusted refresh path for completed reviews so the shadow-week results reflect the finished review.

greptile-apps · 2026-10-19T11:12:32Z

+    for line in _meaningful(f.added) + _meaningful(f.removed):
+        if WORKFLOW_SENSITIVE.search(line):
+            return Reason("T2-hard", "workflow-permissions-secrets-triggers", f"`{f.path}`: `{line.strip()[:80]}`")
+    added_keys = {_key(l) for l in _meaningful(f.added)}
+    for line in _meaningful(f.removed):
+        if not (BUMPABLE.search(_key(line)) and _key(line) in added_keys):
+            return Reason("T2-hard", "workflow-weakened", f"`{f.path}` removes `{line.strip()[:80]}`")
+    return Reason("T2-soft", "workflow-additive", f"`{f.path}`: additive edit or pinned version bump")


Disabled gates look additive

workflow_reason treats additions as harmless unless a line matches WORKFLOW_SENSITIVE. Adding if: false to the existing contracts job, or continue-on-error: true to its check step, returns T2-soft even though it disables a gate. Adding a trigger below an unchanged on: also escapes the trigger check. Compare the workflow structure before and after the edit so these changes receive T2-hard rather than understating their risk.

greptile-apps · 2026-10-19T11:12:32Z

+    if f.status == "D":
+        return Reason("T2-hard", "workflow-weakened", f"deletes workflow `{f.path}`")


Moved workflows look additive

Moving .github/workflows/test.yml outside .github/workflows/ without changing its contents returns T2-soft. Git records a rename with no added or removed lines, but this function only treats D as a removed workflow. GitHub stops running the moved file, so the rename removes all its checks. Treat moves out of the workflow directory, including moves to an unsupported extension, as workflow removal.

greptile-apps · 2026-10-19T11:12:32Z

+        elif line.startswith("+++ ") or line.startswith("--- ") or line.startswith("@@"):
+            continue


Removed comments go uncounted

parse_diff drops changed content that looks like a file header. Removing a SQL or Lua comment such as -- explanation produces --- explanation in the diff, which this branch skips even inside a hunk. Removing 20 such comments therefore misses the comment-pruning HOLD rule. Track whether the parser is inside a hunk before skipping file headers.

A successful Greptile Review check is not a pass on its own. The latest summary from greptile-apps[bot] must score at least 4/5 on the current head, and that bot must not have a P0 or P1 inline comment on that SHA. A non-pass raises the tier one step and stops at T2-hard. The claude-code-mode#47 review (score 0/5, P1 on 5075dbc) is a fixture and is not a pass. Co-authored-by: Gabe <gpavanello@thefork.com>

Add advisory review and risk checks for the shadow week

f711b0c

github-actions Bot added the risk:t2-hard label Oct 18, 2026

Harness and others added 2 commits October 19, 2026 10:58

Remove the claude -p review workflow

dc4de2b

The review job ran on pull_request with API secrets, so a branch could read them or post its own result. Greptile reviews these PRs instead. Closes gabe4coding/puredori#12 Co-authored-by: Gabe <gpavanello@thefork.com>

greptile-apps Bot reviewed Oct 19, 2026

View reviewed changes

gabe4coding merged commit ad0cf2b into main Oct 19, 2026
7 checks passed

Navigation Menu

Repository navigation

Add advisory review and risk checks for the shadow week - #9

Add advisory review and risk checks for the shadow week#9
gabe4coding merged 4 commits into
mainfrom
bots/validation-shadow-week

gabe-bots Bot commented Oct 18, 2026

Uh oh!

gabe-bots Bot commented Oct 19, 2026

Uh oh!

greptile-apps Bot commented Oct 19, 2026 •

edited

Loading

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

greptile-apps Bot Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		pull_request_target:
		types: [opened, synchronize, reopened, ready_for_review, edited]

		if f.status == "D":
		return Reason("T2-hard", "workflow-weakened", f"deletes workflow `{f.path}`")

		elif line.startswith("+++ ") or line.startswith("--- ") or line.startswith("@@"):
		continue

Conversation

gabe-bots Bot commented Oct 18, 2026

Intent

Evidence

Uh oh!

gabe-bots Bot commented Oct 19, 2026

Uh oh!

greptile-apps Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

greptile-apps Bot commented Oct 19, 2026 •

edited

Loading