Skip to content

feat(warden): Wait for ready for review before running Warden - #231

Draft
sentry-junior[bot] wants to merge 1 commit into
mainfrom
warden/require-ready-for-review
Draft

sentry-junior[bot] wants to merge 1 commit into
mainfrom
warden/require-ready-for-review

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Warden runs as a ruleset-required workflow, and GitHub ignores types for those: only opened, synchronize and reopened fire (docs). Warden never sees ready_for_review, so a skill with draft = false passes green on a draft, and a PR marked ready with no new push can merge without review. #229 and #230 can't fix this, for that reason.

This makes a draft-time run count as "not yet":

  • New first step before Analyze: .github/scripts/warden-require-ready.sh asks GitHub for the PR's live draft state and fails the job on drafts, before any model call. Drafts stay cheap. A PR marked ready stays blocked until someone re-runs the Warden check or pushes. The script reads live state rather than the event payload, so a re-run passes the gate.
  • incremental = true in [defaults]: incrementalApp was already set, but incremental was never on, so security-review re-reviewed every hunk on every push. Re-runs and later pushes now skip hunks a clean run already reviewed. If there is no baseline, Warden reviews the full PR.

Open decisions

  • Red checks on drafts. Every draft shows a failing Warden check until it's marked ready, and the author then has to re-run the check or push. Is that friction OK org-wide? Roll out in Evaluate mode first.
  • Repos with draft = false in their own warden.toml need to drop it. A re-run reuses the original event payload (expected GitHub behavior, not tested here), so Warden still thinks the PR is a draft and skips those skills with a green result. Drafts no longer reach Warden after this change, so draft = false isn't needed anyway.
  • integrity-review still has a gap. On synchronize it only reviews hunks from that one push (integrity-gate.ts). When a draft-time run fails at the gate and is re-run, integrity-review only covers the last push, not the commits made while the PR was a draft. It isn't in the org config yet. Before it's enforced, either keep it out of this gate or change Warden to scope from the last completed integrity-review run.
  • Getting the script. The step runs the script from the .warden-org checkout of this repo's default branch, so the script has to merge before or together with the ruleset picking up this workflow version.

via Jeffrey Hung.

--

View Session [Sentry]

Ruleset-required workflows only receive opened, synchronize, and reopened,
so a passing run on a draft lets a PR marked ready merge unreviewed. Fail
draft runs before any model call and check the live draft state, so a
re-run after marking the PR ready proceeds. Turn on incremental review so
re-runs and later pushes skip hunks a clean run already covered.

Co-Authored-By: Jeffrey Hung <jeffrey.hung@sentry.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants