Skip to content

Add repo-tuned Warden review skills - #2939

Merged
bitsandfoxes merged 7 commits into
mainfrom
warden/review-skills
Nov 16, 2026
Merged

bitsandfoxes merged 7 commits into
mainfrom
warden/review-skills

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Adds two repo-tuned Warden skills and a warden.toml, so Warden can take over what Cursor Bugbot catches in this repo. Today Warden only runs the org-level security-review here, and that skill covered none of Bugbot's fixed findings.

#skip-changelog

What Bugbot found (last 6 months)

Scope: merged PRs from 2026-05-11 to 2026-11-11. That is 194 merged PRs, 100 of them not dependency-bot PRs. Bugbot left 94 inline findings on 42 PRs.

Category Fixed Dismissed Ignored Unclear Total
Correctness 28 6 7 1 42
CI / build / tooling 26 5 7 2 40
Test quality 4 1 0 0 5
Quality 3 1 3 0 7
Security 0 0 0 0 0
Total 61 13 17 3 94

How the outcomes were decided:

  • Fixed: Bugbot resolved the thread itself after a later commit (43), or a later commit in the PR changed the flagged lines and nobody objected (18; this part is inferred).
  • Dismissed: the author replied that it was a false positive or intentional.
  • Ignored: no code change and no reply. This includes "follow-up" deferrals.

CI/build is 43% of the fixed findings here, much more than the cross-repo average. So it gets its own skill.

Recurring fixed patterns:

  • Native entry-point drift: a C# P/Invoke with no matching function in the NoOp bridge, the Switch stub, or the native side (#2679, #2720, #2764).
  • Platform and build-target gaps: Dedicated Server read as Standalone, GetSubtarget called on non-standalone builds, arm64 vs ARM64, Switch 2, nameof(target) (#2858, #2882, #2894, #2834).
  • Build state that outlives the build: UnityLinker args not removed or user args stripped, and Xcode Append exports keeping stale bridges or upload phases (#2805, #2885).
  • Option wiring: an option missing from one init path, or Experimental.* vs top-level mix-ups (#2749, #2687).
  • CI producer/consumer drift and runner images: artifacts that wait-for-artifact polls for but nothing produces anymore, missing actions: read, Xcode or simulator pins that the image no longer has (#2701, #2665, #2871).
  • PowerShell false success: native command failures ignored, lexical version sort, partial downloads that block retries (#2727, #2783).

The dismissed findings became "What not to report" rules in the skills:

  • claims about sentry-cocoa or sentry-java APIs (#2767, #2778)
  • stale files in fresh builds
  • Unity callback lifecycle guesses (#2805)
  • PowerShell -in semantics (#2805)
  • implausible races
  • forwarded CLI arguments
  • failures in pinned environments that already fail loudly

Skills

  • unity-sdk-review covers src/**/*.cs, package-dev C#/ObjC/C/ProGuard sources, *.asmdef, and test/**/*.cs, but not test/Scripts.Integration.Test. It is tuned to the patterns above and also checks threading/GPU readback, batch-mode assumptions, and tests that hide regressions. It encodes facts that change a verdict: src/ is built once against Unity 2021.3, so #if UNITY_* never applies there; iOS and macOS share the Cocoa code; the Warden checkout has no submodules. Bridge-parity and Editor build-pipeline details are in references/, which the skill reads only when a hunk needs them.
  • unity-ci-review covers .github/**, scripts/**, *.ps1/*.sh, MSBuild props/targets/csproj, build/**, package metadata, test/Scripts.*, and the sample Assets/Editor build scripts. PowerShell details are in references/powershell.md.

The two skills' paths do not overlap, so no hunk is reviewed twice. Out of 94 Bugbot findings, 84 fall in scope (57 of 61 fixed). Not covered: CHANGELOG.md, sample ProjectSettings.asset and SentryOptions.asset files, and one sample script.

The built-in code-review is not enabled. It is generic: its references cover JS/TS, Python, and workflows, not C#, Unity, IL2CPP, or PowerShell. Running it next to these skills would review every hunk twice on a high-effort model. The repo skills keep its "bugs only, prove it" bar and its severity scale.

Config

  • security-review comes from the org base and is not redeclared. Warden ignores a repo skill with the same name as a base skill.
  • Model, effort, runtime, and incrementalApp are inherited from the base config, with no overrides.
  • Repo skills do not inherit the base reportOn, failOn, or ignorePaths. See inheritRepoLayerDefaults and resolveLayeredSkillConfigs in packages/warden/src/config/loader.ts, and the loader test "inherits base execution defaults without inheriting base policy defaults". So warden.toml repeats the org policy values in [defaults]: reportOn low, failOn off, failCheck false, requestChanges false. Findings of every severity are posted as comments and nothing blocks a PR. Low is on so Warden can be compared fairly with Bugbot, which got low-severity findings fixed too. The base ignorePaths still does not apply, which is why the test-focused rules can see test/**.
  • incremental = true on both skills, with triggers on opened/synchronize/reopened.

Cost

  • Each SKILL.md is about 70 lines (6.6k to 7.8k characters), the same size as the built-in code-review prompt (7.5k).
  • Warden prompts per hunk, so path scoping and incremental review are the main savings.
  • Both skills define a narrow low level (a real bug with a small blast radius) and exclude style, dead code, and suspicion, so reporting at low does not turn into a lint pass.

Validation

  • Built getsentry/warden locally (pnpm install && pnpm build, v0.53.0).
  • Loaded this repo's config layered on the org warden.toml with loadLayeredWardenConfig and resolveLayeredSkillConfigs, and resolved each skill with resolveSkillAsync.
    • Both skills load from .agents/skills/.
    • There were no duplicate-skill warnings.
    • Each skill inherits grok-4.5, effort high, runtime pi, and incrementalApp sentry-warden.
  • With Warden's matchGlob, the skills match 239 and 134 tracked files with zero overlap.
  • No live model run was possible (no API key here). The first real PRs will show signal and noise.

Open questions

  • Should the samples/ assets (serialized options, ProjectSettings team IDs) be in scope? One fixed and three ignored findings came from there.
  • Should Bugbot be turned off once this has run for a few PRs?
  • Triggers do not set draft, so draft PRs are reviewed too, the same as getsentry/sentry. Each push to a draft costs a review run; incremental = true keeps it to what changed since the last run.

via amir.

--

View Session [Sentry]

Co-Authored-By: Amir Mujacic <amir.mujacic@sentry.io>
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
Messages
📖 Do not forget to update Sentry-docs with your feature once the pull request gets approved.

Generated by 🚫 dangerJS against ef3bd39

sentry-junior Bot and others added 2 commits November 11, 2026 11:55
Co-Authored-By: Amir Mujacic <amir.mujacic@sentry.io>
Co-Authored-By: Amir Mujacic <amir.mujacic@sentry.io>
@sentry-junior
sentry-junior Bot marked this pull request as ready for review November 11, 2026 12:13
Co-Authored-By: Amir Mujacic <amir.mujacic@sentry.io>
sentry-junior Bot and others added 3 commits November 11, 2026 12:21
The org Warden workflow does not start on ready_for_review, so this action never matches.

Co-Authored-By: Amir Mujacic <amir.mujacic@sentry.io>
Match getsentry/sentry: drop draft = false so skills also run on draft PRs.

Co-Authored-By: Amir Mujacic <amir.mujacic@sentry.io>
@bitsandfoxes
bitsandfoxes merged commit 555e3f8 into main Nov 16, 2026
30 checks passed
@bitsandfoxes
bitsandfoxes deleted the warden/review-skills branch November 16, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant