Skip to content

fix(always-on): a SessionStart hook that didn't run can't read as zero - #17

Open
DojoGenesis wants to merge 1 commit into
mainfrom
fix/dgs-261-always-on-false-pass
Open

DojoGenesis wants to merge 1 commit into
mainfrom
fix/dgs-261-always-on-false-pass

Conversation

@DojoGenesis

Copy link
Copy Markdown
Owner

scripts/always_on.py is the gate for the suite's 2,000-token budget. On Windows it read low and could pass while over budget. It now measures correctly there, and a hook that didn't run stops the check instead of counting as zero.

Cause

injection_chars() ran the raw hooks.json command with shell=True, and only added to the total when the hook exited 0. A hook that failed counted as 0 tokens, with no error.

On Windows that shell is cmd.exe, which leaves ${CLAUDE_PLUGIN_ROOT} as it is, so the hook exits 2. That happened to every SessionStart hook in the marketplace.

Before (Windows) After (Windows)
dojo-protocol SessionStart 0 ~568
Suite total ~1,288 ~1,856
Fixture hook that exits 1, real weight over budget script exits 0: suite ~1500 tokens, 0 over budget script exits 2: s1: SessionStart hook exited 1: note from the hook

The fix

  • Expansion: the script fills in ${CLAUDE_PLUGIN_ROOT} before the shell sees the command.
  • Non-zero exit: this is now a RunError (exit 2) naming the plugin and the last line of the hook's stderr. No marketplace hook is meant to exit non-zero: dojo-protocol, bring-loop and kata-harness are the only three SessionStart hooks, and all three end in sys.exit(0).
  • Absolute plugin path (not in the original issue): hooks run from a temporary project, so --repo . also read as zero, on POSIX too. Without this it would now be a hard error.

Tests

The stand-in claude in test_always_on.py was a #! file, which Windows can't start, so 4 of its 5 tests never reached the code on the platform with this bug. It now gets a .cmd launcher there. Six new tests:

  • a hook that fails
  • a command that can't start
  • a hook that exits clean and silent (must stay zero, so this isn't over-tightened)
  • the command handed to the shell
  • a relative --repo
  • the marketplace's real hooks through the real shell, with dojo-protocol pinned to the size of PROTOCOL.md
Self-tests, full suite Failures on main Failures on this branch Newly failing
Windows, Python 3.14 234 + 2 errors, of 238 230 + 2 errors, of 244 none
WSL, Python 3.12 2, of 296 2, of 302 none

Most of this suite already fails on Windows on main, so that row only shows nothing new broke; WSL is the meaningful pass, and its 2 failures (test_unittest_with_zero_tests_fails, test_python_310_apis) also fail on main. Compared by test name against a clean tree. Run against the old always_on.py, the new test file fails 7 tests on Windows and 4 on WSL.

release-check.sh --only always-on on Windows reads PASS, suite ~1856 tokens. The fresh table is byte-identical to the one in ALWAYS-ON.md, so the numbers there didn't need a refresh; only one sentence on the method was added.

Limits

  • A hook that crashes inside its own try/except still reads as zero on the always-on row. All three hooks are fail-open: they exit 0 and print nothing. An exit code can't tell that from a hook that's quiet by design, such as bring-loop in a project that doesn't use it. The new marketplace test catches this on the self-test row for dojo-protocol, the only hook that adds text by default (checked by breaking the hook in a scratch copy: {} != {'dojo-protocol': 1987}). bring-loop and kata-harness have no such check, but they add nothing in an empty project anyway.
  • Not run: the denylist row, because its local lists aren't on this machine. Nothing ran on a Mac; WSL's Linux shell is the only POSIX coverage, and it has no claude, so the real always_on.py table was measured on Windows only.
  • This measures a hook's text, not a live session. It doesn't check how Claude Code itself runs hooks on Windows.

Fixes DGS-261

🤖 Generated with Claude Code

injection_chars() ran the raw hooks.json command with shell=True and added
to the total only when the hook exited 0, so a failed hook counted as 0
tokens with no error. On Windows that was every hook: cmd.exe leaves
${CLAUDE_PLUGIN_ROOT} as it is, the hook exits 2, and the always-on row read
PASS at ~1,288 when the suite was ~1,856, so it could have passed over budget.

- Fill in ${CLAUDE_PLUGIN_ROOT} before the shell sees the command.
- A hook that exits non-zero is now a RunError (exit 2) that names the plugin
  and the last line of its stderr. No marketplace SessionStart hook is meant
  to exit non-zero: dojo-protocol, bring-loop and kata-harness all end in
  sys.exit(0).
- Make the plugin path absolute. Hooks run from a temporary project, so a
  relative --repo also read as zero, on POSIX too.

Tests: the stand-in claude in test_always_on.py was a #! file Windows can't
start, so those tests never ran on the platform with this bug; it now gets a
.cmd launcher there. New tests cover a failing hook, a command that can't
start, a hook that exits clean and silent, the command handed to the shell, a
relative --repo, and the marketplace's real hooks through the real shell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants