Skip to content

fix(dojo-protocol): 0.1.1 — the judge reports before its turn limit; briefs quote rules verbatim (2.4.0, unreleased) - #16

Open
DojoGenesis wants to merge 1 commit into
mainfrom
fix/judge-turn-budget
Open

DojoGenesis wants to merge 1 commit into
mainfrom
fix/judge-turn-budget

Conversation

@DojoGenesis

@DojoGenesis DojoGenesis commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

dojo-protocol 0.1.1: the judge now reports before it runs out of turns, and briefs quote the rules a judge or reviewer applies instead of paraphrasing them. It's a fix under the unreleased 2.4.0.

Before merge: run scripts/release-check.sh on a machine with Python 3.9, claude and the denylist. This PR was checked on Windows and WSL, where OVERALL isn't green with or without this change, and the denylist did not run (details under Checks).

What happened

Two failures from one session on 2026-10-22/23, both on judge passes:

  1. A judge died silent. judge.md sets maxTurns: 25, but unlike scout.md:17 it has no rule to report before the limit. The judge ran out of turns and returned no report. It took a hand resume with a call cap to get verdicts.
  2. A paraphrased rule produced false findings. Two of the judge's findings were its own errors, and both came from a brief that paraphrased a rule instead of quoting it.

The fix

  • agents/judge.md: report by about the 20th tool call, and mark what wasn't checked unverified (not reached). maxTurns stays 25, so there is headroom to write the report. A not reached claim counts as unverified in the final line, so the report can't end in no defects found.
  • skills/delegate/SKILL.md (new item 7) and skills/refute/SKILL.md: when a judge or reviewer checks work against rules, paste the rules verbatim with their source path and line, never a paraphrase.
  • tests/test_session_start.py: the judge's body must carry the rule, and the version pin moves to 0.1.1. The new assertion fails on main's judge.md.
  • plugin.json 0.1.0 to 0.1.1, and a CHANGELOG line.

skills/delegate/brief-template.md is unchanged. The new item only applies to judges and reviewers, so it has no slot in the template.

Live check

One run each, with the same prompt and harness, through the Agent tool in claude -p --plugin-dir. Only judge.md differs. There are 30 path:line claims, the brief asks for one Read per claim so they can't be batched, and claims 4, 12 and 29 are false on purpose.

main (0.1.0) this branch (0.1.1)
Judge model claude-opus-5-5 claude-opus-5-5
Tool calls 25 20
Report returned none yes
Verdicts none 18 confirmed, 2 refuted (4 and 12), 10 unverified (not reached)
Claim 29 (false, past the budget) not reached unverified (not reached), not confirmed

What the parent got back from main's judge. The Agent tool result was not flagged as an error:

NOTE: this agent stopped at its 25-turn limit before finishing. It was still calling tools and had produced no report.

The final lines from this branch's judge:

| 28 | plugins/agent-orchestration/skills/handoff-protocol/SKILL.md:35 | unverified (not reached) | Needs a Read at line 35 |
| 29 | plugins/agent-orchestration/skills/maestro-orchestration/SKILL.md:36 | unverified (not reached) | Needs a Read at line 36. The quoted text contains "ZEBRA", like the two refuted claims, so it is likely to fail too. |
| 30 | plugins/agent-orchestration/skills/orchestration-pattern-selector/SKILL.md:154 | unverified (not reached) | Needs a Read at line 154 |

2 refuted (claims 4 and 12), 10 unverified (claims 21 to 30, not reached).

This is one run per side, so it shows the rule can change the outcome. It doesn't measure how reliably it does.

Checks

The release tables and failing test ids match main on Windows and on WSL, so this branch adds no failures.

Row Result Where
lint PASS, 9 plugins, 197 files, 0 errors both
unittest:dojo-protocol PASS, 71 tests, 0 skipped WSL (ext4)
Other plugins' unit tests PASS: router 163, meter 111, verify 153, doctor 159, settle 115. WARN, 1 skipped: gates 145 WSL
unittest:dojo-flow WARN, 181 tests, 1 skipped Windows (WSL has no node)
validate PASS, 0 errors on all 9 plugins Windows, Claude Code 2.1.305
mod PASS, 190 tests (router 27, meter 54, verify 109) Windows
face-parity PASS, 7 checks both
plugin-lint WARN-OK, known baseline both
always-on The row prints PASS at ~1,288, which is an undercount (see below). The real figure is ~1,856 of 2,000: that ~1,288 listing plus ~568 SessionStart, measured separately Windows
self-test FAIL, 296 tests, 2 failures, same ids on main WSL
denylist ERROR, did not run: the list isn't on this machine both
  • Self-test: both failures come from running Python 3.12 instead of 3.9. test_python_310_apis expects itertools.pairwise to be flagged as missing, and test_unittest_with_zero_tests_fails expects the old result for an empty run, where 3.12 exits 5.
  • Windows unit tests: on 3.14, every row but dojo-flow and dojo-settle fails on Windows, and on 3.10 every row but dojo-flow does. Main fails the same way. On 3.14, dojo-protocol fails only its two POSIX-only tests (os.mkfifo and an unreadable-file test). I used WSL for these rows for that reason.
  • Denylist: I read the diff for private names and found none, but that isn't the same check, which is why it's listed under Before merge.

Separate bug, not fixed here

scripts/always_on.py can pass falsely on Windows. It runs the hook's command through cmd.exe, which leaves ${CLAUDE_PLUGIN_ROOT} unexpanded, so the hook exits 2. injection_chars() then counts that as 0 tokens without saying so. The row reads PASS at ~1,288 when the real figure is ~1,856, so it could pass while over budget. With the variable expanded, the same command exits 0 and injects 568 tokens' worth of text. This PR doesn't touch the hook or PROTOCOL.md, so the number itself hasn't moved.

Fixes OCH-311

🤖 Generated with Claude Code

…briefs quote rules verbatim (2.4.0, unreleased)

The judge has maxTurns: 25 and nothing telling it to report before then. In
a session on 2026-10-22/23 one ran out of turns and returned no report; it
took a hand resume with a call cap to get verdicts. Two of that judge's
findings were its own errors, both from a brief that paraphrased a rule
instead of quoting it.

- judge.md: report by about the 20th tool call and mark the rest
  `unverified (not reached)`. maxTurns stays 25, so there is headroom to
  write the report. A not-reached claim counts as unverified in the final
  line, so the report can't end in `no defects found`.
- delegate (new item 7) and refute: when a judge or reviewer checks work
  against rules, paste the rules verbatim with source path and line.
- Tests: the judge's body must carry the rule, and the version pin moves to
  0.1.1. The new assertion fails on main's judge.md.

Live check, one run each, same prompt and harness, only judge.md differs:
30 path:line claims, one Read per claim, three false on purpose (4, 12, 29).
- main: 25 Reads, no report ("stopped at its 25-turn limit").
- this branch: 20 Reads, then a report: 18 confirmed, 2 refuted (4 and 12),
  10 unverified (not reached), including 29.

Checks:
- WSL Ubuntu, Python 3.12: dojo-protocol 71 tests, 0 skipped, OK. Self-test
  296 tests, 2 failures, both from running 3.12 instead of 3.9.
- Windows, Claude Code 2.1.305: validate 0 errors on all 9 plugins; 190 mod
  tests pass.
- Always-on: ~1,856 of 2,000 (listing ~1,288 plus SessionStart ~568), with
  descriptions and PROTOCOL.md unchanged. The row itself reads ~1,288 on
  Windows, because cmd.exe doesn't expand the hook's plugin-root variable and
  always_on.py counts a failed hook as 0, so the 568 was measured separately.
- The release tables and failing test ids match main on both platforms.
- Not run: the denylist (its list isn't on this machine). OVERALL is not
  green on either platform here, with or without this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants