Repository navigation
fix(dojo-protocol): 0.1.1 — the judge reports before its turn limit; briefs quote rules verbatim (2.4.0, unreleased) - #16
Open
DojoGenesis wants to merge 1 commit into
Open
DojoGenesis wants to merge 1 commit into
DojoGenesis wants to merge 1 commit into
Conversation
…briefs quote rules verbatim (2.4.0, unreleased)
The judge has maxTurns: 25 and nothing telling it to report before then. In
a session on 2026-10-22/23 one ran out of turns and returned no report; it
took a hand resume with a call cap to get verdicts. Two of that judge's
findings were its own errors, both from a brief that paraphrased a rule
instead of quoting it.
- judge.md: report by about the 20th tool call and mark the rest
`unverified (not reached)`. maxTurns stays 25, so there is headroom to
write the report. A not-reached claim counts as unverified in the final
line, so the report can't end in `no defects found`.
- delegate (new item 7) and refute: when a judge or reviewer checks work
against rules, paste the rules verbatim with source path and line.
- Tests: the judge's body must carry the rule, and the version pin moves to
0.1.1. The new assertion fails on main's judge.md.
Live check, one run each, same prompt and harness, only judge.md differs:
30 path:line claims, one Read per claim, three false on purpose (4, 12, 29).
- main: 25 Reads, no report ("stopped at its 25-turn limit").
- this branch: 20 Reads, then a report: 18 confirmed, 2 refuted (4 and 12),
10 unverified (not reached), including 29.
Checks:
- WSL Ubuntu, Python 3.12: dojo-protocol 71 tests, 0 skipped, OK. Self-test
296 tests, 2 failures, both from running 3.12 instead of 3.9.
- Windows, Claude Code 2.1.305: validate 0 errors on all 9 plugins; 190 mod
tests pass.
- Always-on: ~1,856 of 2,000 (listing ~1,288 plus SessionStart ~568), with
descriptions and PROTOCOL.md unchanged. The row itself reads ~1,288 on
Windows, because cmd.exe doesn't expand the hook's plugin-root variable and
always_on.py counts a failed hook as 0, so the 568 was measured separately.
- The release tables and failing test ids match main on both platforms.
- Not run: the denylist (its list isn't on this machine). OVERALL is not
green on either platform here, with or without this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dojo-protocol 0.1.1: the judge now reports before it runs out of turns, and briefs quote the rules a judge or reviewer applies instead of paraphrasing them. It's a fix under the unreleased 2.4.0.
Before merge: run
scripts/release-check.shon a machine with Python 3.9,claudeand the denylist. This PR was checked on Windows and WSL, whereOVERALLisn't green with or without this change, and the denylist did not run (details under Checks).What happened
Two failures from one session on 2026-10-22/23, both on judge passes:
judge.mdsetsmaxTurns: 25, but unlikescout.md:17it has no rule to report before the limit. The judge ran out of turns and returned no report. It took a hand resume with a call cap to get verdicts.The fix
agents/judge.md: report by about the 20th tool call, and mark what wasn't checkedunverified (not reached).maxTurnsstays 25, so there is headroom to write the report. Anot reachedclaim counts as unverified in the final line, so the report can't end inno defects found.skills/delegate/SKILL.md(new item 7) andskills/refute/SKILL.md: when a judge or reviewer checks work against rules, paste the rules verbatim with their source path and line, never a paraphrase.tests/test_session_start.py: the judge's body must carry the rule, and the version pin moves to 0.1.1. The new assertion fails on main'sjudge.md.plugin.json0.1.0 to 0.1.1, and a CHANGELOG line.skills/delegate/brief-template.mdis unchanged. The new item only applies to judges and reviewers, so it has no slot in the template.Live check
One run each, with the same prompt and harness, through the Agent tool in
claude -p --plugin-dir. Onlyjudge.mddiffers. There are 30path:lineclaims, the brief asks for one Read per claim so they can't be batched, and claims 4, 12 and 29 are false on purpose.unverified (not reached)unverified (not reached), not confirmedWhat the parent got back from main's judge. The Agent tool result was not flagged as an error:
The final lines from this branch's judge:
This is one run per side, so it shows the rule can change the outcome. It doesn't measure how reliably it does.
Checks
The release tables and failing test ids match main on Windows and on WSL, so this branch adds no failures.
unittest:dojo-protocolunittest:dojo-flowtest_python_310_apisexpectsitertools.pairwiseto be flagged as missing, andtest_unittest_with_zero_tests_failsexpects the old result for an empty run, where 3.12 exits 5.dojo-protocolfails only its two POSIX-only tests (os.mkfifoand an unreadable-file test). I used WSL for these rows for that reason.Separate bug, not fixed here
scripts/always_on.pycan pass falsely on Windows. It runs the hook's command throughcmd.exe, which leaves${CLAUDE_PLUGIN_ROOT}unexpanded, so the hook exits 2.injection_chars()then counts that as 0 tokens without saying so. The row readsPASSat ~1,288 when the real figure is ~1,856, so it could pass while over budget. With the variable expanded, the same command exits 0 and injects 568 tokens' worth of text. This PR doesn't touch the hook orPROTOCOL.md, so the number itself hasn't moved.Fixes OCH-311
🤖 Generated with Claude Code