Skip to content

Add the verification loop and a /verify recipe - #51

Merged
gabe4coding merged 2 commits into
mainfrom
claude/validation-loop-verify-c56a3b
Oct 13, 2026
Merged

gabe4coding merged 2 commits into
mainfrom
claude/validation-loop-verify-c56a3b

Conversation

@gabe4coding

@gabe4coding gabe4coding commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

Intent

Add the code-mode / shall verification loop to retroagent. Before each commit, the agent runs /simplify, then /code-review, then scripts/test --all, then /verify. /verify uses a project recipe, .claude/skills/verify/ (SKILL.md and drive.py). It runs this checkout's kb and the plugin hooks (headless claude -p --plugin-dir .) against a temporary sandbox data repo, never the real one.

Drift

  • The plan said "no drive script": the agent was to run each command by hand. If one command left out KB_CONFIG, it reached the real data repo. So drive.py came back. It sets KB_CONFIG and KB_ROOT (and HOME for kb runs) for each run, and it refuses a folder that has no sandbox marker.
  • No TUI drive for the decide mod. The recipe uses claude plugin validate . and claude plugin test ., and the hook run shows the mod load in pluginLog.
  • The /code-review fixes (about 20 lines) changed code. I did not run /simplify and /code-review again for them. I checked them inline and with the drive.

Decisions

  • The loop order and the "start again at step 1" rule are the same as in code-mode and shall. The efficiency review said that a restart after each small fix costs a full review. I kept the rule as you asked for it.
  • A hook session keeps the real HOME, because the Claude login needs it. So kb-hint in a hook run can use your local embedding model (method: semantic). A kb run gets the sandbox HOME.
  • drive.py reuses tests/fixtures.py (init_remote, clone, make_claude_tree, make_codex_tree, git) and kb.adapters.common.iter_records.
  • Door: two-way. The change adds dev tooling and rule text only. No src/kb/ code, hook or skill changes.

Evidence

Before: main has no .claude/skills/verify/ (git ls-tree -r --name-only main .claude prints nothing). /verify has no recipe and nothing to drive.

After: real drive.py runs on 2026-10-13. Each run used one sandbox under $TMPDIR/ra-verify-*.

new → first sync in the sandbox:

2026-10-13T15:20:11Z verify: 3 sessions, 0 summaries, 1 redactions, pushed

kb <dir> -- find flaky:

page     2026-10-01 project demo-app           demo-app · …lock and rerun (5e6f7a8b) ## Open threads - Retry the «flaky» fetch test (5e6f7a8b) [kb page demo-app]
e94ad30f 2026-10-06 claude demo               ↳55555555 Explore tests · Find «flaky» tests [turn 1]
55555555 2026-10-06 claude demo               Fix flaky motion test · Fix the «flaky» test in tests/motion.spec.ts [turn 1]

hook <dir> "Run exactly this Bash command: sh -c 'echo \"ModuleNotFoundError: No module named yaml\" >&2; exit 1'" ($0.0028):

  • SessionStart (kb-hook): Project page of demo-app: kb page demo-app (1 open thread, 2 errors → fixes). ... · - [s-000001] Rules · Wait for CI with a watch, not a sleep loop · 1 retro suggestion waits for the user to accept or reject ...
  • PostToolUseFailure (kb-hint): retroagent: seen before → make checkfails: ModuleNotFoundError no module named yaml → install the dev extras withpip install -e .[dev] (kb summary 1a2b3c4d)
  • pluginLog: hooks module retroagent@inline loaded ..., $.process.run (retroagent): bin/kb exited 0 in 150ms (the decide mod's kb decide --json), PreToolUse:Bash ["bin/kb-guard"] ... status 0 (90ms), PostToolUseFailure:Bash ["bin/kb-hint"] ... status 0 (204ms)
  • data.log and data.remoteLog did not change (auto_sync: false).

kb-guard deny, from a run of the first version of drive.py (15:12, before the review changes; the session code is the same apart from the log file names):

PreToolUse:Bash hook error: retroagent: only the user answers `kb decide` questions, in their own terminal. Tell the user which answer you suggest and why; do not run it.

SessionStart sync with new --config '{"auto_sync": true, "debounce_minutes": 0}', then kb <dir> -- status: last sync: 2026-10-13T15:14:29Z (the first sync of new ran at 15:14:06).

Guards: kb <dir> on the temp folder itself → not a sandbox: /private/var/folders/.../T. kb <dir> --timeout 0.01 -- status → "exit": "timeout" with the evidence JSON. The drive found this bug: before the fix, --timeout went to kb as an argument.

scripts/test --all: 1726 passed in 81.06s.

Reading map

Config:

  • .claude/skills/verify/seed/pages/suggestions.json, .claude/skills/verify/seed/pages/decisions.json (sandbox seed data, made up)

Error handling:

  • .claude/skills/verify/drive.py:71 (raise SystemExit for a folder that is not a sandbox), :84-90 (kb timeout), :150 and :182 (session kill timer), :159 (BrokenPipeError), :168 (JSONDecodeError)

Stack

Step 1/1. Previous: none. Next: none.

Story

Add the verification loop (/simplify, /code-review, checks, /verify) and a /verify recipe for retroagent.

# Step Evidence PR
1 Loop rule in CODING_STANDARDS.md and docs/development.mdx. /verify recipe: .claude/skills/verify/SKILL.md, drive.py (new, kb, hook, clean) and the seed pages. Real drive.py output for the kb and hook probes. scripts/test --all passes. this PR (done)

🤖 Generated with Claude Code

gabrypavanello and others added 2 commits October 13, 2026 17:08
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before each commit, the agent runs /simplify, /code-review, scripts/test --all and /verify, as in code-mode and shall. The /verify recipe (.claude/skills/verify/) has drive.py. It makes a sandbox data repo with seed pages and the fake transcripts of tests/fixtures.py. Then it runs this checkout's kb, or headless Claude Code with this checkout as the plugin. Every run gets the sandbox KB_CONFIG and KB_ROOT, and a folder without the sandbox marker is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gabe4coding
gabe4coding marked this pull request as ready for review October 13, 2026 15:26
@gabe4coding
gabe4coding merged commit dcb39c8 into main Oct 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants