From e755ddd886569439b79f2869f9614381c7d26b53 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 30 Aug 2026 22:09:43 +0000 Subject: [PATCH 1/6] sizing: add the review-cost model (batch epic phase 0a) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Difficulty measures blast radius. It cannot answer the question a batch has to be planned against, which is what a task costs the HUMAN once it lands. Four read-only outputs on the sizing faculty now answer it: Consequence: notify | glance | judge — how much review the work needs Witness: free text — the machine-checkable claim that makes it reviewable Review-minutes: integer — a seed, not a measurement Unattended: ready | needs-slicing | never Graded by rules over repo class (from repos.yaml, where repo identity is already declared once) and over surface, never by an agent's reading of its own work — the ledger's base rate for an agent mis-scoping its own change is 20% (68 of 332 records in 2026-08 carry a correction or a retraction), so self-assessment is not an input. THE RULE THAT CARRIES THE MODEL: no Witness means judge. Without it the field would be aspirational — a prompt could claim a cheap tier while offering the reviewer nothing but the diff. With it, choosing a cheap tier means committing at conception to producing evidence, which is what actually makes work reviewable in minutes. Measured over the 153 backlog prompts today: 151 grade judge, because 3 carry a witness. Given one, the same backlog grades 33 notify / 104 glance / 16 judge. The whole distance between "everything costs a PI's hour" and "a fifth of it costs nothing" is whether prompts declare what will make them checkable. That number is the case for the field, and it is in the faculty's AGENTS.md. Unattended is deliberately not difficulty renamed: needs-slicing keys off the compaction rule (a task that would need context compaction to finish is too big to run unattended), which is measured rather than cautious. A single-repo large task still grades ready; a large one across four repos does not. Precedence is the module's existing rule, extended: declared beats derived, with the derived value returned alongside so the split is reported rather than silently resolved. One case is called out in the surface rather than hidden — a prompt the heuristic reads too-large and the author calls medium still grades ready, but says so. Known limit, documented rather than hidden: the judged-surface test is keyword matching. Fenced and inline code are masked (a prompt quoting a surface is documenting it, not touching it — the same rule declared_header applies), but prose describing a surface still trips it, and the prompts specifying this model are the worst offenders. The error is in the safe direction, and the keyword list is kept narrow because a loose one does not fail safe: it grades everything judge and the model stops discriminating. 10 new tests including a golden pin; all three load-bearing guards were confirmed to FAIL against a deliberately broken tree before being trusted. 654 pass. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RQeMJZznA3xTQXX4PqWg1v --- agents/faculties/sizing/AGENTS.md | 63 +++++- agents/faculties/sizing/_sizing.py | 256 +++++++++++++++++++++- tests/data/sizing_review_cost_golden.json | 30 +++ tests/test_sizing_review_cost.py | 182 +++++++++++++++ 4 files changed, 524 insertions(+), 7 deletions(-) create mode 100644 tests/data/sizing_review_cost_golden.json create mode 100644 tests/test_sizing_review_cost.py diff --git a/agents/faculties/sizing/AGENTS.md b/agents/faculties/sizing/AGENTS.md index 3ff55e1..2a5ea13 100644 --- a/agents/faculties/sizing/AGENTS.md +++ b/agents/faculties/sizing/AGENTS.md @@ -10,9 +10,68 @@ A PyAutoBrain read-only reasoning faculty. It owns the organism's single difficulty heuristic so the number is defined **once** and shared, never -recomputed by a divergent copy. +recomputed by a divergent copy — and, since 2026-08-30, the **review-cost +model** beside it. -It also owns the **precedence rule** over that heuristic. `estimate_difficulty` +## The review-cost model + +`Difficulty:` measures blast radius. It cannot answer the only question a batch +can be planned against, which is what a task costs the **human** once it lands. +Three outputs answer that, plus one that asks whether the human is needed at all: + +| Output | Header | Values | What it is | +|---|---|---|---| +| `consequence` | `Consequence:` | `notify` / `glance` / `judge` | how much review the work needs | +| `witness` | `Witness:` | free text | the machine-checkable claim that makes it reviewable | +| `review_minutes` | `Review-minutes:` | integer | a **seed**, not a measurement (see below) | +| `unattended` | `Unattended:` | `ready` / `needs-slicing` / `never` | can it finish without a human | + +Graded by **rules over repo class and surface**, never by an agent's reading of +its own work: the ledger's base rate for an agent mis-scoping its own change is +20% (68 of 332 records in 2026-08 carry a correction or a retraction), so +self-assessment is not an input. Repo class comes from `PyAutoMind/repos.yaml`, +where repo identity is already declared once. + +**The rule that carries the model: no `Witness:` means `judge`.** Without it the +field would be aspirational — a prompt could claim a cheap tier while offering +the reviewer nothing but the diff. With it, choosing a cheap tier means +committing at conception to producing evidence, which is what actually makes +work reviewable in minutes. A prompt carrying no witness grades `judge` however +small it looks, and that is the intended behaviour, not a gap. + +Measured over the 153 backlog prompts on the day this shipped: **151 grade +`judge`, because 3 carry a witness.** Given one, the same backlog grades 33 +`notify` / 104 `glance` / 16 `judge`. The entire distance between "everything +costs a PI's hour" and "a fifth of it costs nothing" is whether prompts declare +what will make them checkable. + +`unattended` is deliberately not difficulty renamed. `needs-slicing` keys off the +**compaction rule** — a task that would need context compaction to finish is too +big to run unattended — which is measured rather than cautious +(`anthropics/claude-code#54393`, a postmortem of five consecutive failed +autonomous overnight runs, names "good plan → compact → garbage drift" as a +primary failure primitive, and nothing downstream of the run catches it). + +`review_minutes` is a **seed awaiting calibration**, tier-driven with one nudge +for size. The honest numbers come from the batch records, which carry the minutes +the human actually spent. Never read a value from here as evidence about how long +anything took. + +**Known limit, stated rather than hidden.** The judged-surface test is keyword +matching over the prompt's prose. Fenced and inline code are masked — a prompt +*quoting* a surface is documenting it, not touching it, the same rule +`declared_header` applies — but prose that *describes* a judged surface still +trips it. The prompts specifying this very model are the worst offenders. The +error is in the safe direction (a false `judge` costs review time, a false +`notify` costs a bad merge), and the keyword list is kept deliberately narrow +because a loose one does not fail safe in the way it first appears: it grades +everything `judge`, the cheap tiers starve, and the model stops discriminating +at all. + +## Precedence + +It also owns the **precedence rule** over the heuristic, and over every output +of the review-cost model above. `estimate_difficulty` derives; `effective_difficulty` reconciles — a **declared** `Difficulty:` wins, and the derived level comes back alongside it so a disagreement is reported rather than silently resolved (the disagreement is evidence about the diff --git a/agents/faculties/sizing/_sizing.py b/agents/faculties/sizing/_sizing.py index 93a110b..418d52d 100755 --- a/agents/faculties/sizing/_sizing.py +++ b/agents/faculties/sizing/_sizing.py @@ -418,13 +418,50 @@ def empty_discovery_reason(mind: Path, work_type: str) -> str: # place, and gives the bug/refactor conductors the same reading for free. DIFFICULTY_LEVELS = ("small", "medium", "large", "too-large") AUTONOMY_LEVELS = ("safe", "supervised", "human-required") + +# --- the review-cost model (batch epic phase 0a) ----------------------------- +# `Difficulty:` is static blast radius. It cannot answer the only question a +# batch needs answered, which is what a task will cost the HUMAN once it lands. +# These three do. They are graded by RULES over repo class and surface, never by +# an agent's reading of its own work — the ledger's own base rate for an agent +# mis-scoping its own change is 20% (68 of 332 records in 2026-08 carry a +# correction or a retraction), so self-assessment is not an input here. +CONSEQUENCE_TIERS = ("notify", "glance", "judge") +UNATTENDED_LEVELS = ("ready", "needs-slicing", "never") + +# Surfaces that make a change a PI's decision whatever repo it lives in: a +# public API, a default value, an error contract, a science-policy call, or an +# external contributor's request. Judged from the prompt's own words. +# Kept deliberately specific. A loose keyword here does not fail safe in the way +# it first appears: it grades everything `judge`, the tier starves, and the model +# stops discriminating at all. `raise` alone matched "praise" and every prose +# mention of raising a question; `raises ` earns its place, bare `raise` does not. +JUDGE_SURFACE_KEYWORDS = [ + "public api", "default value", "defaults to", "error contract", + "raises ", "science policy", "external contributor", "external reporter", + "user-filed", "reported by", "backwards-compat", "backwards compat", + "breaking change", +] +# A witness claiming exact equality is the one machine-checkable claim strong +# enough to carry a behaviour-preserving change on its own. +BYTE_EQUALITY_MARKERS = [ + "byte-equal", "byte equality", "byte-identical", "bit-identical", + "bit identical", "byte-for-byte", "identical output", "unchanged output", +] +# Repo categories (PyAutoMind/repos.yaml) whose contents nobody outside this +# workshop consumes. Everything else is somebody's dependency. +INTERNAL_REPO_CATEGORIES = frozenset({"organ", "admin"}) +# Seeds, NOT measurements. Phase 7's batch records carry the minutes the human +# actually spent, and those are what will correct these. +REVIEW_MINUTES_SEED = {"notify": 0, "glance": 3, "judge": 20} # `medium` is not a documented Priority: value but occurs in the live backlog; # read it as normal rather than dropping the prompt's stated intent. PRIORITY_RANK = {"high": 0, "normal": 1, "medium": 1, "low": 2} DEFAULT_PRIORITY_RANK = 1 _HEADER_KEY_RE = re.compile( - r"^\s*(difficulty|type|autonomy|status|priority|blocked-by|closes-when)" + r"^\s*(difficulty|type|autonomy|status|priority|blocked-by|closes-when" + r"|consequence|witness|review-minutes|unattended)" r"\s*:\s*(.+?)\s*$", re.I ) @@ -446,7 +483,9 @@ def declared_header(text: str) -> dict: """ out = {"declared_difficulty": None, "declared_type": None, "declared_autonomy": None, "status": None, - "priority": None, "blocked_by": [], "closes_when": []} + "priority": None, "blocked_by": [], "closes_when": [], + "declared_consequence": None, "witness": None, + "declared_review_minutes": None, "declared_unattended": None} in_fence = False for line in text.splitlines(): if line.lstrip().startswith("```"): @@ -457,7 +496,11 @@ def declared_header(text: str) -> dict: m = _HEADER_KEY_RE.match(line) if not m: continue - key, value = m.group(1).lower(), _strip_trailing_comment(m.group(2)) + key = m.group(1).lower() + # `Witness:` is free text and routinely carries a `PR #123` reference, + # which the trailing-comment split would truncate. Take it raw. + value = m.group(2).strip() if key == "witness" \ + else _strip_trailing_comment(m.group(2)) if not value: continue if key == "difficulty": @@ -480,6 +523,20 @@ def declared_header(text: str) -> dict: out["blocked_by"].append(value) elif key == "closes-when": out["closes_when"].append(value) + elif key == "consequence": + v = value.lower() + if v in CONSEQUENCE_TIERS and out["declared_consequence"] is None: + out["declared_consequence"] = v + elif key == "witness" and out["witness"] is None: + out["witness"] = value + elif key == "review-minutes" and out["declared_review_minutes"] is None: + m2 = re.search(r"\d+", value) + if m2: + out["declared_review_minutes"] = int(m2.group(0)) + elif key == "unattended": + v = _norm_level(value) + if v in UNATTENDED_LEVELS and out["declared_unattended"] is None: + out["declared_unattended"] = v return out @@ -701,6 +758,159 @@ def estimate_difficulty(p: dict): return level, score, factors +def _repo_categories_of(p: dict) -> set: + """The body-map categories of the repos this prompt names. + + Repo *identity* is declared once, in `PyAutoMind/repos.yaml`. Reading the + category from there rather than keeping a second list here is the same rule + that governs every other repo fact in this module: one source, or the two + drift and whichever the reader happened to consult decides the answer. + """ + cats = _body_map_categories() + out = set() + for r in p["repos"]: + for name, spec in _body_map_specs().items(): + if canonical_key(name, spec) == r: + out.add(cats.get(name, "?")) + break + return out + + +def estimate_consequence(p: dict, factors: dict | None = None): + """Heuristic consequence tier -> (tier, reasons). + + `notify` costs the human nothing, `glance` costs a couple of minutes reading + the WITNESS, `judge` costs a PI's quarter-hour. The rules are ordered and the + first match wins, so every uncertain case falls through to `judge`: the tier + decides how little review something gets, and the safe direction for a + grader that is unsure is always more. + + The load-bearing rule is the last fallthrough — **no witness means judge**. + Without it the field would be aspirational: a prompt could claim a cheap tier + while offering the reviewer nothing to check but the diff. With it, choosing + a cheap tier means committing at CONCEPTION to producing evidence, which is + what actually makes work reviewable in minutes. The corollary is deliberate: + a prompt carrying no `Witness:` grades `judge` however small it looks. + """ + if factors is None: + _, _, factors = estimate_difficulty(p) + # Mask fenced blocks and inline code first, on the module's own standing + # rule: a prompt QUOTING a surface is documenting it, not touching it. The + # prompts that define this very model are the worst offenders — they name + # every judged surface in prose while touching none of them. + text = _mask_code(p["text"]).lower() + witness = (p.get("witness") or "").strip() + reasons = [] + + # 1. Work types whose deliverable is never a quietly-mergeable diff. + if p["work_type"] in ("release", HUMAN_REVIEW): + return "judge", [f"work-type {p['work_type']} is a human act by contract"] + + # 2. Surfaces that are a PI's call wherever they live. + surface = _hits(text, JUDGE_SURFACE_KEYWORDS) + if surface: + return "judge", [f"names a judged surface: {', '.join(surface[:3])}"] + + # 3. The default that makes the model bite. + if not witness: + return "judge", ["no Witness: declared — nothing to check but the diff"] + reasons.append(f"witness declared: {witness[:60]}") + + cats = _repo_categories_of(p) + internal = bool(cats) and cats <= INTERNAL_REPO_CATEGORIES + + # 4. A behaviour-preserving change proving exact equality reviews itself. + if p["work_type"] == "refactor" and _hits(witness.lower(), BYTE_EQUALITY_MARKERS): + return "notify", reasons + ["refactor with a byte-equality witness"] + + # 5. Work nobody outside this workshop consumes. + if internal: + return "notify", reasons + [f"internal repos only ({', '.join(sorted(cats))})"] + if p["work_type"] in ("docs", "test") and not (factors["library_repos"]): + return "notify", reasons + [f"{p['work_type']}, no library repo touched"] + + return "glance", reasons + ["witnessed change to a consumed repo"] + + +def estimate_unattended(p: dict, level: str, factors: dict | None = None): + """Can this finish without the human? -> (grade, reasons). + + Deliberately NOT difficulty renamed. Difficulty asks how big the blast + radius is; this asks whether one unattended run can carry the task to + PR-open. The rule that separates them is the compaction rule: **a task that + would need context compaction to finish is too big to run unattended.** + That is measured, not cautious — `anthropics/claude-code#54393`, a + postmortem of five consecutive failed autonomous overnight runs, names + "good plan -> compact -> garbage drift" as a primary failure primitive, and + nothing downstream of the run catches it. + """ + if factors is None: + _, _, factors = estimate_difficulty(p) + if p["work_type"] in ("release", HUMAN_REVIEW): + return "never", [f"work-type {p['work_type']} is human-driven by contract"] + if p.get("declared_autonomy") == "human-required": + return "never", ["declared Autonomy: human-required"] + if level == "too-large": + return "needs-slicing", ["too-large is a routing signal, not a grade"] + if level == "large" and factors["repos_affected"] > 2: + return "needs-slicing", [ + f"large across {factors['repos_affected']} repos — would compact"] + return "ready", ["fits one unattended run"] + + +def estimate_review_minutes(tier: str, level: str) -> int: + """A SEED, not a measurement. + + Everything here is derived from the tier plus a nudge for size; the honest + numbers come from phase 7's batch records, which carry the minutes the human + actually spent. Read a value from this function as "what to plan with until + we have measured", and never as evidence about how long anything took. + """ + minutes = REVIEW_MINUTES_SEED.get(tier, REVIEW_MINUTES_SEED["judge"]) + if tier == "judge" and level in ("large", "too-large"): + minutes += 5 + return minutes + + +def effective_consequence(p: dict, factors: dict | None = None): + """(tier, reasons, derived_tier) — the DECLARED tier wins. + + Same precedence rule as `effective_difficulty`, for the same reason: a value + the author declared is a judgement the heuristic does not have, and a + disagreement is evidence about the heuristic worth reporting rather than + silently resolving. + """ + derived, reasons = estimate_consequence(p, factors) + return p.get("declared_consequence") or derived, reasons, derived + + +def effective_unattended(p: dict, level: str, factors: dict | None = None, + derived_level: str | None = None): + """(grade, reasons, derived_grade) — the DECLARED grade wins. + + `derived_level` is the difficulty the heuristic derived, as opposed to the + one the author declared. It changes no verdict — declared still wins, per + the module's precedence rule — but a prompt the heuristic reads as + `too-large` and the author calls `medium` is exactly the case the compaction + rule exists to catch, so the tension is named in the reasons rather than + disappearing behind the precedence. + """ + derived, reasons = estimate_unattended(p, level, factors) + grade = p.get("declared_unattended") or derived + if grade == "ready" and derived_level == "too-large": + reasons = reasons + [ + "CAUTION: the heuristic derives too-large; the declared level won." + " Re-read before dispatching this unattended"] + return grade, reasons, derived + + +def effective_review_minutes(p: dict, tier: str, level: str): + """(minutes, derived_minutes) — a DECLARED estimate wins.""" + derived = estimate_review_minutes(tier, level) + declared = p.get("declared_review_minutes") + return (declared if declared is not None else derived), derived + + def effective_difficulty(p: dict): """(level, score, factors, derived_level) — the DECLARED level wins. @@ -729,6 +939,18 @@ def effective_difficulty(p: dict): # prompt without dispatching anything. It writes nothing. +def _disagree(field: str, declared, derived) -> None: + """Report a declared/derived split rather than resolving it silently. + + The precedence rule is that declared wins; the point of printing the other + value is that the split is evidence about the heuristic, and the heuristic + only improves if somebody sees it. + """ + if declared != derived: + print(f" ! declared {declared} but derived {derived}" + f" — declared wins; the disagreement is worth a look") + + def _main(argv=None): import argparse import json @@ -747,7 +969,11 @@ def _main(argv=None): args = ap.parse_args(argv) p = parse_prompt(Path(args.prompt).resolve(), args.mind.resolve()) - level, score, factors = estimate_difficulty(p) + level, score, factors, derived_level = effective_difficulty(p) + tier, tier_why, derived_tier = effective_consequence(p, factors) + grade, grade_why, derived_grade = effective_unattended( + p, level, factors, derived_level) + minutes, derived_minutes = effective_review_minutes(p, tier, level) surface = { "path": p["path"], "work_type": p["work_type"], @@ -755,17 +981,37 @@ def _main(argv=None): "repos": p["repos"], "lines": p["lines"], "words": p["words"], - "difficulty": {"level": level, "score": score, "factors": factors}, + "difficulty": {"level": level, "derived": derived_level, + "score": score, "factors": factors}, + "consequence": {"tier": tier, "derived": derived_tier, + "reasons": tier_why}, + "witness": p.get("witness"), + "review_minutes": {"minutes": minutes, "derived": derived_minutes, + "note": "a seed, not a measurement"}, + "unattended": {"grade": grade, "derived": derived_grade, + "reasons": grade_why}, } if args.json: print(json.dumps(surface, indent=2)) return + witness = p.get("witness") print(f"SizingSurface: {p['path']}") print(f" work-type : {p['work_type']}") print(f" target : {p['target']}") print(f" repos : {', '.join(p['repos']) or '(none)'}") print(f" size : {p['lines']} lines / {p['words']} words") print(f" difficulty: {level} (score {score})") + _disagree("difficulty", level, derived_level) + print(f" witness : {witness if witness else '(none declared)'}") + print(f" consequence: {tier} — {tier_why[0] if tier_why else ''}") + _disagree("consequence", tier, derived_tier) + print(f" review : ~{minutes} min (seed, not a measurement)") + if derived_minutes != minutes: + print(f" ! declared {minutes} but seeded {derived_minutes}") + print(f" unattended: {grade} — {grade_why[0] if grade_why else ''}") + for extra in grade_why[1:]: + print(f" ! {extra}") + _disagree("unattended", grade, derived_grade) if __name__ == "__main__": diff --git a/tests/data/sizing_review_cost_golden.json b/tests/data/sizing_review_cost_golden.json new file mode 100644 index 0000000..f1b03ea --- /dev/null +++ b/tests/data/sizing_review_cost_golden.json @@ -0,0 +1,30 @@ +{ + "api_change": { + "consequence": "judge", + "derived_consequence": "judge", + "difficulty": "small", + "review_minutes": 20, + "unattended": "ready" + }, + "doc_no_witness": { + "consequence": "judge", + "derived_consequence": "judge", + "difficulty": "small", + "review_minutes": 20, + "unattended": "ready" + }, + "doc_with_witness": { + "consequence": "notify", + "derived_consequence": "notify", + "difficulty": "small", + "review_minutes": 0, + "unattended": "ready" + }, + "refactor_byte_equal": { + "consequence": "notify", + "derived_consequence": "notify", + "difficulty": "small", + "review_minutes": 0, + "unattended": "ready" + } +} diff --git a/tests/test_sizing_review_cost.py b/tests/test_sizing_review_cost.py new file mode 100644 index 0000000..8f0be32 --- /dev/null +++ b/tests/test_sizing_review_cost.py @@ -0,0 +1,182 @@ +"""tests/test_sizing_review_cost.py — the review-cost model (batch epic 0a). + +`Difficulty:` measures blast radius. These four outputs measure what a task +costs the HUMAN, which is the only quantity a batch can be planned against. + +Two rules carry the whole model and each gets its own test: + + * **no `Witness:` means `judge`** — without it a prompt could claim a cheap + tier while offering the reviewer nothing but the diff, and the field would + be aspirational rather than load-bearing; + * **declared beats derived** — the module's standing precedence rule, which + three conductors re-implemented and two got wrong (PyAutoBrain#217, #274). + +The golden file pins the grade for a fixed sample so a change to the heuristic +has to be seen and accepted rather than absorbed. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "agents" / "faculties" / "sizing")) +from _sizing import ( # noqa: E402 + CONSEQUENCE_TIERS, + UNATTENDED_LEVELS, + effective_consequence, + effective_difficulty, + effective_review_minutes, + effective_unattended, + estimate_review_minutes, + parse_prompt, +) + +GOLDEN = Path(__file__).parent / "data" / "sizing_review_cost_golden.json" + + +def _prompt(tmp_path: Path, body: str, rel: str = "draft/feature/pyautobrain/t.md"): + path = tmp_path / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(body) + return parse_prompt(path, tmp_path) + + +def _grade(p): + level, _score, factors, derived_level = effective_difficulty(p) + tier, tier_why, derived_tier = effective_consequence(p, factors) + grade, _why, _dg = effective_unattended(p, level, factors, derived_level) + minutes, _dm = effective_review_minutes(p, tier, level) + return {"difficulty": level, "consequence": tier, "derived_consequence": derived_tier, + "review_minutes": minutes, "unattended": grade, "why": tier_why[0]} + + +HEADERLESS_DOC = """# Tidy the Brain README + +Type: docs +Target: PyAutoBrain +Repos: +- PyAutoBrain +Difficulty: small +Priority: normal +Status: draft + +Rewrite two paragraphs. Nothing else. +""" + + +def test_no_witness_grades_judge(tmp_path): + """The load-bearing default. A docs change to an organ repo is about as + consequence-free as work gets — and it still costs a PI's hour, because + nothing was promised that a reviewer could check.""" + g = _grade(_prompt(tmp_path, HEADERLESS_DOC)) + assert g["consequence"] == "judge" + assert "no Witness" in g["why"] + assert g["review_minutes"] == 20 + + +def test_witness_unlocks_the_cheap_tier(tmp_path): + """The same prompt, with a witness, is the same work — but now reviewable + in the time it takes to read one line, so it grades `notify`.""" + body = HEADERLESS_DOC.replace( + "Status: draft", + "Status: draft\nWitness: every link in the file resolves; docs build clean") + g = _grade(_prompt(tmp_path, body)) + assert g["consequence"] == "notify" + assert g["review_minutes"] == 0 + + +def test_judged_surface_beats_a_witness(tmp_path): + """A witness makes work checkable; it does not make an API decision + somebody else's to make. Surface outranks evidence.""" + body = HEADERLESS_DOC.replace( + "Status: draft", + "Status: draft\nWitness: byte-identical output on the full suite" + ).replace("Rewrite two paragraphs. Nothing else.", + "Change the default value of over_sample_size.") + assert _grade(_prompt(tmp_path, body))["consequence"] == "judge" + + +def test_declared_beats_derived_and_reports_the_split(tmp_path): + """Precedence, and the reason the derived value is returned at all: the + disagreement is evidence about the heuristic and has to stay visible.""" + body = HEADERLESS_DOC.replace( + "Status: draft", "Status: draft\nConsequence: judge") + g = _grade(_prompt(tmp_path, body)) + assert g["consequence"] == "judge" # declared wins + assert g["derived_consequence"] == "judge" # no witness -> judge anyway + + body2 = HEADERLESS_DOC.replace( + "Status: draft", + "Status: draft\nConsequence: judge\nWitness: docs build clean") + g2 = _grade(_prompt(tmp_path, body2)) + assert g2["consequence"] == "judge" # declared still wins + assert g2["derived_consequence"] == "notify" # and the split is reported + + +def test_prose_describing_a_surface_is_not_touching_one(tmp_path): + """A prompt quoting a judged surface inside a code fence is documenting it. + Same rule, same reason, as `declared_header`'s fenced-block skip.""" + body = HEADERLESS_DOC.replace( + "Rewrite two paragraphs. Nothing else.", + "Document the guard:\n\n```\nraises ValueError on a bad default value\n```\n" + ).replace("Status: draft", "Status: draft\nWitness: docs build clean") + assert _grade(_prompt(tmp_path, body))["consequence"] == "notify" + + +def test_unattended_is_not_difficulty_renamed(tmp_path): + """`needs-slicing` keys off the compaction rule, not off size alone: a + single-repo `large` task still fits one run.""" + body = HEADERLESS_DOC.replace("Difficulty: small", "Difficulty: large") + assert _grade(_prompt(tmp_path, body))["unattended"] == "ready" + + wide = HEADERLESS_DOC.replace("Difficulty: small", "Difficulty: large").replace( + "Repos:\n- PyAutoBrain", + "Repos:\n- PyAutoBrain\n- PyAutoMind\n- PyAutoHeart\n- PyAutoHands") + assert _grade(_prompt(tmp_path, wide))["unattended"] == "needs-slicing" + + +def test_human_required_is_never_unattended(tmp_path): + body = HEADERLESS_DOC.replace("Priority: normal", + "Autonomy: human-required\nPriority: normal") + assert _grade(_prompt(tmp_path, body))["unattended"] == "never" + + +def test_review_minutes_are_a_seed_not_a_measurement(): + """Documented as a seed, and shaped like one: tier-driven, with a single + nudge for size. Anything more precise would be inventing certainty.""" + assert estimate_review_minutes("notify", "small") == 0 + assert estimate_review_minutes("glance", "small") == 3 + assert estimate_review_minutes("judge", "small") == 20 + assert estimate_review_minutes("judge", "too-large") == 25 + + +def test_vocabularies_are_closed(): + assert CONSEQUENCE_TIERS == ("notify", "glance", "judge") + assert UNATTENDED_LEVELS == ("ready", "needs-slicing", "never") + + +def test_golden_sample_is_unchanged(tmp_path): + """Pins the grade for a fixed sample. A heuristic change that moves these + is not necessarily wrong — but it has to be looked at and re-accepted, + which is the entire job of a golden file.""" + samples = { + "doc_no_witness": HEADERLESS_DOC, + "doc_with_witness": HEADERLESS_DOC.replace( + "Status: draft", "Status: draft\nWitness: docs build clean"), + "api_change": HEADERLESS_DOC.replace( + "Rewrite two paragraphs. Nothing else.", + "Change the public API of Grid2D.").replace( + "Status: draft", "Status: draft\nWitness: 900 tests pass"), + "refactor_byte_equal": HEADERLESS_DOC.replace("Type: docs", "Type: refactor").replace( + "Status: draft", "Status: draft\nWitness: ids bit-identical, 62 -> 9.7 ms"), + } + got = {k: _grade(_prompt(tmp_path, v, f"draft/feature/pyautobrain/{k}.md")) + for k, v in samples.items()} + for g in got.values(): + g.pop("why") + if not GOLDEN.exists(): # first run writes the pin + GOLDEN.parent.mkdir(parents=True, exist_ok=True) + GOLDEN.write_text(json.dumps(got, indent=2, sort_keys=True) + "\n") + assert got == json.loads(GOLDEN.read_text()) From f18b6bf97b9a8aa140f836cf1dee3d99a35bacc6 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 30 Aug 2026 22:22:31 +0000 Subject: [PATCH 2/6] intake: write the review-cost model; drop the multi-repo autonomy trigger MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Intake now derives and writes Consequence:, Review-minutes: and Unattended: at conception, and reads a Witness: when the author declares one. Witness is the one field nothing derives: an invented one is plausible prose with nothing behind it, which is worse than none, because the value of the field is that its absence is informative. So it is kept OUT of HEADER_FIELDS — `intake formalise` writes every field it finds missing there, and would have auto-invented one across the whole backlog. Consequence and Review-minutes are derived and do join the hygiene set. When no witness is declared, the IntakeDecision says so and explains what it costs: the prompt grades judge, a PI's quarter-hour, whatever its size. infer_autonomy no longer returns supervised on repo_count > 1. Repo count is blast radius, already priced at +2 per repo in estimate_difficulty; this field is supposed to encode whether a human's judgement is needed, and a change across four repos mechanically needs no more of it than the same change in one. TWO CORRECTIONS TO THE PLAN THIS IMPLEMENTS, both measured here. The epic asserted that this one rule caused 120 of 137 prompts to read supervised. It does not. Re-deriving all 153 draft prompts: dropping the trigger takes safe 30 -> 55 and supervised 117 -> 92, and repo_count > 1 is the SOLE trigger for 25 prompts — the largest single one, ahead of large-or-above (20) and architectural risk (17), but nothing like 120. Those 120 are declared levels written by earlier intake runs, and the triggers overlap heavily. The change is right on its merits; it is not the unblocking it was taken for, and where that actually lives is the ship-sign-off change in phase 3 — 19 of the 46 parked rows are the contract park supervised imposes at ship, which no grading change touches. The first draft replaced repo_count with human_judgement, on the reasoning that ambiguity is what predicts a park. Measured, it made things worse: safe fell to 24, because the ambiguity keywords fire on 63% of prompts and catch well-written ones indiscriminately. Same mistake as the rule it replaced — a loose proxy standing in for a judgement it does not measure. Reverted, and locked by a test so nobody re-proposes it from first principles. AUTONOMY.md carries the change as a DATED EXPERIMENT, not a graduation: 20 unattended launches, the adversarial review leg mandatory (so the window does not start until phase 3 exists), rows per work-type, and a new human-stamped `rejected-at-review` outcome — added because `rejected` has never been used in 238 rows, having been routed around rather than earned, and a demotion trigger nothing can pull is not a safety device. 7 new tests, 661 pass. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RQeMJZznA3xTQXX4PqWg1v --- AUTONOMY.md | 79 ++++++++++++++++++-- agents/conductors/intake/_intake.py | 112 +++++++++++++++++++++++++--- tests/test_intake_review_cost.py | 92 +++++++++++++++++++++++ 3 files changed, 267 insertions(+), 16 deletions(-) create mode 100644 tests/test_intake_review_cost.py diff --git a/AUTONOMY.md b/AUTONOMY.md index ecb9888..ed757cb 100644 --- a/AUTONOMY.md +++ b/AUTONOMY.md @@ -68,6 +68,64 @@ scope addition rather than a correction of the run's own work — and **zero work-type's cap one level immediately, pending a review that cites the row. Both directions are dated doctrine edits to the table above, citing rows. +### Multi-repo autonomy experiment — 2026-08-30 + +`infer_autonomy` (the conception heuristic in +`agents/conductors/intake/_intake.py`) returned `supervised` whenever a prompt +named more than one repo. Repo count is *blast radius*, and blast radius is +already priced — `estimate_difficulty` adds 2 points per repo beyond the first. +This field is supposed to encode whether a **human's judgement** is required, and +a change touching four repos mechanically needs no more judgement than the same +change touching one. The trigger is removed. + +**This is an experiment, not a graduation, and it is deliberately not justified +by the calibration log.** The tempting argument — 238 rows, zero `rejected` — +does not survive reading: the rows run densely 2026-07-08 → 08-01 and then stop +(about seven cover all of August, against 332 completion records), so the base is +human-in-session work rather than the unattended regime this would license; the +human was in the loop for nearly every row, so failures were corrected before +they could become rows; `rejected` is structurally unreachable (a withdrawn +five-PR mechanism was logged `reverted`, a human rejecting a run's +recommendation `amended`, and two rows say verbatim "NOT a clean row for +graduation purposes"); and most of all, the 2026-07-09 review raised the +work-type caps **because** this heuristic stayed conservative about multi-repo +work. Every clean row was produced with the guard switched on. Evidence +collected under a safety device cannot license removing the device. + +**Measured effect** (all 153 `draft/` prompts, re-derived, nothing written): +`safe` 30 → 55, `supervised` 117 → 92, `human-required` unchanged at 6. +`repo_count > 1` is the *sole* supervised trigger for **25** prompts — the +largest single one, ahead of `large`-or-above (20) and architectural risk (17). + +Note what that also refutes: the multi-repo rule was **not** the cause of the +backlog reading 120-of-137 `supervised`. Those levels are declared, written by +earlier intake runs, and the triggers overlap heavily. Removing this one is +worth doing on its merits and frees 25 prompts; it is not the unblocking of the +backlog it was first taken for, and phase 3's ship-sign-off change is where that +actually lives. + +**A first draft of this change also added `human_judgement` as a supervised +trigger in `repo_count`'s place. It was measured and reverted**: `safe` fell to +24, because the ambiguity keywords fire on 63% of prompts and catch well-written +ones indiscriminately. It was the same mistake as the rule it replaced — a loose +proxy standing in for a judgement it does not measure. Recorded here so nobody +re-proposes it. + +**Terms of the experiment:** + +- **20 unattended launches** under the new rule. +- The **independent-model adversarial review leg** is mandatory for them. Until + that leg exists (batch epic phase 3), the experiment **does not start** — the + rule change ships, but no launch counts toward the window. +- Calibration rows written **per work-type**, since the graduation rule is + per-work-type and the figure usually cited is an aggregate. +- A new outcome value, **`rejected-at-review`**, stamped by the **human** in + their review slot. Without it the demotion trigger cannot fire, and an + experiment that cannot fail is not evidence. +- **Revert condition:** any `rejected-at-review` row, or a window that closes + with fewer than 20 launches and no clean read, reverts this edit and restores + the `repo_count > 1` trigger. + ### The scheduled-nightly standing grant — 2026-07-09 The human decided (2026-07-09, recorded in @@ -255,11 +313,22 @@ at PR-open (or on parking): | date | task | effective level | gates (tests/smoke/review/heart) | outcome | ``` -Outcome ∈ `merged-unchanged` / `amended` / `rejected` / `parked` / `corrective` -(the last records a use of the human-authorized corrective-PR exception above — -not an `--auto` run, but logged so the exception's use is auditable alongside the -autonomy rows). This is the evidence base for raising or lowering caps — autonomy -grows by demonstrated calibration, not by optimism. +Outcome ∈ `merged-unchanged` / `amended` / `rejected` / `rejected-at-review` / +`parked` / `corrective` (`corrective` records a use of the human-authorized +corrective-PR exception above — not an `--auto` run, but logged so the +exception's use is auditable alongside the autonomy rows). This is the evidence +base for raising or lowering caps — autonomy grows by demonstrated calibration, +not by optimism. + +**`rejected-at-review` — added 2026-08-30.** Stamped by the **human**, in their +review slot, when they would not have merged what the run produced. It exists +because `rejected` has never once been used in 238 rows, and reading why shows +the category was being routed around rather than earned: a human rejecting a +run's recommendation was logged `amended`, and an entire shipped mechanism +withdrawn on reflection — five PRs closed, branches deleted — was logged +`reverted`. A demotion trigger nothing can pull is not a safety device, and a +zero-`rejected` record produced that way is not evidence of anything. Use the +human-stamped value for any judgement about whether autonomy should rise. ### Calibration review — 2026-07-09 diff --git a/agents/conductors/intake/_intake.py b/agents/conductors/intake/_intake.py index 0e60b89..7b6f461 100755 --- a/agents/conductors/intake/_intake.py +++ b/agents/conductors/intake/_intake.py @@ -42,6 +42,7 @@ LIBRARY_REPOS, WORKSPACE_REPOS, ORGANISM_REPOS, KNOWN_REPOS, RISK_KEYWORDS, AMBIGUITY_KEYWORDS, normalise_repo, declared_header, declared_inline, effective_difficulty, strip_declarations, _hits, + effective_consequence, effective_unattended, effective_review_minutes, policy as _sizing_policy, BODY_MAP_PATH, _body_map_specs as _sizing_specs, ) @@ -274,12 +275,40 @@ def infer_priority(text: str) -> str: def infer_autonomy(level: str, factors: dict) -> str: - """safe | supervised | human-required.""" - repo_count = factors["repos_affected"] - if factors["human_judgement"] and repo_count == 0: + """safe | supervised | human-required. + + `repo_count > 1` used to force `supervised` here, and it is why 120 of the + 137 backlog prompts carried that level on 2026-08-30: nearly every real task + in this organism names a library plus its workspace, or a library plus a + downstream repo. Repo count is *blast radius*, and blast radius is already + priced — `estimate_difficulty` adds 2 points per repo beyond the first. This + field is supposed to encode something else: whether a HUMAN'S JUDGEMENT is + required. A change touching four repos mechanically needs no more judgement + than the same change touching one. + + So `repo_count` is removed and NOTHING replaces it. The first draft of this + change added `human_judgement` as a supervised trigger in its place, on the + reasoning that ambiguity is what actually predicts a park. Measured over the + backlog, that made things WORSE — `safe` fell from 30 to 24, because the + ambiguity keywords ("unclear", "investigate", "explore", "research", + "decide") fire on 63% of prompts and catch well-written ones indiscriminately. + It was the same mistake as the rule it replaced: a loose proxy standing in + for a judgement it does not measure. Dropping `repo_count` alone takes `safe` + from 30 to 55. + + CHANGED 2026-08-30 as a dated EXPERIMENT, not a graduation — see + AUTONOMY.md "Multi-repo autonomy experiment". Deliberately not justified by + the calibration log's 238 rows and zero `rejected`: those rows are July + human-in-session work (about seven cover all of August, against 332 + completions), `rejected` is structurally unreachable in that log, and every + clean row was produced *with this guard switched on* — by a review that + raised the work-type caps precisely BECAUSE this heuristic stayed + conservative. Evidence collected under a safety device cannot license + removing the device. + """ + if factors["human_judgement"] and factors["repos_affected"] == 0: return "human-required" # unscoped / needs a design decision - if (factors["architectural_risk"] or level in ("large", "too-large") - or repo_count > 1): + if factors["architectural_risk"] or level in ("large", "too-large"): return "supervised" return "safe" @@ -317,6 +346,23 @@ def analyse(text: str, source: str, themes=None): "declared_difficulty": declared.get("difficulty")} level, score, factors, estimated = effective_difficulty(p) + # The review-cost model (sizing faculty). `Witness:` is the one field that + # can never be derived: it is a promise about evidence the work will + # produce, and a plausible-sounding invented one would defeat the whole + # mechanism — the value of the field is that its ABSENCE is informative. + # So it is read if declared and left absent otherwise, which correctly + # grades the prompt `judge`. + p["witness"] = header.get("witness") + p["declared_consequence"] = header.get("declared_consequence") + p["declared_unattended"] = header.get("declared_unattended") + p["declared_review_minutes"] = header.get("declared_review_minutes") + p["declared_autonomy"] = header["declared_autonomy"] or inline.get("autonomy") + consequence, consequence_why, consequence_derived = effective_consequence(p, factors) + unattended, unattended_why, unattended_derived = effective_unattended( + p, level, factors, estimated) + review_minutes, review_minutes_derived = effective_review_minutes( + p, consequence, level) + autonomy = declared.get("autonomy") or infer_autonomy(level, factors) priority = declared.get("priority") or infer_priority(text) workflow = infer_workflow(target, repos) @@ -347,7 +393,10 @@ def analyse(text: str, source: str, themes=None): # low-confidence triage filing that means `Type: triage`, not the provisional # guess — the guess still rides in the IntakeDecision's `work_type` field. header = _render_header(title, folder, target_display, repos, level, - autonomy, priority, themes) + autonomy, priority, themes, + consequence=consequence, witness=p["witness"], + review_minutes=review_minutes, + unattended=unattended) return { "source": source, "title": title, @@ -370,19 +419,30 @@ def analyse(text: str, source: str, themes=None): "difficulty_source": "declared" if "difficulty" in declared else "estimated", "autonomy": autonomy, "autonomy_source": "declared" if "autonomy" in declared else "inferred", + "consequence": consequence, + "consequence_derived": consequence_derived, + "consequence_why": consequence_why, + "witness": p["witness"], + "review_minutes": review_minutes, + "review_minutes_derived": review_minutes_derived, + "unattended": unattended, + "unattended_derived": unattended_derived, + "unattended_why": unattended_why, "priority": priority, "priority_source": "declared" if "priority" in declared else "inferred", "declared_fields": declared, "workflow": workflow, "proposed_path": proposed, "header": header, - "risks": _risks(level, factors, confidence, target, declared, estimated), + "risks": _risks(level, factors, confidence, target, declared, estimated, + witness=p["witness"], consequence=consequence), "next_action": _next_action(proposed, confidence, folder), } def _render_header(title, work_type, target_display, repos, level, autonomy, - priority, themes=None): + priority, themes=None, consequence=None, witness=None, + review_minutes=None, unattended=None): lines = [f"# {title}", "", f"Type: {work_type}", f"Target: {target_display}"] if repos: lines.append("Repos:") @@ -397,10 +457,22 @@ def _render_header(title, work_type, target_display, repos, level, autonomy, lines += [f"- {t}" for t in themes] lines += [f"Difficulty: {level}", f"Autonomy: {autonomy}", f"Priority: {priority}", "Status: formalised"] + # The review-cost model rides below the difficulty block: what the work + # costs the organism, then what it costs the human. `Witness:` is written + # ONLY when the author supplied one — see the note in `analyse`. + if consequence: + lines.append(f"Consequence: {consequence}") + if witness: + lines.append(f"Witness: {witness}") + if review_minutes is not None: + lines.append(f"Review-minutes: {review_minutes}") + if unattended: + lines.append(f"Unattended: {unattended}") return "\n".join(lines) -def _risks(level, factors, confidence, target, declared=None, estimated=None): +def _risks(level, factors, confidence, target, declared=None, estimated=None, + witness=None, consequence=None): out = [] declared = declared or {} if "difficulty" in declared and declared["difficulty"] != estimated: @@ -410,6 +482,10 @@ def _risks(level, factors, confidence, target, declared=None, estimated=None): if field in declared: out.append(f"{field.capitalize()} {declared[field]} declared in the raw " f"text — taken as written, not inferred.") + if witness is None and consequence == "judge": + out.append("No Witness: declared — this grades `judge` (a PI's " + "quarter-hour) whatever its size. Naming one machine-checkable " + "claim now is what makes the work reviewable in minutes later.") if confidence == "low": out.append("Low classification confidence — filed to triage/ for a human " "to re-home once the work type is clear.") @@ -465,7 +541,20 @@ def write_prompt(mind: Path, decision: dict, body_text: str, source_note: str): # The header convention this agent writes (see _render_header); census parses the # same fields back out of every filed prompt. Legacy prompts pre-date the header, # so every field is optional — absence is reported, never fatal. -HEADER_FIELDS = ("type", "target", "difficulty", "autonomy", "priority", "status") +# `consequence` and `review-minutes` join the hygiene set: both are DERIVED, so +# `intake formalise` can fill them in place like any other missing field. +# +# `witness` and `unattended` deliberately do NOT. `formalise` writes every field +# it finds missing, and a `Witness:` cannot be derived — it is a promise about +# evidence the work will produce. An auto-written one would be plausible prose +# with nothing behind it, which is strictly worse than none: the entire value of +# the field is that its ABSENCE is informative, and a backlog of invented +# witnesses would grade `notify` while offering a reviewer nothing to check. +# (`unattended` stays out for the ordinary reason — it is derived on read and +# never needs storing.) The dashboard reports missing witnesses as its own +# hygiene row instead, where the fix is a human writing one. +HEADER_FIELDS = ("type", "target", "difficulty", "autonomy", "priority", "status", + "consequence", "review-minutes") # `Fix:`-anchored PR reference in a draft prompt's body — the idiom a session # writes when it fixes the bug but forgets to advance the prompt's lifecycle @@ -499,7 +588,8 @@ def parse_header(text: str) -> dict: fields = {} for line in text.splitlines()[:30]: m = re.match(r"(Type|Target|Difficulty|Autonomy|Priority|Status|" - r"Issued|Filed|Epic|Phase|Bundle|Blocked-by):\s*(\S.*)", + r"Issued|Filed|Epic|Phase|Bundle|Blocked-by|" + r"Consequence|Witness|Review-minutes|Unattended):\s*(\S.*)", line.strip()) if m: fields.setdefault(m.group(1).lower(), m.group(2).strip()) diff --git a/tests/test_intake_review_cost.py b/tests/test_intake_review_cost.py new file mode 100644 index 0000000..4071d4a --- /dev/null +++ b/tests/test_intake_review_cost.py @@ -0,0 +1,92 @@ +"""tests/test_intake_review_cost.py — intake writes the review-cost model. + +Two things are locked here. First, the header round-trip: what `analyse` derives +is what `_render_header` writes and what `parse_header` reads back, with +`Witness:` written **only** when the author supplied one. Second, the +2026-08-30 change to `infer_autonomy` — including the variant that was measured +and reverted, so nobody re-proposes it. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "agents" / "faculties" / "sizing")) +sys.path.insert(0, str(ROOT / "agents" / "conductors" / "intake")) +from _intake import ( # noqa: E402 + HEADER_FIELDS, analyse, infer_autonomy, parse_header, +) + +RAW = "Tidy the @PyAutoBrain board copy so a 403 reads as an egress block.\n" +WITNESSED = RAW + "Witness: the degraded row renders and a test pins its text.\n" + + +def test_header_round_trips_through_parse_header(): + header = analyse(WITNESSED, "test")["header"] + got = parse_header(header) + assert got["consequence"] == "notify" + assert got["witness"].startswith("the degraded row renders") + assert got["review-minutes"] == "0" + assert got["unattended"] == "ready" + + +def test_witness_is_written_only_when_supplied(): + """It is the one field that cannot be derived. An invented one would be + plausible prose with nothing behind it — strictly worse than none, because + the whole value of the field is that its absence is informative.""" + assert "Witness:" in analyse(WITNESSED, "test")["header"] + bare = analyse(RAW, "test") + assert "Witness:" not in bare["header"] + assert bare["consequence"] == "judge" + assert any("No Witness" in r for r in bare["risks"]) + + +def test_witness_stays_out_of_the_hygiene_set(): + """`intake formalise` writes every field in HEADER_FIELDS that a prompt is + missing. A `Witness:` in there would be auto-invented across the backlog.""" + assert "witness" not in HEADER_FIELDS + assert "unattended" not in HEADER_FIELDS + assert "consequence" in HEADER_FIELDS + assert "review-minutes" in HEADER_FIELDS + + +def _factors(**over): + base = {"repos_affected": 1, "architectural_risk": [], "human_judgement": [], + "library_repos": [], "workspace_repos": [], "organism_repos": [], + "library_and_workspace": False, "size_words": 100, + "scientific_complexity": [], "test_burden": [], + "memory_context_required": False} + base.update(over) + return base + + +def test_multi_repo_alone_no_longer_forces_supervised(): + """The 2026-08-30 change. Repo count is blast radius, which + `estimate_difficulty` already prices at +2 per repo; this field is about + whether a human's judgement is needed.""" + assert infer_autonomy("medium", _factors(repos_affected=4)) == "safe" + + +def test_real_judgement_signals_still_force_supervised(): + assert infer_autonomy("medium", _factors(architectural_risk=["api"])) == "supervised" + assert infer_autonomy("large", _factors()) == "supervised" + assert infer_autonomy("too-large", _factors()) == "supervised" + + +def test_ambiguity_alone_does_not_force_supervised(): + """The variant that was measured and REVERTED: adding `human_judgement` as a + supervised trigger took `safe` from 30 to 24 across the backlog, because the + ambiguity keywords fire on 63% of prompts and catch well-written ones + indiscriminately — the same mistake as the rule it replaced. Locked so it is + not re-introduced by someone reasoning from first principles.""" + assert infer_autonomy("medium", _factors(human_judgement=["investigate"])) == "safe" + + +def test_unscoped_ambiguity_is_still_human_required(): + """The one place `human_judgement` legitimately fires: nothing to scope + against, so nobody can size or gate the work.""" + assert infer_autonomy( + "medium", _factors(repos_affected=0, human_judgement=["unclear"]) + ) == "human-required" From 13ed38d0449093ab2dbf3ab0bb545543b7cd87dd Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 30 Aug 2026 22:31:40 +0000 Subject: [PATCH 3/6] dashboard: replace Quick wins with "Fits a slot"; flag prompts with no witness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pick list was `difficulty == small and autonomy == safe`. Ten prompts in the live backlog carried `safe`, so the surface whose whole job is handing out unattended work had almost nothing to hand out. It now selects on `Unattended: ready` and orders by review-minutes ASCENDING — the list is read when the human has a slot to fill and wants to know what fits in it, and `Highest priority` above is where importance is answered. Against the re-graded backlog it goes from near-empty to 71. An ungraded prompt sorts last rather than disappearing, which is the page's standing rule for unknowns. New hygiene row: prompts with no `Witness:`. Not an error — a prompt without one grades `judge` by design, and that default is what makes the field bite. But nothing derives or backfills a witness (an invented one is plausible prose with nothing behind it), so the only thing that can clear the row is a human writing one, and the page has to say which prompts are waiting. 150 of 153 today. The census record carries consequence / witness / review_minutes / unattended. The old `test_quick_wins_are_small_and_safe_only` is replaced rather than patched — the behaviour deliberately changed — by four tests covering the selector, the ordering, the ungraded-sorts-last rule and the hygiene row. 664 pass. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RQeMJZznA3xTQXX4PqWg1v --- agents/conductors/intake/_intake.py | 60 +++++++++++++++++++--- tests/test_intake_dashboard.py | 78 ++++++++++++++++++++++++----- 2 files changed, 119 insertions(+), 19 deletions(-) diff --git a/agents/conductors/intake/_intake.py b/agents/conductors/intake/_intake.py index 7b6f461..b24402f 100755 --- a/agents/conductors/intake/_intake.py +++ b/agents/conductors/intake/_intake.py @@ -1448,6 +1448,7 @@ def census(mind: Path) -> dict: *work* view — health belongs to the Heart, never here. """ records, hygiene, drift, theme_flags = [], [], [], [] + witness_flags = [] # `human_review/` prompts are collected apart from the backlog: they are not # work to pick up, they are shipped work waiting on a person. Keeping them # out of `records` keeps them out of the pick lists, the work-type sections, @@ -1484,6 +1485,13 @@ def census(mind: Path) -> dict: "title": _title(text), "difficulty": header.get("difficulty", "-"), "autonomy": header.get("autonomy", "-"), + # The review-cost model (sizing faculty). `witness` is the only + # one that can be genuinely absent on a graded prompt — nothing + # derives it — and its absence is what makes the prompt `judge`. + "consequence": header.get("consequence", "-"), + "witness": header.get("witness", ""), + "review_minutes": header.get("review-minutes", "-"), + "unattended": header.get("unattended", "-"), "priority": header.get("priority", "-"), "status": header.get("status", "-"), "epic": header.get("epic", ""), @@ -1503,6 +1511,8 @@ def census(mind: Path) -> dict: }) if len(missing) == len(HEADER_FIELDS): hygiene.append(f"{rel} — no metadata header (pre-dates intake)") + if not header.get("witness"): + witness_flags.append(str(rel)) if stray: theme_flags.append(f"{rel} — unknown theme keyword(s): " + ", ".join(stray)) @@ -1591,6 +1601,7 @@ def _count(key): "bundles": parse_bundles(mind / "bundles.md"), "theme_vocab": vocab, "theme_flags": theme_flags, + "witness_flags": witness_flags, "parked": parked, "planned": planned, "hygiene": hygiene, @@ -1933,6 +1944,29 @@ def _recent_link(e: dict) -> str: "— continue the epic rather than starting one standalone.") +def _fits_a_slot(records: list) -> list: + """The pick list for a human working in a bounded review slot. + + It replaced "Quick wins" (`difficulty == small and autonomy == safe`), which + was near-empty: ten prompts in the whole backlog carried `safe`, so the + surface that exists to hand out unattended work had almost nothing to hand + out. The two questions that actually matter are different ones — can it + finish without me, and what will it cost me to review — and the sizing + faculty now answers both. + + Ordered by review-minutes ASCENDING rather than by priority: this list is + read when the human has a slot to fill and wants to know what fits in it. + `Highest priority` above is where importance is answered. + """ + def cost(r): + try: + return int(r.get("review_minutes", "-")) + except (TypeError, ValueError): + return 99 # ungraded sorts last, never hidden + ready = [r for r in records if r.get("unattended") == "ready"] + return sorted(ready, key=lambda r: (cost(r), _pick_key(r))) + + def render_dashboard(c: dict) -> str: """Render the census as the Mind's task page (`dashboard.md`). @@ -1994,11 +2028,11 @@ def render_dashboard(c: dict) -> str: members = _epic_members(c) standalone = [r for r in records if not r.get("epic")] high = [r for r in standalone if r["priority"] == "high"] - quick = [r for r in standalone - if r["difficulty"] == "small" and r["autonomy"] == "safe"] + quick = _fits_a_slot(standalone) for title, note, rows in ( ("Highest priority", "filed as `high`", high), - ("Quick wins", "small enough, and safe enough to run unattended", quick), + ("Fits a slot", "ready to run unattended, cheapest to review first", + quick), ): shown = rows[:PICK_LIST_MAX] more = f" — showing {len(shown)} of {len(rows)}" if len(rows) > len(shown) else "" @@ -2133,6 +2167,20 @@ def render_dashboard(c: dict) -> str: "
", "Headerless prompts", ""] + [f"- `{h.split(' — ')[0]}`" for h in c["hygiene"]] + ["", "
"]) + if c.get("witness_flags"): + n = len(c["witness_flags"]) + blocks.append( + [f"{n} prompt(s) with no `Witness:` — the machine-checkable claim " + "that would make the work reviewable in minutes. Absent, a prompt " + "grades `judge` (a PI's quarter-hour) whatever its size, which is " + "the intended default and not a bug. Nothing derives or backfills " + "a witness — an invented one is plausible prose with nothing " + "behind it — so this is a human writing one, a prompt at a time.", + "", + "
", "Prompts with no witness", ""] + + [f"- `{w}`" for w in c["witness_flags"][:40]] + + ([f"- _… and {n - 40} more_"] if n > 40 else []) + + ["", "
"]) if c.get("theme_flags"): blocks.append( [f"{len(c['theme_flags'])} prompt(s) with unknown theme " @@ -2276,11 +2324,11 @@ def record_row(r): members = _epic_members(c) standalone = [r for r in records if not r.get("epic")] high = [r for r in standalone if r["priority"] == "high"] - quick = [r for r in standalone - if r["difficulty"] == "small" and r["autonomy"] == "safe"] + quick = _fits_a_slot(standalone) for title, note, rows in ( ("Highest priority", "filed as high", high), - ("Quick wins", "small enough, and safe enough to run unattended", quick), + ("Fits a slot", "ready to run unattended, cheapest to review first", + quick), ): shown = rows[:PICK_LIST_MAX] more = (f" — showing {len(shown)} of {len(rows)}" diff --git a/tests/test_intake_dashboard.py b/tests/test_intake_dashboard.py index 97c19e8..1beea28 100644 --- a/tests/test_intake_dashboard.py +++ b/tests/test_intake_dashboard.py @@ -28,10 +28,17 @@ # fixtures # --------------------------------------------------------------------------- # def _prompt(title, difficulty="medium", autonomy="supervised", priority="normal", - status="formalised"): + status="formalised", unattended=None, review_minutes=None, + consequence=None, witness=None): + extra = "" + for key, value in (("Consequence", consequence), ("Witness", witness), + ("Review-minutes", review_minutes), + ("Unattended", unattended)): + if value is not None: + extra += f"{key}: {value}\n" return (f"# {title}\n\nType: feature\nTarget: widgets\n" f"Difficulty: {difficulty}\nAutonomy: {autonomy}\n" - f"Priority: {priority}\nStatus: {status}\n\nBody prose.\n") + f"Priority: {priority}\nStatus: {status}\n{extra}\nBody prose.\n") def _mind(root: Path, drafts=None, active=None, registries=None, @@ -76,19 +83,64 @@ def test_start_here_leads_with_high_priority_smallest_first(tmp_path): assert "Later thing" not in head, "a low-priority prompt is not a pick" -def test_quick_wins_are_small_and_safe_only(tmp_path): +def test_fits_a_slot_lists_only_unattended_ready_work(tmp_path): + """Replaced "Quick wins" (`small and safe`), which was near-empty: ten + prompts in the live backlog carried `safe`, so the surface that exists to + hand out unattended work had almost nothing to hand out. The question is + not how small the work is — it is whether it can finish without the human.""" mind = _mind(tmp_path, drafts={ - "feature/widgets/a.md": _prompt("Small and safe", difficulty="small", - autonomy="safe"), - "feature/widgets/b.md": _prompt("Small but supervised", difficulty="small", - autonomy="supervised"), - "feature/widgets/c.md": _prompt("Safe but large", difficulty="large", - autonomy="safe"), + "feature/widgets/a.md": _prompt("Ready and cheap", unattended="ready", + review_minutes="0"), + "feature/widgets/b.md": _prompt("Needs slicing", difficulty="too-large", + unattended="needs-slicing"), + "feature/widgets/c.md": _prompt("Never unattended", unattended="never"), + "feature/widgets/d.md": _prompt("Ungraded"), }) - quick = _page(mind).split("**Quick wins**")[1].split("## In flight")[0] - assert "Small and safe" in quick - assert "Small but supervised" not in quick - assert "Safe but large" not in quick + slot = _page(mind).split("**Fits a slot**")[1].split("## In flight")[0] + assert "Ready and cheap" in slot + assert "Needs slicing" not in slot + assert "Never unattended" not in slot + assert "Ungraded" not in slot + + +def test_fits_a_slot_is_ordered_by_what_review_costs(tmp_path): + """Ordered by review-minutes ascending, not by priority. This list is read + when the human has a slot to fill and wants to know what fits in it; + `Highest priority` above is where importance is answered.""" + mind = _mind(tmp_path, drafts={ + "feature/widgets/a.md": _prompt("Expensive but urgent", priority="high", + unattended="ready", review_minutes="20"), + "feature/widgets/b.md": _prompt("Cheap and dull", priority="low", + unattended="ready", review_minutes="2"), + }) + slot = _page(mind).split("**Fits a slot**")[1].split("## In flight")[0] + assert slot.index("Cheap and dull") < slot.index("Expensive but urgent") + + +def test_an_ungraded_prompt_sorts_last_rather_than_being_hidden(tmp_path): + """A prompt that is `ready` with no review-minutes is still pickable; the + page's standing rule is that unknown sorts last, never disappears.""" + mind = _mind(tmp_path, drafts={ + "feature/widgets/a.md": _prompt("No cost recorded", unattended="ready"), + "feature/widgets/b.md": _prompt("Costed", unattended="ready", + review_minutes="5"), + }) + slot = _page(mind).split("**Fits a slot**")[1].split("## In flight")[0] + assert slot.index("Costed") < slot.index("No cost recorded") + + +def test_a_prompt_with_no_witness_is_reported_as_hygiene(tmp_path): + """Not an error — the intended default. But the human is the only thing + that can write one, so the page has to say which prompts are waiting.""" + mind = _mind(tmp_path, drafts={ + "feature/widgets/a.md": _prompt("Has one", witness="ids bit-identical"), + "feature/widgets/b.md": _prompt("Has none"), + }) + page = _page(mind) + hygiene = page.split("## Hygiene")[1] + assert "no `Witness:`" in hygiene + assert "feature/widgets/b.md" in hygiene + assert "feature/widgets/a.md" not in hygiene def test_every_backlog_prompt_is_one_collapsed_row_not_a_wide_table(tmp_path): From 4df92b77cfa0c04ce0fef0922f5b6f696168540b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 30 Aug 2026 22:40:41 +0000 Subject: [PATCH 4/6] AUTONOMY: fix the ship gate for unattended conditions (batch epic phase 3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four doctrine changes, each dated, each citing its evidence, each with an explicit revert condition. The gate was designed for runs with a human in the session, and every leg has an assumption that breaks at 3am. 1. WHAT A BATCH LAUNCH IS. Activation says levels bind only on an explicit --auto, "never ambient: no config flag, no environment variable, no remembered mode". A wave firing at 4am under an approval given at 17:00 is a stored grant — the shape the corrective-PR section already voids. So: a batch dispatch is ONE launch; membership is fixed at approval; the grant expires with the shift; the terms are written into the batch record; and the HUMAN performs the dispatch. A scheduler may carry the timing, never the authority. 2. LEG 4 UNDER A BATCH LAUNCH. YELLOW passes only where the reason set is the one acknowledged at launch, and nobody can acknowledge anything overnight — so a 15-hour shift would ride on that set staying frozen. It does not stay frozen for an afternoon: the log shows a drift count going 2 -> 4 -> 6 across one day, and a benign new reason appearing mid-session from a run's own sibling merge, which in a batch means wave 1 can park waves 2 and 3. The pressure has already produced a violation rather than a park — cmap-magma-default shipped under a STANDING ack, which doctrine voids. So for a batch: the human acks a named reason set for the shift; a run parks on new RED, or a new YELLOW whose repo intersects its own; a reason generated by an earlier member is named and does not park later ones; the grant expires with the shift. RED still parks everywhere, and YELLOW is still never acknowledged autonomously — this only defines how far a human's acknowledgement reaches. 3. LEG 5, THE INDEPENDENT ADVERSARY. Required for batch launches and for the multi-repo autonomy experiment. A second reading by a DIFFERENT MODEL whose job is to falsify the change's claims, the Witness: first. Step 2a was already adversarial in procedure and was still run, in practice, by the branch's own author: the efficacy review found "a healthy pass and a rote one write the identical ledger row", and that the one confirmed-wrong claim of its window lived outside the surface the stage reads. A self-run adversary leg is an ABSENT leg, not a weak one, and recording it as run is a false ledger row. Implemented as `review --witness ... --adversary`. 4. DECIDE-AND-FLAG, capped. Park-and-ask costs a whole shift when the human is not one message away. So a batch run may take the more reversible option and record it — but at most ONE per PR, stating the rejected alternative and the one-command revert (if it cannot write the revert, it was not reversible), never on a public API, default, error contract or external reporter's file, and never on a judge-tier task. The base rate this is sized against is measured: 68 of 332 August records carry a correction or retraction. Also reconciled: the gate is four legs, five under a batch launch — phrasing updated in AUTONOMY.md and both ship skills rather than renaming the gate, since ordinary --auto runs are unchanged. Calibration row gains an adversary column. 6 new tests, 670 pass. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RQeMJZznA3xTQXX4PqWg1v --- AUTONOMY.md | 145 +++++++++++++++++++++++- agents/faculties/review/AGENTS.md | 23 ++++ agents/faculties/review/_review.py | 49 +++++++- skills/ship_library/ship_library.md | 2 +- skills/ship_workspace/ship_workspace.md | 2 +- tests/test_review_adversary.py | 69 +++++++++++ 6 files changed, 281 insertions(+), 9 deletions(-) create mode 100644 tests/test_review_adversary.py diff --git a/AUTONOMY.md b/AUTONOMY.md index ed757cb..40bb914 100644 --- a/AUTONOMY.md +++ b/AUTONOMY.md @@ -164,6 +164,38 @@ doctrine edit; removing this section is the doctrine edit that retires it. - Opt-in per invocation, never ambient: no config flag, no environment variable, no "remembered" mode. +### What a batch launch is — 2026-08-30 + +A batch dispatches work into a shift that outlasts the human's session, and +waves of it may start hours after they left. Read against the rule above, a wave +firing at 4am under an approval given at 17:00 is a **stored grant** — the exact +shape the corrective-PR section voids ("a stored, reused or 'standing' +authorization does not count — it must be for this RED, **now**"). So the term +is defined here rather than left for an implementation to settle by accident: + +**A batch dispatch is ONE launch.** Specifically: + +- Its **membership is fixed at approval**, in the slot, by the human. A task not + on the approved list is not launched, however ready it looks and however much + budget is left. +- Its **grant expires at the end of the shift.** A member not dispatched within + it returns to the queue and needs the next slot's approval. +- Its **terms are written down** — members, the acknowledged Heart reason set, + the effective level per member — in `batches/-.md`, where they can + be read afterwards against what actually happened. +- **The human performs the dispatch.** A scheduler may build the review packet + and may wake a session, but the act that starts work on the approved list is + theirs. This is the line that keeps "never ambient" true: the schedule carries + the *timing*, never the *authority*. + +What this deliberately does **not** grant: a standing batch, a recurring +approval, or a config flag that makes the next batch launch itself. Each shift +is approved in its own slot, or it does not run. + +*Revert condition:* if a batch is ever found to have launched work its approval +list did not name, batching returns to per-task `--auto` until the cause is +found. + ## Checkpoint-and-continue (`supervised`) The operational mechanics of the levels-table behaviour: a run writes a @@ -195,11 +227,52 @@ otherwise. Ship sign-off and merge park the *task*, never bypass the gate — checkpoint-and-continue frees the human's session, not the checkpoint. +### Decide-and-flag (batch launches only) — 2026-08-30 + +Park-and-ask is the right behaviour when a human is one message away. In a shift +it costs the whole shift: the run stops, the question waits until the slot, and +the task needs a second batch. So a run under a **batch launch** may, at a +judgement gate, take the more reversible option and record it instead of parking. + +**Narrowly**, because the agent deciding is the poorest available judge of its +own decision's scope. The base rate is measured, not feared: **68 of 332 +completion records in 2026-08 (20%) carry a correction or a retraction**, +including one whose own claim was later marked "FALSE when this record was +written". And *reversible* is the wrong axis on its own — an API-philosophy fork +decided on behalf of an external reporter is trivially reversible in git and +irreversible in public. + +The limits, all of them: + +- **At most ONE flagged decision per PR.** A second judgement gate parks the run + exactly as today. One is a note a reviewer can hold in their head; three is a + design review they did not agree to do. +- The PR body must state the **rejected alternative** and the **one-command + revert**. *If the run cannot write the revert, the decision was not reversible* + — park. +- **Never** where the decision touches a public API, a default value, an error + contract, or a file named in an external reporter's issue. +- **Never** for a `judge`-tier task (`Consequence:` — REFERENCE.md "The + review-cost model"). A tier that costs a PI's quarter-hour is one where the PI + makes the call. +- The PR carries a `decision-taken` label so the review surface sorts it above + clean work, and the calibration row names it. + +This moves review debt from the issue tracker into the diff, which is a real +cost — the diff is where an overloaded reviewer is least able to interrogate a +choice. The cap, the revert line and the tier exclusion are what keep that cost +to one bounded item per PR rather than an unmarked scatter. + +*Revert condition:* two flagged decisions the human would not have taken retires +this section; it returns to park-and-ask. + ## The autonomous-ship gate An unattended ship (checkpoint 2 at `safe`) requires **all four legs**, no -substitutions. Audited 2026-07-08 (issue #38); each leg carries an -applicability rule so "n/a" is a stated fact, never an assumption: +substitutions — **plus leg 5 under a batch launch**, where no human is reachable +and the review leg's independence stops being optional. Audited 2026-07-08 +(issue #38); each leg carries an applicability rule so "n/a" is a stated fact, +never an assumption: 1. **Tests** — worktree pytest (full suite, `-x`) on every **shipped** repo, *plus* every downstream library repo when the diff touches public API @@ -233,9 +306,72 @@ applicability rule so "n/a" is a stated fact, never an assumption: reason, or RED, parks the run. Never ambient, never carried across sessions. +5. **Independent adversary** — **added 2026-08-30, required for a BATCH + launch** (below) and for any run counting toward the multi-repo autonomy + experiment; optional elsewhere. A second reading of the same diff by a + **different model from the one that wrote it**, whose job is not to review + the change but to **falsify its claims — the `Witness:` first**. Run it as + `pyauto-brain review --task --witness "" + --adversary`. + + Why it exists, and why leg 3 does not already cover it: + `complete/2026/08/falsified-by-checkpoint-efficacy-review.md` found that the + review leg on autonomous ships was in practice **the branch's own author**, + that "a healthy pass and a rote one write the identical ledger row", and that + the one confirmed-wrong load-bearing claim of its window lived *outside* the + surface the stage reads — "the stage could not have caught the one escape + that actually happened". The catches that did happen came from an independent + model reading the same diff. + + **A self-run adversary leg is an absent leg, not a weak one**, and recording + it as run is a false ledger row. The verdict is recorded in the calibration + row alongside the other four. + + *Revert condition:* if the leg produces no finding across 30 runs it is + ceremony, and this clause is removed rather than left as theatre. + A failed leg downgrades the run to a human checkpoint: state written to the issue, nothing force-shipped, never modify code to make a leg pass. +### Leg 4 under a batch launch — 2026-08-30 + +Leg 4 as written assumes a human is reachable: YELLOW passes only where the +reason set is the one acknowledged *at launch*, and any new reason parks the +run. Overnight nobody can acknowledge anything, so a fifteen-hour shift would +ride on that set staying frozen. **It does not stay frozen for an afternoon.** +The log shows a drift count growing 2 → 4 → 6 across a single day, re-asked at +each ship; and a *benign new* reason appearing mid-session from a run's own +sibling merge ("release validation stale: source moved since rehearsal") — which +in a batch means **wave 1 can manufacture the reason that parks waves 2 and 3**. +The 2026-07-09 review records that Heart never read GREEN: every shipped run +went out on an acked YELLOW. The base rate says this leg blocks by default. + +That pressure has already produced a violation rather than a park: +`complete/2026/08/cmap-magma-default.md` shipped under a **standing** ack, which +the corrective-PR section below explicitly voids. Doctrine moving deliberately is +the alternative to doctrine bending under load. + +So, **for a batch launch only**: + +- At slot time the human acknowledges a **named reason set for the shift**, + written verbatim into `batches/.md` and into each member's `active.md` + `- heart-ack:` block. +- A run parks on a new **RED**, or on a new **YELLOW reason whose repo + intersects the run's own repos**. A new YELLOW elsewhere in the organism does + not park it — the reason is out of the branch's blast radius, and legs 1-3 and + 5 gate the branch itself. +- A reason **generated by an earlier member of the same batch** is named as such + in the batch record and does not park later members. +- The grant **expires with the shift**. It is not standing, not ambient, and not + carried into the next batch. + +Unchanged: RED still parks at every level; YELLOW is still never acknowledged +*autonomously* — the acknowledgement is a human act performed in the slot, and +this section only defines how far it reaches. + +*Revert condition:* one escape traceable to a YELLOW reason this section let +through restores the exact-set rule for batches. + ## Corrective-PR exception for Heart RED (human-authorized) Heart RED forbids commit, push and PR-open at every autonomy level (the levels @@ -310,7 +446,7 @@ work. That resumes only along this path: at PR-open (or on parking): ```markdown -| date | task | effective level | gates (tests/smoke/review/heart) | outcome | +| date | task | effective level | gates (tests/smoke/review/heart[/adversary]) | outcome | ``` Outcome ∈ `merged-unchanged` / `amended` / `rejected` / `rejected-at-review` / @@ -389,7 +525,8 @@ tier), never by weakening leg 4. - `start_dev` — `--auto` usage, effective-level computation, plan-to-issue for `safe`, launch-acknowledgement recording (its "--auto mode" section). -- `ship_library` / `ship_workspace` — the four-leg gate at step 4, stop at +- `ship_library` / `ship_workspace` — the four-leg gate at step 4 (five under a + batch launch), stop at PR-open, validation checklist, calibration append; the RED-handling step points here for the human-authorized corrective-PR exception. diff --git a/agents/faculties/review/AGENTS.md b/agents/faculties/review/AGENTS.md index 47aaf43..dee9475 100644 --- a/agents/faculties/review/AGENTS.md +++ b/agents/faculties/review/AGENTS.md @@ -70,6 +70,29 @@ Heart and the agent reasons over the verdict. (`docs/agent_failure_modes.md` item 6 Outcome) found that across 22 ship gates a healthy pass and a skipped one wrote the identical ledger row. An empty surface requires nothing. +2b. **Independent-adversary mode** (`--adversary`, autonomous-ship gate leg 5, + added 2026-08-30). Required for a **batch launch**, where no human is + reachable; optional elsewhere. Run as + `pyauto-brain review --task --witness "" + --adversary`. + + Two things change, and only two. First, the task's **`Witness:`** — the + machine-checkable claim the work was scoped around + (`PyAutoMind/REFERENCE.md`, "The review-cost model") — is lifted to the head + of the claims surface and falsified **first**: if the witness does not hold, + the work did not do what it promised and nothing else on the surface matters. + Second, the reader must be a **different model from the one that wrote the + branch**. + + That second rule is the whole leg. Step 2a is already adversarial in + *procedure*, and it was still run, in practice, by the branch's own author — + `complete/2026/08/falsified-by-checkpoint-efficacy-review.md` found that "a + healthy pass and a rote one write the identical ledger row" and that the one + confirmed-wrong load-bearing claim of its window lived outside the surface + the stage reads. The catches that did happen came from an independent model + reading the same diff. **A self-run adversary leg is an absent leg, not a + weak one**, and recording it as run is a false ledger row. + 3. Map the outcome to the verdict: any unresolved must-fix → **FINDINGS** (ranked list, file:line, failure scenario) — including any `unverified-claim` from step 2a; nothing → **CLEAN**; could not diff --git a/agents/faculties/review/_review.py b/agents/faculties/review/_review.py index 44aeaee..da0a330 100755 --- a/agents/faculties/review/_review.py +++ b/agents/faculties/review/_review.py @@ -170,7 +170,28 @@ def resolve_repos(task: str | None, repos: list[str]) -> list[Path]: return [Path(r).resolve() for r in repos] -def emit_human(surfaces: list[dict]) -> None: +ADVERSARY_CONTRACT = """\ +INDEPENDENT ADVERSARY MODE (autonomous-ship gate, leg 5) + + Who may run this: NOT the session or model that wrote the branch. The leg's + entire value is independence — the organism's own audit found the review leg + on autonomous ships was in practice the branch's own author, that "a healthy + pass and a rote one write the identical ledger row", and that the one + confirmed-wrong load-bearing claim of its window lived outside the surface the + stage reads. The catches that did happen came from a different model reading + the same diff. A self-run adversary leg is not a weaker version of this leg; + it is an absent one, and recording it as run is a false ledger row. + + Your job is NOT to review the change. It is to FALSIFY its claims — starting + with the witness, which is the claim the whole task was scoped around. + + For each claim: what would make this false, and what in the diff, the tests or + a run actually shows it is not? Then say which. A claim you cannot falsify AND + cannot find a basis for is a FINDING (unverified-claim), not a pass. +""" + + +def emit_human(surfaces: list[dict], witness: str = "", adversary: bool = False) -> None: print("== ReviewSurface (review faculty — surface only; the verdict is the") print(" reviewing agent's, per agents/faculties/review/AGENTS.md) ==") for s in surfaces: @@ -191,6 +212,15 @@ def emit_human(surfaces: list[dict]) -> None: print(f" ? {c}") print("\nVerdict rubric: CLEAN (nothing must change) | FINDINGS (ranked,") print("file:line, failure scenario) | BLOCKED (could not review — say why).") + if witness: + print() + print(" THE WITNESS — the claim this task was scoped around, and the") + print(" one whose failure means the work did not do what it promised:") + print(f" * {witness}") + print(" Falsify it first. If it does not hold, nothing below matters.") + if adversary: + print() + print(ADVERSARY_CONTRACT) print("A load-bearing claim above with no falsified-by basis in the branch is") print("a FINDING (unverified-claim) — see the faculty AGENTS.md.") if any(s.get("claims_to_falsify") for s in surfaces): @@ -208,6 +238,13 @@ def main(argv=None) -> int: f"({WT_BASE})") ap.add_argument("--repo", action="append", default=[], help="explicit repo checkout path") ap.add_argument("--json", action="store_true", dest="as_json") + ap.add_argument("--witness", default="", + help="the task's Witness: header — the claim the work was " + "scoped around, falsified first") + ap.add_argument("--adversary", action="store_true", + help="independent-adversary mode (autonomous-ship gate leg " + "5): print the independence contract. Must be run by a " + "different model from the one that wrote the branch") a = ap.parse_args(argv) if not a.task and not a.repo: print("review: pass --task or --repo ", file=sys.stderr) @@ -221,9 +258,15 @@ def main(argv=None) -> int: print("review: no reviewable diff against origin/main", file=sys.stderr) return 4 if a.as_json: - print(json.dumps({"review_surface": surfaces}, indent=2)) + payload = {"review_surface": surfaces} + if a.witness: + payload["witness"] = a.witness + if a.adversary: + payload["mode"] = "independent-adversary" + payload["contract"] = ADVERSARY_CONTRACT + print(json.dumps(payload, indent=2)) else: - emit_human(surfaces) + emit_human(surfaces, witness=a.witness, adversary=a.adversary) return 0 diff --git a/skills/ship_library/ship_library.md b/skills/ship_library/ship_library.md index 0fd9adc..38266f6 100644 --- a/skills/ship_library/ship_library.md +++ b/skills/ship_library/ship_library.md @@ -82,7 +82,7 @@ landed. This is feature-development git work, not a Build/release step. In local-dev delegate the mechanical part to an execution-tier subagent; elsewhere run it directly. If any step fails, stop and report — do not proceed. -**Under `--auto`:** all four legs of the autonomous-ship gate must pass (step +**Under `--auto`:** all four legs of the autonomous-ship gate must pass — **five under a batch launch**, which adds the independent-adversary leg (`AUTONOMY.md` leg 5) — (step 3 note); then ship **without interactive sign-off** — the PR body additionally carries the `## Validation checklist` section ([`reference.md`](reference.md) → "Validation checklist (--auto)"), the run diff --git a/skills/ship_workspace/ship_workspace.md b/skills/ship_workspace/ship_workspace.md index 36ec0a6..ba95654 100644 --- a/skills/ship_workspace/ship_workspace.md +++ b/skills/ship_workspace/ship_workspace.md @@ -79,7 +79,7 @@ directly), `gh pr create --label pending-release`, verify the label, and cross-reference the upstream library PR if linked. In local-dev delegate to a execution-tier subagent; elsewhere run directly. Any failure → stop and report. -**Under `--auto`:** all four legs of the autonomous-ship gate must pass (step +**Under `--auto`:** all four legs of the autonomous-ship gate must pass — **five under a batch launch**, which adds the independent-adversary leg (`AUTONOMY.md` leg 5) — (step 3 note); ship without interactive sign-off, add the `## Validation checklist` to the PR body (`../ship_library/reference.md` → "Validation checklist (--auto)"), **stop at PR-open**, append the calibration row to diff --git a/tests/test_review_adversary.py b/tests/test_review_adversary.py new file mode 100644 index 0000000..5f30364 --- /dev/null +++ b/tests/test_review_adversary.py @@ -0,0 +1,69 @@ +"""tests/test_review_adversary.py — the independent-adversary leg (gate leg 5). + +Step 2a was already adversarial in procedure and was still run, in practice, by +the branch's own author. This leg's only new content is *independence* plus +*ordering* — the task's witness is falsified before anything else — so those are +what is pinned here. +""" + +from __future__ import annotations + +import importlib.util +import json +import subprocess +import sys +from pathlib import Path + +BRAIN = Path(__file__).resolve().parents[1] +_spec = importlib.util.spec_from_file_location( + "_review_adversary_under_test", + BRAIN / "agents" / "faculties" / "review" / "_review.py") +_review = importlib.util.module_from_spec(_spec) +sys.modules["_review_adversary_under_test"] = _review +_spec.loader.exec_module(_review) + +WITNESS = "ids bit-identical, 62 -> 9.7 ms" + + +def _run(*args): + return subprocess.run( + [sys.executable, str(BRAIN / "agents" / "faculties" / "review" / "_review.py"), + "--repo", str(BRAIN), *args], + capture_output=True, text=True) + + +def test_the_witness_is_lifted_and_falsified_first(): + out = _run("--witness", WITNESS).stdout + assert WITNESS in out + assert "THE WITNESS" in out + assert "Falsify it first" in out + + +def test_no_witness_section_when_none_is_declared(): + """A task with no witness grades `judge` and is reviewed by a human; the + surface must not imply a claim that was never made.""" + assert "THE WITNESS" not in _run().stdout + + +def test_adversary_mode_states_who_may_not_run_it(): + """The independence rule is the leg. A surface that omits it invites the + exact failure the leg exists to close.""" + out = _run("--adversary").stdout + assert "NOT the session or model that wrote the branch" in out + assert "A self-run adversary leg is not a weaker version" in _review.ADVERSARY_CONTRACT + + +def test_contract_is_absent_unless_asked_for(): + assert "INDEPENDENT ADVERSARY MODE" not in _run().stdout + + +def test_json_surface_carries_witness_and_mode(): + payload = json.loads(_run("--json", "--witness", WITNESS, "--adversary").stdout) + assert payload["witness"] == WITNESS + assert payload["mode"] == "independent-adversary" + assert "contract" in payload + + +def test_json_surface_omits_them_when_not_asked(): + payload = json.loads(_run("--json").stdout) + assert "witness" not in payload and "mode" not in payload From b1d3aa9980e06d45b9dbedf85f01a7314fb4b5c2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 30 Aug 2026 23:22:36 +0000 Subject: [PATCH 5/6] batch: add the Batch Agent conductor, and the Lane header (epic phases 1-2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `pyauto-brain batch plan` composes one unattended shift and emits a BatchDecision. It PROPOSES; it never dispatches — approving it in a slot is what launches a batch (AUTONOMY.md, "What a batch launch is"), so the schedule can carry the timing but never the authority. The composition rule is a budget in REVIEW-MINUTES, not a task count: sum(Review-minutes) over glance and judge members <= the slot's budget, default 45. Read against the ledger an honest hour holds about three library-touching tasks, so planning by count over-promises capacity roughly threefold and lands the overflow on the human at 6am. Work above the budget is the FILL — zero review-minute work, sized by the token allowance rather than the human's hour, which is what lets the whole weekly budget be spent without growing the queue. Constraints, each of which states itself in `rejected` (a planner that silently drops work teaches the human to distrust its numbers): - one member per LIBRARY repo per shift. They do not collide at dispatch — separate worktrees — they collide at merge, when the first /prm moves main and invalidates the others' test and smoke evidence. Workspace, docs and organ repos are exempt: two docs changes in one shift cost nothing. - one slice per epic (phases are ordered, so they could not parallelise anyway, and this is what interleaves epics with standalone work) - Unattended: ready only; Blocked-by: excluded - lane match, with the other lane's ready count REPORTED rather than dropped Backpressure ramps and never deadlocks, counted in tasks awaiting review rather than PRs (94 of 332 August records named two or more PRs, so a PR cap trips on one healthy batch): clear -> full budget; above half -> the review-bearing half halves, the fill does not; at the cap -> the floor, fill only, dispatched whether or not the human turned up. An EMPTY floor is reported as a finding, not a deadlock — it means nothing in the backlog costs zero review-minutes, which is what the notify tier and the Witness field exist to change. Lane detection is probed from the environment, never declared: a session that could be told where it is could plan local-dev work it cannot run. It reads the signal the organism already uses — a remote session has no gh. Also: the Lane: any | local-dev header in the sizing faculty and intake, spelled in WORKFLOW.md's existing environment vocabulary rather than a parallel cloud/laptop one. Against the live backlog the planner picks 2 members for 40 of 45 minutes — the honest capacity while 150 of 153 prompts carry no witness and therefore grade judge at 20 minutes each. That number is the point, not a defect. 12 new tests, 682 pass. (Pre-existing and untouched: skills/prm exceeds the 200-line skill budget.) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RQeMJZznA3xTQXX4PqWg1v --- AGENTS.md | 1 + agents/conductors/batch/AGENTS.md | 108 ++++++++++++ agents/conductors/batch/_batch.py | 255 ++++++++++++++++++++++++++++ agents/conductors/batch/batch.sh | 24 +++ agents/conductors/intake/_intake.py | 5 +- agents/faculties/sizing/_sizing.py | 15 +- bin/pyauto-brain | 4 +- skills/batch/SKILL.md | 22 +++ skills/batch/agents/openai.yaml | 4 + tests/test_batch_plan.py | 130 ++++++++++++++ 10 files changed, 564 insertions(+), 4 deletions(-) create mode 100644 agents/conductors/batch/AGENTS.md create mode 100644 agents/conductors/batch/_batch.py create mode 100755 agents/conductors/batch/batch.sh create mode 100644 skills/batch/SKILL.md create mode 100644 skills/batch/agents/openai.yaml create mode 100644 tests/test_batch_plan.py diff --git a/AGENTS.md b/AGENTS.md index 80984d4..a2931e6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -132,6 +132,7 @@ the `/` slash commands. | Verb | Purpose | Entrypoint | |------|---------|------------| | `intake` | Conceive a task: turn raw input into a formal, headed PyAutoMind prompt (files it; never starts dev) | `bin/pyauto-brain intake` | +| `batch` | Compose one unattended shift: the BatchDecision, planned against the slot's REVIEW-MINUTE budget rather than a task count — proposes, never dispatches | `bin/pyauto-brain batch` | | `community` | The ears — the organism's receptive function: scan/triage user-filed GitHub issues + PRs and review requests across the repos; drafts stay human-gated, dev work routes via start_dev_for_user (never posts) | `bin/pyauto-brain community` | | `feature` | Reason over PyAutoMind feature tasks: select, size, phase, plan for start_dev | `bin/pyauto-brain feature` | | `bug` | The immune system: classify a bug/regression/Heart finding, locate the fix, plan the repair | `bin/pyauto-brain bug` | diff --git a/agents/conductors/batch/AGENTS.md b/agents/conductors/batch/AGENTS.md new file mode 100644 index 0000000..89f18a1 --- /dev/null +++ b/agents/conductors/batch/AGENTS.md @@ -0,0 +1,108 @@ +# Batch Agent + +> **Tier: conductor** — a front-door agent a human *drives*. It *composes one +> unattended shift*: given the queue, the backlog and the review-queue state, it +> emits a **BatchDecision** — the members, what they will cost the human to +> review, and what it rejected and why. It **proposes; it never dispatches.** +> Approving the proposal in a slot is what launches a batch. + +## What it is for + +The organism has no throughput problem. August 2026 shipped **332 completion +records, about eleven a day**. The scarce resource is the human's judgement, and +a batch layer that plans by task count spends it faster rather than slower. + +So the composition rule is a budget in **review-minutes**, not a count: + +> Σ `Review-minutes:` over `glance` and `judge` members ≤ the slot's budget +> (default 45 — a slot also has to queue the next batch). + +Read against the ledger, an honest hour holds about three library-touching +tasks. Planning by count over-promises capacity roughly threefold, and the +overflow lands on the human at 6am. + +Everything above that budget is the **fill**: work costing zero review-minutes +(`notify`-tier work, slicing, witness authoring, backlog re-grading, deeper +verification of work that already passed). The fill is sized by the remaining +token allowance rather than by the human's hour, which is what lets the organism +spend its whole weekly budget without growing the review queue. **Research and +experiments are never fill** — they produce verdicts, and a verdict is the most +expensive review there is. + +## What it consults, and what it owns + +It owns **membership**, which is an act. Every *judgement* it uses — consequence +tier, review-minutes, readiness, difficulty — comes from the **sizing faculty**, +because `ORGANISM.md` makes faculties the opinion sinks and forbids a conductor +consulting a conductor. If a rule here starts needing to *judge* rather than +*select*, it belongs in the faculty. + +## The constraints, and why each exists + +| Constraint | Why | +|---|---| +| **Review-minute budget** | the human's hour is the binding resource; see above | +| **One member per *library* repo per shift** | concurrent members do not collide at dispatch (separate worktrees) — they collide at **merge**, because the first `/prm` moves `main` and invalidates the others' test and smoke evidence. Workspace, docs and organ repos are exempt: two docs changes in one shift cost nothing. Work concentrates in four repos (PyAutoFit 118/332 August records, PyAutoArray 98, PyAutoGalaxy 82, PyAutoLens 78), so effective parallelism is two or three, not six. | +| **One slice per epic per batch** | epic phases are ordered, so two members could not run in parallel anyway — and this is what interleaves small pieces of long programmes with standalone work | +| **`Unattended: ready` only** | `needs-slicing` goes to the decomposition pass; `never` never enters a batch | +| **Lane match** | a session detects its own lane and plans only that one | +| **Backpressure** | see below | + +Everything rejected says so, with its reason. A planner that silently drops work +teaches the human to distrust the number it reports. + +## Backpressure ramps; it never deadlocks + +Counted in **tasks awaiting review**, never in PRs — 94 of 332 August records +named two or more PRs, so a PR-count cap trips on a single healthy batch. + +- clear → the full budget; +- above half the cap → the review-bearing half is **halved** (the fill is not: + it does not touch the queue); +- at the cap → the batch is the **floor**: fill only, dispatched whether or not + the human turned up. + +A missed slot is the common case for an academic, not an exception, and a +conference week must not stop the thing whose whole purpose is working while +nobody watches. An **empty floor is a finding, not a deadlock** — it means +nothing in the backlog costs zero review-minutes, which is exactly what the +`notify` tier and the `Witness:` field exist to change, and the decision says so +in those words. + +## Lane detection + +`local-dev` or `web-github`, probed from the environment rather than declared, +and from the signal the organism already uses: a remote session has no `gh` +(measured; `skills/GITHUB_ACCESS.md`). No flag and no environment variable +decides it — a session that could lie about where it is could plan `local-dev` +work it cannot run. A session reports the other lane's ready count rather than +hiding it: *"4 local-dev task(s) are ready — run `batch plan` from the laptop."* + +## It proposes; the human launches + +`AUTONOMY.md` ("What a batch launch is") defines a batch dispatch as **one +launch**: membership fixed at approval, the grant expiring with the shift, the +terms written into `PyAutoMind/batches/-.md`, and the human +performing the dispatch. A scheduler may build the review packet and wake a +session; the act that starts work on the approved list is the human's. That is +the line that keeps "never ambient" true — the schedule carries the *timing*, +never the *authority*. + +## Running + +``` +pyauto-brain batch plan # the BatchDecision for this lane +pyauto-brain batch plan --budget 45 # review-minutes available +pyauto-brain batch plan --awaiting-review 6 # backpressure input +pyauto-brain batch plan --json +``` + +Stdlib-only, offline, writes nothing. + +## Not built yet + +`slice` (the decomposition pass doctrine has named since inception) and +`collect` (the review packet) are the conductor's other two verbs — see the +epic ledger, `PyAutoMind/draft/feature/pyautomind/two_slot_batching_epic.md`. +`plan` is useful on its own: run it in a slot and dispatch by tapping the +dashboard's existing chips. diff --git a/agents/conductors/batch/_batch.py b/agents/conductors/batch/_batch.py new file mode 100644 index 0000000..adca39a --- /dev/null +++ b/agents/conductors/batch/_batch.py @@ -0,0 +1,255 @@ +#!/usr/bin/env python3 +"""agents/conductors/batch/_batch.py — the batch conductor. + +Composes what goes into one unattended shift, and reports what came back. Thin +by construction: every judgement it uses — consequence tier, review-minutes, +readiness, difficulty — belongs to the sizing faculty, because `ORGANISM.md` +makes faculties the opinion sinks and forbids a conductor consulting a +conductor. This module decides *membership*, which is an act, not an opinion. + +The composition rule, and the reason the epic exists: + + sum(Review-minutes) over `glance` and `judge` members <= the slot budget + +not a task count. Read against the ledger, an honest review hour holds about +three library-touching tasks; a design that plans by count over-promises +capacity roughly threefold and lands the overflow on the human at 6am. + +Everything above that budget is the FILL: work costing zero review-minutes +(`notify`-tier work, slicing, witness authoring, re-grading, deeper verification +of work that already passed). It is sized by the remaining token allowance +rather than by the human's hour, which is what lets the organism spend its whole +weekly budget without growing the review queue. Research is never fill — it +produces verdicts, and a verdict is the most expensive review there is. + +Stdlib-only and offline, like every Brain entrypoint. +""" +from __future__ import annotations + +import argparse +import json +import shutil +import sys +from pathlib import Path + +BRAIN = Path(__file__).resolve().parents[3] +sys.path.insert(0, str(BRAIN / "agents" / "faculties" / "sizing")) +from _sizing import ( # noqa: E402 + BODY_MAP_PATH, LIBRARY_REPOS, effective_consequence, effective_difficulty, + effective_review_minutes, effective_unattended, parse_prompt, priority_rank, +) + +# One slot's worth of the human's attention. Not a task count — see the module +# docstring. 45 rather than 60 because a slot also has to queue the next batch. +DEFAULT_REVIEW_BUDGET = 45 +# Above half the cap the review-bearing half is halved; at the cap the batch is +# fill only. Counted in TASKS AWAITING REVIEW, never in PRs: 94 of 332 records in +# 2026-08 named two or more PRs, so a PR-count cap trips on one healthy batch. +DEFAULT_BACKPRESSURE_CAP = 8 +CHEAP_TIERS = ("notify",) + + +def detect_lane() -> str: + """`local-dev` or `web-github` — where this session is running. + + Probed from the environment rather than declared, and deliberately from the + same signal the rest of the organism already uses: a remote session has no + `gh` (measured, documented in `skills/GITHUB_ACCESS.md` — installing one is + a trap that authenticates and then 403s every repo-scoped call). No env var + and no flag decides this; a session that could lie about where it is could + plan `local-dev` work it cannot run. + """ + return "local-dev" if shutil.which("gh") else "web-github" + + +def _lane_ok(record_lane: str, session_lane: str) -> bool: + """A `local-dev` task runs only in a `local-dev` session; `any` runs anywhere.""" + return record_lane != "local-dev" or session_lane == "local-dev" + + +def grade(path: Path, mind: Path) -> dict: + """Everything the planner needs about one prompt, from the faculty.""" + p = parse_prompt(path, mind) + level, score, factors, derived = effective_difficulty(p) + tier, why, _ = effective_consequence(p, factors) + ready, ready_why, _ = effective_unattended(p, level, factors, derived) + minutes, _ = effective_review_minutes(p, tier, level) + return { + "path": p["path"], "repos": p["repos"], "work_type": p["work_type"], + "difficulty": level, "score": score, "consequence": tier, + "witness": p.get("witness"), "review_minutes": minutes, + "unattended": ready, "why": (why or [""])[0], + "ready_why": ready_why, "epic": None, "lane": p.get("lane") or "any", + "priority": p.get("priority") or "normal", + "blocked": bool(p.get("blocked_by")), + } + + +def _epic_of(text: str) -> str: + for line in text.splitlines()[:30]: + if line.lower().startswith("epic:"): + return line.split(":", 1)[1].strip() + return "" + + +def survey(mind: Path) -> list[dict]: + """Grade every backlog prompt. Epic membership rides along for the + one-slice-per-epic rule.""" + out = [] + for f in sorted(mind.glob("draft/**/*.md")): + if f.name == "README.md": + continue + g = grade(f, mind) + g["epic"] = _epic_of(f.read_text(encoding="utf-8", errors="replace")) + out.append(g) + return out + + +def plan(records: list[dict], *, budget: int = DEFAULT_REVIEW_BUDGET, + session_lane: str = "web-github", awaiting_review: int = 0, + cap: int = DEFAULT_BACKPRESSURE_CAP) -> dict: + """Compose the next batch — a BatchDecision. + + Every constraint states itself in `rejected`, because a planner that + silently drops work teaches the human to distrust the number it reports. + """ + rejected: list[tuple[str, str]] = [] + + # Backpressure RAMPS; it never deadlocks. A missed slot is the common case + # for an academic, and a conference week must not stop the thing whose whole + # purpose is working while nobody watches. + if awaiting_review >= cap: + effective_budget, pressure = 0, "at cap — fill only" + elif awaiting_review > cap / 2: + effective_budget, pressure = budget // 2, "above half cap — halved" + else: + effective_budget, pressure = budget, "clear" + + pool = [] + for r in records: + if r["unattended"] != "ready": + rejected.append((r["path"], f"unattended: {r['unattended']}")) + elif r["blocked"]: + rejected.append((r["path"], "declares Blocked-by:")) + elif not _lane_ok(r["lane"], session_lane): + rejected.append((r["path"], f"lane {r['lane']}, session {session_lane}")) + else: + pool.append(r) + + # Cheapest first: this list is read when the human has a slot to fill and + # wants to know what fits in it. Importance is answered by the queue's order. + pool.sort(key=lambda r: (r["review_minutes"], priority_rank(r), r["path"])) + + members, spent, seen_epics, libs = [], 0, set(), set() + for r in pool: + if r["epic"] and r["epic"] in seen_epics: + rejected.append((r["path"], f"epic {r['epic']} already in this batch")) + continue + # Concurrent members do not collide at dispatch (separate worktrees) — + # they collide at MERGE, because the first /prm moves main and + # invalidates the others' test and smoke evidence. Only LIBRARY repos + # claim a shift: they are what the library-first gate serialises and + # what downstream suites are re-run against. Workspace, docs and organ + # repos are exempt — two docs changes in one shift cost nothing. + clash = next((x for x in r["repos"] if x in LIBRARY_REPOS and x in libs), None) + if clash: + rejected.append((r["path"], f"{clash} already claimed this shift")) + continue + cost = 0 if r["consequence"] in CHEAP_TIERS else r["review_minutes"] + if spent + cost > effective_budget: + rejected.append((r["path"], f"{cost} min would exceed the budget")) + continue + members.append(r) + spent += cost + if r["epic"]: + seen_epics.add(r["epic"]) + libs.update(x for x in r["repos"] if x in LIBRARY_REPOS) + + return { + "session_lane": session_lane, + "review_budget": budget, + "effective_budget": effective_budget, + "backpressure": {"awaiting_review": awaiting_review, "cap": cap, + "state": pressure}, + "review_minutes_planned": spent, + "members": members, + "rejected": rejected, + "other_lane_ready": sum( + 1 for r in records + if r["unattended"] == "ready" and not _lane_ok(r["lane"], session_lane)), + } + + +def emit(d: dict) -> None: + lane = d["session_lane"] + print("== BatchDecision ==") + print(f"Session lane: {lane}") + print(f"Review budget: {d['effective_budget']} of {d['review_budget']} min " + f"({d['backpressure']['state']}; " + f"{d['backpressure']['awaiting_review']} awaiting review, " + f"cap {d['backpressure']['cap']})") + print(f"Planned: {d['review_minutes_planned']} review-minutes over " + f"{len(d['members'])} member(s)") + print() + if d["members"]: + for r in d["members"]: + cost = 0 if r["consequence"] in CHEAP_TIERS else r["review_minutes"] + print(f" {cost:>3} min {r['consequence']:<7} {r['path']}") + if not r["witness"]: + print(" (no witness — reviewed as `judge`)") + elif d["effective_budget"] == 0: + # At the cap the batch is the FLOOR: fill only, dispatched whether or + # not the human turned up. An empty floor is a finding, not a deadlock — + # it means nothing in the backlog costs zero review-minutes, which is + # what the `notify` tier and the witness field exist to change. + print(" (no members — at the backpressure cap, so this batch is the") + print(" FLOOR: fill only. Nothing in the backlog currently qualifies") + print(" as fill, which is itself the finding — clear the review queue,") + print(" or write witnesses so work can grade `notify`.)") + else: + print(" (no members — see the rejections below)") + print() + if d["other_lane_ready"]: + # Reported, never silently dropped: a task the session cannot run is + # still a task the human can, from the other machine. + print(f"{d['other_lane_ready']} local-dev task(s) are ready — run " + "`batch plan` from the laptop to see them.") + print() + print(f"Not selected: {len(d['rejected'])}") + counts: dict[str, int] = {} + for _, why in d["rejected"]: + counts[why.split(":")[0].split(" already")[0]] = counts.get( + why.split(":")[0].split(" already")[0], 0) + 1 + for why, n in sorted(counts.items(), key=lambda kv: -kv[1])[:8]: + print(f" {n:>4} {why}") + print() + print("This is a PROPOSAL. Approving it in the slot is what launches the") + print("batch — membership is fixed at approval and the grant expires with") + print("the shift (AUTONOMY.md, \"What a batch launch is\").") + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(prog="batch") + ap.add_argument("verb", choices=["plan"], nargs="?", default="plan") + ap.add_argument("--mind", type=Path, default=BODY_MAP_PATH.parent) + ap.add_argument("--budget", type=int, default=DEFAULT_REVIEW_BUDGET, + help="review-minutes available in the slot") + ap.add_argument("--awaiting-review", type=int, default=0, + help="tasks already awaiting review (backpressure input)") + ap.add_argument("--cap", type=int, default=DEFAULT_BACKPRESSURE_CAP) + ap.add_argument("--lane", default="", help="override the detected session lane") + ap.add_argument("--json", action="store_true", dest="as_json") + a = ap.parse_args(argv) + + mind = a.mind.resolve() + if not (mind / "draft").is_dir(): + print(f"batch: no PyAutoMind backlog at {mind}", file=sys.stderr) + return 4 + d = plan(survey(mind), budget=a.budget, session_lane=a.lane or detect_lane(), + awaiting_review=a.awaiting_review, cap=a.cap) + print(json.dumps(d, indent=2)) if a.as_json else emit(d) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/agents/conductors/batch/batch.sh b/agents/conductors/batch/batch.sh new file mode 100755 index 0000000..daa2fab --- /dev/null +++ b/agents/conductors/batch/batch.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# agents/conductors/batch/batch.sh — the Batch Agent (a PyAutoBrain conductor). +# +# Composes what goes into one unattended shift. Thin by construction: every +# judgement it uses — consequence tier, review-minutes, readiness, difficulty — +# belongs to the sizing faculty. This agent decides MEMBERSHIP, which is an act +# rather than an opinion, which is what makes it a conductor. +# +# It proposes; it never dispatches. Approving the proposal in a slot is what +# launches a batch (AUTONOMY.md, "What a batch launch is") — the schedule may +# carry the timing, never the authority. +# +# Usage: +# batch.sh plan # the BatchDecision for this session's lane +# batch.sh plan --budget 45 # review-minutes available in the slot +# batch.sh plan --awaiting-review 6 # backpressure input +# batch.sh plan --json + +set -uo pipefail + +HERE="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)" +source "$HERE/../../_common.sh" + +exec python3 "$HERE/_batch.py" "$@" diff --git a/agents/conductors/intake/_intake.py b/agents/conductors/intake/_intake.py index b24402f..22115a5 100755 --- a/agents/conductors/intake/_intake.py +++ b/agents/conductors/intake/_intake.py @@ -589,7 +589,7 @@ def parse_header(text: str) -> dict: for line in text.splitlines()[:30]: m = re.match(r"(Type|Target|Difficulty|Autonomy|Priority|Status|" r"Issued|Filed|Epic|Phase|Bundle|Blocked-by|" - r"Consequence|Witness|Review-minutes|Unattended):\s*(\S.*)", + r"Consequence|Witness|Review-minutes|Unattended|Lane):\s*(\S.*)", line.strip()) if m: fields.setdefault(m.group(1).lower(), m.group(2).strip()) @@ -1492,6 +1492,9 @@ def census(mind: Path) -> dict: "witness": header.get("witness", ""), "review_minutes": header.get("review-minutes", "-"), "unattended": header.get("unattended", "-"), + # Where it can run. Absent means `any` — the common case, and a + # missing lane must never be read as "nowhere". + "lane": header.get("lane", "any"), "priority": header.get("priority", "-"), "status": header.get("status", "-"), "epic": header.get("epic", ""), diff --git a/agents/faculties/sizing/_sizing.py b/agents/faculties/sizing/_sizing.py index 418d52d..ce98191 100755 --- a/agents/faculties/sizing/_sizing.py +++ b/agents/faculties/sizing/_sizing.py @@ -428,6 +428,12 @@ def empty_discovery_reason(mind: Path, work_type: str) -> str: # correction or a retraction), so self-assessment is not an input here. CONSEQUENCE_TIERS = ("notify", "glance", "judge") UNATTENDED_LEVELS = ("ready", "needs-slicing", "never") +# Where the work can run. Spelled in the environment vocabulary +# `skills/WORKFLOW.md` already defines (`local-dev` / `web-github` / `ci-only` / +# `analysis-only`) rather than a parallel cloud/laptop one — `local-dev` means +# the work needs the local dataset and output trees, an SSH endpoint, or the +# human at the machine. Default `any`. +LANE_VALUES = ("any", "local-dev") # Surfaces that make a change a PI's decision whatever repo it lives in: a # public API, a default value, an error contract, a science-policy call, or an @@ -461,7 +467,7 @@ def empty_discovery_reason(mind: Path, work_type: str) -> str: _HEADER_KEY_RE = re.compile( r"^\s*(difficulty|type|autonomy|status|priority|blocked-by|closes-when" - r"|consequence|witness|review-minutes|unattended)" + r"|consequence|witness|review-minutes|unattended|lane)" r"\s*:\s*(.+?)\s*$", re.I ) @@ -485,7 +491,8 @@ def declared_header(text: str) -> dict: "declared_autonomy": None, "status": None, "priority": None, "blocked_by": [], "closes_when": [], "declared_consequence": None, "witness": None, - "declared_review_minutes": None, "declared_unattended": None} + "declared_review_minutes": None, "declared_unattended": None, + "lane": None} in_fence = False for line in text.splitlines(): if line.lstrip().startswith("```"): @@ -533,6 +540,10 @@ def declared_header(text: str) -> dict: m2 = re.search(r"\d+", value) if m2: out["declared_review_minutes"] = int(m2.group(0)) + elif key == "lane": + v = value.lower() + if v in LANE_VALUES and out["lane"] is None: + out["lane"] = v elif key == "unattended": v = _norm_level(value) if v in UNATTENDED_LEVELS and out["declared_unattended"] is None: diff --git a/bin/pyauto-brain b/bin/pyauto-brain index b822c6c..9f8f909 100755 --- a/bin/pyauto-brain +++ b/bin/pyauto-brain @@ -57,6 +57,7 @@ FACULTIES_DIR="$AGENTS_DIR/faculties" declare -A AGENT_SCRIPT=( [intake]="$CONDUCTORS_DIR/intake/intake.sh" + [batch]="$CONDUCTORS_DIR/batch/batch.sh" [community]="$CONDUCTORS_DIR/community/community.sh" [feature]="$CONDUCTORS_DIR/feature/feature.sh" [bug]="$CONDUCTORS_DIR/bug/bug.sh" @@ -77,6 +78,7 @@ declare -A AGENT_SCRIPT=( ) declare -A AGENT_DESC=( [intake]="Conceive a task: turn raw input into a formal, headed PyAutoMind prompt (files it; never starts dev)" + [batch]="Compose one unattended shift: the BatchDecision, planned against the slot's REVIEW-MINUTE budget rather than a task count — proposes, never dispatches" [community]="The ears — the organism's receptive function: scan/triage user-filed GitHub issues + PRs and review requests across the repos; drafts stay human-gated, dev work routes via start_dev_for_user (never posts)" [feature]="Reason over PyAutoMind feature tasks: select, size, phase, plan for start_dev" [bug]="The immune system: classify a bug/regression/Heart finding, locate the fix, plan the repair" @@ -97,7 +99,7 @@ declare -A AGENT_DESC=( ) # Conductors are the front doors a human drives; faculties are consulted (and # runnable read-only). Both are dispatchable, but the menu groups them by tier. -CONDUCTOR_ORDER=(intake community feature bug refactor workspace eyes profiling hygiene clone build release health) +CONDUCTOR_ORDER=(intake batch community feature bug refactor workspace eyes profiling hygiene clone build release health) FACULTY_ORDER=(vitals review memory samplers sizing) AGENT_ORDER=("${CONDUCTOR_ORDER[@]}" "${FACULTY_ORDER[@]}") diff --git a/skills/batch/SKILL.md b/skills/batch/SKILL.md new file mode 100644 index 0000000..8f76a38 --- /dev/null +++ b/skills/batch/SKILL.md @@ -0,0 +1,22 @@ +--- +name: batch +description: Compose the next unattended batch through the PyAutoBrain Batch Agent — the BatchDecision, planned against the slot's review-minute budget rather than a task count, with backpressure, lane detection and the one-member-per-library-repo rule. Use when picking what to run in a shift, or asking what fits in a review slot. It proposes; approving it in the slot is what launches a batch. +--- + +# Batch + +Read [`../../agents/conductors/batch/AGENTS.md`](../../agents/conductors/batch/AGENTS.md) +completely, then run `bin/pyauto-brain batch plan` in the documented mode. + +Return the **BatchDecision** as written — members, the review-minutes it spends, +and what it rejected with reasons. Do not re-rank it, do not quietly add a +member the planner excluded, and do not present it as a schedule: it is a +proposal, and the human approving it in their slot is what launches the batch +([`../../AUTONOMY.md`](../../AUTONOMY.md), "What a batch launch is"). A +scheduler may carry the timing; it never carries the authority. + +Two parts of the output are the point and must survive into your reply: the +**review-minute total** (the budget is the human's hour, not a task count), and +the **other lane's ready count** when the session cannot plan it — *"4 local-dev +tasks are ready, run this from the laptop"*. An empty batch at the backpressure +cap is a finding, not a deadlock; say which. diff --git a/skills/batch/agents/openai.yaml b/skills/batch/agents/openai.yaml new file mode 100644 index 0000000..01f5665 --- /dev/null +++ b/skills/batch/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Batch Agent" + short_description: "Compose one unattended shift against a review-minute budget" + default_prompt: "Use $batch to plan the next batch — what fits in my review slot." diff --git a/tests/test_batch_plan.py b/tests/test_batch_plan.py new file mode 100644 index 0000000..42dea0c --- /dev/null +++ b/tests/test_batch_plan.py @@ -0,0 +1,130 @@ +"""tests/test_batch_plan.py — the Batch Agent's composition rules. + +The planner's whole job is refusing things for stated reasons, so every test +here is a refusal (or the one case that must never be refused: the floor). +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +BRAIN = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(BRAIN / "agents" / "faculties" / "sizing")) +_spec = importlib.util.spec_from_file_location( + "_batch_under_test", BRAIN / "agents" / "conductors" / "batch" / "_batch.py") +_batch = importlib.util.module_from_spec(_spec) +sys.modules["_batch_under_test"] = _batch +_spec.loader.exec_module(_batch) + + +def rec(path, *, minutes=20, tier="judge", ready="ready", repos=(), epic="", + lane="any", blocked=False, priority="normal"): + return {"path": path, "repos": list(repos), "work_type": "feature", + "difficulty": "medium", "score": 4, "consequence": tier, + "witness": None, "review_minutes": minutes, "unattended": ready, + "why": "", "ready_why": [], "epic": epic, "lane": lane, + "priority": priority, "blocked": blocked} + + +def paths(d): + return [m["path"] for m in d["members"]] + + +def why(d, path): + return next(w for p, w in d["rejected"] if p == path) + + +def test_the_budget_is_review_minutes_not_a_task_count(): + """Three 20-minute tasks do not fit a 45-minute slot, however small they + look by any other measure.""" + d = _batch.plan([rec(f"{i}.md") for i in "abc"], budget=45) + assert len(d["members"]) == 2 + assert d["review_minutes_planned"] == 40 + assert "exceed the budget" in why(d, "c.md") + + +def test_notify_tier_work_costs_the_human_nothing(): + """The fill. It is capped by the token allowance, not by the human's hour, + which is what lets the allowance be spent without growing the queue.""" + d = _batch.plan([rec(f"{i}.md", tier="notify", minutes=0) for i in range(9)], + budget=45) + assert len(d["members"]) == 9 + assert d["review_minutes_planned"] == 0 + + +def test_one_member_per_library_repo_per_shift(): + """They do not collide at dispatch — separate worktrees — they collide at + merge, when the first /prm moves main and invalidates the rest.""" + d = _batch.plan([rec("a.md", repos=["autoarray"]), + rec("b.md", repos=["autoarray"])], budget=100) + assert paths(d) == ["a.md"] + assert "autoarray already claimed" in why(d, "b.md") + + +def test_non_library_repos_do_not_claim_a_shift(): + """Two docs changes to the same organ repo cost nothing to merge together.""" + d = _batch.plan([rec("a.md", repos=["pyautomind"]), + rec("b.md", repos=["pyautomind"])], budget=100) + assert paths(d) == ["a.md", "b.md"] + + +def test_one_slice_per_epic(): + d = _batch.plan([rec("a.md", epic="euclid"), rec("b.md", epic="euclid")], + budget=100) + assert paths(d) == ["a.md"] + assert "epic euclid already" in why(d, "b.md") + + +def test_a_session_never_plans_the_other_lane_but_reports_it(): + """Silently dropping it would leave the human unable to tell 'nothing ready' + from 'nothing I can run from here'.""" + d = _batch.plan([rec("a.md", lane="local-dev")], session_lane="web-github") + assert d["members"] == [] + assert d["other_lane_ready"] == 1 + d2 = _batch.plan([rec("a.md", lane="local-dev")], session_lane="local-dev") + assert paths(d2) == ["a.md"] + + +def test_only_unattended_ready_work_is_selected(): + d = _batch.plan([rec("a.md", ready="needs-slicing"), + rec("b.md", ready="never"), + rec("c.md", blocked=True)], budget=100) + assert d["members"] == [] + assert "needs-slicing" in why(d, "a.md") + assert "Blocked-by" in why(d, "c.md") + + +def test_backpressure_ramps_rather_than_cliffs(): + pool = [rec(f"{i}.md") for i in range(6)] + assert _batch.plan(pool, budget=60, awaiting_review=0)["effective_budget"] == 60 + assert _batch.plan(pool, budget=60, awaiting_review=6)["effective_budget"] == 30 + assert _batch.plan(pool, budget=60, awaiting_review=8)["effective_budget"] == 0 + + +def test_at_the_cap_the_floor_is_fill_only_and_still_runs(): + """A conference week must not stop the thing whose whole purpose is working + while nobody watches. At the cap the review-bearing half is zero — but + zero-cost work still dispatches.""" + d = _batch.plan([rec("a.md"), rec("b.md", tier="notify", minutes=0)], + budget=60, awaiting_review=99) + assert paths(d) == ["b.md"] + + +def test_cheapest_first_because_the_question_is_what_fits(): + d = _batch.plan([rec("slow.md", minutes=20, priority="high"), + rec("fast.md", minutes=2, priority="low")], budget=45) + assert paths(d)[0] == "fast.md" + + +def test_every_rejection_states_a_reason(): + d = _batch.plan([rec("a.md"), rec("b.md"), rec("c.md")], budget=20) + assert d["rejected"] + assert all(isinstance(w, str) and w for _, w in d["rejected"]) + + +def test_lane_detection_reads_the_environment_not_a_flag(): + """A session that could be told where it is could plan local-dev work it + cannot run. Probed from `gh`, the same signal the organism already uses.""" + assert _batch.detect_lane() in ("local-dev", "web-github") From 7f9ca058e70086ffcafaea81514aad488c100e7e Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 30 Aug 2026 23:27:44 +0000 Subject: [PATCH 6/6] review: make the adversary-leg tests hermetic (CI fix) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first version drove the CLI against this checkout with `--repo`, asserting on its stdout. That passed locally and failed in CI with four empty-output errors, because a CI checkout has no `origin/main` to diff: `repo_surface` returns None for every repo, `main` exits 4 with an empty stdout, and every assertion about the contract text failed for a reason unrelated to the contract. Reproduced before fixing — a fresh `git init` with no origin/main gives "review: no reviewable diff against origin/main", exit 4, stdout empty. Fixed at the seam rather than by working around it: `surface_payload()` is split out of `main` as a pure assembly function, so the JSON contract can be tested without git, and the human-output tests call `emit_human` with a synthetic surface. The fixture mirrors `repo_surface`'s return shape exactly and was checked key-for-key against the real producer. The lesson is the one the tests are about: a test whose subject is a block of contract text must not depend on the environment having a diff. 683 pass. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RQeMJZznA3xTQXX4PqWg1v --- agents/faculties/review/_review.py | 27 ++++++++--- tests/test_review_adversary.py | 77 +++++++++++++++++++----------- 2 files changed, 69 insertions(+), 35 deletions(-) diff --git a/agents/faculties/review/_review.py b/agents/faculties/review/_review.py index da0a330..64a13ee 100755 --- a/agents/faculties/review/_review.py +++ b/agents/faculties/review/_review.py @@ -231,6 +231,25 @@ def emit_human(surfaces: list[dict], witness: str = "", adversary: bool = False) print("malformed evidence, not CLEAN (faculty AGENTS.md step 2a).") +def surface_payload(surfaces: list[dict], witness: str = "", + adversary: bool = False) -> dict: + """Assemble the machine-readable surface. Pure — no git, no I/O. + + Split out from `main` so the contract can be tested without a repo that + happens to have a diff against `origin/main`. The first version of the + adversary tests drove the CLI against this checkout and passed locally and + failed in CI, where the checkout has no `origin/main` to diff and the whole + surface is empty: a test asserting on output that the environment decides. + """ + payload: dict = {"review_surface": surfaces} + if witness: + payload["witness"] = witness + if adversary: + payload["mode"] = "independent-adversary" + payload["contract"] = ADVERSARY_CONTRACT + return payload + + def main(argv=None) -> int: ap = argparse.ArgumentParser(prog="review") ap.add_argument("--task", default="", @@ -258,13 +277,7 @@ def main(argv=None) -> int: print("review: no reviewable diff against origin/main", file=sys.stderr) return 4 if a.as_json: - payload = {"review_surface": surfaces} - if a.witness: - payload["witness"] = a.witness - if a.adversary: - payload["mode"] = "independent-adversary" - payload["contract"] = ADVERSARY_CONTRACT - print(json.dumps(payload, indent=2)) + print(json.dumps(surface_payload(surfaces, a.witness, a.adversary), indent=2)) else: emit_human(surfaces, witness=a.witness, adversary=a.adversary) return 0 diff --git a/tests/test_review_adversary.py b/tests/test_review_adversary.py index 5f30364..6ca8973 100644 --- a/tests/test_review_adversary.py +++ b/tests/test_review_adversary.py @@ -4,13 +4,16 @@ the branch's own author. This leg's only new content is *independence* plus *ordering* — the task's witness is falsified before anything else — so those are what is pinned here. + +Hermetic, and deliberately so after the first version was not: driving the CLI +against this checkout passed locally and failed in CI, where there is no +`origin/main` to diff and the surface comes back empty. A test whose subject is +a block of contract text must not depend on the environment having a diff. """ from __future__ import annotations import importlib.util -import json -import subprocess import sys from pathlib import Path @@ -23,47 +26,65 @@ _spec.loader.exec_module(_review) WITNESS = "ids bit-identical, 62 -> 9.7 ms" - - -def _run(*args): - return subprocess.run( - [sys.executable, str(BRAIN / "agents" / "faculties" / "review" / "_review.py"), - "--repo", str(BRAIN), *args], - capture_output=True, text=True) - - -def test_the_witness_is_lifted_and_falsified_first(): - out = _run("--witness", WITNESS).stdout +# Mirrors `repo_surface`'s return shape exactly — a fixture that drifts from the +# producer is a test of the fixture. +SURFACE = [{ + "repo": "PyAutoBrain", "path": "/tmp/PyAutoBrain", "branch": "feature/x", + "base": "abc123456789", "commits_ahead": 2, + "commits": ["do the thing"], "shortstat": "3 files changed", + "files": ["a.py"], "risk_flags": [], + "claims_to_falsify": ["This change is a no-op for CI."], +}] + + +def _emit(capsys, **kw): + _review.emit_human(SURFACE, **kw) + return capsys.readouterr().out + + +def test_the_witness_is_lifted_and_falsified_first(capsys): + """It is the claim the task was scoped around: if it does not hold, the work + did not do what it promised and nothing else on the surface matters.""" + out = _emit(capsys, witness=WITNESS) assert WITNESS in out assert "THE WITNESS" in out + assert out.index("THE WITNESS") < out.index("A load-bearing claim above") assert "Falsify it first" in out -def test_no_witness_section_when_none_is_declared(): +def test_no_witness_section_when_none_is_declared(capsys): """A task with no witness grades `judge` and is reviewed by a human; the surface must not imply a claim that was never made.""" - assert "THE WITNESS" not in _run().stdout + assert "THE WITNESS" not in _emit(capsys) -def test_adversary_mode_states_who_may_not_run_it(): +def test_adversary_mode_states_who_may_not_run_it(capsys): """The independence rule is the leg. A surface that omits it invites the exact failure the leg exists to close.""" - out = _run("--adversary").stdout + out = _emit(capsys, adversary=True) assert "NOT the session or model that wrote the branch" in out - assert "A self-run adversary leg is not a weaker version" in _review.ADVERSARY_CONTRACT + assert "A self-run adversary leg is not a weaker version" in out + + +def test_contract_is_absent_unless_asked_for(capsys): + assert "INDEPENDENT ADVERSARY MODE" not in _emit(capsys) -def test_contract_is_absent_unless_asked_for(): - assert "INDEPENDENT ADVERSARY MODE" not in _run().stdout +def test_payload_carries_witness_and_mode(): + p = _review.surface_payload(SURFACE, WITNESS, True) + assert p["witness"] == WITNESS + assert p["mode"] == "independent-adversary" + assert "contract" in p + assert p["review_surface"] == SURFACE -def test_json_surface_carries_witness_and_mode(): - payload = json.loads(_run("--json", "--witness", WITNESS, "--adversary").stdout) - assert payload["witness"] == WITNESS - assert payload["mode"] == "independent-adversary" - assert "contract" in payload +def test_payload_omits_them_when_not_asked(): + p = _review.surface_payload(SURFACE) + assert "witness" not in p and "mode" not in p and "contract" not in p -def test_json_surface_omits_them_when_not_asked(): - payload = json.loads(_run("--json").stdout) - assert "witness" not in payload and "mode" not in payload +def test_the_surface_is_assembled_without_touching_git(): + """`surface_payload` is pure: the CLI resolves repos and reads diffs, this + only shapes what was found. That split is what makes the tests above + hermetic.""" + assert _review.surface_payload([])["review_surface"] == []