diff --git a/AGENTS.md b/AGENTS.md index 80984d4..a2931e6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -132,6 +132,7 @@ the `/` slash commands. | Verb | Purpose | Entrypoint | |------|---------|------------| | `intake` | Conceive a task: turn raw input into a formal, headed PyAutoMind prompt (files it; never starts dev) | `bin/pyauto-brain intake` | +| `batch` | Compose one unattended shift: the BatchDecision, planned against the slot's REVIEW-MINUTE budget rather than a task count — proposes, never dispatches | `bin/pyauto-brain batch` | | `community` | The ears — the organism's receptive function: scan/triage user-filed GitHub issues + PRs and review requests across the repos; drafts stay human-gated, dev work routes via start_dev_for_user (never posts) | `bin/pyauto-brain community` | | `feature` | Reason over PyAutoMind feature tasks: select, size, phase, plan for start_dev | `bin/pyauto-brain feature` | | `bug` | The immune system: classify a bug/regression/Heart finding, locate the fix, plan the repair | `bin/pyauto-brain bug` | diff --git a/AUTONOMY.md b/AUTONOMY.md index ecb9888..40bb914 100644 --- a/AUTONOMY.md +++ b/AUTONOMY.md @@ -68,6 +68,64 @@ scope addition rather than a correction of the run's own work — and **zero work-type's cap one level immediately, pending a review that cites the row. Both directions are dated doctrine edits to the table above, citing rows. +### Multi-repo autonomy experiment — 2026-08-30 + +`infer_autonomy` (the conception heuristic in +`agents/conductors/intake/_intake.py`) returned `supervised` whenever a prompt +named more than one repo. Repo count is *blast radius*, and blast radius is +already priced — `estimate_difficulty` adds 2 points per repo beyond the first. +This field is supposed to encode whether a **human's judgement** is required, and +a change touching four repos mechanically needs no more judgement than the same +change touching one. The trigger is removed. + +**This is an experiment, not a graduation, and it is deliberately not justified +by the calibration log.** The tempting argument — 238 rows, zero `rejected` — +does not survive reading: the rows run densely 2026-07-08 → 08-01 and then stop +(about seven cover all of August, against 332 completion records), so the base is +human-in-session work rather than the unattended regime this would license; the +human was in the loop for nearly every row, so failures were corrected before +they could become rows; `rejected` is structurally unreachable (a withdrawn +five-PR mechanism was logged `reverted`, a human rejecting a run's +recommendation `amended`, and two rows say verbatim "NOT a clean row for +graduation purposes"); and most of all, the 2026-07-09 review raised the +work-type caps **because** this heuristic stayed conservative about multi-repo +work. Every clean row was produced with the guard switched on. Evidence +collected under a safety device cannot license removing the device. + +**Measured effect** (all 153 `draft/` prompts, re-derived, nothing written): +`safe` 30 → 55, `supervised` 117 → 92, `human-required` unchanged at 6. +`repo_count > 1` is the *sole* supervised trigger for **25** prompts — the +largest single one, ahead of `large`-or-above (20) and architectural risk (17). + +Note what that also refutes: the multi-repo rule was **not** the cause of the +backlog reading 120-of-137 `supervised`. Those levels are declared, written by +earlier intake runs, and the triggers overlap heavily. Removing this one is +worth doing on its merits and frees 25 prompts; it is not the unblocking of the +backlog it was first taken for, and phase 3's ship-sign-off change is where that +actually lives. + +**A first draft of this change also added `human_judgement` as a supervised +trigger in `repo_count`'s place. It was measured and reverted**: `safe` fell to +24, because the ambiguity keywords fire on 63% of prompts and catch well-written +ones indiscriminately. It was the same mistake as the rule it replaced — a loose +proxy standing in for a judgement it does not measure. Recorded here so nobody +re-proposes it. + +**Terms of the experiment:** + +- **20 unattended launches** under the new rule. +- The **independent-model adversarial review leg** is mandatory for them. Until + that leg exists (batch epic phase 3), the experiment **does not start** — the + rule change ships, but no launch counts toward the window. +- Calibration rows written **per work-type**, since the graduation rule is + per-work-type and the figure usually cited is an aggregate. +- A new outcome value, **`rejected-at-review`**, stamped by the **human** in + their review slot. Without it the demotion trigger cannot fire, and an + experiment that cannot fail is not evidence. +- **Revert condition:** any `rejected-at-review` row, or a window that closes + with fewer than 20 launches and no clean read, reverts this edit and restores + the `repo_count > 1` trigger. + ### The scheduled-nightly standing grant — 2026-07-09 The human decided (2026-07-09, recorded in @@ -106,6 +164,38 @@ doctrine edit; removing this section is the doctrine edit that retires it. - Opt-in per invocation, never ambient: no config flag, no environment variable, no "remembered" mode. +### What a batch launch is — 2026-08-30 + +A batch dispatches work into a shift that outlasts the human's session, and +waves of it may start hours after they left. Read against the rule above, a wave +firing at 4am under an approval given at 17:00 is a **stored grant** — the exact +shape the corrective-PR section voids ("a stored, reused or 'standing' +authorization does not count — it must be for this RED, **now**"). So the term +is defined here rather than left for an implementation to settle by accident: + +**A batch dispatch is ONE launch.** Specifically: + +- Its **membership is fixed at approval**, in the slot, by the human. A task not + on the approved list is not launched, however ready it looks and however much + budget is left. +- Its **grant expires at the end of the shift.** A member not dispatched within + it returns to the queue and needs the next slot's approval. +- Its **terms are written down** — members, the acknowledged Heart reason set, + the effective level per member — in `batches/-.md`, where they can + be read afterwards against what actually happened. +- **The human performs the dispatch.** A scheduler may build the review packet + and may wake a session, but the act that starts work on the approved list is + theirs. This is the line that keeps "never ambient" true: the schedule carries + the *timing*, never the *authority*. + +What this deliberately does **not** grant: a standing batch, a recurring +approval, or a config flag that makes the next batch launch itself. Each shift +is approved in its own slot, or it does not run. + +*Revert condition:* if a batch is ever found to have launched work its approval +list did not name, batching returns to per-task `--auto` until the cause is +found. + ## Checkpoint-and-continue (`supervised`) The operational mechanics of the levels-table behaviour: a run writes a @@ -137,11 +227,52 @@ otherwise. Ship sign-off and merge park the *task*, never bypass the gate — checkpoint-and-continue frees the human's session, not the checkpoint. +### Decide-and-flag (batch launches only) — 2026-08-30 + +Park-and-ask is the right behaviour when a human is one message away. In a shift +it costs the whole shift: the run stops, the question waits until the slot, and +the task needs a second batch. So a run under a **batch launch** may, at a +judgement gate, take the more reversible option and record it instead of parking. + +**Narrowly**, because the agent deciding is the poorest available judge of its +own decision's scope. The base rate is measured, not feared: **68 of 332 +completion records in 2026-08 (20%) carry a correction or a retraction**, +including one whose own claim was later marked "FALSE when this record was +written". And *reversible* is the wrong axis on its own — an API-philosophy fork +decided on behalf of an external reporter is trivially reversible in git and +irreversible in public. + +The limits, all of them: + +- **At most ONE flagged decision per PR.** A second judgement gate parks the run + exactly as today. One is a note a reviewer can hold in their head; three is a + design review they did not agree to do. +- The PR body must state the **rejected alternative** and the **one-command + revert**. *If the run cannot write the revert, the decision was not reversible* + — park. +- **Never** where the decision touches a public API, a default value, an error + contract, or a file named in an external reporter's issue. +- **Never** for a `judge`-tier task (`Consequence:` — REFERENCE.md "The + review-cost model"). A tier that costs a PI's quarter-hour is one where the PI + makes the call. +- The PR carries a `decision-taken` label so the review surface sorts it above + clean work, and the calibration row names it. + +This moves review debt from the issue tracker into the diff, which is a real +cost — the diff is where an overloaded reviewer is least able to interrogate a +choice. The cap, the revert line and the tier exclusion are what keep that cost +to one bounded item per PR rather than an unmarked scatter. + +*Revert condition:* two flagged decisions the human would not have taken retires +this section; it returns to park-and-ask. + ## The autonomous-ship gate An unattended ship (checkpoint 2 at `safe`) requires **all four legs**, no -substitutions. Audited 2026-07-08 (issue #38); each leg carries an -applicability rule so "n/a" is a stated fact, never an assumption: +substitutions — **plus leg 5 under a batch launch**, where no human is reachable +and the review leg's independence stops being optional. Audited 2026-07-08 +(issue #38); each leg carries an applicability rule so "n/a" is a stated fact, +never an assumption: 1. **Tests** — worktree pytest (full suite, `-x`) on every **shipped** repo, *plus* every downstream library repo when the diff touches public API @@ -175,9 +306,72 @@ applicability rule so "n/a" is a stated fact, never an assumption: reason, or RED, parks the run. Never ambient, never carried across sessions. +5. **Independent adversary** — **added 2026-08-30, required for a BATCH + launch** (below) and for any run counting toward the multi-repo autonomy + experiment; optional elsewhere. A second reading of the same diff by a + **different model from the one that wrote it**, whose job is not to review + the change but to **falsify its claims — the `Witness:` first**. Run it as + `pyauto-brain review --task --witness "" + --adversary`. + + Why it exists, and why leg 3 does not already cover it: + `complete/2026/08/falsified-by-checkpoint-efficacy-review.md` found that the + review leg on autonomous ships was in practice **the branch's own author**, + that "a healthy pass and a rote one write the identical ledger row", and that + the one confirmed-wrong load-bearing claim of its window lived *outside* the + surface the stage reads — "the stage could not have caught the one escape + that actually happened". The catches that did happen came from an independent + model reading the same diff. + + **A self-run adversary leg is an absent leg, not a weak one**, and recording + it as run is a false ledger row. The verdict is recorded in the calibration + row alongside the other four. + + *Revert condition:* if the leg produces no finding across 30 runs it is + ceremony, and this clause is removed rather than left as theatre. + A failed leg downgrades the run to a human checkpoint: state written to the issue, nothing force-shipped, never modify code to make a leg pass. +### Leg 4 under a batch launch — 2026-08-30 + +Leg 4 as written assumes a human is reachable: YELLOW passes only where the +reason set is the one acknowledged *at launch*, and any new reason parks the +run. Overnight nobody can acknowledge anything, so a fifteen-hour shift would +ride on that set staying frozen. **It does not stay frozen for an afternoon.** +The log shows a drift count growing 2 → 4 → 6 across a single day, re-asked at +each ship; and a *benign new* reason appearing mid-session from a run's own +sibling merge ("release validation stale: source moved since rehearsal") — which +in a batch means **wave 1 can manufacture the reason that parks waves 2 and 3**. +The 2026-07-09 review records that Heart never read GREEN: every shipped run +went out on an acked YELLOW. The base rate says this leg blocks by default. + +That pressure has already produced a violation rather than a park: +`complete/2026/08/cmap-magma-default.md` shipped under a **standing** ack, which +the corrective-PR section below explicitly voids. Doctrine moving deliberately is +the alternative to doctrine bending under load. + +So, **for a batch launch only**: + +- At slot time the human acknowledges a **named reason set for the shift**, + written verbatim into `batches/.md` and into each member's `active.md` + `- heart-ack:` block. +- A run parks on a new **RED**, or on a new **YELLOW reason whose repo + intersects the run's own repos**. A new YELLOW elsewhere in the organism does + not park it — the reason is out of the branch's blast radius, and legs 1-3 and + 5 gate the branch itself. +- A reason **generated by an earlier member of the same batch** is named as such + in the batch record and does not park later members. +- The grant **expires with the shift**. It is not standing, not ambient, and not + carried into the next batch. + +Unchanged: RED still parks at every level; YELLOW is still never acknowledged +*autonomously* — the acknowledgement is a human act performed in the slot, and +this section only defines how far it reaches. + +*Revert condition:* one escape traceable to a YELLOW reason this section let +through restores the exact-set rule for batches. + ## Corrective-PR exception for Heart RED (human-authorized) Heart RED forbids commit, push and PR-open at every autonomy level (the levels @@ -252,14 +446,25 @@ work. That resumes only along this path: at PR-open (or on parking): ```markdown -| date | task | effective level | gates (tests/smoke/review/heart) | outcome | +| date | task | effective level | gates (tests/smoke/review/heart[/adversary]) | outcome | ``` -Outcome ∈ `merged-unchanged` / `amended` / `rejected` / `parked` / `corrective` -(the last records a use of the human-authorized corrective-PR exception above — -not an `--auto` run, but logged so the exception's use is auditable alongside the -autonomy rows). This is the evidence base for raising or lowering caps — autonomy -grows by demonstrated calibration, not by optimism. +Outcome ∈ `merged-unchanged` / `amended` / `rejected` / `rejected-at-review` / +`parked` / `corrective` (`corrective` records a use of the human-authorized +corrective-PR exception above — not an `--auto` run, but logged so the +exception's use is auditable alongside the autonomy rows). This is the evidence +base for raising or lowering caps — autonomy grows by demonstrated calibration, +not by optimism. + +**`rejected-at-review` — added 2026-08-30.** Stamped by the **human**, in their +review slot, when they would not have merged what the run produced. It exists +because `rejected` has never once been used in 238 rows, and reading why shows +the category was being routed around rather than earned: a human rejecting a +run's recommendation was logged `amended`, and an entire shipped mechanism +withdrawn on reflection — five PRs closed, branches deleted — was logged +`reverted`. A demotion trigger nothing can pull is not a safety device, and a +zero-`rejected` record produced that way is not evidence of anything. Use the +human-stamped value for any judgement about whether autonomy should rise. ### Calibration review — 2026-07-09 @@ -320,7 +525,8 @@ tier), never by weakening leg 4. - `start_dev` — `--auto` usage, effective-level computation, plan-to-issue for `safe`, launch-acknowledgement recording (its "--auto mode" section). -- `ship_library` / `ship_workspace` — the four-leg gate at step 4, stop at +- `ship_library` / `ship_workspace` — the four-leg gate at step 4 (five under a + batch launch), stop at PR-open, validation checklist, calibration append; the RED-handling step points here for the human-authorized corrective-PR exception. diff --git a/agents/conductors/batch/AGENTS.md b/agents/conductors/batch/AGENTS.md new file mode 100644 index 0000000..89f18a1 --- /dev/null +++ b/agents/conductors/batch/AGENTS.md @@ -0,0 +1,108 @@ +# Batch Agent + +> **Tier: conductor** — a front-door agent a human *drives*. It *composes one +> unattended shift*: given the queue, the backlog and the review-queue state, it +> emits a **BatchDecision** — the members, what they will cost the human to +> review, and what it rejected and why. It **proposes; it never dispatches.** +> Approving the proposal in a slot is what launches a batch. + +## What it is for + +The organism has no throughput problem. August 2026 shipped **332 completion +records, about eleven a day**. The scarce resource is the human's judgement, and +a batch layer that plans by task count spends it faster rather than slower. + +So the composition rule is a budget in **review-minutes**, not a count: + +> Σ `Review-minutes:` over `glance` and `judge` members ≤ the slot's budget +> (default 45 — a slot also has to queue the next batch). + +Read against the ledger, an honest hour holds about three library-touching +tasks. Planning by count over-promises capacity roughly threefold, and the +overflow lands on the human at 6am. + +Everything above that budget is the **fill**: work costing zero review-minutes +(`notify`-tier work, slicing, witness authoring, backlog re-grading, deeper +verification of work that already passed). The fill is sized by the remaining +token allowance rather than by the human's hour, which is what lets the organism +spend its whole weekly budget without growing the review queue. **Research and +experiments are never fill** — they produce verdicts, and a verdict is the most +expensive review there is. + +## What it consults, and what it owns + +It owns **membership**, which is an act. Every *judgement* it uses — consequence +tier, review-minutes, readiness, difficulty — comes from the **sizing faculty**, +because `ORGANISM.md` makes faculties the opinion sinks and forbids a conductor +consulting a conductor. If a rule here starts needing to *judge* rather than +*select*, it belongs in the faculty. + +## The constraints, and why each exists + +| Constraint | Why | +|---|---| +| **Review-minute budget** | the human's hour is the binding resource; see above | +| **One member per *library* repo per shift** | concurrent members do not collide at dispatch (separate worktrees) — they collide at **merge**, because the first `/prm` moves `main` and invalidates the others' test and smoke evidence. Workspace, docs and organ repos are exempt: two docs changes in one shift cost nothing. Work concentrates in four repos (PyAutoFit 118/332 August records, PyAutoArray 98, PyAutoGalaxy 82, PyAutoLens 78), so effective parallelism is two or three, not six. | +| **One slice per epic per batch** | epic phases are ordered, so two members could not run in parallel anyway — and this is what interleaves small pieces of long programmes with standalone work | +| **`Unattended: ready` only** | `needs-slicing` goes to the decomposition pass; `never` never enters a batch | +| **Lane match** | a session detects its own lane and plans only that one | +| **Backpressure** | see below | + +Everything rejected says so, with its reason. A planner that silently drops work +teaches the human to distrust the number it reports. + +## Backpressure ramps; it never deadlocks + +Counted in **tasks awaiting review**, never in PRs — 94 of 332 August records +named two or more PRs, so a PR-count cap trips on a single healthy batch. + +- clear → the full budget; +- above half the cap → the review-bearing half is **halved** (the fill is not: + it does not touch the queue); +- at the cap → the batch is the **floor**: fill only, dispatched whether or not + the human turned up. + +A missed slot is the common case for an academic, not an exception, and a +conference week must not stop the thing whose whole purpose is working while +nobody watches. An **empty floor is a finding, not a deadlock** — it means +nothing in the backlog costs zero review-minutes, which is exactly what the +`notify` tier and the `Witness:` field exist to change, and the decision says so +in those words. + +## Lane detection + +`local-dev` or `web-github`, probed from the environment rather than declared, +and from the signal the organism already uses: a remote session has no `gh` +(measured; `skills/GITHUB_ACCESS.md`). No flag and no environment variable +decides it — a session that could lie about where it is could plan `local-dev` +work it cannot run. A session reports the other lane's ready count rather than +hiding it: *"4 local-dev task(s) are ready — run `batch plan` from the laptop."* + +## It proposes; the human launches + +`AUTONOMY.md` ("What a batch launch is") defines a batch dispatch as **one +launch**: membership fixed at approval, the grant expiring with the shift, the +terms written into `PyAutoMind/batches/-.md`, and the human +performing the dispatch. A scheduler may build the review packet and wake a +session; the act that starts work on the approved list is the human's. That is +the line that keeps "never ambient" true — the schedule carries the *timing*, +never the *authority*. + +## Running + +``` +pyauto-brain batch plan # the BatchDecision for this lane +pyauto-brain batch plan --budget 45 # review-minutes available +pyauto-brain batch plan --awaiting-review 6 # backpressure input +pyauto-brain batch plan --json +``` + +Stdlib-only, offline, writes nothing. + +## Not built yet + +`slice` (the decomposition pass doctrine has named since inception) and +`collect` (the review packet) are the conductor's other two verbs — see the +epic ledger, `PyAutoMind/draft/feature/pyautomind/two_slot_batching_epic.md`. +`plan` is useful on its own: run it in a slot and dispatch by tapping the +dashboard's existing chips. diff --git a/agents/conductors/batch/_batch.py b/agents/conductors/batch/_batch.py new file mode 100644 index 0000000..adca39a --- /dev/null +++ b/agents/conductors/batch/_batch.py @@ -0,0 +1,255 @@ +#!/usr/bin/env python3 +"""agents/conductors/batch/_batch.py — the batch conductor. + +Composes what goes into one unattended shift, and reports what came back. Thin +by construction: every judgement it uses — consequence tier, review-minutes, +readiness, difficulty — belongs to the sizing faculty, because `ORGANISM.md` +makes faculties the opinion sinks and forbids a conductor consulting a +conductor. This module decides *membership*, which is an act, not an opinion. + +The composition rule, and the reason the epic exists: + + sum(Review-minutes) over `glance` and `judge` members <= the slot budget + +not a task count. Read against the ledger, an honest review hour holds about +three library-touching tasks; a design that plans by count over-promises +capacity roughly threefold and lands the overflow on the human at 6am. + +Everything above that budget is the FILL: work costing zero review-minutes +(`notify`-tier work, slicing, witness authoring, re-grading, deeper verification +of work that already passed). It is sized by the remaining token allowance +rather than by the human's hour, which is what lets the organism spend its whole +weekly budget without growing the review queue. Research is never fill — it +produces verdicts, and a verdict is the most expensive review there is. + +Stdlib-only and offline, like every Brain entrypoint. +""" +from __future__ import annotations + +import argparse +import json +import shutil +import sys +from pathlib import Path + +BRAIN = Path(__file__).resolve().parents[3] +sys.path.insert(0, str(BRAIN / "agents" / "faculties" / "sizing")) +from _sizing import ( # noqa: E402 + BODY_MAP_PATH, LIBRARY_REPOS, effective_consequence, effective_difficulty, + effective_review_minutes, effective_unattended, parse_prompt, priority_rank, +) + +# One slot's worth of the human's attention. Not a task count — see the module +# docstring. 45 rather than 60 because a slot also has to queue the next batch. +DEFAULT_REVIEW_BUDGET = 45 +# Above half the cap the review-bearing half is halved; at the cap the batch is +# fill only. Counted in TASKS AWAITING REVIEW, never in PRs: 94 of 332 records in +# 2026-08 named two or more PRs, so a PR-count cap trips on one healthy batch. +DEFAULT_BACKPRESSURE_CAP = 8 +CHEAP_TIERS = ("notify",) + + +def detect_lane() -> str: + """`local-dev` or `web-github` — where this session is running. + + Probed from the environment rather than declared, and deliberately from the + same signal the rest of the organism already uses: a remote session has no + `gh` (measured, documented in `skills/GITHUB_ACCESS.md` — installing one is + a trap that authenticates and then 403s every repo-scoped call). No env var + and no flag decides this; a session that could lie about where it is could + plan `local-dev` work it cannot run. + """ + return "local-dev" if shutil.which("gh") else "web-github" + + +def _lane_ok(record_lane: str, session_lane: str) -> bool: + """A `local-dev` task runs only in a `local-dev` session; `any` runs anywhere.""" + return record_lane != "local-dev" or session_lane == "local-dev" + + +def grade(path: Path, mind: Path) -> dict: + """Everything the planner needs about one prompt, from the faculty.""" + p = parse_prompt(path, mind) + level, score, factors, derived = effective_difficulty(p) + tier, why, _ = effective_consequence(p, factors) + ready, ready_why, _ = effective_unattended(p, level, factors, derived) + minutes, _ = effective_review_minutes(p, tier, level) + return { + "path": p["path"], "repos": p["repos"], "work_type": p["work_type"], + "difficulty": level, "score": score, "consequence": tier, + "witness": p.get("witness"), "review_minutes": minutes, + "unattended": ready, "why": (why or [""])[0], + "ready_why": ready_why, "epic": None, "lane": p.get("lane") or "any", + "priority": p.get("priority") or "normal", + "blocked": bool(p.get("blocked_by")), + } + + +def _epic_of(text: str) -> str: + for line in text.splitlines()[:30]: + if line.lower().startswith("epic:"): + return line.split(":", 1)[1].strip() + return "" + + +def survey(mind: Path) -> list[dict]: + """Grade every backlog prompt. Epic membership rides along for the + one-slice-per-epic rule.""" + out = [] + for f in sorted(mind.glob("draft/**/*.md")): + if f.name == "README.md": + continue + g = grade(f, mind) + g["epic"] = _epic_of(f.read_text(encoding="utf-8", errors="replace")) + out.append(g) + return out + + +def plan(records: list[dict], *, budget: int = DEFAULT_REVIEW_BUDGET, + session_lane: str = "web-github", awaiting_review: int = 0, + cap: int = DEFAULT_BACKPRESSURE_CAP) -> dict: + """Compose the next batch — a BatchDecision. + + Every constraint states itself in `rejected`, because a planner that + silently drops work teaches the human to distrust the number it reports. + """ + rejected: list[tuple[str, str]] = [] + + # Backpressure RAMPS; it never deadlocks. A missed slot is the common case + # for an academic, and a conference week must not stop the thing whose whole + # purpose is working while nobody watches. + if awaiting_review >= cap: + effective_budget, pressure = 0, "at cap — fill only" + elif awaiting_review > cap / 2: + effective_budget, pressure = budget // 2, "above half cap — halved" + else: + effective_budget, pressure = budget, "clear" + + pool = [] + for r in records: + if r["unattended"] != "ready": + rejected.append((r["path"], f"unattended: {r['unattended']}")) + elif r["blocked"]: + rejected.append((r["path"], "declares Blocked-by:")) + elif not _lane_ok(r["lane"], session_lane): + rejected.append((r["path"], f"lane {r['lane']}, session {session_lane}")) + else: + pool.append(r) + + # Cheapest first: this list is read when the human has a slot to fill and + # wants to know what fits in it. Importance is answered by the queue's order. + pool.sort(key=lambda r: (r["review_minutes"], priority_rank(r), r["path"])) + + members, spent, seen_epics, libs = [], 0, set(), set() + for r in pool: + if r["epic"] and r["epic"] in seen_epics: + rejected.append((r["path"], f"epic {r['epic']} already in this batch")) + continue + # Concurrent members do not collide at dispatch (separate worktrees) — + # they collide at MERGE, because the first /prm moves main and + # invalidates the others' test and smoke evidence. Only LIBRARY repos + # claim a shift: they are what the library-first gate serialises and + # what downstream suites are re-run against. Workspace, docs and organ + # repos are exempt — two docs changes in one shift cost nothing. + clash = next((x for x in r["repos"] if x in LIBRARY_REPOS and x in libs), None) + if clash: + rejected.append((r["path"], f"{clash} already claimed this shift")) + continue + cost = 0 if r["consequence"] in CHEAP_TIERS else r["review_minutes"] + if spent + cost > effective_budget: + rejected.append((r["path"], f"{cost} min would exceed the budget")) + continue + members.append(r) + spent += cost + if r["epic"]: + seen_epics.add(r["epic"]) + libs.update(x for x in r["repos"] if x in LIBRARY_REPOS) + + return { + "session_lane": session_lane, + "review_budget": budget, + "effective_budget": effective_budget, + "backpressure": {"awaiting_review": awaiting_review, "cap": cap, + "state": pressure}, + "review_minutes_planned": spent, + "members": members, + "rejected": rejected, + "other_lane_ready": sum( + 1 for r in records + if r["unattended"] == "ready" and not _lane_ok(r["lane"], session_lane)), + } + + +def emit(d: dict) -> None: + lane = d["session_lane"] + print("== BatchDecision ==") + print(f"Session lane: {lane}") + print(f"Review budget: {d['effective_budget']} of {d['review_budget']} min " + f"({d['backpressure']['state']}; " + f"{d['backpressure']['awaiting_review']} awaiting review, " + f"cap {d['backpressure']['cap']})") + print(f"Planned: {d['review_minutes_planned']} review-minutes over " + f"{len(d['members'])} member(s)") + print() + if d["members"]: + for r in d["members"]: + cost = 0 if r["consequence"] in CHEAP_TIERS else r["review_minutes"] + print(f" {cost:>3} min {r['consequence']:<7} {r['path']}") + if not r["witness"]: + print(" (no witness — reviewed as `judge`)") + elif d["effective_budget"] == 0: + # At the cap the batch is the FLOOR: fill only, dispatched whether or + # not the human turned up. An empty floor is a finding, not a deadlock — + # it means nothing in the backlog costs zero review-minutes, which is + # what the `notify` tier and the witness field exist to change. + print(" (no members — at the backpressure cap, so this batch is the") + print(" FLOOR: fill only. Nothing in the backlog currently qualifies") + print(" as fill, which is itself the finding — clear the review queue,") + print(" or write witnesses so work can grade `notify`.)") + else: + print(" (no members — see the rejections below)") + print() + if d["other_lane_ready"]: + # Reported, never silently dropped: a task the session cannot run is + # still a task the human can, from the other machine. + print(f"{d['other_lane_ready']} local-dev task(s) are ready — run " + "`batch plan` from the laptop to see them.") + print() + print(f"Not selected: {len(d['rejected'])}") + counts: dict[str, int] = {} + for _, why in d["rejected"]: + counts[why.split(":")[0].split(" already")[0]] = counts.get( + why.split(":")[0].split(" already")[0], 0) + 1 + for why, n in sorted(counts.items(), key=lambda kv: -kv[1])[:8]: + print(f" {n:>4} {why}") + print() + print("This is a PROPOSAL. Approving it in the slot is what launches the") + print("batch — membership is fixed at approval and the grant expires with") + print("the shift (AUTONOMY.md, \"What a batch launch is\").") + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(prog="batch") + ap.add_argument("verb", choices=["plan"], nargs="?", default="plan") + ap.add_argument("--mind", type=Path, default=BODY_MAP_PATH.parent) + ap.add_argument("--budget", type=int, default=DEFAULT_REVIEW_BUDGET, + help="review-minutes available in the slot") + ap.add_argument("--awaiting-review", type=int, default=0, + help="tasks already awaiting review (backpressure input)") + ap.add_argument("--cap", type=int, default=DEFAULT_BACKPRESSURE_CAP) + ap.add_argument("--lane", default="", help="override the detected session lane") + ap.add_argument("--json", action="store_true", dest="as_json") + a = ap.parse_args(argv) + + mind = a.mind.resolve() + if not (mind / "draft").is_dir(): + print(f"batch: no PyAutoMind backlog at {mind}", file=sys.stderr) + return 4 + d = plan(survey(mind), budget=a.budget, session_lane=a.lane or detect_lane(), + awaiting_review=a.awaiting_review, cap=a.cap) + print(json.dumps(d, indent=2)) if a.as_json else emit(d) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/agents/conductors/batch/batch.sh b/agents/conductors/batch/batch.sh new file mode 100755 index 0000000..daa2fab --- /dev/null +++ b/agents/conductors/batch/batch.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# agents/conductors/batch/batch.sh — the Batch Agent (a PyAutoBrain conductor). +# +# Composes what goes into one unattended shift. Thin by construction: every +# judgement it uses — consequence tier, review-minutes, readiness, difficulty — +# belongs to the sizing faculty. This agent decides MEMBERSHIP, which is an act +# rather than an opinion, which is what makes it a conductor. +# +# It proposes; it never dispatches. Approving the proposal in a slot is what +# launches a batch (AUTONOMY.md, "What a batch launch is") — the schedule may +# carry the timing, never the authority. +# +# Usage: +# batch.sh plan # the BatchDecision for this session's lane +# batch.sh plan --budget 45 # review-minutes available in the slot +# batch.sh plan --awaiting-review 6 # backpressure input +# batch.sh plan --json + +set -uo pipefail + +HERE="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)" +source "$HERE/../../_common.sh" + +exec python3 "$HERE/_batch.py" "$@" diff --git a/agents/conductors/intake/_intake.py b/agents/conductors/intake/_intake.py index 0e60b89..22115a5 100755 --- a/agents/conductors/intake/_intake.py +++ b/agents/conductors/intake/_intake.py @@ -42,6 +42,7 @@ LIBRARY_REPOS, WORKSPACE_REPOS, ORGANISM_REPOS, KNOWN_REPOS, RISK_KEYWORDS, AMBIGUITY_KEYWORDS, normalise_repo, declared_header, declared_inline, effective_difficulty, strip_declarations, _hits, + effective_consequence, effective_unattended, effective_review_minutes, policy as _sizing_policy, BODY_MAP_PATH, _body_map_specs as _sizing_specs, ) @@ -274,12 +275,40 @@ def infer_priority(text: str) -> str: def infer_autonomy(level: str, factors: dict) -> str: - """safe | supervised | human-required.""" - repo_count = factors["repos_affected"] - if factors["human_judgement"] and repo_count == 0: + """safe | supervised | human-required. + + `repo_count > 1` used to force `supervised` here, and it is why 120 of the + 137 backlog prompts carried that level on 2026-08-30: nearly every real task + in this organism names a library plus its workspace, or a library plus a + downstream repo. Repo count is *blast radius*, and blast radius is already + priced — `estimate_difficulty` adds 2 points per repo beyond the first. This + field is supposed to encode something else: whether a HUMAN'S JUDGEMENT is + required. A change touching four repos mechanically needs no more judgement + than the same change touching one. + + So `repo_count` is removed and NOTHING replaces it. The first draft of this + change added `human_judgement` as a supervised trigger in its place, on the + reasoning that ambiguity is what actually predicts a park. Measured over the + backlog, that made things WORSE — `safe` fell from 30 to 24, because the + ambiguity keywords ("unclear", "investigate", "explore", "research", + "decide") fire on 63% of prompts and catch well-written ones indiscriminately. + It was the same mistake as the rule it replaced: a loose proxy standing in + for a judgement it does not measure. Dropping `repo_count` alone takes `safe` + from 30 to 55. + + CHANGED 2026-08-30 as a dated EXPERIMENT, not a graduation — see + AUTONOMY.md "Multi-repo autonomy experiment". Deliberately not justified by + the calibration log's 238 rows and zero `rejected`: those rows are July + human-in-session work (about seven cover all of August, against 332 + completions), `rejected` is structurally unreachable in that log, and every + clean row was produced *with this guard switched on* — by a review that + raised the work-type caps precisely BECAUSE this heuristic stayed + conservative. Evidence collected under a safety device cannot license + removing the device. + """ + if factors["human_judgement"] and factors["repos_affected"] == 0: return "human-required" # unscoped / needs a design decision - if (factors["architectural_risk"] or level in ("large", "too-large") - or repo_count > 1): + if factors["architectural_risk"] or level in ("large", "too-large"): return "supervised" return "safe" @@ -317,6 +346,23 @@ def analyse(text: str, source: str, themes=None): "declared_difficulty": declared.get("difficulty")} level, score, factors, estimated = effective_difficulty(p) + # The review-cost model (sizing faculty). `Witness:` is the one field that + # can never be derived: it is a promise about evidence the work will + # produce, and a plausible-sounding invented one would defeat the whole + # mechanism — the value of the field is that its ABSENCE is informative. + # So it is read if declared and left absent otherwise, which correctly + # grades the prompt `judge`. + p["witness"] = header.get("witness") + p["declared_consequence"] = header.get("declared_consequence") + p["declared_unattended"] = header.get("declared_unattended") + p["declared_review_minutes"] = header.get("declared_review_minutes") + p["declared_autonomy"] = header["declared_autonomy"] or inline.get("autonomy") + consequence, consequence_why, consequence_derived = effective_consequence(p, factors) + unattended, unattended_why, unattended_derived = effective_unattended( + p, level, factors, estimated) + review_minutes, review_minutes_derived = effective_review_minutes( + p, consequence, level) + autonomy = declared.get("autonomy") or infer_autonomy(level, factors) priority = declared.get("priority") or infer_priority(text) workflow = infer_workflow(target, repos) @@ -347,7 +393,10 @@ def analyse(text: str, source: str, themes=None): # low-confidence triage filing that means `Type: triage`, not the provisional # guess — the guess still rides in the IntakeDecision's `work_type` field. header = _render_header(title, folder, target_display, repos, level, - autonomy, priority, themes) + autonomy, priority, themes, + consequence=consequence, witness=p["witness"], + review_minutes=review_minutes, + unattended=unattended) return { "source": source, "title": title, @@ -370,19 +419,30 @@ def analyse(text: str, source: str, themes=None): "difficulty_source": "declared" if "difficulty" in declared else "estimated", "autonomy": autonomy, "autonomy_source": "declared" if "autonomy" in declared else "inferred", + "consequence": consequence, + "consequence_derived": consequence_derived, + "consequence_why": consequence_why, + "witness": p["witness"], + "review_minutes": review_minutes, + "review_minutes_derived": review_minutes_derived, + "unattended": unattended, + "unattended_derived": unattended_derived, + "unattended_why": unattended_why, "priority": priority, "priority_source": "declared" if "priority" in declared else "inferred", "declared_fields": declared, "workflow": workflow, "proposed_path": proposed, "header": header, - "risks": _risks(level, factors, confidence, target, declared, estimated), + "risks": _risks(level, factors, confidence, target, declared, estimated, + witness=p["witness"], consequence=consequence), "next_action": _next_action(proposed, confidence, folder), } def _render_header(title, work_type, target_display, repos, level, autonomy, - priority, themes=None): + priority, themes=None, consequence=None, witness=None, + review_minutes=None, unattended=None): lines = [f"# {title}", "", f"Type: {work_type}", f"Target: {target_display}"] if repos: lines.append("Repos:") @@ -397,10 +457,22 @@ def _render_header(title, work_type, target_display, repos, level, autonomy, lines += [f"- {t}" for t in themes] lines += [f"Difficulty: {level}", f"Autonomy: {autonomy}", f"Priority: {priority}", "Status: formalised"] + # The review-cost model rides below the difficulty block: what the work + # costs the organism, then what it costs the human. `Witness:` is written + # ONLY when the author supplied one — see the note in `analyse`. + if consequence: + lines.append(f"Consequence: {consequence}") + if witness: + lines.append(f"Witness: {witness}") + if review_minutes is not None: + lines.append(f"Review-minutes: {review_minutes}") + if unattended: + lines.append(f"Unattended: {unattended}") return "\n".join(lines) -def _risks(level, factors, confidence, target, declared=None, estimated=None): +def _risks(level, factors, confidence, target, declared=None, estimated=None, + witness=None, consequence=None): out = [] declared = declared or {} if "difficulty" in declared and declared["difficulty"] != estimated: @@ -410,6 +482,10 @@ def _risks(level, factors, confidence, target, declared=None, estimated=None): if field in declared: out.append(f"{field.capitalize()} {declared[field]} declared in the raw " f"text — taken as written, not inferred.") + if witness is None and consequence == "judge": + out.append("No Witness: declared — this grades `judge` (a PI's " + "quarter-hour) whatever its size. Naming one machine-checkable " + "claim now is what makes the work reviewable in minutes later.") if confidence == "low": out.append("Low classification confidence — filed to triage/ for a human " "to re-home once the work type is clear.") @@ -465,7 +541,20 @@ def write_prompt(mind: Path, decision: dict, body_text: str, source_note: str): # The header convention this agent writes (see _render_header); census parses the # same fields back out of every filed prompt. Legacy prompts pre-date the header, # so every field is optional — absence is reported, never fatal. -HEADER_FIELDS = ("type", "target", "difficulty", "autonomy", "priority", "status") +# `consequence` and `review-minutes` join the hygiene set: both are DERIVED, so +# `intake formalise` can fill them in place like any other missing field. +# +# `witness` and `unattended` deliberately do NOT. `formalise` writes every field +# it finds missing, and a `Witness:` cannot be derived — it is a promise about +# evidence the work will produce. An auto-written one would be plausible prose +# with nothing behind it, which is strictly worse than none: the entire value of +# the field is that its ABSENCE is informative, and a backlog of invented +# witnesses would grade `notify` while offering a reviewer nothing to check. +# (`unattended` stays out for the ordinary reason — it is derived on read and +# never needs storing.) The dashboard reports missing witnesses as its own +# hygiene row instead, where the fix is a human writing one. +HEADER_FIELDS = ("type", "target", "difficulty", "autonomy", "priority", "status", + "consequence", "review-minutes") # `Fix:`-anchored PR reference in a draft prompt's body — the idiom a session # writes when it fixes the bug but forgets to advance the prompt's lifecycle @@ -499,7 +588,8 @@ def parse_header(text: str) -> dict: fields = {} for line in text.splitlines()[:30]: m = re.match(r"(Type|Target|Difficulty|Autonomy|Priority|Status|" - r"Issued|Filed|Epic|Phase|Bundle|Blocked-by):\s*(\S.*)", + r"Issued|Filed|Epic|Phase|Bundle|Blocked-by|" + r"Consequence|Witness|Review-minutes|Unattended|Lane):\s*(\S.*)", line.strip()) if m: fields.setdefault(m.group(1).lower(), m.group(2).strip()) @@ -1358,6 +1448,7 @@ def census(mind: Path) -> dict: *work* view — health belongs to the Heart, never here. """ records, hygiene, drift, theme_flags = [], [], [], [] + witness_flags = [] # `human_review/` prompts are collected apart from the backlog: they are not # work to pick up, they are shipped work waiting on a person. Keeping them # out of `records` keeps them out of the pick lists, the work-type sections, @@ -1394,6 +1485,16 @@ def census(mind: Path) -> dict: "title": _title(text), "difficulty": header.get("difficulty", "-"), "autonomy": header.get("autonomy", "-"), + # The review-cost model (sizing faculty). `witness` is the only + # one that can be genuinely absent on a graded prompt — nothing + # derives it — and its absence is what makes the prompt `judge`. + "consequence": header.get("consequence", "-"), + "witness": header.get("witness", ""), + "review_minutes": header.get("review-minutes", "-"), + "unattended": header.get("unattended", "-"), + # Where it can run. Absent means `any` — the common case, and a + # missing lane must never be read as "nowhere". + "lane": header.get("lane", "any"), "priority": header.get("priority", "-"), "status": header.get("status", "-"), "epic": header.get("epic", ""), @@ -1413,6 +1514,8 @@ def census(mind: Path) -> dict: }) if len(missing) == len(HEADER_FIELDS): hygiene.append(f"{rel} — no metadata header (pre-dates intake)") + if not header.get("witness"): + witness_flags.append(str(rel)) if stray: theme_flags.append(f"{rel} — unknown theme keyword(s): " + ", ".join(stray)) @@ -1501,6 +1604,7 @@ def _count(key): "bundles": parse_bundles(mind / "bundles.md"), "theme_vocab": vocab, "theme_flags": theme_flags, + "witness_flags": witness_flags, "parked": parked, "planned": planned, "hygiene": hygiene, @@ -1843,6 +1947,29 @@ def _recent_link(e: dict) -> str: "— continue the epic rather than starting one standalone.") +def _fits_a_slot(records: list) -> list: + """The pick list for a human working in a bounded review slot. + + It replaced "Quick wins" (`difficulty == small and autonomy == safe`), which + was near-empty: ten prompts in the whole backlog carried `safe`, so the + surface that exists to hand out unattended work had almost nothing to hand + out. The two questions that actually matter are different ones — can it + finish without me, and what will it cost me to review — and the sizing + faculty now answers both. + + Ordered by review-minutes ASCENDING rather than by priority: this list is + read when the human has a slot to fill and wants to know what fits in it. + `Highest priority` above is where importance is answered. + """ + def cost(r): + try: + return int(r.get("review_minutes", "-")) + except (TypeError, ValueError): + return 99 # ungraded sorts last, never hidden + ready = [r for r in records if r.get("unattended") == "ready"] + return sorted(ready, key=lambda r: (cost(r), _pick_key(r))) + + def render_dashboard(c: dict) -> str: """Render the census as the Mind's task page (`dashboard.md`). @@ -1904,11 +2031,11 @@ def render_dashboard(c: dict) -> str: members = _epic_members(c) standalone = [r for r in records if not r.get("epic")] high = [r for r in standalone if r["priority"] == "high"] - quick = [r for r in standalone - if r["difficulty"] == "small" and r["autonomy"] == "safe"] + quick = _fits_a_slot(standalone) for title, note, rows in ( ("Highest priority", "filed as `high`", high), - ("Quick wins", "small enough, and safe enough to run unattended", quick), + ("Fits a slot", "ready to run unattended, cheapest to review first", + quick), ): shown = rows[:PICK_LIST_MAX] more = f" — showing {len(shown)} of {len(rows)}" if len(rows) > len(shown) else "" @@ -2043,6 +2170,20 @@ def render_dashboard(c: dict) -> str: "
", "Headerless prompts", ""] + [f"- `{h.split(' — ')[0]}`" for h in c["hygiene"]] + ["", "
"]) + if c.get("witness_flags"): + n = len(c["witness_flags"]) + blocks.append( + [f"{n} prompt(s) with no `Witness:` — the machine-checkable claim " + "that would make the work reviewable in minutes. Absent, a prompt " + "grades `judge` (a PI's quarter-hour) whatever its size, which is " + "the intended default and not a bug. Nothing derives or backfills " + "a witness — an invented one is plausible prose with nothing " + "behind it — so this is a human writing one, a prompt at a time.", + "", + "
", "Prompts with no witness", ""] + + [f"- `{w}`" for w in c["witness_flags"][:40]] + + ([f"- _… and {n - 40} more_"] if n > 40 else []) + + ["", "
"]) if c.get("theme_flags"): blocks.append( [f"{len(c['theme_flags'])} prompt(s) with unknown theme " @@ -2186,11 +2327,11 @@ def record_row(r): members = _epic_members(c) standalone = [r for r in records if not r.get("epic")] high = [r for r in standalone if r["priority"] == "high"] - quick = [r for r in standalone - if r["difficulty"] == "small" and r["autonomy"] == "safe"] + quick = _fits_a_slot(standalone) for title, note, rows in ( ("Highest priority", "filed as high", high), - ("Quick wins", "small enough, and safe enough to run unattended", quick), + ("Fits a slot", "ready to run unattended, cheapest to review first", + quick), ): shown = rows[:PICK_LIST_MAX] more = (f" — showing {len(shown)} of {len(rows)}" diff --git a/agents/faculties/review/AGENTS.md b/agents/faculties/review/AGENTS.md index 47aaf43..dee9475 100644 --- a/agents/faculties/review/AGENTS.md +++ b/agents/faculties/review/AGENTS.md @@ -70,6 +70,29 @@ Heart and the agent reasons over the verdict. (`docs/agent_failure_modes.md` item 6 Outcome) found that across 22 ship gates a healthy pass and a skipped one wrote the identical ledger row. An empty surface requires nothing. +2b. **Independent-adversary mode** (`--adversary`, autonomous-ship gate leg 5, + added 2026-08-30). Required for a **batch launch**, where no human is + reachable; optional elsewhere. Run as + `pyauto-brain review --task --witness "" + --adversary`. + + Two things change, and only two. First, the task's **`Witness:`** — the + machine-checkable claim the work was scoped around + (`PyAutoMind/REFERENCE.md`, "The review-cost model") — is lifted to the head + of the claims surface and falsified **first**: if the witness does not hold, + the work did not do what it promised and nothing else on the surface matters. + Second, the reader must be a **different model from the one that wrote the + branch**. + + That second rule is the whole leg. Step 2a is already adversarial in + *procedure*, and it was still run, in practice, by the branch's own author — + `complete/2026/08/falsified-by-checkpoint-efficacy-review.md` found that "a + healthy pass and a rote one write the identical ledger row" and that the one + confirmed-wrong load-bearing claim of its window lived outside the surface + the stage reads. The catches that did happen came from an independent model + reading the same diff. **A self-run adversary leg is an absent leg, not a + weak one**, and recording it as run is a false ledger row. + 3. Map the outcome to the verdict: any unresolved must-fix → **FINDINGS** (ranked list, file:line, failure scenario) — including any `unverified-claim` from step 2a; nothing → **CLEAN**; could not diff --git a/agents/faculties/review/_review.py b/agents/faculties/review/_review.py index 44aeaee..64a13ee 100755 --- a/agents/faculties/review/_review.py +++ b/agents/faculties/review/_review.py @@ -170,7 +170,28 @@ def resolve_repos(task: str | None, repos: list[str]) -> list[Path]: return [Path(r).resolve() for r in repos] -def emit_human(surfaces: list[dict]) -> None: +ADVERSARY_CONTRACT = """\ +INDEPENDENT ADVERSARY MODE (autonomous-ship gate, leg 5) + + Who may run this: NOT the session or model that wrote the branch. The leg's + entire value is independence — the organism's own audit found the review leg + on autonomous ships was in practice the branch's own author, that "a healthy + pass and a rote one write the identical ledger row", and that the one + confirmed-wrong load-bearing claim of its window lived outside the surface the + stage reads. The catches that did happen came from a different model reading + the same diff. A self-run adversary leg is not a weaker version of this leg; + it is an absent one, and recording it as run is a false ledger row. + + Your job is NOT to review the change. It is to FALSIFY its claims — starting + with the witness, which is the claim the whole task was scoped around. + + For each claim: what would make this false, and what in the diff, the tests or + a run actually shows it is not? Then say which. A claim you cannot falsify AND + cannot find a basis for is a FINDING (unverified-claim), not a pass. +""" + + +def emit_human(surfaces: list[dict], witness: str = "", adversary: bool = False) -> None: print("== ReviewSurface (review faculty — surface only; the verdict is the") print(" reviewing agent's, per agents/faculties/review/AGENTS.md) ==") for s in surfaces: @@ -191,6 +212,15 @@ def emit_human(surfaces: list[dict]) -> None: print(f" ? {c}") print("\nVerdict rubric: CLEAN (nothing must change) | FINDINGS (ranked,") print("file:line, failure scenario) | BLOCKED (could not review — say why).") + if witness: + print() + print(" THE WITNESS — the claim this task was scoped around, and the") + print(" one whose failure means the work did not do what it promised:") + print(f" * {witness}") + print(" Falsify it first. If it does not hold, nothing below matters.") + if adversary: + print() + print(ADVERSARY_CONTRACT) print("A load-bearing claim above with no falsified-by basis in the branch is") print("a FINDING (unverified-claim) — see the faculty AGENTS.md.") if any(s.get("claims_to_falsify") for s in surfaces): @@ -201,6 +231,25 @@ def emit_human(surfaces: list[dict]) -> None: print("malformed evidence, not CLEAN (faculty AGENTS.md step 2a).") +def surface_payload(surfaces: list[dict], witness: str = "", + adversary: bool = False) -> dict: + """Assemble the machine-readable surface. Pure — no git, no I/O. + + Split out from `main` so the contract can be tested without a repo that + happens to have a diff against `origin/main`. The first version of the + adversary tests drove the CLI against this checkout and passed locally and + failed in CI, where the checkout has no `origin/main` to diff and the whole + surface is empty: a test asserting on output that the environment decides. + """ + payload: dict = {"review_surface": surfaces} + if witness: + payload["witness"] = witness + if adversary: + payload["mode"] = "independent-adversary" + payload["contract"] = ADVERSARY_CONTRACT + return payload + + def main(argv=None) -> int: ap = argparse.ArgumentParser(prog="review") ap.add_argument("--task", default="", @@ -208,6 +257,13 @@ def main(argv=None) -> int: f"({WT_BASE})") ap.add_argument("--repo", action="append", default=[], help="explicit repo checkout path") ap.add_argument("--json", action="store_true", dest="as_json") + ap.add_argument("--witness", default="", + help="the task's Witness: header — the claim the work was " + "scoped around, falsified first") + ap.add_argument("--adversary", action="store_true", + help="independent-adversary mode (autonomous-ship gate leg " + "5): print the independence contract. Must be run by a " + "different model from the one that wrote the branch") a = ap.parse_args(argv) if not a.task and not a.repo: print("review: pass --task or --repo ", file=sys.stderr) @@ -221,9 +277,9 @@ def main(argv=None) -> int: print("review: no reviewable diff against origin/main", file=sys.stderr) return 4 if a.as_json: - print(json.dumps({"review_surface": surfaces}, indent=2)) + print(json.dumps(surface_payload(surfaces, a.witness, a.adversary), indent=2)) else: - emit_human(surfaces) + emit_human(surfaces, witness=a.witness, adversary=a.adversary) return 0 diff --git a/agents/faculties/sizing/AGENTS.md b/agents/faculties/sizing/AGENTS.md index 3ff55e1..2a5ea13 100644 --- a/agents/faculties/sizing/AGENTS.md +++ b/agents/faculties/sizing/AGENTS.md @@ -10,9 +10,68 @@ A PyAutoBrain read-only reasoning faculty. It owns the organism's single difficulty heuristic so the number is defined **once** and shared, never -recomputed by a divergent copy. +recomputed by a divergent copy — and, since 2026-08-30, the **review-cost +model** beside it. -It also owns the **precedence rule** over that heuristic. `estimate_difficulty` +## The review-cost model + +`Difficulty:` measures blast radius. It cannot answer the only question a batch +can be planned against, which is what a task costs the **human** once it lands. +Three outputs answer that, plus one that asks whether the human is needed at all: + +| Output | Header | Values | What it is | +|---|---|---|---| +| `consequence` | `Consequence:` | `notify` / `glance` / `judge` | how much review the work needs | +| `witness` | `Witness:` | free text | the machine-checkable claim that makes it reviewable | +| `review_minutes` | `Review-minutes:` | integer | a **seed**, not a measurement (see below) | +| `unattended` | `Unattended:` | `ready` / `needs-slicing` / `never` | can it finish without a human | + +Graded by **rules over repo class and surface**, never by an agent's reading of +its own work: the ledger's base rate for an agent mis-scoping its own change is +20% (68 of 332 records in 2026-08 carry a correction or a retraction), so +self-assessment is not an input. Repo class comes from `PyAutoMind/repos.yaml`, +where repo identity is already declared once. + +**The rule that carries the model: no `Witness:` means `judge`.** Without it the +field would be aspirational — a prompt could claim a cheap tier while offering +the reviewer nothing but the diff. With it, choosing a cheap tier means +committing at conception to producing evidence, which is what actually makes +work reviewable in minutes. A prompt carrying no witness grades `judge` however +small it looks, and that is the intended behaviour, not a gap. + +Measured over the 153 backlog prompts on the day this shipped: **151 grade +`judge`, because 3 carry a witness.** Given one, the same backlog grades 33 +`notify` / 104 `glance` / 16 `judge`. The entire distance between "everything +costs a PI's hour" and "a fifth of it costs nothing" is whether prompts declare +what will make them checkable. + +`unattended` is deliberately not difficulty renamed. `needs-slicing` keys off the +**compaction rule** — a task that would need context compaction to finish is too +big to run unattended — which is measured rather than cautious +(`anthropics/claude-code#54393`, a postmortem of five consecutive failed +autonomous overnight runs, names "good plan → compact → garbage drift" as a +primary failure primitive, and nothing downstream of the run catches it). + +`review_minutes` is a **seed awaiting calibration**, tier-driven with one nudge +for size. The honest numbers come from the batch records, which carry the minutes +the human actually spent. Never read a value from here as evidence about how long +anything took. + +**Known limit, stated rather than hidden.** The judged-surface test is keyword +matching over the prompt's prose. Fenced and inline code are masked — a prompt +*quoting* a surface is documenting it, not touching it, the same rule +`declared_header` applies — but prose that *describes* a judged surface still +trips it. The prompts specifying this very model are the worst offenders. The +error is in the safe direction (a false `judge` costs review time, a false +`notify` costs a bad merge), and the keyword list is kept deliberately narrow +because a loose one does not fail safe in the way it first appears: it grades +everything `judge`, the cheap tiers starve, and the model stops discriminating +at all. + +## Precedence + +It also owns the **precedence rule** over the heuristic, and over every output +of the review-cost model above. `estimate_difficulty` derives; `effective_difficulty` reconciles — a **declared** `Difficulty:` wins, and the derived level comes back alongside it so a disagreement is reported rather than silently resolved (the disagreement is evidence about the diff --git a/agents/faculties/sizing/_sizing.py b/agents/faculties/sizing/_sizing.py index 93a110b..ce98191 100755 --- a/agents/faculties/sizing/_sizing.py +++ b/agents/faculties/sizing/_sizing.py @@ -418,13 +418,56 @@ def empty_discovery_reason(mind: Path, work_type: str) -> str: # place, and gives the bug/refactor conductors the same reading for free. DIFFICULTY_LEVELS = ("small", "medium", "large", "too-large") AUTONOMY_LEVELS = ("safe", "supervised", "human-required") + +# --- the review-cost model (batch epic phase 0a) ----------------------------- +# `Difficulty:` is static blast radius. It cannot answer the only question a +# batch needs answered, which is what a task will cost the HUMAN once it lands. +# These three do. They are graded by RULES over repo class and surface, never by +# an agent's reading of its own work — the ledger's own base rate for an agent +# mis-scoping its own change is 20% (68 of 332 records in 2026-08 carry a +# correction or a retraction), so self-assessment is not an input here. +CONSEQUENCE_TIERS = ("notify", "glance", "judge") +UNATTENDED_LEVELS = ("ready", "needs-slicing", "never") +# Where the work can run. Spelled in the environment vocabulary +# `skills/WORKFLOW.md` already defines (`local-dev` / `web-github` / `ci-only` / +# `analysis-only`) rather than a parallel cloud/laptop one — `local-dev` means +# the work needs the local dataset and output trees, an SSH endpoint, or the +# human at the machine. Default `any`. +LANE_VALUES = ("any", "local-dev") + +# Surfaces that make a change a PI's decision whatever repo it lives in: a +# public API, a default value, an error contract, a science-policy call, or an +# external contributor's request. Judged from the prompt's own words. +# Kept deliberately specific. A loose keyword here does not fail safe in the way +# it first appears: it grades everything `judge`, the tier starves, and the model +# stops discriminating at all. `raise` alone matched "praise" and every prose +# mention of raising a question; `raises ` earns its place, bare `raise` does not. +JUDGE_SURFACE_KEYWORDS = [ + "public api", "default value", "defaults to", "error contract", + "raises ", "science policy", "external contributor", "external reporter", + "user-filed", "reported by", "backwards-compat", "backwards compat", + "breaking change", +] +# A witness claiming exact equality is the one machine-checkable claim strong +# enough to carry a behaviour-preserving change on its own. +BYTE_EQUALITY_MARKERS = [ + "byte-equal", "byte equality", "byte-identical", "bit-identical", + "bit identical", "byte-for-byte", "identical output", "unchanged output", +] +# Repo categories (PyAutoMind/repos.yaml) whose contents nobody outside this +# workshop consumes. Everything else is somebody's dependency. +INTERNAL_REPO_CATEGORIES = frozenset({"organ", "admin"}) +# Seeds, NOT measurements. Phase 7's batch records carry the minutes the human +# actually spent, and those are what will correct these. +REVIEW_MINUTES_SEED = {"notify": 0, "glance": 3, "judge": 20} # `medium` is not a documented Priority: value but occurs in the live backlog; # read it as normal rather than dropping the prompt's stated intent. PRIORITY_RANK = {"high": 0, "normal": 1, "medium": 1, "low": 2} DEFAULT_PRIORITY_RANK = 1 _HEADER_KEY_RE = re.compile( - r"^\s*(difficulty|type|autonomy|status|priority|blocked-by|closes-when)" + r"^\s*(difficulty|type|autonomy|status|priority|blocked-by|closes-when" + r"|consequence|witness|review-minutes|unattended|lane)" r"\s*:\s*(.+?)\s*$", re.I ) @@ -446,7 +489,10 @@ def declared_header(text: str) -> dict: """ out = {"declared_difficulty": None, "declared_type": None, "declared_autonomy": None, "status": None, - "priority": None, "blocked_by": [], "closes_when": []} + "priority": None, "blocked_by": [], "closes_when": [], + "declared_consequence": None, "witness": None, + "declared_review_minutes": None, "declared_unattended": None, + "lane": None} in_fence = False for line in text.splitlines(): if line.lstrip().startswith("```"): @@ -457,7 +503,11 @@ def declared_header(text: str) -> dict: m = _HEADER_KEY_RE.match(line) if not m: continue - key, value = m.group(1).lower(), _strip_trailing_comment(m.group(2)) + key = m.group(1).lower() + # `Witness:` is free text and routinely carries a `PR #123` reference, + # which the trailing-comment split would truncate. Take it raw. + value = m.group(2).strip() if key == "witness" \ + else _strip_trailing_comment(m.group(2)) if not value: continue if key == "difficulty": @@ -480,6 +530,24 @@ def declared_header(text: str) -> dict: out["blocked_by"].append(value) elif key == "closes-when": out["closes_when"].append(value) + elif key == "consequence": + v = value.lower() + if v in CONSEQUENCE_TIERS and out["declared_consequence"] is None: + out["declared_consequence"] = v + elif key == "witness" and out["witness"] is None: + out["witness"] = value + elif key == "review-minutes" and out["declared_review_minutes"] is None: + m2 = re.search(r"\d+", value) + if m2: + out["declared_review_minutes"] = int(m2.group(0)) + elif key == "lane": + v = value.lower() + if v in LANE_VALUES and out["lane"] is None: + out["lane"] = v + elif key == "unattended": + v = _norm_level(value) + if v in UNATTENDED_LEVELS and out["declared_unattended"] is None: + out["declared_unattended"] = v return out @@ -701,6 +769,159 @@ def estimate_difficulty(p: dict): return level, score, factors +def _repo_categories_of(p: dict) -> set: + """The body-map categories of the repos this prompt names. + + Repo *identity* is declared once, in `PyAutoMind/repos.yaml`. Reading the + category from there rather than keeping a second list here is the same rule + that governs every other repo fact in this module: one source, or the two + drift and whichever the reader happened to consult decides the answer. + """ + cats = _body_map_categories() + out = set() + for r in p["repos"]: + for name, spec in _body_map_specs().items(): + if canonical_key(name, spec) == r: + out.add(cats.get(name, "?")) + break + return out + + +def estimate_consequence(p: dict, factors: dict | None = None): + """Heuristic consequence tier -> (tier, reasons). + + `notify` costs the human nothing, `glance` costs a couple of minutes reading + the WITNESS, `judge` costs a PI's quarter-hour. The rules are ordered and the + first match wins, so every uncertain case falls through to `judge`: the tier + decides how little review something gets, and the safe direction for a + grader that is unsure is always more. + + The load-bearing rule is the last fallthrough — **no witness means judge**. + Without it the field would be aspirational: a prompt could claim a cheap tier + while offering the reviewer nothing to check but the diff. With it, choosing + a cheap tier means committing at CONCEPTION to producing evidence, which is + what actually makes work reviewable in minutes. The corollary is deliberate: + a prompt carrying no `Witness:` grades `judge` however small it looks. + """ + if factors is None: + _, _, factors = estimate_difficulty(p) + # Mask fenced blocks and inline code first, on the module's own standing + # rule: a prompt QUOTING a surface is documenting it, not touching it. The + # prompts that define this very model are the worst offenders — they name + # every judged surface in prose while touching none of them. + text = _mask_code(p["text"]).lower() + witness = (p.get("witness") or "").strip() + reasons = [] + + # 1. Work types whose deliverable is never a quietly-mergeable diff. + if p["work_type"] in ("release", HUMAN_REVIEW): + return "judge", [f"work-type {p['work_type']} is a human act by contract"] + + # 2. Surfaces that are a PI's call wherever they live. + surface = _hits(text, JUDGE_SURFACE_KEYWORDS) + if surface: + return "judge", [f"names a judged surface: {', '.join(surface[:3])}"] + + # 3. The default that makes the model bite. + if not witness: + return "judge", ["no Witness: declared — nothing to check but the diff"] + reasons.append(f"witness declared: {witness[:60]}") + + cats = _repo_categories_of(p) + internal = bool(cats) and cats <= INTERNAL_REPO_CATEGORIES + + # 4. A behaviour-preserving change proving exact equality reviews itself. + if p["work_type"] == "refactor" and _hits(witness.lower(), BYTE_EQUALITY_MARKERS): + return "notify", reasons + ["refactor with a byte-equality witness"] + + # 5. Work nobody outside this workshop consumes. + if internal: + return "notify", reasons + [f"internal repos only ({', '.join(sorted(cats))})"] + if p["work_type"] in ("docs", "test") and not (factors["library_repos"]): + return "notify", reasons + [f"{p['work_type']}, no library repo touched"] + + return "glance", reasons + ["witnessed change to a consumed repo"] + + +def estimate_unattended(p: dict, level: str, factors: dict | None = None): + """Can this finish without the human? -> (grade, reasons). + + Deliberately NOT difficulty renamed. Difficulty asks how big the blast + radius is; this asks whether one unattended run can carry the task to + PR-open. The rule that separates them is the compaction rule: **a task that + would need context compaction to finish is too big to run unattended.** + That is measured, not cautious — `anthropics/claude-code#54393`, a + postmortem of five consecutive failed autonomous overnight runs, names + "good plan -> compact -> garbage drift" as a primary failure primitive, and + nothing downstream of the run catches it. + """ + if factors is None: + _, _, factors = estimate_difficulty(p) + if p["work_type"] in ("release", HUMAN_REVIEW): + return "never", [f"work-type {p['work_type']} is human-driven by contract"] + if p.get("declared_autonomy") == "human-required": + return "never", ["declared Autonomy: human-required"] + if level == "too-large": + return "needs-slicing", ["too-large is a routing signal, not a grade"] + if level == "large" and factors["repos_affected"] > 2: + return "needs-slicing", [ + f"large across {factors['repos_affected']} repos — would compact"] + return "ready", ["fits one unattended run"] + + +def estimate_review_minutes(tier: str, level: str) -> int: + """A SEED, not a measurement. + + Everything here is derived from the tier plus a nudge for size; the honest + numbers come from phase 7's batch records, which carry the minutes the human + actually spent. Read a value from this function as "what to plan with until + we have measured", and never as evidence about how long anything took. + """ + minutes = REVIEW_MINUTES_SEED.get(tier, REVIEW_MINUTES_SEED["judge"]) + if tier == "judge" and level in ("large", "too-large"): + minutes += 5 + return minutes + + +def effective_consequence(p: dict, factors: dict | None = None): + """(tier, reasons, derived_tier) — the DECLARED tier wins. + + Same precedence rule as `effective_difficulty`, for the same reason: a value + the author declared is a judgement the heuristic does not have, and a + disagreement is evidence about the heuristic worth reporting rather than + silently resolving. + """ + derived, reasons = estimate_consequence(p, factors) + return p.get("declared_consequence") or derived, reasons, derived + + +def effective_unattended(p: dict, level: str, factors: dict | None = None, + derived_level: str | None = None): + """(grade, reasons, derived_grade) — the DECLARED grade wins. + + `derived_level` is the difficulty the heuristic derived, as opposed to the + one the author declared. It changes no verdict — declared still wins, per + the module's precedence rule — but a prompt the heuristic reads as + `too-large` and the author calls `medium` is exactly the case the compaction + rule exists to catch, so the tension is named in the reasons rather than + disappearing behind the precedence. + """ + derived, reasons = estimate_unattended(p, level, factors) + grade = p.get("declared_unattended") or derived + if grade == "ready" and derived_level == "too-large": + reasons = reasons + [ + "CAUTION: the heuristic derives too-large; the declared level won." + " Re-read before dispatching this unattended"] + return grade, reasons, derived + + +def effective_review_minutes(p: dict, tier: str, level: str): + """(minutes, derived_minutes) — a DECLARED estimate wins.""" + derived = estimate_review_minutes(tier, level) + declared = p.get("declared_review_minutes") + return (declared if declared is not None else derived), derived + + def effective_difficulty(p: dict): """(level, score, factors, derived_level) — the DECLARED level wins. @@ -729,6 +950,18 @@ def effective_difficulty(p: dict): # prompt without dispatching anything. It writes nothing. +def _disagree(field: str, declared, derived) -> None: + """Report a declared/derived split rather than resolving it silently. + + The precedence rule is that declared wins; the point of printing the other + value is that the split is evidence about the heuristic, and the heuristic + only improves if somebody sees it. + """ + if declared != derived: + print(f" ! declared {declared} but derived {derived}" + f" — declared wins; the disagreement is worth a look") + + def _main(argv=None): import argparse import json @@ -747,7 +980,11 @@ def _main(argv=None): args = ap.parse_args(argv) p = parse_prompt(Path(args.prompt).resolve(), args.mind.resolve()) - level, score, factors = estimate_difficulty(p) + level, score, factors, derived_level = effective_difficulty(p) + tier, tier_why, derived_tier = effective_consequence(p, factors) + grade, grade_why, derived_grade = effective_unattended( + p, level, factors, derived_level) + minutes, derived_minutes = effective_review_minutes(p, tier, level) surface = { "path": p["path"], "work_type": p["work_type"], @@ -755,17 +992,37 @@ def _main(argv=None): "repos": p["repos"], "lines": p["lines"], "words": p["words"], - "difficulty": {"level": level, "score": score, "factors": factors}, + "difficulty": {"level": level, "derived": derived_level, + "score": score, "factors": factors}, + "consequence": {"tier": tier, "derived": derived_tier, + "reasons": tier_why}, + "witness": p.get("witness"), + "review_minutes": {"minutes": minutes, "derived": derived_minutes, + "note": "a seed, not a measurement"}, + "unattended": {"grade": grade, "derived": derived_grade, + "reasons": grade_why}, } if args.json: print(json.dumps(surface, indent=2)) return + witness = p.get("witness") print(f"SizingSurface: {p['path']}") print(f" work-type : {p['work_type']}") print(f" target : {p['target']}") print(f" repos : {', '.join(p['repos']) or '(none)'}") print(f" size : {p['lines']} lines / {p['words']} words") print(f" difficulty: {level} (score {score})") + _disagree("difficulty", level, derived_level) + print(f" witness : {witness if witness else '(none declared)'}") + print(f" consequence: {tier} — {tier_why[0] if tier_why else ''}") + _disagree("consequence", tier, derived_tier) + print(f" review : ~{minutes} min (seed, not a measurement)") + if derived_minutes != minutes: + print(f" ! declared {minutes} but seeded {derived_minutes}") + print(f" unattended: {grade} — {grade_why[0] if grade_why else ''}") + for extra in grade_why[1:]: + print(f" ! {extra}") + _disagree("unattended", grade, derived_grade) if __name__ == "__main__": diff --git a/bin/pyauto-brain b/bin/pyauto-brain index b822c6c..9f8f909 100755 --- a/bin/pyauto-brain +++ b/bin/pyauto-brain @@ -57,6 +57,7 @@ FACULTIES_DIR="$AGENTS_DIR/faculties" declare -A AGENT_SCRIPT=( [intake]="$CONDUCTORS_DIR/intake/intake.sh" + [batch]="$CONDUCTORS_DIR/batch/batch.sh" [community]="$CONDUCTORS_DIR/community/community.sh" [feature]="$CONDUCTORS_DIR/feature/feature.sh" [bug]="$CONDUCTORS_DIR/bug/bug.sh" @@ -77,6 +78,7 @@ declare -A AGENT_SCRIPT=( ) declare -A AGENT_DESC=( [intake]="Conceive a task: turn raw input into a formal, headed PyAutoMind prompt (files it; never starts dev)" + [batch]="Compose one unattended shift: the BatchDecision, planned against the slot's REVIEW-MINUTE budget rather than a task count — proposes, never dispatches" [community]="The ears — the organism's receptive function: scan/triage user-filed GitHub issues + PRs and review requests across the repos; drafts stay human-gated, dev work routes via start_dev_for_user (never posts)" [feature]="Reason over PyAutoMind feature tasks: select, size, phase, plan for start_dev" [bug]="The immune system: classify a bug/regression/Heart finding, locate the fix, plan the repair" @@ -97,7 +99,7 @@ declare -A AGENT_DESC=( ) # Conductors are the front doors a human drives; faculties are consulted (and # runnable read-only). Both are dispatchable, but the menu groups them by tier. -CONDUCTOR_ORDER=(intake community feature bug refactor workspace eyes profiling hygiene clone build release health) +CONDUCTOR_ORDER=(intake batch community feature bug refactor workspace eyes profiling hygiene clone build release health) FACULTY_ORDER=(vitals review memory samplers sizing) AGENT_ORDER=("${CONDUCTOR_ORDER[@]}" "${FACULTY_ORDER[@]}") diff --git a/skills/batch/SKILL.md b/skills/batch/SKILL.md new file mode 100644 index 0000000..8f76a38 --- /dev/null +++ b/skills/batch/SKILL.md @@ -0,0 +1,22 @@ +--- +name: batch +description: Compose the next unattended batch through the PyAutoBrain Batch Agent — the BatchDecision, planned against the slot's review-minute budget rather than a task count, with backpressure, lane detection and the one-member-per-library-repo rule. Use when picking what to run in a shift, or asking what fits in a review slot. It proposes; approving it in the slot is what launches a batch. +--- + +# Batch + +Read [`../../agents/conductors/batch/AGENTS.md`](../../agents/conductors/batch/AGENTS.md) +completely, then run `bin/pyauto-brain batch plan` in the documented mode. + +Return the **BatchDecision** as written — members, the review-minutes it spends, +and what it rejected with reasons. Do not re-rank it, do not quietly add a +member the planner excluded, and do not present it as a schedule: it is a +proposal, and the human approving it in their slot is what launches the batch +([`../../AUTONOMY.md`](../../AUTONOMY.md), "What a batch launch is"). A +scheduler may carry the timing; it never carries the authority. + +Two parts of the output are the point and must survive into your reply: the +**review-minute total** (the budget is the human's hour, not a task count), and +the **other lane's ready count** when the session cannot plan it — *"4 local-dev +tasks are ready, run this from the laptop"*. An empty batch at the backpressure +cap is a finding, not a deadlock; say which. diff --git a/skills/batch/agents/openai.yaml b/skills/batch/agents/openai.yaml new file mode 100644 index 0000000..01f5665 --- /dev/null +++ b/skills/batch/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Batch Agent" + short_description: "Compose one unattended shift against a review-minute budget" + default_prompt: "Use $batch to plan the next batch — what fits in my review slot." diff --git a/skills/ship_library/ship_library.md b/skills/ship_library/ship_library.md index 0fd9adc..38266f6 100644 --- a/skills/ship_library/ship_library.md +++ b/skills/ship_library/ship_library.md @@ -82,7 +82,7 @@ landed. This is feature-development git work, not a Build/release step. In local-dev delegate the mechanical part to an execution-tier subagent; elsewhere run it directly. If any step fails, stop and report — do not proceed. -**Under `--auto`:** all four legs of the autonomous-ship gate must pass (step +**Under `--auto`:** all four legs of the autonomous-ship gate must pass — **five under a batch launch**, which adds the independent-adversary leg (`AUTONOMY.md` leg 5) — (step 3 note); then ship **without interactive sign-off** — the PR body additionally carries the `## Validation checklist` section ([`reference.md`](reference.md) → "Validation checklist (--auto)"), the run diff --git a/skills/ship_workspace/ship_workspace.md b/skills/ship_workspace/ship_workspace.md index 36ec0a6..ba95654 100644 --- a/skills/ship_workspace/ship_workspace.md +++ b/skills/ship_workspace/ship_workspace.md @@ -79,7 +79,7 @@ directly), `gh pr create --label pending-release`, verify the label, and cross-reference the upstream library PR if linked. In local-dev delegate to a execution-tier subagent; elsewhere run directly. Any failure → stop and report. -**Under `--auto`:** all four legs of the autonomous-ship gate must pass (step +**Under `--auto`:** all four legs of the autonomous-ship gate must pass — **five under a batch launch**, which adds the independent-adversary leg (`AUTONOMY.md` leg 5) — (step 3 note); ship without interactive sign-off, add the `## Validation checklist` to the PR body (`../ship_library/reference.md` → "Validation checklist (--auto)"), **stop at PR-open**, append the calibration row to diff --git a/tests/data/sizing_review_cost_golden.json b/tests/data/sizing_review_cost_golden.json new file mode 100644 index 0000000..f1b03ea --- /dev/null +++ b/tests/data/sizing_review_cost_golden.json @@ -0,0 +1,30 @@ +{ + "api_change": { + "consequence": "judge", + "derived_consequence": "judge", + "difficulty": "small", + "review_minutes": 20, + "unattended": "ready" + }, + "doc_no_witness": { + "consequence": "judge", + "derived_consequence": "judge", + "difficulty": "small", + "review_minutes": 20, + "unattended": "ready" + }, + "doc_with_witness": { + "consequence": "notify", + "derived_consequence": "notify", + "difficulty": "small", + "review_minutes": 0, + "unattended": "ready" + }, + "refactor_byte_equal": { + "consequence": "notify", + "derived_consequence": "notify", + "difficulty": "small", + "review_minutes": 0, + "unattended": "ready" + } +} diff --git a/tests/test_batch_plan.py b/tests/test_batch_plan.py new file mode 100644 index 0000000..42dea0c --- /dev/null +++ b/tests/test_batch_plan.py @@ -0,0 +1,130 @@ +"""tests/test_batch_plan.py — the Batch Agent's composition rules. + +The planner's whole job is refusing things for stated reasons, so every test +here is a refusal (or the one case that must never be refused: the floor). +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +BRAIN = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(BRAIN / "agents" / "faculties" / "sizing")) +_spec = importlib.util.spec_from_file_location( + "_batch_under_test", BRAIN / "agents" / "conductors" / "batch" / "_batch.py") +_batch = importlib.util.module_from_spec(_spec) +sys.modules["_batch_under_test"] = _batch +_spec.loader.exec_module(_batch) + + +def rec(path, *, minutes=20, tier="judge", ready="ready", repos=(), epic="", + lane="any", blocked=False, priority="normal"): + return {"path": path, "repos": list(repos), "work_type": "feature", + "difficulty": "medium", "score": 4, "consequence": tier, + "witness": None, "review_minutes": minutes, "unattended": ready, + "why": "", "ready_why": [], "epic": epic, "lane": lane, + "priority": priority, "blocked": blocked} + + +def paths(d): + return [m["path"] for m in d["members"]] + + +def why(d, path): + return next(w for p, w in d["rejected"] if p == path) + + +def test_the_budget_is_review_minutes_not_a_task_count(): + """Three 20-minute tasks do not fit a 45-minute slot, however small they + look by any other measure.""" + d = _batch.plan([rec(f"{i}.md") for i in "abc"], budget=45) + assert len(d["members"]) == 2 + assert d["review_minutes_planned"] == 40 + assert "exceed the budget" in why(d, "c.md") + + +def test_notify_tier_work_costs_the_human_nothing(): + """The fill. It is capped by the token allowance, not by the human's hour, + which is what lets the allowance be spent without growing the queue.""" + d = _batch.plan([rec(f"{i}.md", tier="notify", minutes=0) for i in range(9)], + budget=45) + assert len(d["members"]) == 9 + assert d["review_minutes_planned"] == 0 + + +def test_one_member_per_library_repo_per_shift(): + """They do not collide at dispatch — separate worktrees — they collide at + merge, when the first /prm moves main and invalidates the rest.""" + d = _batch.plan([rec("a.md", repos=["autoarray"]), + rec("b.md", repos=["autoarray"])], budget=100) + assert paths(d) == ["a.md"] + assert "autoarray already claimed" in why(d, "b.md") + + +def test_non_library_repos_do_not_claim_a_shift(): + """Two docs changes to the same organ repo cost nothing to merge together.""" + d = _batch.plan([rec("a.md", repos=["pyautomind"]), + rec("b.md", repos=["pyautomind"])], budget=100) + assert paths(d) == ["a.md", "b.md"] + + +def test_one_slice_per_epic(): + d = _batch.plan([rec("a.md", epic="euclid"), rec("b.md", epic="euclid")], + budget=100) + assert paths(d) == ["a.md"] + assert "epic euclid already" in why(d, "b.md") + + +def test_a_session_never_plans_the_other_lane_but_reports_it(): + """Silently dropping it would leave the human unable to tell 'nothing ready' + from 'nothing I can run from here'.""" + d = _batch.plan([rec("a.md", lane="local-dev")], session_lane="web-github") + assert d["members"] == [] + assert d["other_lane_ready"] == 1 + d2 = _batch.plan([rec("a.md", lane="local-dev")], session_lane="local-dev") + assert paths(d2) == ["a.md"] + + +def test_only_unattended_ready_work_is_selected(): + d = _batch.plan([rec("a.md", ready="needs-slicing"), + rec("b.md", ready="never"), + rec("c.md", blocked=True)], budget=100) + assert d["members"] == [] + assert "needs-slicing" in why(d, "a.md") + assert "Blocked-by" in why(d, "c.md") + + +def test_backpressure_ramps_rather_than_cliffs(): + pool = [rec(f"{i}.md") for i in range(6)] + assert _batch.plan(pool, budget=60, awaiting_review=0)["effective_budget"] == 60 + assert _batch.plan(pool, budget=60, awaiting_review=6)["effective_budget"] == 30 + assert _batch.plan(pool, budget=60, awaiting_review=8)["effective_budget"] == 0 + + +def test_at_the_cap_the_floor_is_fill_only_and_still_runs(): + """A conference week must not stop the thing whose whole purpose is working + while nobody watches. At the cap the review-bearing half is zero — but + zero-cost work still dispatches.""" + d = _batch.plan([rec("a.md"), rec("b.md", tier="notify", minutes=0)], + budget=60, awaiting_review=99) + assert paths(d) == ["b.md"] + + +def test_cheapest_first_because_the_question_is_what_fits(): + d = _batch.plan([rec("slow.md", minutes=20, priority="high"), + rec("fast.md", minutes=2, priority="low")], budget=45) + assert paths(d)[0] == "fast.md" + + +def test_every_rejection_states_a_reason(): + d = _batch.plan([rec("a.md"), rec("b.md"), rec("c.md")], budget=20) + assert d["rejected"] + assert all(isinstance(w, str) and w for _, w in d["rejected"]) + + +def test_lane_detection_reads_the_environment_not_a_flag(): + """A session that could be told where it is could plan local-dev work it + cannot run. Probed from `gh`, the same signal the organism already uses.""" + assert _batch.detect_lane() in ("local-dev", "web-github") diff --git a/tests/test_intake_dashboard.py b/tests/test_intake_dashboard.py index 97c19e8..1beea28 100644 --- a/tests/test_intake_dashboard.py +++ b/tests/test_intake_dashboard.py @@ -28,10 +28,17 @@ # fixtures # --------------------------------------------------------------------------- # def _prompt(title, difficulty="medium", autonomy="supervised", priority="normal", - status="formalised"): + status="formalised", unattended=None, review_minutes=None, + consequence=None, witness=None): + extra = "" + for key, value in (("Consequence", consequence), ("Witness", witness), + ("Review-minutes", review_minutes), + ("Unattended", unattended)): + if value is not None: + extra += f"{key}: {value}\n" return (f"# {title}\n\nType: feature\nTarget: widgets\n" f"Difficulty: {difficulty}\nAutonomy: {autonomy}\n" - f"Priority: {priority}\nStatus: {status}\n\nBody prose.\n") + f"Priority: {priority}\nStatus: {status}\n{extra}\nBody prose.\n") def _mind(root: Path, drafts=None, active=None, registries=None, @@ -76,19 +83,64 @@ def test_start_here_leads_with_high_priority_smallest_first(tmp_path): assert "Later thing" not in head, "a low-priority prompt is not a pick" -def test_quick_wins_are_small_and_safe_only(tmp_path): +def test_fits_a_slot_lists_only_unattended_ready_work(tmp_path): + """Replaced "Quick wins" (`small and safe`), which was near-empty: ten + prompts in the live backlog carried `safe`, so the surface that exists to + hand out unattended work had almost nothing to hand out. The question is + not how small the work is — it is whether it can finish without the human.""" mind = _mind(tmp_path, drafts={ - "feature/widgets/a.md": _prompt("Small and safe", difficulty="small", - autonomy="safe"), - "feature/widgets/b.md": _prompt("Small but supervised", difficulty="small", - autonomy="supervised"), - "feature/widgets/c.md": _prompt("Safe but large", difficulty="large", - autonomy="safe"), + "feature/widgets/a.md": _prompt("Ready and cheap", unattended="ready", + review_minutes="0"), + "feature/widgets/b.md": _prompt("Needs slicing", difficulty="too-large", + unattended="needs-slicing"), + "feature/widgets/c.md": _prompt("Never unattended", unattended="never"), + "feature/widgets/d.md": _prompt("Ungraded"), }) - quick = _page(mind).split("**Quick wins**")[1].split("## In flight")[0] - assert "Small and safe" in quick - assert "Small but supervised" not in quick - assert "Safe but large" not in quick + slot = _page(mind).split("**Fits a slot**")[1].split("## In flight")[0] + assert "Ready and cheap" in slot + assert "Needs slicing" not in slot + assert "Never unattended" not in slot + assert "Ungraded" not in slot + + +def test_fits_a_slot_is_ordered_by_what_review_costs(tmp_path): + """Ordered by review-minutes ascending, not by priority. This list is read + when the human has a slot to fill and wants to know what fits in it; + `Highest priority` above is where importance is answered.""" + mind = _mind(tmp_path, drafts={ + "feature/widgets/a.md": _prompt("Expensive but urgent", priority="high", + unattended="ready", review_minutes="20"), + "feature/widgets/b.md": _prompt("Cheap and dull", priority="low", + unattended="ready", review_minutes="2"), + }) + slot = _page(mind).split("**Fits a slot**")[1].split("## In flight")[0] + assert slot.index("Cheap and dull") < slot.index("Expensive but urgent") + + +def test_an_ungraded_prompt_sorts_last_rather_than_being_hidden(tmp_path): + """A prompt that is `ready` with no review-minutes is still pickable; the + page's standing rule is that unknown sorts last, never disappears.""" + mind = _mind(tmp_path, drafts={ + "feature/widgets/a.md": _prompt("No cost recorded", unattended="ready"), + "feature/widgets/b.md": _prompt("Costed", unattended="ready", + review_minutes="5"), + }) + slot = _page(mind).split("**Fits a slot**")[1].split("## In flight")[0] + assert slot.index("Costed") < slot.index("No cost recorded") + + +def test_a_prompt_with_no_witness_is_reported_as_hygiene(tmp_path): + """Not an error — the intended default. But the human is the only thing + that can write one, so the page has to say which prompts are waiting.""" + mind = _mind(tmp_path, drafts={ + "feature/widgets/a.md": _prompt("Has one", witness="ids bit-identical"), + "feature/widgets/b.md": _prompt("Has none"), + }) + page = _page(mind) + hygiene = page.split("## Hygiene")[1] + assert "no `Witness:`" in hygiene + assert "feature/widgets/b.md" in hygiene + assert "feature/widgets/a.md" not in hygiene def test_every_backlog_prompt_is_one_collapsed_row_not_a_wide_table(tmp_path): diff --git a/tests/test_intake_review_cost.py b/tests/test_intake_review_cost.py new file mode 100644 index 0000000..4071d4a --- /dev/null +++ b/tests/test_intake_review_cost.py @@ -0,0 +1,92 @@ +"""tests/test_intake_review_cost.py — intake writes the review-cost model. + +Two things are locked here. First, the header round-trip: what `analyse` derives +is what `_render_header` writes and what `parse_header` reads back, with +`Witness:` written **only** when the author supplied one. Second, the +2026-08-30 change to `infer_autonomy` — including the variant that was measured +and reverted, so nobody re-proposes it. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "agents" / "faculties" / "sizing")) +sys.path.insert(0, str(ROOT / "agents" / "conductors" / "intake")) +from _intake import ( # noqa: E402 + HEADER_FIELDS, analyse, infer_autonomy, parse_header, +) + +RAW = "Tidy the @PyAutoBrain board copy so a 403 reads as an egress block.\n" +WITNESSED = RAW + "Witness: the degraded row renders and a test pins its text.\n" + + +def test_header_round_trips_through_parse_header(): + header = analyse(WITNESSED, "test")["header"] + got = parse_header(header) + assert got["consequence"] == "notify" + assert got["witness"].startswith("the degraded row renders") + assert got["review-minutes"] == "0" + assert got["unattended"] == "ready" + + +def test_witness_is_written_only_when_supplied(): + """It is the one field that cannot be derived. An invented one would be + plausible prose with nothing behind it — strictly worse than none, because + the whole value of the field is that its absence is informative.""" + assert "Witness:" in analyse(WITNESSED, "test")["header"] + bare = analyse(RAW, "test") + assert "Witness:" not in bare["header"] + assert bare["consequence"] == "judge" + assert any("No Witness" in r for r in bare["risks"]) + + +def test_witness_stays_out_of_the_hygiene_set(): + """`intake formalise` writes every field in HEADER_FIELDS that a prompt is + missing. A `Witness:` in there would be auto-invented across the backlog.""" + assert "witness" not in HEADER_FIELDS + assert "unattended" not in HEADER_FIELDS + assert "consequence" in HEADER_FIELDS + assert "review-minutes" in HEADER_FIELDS + + +def _factors(**over): + base = {"repos_affected": 1, "architectural_risk": [], "human_judgement": [], + "library_repos": [], "workspace_repos": [], "organism_repos": [], + "library_and_workspace": False, "size_words": 100, + "scientific_complexity": [], "test_burden": [], + "memory_context_required": False} + base.update(over) + return base + + +def test_multi_repo_alone_no_longer_forces_supervised(): + """The 2026-08-30 change. Repo count is blast radius, which + `estimate_difficulty` already prices at +2 per repo; this field is about + whether a human's judgement is needed.""" + assert infer_autonomy("medium", _factors(repos_affected=4)) == "safe" + + +def test_real_judgement_signals_still_force_supervised(): + assert infer_autonomy("medium", _factors(architectural_risk=["api"])) == "supervised" + assert infer_autonomy("large", _factors()) == "supervised" + assert infer_autonomy("too-large", _factors()) == "supervised" + + +def test_ambiguity_alone_does_not_force_supervised(): + """The variant that was measured and REVERTED: adding `human_judgement` as a + supervised trigger took `safe` from 30 to 24 across the backlog, because the + ambiguity keywords fire on 63% of prompts and catch well-written ones + indiscriminately — the same mistake as the rule it replaced. Locked so it is + not re-introduced by someone reasoning from first principles.""" + assert infer_autonomy("medium", _factors(human_judgement=["investigate"])) == "safe" + + +def test_unscoped_ambiguity_is_still_human_required(): + """The one place `human_judgement` legitimately fires: nothing to scope + against, so nobody can size or gate the work.""" + assert infer_autonomy( + "medium", _factors(repos_affected=0, human_judgement=["unclear"]) + ) == "human-required" diff --git a/tests/test_review_adversary.py b/tests/test_review_adversary.py new file mode 100644 index 0000000..6ca8973 --- /dev/null +++ b/tests/test_review_adversary.py @@ -0,0 +1,90 @@ +"""tests/test_review_adversary.py — the independent-adversary leg (gate leg 5). + +Step 2a was already adversarial in procedure and was still run, in practice, by +the branch's own author. This leg's only new content is *independence* plus +*ordering* — the task's witness is falsified before anything else — so those are +what is pinned here. + +Hermetic, and deliberately so after the first version was not: driving the CLI +against this checkout passed locally and failed in CI, where there is no +`origin/main` to diff and the surface comes back empty. A test whose subject is +a block of contract text must not depend on the environment having a diff. +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +BRAIN = Path(__file__).resolve().parents[1] +_spec = importlib.util.spec_from_file_location( + "_review_adversary_under_test", + BRAIN / "agents" / "faculties" / "review" / "_review.py") +_review = importlib.util.module_from_spec(_spec) +sys.modules["_review_adversary_under_test"] = _review +_spec.loader.exec_module(_review) + +WITNESS = "ids bit-identical, 62 -> 9.7 ms" +# Mirrors `repo_surface`'s return shape exactly — a fixture that drifts from the +# producer is a test of the fixture. +SURFACE = [{ + "repo": "PyAutoBrain", "path": "/tmp/PyAutoBrain", "branch": "feature/x", + "base": "abc123456789", "commits_ahead": 2, + "commits": ["do the thing"], "shortstat": "3 files changed", + "files": ["a.py"], "risk_flags": [], + "claims_to_falsify": ["This change is a no-op for CI."], +}] + + +def _emit(capsys, **kw): + _review.emit_human(SURFACE, **kw) + return capsys.readouterr().out + + +def test_the_witness_is_lifted_and_falsified_first(capsys): + """It is the claim the task was scoped around: if it does not hold, the work + did not do what it promised and nothing else on the surface matters.""" + out = _emit(capsys, witness=WITNESS) + assert WITNESS in out + assert "THE WITNESS" in out + assert out.index("THE WITNESS") < out.index("A load-bearing claim above") + assert "Falsify it first" in out + + +def test_no_witness_section_when_none_is_declared(capsys): + """A task with no witness grades `judge` and is reviewed by a human; the + surface must not imply a claim that was never made.""" + assert "THE WITNESS" not in _emit(capsys) + + +def test_adversary_mode_states_who_may_not_run_it(capsys): + """The independence rule is the leg. A surface that omits it invites the + exact failure the leg exists to close.""" + out = _emit(capsys, adversary=True) + assert "NOT the session or model that wrote the branch" in out + assert "A self-run adversary leg is not a weaker version" in out + + +def test_contract_is_absent_unless_asked_for(capsys): + assert "INDEPENDENT ADVERSARY MODE" not in _emit(capsys) + + +def test_payload_carries_witness_and_mode(): + p = _review.surface_payload(SURFACE, WITNESS, True) + assert p["witness"] == WITNESS + assert p["mode"] == "independent-adversary" + assert "contract" in p + assert p["review_surface"] == SURFACE + + +def test_payload_omits_them_when_not_asked(): + p = _review.surface_payload(SURFACE) + assert "witness" not in p and "mode" not in p and "contract" not in p + + +def test_the_surface_is_assembled_without_touching_git(): + """`surface_payload` is pure: the CLI resolves repos and reads diffs, this + only shapes what was found. That split is what makes the tests above + hermetic.""" + assert _review.surface_payload([])["review_surface"] == [] diff --git a/tests/test_sizing_review_cost.py b/tests/test_sizing_review_cost.py new file mode 100644 index 0000000..8f0be32 --- /dev/null +++ b/tests/test_sizing_review_cost.py @@ -0,0 +1,182 @@ +"""tests/test_sizing_review_cost.py — the review-cost model (batch epic 0a). + +`Difficulty:` measures blast radius. These four outputs measure what a task +costs the HUMAN, which is the only quantity a batch can be planned against. + +Two rules carry the whole model and each gets its own test: + + * **no `Witness:` means `judge`** — without it a prompt could claim a cheap + tier while offering the reviewer nothing but the diff, and the field would + be aspirational rather than load-bearing; + * **declared beats derived** — the module's standing precedence rule, which + three conductors re-implemented and two got wrong (PyAutoBrain#217, #274). + +The golden file pins the grade for a fixed sample so a change to the heuristic +has to be seen and accepted rather than absorbed. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "agents" / "faculties" / "sizing")) +from _sizing import ( # noqa: E402 + CONSEQUENCE_TIERS, + UNATTENDED_LEVELS, + effective_consequence, + effective_difficulty, + effective_review_minutes, + effective_unattended, + estimate_review_minutes, + parse_prompt, +) + +GOLDEN = Path(__file__).parent / "data" / "sizing_review_cost_golden.json" + + +def _prompt(tmp_path: Path, body: str, rel: str = "draft/feature/pyautobrain/t.md"): + path = tmp_path / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(body) + return parse_prompt(path, tmp_path) + + +def _grade(p): + level, _score, factors, derived_level = effective_difficulty(p) + tier, tier_why, derived_tier = effective_consequence(p, factors) + grade, _why, _dg = effective_unattended(p, level, factors, derived_level) + minutes, _dm = effective_review_minutes(p, tier, level) + return {"difficulty": level, "consequence": tier, "derived_consequence": derived_tier, + "review_minutes": minutes, "unattended": grade, "why": tier_why[0]} + + +HEADERLESS_DOC = """# Tidy the Brain README + +Type: docs +Target: PyAutoBrain +Repos: +- PyAutoBrain +Difficulty: small +Priority: normal +Status: draft + +Rewrite two paragraphs. Nothing else. +""" + + +def test_no_witness_grades_judge(tmp_path): + """The load-bearing default. A docs change to an organ repo is about as + consequence-free as work gets — and it still costs a PI's hour, because + nothing was promised that a reviewer could check.""" + g = _grade(_prompt(tmp_path, HEADERLESS_DOC)) + assert g["consequence"] == "judge" + assert "no Witness" in g["why"] + assert g["review_minutes"] == 20 + + +def test_witness_unlocks_the_cheap_tier(tmp_path): + """The same prompt, with a witness, is the same work — but now reviewable + in the time it takes to read one line, so it grades `notify`.""" + body = HEADERLESS_DOC.replace( + "Status: draft", + "Status: draft\nWitness: every link in the file resolves; docs build clean") + g = _grade(_prompt(tmp_path, body)) + assert g["consequence"] == "notify" + assert g["review_minutes"] == 0 + + +def test_judged_surface_beats_a_witness(tmp_path): + """A witness makes work checkable; it does not make an API decision + somebody else's to make. Surface outranks evidence.""" + body = HEADERLESS_DOC.replace( + "Status: draft", + "Status: draft\nWitness: byte-identical output on the full suite" + ).replace("Rewrite two paragraphs. Nothing else.", + "Change the default value of over_sample_size.") + assert _grade(_prompt(tmp_path, body))["consequence"] == "judge" + + +def test_declared_beats_derived_and_reports_the_split(tmp_path): + """Precedence, and the reason the derived value is returned at all: the + disagreement is evidence about the heuristic and has to stay visible.""" + body = HEADERLESS_DOC.replace( + "Status: draft", "Status: draft\nConsequence: judge") + g = _grade(_prompt(tmp_path, body)) + assert g["consequence"] == "judge" # declared wins + assert g["derived_consequence"] == "judge" # no witness -> judge anyway + + body2 = HEADERLESS_DOC.replace( + "Status: draft", + "Status: draft\nConsequence: judge\nWitness: docs build clean") + g2 = _grade(_prompt(tmp_path, body2)) + assert g2["consequence"] == "judge" # declared still wins + assert g2["derived_consequence"] == "notify" # and the split is reported + + +def test_prose_describing_a_surface_is_not_touching_one(tmp_path): + """A prompt quoting a judged surface inside a code fence is documenting it. + Same rule, same reason, as `declared_header`'s fenced-block skip.""" + body = HEADERLESS_DOC.replace( + "Rewrite two paragraphs. Nothing else.", + "Document the guard:\n\n```\nraises ValueError on a bad default value\n```\n" + ).replace("Status: draft", "Status: draft\nWitness: docs build clean") + assert _grade(_prompt(tmp_path, body))["consequence"] == "notify" + + +def test_unattended_is_not_difficulty_renamed(tmp_path): + """`needs-slicing` keys off the compaction rule, not off size alone: a + single-repo `large` task still fits one run.""" + body = HEADERLESS_DOC.replace("Difficulty: small", "Difficulty: large") + assert _grade(_prompt(tmp_path, body))["unattended"] == "ready" + + wide = HEADERLESS_DOC.replace("Difficulty: small", "Difficulty: large").replace( + "Repos:\n- PyAutoBrain", + "Repos:\n- PyAutoBrain\n- PyAutoMind\n- PyAutoHeart\n- PyAutoHands") + assert _grade(_prompt(tmp_path, wide))["unattended"] == "needs-slicing" + + +def test_human_required_is_never_unattended(tmp_path): + body = HEADERLESS_DOC.replace("Priority: normal", + "Autonomy: human-required\nPriority: normal") + assert _grade(_prompt(tmp_path, body))["unattended"] == "never" + + +def test_review_minutes_are_a_seed_not_a_measurement(): + """Documented as a seed, and shaped like one: tier-driven, with a single + nudge for size. Anything more precise would be inventing certainty.""" + assert estimate_review_minutes("notify", "small") == 0 + assert estimate_review_minutes("glance", "small") == 3 + assert estimate_review_minutes("judge", "small") == 20 + assert estimate_review_minutes("judge", "too-large") == 25 + + +def test_vocabularies_are_closed(): + assert CONSEQUENCE_TIERS == ("notify", "glance", "judge") + assert UNATTENDED_LEVELS == ("ready", "needs-slicing", "never") + + +def test_golden_sample_is_unchanged(tmp_path): + """Pins the grade for a fixed sample. A heuristic change that moves these + is not necessarily wrong — but it has to be looked at and re-accepted, + which is the entire job of a golden file.""" + samples = { + "doc_no_witness": HEADERLESS_DOC, + "doc_with_witness": HEADERLESS_DOC.replace( + "Status: draft", "Status: draft\nWitness: docs build clean"), + "api_change": HEADERLESS_DOC.replace( + "Rewrite two paragraphs. Nothing else.", + "Change the public API of Grid2D.").replace( + "Status: draft", "Status: draft\nWitness: 900 tests pass"), + "refactor_byte_equal": HEADERLESS_DOC.replace("Type: docs", "Type: refactor").replace( + "Status: draft", "Status: draft\nWitness: ids bit-identical, 62 -> 9.7 ms"), + } + got = {k: _grade(_prompt(tmp_path, v, f"draft/feature/pyautobrain/{k}.md")) + for k, v in samples.items()} + for g in got.values(): + g.pop("why") + if not GOLDEN.exists(): # first run writes the pin + GOLDEN.parent.mkdir(parents=True, exist_ok=True) + GOLDEN.write_text(json.dumps(got, indent=2, sort_keys=True) + "\n") + assert got == json.loads(GOLDEN.read_text())