Make the paired evaluator harness-agnostic

.eval-output/README.md

@@ -0,0 +1,37 @@
+# Local evaluation artifacts
+
+This directory is gitignored except for this README.
+
+Put live `--output` and `--output` calibration directories here:
+
+```text
+.eval-output/<run-name>/
+```
+
+Or use any path outside the repository. Do not commit run directories.
+
+## What to keep and inspect
+
+Retained evidence for a paired run:
+
+- `run.json`, `config.json`, `tasks.jsonl`
+- `task-<id>/trial-NNN/report.json` and `report.md`
+- `control/` and `treatment/`: `response.md`, `trace.jsonl`, `stderr.txt`
+- judge artifacts under `judge-NNN/` and `pairwise-NNN/` when a rubric ran
+
+Those files are the evaluation record. Read them locally. Share only after
+review; they can contain prompts, model output, and harness metadata.
+
+## What is not evidence
+
+Harness runtime homes are copied or created for the process, then deleted
+when the run finishes cleanly:
+
+- `cursor-home/` (includes `chats/**/store.db`)
+- `codex-home/` (may include a copied `auth.json` during the run)
+- `claude-home/`, `hermes-home/`, `agy-home/`
+- per-invocation `home/` directories
+
+If those directories are still here, the run was interrupted or predates
+cleanup. Do not commit them. SQLite chat databases are especially noisy in
+`git status` and are not part of the comparison report.

.gitignore

@@ -1,7 +1,13 @@
+.DS_Store
 .agents/
-.eval-runs/
 .pytest_cache/
 .ruff_cache/
 __pycache__/
 *.py[cod]
 node_modules/
+
+# Live evaluation artifacts. Inspect AI ignores logs/; Cursor agent-trace
+# ignores .agent-trace/. Keep harness traces and Cursor chat DBs local.
+.eval-output/*
+!.eval-output/README.md
+.eval-runs/

AGENTS.md

@@ -4,8 +4,5 @@
 
 Follow the maintainability principles in [ZEN.md](ZEN.md).
 
-## Next change
 
-Harness adapters, judging, and domain tests now have owners. `skill_eval_loop.py` is still ~2k lines; load models still serialize to dicts before the rest of the pipeline. Do not split further until a later change needs a new owner.
 
-The remaining evidence gate is Task 10: a repeated-trial promotion run on an independently controlled, human-labeled holdout. Do not invent that holdout here.

README.md

@@ -1,10 +1,14 @@
 # skill-eval-loop
 
-`skill-eval-loop` is a self-contained Python 3 Agent Skill that measures
-whether explicitly applying one local skill changes task outcomes. The control
-receives the original task. The treatment receives the exact hashed skill's
-`SKILL.md` instructions in its prompt, with the installed payload available for
-referenced files. The runner retains the raw evidence and a comparison report.
+`skill-eval-loop` measures one question: for this skill, these JSONL tasks,
+this CLI harness, and this opaque model id, does injecting `SKILL.md` change
+the outcome versus the same prompt with no skill?
+
+The runner does not talk to a model vendor. It runs whatever binary
+`--harness` / `--harness-bin` names, passes `--model` through unchanged, and
+keeps traces. `--harness script` is the escape hatch for any other program.
+Control gets the raw task. Treatment gets the hashed skill instructions.
+Reports are derived from retained evidence.
 
 ## Install
 
@@ -21,8 +25,19 @@ The public launcher requires Python 3 and no package installation:
 ```bash
 EVALUATOR="$PWD/.agents/skills/skill-eval-loop/scripts/skill-eval-loop"
 "$EVALUATOR" healthcheck
+"$EVALUATOR" models --harness pi
 ```
 
+Copy `--model` and `--judge-model` from that harness's listing. If the listing
+is non-empty, `run` and `calibrate` reject ids that are not on it. An empty
+listing does not reject. Omit `--harness-bin` when the CLI is on `PATH` or in a
+standard location such as `~/.local/bin`. Relative, `~/`, and symlink paths are
+resolved to canonical absolute paths before validation and recording. Re-use an
+output directory with `--force`. The default timeout is 300 seconds. For a
+quality-complete rubric run, use the same `--harness` for student and judge: a
+different `--judge-harness` can mark a run independent, but that calibration
+cannot bind.
+
 ## Run an evaluation
 
 Create a JSONL task file. Every non-empty line needs a unique, path-safe `id`,
@@ -36,14 +51,12 @@ Run a side-effect-free plan before a live invocation:
 
 ```bash
 "$EVALUATOR" run \
-  --skill /absolute/path/to/target-skill \
-  --tasks /absolute/path/to/tasks.jsonl \
-  --output /absolute/path/to/fresh-run \
-  --harness codex \
-  --harness-bin /absolute/path/to/codex \
+  --skill ./skills/target-skill \
+  --tasks ./tasks.jsonl \
+  --output "$PWD/.eval-output/fresh-run" \
+  --harness pi \
   --model exact-model-id \
   --trials 1 \
-  --timeout-seconds 300 \
   --dry-run
 ```
 
@@ -67,32 +80,65 @@ The checked-in development benchmark evaluates Vercel's
 Fetch that exact revision into a controlled local directory and pass the
 absolute skill subpath plus the checked-in task file to `run --dry-run`. Reject
 the plan if the revision or payload hash differs. The public task file is
-development evidence, not a secret client holdout.
+development evidence, not a secret client holdout. The 2026-09-01 Cursor run
+showed v1 is saturated for `claude-sonnet-5-medium`: both conditions already
+met the rubric.
+
+### Quality dataset (v2)
+
+Use `tasks/react-best-practices-v2.jsonl` with
+`tests/fixtures/calibration/react-review-v1.json`. Eight review-in-prompt
+examples share binary dimensions `primary_diagnosis`, `actionable_fix`, and
+`grounded_claims`. Bound calibration must cover every task rubric dimension
+name. Run at least 3 trials on the same harness pair. This remains a
+development experiment until humans review a sample of transcripts.
+
+- expected v2 task SHA-256:
+  `9c3a558691d2507ccb332d8f20d25422f52f4ab9602b479c0e2e0c3a498e959b`
+- expected react-review calibration fixture SHA-256:
+  `5b093a444351abe683dbe3177f0c1ef93f161e02c466a94b9b0376de599cfd06`
+
+The 2026-09-01 same-harness Cursor v2 run
+(`.eval-output/react-best-practices-v2-cursor/`) used the previous task SHA
+and is retained as provisional evidence. `memo-default-callback` was then
+rewritten so the gold label matches React memo semantics: Header's inline
+default is passed into a memoized child.
+
+The 2026-09-02 bound independent run
+(`.eval-output/react-best-practices-v2-cursor-agy-judge/`) used that SHA,
+`claude-sonnet-5-medium` on `cursor-agent`, and `gemini-3.6-flash-high` on
+`antigravity` after a 3/3 accepted same-harness calibration of that judge id.
+It exited 0 with `quality_status: independent`. Across 24 paired trials,
+`quality_outcome` was control 15, tie 4, treatment 2, inconsistent 3.
+Per-output binary scores were at ceiling for both conditions
+(`primary_diagnosis` and `grounded_claims` 24/24; `actionable_fix` control
+23/24, treatment 24/24). Pairwise leftover scoring is not a skill win. This
+is not a promotion claim.
 
 For rubric tasks, also pass `--judge-model` with a different exact model
-identifier and `--calibration /absolute/path/to/calibration.json` from an
-accepted calibrate run. The runner judges each condition only after
-deterministic gates pass. A valid same-provider judgment is
-`provisional_non_independent`; a timeout, failed gate, malformed response, or
+identifier and `--calibration` from an accepted calibrate run on that same
+harness pair. The runner judges each condition only after deterministic gates
+pass. Color-choice fixtures cannot bind v2. A valid same-harness judgment
+is `provisional_non_independent`; a timeout, failed gate, malformed response, or
 identity mismatch is `unknown`. A missing trace-reported model is unattested,
 not a quality unknown. Omitting `--calibration` is allowed, but a rubric run
 then remains quality-incomplete and cannot exit `0`.
 
-The runner invokes Codex sequentially in read-only mode, emitting invocation
-progress to stderr. Odd trials run control first; even trials run treatment
-first. The evaluator injects the exact `SKILL.md` text itself, so treatment
-exposure does not depend on model-side discovery. Target, judge, and calibration
-invocations share one lifecycle that uses cleaned OS-temporary workspaces outside
-the evaluator repository. It retains `run.json`, the
-planned configuration, tasks, condition responses, traces, stderr, and a
+The runner invokes the configured harness sequentially in read-only mode,
+emitting invocation start, 15-second heartbeats, and finish lines to stderr. Odd trials run control first; even
+trials run treatment first. The evaluator injects the exact `SKILL.md` text
+itself, so treatment exposure does not depend on model-side discovery. Target,
+judge, and calibration invocations share one lifecycle that uses cleaned
+OS-temporary workspaces outside the evaluator repository. It retains `run.json`,
+the planned configuration, tasks, condition responses, traces, stderr, and a
 JSON/Markdown report for every pair.
 
 `runner_valid` means the runner held its declared variables, isolation checks,
 and treatment activation. It is not a general quality claim. Read both transcripts before
 interpreting `treatment_only`, `both_pass`, `control_only`, or `both_fail`.
 
 JSON and Markdown reports expose evaluator-recorded instruction delivery plus
-optional trace telemetry when Codex also reads the installed skill, rolled-up timing and token usage,
+optional trace telemetry when the harness also reads the installed skill, rolled-up timing and token usage,
 calibration (`not_run`, or `accepted` plus `fixtures_sha256` when a bound
 calibration is supplied), every judged dimension, `quality_status`, and
 `quality_outcome`. Deterministic-only reports say semantic quality was not
@@ -110,9 +156,8 @@ Calibrate the pairwise judge against versioned human-labeled
 ```bash
 python3 skills/skill-eval-loop/scripts/skill_eval_loop.py calibrate \
   --fixtures /absolute/path/to/calibration/v1.json \
-  --output /absolute/path/to/fresh-calibration \
-  --harness codex \
-  --harness-bin /absolute/path/to/codex \
+  --output "$PWD/.eval-output/fresh-calibration" \
+  --harness pi \
   --model exact-model-id \
   --judge-model exact-judge-model-id \
   --dry-run
@@ -134,17 +179,38 @@ owner.
 
 ## Boundaries
 
-The minimum runner supports Codex, deterministic graders, a provisional
-same-provider rubric judge, blinded pairwise comparison, human-labeled
-calibration fixtures, and hash-bound two-reviewer promotion evidence. It records
-operator-supplied cost; it does not discover pricing, provide an independent
-automated judge, authenticate human identities, run in parallel, discover
-providers, or adapt other harnesses.
-
-Live evaluation is a trusted local-operator workflow. The configured harness
-and Codex executable can read the run-local Codex credentials and therefore
-must be trusted. This project does not sandbox hostile executables. Keep raw
-run directories local and inspect them before sharing any evidence.
+The supported path is: list enumerable model ids, use those ids on the harness
+that will call them, keep student and judge on the same harness, calibrate,
+dry-run, live-run, and read the retained traces. Deterministic graders can
+complete without a judge.
+
+Adapters exist for Codex, Claude Code, Cursor Agent, Muse, Hermes, Pi,
+Antigravity, and a custom script. CI proves mechanics with fake Codex and
+`script`. Local live dogfood on 2026-09-01 succeeded on cursor-agent, pi,
+muse, antigravity, hermes, and codex. Claude is installed but not logged in
+on this machine (`claude auth status` reports `loggedIn: false`); the adapter
+treats that JSON error as a failed invocation rather than a model answer.
+`models` enumerates a harness when that CLI can list ids (Muse catalog,
+Cursor Agent `--list-models`, Pi `--list-models`, Antigravity `agy models`).
+An empty listing does not reject an id. `--harness script` wraps any other
+binary.
+
+`--promotion` plus `prepare-review` / `finalize-review` implement a human
+review workflow. They do not prove an independent holdout or complete a
+promotion claim. Tasks run in empty temp workspaces, so repository-editing
+evals are not quality evidence.
+
+The runner records operator-supplied cost. It does not pick models, discover
+providers, price calls, authenticate reviewers, or run in parallel.
+
+Live evaluation is a trusted local-operator workflow. Write `--output` under
+the gitignored `.eval-output/` directory or outside the repo. The runner
+retains reports, responses, traces, and stderr. Harness homes (`cursor-home`,
+`codex-home`, and Cursor `chats/**/store.db`) are deleted after a clean run;
+leftovers from interrupted runs are still not git material. The configured
+harness executable can read local credentials and therefore must be trusted.
+This project does not sandbox hostile executables. Inspect raw runs locally
+before sharing any evidence.
 
 ## Development
 

docs/minimum-eval-contract.md

@@ -22,15 +22,16 @@ Changing any component creates a different evaluation.
 
 A run requires:
 
-- an absolute skill directory;
-- either an absolute newline-delimited JSON task file or the target-owned
+- a skill directory (CLI paths may be relative or `~/`; retained evidence stores
+  the canonical absolute path);
+- either a newline-delimited JSON task file or the target-owned
   `evals/tasks.jsonl` file;
 - a supported harness and its resolved executable;
 - an exact target model identifier;
 - a different exact judge model identifier when a task uses a rubric;
 - a positive trial count;
-- a positive timeout;
-- an absolute output directory;
+- a positive timeout (CLI default 300 seconds);
+- an output directory (same path-resolution rule as the skill path);
 - an exact judge model when any task uses a rubric grader.
 
 The retained configuration also records the harness version, hashes of the