Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
28e03e9
docs(run-to-fix): write the adapter contract
opPO333 Nov 17, 2026
df72eef
docs(run-to-fix): name the placeholders each adapter command needs
opPO333 Nov 17, 2026
7530ae9
test(run-to-fix): cover read and classify on hand-made jobs
opPO333 Nov 17, 2026
c1caf99
test(run-to-fix): look for agent class columns by their prefix
opPO333 Nov 17, 2026
7600676
feat(run-to-fix): read a run and classify its trials
opPO333 Nov 17, 2026
09e4a2f
test(run-to-fix): cover compare, the exact tests and the stop rule
opPO333 Nov 17, 2026
eb4d087
feat(run-to-fix): compare a new run and apply the stop rule
opPO333 Nov 17, 2026
a142991
fix(run-to-fix): print a negative budget left with its sign first
opPO333 Nov 17, 2026
c913145
test(run-to-fix): cover candidates, checks, both stops and the trial run
opPO333 Nov 17, 2026
508a5ba
feat(run-to-fix): check candidates and write both decision pages
opPO333 Nov 17, 2026
4c0aa8e
test(run-to-fix): count the trial run in the money a round has spent
opPO333 Nov 17, 2026
b2070e7
fix(run-to-fix): count the trial run in the money a round has spent
opPO333 Nov 17, 2026
045b6a7
docs(run-to-fix): say that the run of tasks is always written for a p…
opPO333 Nov 17, 2026
efd58da
test(run-to-fix): count every trial run of a round in money and in ta…
opPO333 Nov 17, 2026
b039633
fix(run-to-fix): count every trial run of a round
opPO333 Nov 17, 2026
e5b6cac
docs(run-to-fix): write the procedure, the classes and the files of t…
opPO333 Nov 17, 2026
1964839
test(run-to-fix): mask quoted strings in exception signatures
opPO333 Nov 17, 2026
ae90436
feat(run-to-fix): mask quoted strings in exception signatures
opPO333 Nov 17, 2026
b5deae4
docs(run-to-fix): describe the quote mask, the group example and the …
opPO333 Nov 17, 2026
7844841
docs(run-to-fix): state the effect of the quote mask in its own sentence
opPO333 Nov 17, 2026
960cb28
test(run-to-fix): name the runs of a round page by their job folders
opPO333 Nov 18, 2026
275f7c6
fix(run-to-fix): name the runs of a round page by their job folders
opPO333 Nov 18, 2026
9785255
docs(run-to-fix): say how the runs of compare-metrics.json are labelled
opPO333 Nov 18, 2026
fc0bc93
test(run-to-fix): price each task by its recorded cost, and two small…
opPO333 Nov 18, 2026
548fc8f
fix(run-to-fix): price each task of a run by its recorded cost
opPO333 Nov 18, 2026
25d9f95
test(run-to-fix): pin which braces in a command line are placeholders
opPO333 Nov 18, 2026
895aaef
docs(run-to-fix): describe the price of a run, the shell the loop nee…
opPO333 Nov 18, 2026
0476f1f
test(run-to-fix): refuse a loop folder that git would list as untracked
opPO333 Nov 18, 2026
f8acef2
fix(run-to-fix): refuse a loop folder that git would list as untracked
opPO333 Nov 18, 2026
3b84239
docs(run-to-fix): say where a loop folder may lie and how read checks it
opPO333 Nov 18, 2026
b280125
test(run-to-fix): read a loop folder outside a repository under a tra…
opPO333 Nov 18, 2026
8499bf4
fix(run-to-fix): run git in the C locale when read checks the loop fo…
opPO333 Nov 18, 2026
5aab4c5
test(run-to-fix): replay before the fix, and the effect a candidate p…
opPO333 Nov 19, 2026
f8fce30
feat(run-to-fix): replay before the fix, and the effect a candidate p…
opPO333 Nov 19, 2026
9df77ed
docs(run-to-fix): describe the ref before a fix, the effect of a cand…
opPO333 Nov 19, 2026
b3c2e80
test(run-to-fix): count the same trials as run-analysis when a standa…
opPO333 Nov 19, 2026
48bb0c3
test(run-to-fix): use a made-up agent's error and made-up counts
opPO333 Nov 19, 2026
43d263f
test(run-to-fix): say why a trial run has no guards, and what a repor…
opPO333 Nov 19, 2026
647879e
fix(run-to-fix): say why a trial run has no guards, and what a report…
opPO333 Nov 19, 2026
c162efe
docs(run-to-fix): name the skill in AGENTS.md and show a report predi…
opPO333 Nov 19, 2026
e9cd01a
test(run-to-fix): pin that a round and a trial run need a budget
opPO333 Nov 19, 2026
da7edb1
test(run-to-fix): keep the spending of a run unknown while a trial re…
opPO333 Nov 19, 2026
8437b15
fix(run-to-fix): keep the spending of a run unknown while a trial rep…
opPO333 Nov 19, 2026
b41ddeb
test(run-to-fix): count a trial run only for the tasks and commit sel…
opPO333 Nov 19, 2026
433f368
fix(run-to-fix): count a trial run only for the tasks and commit sele…
opPO333 Nov 19, 2026
73e0d35
test(run-to-fix): leave no earlier success behind a failing check, an…
opPO333 Nov 19, 2026
664ad42
fix(run-to-fix): leave no earlier success behind a failing check, and…
opPO333 Nov 19, 2026
981df7d
test(run-to-fix): run a written paid command in paths that hold a spa…
opPO333 Nov 19, 2026
646a886
fix(run-to-fix): run a written paid command in paths that hold a spac…
opPO333 Nov 19, 2026
6b8cf5c
test(run-to-fix): stop the adapter command and remove its worktree wh…
opPO333 Nov 19, 2026
c82a7e4
test(run-to-fix): close the file of the command's process id
opPO333 Nov 19, 2026
6b4e22d
fix(run-to-fix): stop the adapter command and remove its worktree whe…
opPO333 Nov 19, 2026
a80871b
test(run-to-fix): refuse a found class that reads outside its trial t…
opPO333 Nov 19, 2026
495eb29
fix(run-to-fix): refuse a found class that reads outside its trial th…
opPO333 Nov 19, 2026
8a27658
test(run-to-fix): refuse a budget or a cost that is not a finite amou…
opPO333 Nov 19, 2026
983a282
test(run-to-fix): give the amount tests runs of unknown cost, so the …
opPO333 Nov 19, 2026
bb12b04
fix(run-to-fix): refuse a budget or a cost that is not a finite amoun…
opPO333 Nov 19, 2026
4729695
test(run-to-fix): show on stop 1 the trials whose replay keeps their …
opPO333 Nov 19, 2026
919f499
test(run-to-fix): quote the replay's post-processing of kept trials once
opPO333 Nov 19, 2026
0035b45
fix(run-to-fix): show on stop 1 the trials whose replay keeps their r…
opPO333 Nov 19, 2026
2a0f0cc
test(run-to-fix): record a timed-out check, fix the stop rule at read…
opPO333 Nov 19, 2026
9a4da9b
fix(run-to-fix): record a check whose command ran past its time as on…
opPO333 Nov 19, 2026
9bf9d24
fix(run-to-fix): check a candidate id and a cause name wherever they …
opPO333 Nov 19, 2026
e7c0512
fix(run-to-fix): name a report candidate's trials on stop 1 without a…
opPO333 Nov 19, 2026
22eb83c
fix(run-to-fix): record the stop rule when a round starts, and apply …
opPO333 Nov 19, 2026
820ab69
test(run-to-fix): name the expected canary and the expected classes o…
opPO333 Nov 19, 2026
76e5bbf
test(run-to-fix): end the expected canary line with an escaped newline
opPO333 Nov 19, 2026
ba61318
test(run-to-fix): use a made-up verifier line with no personal name
opPO333 Nov 19, 2026
8d05cbb
docs(run-to-fix): describe the stated commit and costs of a trial run…
opPO333 Nov 19, 2026
a59f8f5
docs(run-to-fix): state the stated commit and the unknown cost of a t…
opPO333 Nov 19, 2026
8ed6bf4
test(run-to-fix): count a trial's model calls by the standard's own c…
opPO333 Nov 19, 2026
1cd0551
test(run-to-fix): import json at the top of the classify tests
opPO333 Nov 19, 2026
20c7bfa
fix(run-to-fix): count a trial's model calls by the standard's own co…
opPO333 Nov 19, 2026
9aa6195
test(run-to-fix): name output_limit as switched off when no trial rec…
opPO333 Nov 19, 2026
0a3ab37
fix(run-to-fix): name output_limit as switched off when no trial reco…
opPO333 Nov 19, 2026
1ed8e08
test(run-to-fix): write unit ids into the task lists of a trial run o…
opPO333 Nov 19, 2026
5afc2be
fix(run-to-fix): write unit ids into the task lists of a trial run on…
opPO333 Nov 19, 2026
1548aaa
test(run-to-fix): run stages 1, 2 and 5 for an agent with no checkout…
opPO333 Nov 19, 2026
8af19ea
fix(run-to-fix): run stages 1, 2 and 5 for an agent with no checkout …
opPO333 Nov 19, 2026
f4291c6
docs(run-to-fix): describe the standard's model calls and units, an a…
opPO333 Nov 19, 2026
4afbc08
test(run-to-fix): refuse a recorded cost or budget that is not a fini…
opPO333 Nov 19, 2026
0d8e4db
fix(run-to-fix): refuse a recorded cost or budget that is not a finit…
opPO333 Nov 19, 2026
5a087ac
test(run-to-fix): refuse a check made with another adapter at stop 1,…
opPO333 Nov 19, 2026
676cc5d
fix(run-to-fix): refuse a check made with another adapter at stop 1, …
opPO333 Nov 19, 2026
cdbd0ef
test(run-to-fix): a trial run holds every selected task, and its repe…
opPO333 Nov 19, 2026
6157626
fix(run-to-fix): a trial run holds every selected task, and its repea…
opPO333 Nov 19, 2026
554c9b0
test(run-to-fix): refuse a selected task with no unit id or with two …
opPO333 Nov 19, 2026
c678a23
fix(run-to-fix): choose the unit id of each selected task, and refuse…
opPO333 Nov 19, 2026
e62ce95
test(run-to-fix): the message for a round of another stop rule, a par…
opPO333 Nov 19, 2026
8c15f1d
fix(run-to-fix): tell the person to keep the loop folder and start a …
opPO333 Nov 19, 2026
9d4ab97
fix(run-to-fix): count the trials that record no tokens of a reply, c…
opPO333 Nov 19, 2026
75d16cd
docs(run-to-fix): state the coverage and unit id messages and the mod…
opPO333 Nov 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
210 changes: 210 additions & 0 deletions .claude/skills/run-to-fix/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,210 @@
---
name: run-to-fix
description: Turn a finished benchmark run into checked fix candidates for the agent, as a loop with two stops for a person - classify every trial, find causes backed by counts, prove each cause with a test, check fix candidates on the recorded trials, try the kept fixes on chosen tasks, and compare the new run with exact tests and a stop rule written in advance. Works for any agent and both run paths of Nullius (a Harbor recipe or the standard), with everything that belongs to one agent in an adapter file. Use whenever the user has a finished run and wants to find the agent's errors and fix them, run another round of fixes, decide which fix to keep, or check whether a fix helped. For a one-off analysis or comparison page without fixes, use run-analysis.
---

# Run to fix

Takes a finished run of an agent and turns it into fix candidates that have been checked before any money is spent
on them, then into a new run that says whether the fixes helped. One pass through the five stages below is a round.
A round stops twice and waits for a person: which candidates to keep, and what to run next at what cost. The person
names every budget and starts every command that costs money.

The work splits as in `run-analysis`. The bundled script counts and keeps the records, all in one folder. It reads
only the files both run paths write. Explaining is done by you: reading trials, naming
causes, writing tests and fixes in the agent's code.

## What it needs

- The `run-analysis` skill in a folder beside this one. `loop.py` calls its `runs.py` for `health`, `analyze` and
`render`, and stops with a message naming the missing file when it is absent.
- Python 3 with the standard library, git, and bash on a POSIX system: Linux or WSL. The tests of this skill ran
on Python 3.12.
- A loop folder the person names, outside every git repository or in a folder its repository ignores. `loop.py`
writes only there, and `read` refuses a loop folder that git would list as untracked.
- An adapter, `adapter.json` in the loop folder: one file per agent with its checkout, its commands and the limits
of the run (`references/adapter.md`). For an agent of the Nullius catalogue, draft it from
`catalogue/agents/<key>/agent.json` (`versions.<v>.upstream.repository` and `tag`) and the run's settings, and
ask the person to confirm it. Ask the person for every value you cannot read from a file. An agent with no checkout
of its own takes `"source": null`: stages 1, 2 and 5 run, and the stages that change its code refuse.
- The run: one job folder with one folder per trial. A baseline run to compare it with is optional.

`L` below is `scripts/loop.py` beside this skill. Every command takes `--loop <loop folder>`. `references/files.md`
lists every file the loop writes and reads.

## Stage 1. Read the run

```bash
python3 L read --loop DIR --run JOB [--baseline JOB] --budget-usd B [--pass-threshold 1]
```

The person names `B`, the budget of the round. `read` runs `health` and `analyze` of `run-analysis`, writes
`input.json`, `health.txt` and `metrics.json`, and refuses a run that `health` calls not analysable (no trials, one
shared error, an incomplete or unfinished job) or whose trials name another agent than the adapter. A refusal is the
answer: give its cause, as `run-analysis` describes for a broken run, and stop. After a round whose stop rule said
stop, `read` also refuses until the person's words are passed with `--continue-after-stop "..."`.

## Stage 2. Find causes

```bash
python3 L classify --loop DIR
```

`classify` gives each trial one of seven classes and a signature, groups the failing trials by signature, and
attaches warnings (`references/classes.md`). It writes `trials.csv` and `groups.json`, and runs `commands.classes` of
the adapter for the agent's own classes when the adapter has it.

Then read two or three trials of each group, largest first, as `run-analysis/references/investigation.md` describes.
When you see a pattern the seven classes do not separate, write it as a found class in `found-classes.json` and run
`classify` again, so that the script counts it on every trial, the passing ones included. Write each cause to
`round-NN/causes/<id>.md`: the claim, a line `Rule: <group or found class>` that counts its trials, the trials and
their count, the code lines at fault at `base_ref`, the layer (model, agent, harness or benchmark), and what you did
not check. A group with the warning `fails_on_baseline` points at the benchmark; read one task's verifier and
expected answer before you write that the benchmark is defective.

## Stage 3. Prove and fix

For each cause in the agent layer:

1. On `source.work_branch` of the checkout, commit a test on a recorded trial that fails on the code of `base_ref`.
Then commit one or more fixes, each its own candidate; a candidate is a commit whose history holds the test.
2. Register and check each candidate:

```bash
python3 L candidate --loop DIR --id ID --cause CAUSE --ref FIX --test PATH --test-ref TEST_COMMIT --effect E
python3 L check --loop DIR --candidate ID
python3 L decide --loop DIR --stop 1
```

`--effect` says what the fix does, with no default. `outcome`: failing trials of the rule turn into passes; the rule
counts its failing trials, and their tasks are expected to pass. `report`: the agent's report changes on trials of
the rule and pass or fail stays; the rule counts every trial it matches, no task is expected to pass, and the
candidate needs `commands.replay`.

`check` runs, each in a detached worktree of the commit that is removed afterwards: the test before and after the fix,
`commands.test` against the failures the suite has on `base_ref`, and `commands.replay` on every recorded trial at the
ref before the fix (the candidate's `test_ref`, else `base_ref`) and at the fix. The replay's labels crossed with pass
and fail give the table done and passed / done and failed / not done and passed / not done and failed, and the count of
false successes, failed trials the agent reports as a success. `check` also counts the trials of the cause whose label
the fix changes, by direction. A command named in `paid` is written to `round-NN/commands/` and not started.

**Stop 1.** Show the person `decision-1.md`: one row per candidate with its cause, effect, test, table before and after
with its two refs, the trials of the cause whose label changes, false successes added and diff size; the candidates
whose fix changes no label of their cause, under a heading of their own; and the checks that did not run. Ask which candidates to keep, and record the answer in
the person's words:

```bash
python3 L answer --loop DIR --stop 1 --keep ID[,ID] --text "the person's words"
```

## Stage 4. Build and try

Merge the kept candidates into one commit of the work branch. Then:

```bash
python3 L select --loop DIR --ref COMMIT --canaries N
```

prints the trial run in three sizes with their price: the targets (the tasks the kept `outcome` candidates predict),
plus the guards (passing tasks whose recorded trial ran the changed code: the replay marks it `touched`, or a kept fix
changes its label), plus the canaries
(passing tasks drawn with a fixed seed). Every task gets 3 trials. A price is the sum over the tasks of the mean
recorded cost of each task's trials, times its trials. The cost comes from this round's trial runs on the same commit
when they hold the task, else from the run that was read. A task with no recorded cost takes the mean of the others,
and the printed line counts such tasks. The person picks a size and names its budget, which cannot exceed what the
round has left:

```bash
python3 L select --loop DIR --ref COMMIT --canaries N --size targets|guards|canaries --budget-usd B --text "..."
```

This writes `trial-tasks.txt`, runs `commands.build` and writes `build.json`, and writes the run to
`round-NN/commands/run_tasks.sh`. The person starts it. When its job folder exists:

```bash
python3 L decide --loop DIR --stop 2 --trial-job JOB --code-ref COMMIT [--trial-cost-usd X]
```

`--code-ref` is the commit the person states the trial run ran. A job does not record its code; the page shows the
commit as the person's statement, and `decide` refuses any commit other than the one `select` chose. The job must hold
every selected task and no other, each with one number of trials. The trials per task count over this round's trial
jobs on that commit: a runner that gives each task one trial per job is started three times, and each job goes to
`decide --stop 2`. The page says how many trials per task the jobs hold together and whether the selection is
complete; the targets are tested only when it is. When some of its trials
report no cost, `decide` refuses the run until the person gives its whole cost with `--trial-cost-usd`, at least what
its trials record.

**Stop 2.** Show the person `decision-2.md`. For a kept `report` candidate it says that this version does not read the
agent's report in the new run. The page shows the build, the targets' trials before and in every trial run of this
round on that build with Fisher's exact test, the guards and canaries, the money spent and left, and three options
with their price: stop, the selected tasks again, a full pass. The commands of the last two are written. Record
the choice:

```bash
python3 L answer --loop DIR --stop 2 --choice stop|rerun|full --text "the person's words"
```

A rerun is another trial run of the round: run `decide --stop 2` on its job folder, and its trials and money add to
the earlier ones.

## Stage 5. Compare and record

```bash
python3 L compare --loop DIR --new JOB --against run [--new-cost-usd X]
```

`compare` reads the new run against the round's run (`--against baseline` compares with the round's baseline) and
writes `result.json`: the pass rates with their standard error; the tasks both runs share, won and lost; the verdict
"improved" or "regressed" when an exact two-sided binomial test on won and lost gives p under 0.05, "noise" otherwise;
the targets' trials with Fisher's exact test; each prediction of stage 3 beside its result; and the stop rule. When
some trials of the new run report no cost, the budget part of the stop rule is not checked, unless the person gives the
run's cost with `--new-cost-usd`. `read` records the text of the stop rule, and `compare` refuses a round whose recorded
rule is not the rule of the skill it runs: keep that loop folder as it is and start a new one. It
closes the round and writes `report.html` with `render` of `run-analysis`; fill its prose slots as that skill
describes (`runs.py render --slots`, then `runs.py check`).

The stop rule is fixed before a round starts. No new round when the verdict is noise and no group of failing trials
of the new run has 3 trials or more, or when the budget left is under the price of the cheapest option. That option
is the lowest-priced trial run of one such group's tasks, 3 trials each, each task at the mean cost of its trials in
the new run; `result.json` names the group.

## When the adapter lacks a command

| null command | what the loop does | what is weaker |
|---|---|---|
| `test_one` | the test of a cause does not run | nothing shows that the cause is real or that the fix removes it |
| `test` | the agent's suite does not run | a fix may break other code unseen |
| `replay` | no table before and after, no guards | false successes and the effect on passing trials are unknown until a paid run |
| `build` | the run installs the checkout | the build tried is not recorded by digest |
| `classes` | no extra columns | the seven classes stay the only ones |
| `run_tasks` | no command is written | the person starts the run of `trial-tasks.txt` by other means |
| a null limit | its class is switched off | those trials fall into a later class, usually `verifier_failed` |

Every decision page names the checks that did not run.

## Writing rules

These follow `run-analysis`, with what the loop adds.

- **Every count says what it counts**: trials, tasks or model calls, and of which run.
- **A cause is a claim with a count.** It names the rule that counts its trials and the trials it explains. A
plausible mechanism without a count is a hypothesis; say so.
- **Respect the noise.** Quote the verdict and its p value with any gain. Do not call a round an improvement on a
noise verdict.
- **Separate the layers.** Model, agent, harness or benchmark. A claim that a benchmark is defective needs the check
its warning names.
- **The person decides.** Never start a written command, pick a size or keep a candidate for the person, and store
each answer in the person's words.
- **Nothing sensitive.** No keys, tokens, host addresses or personal identifiers from logs.

## Limits

- `read` decides that a run is not analysable from the opening words of the verdicts `runs.py health` prints. A change
of that wording in `run-analysis` changes what `read` refuses.
- The classes read only shared files. What one agent writes reaches the loop through a found class or
`commands.classes`.
- A trial job gives every task the same number of trials, because `run_tasks` takes one `{repeat}`.
- A loop stopped with SIGTERM or Ctrl+C stops the adapter command it runs and removes its worktree. A loop killed with
SIGKILL cannot: its worktree stays under `<loop folder>/worktrees/`, and `git worktree remove --force <path>` in the
agent's checkout removes it.
- Every amount in US dollars on the command line is a finite number of 0 or more.
25 changes: 25 additions & 0 deletions .claude/skills/run-to-fix/assets/decision-1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Stop 1 of {{ROUND}}: the candidates to keep

{{SUMMARY}}

| candidate | cause | effect | trials of the cause | test before → after the fix | suite failures added | replay before → after (refs) | trials of the cause whose label changes | false successes added | diff |
|---|---|---|---|---|---|---|---|---|---|
{{ROWS}}

Replay counts are of the {{TRIALS}} recorded trials: done and passed / done and failed / not done and passed / not
done and failed. A false success is a failed trial that the agent reports as a success. A trial that keeps its
recorded label is one the replay could not reproduce; its label comes from the recorded run.

## Candidates whose fix changes no label of their cause

{{UNCHANGED}}

## Checks that did not run

{{NOT_RUN}}

## Answer

```
python3 loop.py answer --loop {{LOOP}} --stop 1 --keep IDS --text "the person's words"
```
23 changes: 23 additions & 0 deletions .claude/skills/run-to-fix/assets/decision-2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Stop 2 of {{ROUND}}: what to run next

Build: {{BUILD}}. Code of the trial runs: {{CODE}}. Trial runs on it: {{TRIAL_JOB}}, {{TRIALS}} trials.
{{COVERAGE}} Spent in this round: {{SPENT}}. Budget left:
{{LEFT}} of {{BUDGET}}.

| tasks | trials passed before | trials passed in the trial run | test |
|---|---|---|---|
{{RESULTS}}

| option | what runs | price | command |
|---|---|---|---|
{{OPTIONS}}
{{REPORT_NOTE}}

Each task is priced at the mean cost of its trials in this round's trial runs on this build, else in the run that
was read, times the trials it gets. A task with no recorded cost takes the mean of the others.

## Answer

```
python3 loop.py answer --loop {{LOOP}} --stop 2 --choice stop|rerun|full --text "the person's words"
```
Loading