Skip to content

feat(runs): add the run-to-fix skill, from a finished run to checked fixes and a compared rerun - #244

Draft
opPO333 wants to merge 92 commits into
ThinkFlowLab:mainfrom
opPO333:feat/run-to-fix-skill
Draft

opPO333 wants to merge 92 commits into
ThinkFlowLab:mainfrom
opPO333:feat/run-to-fix-skill

Conversation

@opPO333

@opPO333 opPO333 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Closes #241

What

The skill run-to-fix under .claude/skills/run-to-fix/ takes a finished run to checked fix candidates and a
compared rerun. A round has five stages (read, classify, check, select, compare) and stops twice for a person: which
candidates to keep, and what runs next at what price. The script scripts/loop.py counts and keeps the records in a
loop folder the person names. The assistant reads trials, names causes and writes tests and fixes in the agent's
code. Everything that belongs to one agent lives in one adapter file (references/adapter.md). The skill calls
runs.py of run-analysis for health, metrics and the round page, and imports nothing from app/. AGENTS.md
gets one paragraph on the skill.

Review order: SKILL.md, then references/adapter.md, scripts/loop.py, and the stage modules in stage order
(reading.py, classifying.py, checking.py, selecting.py, deciding.py, comparing.py). Feedback wanted on two
choices: the adapter contract, and the price of a trial run. A price counts each task at the recorded cost of its own
trials.

Why one PR: the change is one new folder that no other code imports, and the author asks the reviewer to agree to one pull request.

Size: verdict: SPLIT: code lines 2356 > hard 500; split the branch or add a Why one PR line from the allowlist
(support lines 3179, 28 files in 2 areas).

Touches

CI and skills: the new folder .claude/skills/run-to-fix/ and one paragraph of AGENTS.md. No file of the
application changes.

Verified

Merge base 0e6ae8cff68380486bb9bca10261e83f40a5bf54 (main), head f4291c6d80281420df2d44d55cf8c9f7b90064d0.

python3 -m unittest discover -s .claude/skills/run-to-fix/tests from the repository root: 154 tests, OK, on
Python 3.12.3. Each change of code was committed after its test, and that test failed on the commit before it.

acceptance box of #241 what proves it
The folder holds SKILL.md, the reference pages, the scripts and their tests; outside it only AGENTS.md changes git diff --stat main...HEAD: 27 files under the folder, and AGENTS.md with 13 added lines
Standard library only; the tests pass with python3 -m unittest the run above; an import scan of every script and test finds the standard library and the skill's own modules, and nothing else
The same trials as a Harbor job and as a standard job give the same classes test_classify.BothRunPaths.test_the_same_columns_for_harbor_and_standard; test_classify.RetriedTries for a standard job with a retried/ folder
No script names an agent; read, classify and compare run for a made-up agent with no command and no limit test_classify.Classes.test_a_made_up_agent_with_no_limits_and_no_commands, and test_compare.Verdicts, whose adapter has no command and no limit; a search of the folder for agent and benchmark names finds none
A paid command is written to a file and not started, for a check and for a trial run test_check.CheckFive.test_a_paid_command_is_written_and_not_started, test_decide.Select.test_choosing_a_size_writes_the_run_and_starts_nothing; test_check.PaidScriptsRun runs a written command in paths that hold a space and a quote
Every budget is an argument with no default value test_read.BudgetHasNoDefault.test_read_needs_the_budget, test_decide.TrialBudgetHasNoDefault.test_a_size_needs_the_budget; test_read.BudgetIsAnAmount and test_decide.AmountsAreFinite refuse nan, inf and negative amounts
The agent's checkout is unchanged after a check, and no worktree is left test_check.Check.test_worktrees_are_removed_and_the_checkout_is_untouched, test_check.Check.test_a_failing_replay_still_removes_its_worktree; test_check.Cancellation stops the command and removes the worktree on SIGTERM and Ctrl+C
Noise at p of 0.05 or more, improved or regressed below it test_compare.Verdicts.test_eight_won_and_twelve_lost_is_noise, test_twelve_won_and_one_lost_is_improved, test_one_won_and_twelve_lost_is_regressed
read refuses a loop folder that git would list as untracked test_read.LoopFolderInARepository (three tests), test_read.LoopFolderWithATranslatedGit
A round on recorded runs of a real agent, and a round on a job of the standard path both below; on the standard path, stages 1, 2 and 5

The findings of the independent review, each fixed after a test that failed before the fix:

  • a trial run whose trials do not all report a cost needs the person's figure before any budget is computed from it
    (test_decide.IncompleteCosts, test_compare.IncompleteNewCost);
  • a trial job counts only when it holds the selected tasks with their trials and the person states the selected
    commit (test_decide.TrialRunIdentity);
  • a check that fails leaves no earlier success, and a changed adapter runs its commands again
    (test_check.StaleEvidence);
  • written paid commands run in paths with a space and a quote (test_check.PaidScriptsRun), a stopped loop stops its
    command (test_check.Cancellation), a found class cannot read outside its trial through a link
    (test_classify.FoundClassConfinement), amounts are finite (test_read.BudgetIsAnAmount,
    test_decide.AmountsAreFinite), and stop 1 shows the trials that keep their recorded label
    (test_decide.KeptRecorded);
  • a timed-out command is recorded as a check that did not finish, the stop rule is recorded when a round starts, ids
    and cause names stay inside the round (test_check.SmallItems, test_compare.StopRuleOfTheRound).

The three mutations the review ran now each fail their test: canaries replaced by an empty list
(test_decide.Select.test_canaries_come_from_a_fixed_seed), no time limit class
(test_classify.BothRunPaths.test_the_same_columns_for_harbor_and_standard), no cleanup in a written paid command
(test_check.PaidScriptsRun).

On recorded runs of a real agent, two Harbor jobs:

  • read and classify: the counts of trials, passes and pairs equal those of run-analysis, and the class of
    each trial equals a classification made by hand.
  • compare of the two runs: the won and lost tasks equal a recount by hand, and the verdict is noise.
  • With the agent's own adapter: candidate, check of two candidates, decide --stop 1 and the proposal of
    select. Each replay table and each count of changed labels equals a recount by hand. After each check the
    agent's repository lists only its own work trees.

On two real jobs of the standard path, an agent of the catalogue on the container backend, the first job the baseline:

  • read, classify with two found classes and compare ran, and their counts match a count by hand on the job
    folders: trials, passes, pairs, won and lost tasks, the class of the failing trial, and each found class.
  • The round found four gaps, each fixed here after a test that failed before the fix: model calls now come from the
    standard's own n_model_calls (test_classify.StandardModelCalls); output_limit is named as switched off when no
    trial records the tokens of a reply (test_classify.OutputLimitUnreadable); the task lists of a trial run hold the
    unit ids that --units of the standard takes (test_decide.StandardUnits); an adapter may have no source, and
    stages 1, 2 and 5 run without a checkout (test_read.NoSource).
  • At the head, read, classify and compare ran again on the same two jobs with an adapter that has no source:
    every trial's model calls equal its n_model_calls, rules_off names the unreadable output_limit, the unit id
    of a task is the one --units takes, and candidate refuses for want of a checkout.

Not verified:

  • stages 3 and 4 on a job of the standard path, and a retried/ folder on a real job; a test covers retried/;
  • a trial run with its comparison: select --size, a build of a real checkout, decide --stop 2 and compare of a
    trial run ran only on the made-up agent of the tests;
  • Python versions other than 3.12.

Risk

Door: two-way (reverts alone). A revert removes one folder and one paragraph of AGENTS.md; no other code imports
the skill.

  • read decides that a run is not analysable from the opening words of the verdicts that runs.py health prints.
    When run-analysis changes that wording, read accepts a run it should refuse, or refuses a healthy one, and
    test_read.ReadRefuses.test_a_systemic_failure_names_the_shared_error fails. classify and compare read the
    trials themselves and stop when their counts differ from those of runs.py.
  • check runs the adapter's commands with bash and makes detached worktrees of the agent's checkout. It removes each
    worktree when its check ends, on failure, on SIGTERM and on Ctrl+C. A loop killed with SIGKILL leaves the worktree
    under the loop folder; git worktree remove --force <path> in the checkout removes it.
  • The commit a trial run ran is the person's statement, since a job does not record its code. A wrong statement
    makes a trial run of other code count for the selected commit.

The adapter is the one file that holds everything about an agent: its
checkout, its commands with their placeholders, the limits of the run and
the commands that cost money. The loop checks each ref in a detached
worktree and removes it when the check ends.
read runs health and analyze of run-analysis, refuses a run that is not
analysable or belongs to another agent, and records the round's input.
classify gives each trial one of seven classes and a signature, groups
the failing trials by signature with their warnings, and adds columns
for found classes and for the agent's own classes.
compare gives the pass rates with their standard error, the won and lost
tasks with an exact two-sided binomial test, the targets' trials with
Fisher's exact test, and the stop rule, then closes the round. read asks
for the person's words before a round that follows a stop. Each stage
lives in its own module, and loop.py parses the command line.
candidate registers a fix of a cause and the trials it predicts. check
runs the cause's test before and after the fix, the agent's suite
against its failures on the base ref, and the replay on both refs, each
in a worktree that is removed when the check ends; a paid command is
written for a person to start. decide writes the pages of stop 1 and
stop 2, answer stores the person's words, and select prices the trial
run, builds it and writes its run command.
decide --stop 2 adds each trial job to the round's list in the ledger.
The money spent in select, decide and compare comes from that list, and
the exact test of the targets reads the trials of every trial run on the
same code.
Two exceptions that differ only in the model text they quote now share a
signature. The mask runs before the detail is cut, a quote that is not
closed masks to the end, and an apostrophe inside a word masks nothing.
Each group keeps the detail of its first trial as written in example.
The baseline warning names the benchmark only for groups of 3 tasks or
more.
… fixes

The made-up trials get their own costs, failed ones more than passing
ones, and the expected prices follow the price of each task. New tests
cover the cheapest group of the stop rule, a task without a cost, a
trial job named as PATH::JOB, and a result that cannot be written.
A run of chosen tasks costs the sum over its tasks of the mean recorded
cost of each task's trials, times the trials it gets. The cost comes from
this round's trial runs on the same commit when they hold the task, else
from the run that was read; compare prices from the new run. A task with
no cost takes the mean of the others, and the line counts such tasks.
select, decide --stop 2 and the stop rule share one function, and the
stop rule names its lowest-priced group.

decide --stop 2 reads the job folder that health found, so PATH::JOB
works. compare writes result.json before the ledger entry. stats.py
loses standard_error, which had no caller.
read asks git, before it writes anything, whether the loop folder lies in
a work tree and whether that repository ignores it. A folder outside
every repository, or an ignored one, is read; any other is refused with
the repository root and the two ways out. Any other answer of git stops
read with git's output.
…name a folder or a file

check and answer take a candidate id only in the form candidate gives
it, and candidate takes a cause only as the name of a file in causes/, so
neither can lead outside the round.
…only that rule

read writes the text of the stop rule into input.json; compare refuses a
round whose recorded rule is not the rule of the skill it runs, so a rule
changed by an update of the skill never decides a round that began under
another.
…unt when its result holds one

A job of the standard writes agent_result.n_model_calls, and its ATIF
trajectory has one step per agent message with no usage per step, so the
steps overcount or undercount the calls. step_limit now reads that count
when result.json holds it, and the rule of run-analysis otherwise.
…rds the tokens of a reply

A job of the standard records no tokens per reply in the files every path
writes. groups.json then lists output_limit under rules_off with that
reason, instead of keeping a rule that can never hold.
… a job of the standard

A trial of the standard names its task as <benchmark>/<unit> and its unit
in standard.unit, and --units of the standard takes the unit. When every
trial of the run has a unit, trial-tasks.txt, rerun-tasks.txt and
full-tasks.txt hold the units; a Harbor run keeps its task names.
…of its own

source in adapter.json may be null. read, classify and compare never open
the agent's code and run without it; candidate, check, select and
decide --stop 2 refuse with a message that names source. An adapter with
no source refuses every command that runs on the code; commands.classes
then runs in the loop folder.
…gent with no checkout, and what a round counts

classes.md: model calls from n_model_calls when result.json holds it,
output_limit switched off when no trial records a reply's tokens, and
found classes counted on the round's run. adapter.md: a null source and
unit ids in the units file. files.md: a round's own run is not money it
spent.
…e amount of 0 or more

A trial's cost_usd, the budget in input.json and the cost of each trial
run in ledger.json are read only as finite amounts of 0 or more; any other
value stops the loop with the file, the trial and the value named, before
a ledger entry or a command is written.
…in the answer and in select

checks.json records the SHA-256 of the adapter it was made with. decide
--stop 1, answer --stop 1 and select read a check only under that same
adapter and otherwise ask for loop.py check again.
…ts count over the round's trial jobs

decide --stop 2 refuses a job that lacks a guard or a canary, and a job whose
tasks have unequal numbers of trials. It counts the trials per task over the
round's trial jobs on the selected commit, so a runner that gives one trial
per job is started three times. trial.json and the page say how many trials
per task the jobs hold together and whether the selection is complete; the
targets are tested only when it is. The rerun option runs every selected task.
… a task with none or with two

run_names gives each task the one unit id its trials give on a job of the
standard. A task whose trials give no unit id while other trials of the job
give one, or two unit ids, is refused before trial-tasks.txt, the ledger or a
command is written. A job of a Harbor recipe keeps its task names.
…tial count of output tokens, and the checkout first at stop 2
…heck the checkout first at stop 2, describe model_calls

rules_off says how many trials record no tokens of one reply when the
adapter sets an output limit. decide --stop 2 checks for the agent's
checkout before it reads the trial job. classes.md and files.md say that
model_calls in trials.csv is the standard's own count where a result holds
one, and that the run-analysis page counts from the trajectory.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skills: take a finished run from its failures to checked fixes and a compared rerun

1 participant