Repository navigation
Conversation
The adapter is the one file that holds everything about an agent: its checkout, its commands with their placeholders, the limits of the run and the commands that cost money. The loop checks each ref in a detached worktree and removes it when the check ends.
read runs health and analyze of run-analysis, refuses a run that is not analysable or belongs to another agent, and records the round's input. classify gives each trial one of seven classes and a signature, groups the failing trials by signature with their warnings, and adds columns for found classes and for the agent's own classes.
compare gives the pass rates with their standard error, the won and lost tasks with an exact two-sided binomial test, the targets' trials with Fisher's exact test, and the stop rule, then closes the round. read asks for the person's words before a round that follows a stop. Each stage lives in its own module, and loop.py parses the command line.
candidate registers a fix of a cause and the trials it predicts. check runs the cause's test before and after the fix, the agent's suite against its failures on the base ref, and the replay on both refs, each in a worktree that is removed when the check ends; a paid command is written for a person to start. decide writes the pages of stop 1 and stop 2, answer stores the person's words, and select prices the trial run, builds it and writes its run command.
decide --stop 2 adds each trial job to the round's list in the ledger. The money spent in select, decide and compare comes from that list, and the exact test of the targets reads the trials of every trial run on the same code.
Two exceptions that differ only in the model text they quote now share a signature. The mask runs before the detail is cut, a quote that is not closed masks to the end, and an apostrophe inside a word masks nothing. Each group keeps the detail of its first trial as written in example. The baseline warning names the benchmark only for groups of 3 tasks or more.
… fixes The made-up trials get their own costs, failed ones more than passing ones, and the expected prices follow the price of each task. New tests cover the cheapest group of the stop rule, a task without a cost, a trial job named as PATH::JOB, and a result that cannot be written.
A run of chosen tasks costs the sum over its tasks of the mean recorded cost of each task's trials, times the trials it gets. The cost comes from this round's trial runs on the same commit when they hold the task, else from the run that was read; compare prices from the new run. A task with no cost takes the mean of the others, and the line counts such tasks. select, decide --stop 2 and the stop rule share one function, and the stop rule names its lowest-priced group. decide --stop 2 reads the job folder that health found, so PATH::JOB works. compare writes result.json before the ledger entry. stats.py loses standard_error, which had no caller.
…ds, and braces in commands
read asks git, before it writes anything, whether the loop folder lies in a work tree and whether that repository ignores it. A folder outside every repository, or an ignored one, is read; any other is refused with the repository root and the two ways out. Any other answer of git stops read with git's output.
…name a folder or a file check and answer take a candidate id only in the form candidate gives it, and candidate takes a cause only as the name of a file in causes/, so neither can lead outside the round.
… count of tasks to pass
…only that rule read writes the text of the stop rule into input.json; compare refuses a round whose recorded rule is not the rule of the skill it runs, so a rule changed by an update of the skill never decides a round that began under another.
…, a stopped loop and the recorded stop rule
…rial run in plain sentences
…ount when its result holds one
…unt when its result holds one A job of the standard writes agent_result.n_model_calls, and its ATIF trajectory has one step per agent message with no usage per step, so the steps overcount or undercount the calls. step_limit now reads that count when result.json holds it, and the rule of run-analysis otherwise.
…ords the tokens of a reply
…rds the tokens of a reply A job of the standard records no tokens per reply in the files every path writes. groups.json then lists output_limit under rules_off with that reason, instead of keeping a rule that can never hold.
…n a job of the standard
… a job of the standard A trial of the standard names its task as <benchmark>/<unit> and its unit in standard.unit, and --units of the standard takes the unit. When every trial of the run has a unit, trial-tasks.txt, rerun-tasks.txt and full-tasks.txt hold the units; a Harbor run keeps its task names.
…of its own source in adapter.json may be null. read, classify and compare never open the agent's code and run without it; candidate, check, select and decide --stop 2 refuse with a message that names source. An adapter with no source refuses every command that runs on the code; commands.classes then runs in the loop folder.
…gent with no checkout, and what a round counts classes.md: model calls from n_model_calls when result.json holds it, output_limit switched off when no trial records a reply's tokens, and found classes counted on the round's run. adapter.md: a null source and unit ids in the units file. files.md: a round's own run is not money it spent.
…te amount of 0 or more
…e amount of 0 or more A trial's cost_usd, the budget in input.json and the cost of each trial run in ledger.json are read only as finite amounts of 0 or more; any other value stops the loop with the file, the trial and the value named, before a ledger entry or a command is written.
… in the answer and in select
…in the answer and in select checks.json records the SHA-256 of the adapter it was made with. decide --stop 1, answer --stop 1 and select read a check only under that same adapter and otherwise ask for loop.py check again.
…ats count over the round's trial jobs
…ts count over the round's trial jobs decide --stop 2 refuses a job that lacks a guard or a canary, and a job whose tasks have unequal numbers of trials. It counts the trials per task over the round's trial jobs on the selected commit, so a runner that gives one trial per job is started three times. trial.json and the page say how many trials per task the jobs hold together and whether the selection is complete; the targets are tested only when it is. The rerun option runs every selected task.
…unit ids on a job of the standard
… a task with none or with two run_names gives each task the one unit id its trials give on a job of the standard. A task whose trials give no unit id while other trials of the job give one, or two unit ids, is refused before trial-tasks.txt, the ledger or a command is written. A job of a Harbor recipe keeps its task names.
…tial count of output tokens, and the checkout first at stop 2
…new one for a round of another stop rule
…heck the checkout first at stop 2, describe model_calls rules_off says how many trials record no tokens of one reply when the adapter sets an output limit. decide --stop 2 checks for the agent's checkout before it reads the trial job. classes.md and files.md say that model_calls in trials.csv is the standard's own count where a result holds one, and that the run-analysis page counts from the trajectory.
…el_calls note without a trailing so
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #241
What
The skill
run-to-fixunder.claude/skills/run-to-fix/takes a finished run to checked fix candidates and acompared rerun. A round has five stages (read, classify, check, select, compare) and stops twice for a person: which
candidates to keep, and what runs next at what price. The script
scripts/loop.pycounts and keeps the records in aloop folder the person names. The assistant reads trials, names causes and writes tests and fixes in the agent's
code. Everything that belongs to one agent lives in one adapter file (
references/adapter.md). The skill callsruns.pyofrun-analysisfor health, metrics and the round page, and imports nothing fromapp/.AGENTS.mdgets one paragraph on the skill.
Review order:
SKILL.md, thenreferences/adapter.md,scripts/loop.py, and the stage modules in stage order(
reading.py,classifying.py,checking.py,selecting.py,deciding.py,comparing.py). Feedback wanted on twochoices: the adapter contract, and the price of a trial run. A price counts each task at the recorded cost of its own
trials.
Why one PR: the change is one new folder that no other code imports, and the author asks the reviewer to agree to one pull request.
Size:
verdict: SPLIT: code lines 2356 > hard 500; split the branch or add a Why one PR line from the allowlist(support lines 3179, 28 files in 2 areas).
Touches
CI and skills: the new folder
.claude/skills/run-to-fix/and one paragraph ofAGENTS.md. No file of theapplication changes.
Verified
Merge base
0e6ae8cff68380486bb9bca10261e83f40a5bf54(main), headf4291c6d80281420df2d44d55cf8c9f7b90064d0.python3 -m unittest discover -s .claude/skills/run-to-fix/testsfrom the repository root: 154 tests, OK, onPython 3.12.3. Each change of code was committed after its test, and that test failed on the commit before it.
SKILL.md, the reference pages, the scripts and their tests; outside it onlyAGENTS.mdchangesgit diff --stat main...HEAD: 27 files under the folder, andAGENTS.mdwith 13 added linespython3 -m unittesttest_classify.BothRunPaths.test_the_same_columns_for_harbor_and_standard;test_classify.RetriedTriesfor a standard job with aretried/foldertest_classify.Classes.test_a_made_up_agent_with_no_limits_and_no_commands, andtest_compare.Verdicts, whose adapter has no command and no limit; a search of the folder for agent and benchmark names finds nonetest_check.CheckFive.test_a_paid_command_is_written_and_not_started,test_decide.Select.test_choosing_a_size_writes_the_run_and_starts_nothing;test_check.PaidScriptsRunruns a written command in paths that hold a space and a quotetest_read.BudgetHasNoDefault.test_read_needs_the_budget,test_decide.TrialBudgetHasNoDefault.test_a_size_needs_the_budget;test_read.BudgetIsAnAmountandtest_decide.AmountsAreFiniterefuse nan, inf and negative amountstest_check.Check.test_worktrees_are_removed_and_the_checkout_is_untouched,test_check.Check.test_a_failing_replay_still_removes_its_worktree;test_check.Cancellationstops the command and removes the worktree on SIGTERM and Ctrl+Ctest_compare.Verdicts.test_eight_won_and_twelve_lost_is_noise,test_twelve_won_and_one_lost_is_improved,test_one_won_and_twelve_lost_is_regressedreadrefuses a loop folder that git would list as untrackedtest_read.LoopFolderInARepository(three tests),test_read.LoopFolderWithATranslatedGitThe findings of the independent review, each fixed after a test that failed before the fix:
(
test_decide.IncompleteCosts,test_compare.IncompleteNewCost);commit (
test_decide.TrialRunIdentity);(
test_check.StaleEvidence);test_check.PaidScriptsRun), a stopped loop stops itscommand (
test_check.Cancellation), a found class cannot read outside its trial through a link(
test_classify.FoundClassConfinement), amounts are finite (test_read.BudgetIsAnAmount,test_decide.AmountsAreFinite), and stop 1 shows the trials that keep their recorded label(
test_decide.KeptRecorded);and cause names stay inside the round (
test_check.SmallItems,test_compare.StopRuleOfTheRound).The three mutations the review ran now each fail their test: canaries replaced by an empty list
(
test_decide.Select.test_canaries_come_from_a_fixed_seed), no time limit class(
test_classify.BothRunPaths.test_the_same_columns_for_harbor_and_standard), no cleanup in a written paid command(
test_check.PaidScriptsRun).On recorded runs of a real agent, two Harbor jobs:
readandclassify: the counts of trials, passes and pairs equal those ofrun-analysis, and the class ofeach trial equals a classification made by hand.
compareof the two runs: the won and lost tasks equal a recount by hand, and the verdict is noise.candidate,checkof two candidates,decide --stop 1and the proposal ofselect. Each replay table and each count of changed labels equals a recount by hand. After each check theagent's repository lists only its own work trees.
On two real jobs of the standard path, an agent of the catalogue on the container backend, the first job the baseline:
read,classifywith two found classes andcompareran, and their counts match a count by hand on the jobfolders: trials, passes, pairs, won and lost tasks, the class of the failing trial, and each found class.
standard's own
n_model_calls(test_classify.StandardModelCalls);output_limitis named as switched off when notrial records the tokens of a reply (
test_classify.OutputLimitUnreadable); the task lists of a trial run hold theunit ids that
--unitsof the standard takes (test_decide.StandardUnits); an adapter may have nosource, andstages 1, 2 and 5 run without a checkout (
test_read.NoSource).read,classifyandcompareran again on the same two jobs with an adapter that has nosource:every trial's model calls equal its
n_model_calls,rules_offnames the unreadableoutput_limit, the unit idof a task is the one
--unitstakes, andcandidaterefuses for want of a checkout.Not verified:
retried/folder on a real job; a test coversretried/;select --size, a build of a real checkout,decide --stop 2andcompareof atrial run ran only on the made-up agent of the tests;
Risk
Door: two-way (reverts alone). A revert removes one folder and one paragraph of
AGENTS.md; no other code importsthe skill.
readdecides that a run is not analysable from the opening words of the verdicts thatruns.py healthprints.When
run-analysischanges that wording,readaccepts a run it should refuse, or refuses a healthy one, andtest_read.ReadRefuses.test_a_systemic_failure_names_the_shared_errorfails.classifyandcompareread thetrials themselves and stop when their counts differ from those of
runs.py.checkruns the adapter's commands with bash and makes detached worktrees of the agent's checkout. It removes eachworktree when its check ends, on failure, on SIGTERM and on Ctrl+C. A loop killed with SIGKILL leaves the worktree
under the loop folder;
git worktree remove --force <path>in the checkout removes it.makes a trial run of other code count for the selected commit.