Repository navigation
Conversation
scripts/household_set.py samples households from the dataset the installed policyengine.py bundle certifies, renames them mc_000 onward, computes every output on the installed policyengine-us with the reference conventions, and writes a board in the frozen layout with batches. On the frozen board's own households it reproduces all 1,935 scored 2.41.0 references exactly. The tool pilot takes --board: the engine and fix pins come from the board's last engine_upgrade revision, and a household set's arms are scored against the same model's no-tools run (--baseline-dir). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
scripts/batch_judge.py writes one arm's answers on a batch as a dashboard payload (each model a board model), so consensus-flags and the two-stage adversary run unchanged: two of three models agreeing off the reference flag a cell. The payload carries what the models saw: the v2-pilot household prompt (checked against each record's sent prompt hash), the v2-pilot definitions, and the condition they answered under. Derivations are pruned engine traces on the household set's own reference system. The adversary takes a payload's outputDefinitions and answerCondition; the default renders both stage headers byte for byte as before. Dry run on today's frozen-board no-tools answers (v2 wording): 23 of 1,935 scored cells flagged, 12 of them among the 2026-10-05 adversary's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-22: "in case youre running any claude api stuff please use the key on my clipboard". With JUDGE_API_KEY_SECRET naming an agent-secret entry, the judges bill that key and nothing else: it refuses a lane config, a lane token, AUDIT_ACCOUNT or the desktop opt-in alongside it; every call runs in a fresh empty config directory removed at exit; the key joins only the claude calls' allowlisted environment; and check-login requires a first-party api_key login from ANTHROPIC_API_KEY in that directory. Sidecars record the mode and the secret's name, never the key. The lane and desktop paths are unchanged. Why now: the desktop login's CLI session had expired, and the Subfleet lanes were closed, so the batch judges had no login to run on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The adversary prompt tells a judge to exclude policybench.org, policyengine.org and GitHub on every search. Codex's search tool takes no blocked_domains, so GPT-6.1 Sol did it with -site: operators, and the audit rejected those stage-1 outputs as searches naming PolicyEngine (rehearsal, 2026-10-22: 2 of 3). searched_for() drops a -site: exclusion of a blocked domain before the contamination check, in both runners' audits. A query that still names the engine or the benchmark (beside exclusions, under site:, -inurl:, or a look-alike domain) is still rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-23: "we can still test before that but i'd like v2 to be clean". household_set.py build takes --purpose v2 (the default) or test. A v2 set needs the installed policyengine.py bundle to certify its own dataset for the installed engine; anything else is refused unless --purpose test. A test set may name a published dataset (--dataset-repo/--dataset-file/ --dataset-revision); it records the release manifest's build engine and compatibility claim, purpose test and certified_pair false. Engines after policyengine-us 2.2.1 ignore Populace's filing_status_input and derive filing status from the household; the build records each sampled household whose stated status differs from the engine's (its scenario and reference carry the engine's). The sampler takes the dataset as an option; the default call is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-23, on what test batch 0 showed about the calculator ("yes go"):
- Names: a name written with spaces (as the prompt labels its facts) or a
benchmark output key (free_school_meals_eligible, head_medicaid_eligible)
is read as its engine variable, and the result's notes say how. Every model
had a call refused for a guessed name in all 10 households.
- Placement: an input set on another entity moves to the one unit that holds
the people it was set for (a group left out becomes one unit of everyone, as
the engine reads it); otherwise it stays an error.
- Overrides: a contribution set as the realized amount is read as its
_desired input (Sol and Opus put "traditional 401k contributions desired"
into traditional_401k_contributions in 6 households, moving tax), and an
input the engine computes and no stated fact sets is flagged as replacing
its formula (Sol set rent, not pre_subsidy_rent).
- Explanations: explain (up to 3 variables) returns the computation steps and
the parameters read, with values and cited sources. Opus overrode a correct
free-meals answer in Louisiana, which pays the reduced-price copay.
Tool version 1 (pe, web+pe) is byte for byte as before. The notices and tool
description now state the board's engine (a household set on 2.60.4 had been
told 2.41.0); the frozen board's text is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-23: "thinking we might want to do all these tests with a cheap model until we're super ready for v2 with the web and tool experiments". batch_judge.py payload takes --models (the frontier three by default). The payload's household prompt is the first tool-contract model's, checked against that model's records, and the payload records which models it holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The calculator's six features (names, placement, overrides, explain, concise, flat) are each on or off, so a study can change one at a time. "pe2" keeps tool version 2 (the first four, its notice byte for byte), "pe3" offers all six, and any other set runs as "pe:" plus its features, e.g. "pe:names+flat". - concise: an explanation gives the value, its direct inputs and up to five parameters unless `detail` is "full". - flat: the household is described as a state, people with a role (head, spouse or dependent) and facts, and household facts; the tool builds the units the benchmark's own situations have and places each household fact on its entity. Tests: the v2 notice is reproduced from its features; each feature adds only its own behavior; flat builds the benchmark's units and refuses what it cannot place; and on the installed engine eight frozen-board households described flat compute every board output exactly as Scenario.to_pe_household does. Five mutants of the flat and arm logic are all caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prompts label some facts in words rather than as the variable name with spaces: employment_income is "gross wages and salaries", hours_worked_last_week "usual weekly hours worked", has_esi "has employer-sponsored insurance" (policybench.prompts.INPUT_LABEL_OVERRIDES). In the 20-household test set nearly every household has at least one; hours appear in 17, employer coverage and premiums in 11 each. Tool version 2's names feature cannot read these, and its notice says the facts carry the variable names. A Flash-Lite smoke test of the flat format met three such errors and fell back to setting employer_sponsored_insurance_premiums to 0 and weekly_hours_worked, which the SNAP work rules do not read. The new labels feature reads a fact under its prompt label (case and punctuation aside, renames applied) and, as the reference situations do, also sets stated usual weekly hours as weekly_hours_worked_before_lsr (PE_INPUT_ALIASES). pe3 now offers all seven features. pe2 is unchanged and keeps its notice as it ran; every other feature set's notice says only what the tool reads. Tests: label reading, the alias (and leaving a set value alone), pe2 unchanged, the variable-name claim only in pe2's notice across every feature set, every prompt label reading back to its engine name, and a differential: twelve frozen-board households described flat with each fact as its prompt writes it compute every board output as the benchmark does. Three mutants (no alias, no renames, strict label matching) are caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The v2-pilot prompt relabels two facts itself ("home energy assistance
received" for spm_unit_energy_subsidy, "employer-paid health insurance
premiums ..." for employer_sponsored_insurance_premiums); the labels
feature now reads them, derived from V2_PILOT_CONVENTIONS so the two
stay in step. In the flat format a person given no name is named by
role (head, spouse, dependent_1, ...): Flash-Lite's first flat call in
two smoke households left the names out.
The label differential now writes each fact as the v2-pilot prompt
does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two of three Flash-Lite smoke households gave the head's rent as a household fact. pre_subsidy_rent is a person variable in policyengine-us, so the flat format refused it. Under the placement feature a person fact given for a household of one now goes to that person, as placement moves a person variable off a one-person unit; with two or more people it is still refused, now saying to give it under the person whose facts list it. The test registry's entities now match the engine's (pre_subsidy_rent is a person variable). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A differential against 5bbd571 (the commit the test batches ran) found version 2 requests carrying detail "full"; only execute_request reads it, through .get, so results were unchanged, but the request now matches exactly. 4,000 generated calls through versions 1 and 2, the notices, descriptions, schemas and 15 model profiles are identical to 5bbd571. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
11 of 12 tasks
GPT-6.1 Sol's review (REQUEST_CHANGES, 7 findings) at 9c43f4f: 1. A model's arguments could turn on features its arm does not offer: the worker read "__tool_features__" (and, as at 5bbd571, "__tool_version__") from the arguments dict. A pool now sends features in a tuple frame no JSON can be, the worker validates them (names, concise beside explain), and keys like "__tool_version__" in a model's arguments are refused as unknown arguments. No run record ever carried them. 2. A resume could mix treatments. Each household record now keeps its tool features and calculator schema hash; a resume compares features, notice and schema, and refuses a non-legacy record without them. 3. Labels missed what the prompt renders. The label index is now built from the prompt's own renderer (_format_input_line, then v2_pilot_text) for every promptable, labeled and reference-stated input: fallback labels, the Marketplace plan line, "state", partnership or S-corp income. Facts the reference drops (PE_DROPPED_INPUTS) are ignored with a note. A label that names two of the engine's inputs is not read. 4. Flat refuses person keys other than name, role and facts. 5. Flat stores household facts under the name written, so normalization compares them once normalized; roles set the relationship inputs and a stated one that contradicts its role is refused. 6. The flat example uses employment_income; placement replaces the wrong-entity error sentence; "full" detail is the trace, capped. 7. Diagnostics read flat people by their default names. Evidence: 210 passed, 4 frozen-engine skips. A labels-only differential builds 30 frozen households from their prompts' facts and matches the reference situation on every output, and so do all 100 frozen and all 20 test-set households in a one-off run, with or without take-up inputs. pe and pe2 are identical to 5bbd571 over 4,000 calls and 2,000 pool messages. 11 mutants of the fixes are caught. An end-to-end Flash-Lite pe3 smoke run had 0 tool errors on 3 households. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- flat: a household fact repeated with a malformed value was dropped without a word. _same_value now reports the repeat's validation problems, and only a repeat validated as equal is dropped quietly. A malformed first value is reported once, by normal validation, and never as a conflict. - The fallback record for a household the harness failed on carried no treatment, so one failure blocked resuming any custom arm. The record is built by harness_error_record, which now includes the notice hash and profile_treatment. resume_treatment_problems checks only the records a resume keeps, since a rerunnable record is replaced under the current profile. pe, web+pe and pe2 still behave exactly as at 5bbd571: 4,000 calls and 2,000 pool messages match. 5 mutants of these fixes were all caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…core A rerun is not sure to replace a rerunnable record: a budget stop, lost workers or a narrower --scenarios leave it in place, and scoring read whatever was there. treatment_problems now checks every record in the folder, rerunnable ones included, and command_score refuses a folder that mixes treatments, rebuilding the notices on the board's engine. Runs written before 6e1578f (test batch 0's frontier calculator arms, b6d835a) were told policyengine-us 2.41.0 while computing on 2.60.4. A folder written wholly so is one treatment and scores with a note; a mix of the two texts is refused, and a resume into it stays refused, since its new records would state the board's engine. All 39 run folders score (628 records); the six frontier calculator folders take the note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) Reviewed head 6b00a6f (GPT-6.1 Sol r4 AGREE); full suite at that head: 218 passed, 4 skipped, ruff clean.
… ceiling scripts/confirmatory_run.py runs the frozen pre-registration's run procedure on the treatment harness (6b00a6f's tool_pilot_argon.py, unchanged) and writes the run records the confirmatory report checks: - two lanes per model, households alternating in draw order, each household's arms in its drawn order, then reruns in passes; - one ToolPool per model with no features of its own, each call framed with its arm's features; - admission against each model's cap and reserve, waiting while a pass's reruns are undecided; - the hard ceiling over every model's spend, through the harness's Spend; - the failure classes and actions, with operator pauses, harness and inconclusive stops, and replaced attempts saved before their reruns; - the 30-second host log and the guard's and power log's lines. Tests follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
30 of 33 tasks
Unit tests for the streams, arm strings and features, the classes and actions, the host evidence and the ceiling. Fake runs (a fake provider, pool and host log, on a fake board, with the windows scaled down) plant reruns, host trouble, an operator pause, a harness stop, the inconclusive stop, a refusal, the hard ceiling, a raising attempt, pool restarts and an interrupt. run_problems recomputes every rule of the run procedure from the files the runner wrote, and a property test checks it over random failures and spend. Two tests run the harness's own conversation under the runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mutant check of 274e1c7 found one fault that the tests missed. Its digest read in Stage.archive used the blocking sha256_file. The new case replaces a prepared final with a FIFO after startup. The archive move must refuse it at once by name, within the 5 s bound, and leave no attempts folder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he abandon suites-r12's first runner pass at 0193811 failed test_an_attempt_past_its_deadline_is_abandoned_and_stops_the_model with 0.02 == 0.01, and the next two passes held 730 passed. The test gave each turn 0.2 s against a 0.3 s deadline, so its second turn spent at 0.4 s, and it expected the waiting thread to abandon the attempt inside that 0.1 s. A wake 0.1 s late put the second turn's cost in the attempt's record, which is the spend so far that the runner's rule asks for. The runner was right and the test's expectation assumed a wake time. A plugin that delays the abandon by 0.25 s reproduces the same assertion at the same line. The first turn now spends at once and the second waits for the abandon itself, through a new `hold` key in FakeProvider. The deadline rises to 1 s and the other model's attempts to 0.2 s, which leaves 0.8 s of room against that model reaching its own deadline. The assertions are unchanged. The test passed 25 of 25 runs alone and 25 of 25 with the 0.25 s delay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eline rules Answers review runner-r18. stage_problems dropped the cost history returned and never compared it with spent_before, and the preflight never read the baseline. Calculator spending could therefore start from an understated ceiling, or on a baseline the report refuses, and the report would then withhold the results. stage_problems now sums every model's validated cost and refuses a spent_before below that sum by more than MONEY_TOLERANCE, the report's value. It refuses an unusable value by the report's money rule: finite and nonnegative, with numeric text and booleans counting as numbers there too. check_baseline is the startup check moved unchanged, so startup and the preflight share one rule. Tests: a full preflight against spend in a final, an archive and an optional model; each unusable spent_before; and each baseline fault. The second hash after a rename gets its own FIFO case, because the earlier case attacks the first hash. Two execution cases fail the second hash and the move back, then check that the other models finish, the lock is released and startup recovers the archive. The deadline test now records the join's timeout: its hold on the second turn hides when the abandon came, so a half deadline passed it. The ceiling test waits for the crossing where it slept 0.3 s against an unbounded competitor. The archive docstring now limits later digest checks to surviving links: after an interrupt, startup rebuilds the last archive's digest from the bytes then present. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers review runner-r19 and item 3 of confirmatory-report-r18. The preflight added each cost to a running total, and the report grouped its total by model, so the two programs rounded differently. A spent_before at the tolerance passed one and failed the other, and $2 of saved spend could round away beside a $1e16 cost. history now lists each saved cost, and exact_spend takes their correctly rounded sum, the same in any order and grouping. The report takes the same sum. An intermediate overflow means a total past every float, which the comparison refuses. Startup's spend uses the same sum. Tests: every order of two sets of costs at the tolerance, costs a running total rounds away, and a total past every float; a spent_before given as numeric text or a boolean, which counts as its number in both programs; and a design without spent_before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review runner-r20 found that restoring a running total at startup alone would pass every test. A new case saves costs whose running total stays under the cap and whose exact sum reaches it, and asserts that no owed attempt starts and that the invocation lock comes free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s release Review runner-r21 found two gaps in the startup test. Rounding each model's costs first escaped it, so a second test saves costs whose subtotals fall one float under the cap and whose exact sum reaches it. The test's lock assertion read bookkeeping, so both tests now take the lock from a second open. `exact_spend`'s docstring limits the claim to a flat list of costs, in any order, and to histories that pass validation. Production code changes only in that docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review runner-r22 found that startup's overflow rule had no test. Replacing `exact_spend(costs)` with `math.fsum(costs)` at startup alone would have raised a raw OverflowError and passed every test, because the existing overflow cases reach the preflight or refuse one invalid cost. A new test saves two valid 1.5e308 finals and requires that startup takes infinity for the total, stops every owed attempt and frees the lock. The helper takes the expected total as an argument. Production code doesn't change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's test job failed 22 confirmatory tests on Linux at 3846666; they had passed on the Mac that ran every earlier suite. Fourteen reached a preflight check that compares the installed policyengine-us with the board's engine, and the lockfile installs 2.41.0 where the fixture board names 2.60.4. Eight reached host_power, which calls macOS's pmset. World already fakes the board's engine, so it now fakes the installed version and the power reading too. Its tests no longer read either from the host. A test that wants a different reading still sets it after World, as the lid-closed test does. The eight stubs in the unreadable-evidence tests and the ones in pool_design and engine_fault repeat World's and go. Production code does not change. A pytest plugin that hides pmset and reports 2.41.0 reproduced CI's 22 failures on this Mac at 3846666, by name. With this commit all 772 runner tests pass under that plugin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
World now reports the installed engine and the power reading, so no test could see the checks that read them, and no earlier test had held those checks with the real values either. An installed engine other than the board's must now refuse in the preflight twice, in its own check and through the no-tools treatment check, and again in main's final check, which must keep the preflight's problems. Each test requires that no provider is called. host_power gets its first test, on the four mixes of power source and lid state, with the text pmset and ioreg print on this Mac. Production code does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max ruled all four on 2026-11-01. Every wording change stays in the v2-pilot rewrites, so the board's v1 prompts do not move. - d1970: the prompt says everyone is a citizen with an SSN unless stated otherwise, and references state both statuses (the engine's defaults). - d1971: the federal credit definition takes the EITC from 26 USC 32's formula, and the two state definitions it reaches say the same. - d1972: household sets round inputs to the dollars the prompt prints before computing references; every amount gets a period; partnership income, educational assistance and capital gain distributions get a tax character; Louisiana's standard deduction is stated outright. - d1029: the four income tax definitions cover the head and spouse's return, and household sets drop households where a dependent has return income. Session call (judge triage Pattern B): the prompt states that rental, farm rent, partnership and S corporation income count as qualified business income, as the engine computes them. Flags set false would part the references from what a calculator returns on the same facts. The partnership label departs from the ruled text where the data contradicts it: two pilot people carry a self-employment line that the engine documents as part of partnership income. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`max` over a date with `key=str` raised TypeError before the assertion ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The canonical-amounts property named its adult "head", so the prompt took the path it uses for children: it lists wages only when they are nonzero. CI's Hypothesis run found a $0.50 wage, which the original prompt prints as "$0" and the canonical scenario, holding 0, leaves out. Adults are named adult1 and adult2, and their wage line always prints, so the exact-text property now holds for an adult1. A child1 with its own wage covers the other path: when the original prompt prints the child's wage as "$0", the canonical prompt drops that line, as intended, since the prompt treats an unlisted amount as 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
GitHub runs #249's CI through draft #250 into main, so it tests this stack merged with main at f076d7c. Run 41354822288 failed four tests before GitHub cancelled it at the 30-minute limit, at 97%. One came from #249, and 2b233b7 fixes it. The other three come from main's move of the board to policyengine-us 2.60.0, and we need to fix them here before this PR lands:
We take main in with a merge commit after the confirmatory run, since a rebase would drop the SHAs the run pins. With main merged, the suite grows to 7,752 tests and outruns ci.yml's 30-minute test limit; main alone takes 17 to 22 minutes. The README at |
Review r1 of #249 found the d1029 filter's list missed four inputs that policyengine-us 2.60.4 counts toward a dependent's gross income: disability_benefits, debt_relief, illicit_income and military_retirement_pay_survivors. The list adds them, and the build now asks the reference engine which inputs raise dependent_gross_income and refuses one the list leaves out. The filter reads each amount as the prompt prints it, so a wage that prints as $0 keeps its household, as its reference computes from $0. Tax-exempt interest counts, since Pennsylvania taxes other states' bond interest and a Pennsylvania dependent files above $33. Social Security counts once a dependent's benefits pass $50,000, where half of them passes 26 U.S.C. 86's $25,000 base amount. The QBI test now checks the prompt's sentence against the engine both ways. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and by losses Sol review r2 found two gaps in d1029's filter. Inputs outside the list can produce gross income together. On policyengine-us 2.60.4 and 2.90.4, an Oklahoma dependent's base-period wages and weeks unemployed give $6,944 of modeled unemployment insurance. The frame now carries the engine's dependent_gross_income, and the filter drops any household where it isn't $0 for a dependent. build drops its one-input probe and instead reads each drawn household's dependent gross income on the reference engine, refusing a draw that kept any. Pennsylvania requires a return from anyone, minors included, with a loss from any transaction as an individual, sole proprietor, partner or S corporation shareholder (2025 PA-40 instructions, page 3). So a listed loss, as the prompt prints it, drops the household too. A missing listed amount reads as $0, since the prompt prints nothing for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A listed column that no promptable input fills (partnership_s_corp_income) stays missing for households without it. The new test checks that the d1029 filter reads it as the prompt does, as nothing, so a mutant that counts missing amounts now fails. The frame-loader test now stubs the promptable-input lookup too, so it passes or fails on the engine count alone, in any order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rules Sol review r2 had it right: the $33 and loss rules sit on page 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… r3) Review r3 found that a dependent's partnership self-employment earnings pass the filter. They aren't gross income, but net earnings of $400 or more require a return (26 U.S.C. 6017), and policyengine-us counts them as the dependent's taxable self-employment income. - List partnership_self_employment_net_earnings, and each part of a listed income that the prompt prints on a line of its own: capital gain distributions, the overtime premium and eleven more. A record that holds a part without the whole now drops. - The build check reads the reference engine's taxable self-employment income as well as its dependent gross income. - A probe test gives a dependent each promptable input in turn and checks that the list holds every input the engine counts. - New tests cover a spouse's income, Social Security across kinds, a missing engine count and each income part by name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine gives each Alaska resident the Permanent Fund Dividend as gross income with no input ($1,000 in 2026), and each prompt says to treat an unlisted amount as 0. d1029's filter already drops every Alaska household with a dependent; this drops the rest, whose references move on income no prompt states. A build stops if its draw holds a state where the engine gives an adult or a child gross income with no input, and a test checks the list against all 51 states. The build-check test gains a spouse, so a check that reads the head alone fails (review r4, finding 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r5 approved 1c8c645 and found five behaviours no test held: - Both of v2's filters keep the weights of the households they leave in. test_the_filters_keep_the_household_weights draws 200 times beside two dropped households of weight 1e6, and a Hypothesis test checks that v2's draw equals an unfiltered draw from the frame without the households the filters leave out (mutants V1 and V2). - The state check's mock engine now pays only people the probe holds, so a probe without a child fails the dependent case (V3), and a loss counts as unstated income (V4). - The build's metadata lists the states the draw left out ("excluded_states": ["AK"]), and the build test checks that household_filter names both filters' sources (V6). The module docstring now names the two filters v2's draw adds to the board's sampler. No change touches policybench/, scripts/tool_pilot_argon.py or the fix modules, so 1c8c645 stays the last commit to touch TREATMENT_PATHS (scripts/confirmatory_run.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Point TREATMENT_COMMIT and the runner's docstring at 1c8c645, the last commit to touch TREATMENT_PATHS, and drop 6b00a6f from the test docstring that described the treatment harness. This commit touches none of TREATMENT_PATHS, so 1c8c645 stays the treatment commit. The 234 tests in tests/test_confirmatory_run.py that CI runs pass locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v2 prompt: carry out d1970, d1971, d1972 and d1029
The merge c0a2885 is now the last commit that touches the treatment's code (tool_pilot_argon.py gains the web+pe3 arm), so TREATMENT_COMMIT moves there from 1c8c645, in the constant and the module docstring. Max ruled on 2026-11-03 (d2229, amended) that the confirmatory run adds a pre-registered web vs web+pe3 stage for GPT-6 Luna, Gemini 3.5 Flash-Lite and Claude Sonnet 5.5, so its harness needs this arm. No other arm's treatment changes (#246 checks each byte for byte against 6b00a6f). The new test runs the preflight's own comparison, so a commit that changes the treatment's code without moving the pin fails in CI, not only at run time (#249's delta review, nit 4). It comes out after the confirmatory run, before this line reaches main (#249, reply 6596821678). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tool pilot: a web+pe3 arm, web search with the version 3 calculator
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP. This builds PolicyBench v2's head-to-head on fresh households from the latest Microcosm build, run in small batches with AI judges between them. Max's direction (2026-10-22):
In this PR so far
scripts/household_set.pybuilds a household set in the frozen board's layout:mc_000onward.latest_final.py, pins checked).The set stays outside the repository until a release, so a web arm can't find its answers.
Check of the reference path.
check-frozenruns the builder on the frozen board's own 100 households with policyengine-us 2.41.0. It reproduces all 1,935 scored references within 1e-6.tool_pilot_argon.py --board DIR:engine_upgraderevision.--baseline-dir), which must use the same prompt version.Progress (2026-10-22)
Batch judge. The batch judge, item 1 of the plan below, is built:
scripts/batch_judge.py: payload and derivations.outputDefinitionsandanswerCondition.JUDGE_API_KEY_SECRET), in a fresh config directory, refused alongside any login.Dry run. On the frozen board's no-tools v2 answers it flagged 23 of 1,935 cells at 2 of 3 models.
Rehearsal (two frozen households, 2.41.0): one full batch ran end to end, covering 12 model-arm runs, scoring, the trigger, the blind two-stage judges (6 valid verdicts) and collection.
Engine upgrade, frozen board on 2.58.0 with latest_final: 6 of 1,984 outputs move.
Judges (Max: "should we use gpt as judge for this?"):
2026-10-23:
v2 waits for a clean 2.60.x bundle (Max).
household_set.py build --purpose v2refuses anything but a certified pair. The Microcosm recertification is tracked in Recertify a published US release for a newer policyengine-us without a rebuild microcosm#1366.Test sets (
--purpose test) may name a published dataset. Test batch 0 ran 10 households from the Route A build 0fb05b6 on 2.60.4, for $17.06.Calculator tool version 2 (the
pe2arm, 6e1578f)._desiredand flags inputs that replace a formula.Tool version 1 is unchanged.
Update (2026-10-23 evening). The interface tests run on cheap models (Haiku 5.5, GPT-6 Luna, Gemini 3.1 Flash-Lite) until v2 is ready. Cheap batch 0 is done; batch 1 waits for a host reservation. Tool v3 (independent calculator features for an ablation pilot, and a finding that the prompts label some facts in words that tool v2 cannot read) is in Tool v3: calculator features one at a time, for the ablation pilot #243, stacked on this branch. It merges here after cheap batch 1, which runs from this branch at 5bbd571.
Plan (to update as it lands)
consensus-flagswith a three-model threshold: 2 of 3 models agreeing on the same wrong answer.adversary-preparewith the v2-pilot prompt and definitions the models saw.adversary-collect.Invariants
referencesrecomputes a set and must match it exactly.check-frozen, and a slow test on three households).🤖 Generated with Claude Code