[MaxGhenis]

Draft: work in progress. This adds the confirmatory runner's web stage for PolicyBench v2's pre-registered H5. It runs web against web+pe3 for GPT-6 Luna, Gemini 3.5 Flash-Lite and Claude Sonnet 5.5 on the calculator stage's 120 households, after that stage ends. It builds on #246 (07ee27f) and changes scripts/confirmatory_run.py, scripts/confirmatory_no_tools.py and the runner's tests.

What the commits do

• f1cbacf, 38721b4, fca2a3f: the web stage. The runner takes each stage's models, arms, folders, profile rule and order stream from a stage table, and leaves the calculator stage's six models, streams, records and failure rules as they were. The web design links the calculator design by path and sha256 and must share its frozen inputs. Preflight starts the web stage only once the calculator stage has ended. That takes a closed host log and a final summary per model, with no request in flight or unresolved. Every attempt must join its record, and the final anchor's push must have succeeded and match the local file.
• eed0ba5: Gemini's search. Preflight refuses a web stage with a Gemini model unless litellm is the version the harness's search patch supports. Tests capture each request body before it leaves the machine and check that both Flash-Lite arms keep Google Search beside their function declarations.
• 78315dd, 4dc830b, 9106545, f92c90a: stage budgets, checked before any provider request.
  • The calculator design's cap must be $150 and the web design's $270, each with a $25 margin to its hard ceiling.
  • Preflight refuses a design whose spent_before already reaches its ceiling. If one started anyway, Ceiling records the crossing when the runner builds it, so the run admits no household.
  • Each model's expected cost, cap and reserve must be finite numbers. Caps must sit within $0.005 of the model's share of the stage's $145 or $120, and reserves within $0.0005 of three times its expected cost per household. Preflight checks these directly and in the calculator design a web design links.
  • The web design's spent_before must reach, less $0.000001, the larger of two flat math.fsum totals: the calculator design's spent_before plus every calculator-stage cost, and every saved no-tools cost plus every calculator-stage cost.

Invariants

For every design the preflight accepts:

• web spent_before >= max(fsum([B, *A]), fsum([*N, *A])) - 0.000001, where B is the calculator design's spent_before, A every calculator-stage cost and N every saved no-tools cost; a total that overflows refuses every finite declaration;
• spent_before < overall_cap + 25, and Ceiling(ceiling, spent_before).exhausted() holds before any admission whenever spent_before >= ceiling;
• for every registered model m, abs(caps[m] - S * e[m] / T) <= 0.005 and abs(reserves[m] - 3 * e[m] / 120) <= 0.0005, with S = 145 or 120 and T = fsum(e) finite and positive.

The study's independent report applies the same rules. A differential test there gives both programs 25 designs and requires the same verdict on each. It then runs an accepted web design on fake providers, and the report must pass its records. Removing any one of the 12 new runner checks fails its named test.

Status and plan

• At 8355176, the runner's confirmatory test files pass in full on a local run: 682 tests in tests/test_confirmatory_run.py, and 25, 11, 50, 205, 200 and 56 in the anchor, board-engine, input-reads, no-tools, no-tools-records and unreadable-evidence files.
• Two independent reviews of 8355176 asked for changes. We're queuing a build that applies three runner fixes we prepared and tested separately, then fixes the remaining findings; this branch will take its commits.
• We'll rewrite the description above, which covers the commits through f92c90a, before this leaves draft.
• After the reviews agree, CI runs through a ci/pr<N>-<sha> branch, since ci.yml runs only for pull requests into main.
• This merges with a merge commit; never squash or rebase v2-microcosm-batches.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Nov 8, 2026 6:24pm UTC