Skip to content

fix(harness): save whether each proof's solution ran and report proved counts (TH-8752) - #320

Draft
KarthikAvinashFI wants to merge 3 commits into
devfrom
feat/TH-8752-proof-honesty
Draft

KarthikAvinashFI wants to merge 3 commits into
devfrom
feat/TH-8752-proof-honesty

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Why

TH-8752: a scenario whose reference solution never ran (its tools have no endpoint in the prove lane, so the steps were assumed) was saved as holds: true and reported as proved. Plan task T05 ("Make scenarios valid and meaningfully graded") needs an unproved case to read as unproved everywhere it is counted.

This PR makes the saved proof say what really ran, and makes the build's Finished message report proved, kept as written and not proved apart.

What changed

File Change
src/fi/alk/harness/prove.py Proof.proved (holds, nothing assumed, nothing deferred), Proof.kept_as_written (not proved only because the lane could not execute something) and Proof.blocked_reason() (first reason, faults before lane limits). LANE_CODES = proof_deferred, solution_assumed, control_not_run.
src/fi/alk/harness/proofs.py Proof entries are format 2: proved, kept_as_written, assumed, solution_steps, deferred, judged_only, blocked_reason, and a why that names the assumed tools. New proof_key(), unprovable_entry(), proof_counts().
src/fi/alk/harness/hosted_chat_entrypoint.py finish_build writes the counts into the build-finished file and its reply reads "N proved, N kept as written (tools), N not proved (reasons). Report exactly these counts".
src/fi/alk/harness/builder.py, nudge.py, scenario_tools.py Every record_proof call passes solution_steps and saves under proof_key().
src/fi/alk/harness/skills/builder/SKILL.md "Finishing a build": say proved only for the count finish_build reports as proved.
tests/harness/proof_world.py Test world helper for the proof tests; its stub returns the scenario-code outcome the prove step reads.
tests/harness/test_proofs.py, tests/harness/test_builder_chat.py New tests (below).

Behavior callouts

  • A proof that holds but assumed a step is saved holds: true, proved: false, kept_as_written: true, with why naming the tools whose steps never ran. Older readers that only look at holds still see true; the platform side reads proved for format 2 (companion PR below).
  • A deferred proof reads proof_deferred, not not_ready: its gates never ran, so they are unset rather than failed.
  • The closing counts are written by code: when the build finished during the turn, the reply gets "Proof counts: N proved, N kept as written, ..." appended, and a reply that claims more proved than the count gets a correction line.
  • No change to which scenarios are kept or refused.

Tests

  • 6 new tests: test_assumed_steps_are_saved_and_not_proved, test_proof_counts_split_kept_as_written, test_unprovable_is_format_two, test_deferred_reads_proof_deferred_not_not_ready, test_holds_failures_map_to_codes, test_finished_message_reports_counts.
  • Run module by module on the stacked lifecycle head (which contains this branch): the touched modules plus every tests/harness module importing changed code (35 modules) and tests/test_harness.py. All new and named tests pass. Failures that exist identically on dev (0203bdf5): test_harness.py 3, test_background_noise.py 2, test_grade.py 1, test_harness_architecture.py 11.
  • Mutation checks across this PR and the stacked lifecycle PR: 32 of 32 mutations killed.

Added on this head:

  • b9b21ee6: the finishing reply carries the code-written counts and corrects an overclaim. Tests test_finished_turn_appends_code_counts, test_overclaim_gets_a_correction, test_exact_claim_gets_no_correction.
  • 99122c5d: test_assumed_steps_are_saved_and_not_proved failed on this branch alone (the stub returned the check outcome type, and the check credited a failed call); the stub now returns the scenario-code outcome and the check reads only calls that succeeded.
  • This branch alone, the targeted harness modules in one process: 2592 pass; three failures, the same on dev (0203bdf5): test_hosted_claude_session_uses_durable_resume_and_restricted_tools, test_a_skill_only_names_tools_its_stage_actually_has, test_the_system_prompt_stays_within_budget.

Live proof

Local stack with production sandbox specs (4 CPU, 8192 MB, restricted egress). Validations re-proved seven existing environments with this code.

Case Result Proof
Cases whose tools run only inside the agent Saved proved: false, assumed tools listed, counted "kept as written", never proved Toolbar line below
Judged-only cases on a voice agent Proved, counted proved Second image
Finished message counts on a fresh build Unit only (test_finished_message_reports_counts); no fresh full build was run on this head Not shown

Counts line: 0 proved, 9 kept as written, 0 not proved on an environment whose cases cannot run their solutions

After validation: 5 proved, 0 kept as written, 1 not proved

Playwright e2e

None in this repo. The platform flows that read these proofs (SIM-E2E-001 to 006 and 008) passed 7 of 7 on the stacked platform lifecycle branch, before its last base merge.

Scope out

  • Proving the reference solution on the real stack, so that agent-bound tools actually run (they stay "kept as written").
  • CLI and SDK display of the counts: later, after the platform contract is released.

Stacked on / merge order

  • Base: dev.
  • Companion: future-agi feat/TH-8752-proof-honesty reads format 2 (proved) and older entries (holds), so either can merge first.
  • Stacked on this PR: agent-learning-kit feat/TH-8918-scenario-lifecycle.

Linear: TH-8752

@KarthikAvinashFI KarthikAvinashFI self-assigned this Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant