Repository navigation
fix(harness): save whether each proof's solution ran and report proved counts (TH-8752) - #320
Draft
KarthikAvinashFI wants to merge 3 commits into
Draft
KarthikAvinashFI wants to merge 3 commits into
KarthikAvinashFI wants to merge 3 commits into
Conversation
…d counts at finish
… and correct an overclaim
…he check reads only calls that succeeded
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
TH-8752: a scenario whose reference solution never ran (its tools have no endpoint in the prove lane, so the steps were assumed) was saved as
holds: trueand reported as proved. Plan task T05 ("Make scenarios valid and meaningfully graded") needs an unproved case to read as unproved everywhere it is counted.This PR makes the saved proof say what really ran, and makes the build's Finished message report proved, kept as written and not proved apart.
What changed
src/fi/alk/harness/prove.pyProof.proved(holds, nothing assumed, nothing deferred),Proof.kept_as_written(not proved only because the lane could not execute something) andProof.blocked_reason()(first reason, faults before lane limits).LANE_CODES=proof_deferred,solution_assumed,control_not_run.src/fi/alk/harness/proofs.pyproved,kept_as_written,assumed,solution_steps,deferred,judged_only,blocked_reason, and awhythat names the assumed tools. Newproof_key(),unprovable_entry(),proof_counts().src/fi/alk/harness/hosted_chat_entrypoint.pyfinish_buildwrites the counts into the build-finished file and its reply reads "N proved, N kept as written (tools), N not proved (reasons). Report exactly these counts".src/fi/alk/harness/builder.py,nudge.py,scenario_tools.pyrecord_proofcall passessolution_stepsand saves underproof_key().src/fi/alk/harness/skills/builder/SKILL.mdfinish_buildreports as proved.tests/harness/proof_world.pytests/harness/test_proofs.py,tests/harness/test_builder_chat.pyBehavior callouts
holds: true, proved: false, kept_as_written: true, withwhynaming the tools whose steps never ran. Older readers that only look atholdsstill seetrue; the platform side readsprovedfor format 2 (companion PR below).proof_deferred, notnot_ready: its gates never ran, so they are unset rather than failed.Tests
test_assumed_steps_are_saved_and_not_proved,test_proof_counts_split_kept_as_written,test_unprovable_is_format_two,test_deferred_reads_proof_deferred_not_not_ready,test_holds_failures_map_to_codes,test_finished_message_reports_counts.tests/harnessmodule importing changed code (35 modules) andtests/test_harness.py. All new and named tests pass. Failures that exist identically ondev(0203bdf5):test_harness.py3,test_background_noise.py2,test_grade.py1,test_harness_architecture.py11.Added on this head:
b9b21ee6: the finishing reply carries the code-written counts and corrects an overclaim. Teststest_finished_turn_appends_code_counts,test_overclaim_gets_a_correction,test_exact_claim_gets_no_correction.99122c5d:test_assumed_steps_are_saved_and_not_provedfailed on this branch alone (the stub returned the check outcome type, and the check credited a failed call); the stub now returns the scenario-code outcome and the check reads only calls that succeeded.dev(0203bdf5):test_hosted_claude_session_uses_durable_resume_and_restricted_tools,test_a_skill_only_names_tools_its_stage_actually_has,test_the_system_prompt_stays_within_budget.Live proof
Local stack with production sandbox specs (4 CPU, 8192 MB, restricted egress). Validations re-proved seven existing environments with this code.
proved: false, assumed tools listed, counted "kept as written", never provedtest_finished_message_reports_counts); no fresh full build was run on this headPlaywright e2e
None in this repo. The platform flows that read these proofs (SIM-E2E-001 to 006 and 008) passed 7 of 7 on the stacked platform lifecycle branch, before its last base merge.
Scope out
Stacked on / merge order
dev.feat/TH-8752-proof-honestyreads format 2 (proved) and older entries (holds), so either can merge first.feat/TH-8918-scenario-lifecycle.Linear: TH-8752