Skip to content

fix(harness): reseal the scenarios that load and name the ones that could not be applied (TH-8929) - #324

Draft
KarthikAvinashFI wants to merge 7 commits into
feat/TH-8918-scenario-lifecyclefrom
fix/TH-8929-partial-apply
Draft

KarthikAvinashFI wants to merge 7 commits into
feat/TH-8918-scenario-lifecyclefrom
fix/TH-8929-partial-apply

Conversation

@KarthikAvinashFI

Copy link
Copy Markdown
Member

Why

TH-8929: when a builder turn changed several scenarios and one of them could not be loaded the way a run loads it, the reseal failed and the whole turn failed with it, so every good change was lost. Now the good changes save, the case that could not be applied keeps its last sealed version (or stays out if it was never sealed), and the platform is told which case it was and why.

What changed

File Change
src/fi/alk/harness/bundle_author_v2.py scenario_seal_problems(authoring) loads each scenario folder alone, as a run does (scenario_source._load_one), and returns {key: first line of why}. reseal_scenarios(authoring, skip=...) reseals without the failing keys; _copy_scenarios keeps a skipped case's last sealed copy, never under a folder name a case written in the same save uses.
src/fi/alk/harness/hosted_chat_entrypoint.py _save_changes checks first, reseals with the failing keys skipped and publishes with failed_scenarios; the reply adds "Saved, except N scenarios that could not be applied: key: reason." A failure of the whole reseal (contract or seed) no longer fails the turn: the change is kept as the chat's unsaved draft and the reply says so.
src/fi/alk/harness/hosted_conversation.py ConversationClient.publish(digest, failed_scenarios=...) sends {"digest", "failed_scenarios": [{"scenario_key", "reason"}]}, at most the platform's 200, and only when there is one.

Behavior callouts

  • Nothing changes for a turn where every scenario loads.
  • A platform that predates future-agi fix/TH-8929-partial-apply refuses the unknown field; the field is sent only when a case failed, so ordinary saves are unaffected.

Tests

Test What it pins
test_builder_chat.py::test_one_broken_scenario_does_not_block_the_save Reseal skips the failing key, publish names it, the reply says saved except one
::test_a_folder_a_run_cannot_load_keeps_its_sealed_version The failing case keeps its last sealed copy
::test_a_held_copy_never_takes_the_folder_of_a_case_written_now A held copy whose folder name a new case now uses is left out instead of breaking the reseal
::test_a_failed_reseal_keeps_the_change_as_a_draft A whole-reseal failure keeps the change as a draft and the turn completes
::test_publish_sends_at_most_the_platform_limit_of_failed_scenarios 250 failures send 200; no failures send no field

Run on this branch (the targeted harness modules, one process): 2647 pass; three failures, the same on dev (0203bdf5). Mutations: reseal without skip, and the held-copy folder guard, each fail their test.

Break-it attempts: every case failing (each named), a never-sealed failing case (left out, the rest sealed), a held copy whose folder a good case took (fixed here: it raised FileExistsError and failed the whole reseal), 250 failures (capped at 200), a multi-line reason (first line only).

Live proof

The platform half is shown live on future-agi fix/TH-8929-partial-apply (one case of twelve reads "Could not apply" with its reason, the others keep their state, and discard clears it). A builder turn that writes an unloadable folder is not reproducible on purpose: the builder's own gate refuses a broken case before it is written, so the reseal path is shown by the tests above.

Stacked on / merge order

  1. agent-learning-kit feat/TH-8918-scenario-lifecycle (feat(harness): negative controls, leaked-answer check, grounding and suite proving (TH-8918) #321), this PR's base
  2. This PR, with future-agi fix/TH-8929-partial-apply

Linear: TH-8929

@KarthikAvinashFI KarthikAvinashFI self-assigned this Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant