[KarthikAvinashFI]

Why

TH-8918 (scenario lifecycle), plan task T05 ("Make scenarios valid and meaningfully graded"): a scenario is only worth running when its checks can fail a wrong behaviour, when nothing the caller or the agent sees gives the answer away, and when what it cites really says what it claims. Before this PR none of the three was checked: a case passed with checks that also passed a wrong refund, a case could quote its own grader text in the caller's instruction, and sources were not recorded at all.

This PR adds the three checks to the prove gate and a suite-wide re-prove that the platform's Validate runs at certify. The platform side (lifecycle states, Validate, archive, restore, source changes) is future-agi feat/TH-8918-scenario-lifecycle.

What changed

File	Change
src/fi/alk/harness/scenario.py	NegativeControl (name, why, steps, must_fail) and ScenarioSource (kinds rule, file, chat_message, evidence_case, requirement, assumption, with the quoted words) on Scenario.
src/fi/alk/harness/prove.py	Plays each negative control like a solution; a control fails when a check it names fails. New blocks no_negative_control, negative_control_passed, answer_leaked, source_unresolved; control_not_run when a control's tool has no endpoint here. _outcome reads a refusal case as no_change only when the reference solution really left the world untouched.
src/fi/alk/harness/leaks.py (new)	Leaked-answer check: an 8-word run of grader-only text (sub-goal what, judged, reference, verifier, comparison, tests, a control's why, evaluator-only files), a multi-word sub-goal name, or a computed expected value the world does not hold, found in the instruction, variables, persona, opening message or an actor's brief. Returns the excerpt and the grader field.
src/fi/alk/harness/grounding.py (new)	Resolves rule, file, requirement and assumption sources against the contract and the world's files (matched_digest as sha256:<hex>); other kinds are left for the platform.
src/fi/alk/harness/prove_suite.py (new)	Re-proves every scenario of a suite, writing each proof as it is made; a crash on one case is broken_check, a run out of budget writes the rest validation_incomplete. Keeps a 300 s margin before the job's deadline.
src/fi/alk/harness/authoring_entrypoint.py	--certify-only --prove-scenarios [--deadline-at] runs the suite re-prove before sealing and writes scenario-proofs.json; a failure there never fails the certify.
src/fi/alk/harness/scenario_tools.py	accept_scenario refuses a case with one of the four codes above, with the reason; controls are checked against the agent's contract like solution steps.
src/fi/alk/harness/proofs.py	Format 2 entries also carry outcome, the controls that ran, the leaks found and the source results.
skills/write-scenarios/SKILL.md, references/refusals.md, skills/builder/SKILL.md	How to write a task-specific negative control, cite sources, and keep grader-only material out of what the caller and agent see.
tests/harness/*, tests/test_harness.py	New and adjusted tests (below). Existing builders that submit through accept_scenario gained a negative control.

Behavior callouts

• Every new scenario must carry at least one negative control; without one it is not kept (no_negative_control). Existing saved scenarios are not rewritten; the platform reads them as legacy.
• A scenario with no sources is kept and shown as "assumed by the builder". Only a cited source that does not resolve blocks it.
• A refusal (no-change) case may hold on an untouched world; its negative control performs the forbidden change, and the checks must fail it.
• Controls whose tools only run inside the agent read control_not_run: the case is kept as written, never proved.
• answer_leaked wins over every lane code; a lane code never hides a fault (test_lane_codes_never_hide_a_fault).
• New CLI flags on the authoring entrypoint (--prove-scenarios, --deadline-at). No new env var.

Tests

• 25 new tests across test_negative_controls.py, test_leaks.py, test_grounding.py, test_prove_suite.py, test_proofs.py, plus adjusted fixtures in test_nudge_turn.py, test_builder_chat.py and tests/test_harness.py.
• Run module by module: the touched modules plus every tests/harness module importing changed code (35 modules) and tests/test_harness.py. All new and named tests pass. Failures identical on dev (0203bdf5): test_harness.py 3, test_background_noise.py 2, test_grade.py 1, test_harness_architecture.py 11.
• Mutation checks (this PR and its base): 32 of 32 killed; 6 weak mutations were redone and test_failure_message_is_not_grader_only strengthened.
• Break-it pass, throwaway tests: deferred plus leaked, assumed plus a control that passed, an unresolved source, format 1 counts, a crash on the first case, a re-recorded proof, a control naming a check that does not exist, an assumed write outcome, the why text. Two real bugs came out of it and are fixed here: a compared literal inside return None if ... == 42.50 else ... was not read by the leak check, and an assumed write step read no_change, so a write case looked like a refusal case.

Live proof

Local stack with production sandbox specs (4 CPU, 8192 MB, restricted egress), builder chat on real demo agents.

Case	Result	Proof
Describe a case; the builder submits without a control	"Not kept. no negative control" three times, until it wrote a task-specific control (a $70 refund instead of the full $140)	Image 1
Ask for a case whose instruction repeats a sub-goal's words	Refused: "what the caller or the agent sees gives the answer away: instruction repeats ... from sub_goal:identifies_caller_before_prescription.what"	Image 2
A rule the case cites is removed	Re-prove reads Draft: "no rule of the agent contains the quoted words"	Image 3
From a policy file / from CSV records	Each case cites the file line or its row	Images 4, 5
A refusal case (refund outside the 30-day window)	Outcome no change; its forbidden refund is the control (Not run on agent-bound tools)	Image 6
A negative control that actually runs	Not shown live: no local environment has tools the prove lane can execute; unit tests test_two_refunds_control_fails, test_control_that_passes_blocks	Not shown

Playwright e2e

None in this repo. The platform flows for the lifecycle (SIM-E2E-008 and SIM-E2E-001 to 006) passed 7 of 7 on the platform branch at 4fb67ad21, with this branch mounted.

Plan lines S3 to S21 and the test pass on this head

Line	Change
S3 judged-only	A case only a judge grades is proved only when the judge fails a control's wrong transcript; otherwise it is kept as written (ad37f0c5).
S4 live controls	For service-bound tools, the reference solution, an empty run and each negative control are played on the live runtime (live_proof.py, d751967e); a check that raises under a control reads broken and a control that cannot be played is no evidence (70e8de2f).
S6 per-case	prove_suite --keys proves only the named cases (445444e0).
S11 sources	The builder's submit refuses a new or changed case that cites nothing and asks for an assumption source (ec69f384); a blank assumption source does not count (2a29511b).
S16 evidence	Each check's proof records what it reads, tool calls or world state (6d08f42c).
S17 policy conflict	Each cited rule is judged against the expected outcome; a contradiction blocks the case, an undecided judge never does (e61ebba3).
S19 must-pass	A scenario carries must_pass, versioned with its content (f76c3f2b, 23010c01).
S21 grader-only	The reference solution's argument values are compared as grader-only material (bf2c058a); 37.5 and 37.50, and values nested in a list or object, count as a leaked answer (d45eba32).

Fixed during the test pass:

Commit	Fix
d333f0c3	Found live: a builder save on an older suite showed "1 added, 12 changed" because every rewritten scenario.json gained "must_pass": false. must_pass is written only when set, so an older suite rewritten as it was reads unchanged.
3b2fbf9c	The builder submit's require_sources=True had no test (a mutation survived); a test now pins it.
e4481805	Ten fixtures submitted builder cases with no source; they now carry an assumption source as the builder writes them. The gate is unchanged.

Tests on this head (the targeted harness modules, one process, the CI extras installed): 2620 pass. Three failures, the same on dev (0203bdf5): test_hosted_claude_session_uses_durable_resume_and_restricted_tools, test_a_skill_only_names_tools_its_stage_actually_has, test_the_system_prompt_stays_within_budget. With the two branches stacked on it: 2658 pass, the same three.

Mutation checks in this pass, all killed: the accept-time sources gate (2 tests fail), the builder submit sources flag (1), must_pass written only when set (1).

Break-it attempts: a blank or whitespace assumption source (refused), an unchanged legacy case with no sources (saves), a changed legacy case with no sources (refused), a judge that cannot decide a policy conflict (not cached, never blocks), a control naming a check the case does not have (refused with the reason).

Live (production-spec local stack, pharmacy voice demo): asked the builder for a case about a prescription number that is not on file; it wrote the case as "not on file" and cited the person's words. Image on the platform PR #4107.

Scope out

• Proving in-process (agent-bound) tools by driving the agent: later, it needs the tool wire format; such cases stay "kept as written". Service-bound tools are proved on the live runtime.
• CLI and SDK parity (validate, archive, sources): later, after the platform contract is released.
• Re-certifying an environment built by an older harness can fail in the sandbox (source_model_fingerprint_mismatch, generated_runtime_entrypoint_ambiguous), in code this PR does not touch.

Stacked on / merge order

• Base: feat/TH-8752-proof-honesty (agent-learning-kit). Merge that first.
• Companion: future-agi feat/TH-8918-scenario-lifecycle passes --prove-scenarios and reads scenario-proofs.json; this PR should merge before or with it.

Linear: TH-8918