feat(harness): negative controls, leaked-answer check, grounding and suite proving (TH-8918) by KarthikAvinashFI · Pull Request #321 · future-agi/agent-learning-kit

feat(harness): negative controls, leaked-answer check, grounding and suite proving at certify

KarthikAvinashFI

committed

70e7401

fix(harness): read compared literals inside a check's return condition and give the proof test world setup outcomes

KarthikAvinashFI

committed

7281e33

test(harness): pin that a raised failure message is not a grader-only leak

KarthikAvinashFI

committed

dd4e8e5

fix(harness): an assumed solution step never reads as a no-change case, and why names the real block

KarthikAvinashFI

committed

36d58c7

Merge branch 'feat/TH-8752-proof-honesty' into feat/TH-8918-scenario-lifecycle

KarthikAvinashFI

committed

32818d1

feat(harness): a judged-only case is proved only when the judge fails a control's wrong transcript, else kept as written

KarthikAvinashFI

committed

ad37f0c

feat(harness): prove_suite proves only the scenario keys passed with --keys

KarthikAvinashFI

committed

445444e

feat(harness): compare the reference solution's argument values as grader-only material

KarthikAvinashFI

committed

bf2c058

feat(harness): judge each cited rule against the expected outcome and block a case that contradicts it

KarthikAvinashFI

committed

e61ebba

feat(harness): the builder refuses a new or changed scenario with no sources and asks for an assumption source

KarthikAvinashFI

committed

ec69f38

feat(harness): a scenario carries must_pass, versioned with its content

KarthikAvinashFI

committed

f76c3f2

docs(harness): tell the scenario writer when to mark a case must_pass

KarthikAvinashFI

committed

23010c0

feat(harness): record what each check reads, tool calls or world state, in its proof

KarthikAvinashFI

committed

6d08f42

feat(harness): prove reference solutions and negative controls on the live runtime for service-bound tools

KarthikAvinashFI

committed

d751967

fix(harness): a check that raises under a negative control reads broken, and a control that cannot be played is no evidence

KarthikAvinashFI

committed

70e8de2

fix(harness): a reference amount written as 37.5 or 37.50, or a value nested in a list or object, counts as a leaked answer

KarthikAvinashFI

committed

d45eba3

fix(harness): a blank assumption source does not count as citing where a new case comes from

KarthikAvinashFI

committed

2a29511

Navigation Menu

Repository navigation

feat(harness): negative controls, leaked-answer check, grounding and suite proving (TH-8918) - #321

feat(harness): negative controls, leaked-answer check, grounding and suite proving (TH-8918)#321
KarthikAvinashFI wants to merge 21 commits into
feat/TH-8752-proof-honestyfrom
feat/TH-8918-scenario-lifecycle

Commits on Nov 14, 2026

feat(harness): negative controls, leaked-answer check, grounding and suite proving at certify

fix(harness): read compared literals inside a check's return condition and give the proof test world setup outcomes

test(harness): pin that a raised failure message is not a grader-only leak

fix(harness): an assumed solution step never reads as a no-change case, and why names the real block

Merge branch 'feat/TH-8752-proof-honesty' into feat/TH-8918-scenario-lifecycle

feat(harness): a judged-only case is proved only when the judge fails a control's wrong transcript, else kept as written

feat(harness): prove_suite proves only the scenario keys passed with --keys

feat(harness): compare the reference solution's argument values as grader-only material

feat(harness): judge each cited rule against the expected outcome and block a case that contradicts it

feat(harness): the builder refuses a new or changed scenario with no sources and asks for an assumption source

feat(harness): a scenario carries must_pass, versioned with its content

docs(harness): tell the scenario writer when to mark a case must_pass

feat(harness): record what each check reads, tool calls or world state, in its proof

feat(harness): prove reference solutions and negative controls on the live runtime for service-bound tools

fix(harness): a check that raises under a negative control reads broken, and a control that cannot be played is no evidence

fix(harness): a reference amount written as 37.5 or 37.50, or a value nested in a list or object, counts as a leaked answer

fix(harness): a blank assumption source does not count as citing where a new case comes from

Commits on Nov 15, 2026

test(harness): builder-submitted cases in the fixtures carry an assumption source, as sources are now required

test(harness): the builder's submit refuses a new case that cites nothing

Merge branch 'feat/TH-8752-proof-honesty' into feat/TH-8918-scenario-lifecycle

fix(harness): write must_pass into a scenario only when it is set, so an older suite rewritten as it was reads unchanged