Skip to content

test(codeql): align receipt fixtures with exact evidence - #2039

Draft
seonghobae wants to merge 9 commits into
fix/codeql-rerun-missing-verdict-redispatchfrom
codex/pr1902-test-contract-successor
Draft

test(codeql): align receipt fixtures with exact evidence#2039
seonghobae wants to merge 9 commits into
fix/codeql-rerun-missing-verdict-redispatchfrom
codex/pr1902-test-contract-successor

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Successor to #1902

This successor preserves the full #1902 delta and adds the missing test-contract repair found by running its exact head.

Root cause

test_codeql_pr_app_receipt_requires_exact_dispatch_evidence declared a field,value parametrization but accepted only tmp_path, so pytest failed during collection with:

function uses no argument 'field'

After adding the parameters, the test still modeled the old one-shot status-only behavior. The updated workflow requires authenticated producer-run evidence, so the forged-status test now supplies an exact failing dispatch artifact and continues to prove that a non-OpenCode status cannot satisfy the verdict.

Evidence

  • predecessor: fix(codeql): recover reruns after missing dispatch verdict #1902 at e9c69e18d0ebab65658041db86d61e7e9ca399c4
  • successor head: 554c31a1
  • focused owner tests: tests/test_codeql_pr_rerun_recovery_contract.py, tests/test_codeql_pr_workflow_contract.py, tests/test_codeql_scan_dispatch_workflow_contract.py
  • observed RED: pytest collection error, then one stale fixture failure
  • after repair: all focused tests completed without failure output; process exited cleanly
  • git diff --check: passed

No workflow gate, security check, or producer evidence requirement was weakened.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae seonghobae added bug Something isn't working priority: high High-priority or P1 work labels Sep 8, 2026 — with ChatGPT Codex Connector
Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact-head follow-up: the focused suite exposed one additional collection error in the same contract file: a test accepted undeclared field,value parameters without parametrization. Successor head 27c660c9 removes those undeclared parameters. The full focused suite also reports 10 stale fixture/assertion failures caused by the new exact producer-evidence and run-wide settlement contract; they remain visible RED findings and are not being hidden or marked passed.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

Copy link
Copy Markdown
Contributor Author

Additional root-cause repair pushed at exact head adfa1d43: the shared _run_verdict_read test helper used truthiness defaults, so explicitly empty producer job/artifact evidence was silently replaced by a successful default fixture. It now defaults only when the argument is None, preserving intentional empty evidence cases. py_compile and focused targeted execution were rerun; the remaining failing assertion is now isolated to stale status-only expectations, not fixture substitution.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact head f90651f3 adds the focused contract repair: malformed or absent producer evidence now explicitly exercises the required fail-closed dispatch path. The six targeted receipt-rejection cases pass (6 passed, 48 deselected, 18.40s).

seonghobae commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Design assurance carryover verification — e37d0c17f1d5ae82877f707a96895cd7be23d320

The branch was non-force restacked onto canonical #1902 base 723d1c368b459ce5646d5ebc91ce9546bbe5fc70. Fresh compare is ahead 8 / behind 0 with files=[]; the PR now has zero effective delta. Canonical #1902 has advanced to 855f0173aae53620f68322e3b218c022fe1c4eed and its body explicitly records equal-or-stronger carryover of this PR’s empty-evidence, forged producer metadata, later-page receipt, malformed/failure-state, and mismatched receipt contracts.

Do not merge this zero-delta branch or use its queued checks as canonical evidence. Keep Draft until #1902’s current head obtains terminal hosted equivalence; then the no-valid-delta / verified-complete-carryover rationale can support retirement without discarding history.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Full focused owner contract suite rerun at exact head f90651f3: 101 passed, 4 failed in 277.38s. Remaining RED is narrowed to one second-page failure-verdict expectation and three coordinator settlement/dispatch fixture-contract cases; no collection errors remain. The four failures are preserved as actionable findings.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact-head 80775ade adds producer gate state to the paginated verdict fixture. The targeted second-page contract now passes for both success and failure (2 passed, 52 deselected, 76.82s). The full suite is reduced to four coordinator/status-contract RED cases, all preserved for the next repair.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact head 1d64118c repairs the coordinator fixtures with a shared exact producer-evidence builder. The three remaining coordinator RED cases now pass (3 passed, 51 deselected, 48.82s), covering pending-language retention, exclusion of already evidenced successful jobs, and no redispatch when all languages have terminal evidence.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact head 1d64118c full focused owner contract suite is GREEN: 105 passed in 306.15s (0:05:06), with no warnings or failures reported. This includes the previously failing receipt and coordinator cases.

Copy link
Copy Markdown
Contributor Author

Verified canonical carryover audit against #1902 exact head 723d1c368b459ce5646d5ebc91ce9546bbe5fc70 / tree f372d1e4217ff0423f76688d3fe5cc23bc48970d.

Every valid semantic delta from this successor is now carried by #1902. The two missing cases—preserving explicit empty producer evidence and rejecting App receipts with substituted exact run metadata—were integrated as executable tests. The remaining failure-state, malformed-evidence, later-page, and incomplete/mismatched receipt semantics are present in equal or stronger canonical contracts. Focused 111 passed; full 3061 passed, 1 skipped, 21 subtests; statement/branch/public-doc 100%.

This PR stays open/Draft until current canonical hosted equivalence is observed. No predecessor review/check evidence is transferred, and no close/retirement is performed in this step.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Non-force restack completed after base e07e8fd5 advanced. Merge commit 9bb6f08 preserves the successor delta and latest base changes. Full focused owner contract suite on the restacked tree is GREEN: 106 passed in 208.89s (3:28), no warnings or failures.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Latest base 723d1c36 advanced while #2039 was under review. Restacked normally without force push; merge commit is exact head e37d0c17. git diff --check and Python compile pass. The successor retains its full delta and the latest base test-contract changes.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact head e37d0c17f1d5ae82877f707a96895cd7be23d320 was revalidated after the final non-force restack onto base 723d1c368b459ce5646d5ebc91ce9546bbe5fc70:

  • Owner full contract suite: 3061 passed, 1 skipped, 21 subtests passed in 488.19s.
  • No test failure or warning output was reported.
  • This replaces the pre-restack local suite receipt; no predecessor evidence was reused.
  • Hosted review/check state and protected merge remain independently required.

…-redispatch' into codex/pr1902-test-contract-successor

Copy link
Copy Markdown
Contributor Author

Non-force restack onto live parent fix/codeql-rerun-missing-verdict-redispatch@ebf054e67f7dbeb7c9b8d0f90e2bc1c151471a2b.

New successor head: 954891a94a27040db05d1329d84d524255a51e7b.
Unique tree versus that parent is empty: every valid #2039 test-contract delta remains on the canonical #1902 owner. Kept Draft until hosted equivalence is observed. No close, no force-push, no Ready conversion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority: high High-priority or P1 work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant