Skip to content

test: pin the type checker's 1 sentinel and the source-dependent receipt labels - #28

Merged
docxology merged 1 commit into
mainfrom
test/p3-conformance-receipt-labels
Nov 12, 2026
Merged

docxology merged 1 commit into
mainfrom
test/p3-conformance-receipt-labels

Conversation

@docxology

@docxology docxology commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

What the review found

#27 made the GNN conformance gate fail closed (validate_gnn_files(...) is True) and
made the receipt result source-dependent, but its tests left two of those claims
unpinned:

  • tests/test_gnn_conformance.py parametrized the rejected outcomes as
    [2, False, None, 0]. The documented sentinels are True / 2 / False, and 1
    was not exercised. 1 is the only value where the shipped identity check and the
    plausible wrong variant outcome != True disagree: 1 != True is False, so
    != True would accept 1 while rejecting 2. Nothing in the suite forbade that
    edit.
  • No test called main(), so the documented split — .github/GNN.md: result is
    repository-content-passed for repository sketches and
    pinned-exemplar-baseline-passed when only the pinned upstream exemplar was checked —
    was unverified by the gate.

Change

tests/test_gnn_conformance.py only:

  • 1 added to the rejected-outcome parametrization.
  • test_main_labels_repository_content_receipt and
    test_main_labels_pinned_exemplar_baseline_receipt call main() on both source
    paths with the existing stand-in gnn module and a local stand-in for
    urllib.request.urlretrieve (no network). They assert source, result,
    models, gnn-version, that the fetch URL ends with EXEMPLAR_PATH and carries the
    pin's revision, and — by parsing the whole captured stdout as one JSON document —
    that the run prints exactly the one JSON line the doc promises.

No production code changed; the merged behaviour already matched the documentation.

Test behaviour before and after

Before (3d1afa3:scripts/gnn_conformance.py, the pre-#27 script, with the new test
file):

FAILED test_type_checker_non_true_outcome_fails[2]  - Failed: DID NOT RAISE SystemExit
FAILED test_type_checker_non_true_outcome_fails[1]  - Failed: DID NOT RAISE SystemExit
FAILED test_main_labels_repository_content_receipt   - assert ('repository', 'verified') == ...
FAILED test_main_labels_pinned_exemplar_baseline_receipt - assert ('pinned-exemplar', 'verified') == ...
8 failed, 1 passed

After (main, this branch): 9 passed.

The [1] case also catches the wrong-variant fix the old test allowed: with
if outcome != True, 2 still raises (old test green) but 1 does not (new case
red).

Gate

./gate.sh arms in a detached worktree of 1031f76 with its own
uv sync --locked --extra dev:

Arm Result
lint pass — all checks, 121 files formatted
test pass — 788 passed, 9 skipped, coverage 89.80% (gate 80%)
build pass — sdist + wheel 3.3.0
program pass — edition scaffold, approved public tree, site render
site pass — render from the verified public tree
docs pass — 241 Markdown files, relative links, docs index, gate arms
gnn pass — pinned GNN 5.0.3; {"source": "pinned-exemplar", "result": "pinned-exemplar-baseline-passed", "models": 1}
sidecar aii_sidecar validate → OK

788 is the previous 785 plus these three tests; no existing test, assertion or
selection was changed.

Hosted Actions remain billing-locked: every run of this branch, and of main at
1031f76 and 17e9345 (PR #27), concludes failure with an empty step list — the
jobs never start. The local gate above is the evidence.

🤖 Generated with Claude Code

@docxology
docxology force-pushed the test/p3-conformance-receipt-labels branch from c8c44c9 to 12bfe49 Compare November 12, 2026 20:24
The merged fail-closed check requires `validate_gnn_files(...) is True`
and labels the receipt by source path, but the suite exercised neither
completely: the non-True parametrization skipped `1`, the only value
that separates an identity check from `!= True`, and no test called
`main()`, so the documented `repository-content-passed` /
`pinned-exemplar-baseline-passed` split was unverified.

- add `1` to the rejected outcomes (it is truthy and equal to True).
- call `main()` on both source paths with a stand-in `gnn` and a local
  exemplar fetch, asserting source, result label and models count.

On the pre-change script both new cases fail: `1` did not raise, and the
receipt said `verified` for either source.
@docxology
docxology force-pushed the test/p3-conformance-receipt-labels branch from 12bfe49 to 1bd96b3 Compare November 12, 2026 20:26
@docxology
docxology merged commit 58d61d5 into main Nov 12, 2026
0 of 9 checks passed
@docxology
docxology deleted the test/p3-conformance-receipt-labels branch November 12, 2026 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant