Skip to content

fix(gnn): require the literal True type-check verdict - #14

Merged
docxology merged 1 commit into
mainfrom
fix/gnn-type-check-verdict
Nov 12, 2026
Merged

docxology merged 1 commit into
mainfrom
fix/gnn-type-check-verdict

Conversation

@docxology

Copy link
Copy Markdown
Collaborator

Defect

_check_types in .github/scripts/gnn_conformance.py tested the type checker's result for truthiness:

if not GNNTypeChecker().validate_gnn_files(model_dir, Path(output_dir)):

validate_gnn_files is tri-state, per its own docstring at the pinned revision 8f293d9 (v5.0.3): True when every file validated, 2 (SUCCESS_WITH_WARNINGS) when one or more files had type/consistency problems or when no GNN file was found, False on a hard failure. 2 is truthy, so an invalid model set passed and the arm printed "result": "verified".

Reproduction

Pinned parser installed exactly as the arm installs it (generalized-notation-notation @ git+…@8f293d9). One sketch — the pinned exemplar with A[2,2,type=float] changed to A[3,3,type=float], i.e. a declared shape that contradicts its 2×2 initialisation — parses cleanly and passes validate_gnn_syntax, and the checker returns 2:

$ /tmp/ke-gnn-venv/bin/python -B .github/scripts/gate.py gnn
⚠️ Type checking completed with warnings (no GNN files found, or some files invalid)
{"gnn-version": "5.0.3", "models": 1, "pin-revision": "8f293d9ba810e6eda5c09842d9c7ed9ffa5f92c1", "result": "verified", "source": "repository"}
GATE-EXIT:0

With this branch the same tree fails:

$ /tmp/ke-gnn-venv/bin/python -B .github/scripts/gate.py gnn
⚠️ Type checking completed with warnings (no GNN files found, or some files invalid)
GNN type checker did not verify the checked sketches: validate_gnn_files returned 2 (True = every file validated, 2 = invalid files or none found, False = hard failure)
GATE-EXIT:1

The empty-model-set path is documented as the same 2:

$ python -c "print(repr(GNNTypeChecker().validate_gnn_files(empty_dir, out)))"
2
_check_types([]) returned normally: gate would print result=verified

Fix

Only the literal True passes; the refusal reports the value the checker returned. No check was weakened and no test removed.

Tests

.github/scripts/tests/test_gnn_conformance.py (6 tests, dependency-free: it stubs gnn in sys.modules, so the tests arm and CI's docs job still need no parser). All five refusal tests are red on main and green here; the True case asserts the copied model set and that the receipt still prints.

Gate

Arm Command Result
links python3.12 .github/scripts/gate.py links ok (65 tracked files)
python / notebooks / checkpoints python3.12 .github/scripts/gate.py archive ok (65 tracked files)
tests python3.12 .github/scripts/gate.py tests ok (28 tests)
fast python3.12 .github/scripts/gate.py fast exit 0
gnn python3.12-in-a-venv .github/scripts/gate.py gnn (pinned parser 5.0.3) exit 0, pinned-exemplar

Hosted Actions are billing-locked for this organisation, so the local gate is the evidence.

`GNNTypeChecker.validate_gnn_files` is tri-state: `True` when every file
validated, `2` when one or more files were invalid or none was found, and
`False` on a hard failure. `_check_types` tested it for truthiness, so `2`
passed and the arm printed `"result": "verified"` for an invalid model set.

Reproduced with the pinned parser (v5.0.3, revision 8f293d9) on a sketch
declaring `A[3,3]` against a 2x2 likelihood matrix: the checker returned `2`
and `python .github/scripts/gate.py gnn` exited 0 with a `verified` receipt.
The arm now passes only on `is True` and the refusal names the returned value.

`.github/scripts/tests/test_gnn_conformance.py` pins `True`, `2`, `False`,
`None` and `0` without needing the parser installed, and proves the arm emits
no receipt for a refused verdict. `.github/GNN.md` and `docs/RUN_GUIDE.md`
record the tri-state contract.
@docxology
docxology merged commit 98a1445 into main Nov 12, 2026
0 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant