Introduce conformance test harness

[henrybear327]

The foundation for executing gvisor and LTP conformance test suite.

---

Summary by cubic

Adds a conformance harness for comparing elfuse with a QEMU Linux reference against checked-in expectations, with hermetic fake tests covering the framework before real suites land.

Harness

• Supports elfuse, qemu, and host backends and records exits, signals, timeouts, and transport failures.
• Uses canonical IDs, JSONC selection and expectation files, bidirectional judging, quarantined retries, isolated reruns, and result reports.
• Verifies pinned payloads with content fingerprints and ELF64 checks, and provides pin-drift checks and expectation seeding.
• Exposes scripts/conformance and Make targets with stable exit codes, including optional-prerequisite skips and --require.

Integration

• Adds hermetic fake-suite selftests to make check; gVisor and LTP providers remain follow-ups.
• Adds a dedicated Conformance workflow with a stable required gate that reaches every macOS job and can report pin drift.
• Extends tests/qemu-runner.sh with stateful start/stop so QEMU can outlive its launching shell, and documents setup and operations.

^{Written for commit 5a47a3b. Summary will update on new commits.}

[jserv]

Merge docs/conformance-design.md into docs/conformance.md as they do overlap.

[cubic-dev-ai]

1 existing issue remains and 27 new issues found across 57 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/workflow-jobs.py">

<violation number="1" location="scripts/workflow-jobs.py:14">
P1: When a block-form `needs` or `runs-on` list contains a comment or blank line between items, `_VALUE` stops at the first item. The checker can then miss a macOS runner or dependency and report a gate as enforced when it is not. Allow comments and blank lines between sequence items and add a regression case.</violation>
</file>

<file name="tests/conformance/ids.py">

<violation number="1" location="tests/conformance/ids.py:20">
P2: is_valid() and parse() accept an id with a single trailing newline because the pattern ends in '$', which in Python regex matches just before a final newline. A line read from discovery or an expectations file that is not stripped (e.g. 'gvisor:foo\n') is therefore reported as canonical and used in matching/slugging, silently diverging from the real id. Anchor the end of the id with \Z instead of $ so only the exact string is accepted.</violation>
</file>

<file name="tests/conformance/selftest/test_update.py">

<violation number="1" location="tests/conformance/selftest/test_update.py:90">
P2: This test depends on filesystem permission bits being enforced, but chmod 0o500 does not stop writes from a process with CAP_DAC_OVERRIDE or euid 0. In this sandbox the write succeeds, refresh returns EXIT_CURRENT, and the assertEqual(rc, update.EXIT_ERROR) at line 95 fails, so the test breaks the suite in root/privileged CI. Skip it when the process can bypass permission checks (e.g. os.geteuid() == 0), or induce the write failure without relying on host permission semantics.</violation>
</file>

<file name="tests/conformance/selftest/test_cli.py">

<violation number="1" location="tests/conformance/selftest/test_cli.py:149">
P2: This selftest is not hermetic: it passes only because build/elfuse is absent. list_ids() with backend 'all' runs through the elfuse backend, and ElfuseBackend.prerequisites() skips only when build/elfuse is missing. On any checkout that has built the binary (`make`), `fake list --backend all` returns 0 instead of 77 and this test fails for a reason unrelated to what it checks. Make the second-backend selection independent of the build, or assert only on a checkout that guarantees the absent-binary state.</violation>
</file>

<file name="mk/conformance.mk">

<violation number="1" location="mk/conformance.mk:52">
P2: When `UPDATE_CHECK=0` is used, this recipe still passes `--check` and refuses to rewrite pins. Enable check mode only when the value equals `1`.</violation>

<violation number="2" location="mk/conformance.mk:52">
P2: When multiple suites are registered and an earlier suite has pin drift, `&&` prevents later suites from being checked or refreshed. Run every suite while preserving the highest-priority failure status.</violation>
</file>

<file name="tests/conformance/backends/elfuse.py">

<violation number="1" location="tests/conformance/backends/elfuse.py:85">
P2: Between the `ps` listing and the `SIGKILL` in reap_orphans there is a TOCTOU window: the listed fork-child can exit and its pid be recycled by an unrelated host process, which the unconditional `os.kill(pid, SIGKILL)` would then kill. Re-check the pid's current session and ppid (e.g. from /proc/<pid>/stat) immediately before signaling, or guard the kill so only a process still in the dead case's session is killed.</violation>
</file>

<file name="tests/conformance/selection.py">

<violation number="1" location="tests/conformance/selection.py:93">
P2: When a selection file sets `"declined": null` (the natural way to say "none"), parse() raises an uncaught TypeError ('NoneType' object is not iterable) instead of a SelectionError. `enabled` is explicitly guarded above, so a malformed `declined` should get the same treatment.</violation>
</file>

<file name="tests/conformance/providers/fake.py">

<violation number="1" location="tests/conformance/providers/fake.py:59">
P3: enumerate() indexes self.data["cases"][entry.group] directly, so an enabled group that is not declared under "cases" (say a rename in either file) raises an uncaught KeyError that aborts selection/dry-run instead of a clear error. Validate that every enabled group is present in the "cases" map, or fall back to an empty set and surface the mismatch as a lint/selection error.</violation>
</file>

<file name="tests/conformance/backends/proc.py">

<violation number="1" location="tests/conformance/backends/proc.py:63">
P2: On a normal exit (no TimeoutExpired), run_local returns without killing or reaping the process group, so any background child the case spawned (e.g. a daemon, a fifo waiter, or `sh -c 'cmd &'`) keeps running after the invocation is recorded. In a suite running thousands of scripts this can leak processes that later block on reused scratch paths or hold locks and interfere with subsequent invocations. The group is only killed in the timeout branch. Kill and reap the group on the normal path too (or document that leaking is intended).</violation>

<violation number="2" location="tests/conformance/backends/proc.py:67">
P3: For a timed-out run, `wall_us` is measured after `_kill_and_reap`, so the recorded wall time includes the up-to-KILL_WAIT_S (30s) kill-and-wait in addition to `timeout_s`. A child stuck in uninterruptible sleep therefore reports a wall time far larger than the configured timeout, which is an inaccurate measured outcome. Capture `wall_us` at the moment the timeout is detected (the wall time the case actually consumed) rather than after the teardown wait.</violation>
</file>

<file name="tests/conformance/runner.py">

<violation number="1" location="tests/conformance/runner.py:104">
P3: When --jobs>1, the ThreadPoolExecutor invokes the user-supplied `log` callable concurrently from worker threads inside run_batch. If the caller passes a non-thread-safe writer (e.g. Cli.out/print in verbose mode, or a buffer that is not just list.append), progress lines can interleave or corrupt output. Make the log calls from parallel workers safe, e.g. route them through a single-threaded writer or document that `log` must be thread-safe.</violation>
</file>

<file name="tests/conformance/backends/qemu.py">

<violation number="1" location="tests/conformance/backends/qemu.py:76">
P2: If qemu-runner ignores SIGTERM for longer than KILL_WAIT_S, the `run.communicate(timeout=KILL_WAIT_S)` inside the timeout handler raises TimeoutExpired instead of raising the intended BackendError. That raw exception bypasses the harness's `except BackendError` handling in start(), aborting the run with a traceback, and because Popen is used as a context manager whose `__exit__` waits without a timeout, the still-alive runner can hang the whole harness. Escalate the second timeout to SIGKILL and raise BackendError.</violation>
</file>

<file name="tests/conformance/backends/ssh.py">

<violation number="1" location="tests/conformance/backends/ssh.py:90">
P3: When `fetch` is set and the connection is lost before the sentinel completes, `run()` returns early without removing the guest's `/tmp/conf.*` scratch dir; because `fetch` non-empty also disables the EXIT trap in `remote_script`, the directory leaks. Install the trap regardless of `fetch` (or clean up before the early return) so a lost transport does not accumulate scratch dirs on a session that persists across cases.</violation>

<violation number="2" location="tests/conformance/backends/ssh.py:103">
P2: A real guest timeout can be misreported as a signal crash. `rc==137` is the guest wrapper's own report that the timeout fired, and it should determine the timeout classification; gating it on `inv.wall_us >= timeout_s * 1_000_000` makes a genuine timeout depend on host wall time, which under any guest-ahead clock skew lands below that threshold and is then turned into `signal` (SIGKILL) by `128 - rc`. Drop the host-wall clause and classify on `rc == 137` alone, leaving the host-side `run_local` timeout as the backstop for the uninterruptible-wait case.</violation>
</file>

<file name="tests/conformance/report.py">

<violation number="1" location="tests/conformance/report.py:50">
P3: Only results.json is written atomically; junit.xml (write_bytes) and summary.txt (write_text) can be left truncated or stale if the run is interrupted between the writes. A downstream JUnit consumer then reads a partial xml. Route all three outputs through an atomic write.</violation>

<violation number="2" location="tests/conformance/report.py:81">
P3: junit() strips XML-invalid control characters from testcase text but not from the name, classname, and message attributes set raw. A control character in a case id or backend silently produces invalid XML (ElementTree leaves it in attribute values unescaped, unlike text). Sanitize the attribute values with _xml_text too.</violation>

<violation number="3" location="tests/conformance/report.py:126">
P2: markdown() renders a bad lane as an error row, but only catches ValueError and KeyError. A non-dict entry in `cases` makes CaseResult.from_dict raise TypeError, and an unreadable results.json raises OSError, so either one crashes the entire CI step summary instead of rendering that lane as an error row. Catch TypeError and OSError alongside the two.</violation>
</file>

<file name="tests/conformance/payload.py">

<violation number="1" location="tests/conformance/payload.py:44">
P2: When builder inputs share a basename, fingerprint() hashes only path.name plus content, so moving a same-named, same-content file between directories does not change the fingerprint and a stale payload goes undetected. Hash each file's relative-to-root path (not just its name) so input identity is captured.</violation>
</file>

<file name="tests/conformance/selftest/test_payload.py">

<violation number="1" location="tests/conformance/selftest/test_payload.py:142">
P3: This test claims to verify atomic rollback but the rejected document fails in check_pins, which write_pins calls before atomic_write. atomic_write is never entered on this path, so its except-BaseException cleanup (os.unlink of the temp file) is never exercised and the single-file-directory assertion trivially passes. Drive a failure from inside atomic_write (e.g. make the directory read-only after a valid check_pins, or call atomic_write directly with an unwritable target) to actually test the rollback branch.</violation>
</file>

<file name="tests/conformance/seed.py">

<violation number="1" location="tests/conformance/seed.py:59">
P2: The group collapse writes `<suite>:<group>/*` whenever every case present in `cases` shares one action and `whole_groups` is true, but it never verifies that `cases` actually covers the whole group; `whole_groups` is just `scope != "test"`. A group enumerated through a pr-scope `only` subset (or any partial run whose scope is not "test") collapses too, so the seeded `*` matcher also pins cases that were not run and may pass, producing false UNEXPECTED_PASS reds later. Check the requested count of cases per group against the group's full membership before collapsing, instead of relying on the scope string.</violation>
</file>

<file name="tests/conformance/backends/base.py">

<violation number="1" location="tests/conformance/backends/base.py:77">
P3: The non-blocking flock failure path reports every OSError as “another session holds the lock”, but only EWOULDBLOCK/EAGAIN (and on Linux EACCES) mean contention. An ENOLCK, EINTR, or unsupported-filesystem error is misreported to the caller as contention, hiding the real cause. Check the errno and only treat contention as BackendError, propagating other OSErrors as-is.</violation>
</file>

<file name="tests/conformance/cli.py">

<violation number="1" location="tests/conformance/cli.py:98">
P2: In `started()`, a `backend.start()` failure is reported through `self.skip()`, which returns the skip code 77 (unless `--require` is set). A present-but-unable-to-boot backend is then indistinguishable from a missing prerequisite, so CI treats a broken VM/reference as a pass and masks real runtime regressions. `make_backend()` already uses 77 correctly for prerequisites; a start failure should be an error instead. Raise `_Exit(EXIT_RED)` (or `_Exit(EXIT_USAGE)` for configuration) when `start()` raises.</violation>
</file>

<file name="tests/qemu-runner.sh">

<violation number="1" location="tests/qemu-runner.sh:294">
P2: When `stop --state-file` is run later against a stale state file, qemu_stop kills whatever pid the pidfile names without verifying it is the qemu process. If that VM already died (guests exit on panic via -no-reboot) and the pid got recycled, `kill`/`kill -9` is sent to an unrelated process. Verify the pid belongs to qemu (e.g. `ps -p "$pid" -o comm=` matches qemu) before killing, and only then clean up the rundir.</violation>
</file>

<file name=".github/workflows/conformance.yml">

<violation number="1" location=".github/workflows/conformance.yml:37">
P3: The workflow-level `CONF_SCOPE` env is computed but never consumed anywhere in this file. Workflow `env:` is scoped to this run and cannot reach build.yml, and no step here invokes `make test-conformance` (the only code path that reads `CONF_SCOPE`, mk/conformance.mk). Either wire it into the upcoming macOS conformance job in this workflow or drop the two lines so the dead scope calculation can't mislead.</violation>
</file>

<file name="tests/conformance/expectations.py">

<violation number="1" location="tests/conformance/expectations.py:21">
P3: The `since` regex `_SINCE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")` accepts impossible dates such as `2026-13-40` or `2026-02-31`, so a typo'd `since` value passes lint and lands in shipped expectations. Validate the parsed month/day ranges (or use `datetime.date.fromisoformat`) after the format check.</violation>

<violation number="2" location="tests/conformance/expectations.py:79">
P3: `resolve()` ends with `assert chosen is not None` and then dereferences `chosen.type`. The invariant (first effective action is `expect_pass` on `"*"`, which `fnmatch` matches against every id) currently guarantees `chosen` is set, but `assert` statements are stripped under `python -O`, turning the guarantee into an `AttributeError: 'NoneType'` on a broken expectations document instead of a clear `ExpectationError`. Prefer an explicit `if chosen is None: raise ExpectationError(...)`.</violation>
</file>

<sub>Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: When a block-form needs or runs-on list contains a comment or blank line between items, _VALUE stops at the first item. The checker can then miss a macOS runner or dependency and report a gate as enforced when it is not. Allow comments and blank lines between sequence items and add a regression case.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/workflow-jobs.py, line 14:

<comment>When a block-form `needs` or `runs-on` list contains a comment or blank line between items, `_VALUE` stops at the first item. The checker can then miss a macOS runner or dependency and report a gate as enforced when it is not. Allow comments and blank lines between sequence items and add a regression case.</comment>

<file context>
@@ -0,0 +1,139 @@
+import re
+import sys
+
+_VALUE = r"[ \t]*(\[[^\]]*\]|[^\n]*(?:\n[ \t]+-[ \t]*[^\n]*)*)"
+
+
</file context>

[cubic-dev-ai]

P2: is_valid() and parse() accept an id with a single trailing newline because the pattern ends in '$', which in Python regex matches just before a final newline. A line read from discovery or an expectations file that is not stripped (e.g. 'gvisor:foo\n') is therefore reported as canonical and used in matching/slugging, silently diverging from the real id. Anchor the end of the id with \Z instead of $ so only the exact string is accepted.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/conformance/ids.py, line 20:

<comment>is_valid() and parse() accept an id with a single trailing newline because the pattern ends in '$', which in Python regex matches just before a final newline. A line read from discovery or an expectations file that is not stripped (e.g. 'gvisor:foo\n') is therefore reported as canonical and used in matching/slugging, silently diverging from the real id. Anchor the end of the id with \Z instead of $ so only the exact string is accepted.</comment>

<file context>
@@ -0,0 +1,64 @@
+
+_ID_RE = re.compile(
+    r"^(?P<suite>[a-z][a-z0-9]*):(?P<group>[A-Za-z0-9_][A-Za-z0-9_.-]*)"
+    r"(?P<case>(/[A-Za-z0-9_.-]+)*)$"
+)
+
</file context>

Suggested change

	r"(?P<case>(/[A-Za-z0-9_.-]+)*)$"
	r"(?P<case>(/[A-Za-z0-9_.-]+)*)\Z"

[cubic-dev-ai]

P2: This test depends on filesystem permission bits being enforced, but chmod 0o500 does not stop writes from a process with CAP_DAC_OVERRIDE or euid 0. In this sandbox the write succeeds, refresh returns EXIT_CURRENT, and the assertEqual(rc, update.EXIT_ERROR) at line 95 fails, so the test breaks the suite in root/privileged CI. Skip it when the process can bypass permission checks (e.g. os.geteuid() == 0), or induce the write failure without relying on host permission semantics.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/conformance/selftest/test_update.py, line 90:

<comment>This test depends on filesystem permission bits being enforced, but chmod 0o500 does not stop writes from a process with CAP_DAC_OVERRIDE or euid 0. In this sandbox the write succeeds, refresh returns EXIT_CURRENT, and the assertEqual(rc, update.EXIT_ERROR) at line 95 fails, so the test breaks the suite in root/privileged CI. Skip it when the process can bypass permission checks (e.g. os.geteuid() == 0), or induce the write failure without relying on host permission semantics.</comment>

<file context>
@@ -0,0 +1,101 @@
+        self.assertEqual(self.path.read_text(), self.before)
+
+    def test_an_unwritable_pins_directory_is_an_error(self):
+        os.chmod(self.tmp.name, 0o500)
+        try:
+            rc = update.refresh(Stub(self.path, "b" * 40), out=self.lines.append)
</file context>

[cubic-dev-ai]

P2: This selftest is not hermetic: it passes only because build/elfuse is absent. list_ids() with backend 'all' runs through the elfuse backend, and ElfuseBackend.prerequisites() skips only when build/elfuse is missing. On any checkout that has built the binary (make), fake list --backend all returns 0 instead of 77 and this test fails for a reason unrelated to what it checks. Make the second-backend selection independent of the build, or assert only on a checkout that guarantees the absent-binary state.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/conformance/selftest/test_cli.py, line 149:

<comment>This selftest is not hermetic: it passes only because build/elfuse is absent. list_ids() with backend 'all' runs through the elfuse backend, and ElfuseBackend.prerequisites() skips only when build/elfuse is missing. On any checkout that has built the binary (`make`), `fake list --backend all` returns 0 instead of 77 and this test fails for a reason unrelated to what it checks. Make the second-backend selection independent of the build, or assert only on a checkout that guarantees the absent-binary state.</comment>

<file context>
@@ -0,0 +1,205 @@
+        self.assertEqual(rc, 2, out)
+        self.assertNotIn("Traceback", out)
+
+    def test_list_on_all_backends_never_boots_the_reference(self):
+        rc, out = self.run_cli("fake", "list", "--backend", "all")
+        self.assertEqual(rc, 77, out)
</file context>

[cubic-dev-ai]

P2: When UPDATE_CHECK=0 is used, this recipe still passes --check and refuses to rewrite pins. Enable check mode only when the value equals 1.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At mk/conformance.mk, line 52:

<comment>When `UPDATE_CHECK=0` is used, this recipe still passes `--check` and refuses to rewrite pins. Enable check mode only when the value equals `1`.</comment>

<file context>
@@ -0,0 +1,52 @@
+## Refresh the conformance pins from upstream (UPDATE_CHECK=1 to report only)
+update-pins:
+	$(if $(CONF_SUITES),,@printf "$(YELLOW)SKIP$(RESET) no conformance suites registered\n")
+	$(foreach s,$(CONF_SUITES),$(CONFORMANCE) $(s) update $(if $(UPDATE_CHECK),--check) $(if $(CONF_REF_$(s)),--ref $(CONF_REF_$(s))) &&) true
</file context>

Suggested change

	$(foreach s,$(CONF_SUITES),$(CONFORMANCE) $(s) update $(if $(UPDATE_CHECK),--check) $(if $(CONF_REF_$(s)),--ref $(CONF_REF_$(s))) &&) true
	$(foreach s,$(CONF_SUITES),$(CONFORMANCE) $(s) update $(if $(filter 1,$(UPDATE_CHECK)),--check) $(if $(CONF_REF_$(s)),--ref $(CONF_REF_$(s))) &&) true

[cubic-dev-ai]

P3: junit() strips XML-invalid control characters from testcase text but not from the name, classname, and message attributes set raw. A control character in a case id or backend silently produces invalid XML (ElementTree leaves it in attribute values unescaped, unlike text). Sanitize the attribute values with _xml_text too.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/conformance/report.py, line 81:

<comment>junit() strips XML-invalid control characters from testcase text but not from the name, classname, and message attributes set raw. A control character in a case id or backend silently produces invalid XML (ElementTree leaves it in attribute values unescaped, unlike text). Sanitize the attribute values with _xml_text too.</comment>

<file context>
@@ -0,0 +1,134 @@
+                       failures=str(sum(1 for c in cases if c.verdict.is_red)),
+                       skipped=str(sum(1 for c in cases if c.verdict is Verdict.FILTERED)))
+    for c in cases:
+        tc = ET.SubElement(suite, "testcase", name=c.id, classname=c.backend,
+                           time="%.3f" % (sum(a.invocation.wall_us for a in c.attempts) / 1e6))
+        if c.verdict.is_red:
</file context>

[cubic-dev-ai]

P3: Only results.json is written atomically; junit.xml (write_bytes) and summary.txt (write_text) can be left truncated or stale if the run is interrupted between the writes. A downstream JUnit consumer then reads a partial xml. Route all three outputs through an atomic write.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/conformance/report.py, line 50:

<comment>Only results.json is written atomically; junit.xml (write_bytes) and summary.txt (write_text) can be left truncated or stale if the run is interrupted between the writes. A downstream JUnit consumer then reads a partial xml. Route all three outputs through an atomic write.</comment>

<file context>
@@ -0,0 +1,134 @@
+    doc = {"schema_version": 1, "run": dict(meta), "gate": gate(cases),
+           "counts": counts(cases), "cases": [c.to_dict() for c in cases]}
+    payload.atomic_write(results_dir / RESULTS, json.dumps(doc, indent=1, sort_keys=True) + "\n")
+    (results_dir / "junit.xml").write_bytes(junit(meta, cases))
+    (results_dir / "summary.txt").write_text("\n".join(summary_lines(meta, cases, results_dir)) + "\n")
+    return doc
</file context>

[cubic-dev-ai]

P3: The workflow-level CONF_SCOPE env is computed but never consumed anywhere in this file. Workflow env: is scoped to this run and cannot reach build.yml, and no step here invokes make test-conformance (the only code path that reads CONF_SCOPE, mk/conformance.mk). Either wire it into the upcoming macOS conformance job in this workflow or drop the two lines so the dead scope calculation can't mislead.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/conformance.yml, line 37:

<comment>The workflow-level `CONF_SCOPE` env is computed but never consumed anywhere in this file. Workflow `env:` is scoped to this run and cannot reach build.yml, and no step here invokes `make test-conformance` (the only code path that reads `CONF_SCOPE`, mk/conformance.mk). Either wire it into the upcoming macOS conformance job in this workflow or drop the two lines so the dead scope calculation can't mislead.</comment>

<file context>
@@ -0,0 +1,131 @@
+  contents: read
+
+env:
+  CONF_SCOPE: ${{ (github.event_name == 'schedule' || inputs.scope == 'full') && 'full' || 'pr' }}
+
+jobs:
</file context>

[cubic-dev-ai]

P3: resolve() ends with assert chosen is not None and then dereferences chosen.type. The invariant (first effective action is expect_pass on "*", which fnmatch matches against every id) currently guarantees chosen is set, but assert statements are stripped under python -O, turning the guarantee into an AttributeError: 'NoneType' on a broken expectations document instead of a clear ExpectationError. Prefer an explicit if chosen is None: raise ExpectationError(...).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/conformance/expectations.py, line 79:

<comment>`resolve()` ends with `assert chosen is not None` and then dereferences `chosen.type`. The invariant (first effective action is `expect_pass` on `"*"`, which `fnmatch` matches against every id) currently guarantees `chosen` is set, but `assert` statements are stripped under `python -O`, turning the guarantee into an `AttributeError: 'NoneType'` on a broken expectations document instead of a clear `ExpectationError`. Prefer an explicit `if chosen is None: raise ExpectationError(...)`.</comment>

<file context>
@@ -0,0 +1,232 @@
+                        quarantined = True
+                    else:
+                        chosen, chosen_matcher = action, m
+        assert chosen is not None
+        return Resolution(
+            type=chosen.type,
</file context>

Suggested change

	assert chosen is not None
	if chosen is None:
	raise ExpectationError("no action matched %r" % test_id)

[cubic-dev-ai]

P3: The since regex _SINCE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$") accepts impossible dates such as 2026-13-40 or 2026-02-31, so a typo'd since value passes lint and lands in shipped expectations. Validate the parsed month/day ranges (or use datetime.date.fromisoformat) after the format check.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/conformance/expectations.py, line 21:

<comment>The `since` regex `_SINCE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")` accepts impossible dates such as `2026-13-40` or `2026-02-31`, so a typo'd `since` value passes lint and lands in shipped expectations. Validate the parsed month/day ranges (or use `datetime.date.fromisoformat`) after the format check.</comment>

<file context>
@@ -0,0 +1,232 @@
+ACTION_TYPES = ("expect_pass", "expect_failure", "expect_conf", "skip", "quarantine")
+_ACTION_KEYS = {"type", "matchers", "reason", "since", "tracking"}
+_TRACKING_RE = re.compile(r"^(#\d+|https?://\S+)$")
+_SINCE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
+SEEDED_PREFIX = "seeded from "
+FLAKY = "flaky.jsonc"
</file context>

[henrybear327]

Closing in favor of the smaller and more reviewable rewrite. (The initial feedback and iteration with Cubic is completed)

[jserv]

Closing in favor of the smaller and more reviewable rewrite. (The initial feedback and iteration with Cubic is completed)

Consider to address the above policy in skills.