Skip to content

ci(nightly): the coverage job fetches the release tags (#2054) - #2055

Open
OffgridwithJD wants to merge 2 commits into
mainfrom
fix/2054-coverage-release-tags
Open

OffgridwithJD wants to merge 2 commits into
mainfrom
fix/2054-coverage-release-tags

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

Fixes the red nightly in #2054. It went red the first night after my #2043 and has stayed red since.

Cause

The coverage job runs from a plain actions/checkout, which fetches no tags. Before #2043 it ran the shell suites, which print a note for a tagless arm. Now it runs the pytest corpus, and there:

  • test_native_upgrade_converge.py's fixture arms (and, via test_harness_deps' no-driver run, test_docs_style.py's changelog arms) record UNMET_PRECONDITION: no release tags in this checkout;
  • test_harness_deps.py::test_the_guard_half_of_the_corpus_runs_without_a_database_driver gets rc 67 and fails.

The result both nights was 1 failed, 5354 passed (runs 42835020343 and 43032199185, coverage-suite-logs artifact). The job's own output didn't say which test failed.

Change

file what
.github/workflows/nightly.yml the coverage job fetches refs/tags/v1.0-alpha* at depth 1, the same step pytest-cluster uses
test/pytest/run_coverage.py prints each FAILED/ERROR line from pytest's short summary, so a red night names its test in the job log
test/pytest/test_coverage_report.py new derived guard test_the_coverage_job_fetches_the_release_tags_its_corpus_reads, plus a failed-test-named check in test_a_corpus_that_ran_nothing_or_failed_is_not_a_clean_report
test/pytest/TESTS.md, test_docs_style.py the test is documented; the docstring no longer says the coverage job is tagless

Evidence

  • New guard: the tag readers come from the corpus itself (files that record the tagless precondition), and the jobs from every workflow that runs run_coverage.py. With main's nightly.yml it's red: nightly.yml:coverage fetches the v1.0-alpha tags, lines before it runs run_coverage.py: got -28, wanted at least 1. Green here.
  • Failed-test naming: red on main's run_coverage.py (got ''), green here.
  • Gate: gpush passed with 895 guards, 5365 collected, census 9660/4894 unchanged and the self-test PASSED. No ledger or expected_tests change.
  • Not yet shown: the real proof is the next nightly going green, which closes nightly deep gate is red #2054 automatically. I haven't re-run the 33-minute corpus locally with tags.

🤖 Generated with Claude Code

actions/checkout fetches no tags. Since run_coverage.py replaced the shell
coverage run (#2043), the corpus's tag arms (native_upgrade_converge's
fixture blobs, and docs_style's changelog arms through test_harness_deps)
record UNMET_PRECONDITION from the plain checkout. The unrunnable
accounting then fails test_harness_deps, and the nightly has been red
every night since (#2054): 1 failed, 5354 passed.

- nightly.yml: the coverage job fetches the v1.0-alpha tags, as
  pytest-cluster does.
- run_coverage.py: each FAILED/ERROR line from pytest's short summary is
  printed, so a red night names its test without fetching the artifact.
- test_coverage_report.py: a derived guard. A corpus file reads the tags,
  so every workflow job that runs run_coverage.py fetches them first. Red
  on main's nightly.yml (gap -28), green here. The failing-corpus test
  now also requires the failed test to be named (red before the fix).

Closes #2054 once the next nightly is green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at d70eceac. What I checked:

  • Red on main, green here. On the branch, both changed tests in test_coverage_report.py pass. With main's nightly.yml and run_coverage.py put back, both fail for the reasons the description gives: nightly.yml:coverage fetches the v1.0-alpha tags, lines before it runs run_coverage.py: got -28, wanted at least 1 and names each failed test in its own output ...: got ''.
  • The cause, reproduced. In a --depth=1 --no-tags clone of d70eceac (0 tags), test_harness_deps.py gives 1 failed, 20 passed. The failure is test_the_guard_half_of_the_corpus_runs_without_a_database_driver. After running the new step's exact command (git fetch --depth=1 origin '+refs/tags/v1.0-alpha*:refs/tags/v1.0-alpha*'), which fetched v1.0-alpha through v1.0-alpha4, the same file gives 21 passed.
  • The diff. The fetch step is identical to pytest-cluster's (nightly.yml:843) and runs before run_coverage.py. The FAILED/ERROR echo is capped at 300 characters per line. The guard skips comment lines, so a commented-out fetch does not satisfy it.

Not checked: the full 33-minute coverage corpus with tags. The next nightly is that proof, as you say.

🤖 Generated with Claude Code

…#2054)

The guards job collects NO_CLUSTER and compares with expected_tests.txt;
the new test_the_coverage_job_fetches_the_release_tags_its_corpus_reads
made it 895.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jdatcmd

jdatcmd commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, one fix is needed before this can merge. I approved before CI had finished, and that was a mistake on my part.

pytest (harness guards, no database) is red at d70eceac (job 130912303568):

ERROR: collected 895 test(s) but expected 894.

The new guard test_the_coverage_job_fetches_the_release_tags_its_corpus_reads makes 895, but test/pytest/expected_tests.txt:367 still says guard_tests 894. Please raise it to 895, with a comment line above it, as the earlier changes have. The description's "no expected_tests change" needs correcting too.

Your push will change the head, so my approval on d70eceac will no longer apply. I'll review the new head again. Everything else in my review stands.

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nightly deep gate is red

2 participants