Repository navigation
Description
User Story
As a maintainer reading CI results, I want one table that shows which environments run which test suites at each CI stage, so I can tell what a green PR proves and what only runs on main or at release time.
Description
Today the answer is spread across five workflow files and a hard-coded JSON list. It takes reading YAML to learn, for example, that:
-
Branch Checks (every PR push and the merge queue) runs no test environment. Its only conformance step is
tasks/scripts/check-conformance-program.sh, which checks that the test lists match. -
Branch E2E (the
test:e2elabel, plustest:e2e-gpuandtest:e2e-kubernetes) runs every row ine2e-suites.yml:- conformance: 6 rows, including the external Docker driver;
- features: 14 rows;
drivers/podman: 8 rows;podman-e2e-integration;- the legacy Docker, Podman, VM and Kubernetes lanes.
-
Release Dev (every push to
main) and Release Qualification run:- conformance (5 rows, without the external driver);
- features;
- the legacy
docker-e2e(arm64) andvm-e2elanes.
They don't run
drivers/podmanor the legacy Kubernetes and Podman lanes, so those run only when someone adds a label to a PR.
The rows live in a rows= JSON string inside .github/workflows/e2e-suites.yml. Callers pick rows with the family and external-drivers inputs. Nothing lists them by stage.
Proposed design:
- Move the rows to one source file (for example
tests/ci-matrix.tomlor YAML). Each row names its environment, installer, suite and runner, plus astagesfield (branch-e2e,release-dev,release-qualification, and laterbranch-checks). e2e-suites.ymlreads the rows from that file and selects them by astageinput instead offamilyandexternal-drivers.- A generated table (stage × environment × suite) goes in the testing docs, with a check in Branch Checks that fails if the table is stale.
- Legacy host lanes (VM, GPU, Kubernetes, Podman scripts) appear in the same table as rows of their own type until they're deleted, so the table is complete.
Context
- Part of test: retire the old e2e test system onto testkit #5281. It makes the coverage changes from the
e2e/rustretirement (test(e2e): decide which e2e/rust tests to port and in what order #5297) visible as they happen. - Several choices need this view: which rows Release Dev should add (
drivers/podmandoesn't run onmaintoday), and the test-selection work in ci(e2e): pick the E2E tests a pull request needs from its diff #5679. - Keep tmachine unaware of tests: the row file belongs to CI and the runner, not tmachine.
Definition of Done
- One source file defines every tmachine row and the stages it runs in. Workflows read it; no workflow hard-codes a row list.
- The testing docs show a generated stage × environment × suite table, including the legacy host lanes while they exist.
- Branch Checks fails when the table and the source file disagree.
- The rows that run at each stage are unchanged by the move, or each change is listed and approved in the PR.
Activity