Repository navigation
Open
Labels
topic:dxMaking OpenShell easy to contribute to for humans and agents.Making OpenShell easy to contribute to for humans and agents.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
Description
User Story
As a contributor adding E2E coverage, I want one guide that tells me where a new test goes, how to write it with testkit, and how to add an environment when no existing one can run it, so I can add and validate a test without reading the runner, tmachine and workflow code.
Description
Part of this is documented, but it's spread out and has gaps:
- Already covered:
tests/testkit/README.md(writing a test,Context, cleanup, programs).tests/suites/conformance/README.md(adding a conformance test, what it may assert).TESTING.md(runner, adding a feature suite, feature environments such as-keycloakand-cdi). - Gaps:
- Where a test goes. No decision guide covers a crate test,
conformance,features/<name>anddrivers/<name>. The guide should also say how to name a feature package: for why it isn't in conformance, not for what it tests. - Adding a driver suite. Only feature suites are documented. A
drivers/<name>workspace (likedrivers/podman) needs its own lockfile and alignment entry, a bundle, and rows. It should also stay self-contained so it can move under RFC 0022. - Adding a base environment or installer. Only
<base>-<feature>environments are documented. A new distro, container engine, driver or network setup needs more, for example the corporate-proxy environment PR6 of test(e2e): decide which e2e/rust tests to port and in what order #5297 adds. That meansconfig.nixbases, installer playbooks, the gateway the installer registers, the disk-cache inputs, and which rows use it. - Shared support crates.
tests/support/has no index. Contributors can't see which fixtures already exist (host-server,container-engine,podman,keycloak) or when to add one. - Validating a new test. There's no single path from local run to CI. It should cover running one test locally against a gateway, then in a tmachine environment (
--filter,--keep), then on CI (a row plus thetest:e2elabel). It should also say which hosts can run which environments: KVM, macOS Hypervisor framework, same architecture only. TESTING.mdstructure. About half of its 878 lines describee2e/rustande2e/python, which are being retired. testkit should be the main path, and the legacy sections should be marked for deletion.
- Where a test goes. No decision guide covers a crate test,
Proposed shape:
TESTING.mdstays the entry point. It gets a short "Adding a test" section with the decision guide and links to the details.- Each part's details live next to its code: testkit, conformance,
tests/suites/features,tests/suites/drivers,tests/support,tests/tmachineandtests/runner. The last four have no README today. - A contributor skill under
.agents/skills/covers the same steps, since agents write most of the ported tests. It points at the docs instead of repeating them.
Context
- Part of test: retire the old e2e test system onto testkit #5281. The
e2e/rustports in test(e2e): decide which e2e/rust tests to port and in what order #5297 keep adding helpers and the occasional environment. They're the first users of this guide and a good check that it's complete. - Once ci(e2e): show which environments run which test suites at each CI stage #5678 moves the rows into one source file, the "add a row" steps here should point at that file.
- Keep tmachine unaware of tests, and keep testkit independent of provisioning. The environment docs describe the contract between them (
OPENSHELL_TEST_*variables), not test details. - Follow
docs/CONTRIBUTING.mdxstyle. This is contributor documentation, so it lives in the repository, not the published docs.
Definition of Done
-
TESTING.mdhas an "Adding a test" section that decides between a crate test, conformance, a feature suite and a driver suite, with naming rules. - The repository documents how to add a driver suite workspace and how to add a base environment or installer, alongside the existing feature-environment steps.
-
tests/support,tests/tmachineandtests/runnereach have a README that says what's there and when to add to it. - A local-to-CI validation path is documented for a new test and for a new environment.
- Legacy
e2e/rustande2e/pythonsections inTESTING.mdare marked as retiring, so they can be deleted with the code. - A contributor skill for writing testkit tests exists and is listed in
CONTRIBUTING.md. - Someone who didn't write the guide follows it to add a test, or the PR6 environment, and the gaps they hit are fixed.
Metadata
Metadata
Assignees
Labels
topic:dxMaking OpenShell easy to contribute to for humans and agents.Making OpenShell easy to contribute to for humans and agents.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
Activity