Skip to content

docs(testing): document how to write testkit tests and add test environments #5683

Open

Description

@matthewgrossman

User Story

As a contributor adding E2E coverage, I want one guide that tells me where a new test goes, how to write it with testkit, and how to add an environment when no existing one can run it, so I can add and validate a test without reading the runner, tmachine and workflow code.

Description

Part of this is documented, but it's spread out and has gaps:

  • Already covered: tests/testkit/README.md (writing a test, Context, cleanup, programs). tests/suites/conformance/README.md (adding a conformance test, what it may assert). TESTING.md (runner, adding a feature suite, feature environments such as -keycloak and -cdi).
  • Gaps:
    • Where a test goes. No decision guide covers a crate test, conformance, features/<name> and drivers/<name>. The guide should also say how to name a feature package: for why it isn't in conformance, not for what it tests.
    • Adding a driver suite. Only feature suites are documented. A drivers/<name> workspace (like drivers/podman) needs its own lockfile and alignment entry, a bundle, and rows. It should also stay self-contained so it can move under RFC 0022.
    • Adding a base environment or installer. Only <base>-<feature> environments are documented. A new distro, container engine, driver or network setup needs more, for example the corporate-proxy environment PR6 of test(e2e): decide which e2e/rust tests to port and in what order #5297 adds. That means config.nix bases, installer playbooks, the gateway the installer registers, the disk-cache inputs, and which rows use it.
    • Shared support crates. tests/support/ has no index. Contributors can't see which fixtures already exist (host-server, container-engine, podman, keycloak) or when to add one.
    • Validating a new test. There's no single path from local run to CI. It should cover running one test locally against a gateway, then in a tmachine environment (--filter, --keep), then on CI (a row plus the test:e2e label). It should also say which hosts can run which environments: KVM, macOS Hypervisor framework, same architecture only.
    • TESTING.md structure. About half of its 878 lines describe e2e/rust and e2e/python, which are being retired. testkit should be the main path, and the legacy sections should be marked for deletion.

Proposed shape:

  1. TESTING.md stays the entry point. It gets a short "Adding a test" section with the decision guide and links to the details.
  2. Each part's details live next to its code: testkit, conformance, tests/suites/features, tests/suites/drivers, tests/support, tests/tmachine and tests/runner. The last four have no README today.
  3. A contributor skill under .agents/skills/ covers the same steps, since agents write most of the ported tests. It points at the docs instead of repeating them.

Context

Definition of Done

  • TESTING.md has an "Adding a test" section that decides between a crate test, conformance, a feature suite and a driver suite, with naming rules.
  • The repository documents how to add a driver suite workspace and how to add a base environment or installer, alongside the existing feature-environment steps.
  • tests/support, tests/tmachine and tests/runner each have a README that says what's there and when to add to it.
  • A local-to-CI validation path is documented for a new test and for a new environment.
  • Legacy e2e/rust and e2e/python sections in TESTING.md are marked as retiring, so they can be deleted with the code.
  • A contributor skill for writing testkit tests exists and is listed in CONTRIBUTING.md.
  • Someone who didn't write the guide follows it to add a test, or the PR6 environment, and the gaps they hit are fixed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic:dxMaking OpenShell easy to contribute to for humans and agents.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions