Repository navigation
ci(e2e): cut fixed setup and teardown costs of the k3s conformance rows - #5608
Open
matthewgrossman wants to merge 3 commits into
Open
matthewgrossman wants to merge 3 commits into
matthewgrossman wants to merge 3 commits into
Conversation
Cache the tmachine setup layers for ubuntu-k3s too, and leave install layers out of the disk cache: they hold the build under test and, on k3s, the gateway's credentials. Install layers now live under disks/install so the cache can take only the setup layers, keyed per environment. Kill QEMU at session teardown instead of powering the guest off, since the session overlay is thrown away. Layer builds still power off cleanly. Pull the HA PostgreSQL fixture image in the k3s setup layer, and run the prebuilt test program with a 180-second per-test limit instead of the 900-second default. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
November 16, 2026 17:12
Member
Author
|
/ok to test d52deca |
This was referenced Nov 16, 2026
…ixed-cost change Leave per-test timeouts to the runner work that owns them, and stop pulling the PostgreSQL fixture image in the k3s setup layer, which saved no measurable time and tied the tmachine cache key to e2e/. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Member
Author
|
/ok to test 9c481ad |
The k3s setup layer boots k3s once, so it holds the cluster CA, token, node password and admin kubeconfig. Stop k3s and delete that state before the layer is saved, so each run's cluster generates its own credentials, and fail the layer build if any credential file remains. Also key the disk cache on ansible_hash.rs, which feeds the setup layer hash. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Member
Author
|
/ok to test d8d1a06 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cut the fixed setup and teardown costs of the k3s conformance rows without touching the tests: cache the k3s setup layers, and kill the throwaway test VM instead of powering it off.
How the k3s disk cache is made safe:
disks/installand are never cached; CI caches only the setup layers.Related Issue
Closes #5531
Changes
Wait for K3sruns before the layer is saved.k3s.yamlnow stops k3s withk3s-killall.sh, deletes that state, keeps the containerd image store, and fails the layer build if any credential file remains, so each run's cluster generates its own credentials on first start. Measured on an arm64 VM: about 1 s added to the layer build and about 2 s to a first start; the node and kube-system pods came up healthy with new credentials on three boots. The install layer, which holds the deployed gateway and the build under test, is still never cached: tmachine keeps it underdisks/install, ande2e-suites.ymlcaches onlydisks/*.qcow2. With no install layers in the cache, the key no longer includes the installer, so rows of one environment share an entry. The key also coversansible_hash.rs, which feeds the setup hash.integration-runner.ymlis unchanged: it no longer runs k3s by default and uses one shared cache key, which is a separate issue.Session::downkills QEMU instead of sendingsystem_powerdownand waiting up to 120 s; the clean shutdown took 3–35 s. This is safe because nothing the test VM writes outlives it:downis called.Follow-ups
hashFileslist: ci(tmachine): key the CI disk cache on tmachine's own layer hash #5627.Testing
cargo clippy -D warningsandcargo nextest runfortests/tmachineandtests/runneractionlintandzizmorone2e-suites.yml(no new findings),nix build .#tmachine-config,nix fmt,mise run markdown:lintTimings
Measured on an earlier head (with the PostgreSQL pre-pull, before the credential wipe). Baseline: runs 43312288101, 43257220972 and 43237169662 (ranges across the three). "Cold" is this PR's first attempt, which built the setup layer. "Cached" is the re-run that restored it (about 6 s to restore 506 MB).
The test phase varies by about ±50 s between runs, which accounts for most of the spread in job time. The new k3s cache entry is 506 MB, against about 1 GB for the old entries that included install layers.
Checklist