Repository navigation
ci(e2e): key the tmachine disk cache on the setup layer hash - #5628
Draft
matthewgrossman wants to merge 3 commits into
Draft
matthewgrossman wants to merge 3 commits into
matthewgrossman wants to merge 3 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Add `tmachine layer-key <environment>`, which prints the hash tmachine names the environment's setup layer after, or `none` when the environment has no setup playbooks, without starting a VM. setup() now derives its layer name from the same function. Key the e2e-suites disk cache on that value instead of a hand-written hashFiles list, so a new setup input changes the key without a workflow edit. Skip the cache for environments without a setup layer. Closes #5627 Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The integration runner cached all of ~/.cache/tmachine, install layers included, under one fixed key per OS and arch, and skipped ubuntu-k3s. Cache only the setup layers and key them on `tmachine layer-key`, as e2e-suites.yml does. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
force-pushed
the
ci/5627-tmachine-layer-key/matthewgrossman
branch
from
November 17, 2026 06:56
9591141 to
4a53efd
Compare
Print the setup-layer hash for every environment instead of `none` for environments without setup playbooks, and drop the workflow checks for it. Expose the existing hash function directly, and drop the tests and the Ansible hashing split that only they needed. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI now asks tmachine for the fingerprint of an environment's prepared disk and uses it as the cache key. Before, CI guessed it from a hand-written file list. When tmachine's inputs change, the cache key now changes with them, with no workflow edit.
Depends on #5608 (this PR is based on its branch, which limits the cache to
~/.cache/tmachine/disks/*.qcow2). Retarget tomainafter #5608 merges.Related Issue
Closes #5627
Background
What tmachine caches. The E2E tests run inside VMs. Before tests can run, tmachine prepares each VM (installs Docker or Podman, test tools, Keycloak and so on), which takes several minutes. It saves the prepared VM as a disk file so it doesn't repeat that work.
How tmachine names the disk. It combines everything that went into the preparation (the OS base image, the Ansible scripts, the environment's playbooks and extra input files such as the Keycloak realm file) into one fingerprint, and names the file
disks/<fingerprint>.qcow2. When any of those inputs change, the fingerprint changes, and tmachine prepares a fresh disk.How CI keeps the disks between runs. CI runners start empty, so the workflow saves the disk files with GitHub's cache at the end of a run and restores them at the start of the next. GitHub files each cache entry under a key. If the key matches, it restores the files. After a matching key, it does not save again, even if the files changed.
The problem. The workflow built its key from its own hand-written list of files, which was supposed to match tmachine's inputs but wasn't checked. When someone added an input to tmachine but not to the list:
The fix. tmachine gets a
layer-keycommand that prints the fingerprint without starting a VM. The workflow runs it first and uses the output as the cache key. The hand-written list is gone.setup()andlayer-keycall the same function, so the key and the disk name can't disagree.Changes
setup_hashintests/tmachine/src/qemu/setup.rsis now the publicsetup_layer_key.setup()still uses it to name the disk.tmachine layer-key <environment>command prints that fingerprint.e2e-suites.yml: a new step builds tmachine and runslayer-key, and the cache key istmachine-<os>-<arch>-<fingerprint>. ThehashFiles(...)list and its comment about keeping it in sync are deleted.integration-runner.yml: this workflow runs on every Branch E2E run aspodman-e2e-integration(it bootsfedora-podman-rootless). It had the same problem in a worse form: one fixed key,tmachine-<os>-<arch>, that never changed, so it kept restoring a stale cache, and it cached all of~/.cache/tmachine, including install layers that hold the commit's build. It also skippedubuntu-k3swithout a stated reason. It now uses the samelayer-keystep, thedisks/*.qcow2path and the same key. The k3s skip is removed: with install layers out of the cache and k3s credentials wiped from the setup layer (ci(e2e): cut fixed setup and teardown costs of the k3s conformance rows #5608), k3s setup disks are as safe to cache here as ine2e-suites.yml. Both workflows produce the same key for the same environment, so they can share a cache entry.Cost and other details
openshell-test-runnerdepends on the same tmachine config, and the prepared disk is built on top of the base image, so the run step already downloads them. On Cachix, the Ubuntu image and what it depends on are about 623 MB. The file actually read for the fingerprint is a small overlay (28 MB for Ubuntu, 48 MB for Fedora), so hashing takes well under a second.e2e-suites.ymldidn't build tmachine itself. The new step addsnix build .#tmachine. Its tools (qemu, ansible, git, sshpass) are the same ones the test runner already downloads. The only new item is the tmachine program. After a change totests/tmachine/srcor its lockfile, no copy is on Cachix yet, so each CI job compiles it. That took about 30 s locally on macOS. WithCACHIX_AUTH_TOKENset, later runs download it.setup()uses the base image directly when an environment has no setup playbooks, so there's no disk to cache.layer-keystill prints a fingerprint for these, and GitHub's cache would log a "nothing to save" warning. No current environment is like this.disks/installand are not cached. The environment name is no longer part of the key. Each environment's fingerprint already differs, and two environments with the same fingerprint would use the same disk anyway.Testing
cargo fmt,cargo clippy --all-targets -- -D warningsandcargo testintests/tmachinepass.cargo checkintests/runner, which depends on the crate, passes.nix run .#tmachine -- layer-keylocally on aarch64-darwin forubuntu-docker-rootful,ubuntu-k3sandfedora-podman-rootless-keycloak. Each printed a 64-character fingerprint in about 1.3 s once built. An unknown environment fails withenvironment "nope" is not defined.actionlintone2e-suites.ymlandintegration-runner.ymlis clean.zizmorreports the same existing findings as the base branch for both.Checklist