Repository navigation
fix(ci): serialize toolbox publishing; two builds could push latest at once - #98
Merged
Merged
Conversation
… at once build-toolbox.yml had no concurrency guard, and that was not a theoretical gap: the #91 and #93 builds ran 25 seconds apart, both pushed `latest`, and for iac-toolbox the EARLIER run's push landed last. `latest` named the older build while the newer digest existed only under its `sha-` tag. The group omits github.ref deliberately. The registry tags are global, so a workflow_dispatch from another branch has to queue behind main rather than get its own lane. cancel-in-progress is false, matching tool-bump.yml and for the same reason. These jobs push and sign six images; killing a run mid-flight can leave the base published and the five domain images not, which is a worse state than a slower queue. Nothing in flight is interrupted, runs serialize, and the newest commit's run finishes last so it wins `latest`. GitHub keeps one pending run per group, so a burst of pushes collapses to the one running plus the newest, which is the right pair to keep. Found while moving the image pins, by comparing what the registry served against what the build log said it published, rather than trusting either on its own. Nothing broke because of it: scripts/check-image-pins.py and the pin mover both resolve the per-commit `sha-` tag precisely so a raced `latest` cannot be pinned. `latest` is still what a human pulling the image by tag gets, which is why this is worth fixing rather than routing around.
garymike
added a commit
that referenced
this pull request
Oct 30, 2026
… the check is green (#99) The mover had been a throwaway script outside the repo, so the one rule that makes it safe to use lived only in whoever last ran it. It moved the pins correctly twice today, and would have churned them a third time an hour later for no benefit. The two scripts answer different questions and routinely disagree: check-image-pins.py was the pinned image built from the current inputs? -> current move-image-pins.py does the pin name one specific build? -> would move Every rebuild publishes a new digest, because the Dockerfiles apt-get install current packages, so "names the newest build" goes false within a week of every pin move while "built from current inputs" stays true. Measured right after #98, whose commit touched no image input: all five digests changed and all five pins stayed current. Running the mover on that state would have repointed every caller at a functionally identical build. So it refuses while the check is green. --force covers the one case the check is blind to on purpose: the weekly rebuild exists to pick up BASE-IMAGE patches, which change what callers run without changing any input this repo tracks, so the check stays green while the pinned image ages. Pulling those through to callers is a real reason to move a pin, and it now has to be stated rather than inferred. Resolves the per-commit sha- tag rather than latest, and verifies every digest's build-inputs label against the repo's own fingerprint before writing, so a wrong digest cannot land even under --force. Both paths were exercised before this commit: refusal wrote nothing and exited 1, --force dry-run listed all five and wrote nothing. check-image-pins.py's failure messages now name the script that fixes them, so a red run says what to do rather than only what is wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
build-toolbox.ymlhad noconcurrency:guard. That was not a theoretical gap:So
latestnamed the older build while the newer digest existed only under itssha-tag.The two choices in here
The group omits
github.ref. The registry tags are global, so aworkflow_dispatchfrom another branch must queue behind main's runs rather than get its own lane.self-scan.ymlkeys on the ref because it is a per-ref check; this publishes to one shared place.cancel-in-progress: false, matchingtool-bump.ymland for the same reason. These jobs push and sign six images. Killing a run mid-flight can leave the base published and the five domain images not, which is a worse state than a slower queue. Nothing in flight is interrupted, runs serialize, and the newest commit's run finishes last so it winslatest. GitHub keeps only one pending run per group, so a burst of pushes collapses to the one running plus the newest, which is the right pair to keep.Why it is worth fixing rather than routing around
Nothing broke because of this race.
scripts/check-image-pins.pyand the pin mover both resolve the per-commitsha-tag precisely so a racedlatestcannot be pinned, which is why #97 landed correct digests. Butlatestis what a human pulling the image by tag gets, and what anything outside this repo's own tooling would reach for.It surfaced from cross-checking what the registry served against what the build log said it published, rather than trusting either on its own. Four of five images agreed; one did not.