Skip to content

fix(ci): serialize toolbox publishing; two builds could push latest at once - #98

Merged
garymike merged 1 commit into
mainfrom
fix/build-toolbox-concurrency
Oct 30, 2026
Merged

garymike merged 1 commit into
mainfrom
fix/build-toolbox-concurrency

Conversation

@garymike

Copy link
Copy Markdown
Owner

build-toolbox.yml had no concurrency: guard. That was not a theoretical gap:

#91 build started  18:31:02
#93 build started  18:31:27      <- 25 seconds later, concurrent

774ab22 (#93) build published iac-toolbox:  c1d133bb0e97
latest resolved to:                         7f5292725111   <- the EARLIER run's push landed last

So latest named the older build while the newer digest existed only under its sha- tag.

The two choices in here

The group omits github.ref. The registry tags are global, so a workflow_dispatch from another branch must queue behind main's runs rather than get its own lane. self-scan.yml keys on the ref because it is a per-ref check; this publishes to one shared place.

cancel-in-progress: false, matching tool-bump.yml and for the same reason. These jobs push and sign six images. Killing a run mid-flight can leave the base published and the five domain images not, which is a worse state than a slower queue. Nothing in flight is interrupted, runs serialize, and the newest commit's run finishes last so it wins latest. GitHub keeps only one pending run per group, so a burst of pushes collapses to the one running plus the newest, which is the right pair to keep.

Why it is worth fixing rather than routing around

Nothing broke because of this race. scripts/check-image-pins.py and the pin mover both resolve the per-commit sha- tag precisely so a raced latest cannot be pinned, which is why #97 landed correct digests. But latest is what a human pulling the image by tag gets, and what anything outside this repo's own tooling would reach for.

It surfaced from cross-checking what the registry served against what the build log said it published, rather than trusting either on its own. Four of five images agreed; one did not.

… at once

build-toolbox.yml had no concurrency guard, and that was not a theoretical gap: the #91 and
#93 builds ran 25 seconds apart, both pushed `latest`, and for iac-toolbox the EARLIER run's
push landed last. `latest` named the older build while the newer digest existed only under
its `sha-` tag.

The group omits github.ref deliberately. The registry tags are global, so a
workflow_dispatch from another branch has to queue behind main rather than get its own lane.

cancel-in-progress is false, matching tool-bump.yml and for the same reason. These jobs push
and sign six images; killing a run mid-flight can leave the base published and the five
domain images not, which is a worse state than a slower queue. Nothing in flight is
interrupted, runs serialize, and the newest commit's run finishes last so it wins `latest`.
GitHub keeps one pending run per group, so a burst of pushes collapses to the one running
plus the newest, which is the right pair to keep.

Found while moving the image pins, by comparing what the registry served against what the
build log said it published, rather than trusting either on its own. Nothing broke because of
it: scripts/check-image-pins.py and the pin mover both resolve the per-commit `sha-` tag
precisely so a raced `latest` cannot be pinned. `latest` is still what a human pulling the
image by tag gets, which is why this is worth fixing rather than routing around.
@garymike
garymike merged commit 278dffe into main Oct 30, 2026
9 checks passed
@garymike
garymike deleted the fix/build-toolbox-concurrency branch October 30, 2026 20:26
garymike added a commit that referenced this pull request Oct 30, 2026
… the check is green (#99)

The mover had been a throwaway script outside the repo, so the one rule that makes it safe to
use lived only in whoever last ran it. It moved the pins correctly twice today, and would
have churned them a third time an hour later for no benefit.

The two scripts answer different questions and routinely disagree:

  check-image-pins.py   was the pinned image built from the current inputs?   -> current
  move-image-pins.py    does the pin name one specific build?                 -> would move

Every rebuild publishes a new digest, because the Dockerfiles apt-get install current
packages, so "names the newest build" goes false within a week of every pin move while "built
from current inputs" stays true. Measured right after #98, whose commit touched no image
input: all five digests changed and all five pins stayed current. Running the mover on that
state would have repointed every caller at a functionally identical build.

So it refuses while the check is green. --force covers the one case the check is blind to on
purpose: the weekly rebuild exists to pick up BASE-IMAGE patches, which change what callers
run without changing any input this repo tracks, so the check stays green while the pinned
image ages. Pulling those through to callers is a real reason to move a pin, and it now has
to be stated rather than inferred.

Resolves the per-commit sha- tag rather than latest, and verifies every digest's build-inputs
label against the repo's own fingerprint before writing, so a wrong digest cannot land even
under --force. Both paths were exercised before this commit: refusal wrote nothing and exited
1, --force dry-run listed all five and wrote nothing.

check-image-pins.py's failure messages now name the script that fixes them, so a red run says
what to do rather than only what is wrong.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant