Repository navigation
feat(ci): fail the build run when the image pins are behind what it published - #88
Merged
Merged
Conversation
…ublished Fourth time today the five reusable-workflow digest pins went stale, so the manual step gets a control. #73 made those pins deliberate on purpose: a rebuild must not silently change what every caller scans with, after betterleaks 1.6.1 -> 1.8.1 in #64 turned green repos red overnight. The cost of "deliberate" is that nothing moves them for you, and nothing reminded anyone either. #80 bumped eight tools and #81 bumped SkillSpector, both published, neither moved the pins, and callers kept getting the previous toolchain while tools.lock said otherwise. #82 was the cleanup. Every toolbox/** change since has put them behind again, including #87 and #86. Each build job now records the digest it published, and a new `pins` job compares those against the pin in the matching reusable workflow. Two design points that are the whole reason this works: It is a SEPARATE job, after base and domain, so publishing is never blocked. It cannot be blocked: the digest to pin does not exist until the publish has happened, so failing the publish on a stale pin would be circular. The images ship, and the run still goes red, so the follow-up is not optional. Same contract as tool-drift.yml: red means a pin is behind, not that anything is broken. It reads steps.build.outputs.digest rather than the registry. Comparing against the `latest` index digest false-reds on images whose index carries attestations, which is exactly why I pulled the equivalent check back out of check-tool-updates.sh in #82 rather than ship a flaky gate. The build already holds both halves, so no lookup and no ambiguity. mcp-review-toolbox is built here but no reusable workflow pins it, so it is reported as unpinned rather than counted as a mismatch. Verified by running the comparison standalone against three scenarios: all pins matching exits 0; two behind exits 1 with both named in ::error:: and marked in the job summary; and no digest files at all exits 1 saying the pins were NOT verified, rather than passing quietly, which is the failure mode that let this go unnoticed in the first place. actionlint clean, zizmor clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
garymike
added a commit
that referenced
this pull request
Oct 21, 2026
…hed (#89) The first refresh prompted by the check rather than by remembering. #88 added a `pins` job to build-toolbox.yml; its own merge triggered a rebuild, the job ran, and it went red naming all five stale pins with the digest each should move to. A true positive on first firing, because #86 and #87 had already left them behind. security-toolbox-base d2f1d6b33d9c -> 9be060794ecb gha-toolbox 7e6558f3b46c -> 4cef3443acbd sast-toolbox 6bf6d9aafd5a -> edfd4db8752a iac-toolbox bc4198b93fcc -> 08e1338df20c skill-audit-toolbox c10bc17412a2 -> f5f99b96668f Digests taken from the job's own annotations, which report steps.build.outputs.digest from the run that published them. No local pull and no registry lookup, so neither the index-vs-manifest ambiguity from #82 nor a hand-copied digest can get in. Verified behind the new pins: base ships trufflehog 3.99.3, betterleaks 1.9.0, osv-scanner 2.7.0 and syft 1.55.0, with pip 26.2.2 vendoring certifi 2026.10.19, so #87's CVE fix is actually in what callers now get. skill-audit ships SkillSpector v2.13.0. One thing the check cannot currently survive: #86's build failed on a cosign/Rekor 409, "an equivalent entry already exists in the transparency log", because toolbox/gha was unchanged so gha-toolbox rebuilt to a byte identical digest that was already signed. A failed domain arm skips the pins job rather than running it. Benign cause, but it means a reproducible rebuild can silence the check. Noted, not fixed here. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
garymike
added a commit
that referenced
this pull request
Oct 21, 2026
…#90) Two faults, one cause, found because #86's build went red on both at once. A rebuild whose inputs have not changed produces a byte-identical digest, and Rekor rejects a second equivalent entry for it with 409 createLogEntryConflict. That is not a signing failure: the entry it conflicts with IS this digest's signature, already in the log from the earlier run. toolbox/gha was untouched by #86, gha-toolbox rebuilt to the same bytes, and the sign step failed. The sign step now matches createLogEntryConflict specifically and treats it as already-signed, with a notice explaining why. Any other cosign failure still fails: verified against four stubbed outcomes, where success and the 409 exit 0, and a missing OIDC token and a registry 403 both exit 1. The second fault is worse, and the 409 is what exposed it. A red domain arm skipped the pins job entirely, because needs: [base, domain] will not run on a partial build. So a reproducible rebuild could silence the very check added in #88 to watch the pins, while five images published with nobody comparing anything. It now runs with if: !cancelled(). That raises the question the first version did not have to answer: what to do about an image that never published. Nothing to compare, so it is reported as "published no digest in this run, so its pin was NOT checked" rather than dropped. Four greens must not read as a clean sweep when six images were expected. Verified on a simulated partial build: the four that published are confirmed current, and the two that did not are named as unchecked. The zero-digest case still fails outright rather than warning. On a totally failed build that is redundant with the build's own red, but the alternative is a job that reports success having compared nothing, which is the failure shape that let the pins drift unnoticed four times in the first place. actionlint clean, zizmor clean. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
garymike
added a commit
that referenced
this pull request
Oct 30, 2026
… never passed (#94) #88 compared each reusable-workflow image pin against the digest the build had just published. It failed on 4 of 4 runs after it landed, including the 2026-10-26 SCHEDULED rebuild where nothing in the repo had changed and all six image jobs passed. The premise was mine and it was wrong: a rebuild with unchanged inputs does not produce a byte-identical digest. The Dockerfiles apt-get install current packages, so a rebuild days apart publishes a new digest. The check could never stay green past the next rebuild, and because the weekly cron rebuilds on its own, "pins behind" stopped carrying information. That is the cry-wolf signal this repo refuses elsewhere. So it now compares something that only moves when a human moves it. build-toolbox.yml stamps every image it publishes with a `git ls-files -s` fingerprint of that image's build inputs: toolbox/base, the image's own context, and toolbox/tools.lock. The check reads that label back off the PINNED image and compares it with the repo's inputs now. Four small registry calls per image, resolving the attestation-bearing index to its one real platform manifest and reading only the config blob, so no layers are pulled. Verified against the live registry before shipping: the chain resolves for all five images, and the shell fingerprint in the workflow and the Python one in the script agree on all five. More precise, too. #93 changed only toolbox/skill-audit/rules/, which under the old check turned all five pins red; under this one only skill-audit.yml is affected. A pinned image with no stamp is reported UNVERIFIABLE rather than passing, so this cannot report success on absent input, which is the failure mode it replaces. Moved out of build-toolbox.yml into its own workflow. A red build should mean the images are broken; the pins being behind is a different fact and should read as one, the same separation tool-drift.yml already uses. Not run on pull_request on purpose: the digest to pin does not exist until the publish has happened, so a PR that bumps a tool could never be green, and a check that cannot pass on the PR that causes it is one people learn to ignore. Also corrects the reproducibility claim in both cosign sign steps. Tolerating a Rekor 409 is still right, but a rebuild is byte-identical only when it hits the buildx cache, not whenever its inputs are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The five reusable-workflow digest pins went stale four times today. This gives the manual step a control.
Why they keep going stale
#73 made those pins deliberate on purpose: a rebuild must not silently change what every caller scans with, after betterleaks 1.6.1→1.8.1 in #64 turned green repos red overnight. The cost of "deliberate" is that nothing moves them for you, and nothing reminded anyone either.
tools.lockclaimed otherwisetoolbox/**, so the pins are behind again right nowEach build job now records the digest it published, and a new
pinsjob compares those against the pin in the matching reusable workflow.Two design points, and they are the reason it works
It is a separate job, after
baseanddomain, so publishing is never blocked. It cannot be blocked: the digest to pin does not exist until the publish has happened, so failing a publish on a stale pin would be circular. The images ship and the run still goes red, so the follow-up is not optional. That is the same contracttool-drift.ymlalready states, where red means a pin is behind rather than something being broken.This is also what kept me from putting it in the drift gate or inline in the build steps. A red
basejob would skipdomainentirely and block five images from publishing.It reads
steps.build.outputs.digest, not the registry. Comparing against thelatestindex digest false-reds on images whose index carries attestations, which is precisely why I pulled the equivalent check back out ofcheck-tool-updates.shin #82 rather than ship a gate that cries wolf. The build already holds both halves, so there is no lookup and no index-versus-manifest ambiguity.mcp-review-toolboxis built here but no reusable workflow pins it, so it is reported as unpinned rather than counted as a mismatch.Verification
I do not trust a guard I have not watched fire, so I extracted the comparison and ran it standalone against three scenarios:
::error::, both marked behind in the job summaryThe third is the one I cared most about. Passing quietly on missing input is the failure mode that let this go unnoticed in the first place, and it is the same shape as several near-misses today: a
| tailswallowing an exit code, a truncatedtail -6hiding the one drifted tool, a shim bug producingPROOF-FIXTURE PASSEDwith blank scores.The job summary renders a table of pinned versus published with short digests for scanning, plus what to do about it. The
::error::keeps the full published digest, because that is the string you paste into the workflow file.actionlintclean,zizmorclean.What to expect on merge
This merge touches
.github/workflows/only, nottoolbox/**, so it will not trigger a rebuild and thepinsjob will not run yet. The nexttoolbox/**change will run it, and it will go red, correctly, because #86 and #87 already left the pins behind. The refresh after that is the last manual one that happens without a prompt.🤖 Generated with Claude Code