Skip to content

feat(ci): fail the build run when the image pins are behind what it published - #88

Merged
garymike merged 1 commit into
mainfrom
feat/pin-check-in-build
Oct 21, 2026
Merged

garymike merged 1 commit into
mainfrom
feat/pin-check-in-build

Conversation

@garymike

Copy link
Copy Markdown
Owner

The five reusable-workflow digest pins went stale four times today. This gives the manual step a control.

Why they keep going stale

#73 made those pins deliberate on purpose: a rebuild must not silently change what every caller scans with, after betterleaks 1.6.1→1.8.1 in #64 turned green repos red overnight. The cost of "deliberate" is that nothing moves them for you, and nothing reminded anyone either.

#80, #81 bumped eight tools and SkillSpector, both published, neither moved the pins
#82 the cleanup, after callers had been on the Oct-1 toolchain while tools.lock claimed otherwise
#87, #86 both touched toolbox/**, so the pins are behind again right now

Each build job now records the digest it published, and a new pins job compares those against the pin in the matching reusable workflow.

Two design points, and they are the reason it works

It is a separate job, after base and domain, so publishing is never blocked. It cannot be blocked: the digest to pin does not exist until the publish has happened, so failing a publish on a stale pin would be circular. The images ship and the run still goes red, so the follow-up is not optional. That is the same contract tool-drift.yml already states, where red means a pin is behind rather than something being broken.

This is also what kept me from putting it in the drift gate or inline in the build steps. A red base job would skip domain entirely and block five images from publishing.

It reads steps.build.outputs.digest, not the registry. Comparing against the latest index digest false-reds on images whose index carries attestations, which is precisely why I pulled the equivalent check back out of check-tool-updates.sh in #82 rather than ship a gate that cries wolf. The build already holds both halves, so there is no lookup and no index-versus-manifest ambiguity.

mcp-review-toolbox is built here but no reusable workflow pins it, so it is reported as unpinned rather than counted as a mismatch.

Verification

I do not trust a guard I have not watched fire, so I extracted the comparison and ran it standalone against three scenarios:

scenario result
all five pins match exit 0, "All reusable-workflow image pins match what this run published"
two pins behind exit 1, both named in ::error::, both marked behind in the job summary
no digest files at all exit 1, "the pins were NOT verified"

The third is the one I cared most about. Passing quietly on missing input is the failure mode that let this go unnoticed in the first place, and it is the same shape as several near-misses today: a | tail swallowing an exit code, a truncated tail -6 hiding the one drifted tool, a shim bug producing PROOF-FIXTURE PASSED with blank scores.

The job summary renders a table of pinned versus published with short digests for scanning, plus what to do about it. The ::error:: keeps the full published digest, because that is the string you paste into the workflow file.

actionlint clean, zizmor clean.

What to expect on merge

This merge touches .github/workflows/ only, not toolbox/**, so it will not trigger a rebuild and the pins job will not run yet. The next toolbox/** change will run it, and it will go red, correctly, because #86 and #87 already left the pins behind. The refresh after that is the last manual one that happens without a prompt.

🤖 Generated with Claude Code

…ublished

Fourth time today the five reusable-workflow digest pins went stale, so the
manual step gets a control.

#73 made those pins deliberate on purpose: a rebuild must not silently change
what every caller scans with, after betterleaks 1.6.1 -> 1.8.1 in #64 turned
green repos red overnight. The cost of "deliberate" is that nothing moves them
for you, and nothing reminded anyone either. #80 bumped eight tools and #81
bumped SkillSpector, both published, neither moved the pins, and callers kept
getting the previous toolchain while tools.lock said otherwise. #82 was the
cleanup. Every toolbox/** change since has put them behind again, including
#87 and #86.

Each build job now records the digest it published, and a new `pins` job
compares those against the pin in the matching reusable workflow.

Two design points that are the whole reason this works:

It is a SEPARATE job, after base and domain, so publishing is never blocked.
It cannot be blocked: the digest to pin does not exist until the publish has
happened, so failing the publish on a stale pin would be circular. The images
ship, and the run still goes red, so the follow-up is not optional. Same
contract as tool-drift.yml: red means a pin is behind, not that anything is
broken.

It reads steps.build.outputs.digest rather than the registry. Comparing
against the `latest` index digest false-reds on images whose index carries
attestations, which is exactly why I pulled the equivalent check back out of
check-tool-updates.sh in #82 rather than ship a flaky gate. The build already
holds both halves, so no lookup and no ambiguity.

mcp-review-toolbox is built here but no reusable workflow pins it, so it is
reported as unpinned rather than counted as a mismatch.

Verified by running the comparison standalone against three scenarios:
all pins matching exits 0; two behind exits 1 with both named in ::error::
and marked in the job summary; and no digest files at all exits 1 saying the
pins were NOT verified, rather than passing quietly, which is the failure mode
that let this go unnoticed in the first place. actionlint clean, zizmor clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@garymike
garymike merged commit 2c9f28a into main Oct 21, 2026
9 checks passed
@garymike
garymike deleted the feat/pin-check-in-build branch October 21, 2026 17:43
garymike added a commit that referenced this pull request Oct 21, 2026
…hed (#89)

The first refresh prompted by the check rather than by remembering. #88 added
a `pins` job to build-toolbox.yml; its own merge triggered a rebuild, the job
ran, and it went red naming all five stale pins with the digest each should
move to. A true positive on first firing, because #86 and #87 had already left
them behind.

  security-toolbox-base   d2f1d6b33d9c -> 9be060794ecb
  gha-toolbox             7e6558f3b46c -> 4cef3443acbd
  sast-toolbox            6bf6d9aafd5a -> edfd4db8752a
  iac-toolbox             bc4198b93fcc -> 08e1338df20c
  skill-audit-toolbox     c10bc17412a2 -> f5f99b96668f

Digests taken from the job's own annotations, which report
steps.build.outputs.digest from the run that published them. No local pull and
no registry lookup, so neither the index-vs-manifest ambiguity from #82 nor a
hand-copied digest can get in.

Verified behind the new pins: base ships trufflehog 3.99.3, betterleaks 1.9.0,
osv-scanner 2.7.0 and syft 1.55.0, with pip 26.2.2 vendoring certifi
2026.10.19, so #87's CVE fix is actually in what callers now get.
skill-audit ships SkillSpector v2.13.0.

One thing the check cannot currently survive: #86's build failed on a
cosign/Rekor 409, "an equivalent entry already exists in the transparency
log", because toolbox/gha was unchanged so gha-toolbox rebuilt to a byte
identical digest that was already signed. A failed domain arm skips the pins
job rather than running it. Benign cause, but it means a reproducible rebuild
can silence the check. Noted, not fixed here.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
garymike added a commit that referenced this pull request Oct 21, 2026
…#90)

Two faults, one cause, found because #86's build went red on both at once.

A rebuild whose inputs have not changed produces a byte-identical digest, and
Rekor rejects a second equivalent entry for it with 409
createLogEntryConflict. That is not a signing failure: the entry it conflicts
with IS this digest's signature, already in the log from the earlier run.
toolbox/gha was untouched by #86, gha-toolbox rebuilt to the same bytes, and
the sign step failed.

The sign step now matches createLogEntryConflict specifically and treats it as
already-signed, with a notice explaining why. Any other cosign failure still
fails: verified against four stubbed outcomes, where success and the 409 exit
0, and a missing OIDC token and a registry 403 both exit 1.

The second fault is worse, and the 409 is what exposed it. A red domain arm
skipped the pins job entirely, because needs: [base, domain] will not run on a
partial build. So a reproducible rebuild could silence the very check added in
#88 to watch the pins, while five images published with nobody comparing
anything. It now runs with if: !cancelled().

That raises the question the first version did not have to answer: what to do
about an image that never published. Nothing to compare, so it is reported as
"published no digest in this run, so its pin was NOT checked" rather than
dropped. Four greens must not read as a clean sweep when six images were
expected. Verified on a simulated partial build: the four that published are
confirmed current, and the two that did not are named as unchecked.

The zero-digest case still fails outright rather than warning. On a totally
failed build that is redundant with the build's own red, but the alternative
is a job that reports success having compared nothing, which is the failure
shape that let the pins drift unnoticed four times in the first place.

actionlint clean, zizmor clean.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
garymike added a commit that referenced this pull request Oct 30, 2026
… never passed (#94)

#88 compared each reusable-workflow image pin against the digest the build had just
published. It failed on 4 of 4 runs after it landed, including the 2026-10-26 SCHEDULED
rebuild where nothing in the repo had changed and all six image jobs passed.

The premise was mine and it was wrong: a rebuild with unchanged inputs does not produce a
byte-identical digest. The Dockerfiles apt-get install current packages, so a rebuild days
apart publishes a new digest. The check could never stay green past the next rebuild, and
because the weekly cron rebuilds on its own, "pins behind" stopped carrying information.
That is the cry-wolf signal this repo refuses elsewhere.

So it now compares something that only moves when a human moves it. build-toolbox.yml
stamps every image it publishes with a `git ls-files -s` fingerprint of that image's build
inputs: toolbox/base, the image's own context, and toolbox/tools.lock. The check reads that
label back off the PINNED image and compares it with the repo's inputs now. Four small
registry calls per image, resolving the attestation-bearing index to its one real platform
manifest and reading only the config blob, so no layers are pulled.

Verified against the live registry before shipping: the chain resolves for all five images,
and the shell fingerprint in the workflow and the Python one in the script agree on all five.

More precise, too. #93 changed only toolbox/skill-audit/rules/, which under the old check
turned all five pins red; under this one only skill-audit.yml is affected.

A pinned image with no stamp is reported UNVERIFIABLE rather than passing, so this cannot
report success on absent input, which is the failure mode it replaces.

Moved out of build-toolbox.yml into its own workflow. A red build should mean the images are
broken; the pins being behind is a different fact and should read as one, the same separation
tool-drift.yml already uses. Not run on pull_request on purpose: the digest to pin does not
exist until the publish has happened, so a PR that bumps a tool could never be green, and a
check that cannot pass on the PR that causes it is one people learn to ignore.

Also corrects the reproducibility claim in both cosign sign steps. Tolerating a Rekor 409 is
still right, but a rebuild is byte-identical only when it hits the buildx cache, not whenever
its inputs are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant