Skip to content

fix(ci): the pin check compared a digest that cannot be stable, so it never passed - #94

Merged
garymike merged 1 commit into
mainfrom
fix/pins-check-build-inputs
Oct 30, 2026
Merged

garymike merged 1 commit into
mainfrom
fix/pins-check-build-inputs

Conversation

@garymike

Copy link
Copy Markdown
Owner

The pin check I added in #88 has failed on every run it ever had.

2026-10-30  push      failure   e4c14c25   <- #92
2026-10-26  schedule  failure   79b45d2    <- nothing in the repo changed
2026-10-21  push      failure   79b45d2
2026-10-21  push      failure   2c9f28ab   <- the commit that added it
2026-10-21  push      success   5e6380b9   <- last green, before it existed

On the scheduled run, all six image jobs passed and only the pin job failed, with no human change involved.

What was wrong

My premise: a rebuild whose inputs have not changed produces a byte-identical digest. It does not. The Dockerfiles run apt-get update && apt-get install, so a rebuild days apart pulls different packages and publishes a different digest. The check could never stay green past the next rebuild, and since the weekly cron rebuilds on its own, it was structurally always-red. "Pins behind" stopped carrying information, which is the cry-wolf signal this repo refuses elsewhere.

What it compares now

Something that only moves when a human moves it. build-toolbox.yml stamps every image it publishes with a git ls-files -s fingerprint of that image's build inputs:

image inputs fingerprinted
security-toolbox-base toolbox/base + toolbox/tools.lock
each domain image toolbox/base + its own context + toolbox/tools.lock

toolbox/base is in every domain stamp because each domain image is built FROM the base by digest.

scripts/check-image-pins.py reads that label back off the pinned image and compares it with the repo's inputs now. It resolves the attestation-bearing index to its one real platform manifest and reads only the config blob, so four small registry calls per image and no layers pulled.

Verified before shipping

  • The registry chain resolves for all five images against the live registry. The index carries one linux/amd64 manifest plus one unknown/unknown attestation manifest, and the script requires exactly one real platform rather than taking the first entry, because picking the wrong member is how two of five pins went in wrong in chore(workflows): move the five image pins alongside the tool bumps #82.
  • The shell fingerprint in the workflow and the Python one in the script agree on all five images, which is the one thing that would otherwise make this permanently red:
  base         MATCH  sh=011ab97fe9e4 py=011ab97fe9e4
  gha          MATCH  sh=b656790438bc py=b656790438bc
  sast         MATCH  sh=1a86b748f2e2 py=1a86b748f2e2
  iac          MATCH  sh=4e730ed850d1 py=4e730ed850d1
  skill-audit  MATCH  sh=9edde01d3094 py=9edde01d3094

Expect this to be red on merge, correctly

Every currently pinned image predates the stamp, so all five report UNVERIFIABLE:

::error::skill-audit.yml pins skill-audit-toolbox@f5f99b96668f, which carries no
io.github.garymike.toolbox.build-inputs label. It was built before this check existed, so what
it holds cannot be verified from the registry. Move the pin to a digest published by a build
that stamps it.

5 of 5 pins are not current.

That is the honest state, not a defect. The pinned images also predate #91's tool bumps, so callers really are scanning with the previous toolchain. A pinned image with no stamp is reported UNVERIFIABLE rather than passing, because a check that reports success on absent input is the failure mode this replaces.

The follow-up is the usual second half: once the build from this merge publishes stamped images, move the five pins to those digests. Each build job now prints the digest to pin in its step summary, which is what that follow-up needs.

Also

  • Moved out of build-toolbox.yml into its own Image pins workflow. A red build should mean the images are broken; the pins being behind is a different fact and now reads as one, the same separation tool-drift.yml already uses. Build toolbox images should go green again on merge.
  • Not run on pull_request, on purpose. The digest to pin does not exist until the publish has happened, so a PR that bumps a tool could never be green here, and a check that cannot pass on the PR that causes it is one people learn to ignore.
  • The per-image digest artifacts existed only to feed the old job and are gone, replaced by a step-summary line.
  • Corrected the reproducibility claim in both cosign sign steps. Tolerating a Rekor 409 is still right, but a rebuild is byte-identical only when it hits the buildx cache, not whenever its inputs are unchanged.

… never passed

#88 compared each reusable-workflow image pin against the digest the build had just
published. It failed on 4 of 4 runs after it landed, including the 2026-10-26 SCHEDULED
rebuild where nothing in the repo had changed and all six image jobs passed.

The premise was mine and it was wrong: a rebuild with unchanged inputs does not produce a
byte-identical digest. The Dockerfiles apt-get install current packages, so a rebuild days
apart publishes a new digest. The check could never stay green past the next rebuild, and
because the weekly cron rebuilds on its own, "pins behind" stopped carrying information.
That is the cry-wolf signal this repo refuses elsewhere.

So it now compares something that only moves when a human moves it. build-toolbox.yml
stamps every image it publishes with a `git ls-files -s` fingerprint of that image's build
inputs: toolbox/base, the image's own context, and toolbox/tools.lock. The check reads that
label back off the PINNED image and compares it with the repo's inputs now. Four small
registry calls per image, resolving the attestation-bearing index to its one real platform
manifest and reading only the config blob, so no layers are pulled.

Verified against the live registry before shipping: the chain resolves for all five images,
and the shell fingerprint in the workflow and the Python one in the script agree on all five.

More precise, too. #93 changed only toolbox/skill-audit/rules/, which under the old check
turned all five pins red; under this one only skill-audit.yml is affected.

A pinned image with no stamp is reported UNVERIFIABLE rather than passing, so this cannot
report success on absent input, which is the failure mode it replaces.

Moved out of build-toolbox.yml into its own workflow. A red build should mean the images are
broken; the pins being behind is a different fact and should read as one, the same separation
tool-drift.yml already uses. Not run on pull_request on purpose: the digest to pin does not
exist until the publish has happened, so a PR that bumps a tool could never be green, and a
check that cannot pass on the PR that causes it is one people learn to ignore.

Also corrects the reproducibility claim in both cosign sign steps. Tolerating a Rekor 409 is
still right, but a rebuild is byte-identical only when it hits the buildx cache, not whenever
its inputs are unchanged.
@garymike
garymike merged commit 02c62e4 into main Oct 30, 2026
9 checks passed
@garymike
garymike deleted the fix/pins-check-build-inputs branch October 30, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant