Skip to content

fix(toolbox): upgrade pip so its vendored certifi is not the CRITICAL - #87

Merged
garymike merged 1 commit into
mainfrom
fix/pip-vendored-certifi-critical
Oct 21, 2026
Merged

garymike merged 1 commit into
mainfrom
fix/pip-vendored-certifi-critical

Conversation

@garymike

Copy link
Copy Markdown
Owner

Unbreaks the Trivy publish gate. It is currently failing on every build, which is what blocks #86.

certifi  GHSA-6q48-6c7w-9gw2  CRITICAL
installed 2026.6.17   fixed 2026.10.19
"certifi bundles emSign root CAs with compromised intermediate CA keys"

Why the Trivy table was misleading

Three different certifi versions were in play, so the reported 2026.6.17 matched nothing I could find by hand:

where version
site-packages/certifi 2026.10.19 already fine, resolves on its own
pip/_vendor/certifi 2026.6.17 the flagged one
main's published image, site-packages 2026.10.18 a red herring while I was comparing

The vulnerable copy is the one pip vendors for its own HTTPS to PyPI. pip install cannot reach it, which is why nothing in tools.lock would have fixed this.

A fix, not an exception

pip 26.2.2 vendors certifi==2026.10.19, the patched release, and 26.2.2 is current on PyPI. .trivy/toolbox.openvex.json already states the rule this follows: a fixable finding is fixed, not excepted. That reasoning is in the Dockerfile comment so the next reader does not have to rediscover why this is not a VEX entry.

Placement and pinning

In the base final stage, because all five domain images inherit FROM ${BASE} and the vulnerable copy came from the shared python:3.14-slim layer rather than from anything they install. One place instead of five.

Pinned as ARG PIP_VERSION=26.2.2 with a tools.lock row rather than a bare --upgrade pip, which keeps house style and plugs into the existing machinery:

  • check pip "$(pinned pip)" "$(pypi_latest pip)" added, and the gate now reports pip: up to date (26.2.2)
  • arg_name("pip") resolves to PIP_VERSION, so apply-tool-bumps.py can move it on a future bump instead of this becoming another pin that only moves by hand

Verification

  • base: pip 26.2.2, vendored certifi==2026.10.19, and all four scanners still report versions, so the upgrade did not break the image
  • derived: a skill-audit image built on it has both certifi copies at 2026.10.19. That was the check worth doing: if the derived pip install had pulled an older certifi into site-packages, the finding would have moved from the vendored copy to the real one rather than being removed. SkillSpector v2.13.0 and semgrep 1.181.0 both still run.
  • drift check exits 0

Scope

No changelog entry, matching #75 and #62, the two previous CVE unbreak-main fixes. That also keeps this off CHANGELOG.md so it does not conflict with #86's 1.7.1 entry.

Nothing here touches detection behaviour. #86 should go green once this lands and its build re-runs.

🤖 Generated with Claude Code

GHSA-6q48-6c7w-9gw2 (CRITICAL): certifi bundles emSign root CAs whose
intermediate keys were compromised. Fixed in certifi 2026.10.19.

The vulnerable copy is pip's vendored one, which `pip install` cannot reach.
Three different versions were in play, which is what made this confusing to
read off a Trivy table:

  site-packages/certifi              2026.10.19  already fine, resolves on its own
  pip/_vendor/certifi                2026.6.17   from the python base layer, flagged

pip 26.2.2 vendors certifi 2026.10.19, so this is a fix and not a VEX
exception. .trivy/toolbox.openvex.json already states the rule: a fixable
finding is fixed, not excepted.

Done in the base final stage because all five domain images inherit
FROM ${BASE}, and the vulnerable copy came from the shared python layer
rather than from anything they install. Pinned as ARG PIP_VERSION with a
tools.lock row rather than a bare --upgrade, so the drift gate tracks it and
apply-tool-bumps.py can move it later: arg_name("pip") is PIP_VERSION, which
matches the applier's convention.

Verified: base reports pip 26.2.2 with vendored certifi 2026.10.19 and all
four scanners still run. A derived skill-audit image built on it has BOTH
copies at 2026.10.19, so the fix survives FROM and the derived pip install
does not pull an older certifi into site-packages, which would have moved the
finding rather than removed it. Drift check reports pip up to date.

No changelog entry, matching #75 and #62, the two previous CVE unbreak-main
fixes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@garymike
garymike merged commit 5e6380b into main Oct 21, 2026
9 checks passed
@garymike
garymike deleted the fix/pip-vendored-certifi-critical branch October 21, 2026 15:22
garymike added a commit that referenced this pull request Oct 21, 2026
…ublished (#88)

Fourth time today the five reusable-workflow digest pins went stale, so the
manual step gets a control.

#73 made those pins deliberate on purpose: a rebuild must not silently change
what every caller scans with, after betterleaks 1.6.1 -> 1.8.1 in #64 turned
green repos red overnight. The cost of "deliberate" is that nothing moves them
for you, and nothing reminded anyone either. #80 bumped eight tools and #81
bumped SkillSpector, both published, neither moved the pins, and callers kept
getting the previous toolchain while tools.lock said otherwise. #82 was the
cleanup. Every toolbox/** change since has put them behind again, including
#87 and #86.

Each build job now records the digest it published, and a new `pins` job
compares those against the pin in the matching reusable workflow.

Two design points that are the whole reason this works:

It is a SEPARATE job, after base and domain, so publishing is never blocked.
It cannot be blocked: the digest to pin does not exist until the publish has
happened, so failing the publish on a stale pin would be circular. The images
ship, and the run still goes red, so the follow-up is not optional. Same
contract as tool-drift.yml: red means a pin is behind, not that anything is
broken.

It reads steps.build.outputs.digest rather than the registry. Comparing
against the `latest` index digest false-reds on images whose index carries
attestations, which is exactly why I pulled the equivalent check back out of
check-tool-updates.sh in #82 rather than ship a flaky gate. The build already
holds both halves, so no lookup and no ambiguity.

mcp-review-toolbox is built here but no reusable workflow pins it, so it is
reported as unpinned rather than counted as a mismatch.

Verified by running the comparison standalone against three scenarios:
all pins matching exits 0; two behind exits 1 with both named in ::error::
and marked in the job summary; and no digest files at all exits 1 saying the
pins were NOT verified, rather than passing quietly, which is the failure mode
that let this go unnoticed in the first place. actionlint clean, zizmor clean.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
garymike added a commit that referenced this pull request Oct 21, 2026
…hed (#89)

The first refresh prompted by the check rather than by remembering. #88 added
a `pins` job to build-toolbox.yml; its own merge triggered a rebuild, the job
ran, and it went red naming all five stale pins with the digest each should
move to. A true positive on first firing, because #86 and #87 had already left
them behind.

  security-toolbox-base   d2f1d6b33d9c -> 9be060794ecb
  gha-toolbox             7e6558f3b46c -> 4cef3443acbd
  sast-toolbox            6bf6d9aafd5a -> edfd4db8752a
  iac-toolbox             bc4198b93fcc -> 08e1338df20c
  skill-audit-toolbox     c10bc17412a2 -> f5f99b96668f

Digests taken from the job's own annotations, which report
steps.build.outputs.digest from the run that published them. No local pull and
no registry lookup, so neither the index-vs-manifest ambiguity from #82 nor a
hand-copied digest can get in.

Verified behind the new pins: base ships trufflehog 3.99.3, betterleaks 1.9.0,
osv-scanner 2.7.0 and syft 1.55.0, with pip 26.2.2 vendoring certifi
2026.10.19, so #87's CVE fix is actually in what callers now get.
skill-audit ships SkillSpector v2.13.0.

One thing the check cannot currently survive: #86's build failed on a
cosign/Rekor 409, "an equivalent entry already exists in the transparency
log", because toolbox/gha was unchanged so gha-toolbox rebuilt to a byte
identical digest that was already signed. A failed domain arm skips the pins
job rather than running it. Benign cause, but it means a reproducible rebuild
can silence the check. Noted, not fixed here.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant