Skip to content

chore(toolbox): bump nine pins, and restate the differentiator SkillSpector v2.13.4 changed - #103

Merged
garymike merged 1 commit into
mainfrom
chore/drift-bump-2026-11-06
Nov 9, 2026
Merged

garymike merged 1 commit into
mainfrom
chore/drift-bump-2026-11-06

Conversation

@garymike

@garymike garymike commented Nov 6, 2026

Copy link
Copy Markdown
Owner

The bump and the docs pass ship together on purpose. v2.13.4 moves the one carrier this gate uniquely covers, so landing the pin alone would leave nine files asserting numbers and a mechanism the shipped version does not have.

Bumped

Tool From To
betterleaks 1.9.0 1.10.0
trufflehog 3.99.7 3.99.8
syft 1.55.0 1.57.0
pip 26.2.2 26.3
snyk-agent-scan 0.6.10 0.6.11
zizmor 1.31.1 1.32.0
semgrep 1.183.0 1.184.0 (both pins)
checkov 3.4.2 3.4.3
skillspector v2.13.2 v2.13.4 = 54824463814a

skillspector is a hand bump by design: its ARG is a commit SHA, so apply-tool-bumps.py refuses it rather than rewriting the lock while matching nothing in the Dockerfile.

What v2.13.4 changed

The claim in README.md and docs/ was that the memory write "is not modelled" and that neither payload variant blocks. Both halves are now false. Measured on the same fixture across the three tags there are images for, rather than inferred from release notes:

tag inert canary to MEMORY.md literal injection string
v2.13.0 0/100, exit 0, no finding 35/100, exit 0, no finding
v2.13.2 0/100, exit 0, no finding 35/100, exit 0, no finding
v2.13.4 27/100, exit 0, MP3 fires 63/100, exit 1, blocks

On the inert fixture v2.13.4 reports exactly one finding:

Issues (1)
  HIGH: MP3 - Memory Manipulation...
    Location: notes.test.ts:8
    Confidence: 85%
    Evidence: carrier_basename=notes.test.ts, carrier_kind=auto_run, destination=MEMORY.md

It names the write, knows the carrier is auto-run, rates it HIGH, and exits 0, because the verdict is a holistic risk_score and 27 is under 50.

Decomposed by holding the mechanism fixed and varying the payload:

variant findings score blocks?
write, inert canary MP3 27/100 no
write, literal injection string MP3, P1, EA2 63/100 yes
write to CLAUDE.md, same string MP3, P1, EA2 63/100 yes
control: same string as SKILL.md prose, no write P1, EA2 27/100 no

The write alone is worth 27 and the injection prose alone is worth 27. Neither half reaches the threshold; only the sum does. So the installable variant is the one whose payload does not read like an injection, which is the easier half to author.

The restated differentiator: not that upstream cannot see the write, but that seeing it does not stop it, and that the payload wording rather than the write decides the verdict. The gate blocks on the write itself, so wording does not move it.

Proof

gate-proof.sh passes 16 of 16 against the image built from these pins. The other four fixture scores are unchanged (64/100, 76/100, 100/100, 51/100), and the live table reads:

  FIXTURE                  SCORE     SKILLSPECTOR
  gecko-demo               64/100    exit 1, BLOCKS
  gecko-hook-demo          76/100    exit 1, BLOCKS
  config-injection-demo    100/100   exit 1, BLOCKS
  sibling-config-demo      51/100    exit 1, BLOCKS
  memory-poisoning-demo    27/100    exit 0, does NOT block

Assertion 4's hard check is unchanged and still passes, but it now sits 23 points from flipping rather than 50, and the header records that.

Corrects the reasoning in 1.7.6, not its conclusion

That entry checked NVIDIA/SkillSpector#893 against the commit graph, found the merge commit 1196 commits ahead of the v2.13.2 tag, and concluded the 0/100 figures were stable. The conclusion held for v2.13.2 and the predicted number was exactly right (27/100), but the method could not have settled it: git still reports 2de9b387...v2.13.4 as diverged, 183 ahead and 1196 behind, because that project cuts releases from separate branches rather than from main. A merge commit's ancestry in main is not a test of what a release does. Running the release against the fixture is, which is what gate-proof.sh does on every build.

Follow-up

The five image pins move in a separate PR once this lands, covering both this bump and #102.

…pector v2.13.4 changed

The bump and the docs pass ship together on purpose. v2.13.4 moves the one carrier this gate
uniquely covers, so landing the pin alone would leave nine files asserting numbers and a
mechanism that the shipped version does not have.

### Bumped

| Tool | From | To |
|---|---|---|
| betterleaks | `1.9.0` | `1.10.0` |
| trufflehog | `3.99.7` | `3.99.8` |
| syft | `1.55.0` | `1.57.0` |
| pip | `26.2.2` | `26.3` |
| snyk-agent-scan | `0.6.10` | `0.6.11` |
| zizmor | `1.31.1` | `1.32.0` |
| semgrep | `1.183.0` | `1.184.0` (both pins) |
| checkov | `3.4.2` | `3.4.3` |
| skillspector | `v2.13.2` | `v2.13.4` = 54824463814a |

skillspector is a hand bump by design: its ARG is a commit SHA, so apply-tool-bumps.py refuses
it rather than rewriting the lock while matching nothing in the Dockerfile.

### What v2.13.4 changed, measured

The claim in README.md and docs/ was that the memory write "is not modelled" and that neither
payload variant blocks. Both halves are now false. Measured on the same fixture across the three
tags there are images for:

| tag | inert canary to MEMORY.md | literal injection string |
|---|---|---|
| v2.13.0 | 0/100, exit 0, no finding | 35/100, exit 0, no finding |
| v2.13.2 | 0/100, exit 0, no finding | 35/100, exit 0, no finding |
| v2.13.4 | 27/100, exit 0, MP3 fires | 63/100, exit 1, blocks |

On the inert fixture v2.13.4 reports exactly one finding: HIGH MP3 - Memory Manipulation, 85%
confidence, evidence carrier_kind=auto_run destination=MEMORY.md. It names the write, rates it
HIGH, and exits 0, because the verdict is a holistic risk_score and 27 is under 50.

Decomposed: the write alone is worth 27, the injection prose alone is worth 27, and only the sum
crosses the threshold. So the installable variant is the one whose payload does not read like an
injection, which is the easier half to author. That is the differentiator now. Not that upstream
cannot see the write, but that seeing it does not stop it, and that the payload wording rather
than the write decides the verdict. The gate blocks on the write, so wording does not move it.

Reframed README.md, docs/threat-model.md, docs/references.md, docs/gecko-vector-walkthrough.md
(Step 6 gains the report excerpt and a findings column), docs/threats/developer-execution.md,
tests/gate-proof.sh, toolbox/skill-audit/Dockerfile and skill-testfile-gate.sh.

gate-proof.sh passes 16 of 16 against the image built from these pins. The other four fixture
scores are unchanged: 64/100, 76/100, 100/100, 51/100. Assertion 4's hard check is unchanged and
still passes, but it now sits 23 points from flipping rather than 50, and the header says so.

### Corrects the reasoning in 1.7.6, not its conclusion

That entry checked NVIDIA/SkillSpector#893 against the commit graph, found the merge commit 1196
commits ahead of the v2.13.2 tag, and concluded the 0/100 figures were stable. The conclusion held
for v2.13.2 and the predicted number was exactly right, but the method could not have settled it:
git still reports 2de9b387...v2.13.4 as diverged, 183 ahead and 1196 behind, because that project
cuts releases from separate branches rather than from main. A merge commit's ancestry in main is
not a test of what a release does. Running the release against the fixture is.
@garymike
garymike merged commit 976ef61 into main Nov 9, 2026
9 checks passed
@garymike
garymike deleted the chore/drift-bump-2026-11-06 branch November 9, 2026 05:02
garymike added a commit that referenced this pull request Nov 9, 2026
…blished (#104)

The follow-up half of #103, and it covers #102 as well: both landed toolbox changes without
moving a pin, so callers have been resolving images built before the nine-tool bump. For
skill-audit-toolbox that gap was load-bearing, because #103's docs assert v2.13.4 numbers while
the pinned image still carried v2.13.2.

Permitted rather than forced. check-image-pins.py reported all 5 of 5 behind before the move,
which is the state the mover's guard requires, so no --force was needed:

    gha-security.yml      inputs 5c137c684483 -> f6a4a72e874a
    iac-security.yml      inputs f24b2cabd9b1 -> 4a28bbf3e20f
    sast.yml              inputs de509bd1ace1 -> 547b6bc2feaf
    security-scan.yml     inputs 2aed68cf9f33 -> b4802091e180
    skill-audit.yml       inputs b011e83c21f6 -> a36fc4fa94cf

Verified the usual way: each digest resolved from the per-commit sha-976ef61d7b38 tag rather than
latest, and its build-inputs label read back out of the registry and compared with the fingerprint
this repo computes now. All five matched before anything was written. check-image-pins.py now
reports all five current.

Verified one step further, because this is the pin #103's claims rest on. Pulled the
skill-audit-toolbox digest callers will now resolve and ran the proof against it, not against a
locally built image:

    SkillSpector v2.13.4, semgrep 1.184.0
    gate-proof.sh: 16 of 16

    gecko-demo               64/100    exit 1, BLOCKS
    gecko-hook-demo          76/100    exit 1, BLOCKS
    config-injection-demo    100/100   exit 1, BLOCKS
    sibling-config-demo      51/100    exit 1, BLOCKS
    memory-poisoning-demo    27/100    exit 0, does NOT block

The 27/100 in README.md and docs/ is now what a caller resolving this pin actually gets.
garymike added a commit that referenced this pull request Nov 9, 2026
…after #103 (#105)

Both are patch releases published after #103's nine-tool bump, so the weekly drift watcher
was red for the right reason. Applied through scripts/apply-tool-bumps.py, which moves
tools.lock and the Dockerfile ARG together rather than letting one of them lie:

    osv-scanner  2.7.0  -> 2.7.1   toolbox/tools.lock, toolbox/base/Dockerfile
    syft         1.57.0 -> 1.57.1  toolbox/tools.lock, toolbox/base/Dockerfile

check-tool-updates.sh --fail-on-drift now exits 0 with all twelve pins current.

Pre-flighted the one failure mode a patch bump has here. The base image verifies each
download against the release's own checksum file, so a renamed asset fails the build
rather than the drift check. Both releases publish the exact names the Dockerfile
constructs: osv-scanner_linux_amd64 with osv-scanner_SHA256SUMS, and
syft_1.57.1_linux_amd64.tar.gz with syft_1.57.1_checksums.txt.

Both pins live in security-toolbox-base, so this rebuilds all five images and the five
workflow pins will need moving once it lands.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant