Repository navigation
fix(gate): agent-memory writes were JS-only, and reads were false-positived - #86
Merged
Merged
Conversation
…itived Found by building the boundary fixtures from NVIDIA/SkillSpector#879 against our own gate rather than only against theirs. Two defects in dev-exec-writes-agent-memory, pulling in opposite directions. False negatives. The pattern required the sink token before the path, which is the JS shape: appendFileSync("MEMORY.md", ...). Python puts the path first and the mode second, so the `open` alternative consumed straight past the filename and could not match a Python write in any mode. pathlib was absent. So was CLAUDE.md, the most-loaded memory file for the agent this gate exists to protect. Measured misses, all now blocked: open("MEMORY.md", "a") was exit 0, now exit 1 open("MEMORY.md", "w") was exit 0, now exit 1 Path("MEMORY.md").write_text(...) was exit 0, now exit 1 appendFileSync("CLAUDE.md", ...) was exit 0, now exit 1 Assertion 7 had been passing on a TypeScript-only fixture, which is how a language-shaped hole stays invisible. One fixture per language now. A false positive. The shell-redirect alternative was `>>?` plus up to 60 characters of anything, and a single `>` is the second character of a JS arrow function, so any `=>` within 60 characters of "MEMORY.md" reported a write. readFileSync("MEMORY.md") inside beforeAll(() => {...}) blocked. Confirmed pre-existing against the published image before touching the rule. Reading agent memory is not the threat; writing to it is. The rule is three branches now, split by where the write intent sits relative to the destination. Eleven shapes checked locally against the compiled regexes, then the full set re-run in-container: five previously-missed shapes block, seven controls stay quiet, and gate-proof passes 12 for 12. Also corrects a sentence I put in the walkthrough yesterday. Step 6 claimed the gate "fires on all four" rows. It fires on two: one miss is the prose control with no write, the other was the CLAUDE.md bug above. Two shapes still pass and are recorded as residuals, not assertions: an assembled destination, and open reached through an indirection. Both need the path resolved rather than matched. Asserting them would turn a weakness into expected behaviour and make the next improvement delete a passing test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
garymike
added a commit
that referenced
this pull request
Oct 21, 2026
…ublished (#88) Fourth time today the five reusable-workflow digest pins went stale, so the manual step gets a control. #73 made those pins deliberate on purpose: a rebuild must not silently change what every caller scans with, after betterleaks 1.6.1 -> 1.8.1 in #64 turned green repos red overnight. The cost of "deliberate" is that nothing moves them for you, and nothing reminded anyone either. #80 bumped eight tools and #81 bumped SkillSpector, both published, neither moved the pins, and callers kept getting the previous toolchain while tools.lock said otherwise. #82 was the cleanup. Every toolbox/** change since has put them behind again, including #87 and #86. Each build job now records the digest it published, and a new `pins` job compares those against the pin in the matching reusable workflow. Two design points that are the whole reason this works: It is a SEPARATE job, after base and domain, so publishing is never blocked. It cannot be blocked: the digest to pin does not exist until the publish has happened, so failing the publish on a stale pin would be circular. The images ship, and the run still goes red, so the follow-up is not optional. Same contract as tool-drift.yml: red means a pin is behind, not that anything is broken. It reads steps.build.outputs.digest rather than the registry. Comparing against the `latest` index digest false-reds on images whose index carries attestations, which is exactly why I pulled the equivalent check back out of check-tool-updates.sh in #82 rather than ship a flaky gate. The build already holds both halves, so no lookup and no ambiguity. mcp-review-toolbox is built here but no reusable workflow pins it, so it is reported as unpinned rather than counted as a mismatch. Verified by running the comparison standalone against three scenarios: all pins matching exits 0; two behind exits 1 with both named in ::error:: and marked in the job summary; and no digest files at all exits 1 saying the pins were NOT verified, rather than passing quietly, which is the failure mode that let this go unnoticed in the first place. actionlint clean, zizmor clean. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
garymike
added a commit
that referenced
this pull request
Oct 21, 2026
…hed (#89) The first refresh prompted by the check rather than by remembering. #88 added a `pins` job to build-toolbox.yml; its own merge triggered a rebuild, the job ran, and it went red naming all five stale pins with the digest each should move to. A true positive on first firing, because #86 and #87 had already left them behind. security-toolbox-base d2f1d6b33d9c -> 9be060794ecb gha-toolbox 7e6558f3b46c -> 4cef3443acbd sast-toolbox 6bf6d9aafd5a -> edfd4db8752a iac-toolbox bc4198b93fcc -> 08e1338df20c skill-audit-toolbox c10bc17412a2 -> f5f99b96668f Digests taken from the job's own annotations, which report steps.build.outputs.digest from the run that published them. No local pull and no registry lookup, so neither the index-vs-manifest ambiguity from #82 nor a hand-copied digest can get in. Verified behind the new pins: base ships trufflehog 3.99.3, betterleaks 1.9.0, osv-scanner 2.7.0 and syft 1.55.0, with pip 26.2.2 vendoring certifi 2026.10.19, so #87's CVE fix is actually in what callers now get. skill-audit ships SkillSpector v2.13.0. One thing the check cannot currently survive: #86's build failed on a cosign/Rekor 409, "an equivalent entry already exists in the transparency log", because toolbox/gha was unchanged so gha-toolbox rebuilt to a byte identical digest that was already signed. A failed domain arm skips the pins job rather than running it. Benign cause, but it means a reproducible rebuild can silence the check. Noted, not fixed here. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
garymike
added a commit
that referenced
this pull request
Oct 21, 2026
…#90) Two faults, one cause, found because #86's build went red on both at once. A rebuild whose inputs have not changed produces a byte-identical digest, and Rekor rejects a second equivalent entry for it with 409 createLogEntryConflict. That is not a signing failure: the entry it conflicts with IS this digest's signature, already in the log from the earlier run. toolbox/gha was untouched by #86, gha-toolbox rebuilt to the same bytes, and the sign step failed. The sign step now matches createLogEntryConflict specifically and treats it as already-signed, with a notice explaining why. Any other cosign failure still fails: verified against four stubbed outcomes, where success and the 409 exit 0, and a missing OIDC token and a registry 403 both exit 1. The second fault is worse, and the 409 is what exposed it. A red domain arm skipped the pins job entirely, because needs: [base, domain] will not run on a partial build. So a reproducible rebuild could silence the very check added in #88 to watch the pins, while five images published with nobody comparing anything. It now runs with if: !cancelled(). That raises the question the first version did not have to answer: what to do about an image that never published. Nothing to compare, so it is reported as "published no digest in this run, so its pin was NOT checked" rather than dropped. Four greens must not read as a clean sweep when six images were expected. Verified on a simulated partial build: the four that published are confirmed current, and the two that did not are named as unchecked. The zero-digest case still fails outright rather than warning. On a totally failed build that is redundant with the build's own red, but the alternative is a job that reports success having compared nothing, which is the failure shape that let the pins drift unnoticed four times in the first place. actionlint clean, zizmor clean. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You asked for the #879 boundary fixtures in the repo. Running them against our gate rather than only against upstream's turned up two defects in
dev-exec-writes-agent-memory, pulling in opposite directions.False negatives: the rule only ever worked in JavaScript
appendFileSync("MEMORY.md", …)appendFileSync("CLAUDE.md", …)open("MEMORY.md", "a")+ writeopen("MEMORY.md", "w")+ writePath("MEMORY.md").write_text(…)Two independent causes.
The pattern required the sink token before the path. That is the JS shape. Python puts the path first and the mode second, so the
openalternative consumed straight past the filename and the destination could never be found after it. That is why even"w"failed, not only"a": no Python write could match this rule in any mode.pathlibwas not covered at all.CLAUDE.mdwas not in the destination list. It heldMEMORY.md,SOUL.md,AGENTS.md,.claude/and.cursor/. The most-loaded memory file for the agent this gate exists to protect was the one destination it did not look for. The rule's message was honest about its own list, so this was a coverage gap rather than a false claim.Assertion 7 had been green for months on a TypeScript-only fixture. That is precisely how a language-shaped hole stays invisible, so the surface now carries one fixture per language.
A false positive: it flagged a write on a file that only reads
The shell-redirect alternative was
>>?followed by up to 60 characters of anything. A single>is the second character of a JS arrow function, so any=>within 60 characters of the stringMEMORY.mdwas reported as a memory write. This blocked:Confirmed pre-existing by running that control against the published image before touching the rule, so it is not something this PR introduced. Reading agent memory is not the threat; writing to it is.
The rule is three branches now
Split by where the write intent sits relative to the destination, which is also what keeps a read quiet:
readFileSynccannot match.open(path, mode),pathlib). The mode must be a write mode, soopen(path, "r")stays quiet.pattern-eitheris new in this file. The comment records why: the orderings genuinely cannot be expressed in one regex, and the previous attempt to do so is what produced both defects.Verification
Eleven shapes checked locally against the compiled regexes (7 must-match, 4 must-not), zero mismatches. Then the full set re-run in-container against a real Semgrep:
report.md, non-auto-run helper writingMEMORY.md,benign-skill,config-injection-benign,memory-read-benigngate-proof.shpasses 12 for 12Three fixtures, three assertions
memory-poisoning-python-demo(10, blocks),memory-poisoning-claude-md-demo(11, blocks),memory-read-benign(12, must not block). Each fixture header records the specific defect it guards, not just what the file does, so the next reader knows why it exists.Two things I did not do
Corrected a false sentence I wrote yesterday. Walkthrough Step 6 ended "The gate blocks on the write itself, which is why it fires on all four." It fires on two of four. One miss is the prose control, which has no write to find; the other was the
CLAUDE.mdrow, which was the bug above rather than an intended negative.Left two shapes passing, as residuals rather than assertions. An assembled destination (
open("MEM" + "ORY" + ".md", "a")) andopenreached through an indirection both still pass. Fixing them needs the path resolved rather than matched, which is a different kind of analysis than this rule pack does. They are recorded in the threat profile's honest residual and deliberately not asserted: a test that asserts the gate does not catch something turns a weakness into expected behaviour, and whoever improves the rule would have to delete a passing test to do it.Worth noting those same two shapes also pass NVIDIA/SkillSpector#893, for the same reason, which is the feedback I left on that PR.
🤖 Generated with Claude Code