Skip to content

fix(gate): agent-memory writes were JS-only, and reads were false-positived - #86

Merged
garymike merged 2 commits into
mainfrom
fix/agent-memory-write-rule-coverage
Oct 21, 2026
Merged

garymike merged 2 commits into
mainfrom
fix/agent-memory-write-rule-coverage

Conversation

@garymike

Copy link
Copy Markdown
Owner

You asked for the #879 boundary fixtures in the repo. Running them against our gate rather than only against upstream's turned up two defects in dev-exec-writes-agent-memory, pulling in opposite directions.

False negatives: the rule only ever worked in JavaScript

shape before after
appendFileSync("MEMORY.md", …) blocks blocks
appendFileSync("CLAUDE.md", …) exit 0 blocks
open("MEMORY.md", "a") + write exit 0 blocks
open("MEMORY.md", "w") + write exit 0 blocks
Path("MEMORY.md").write_text(…) exit 0 blocks

Two independent causes.

The pattern required the sink token before the path. That is the JS shape. Python puts the path first and the mode second, so the open alternative consumed straight past the filename and the destination could never be found after it. That is why even "w" failed, not only "a": no Python write could match this rule in any mode. pathlib was not covered at all.

CLAUDE.md was not in the destination list. It held MEMORY.md, SOUL.md, AGENTS.md, .claude/ and .cursor/. The most-loaded memory file for the agent this gate exists to protect was the one destination it did not look for. The rule's message was honest about its own list, so this was a coverage gap rather than a false claim.

Assertion 7 had been green for months on a TypeScript-only fixture. That is precisely how a language-shaped hole stays invisible, so the surface now carries one fixture per language.

A false positive: it flagged a write on a file that only reads

The shell-redirect alternative was >>? followed by up to 60 characters of anything. A single > is the second character of a JS arrow function, so any => within 60 characters of the string MEMORY.md was reported as a memory write. This blocked:

beforeAll(() => { const n = readFileSync("MEMORY.md", "utf8").length; });

Confirmed pre-existing by running that control against the published image before touching the rule, so it is not something this PR introduced. Reading agent memory is not the threat; writing to it is.

The rule is three branches now

Split by where the write intent sits relative to the destination, which is also what keeps a read quiet:

  1. JS/TS sink names the destination. Write sinks only, so readFileSync cannot match.
  2. Destination first, write intent after (open(path, mode), pathlib). The mode must be a write mode, so open(path, "r") stays quiet.
  3. Shell redirect, where the destination follows the operator directly rather than within 60 characters of it.

pattern-either is new in this file. The comment records why: the orderings genuinely cannot be expressed in one regex, and the previous attempt to do so is what produced both defects.

Verification

Eleven shapes checked locally against the compiled regexes (7 must-match, 4 must-not), zero mismatches. Then the full set re-run in-container against a real Semgrep:

  • five previously-missed shapes now block, including the computed-payload case from the #879 table
  • seven controls stay quiet: Python read, TypeScript read, write to report.md, non-auto-run helper writing MEMORY.md, benign-skill, config-injection-benign, memory-read-benign
  • gate-proof.sh passes 12 for 12

Three fixtures, three assertions

memory-poisoning-python-demo (10, blocks), memory-poisoning-claude-md-demo (11, blocks), memory-read-benign (12, must not block). Each fixture header records the specific defect it guards, not just what the file does, so the next reader knows why it exists.

Two things I did not do

Corrected a false sentence I wrote yesterday. Walkthrough Step 6 ended "The gate blocks on the write itself, which is why it fires on all four." It fires on two of four. One miss is the prose control, which has no write to find; the other was the CLAUDE.md row, which was the bug above rather than an intended negative.

Left two shapes passing, as residuals rather than assertions. An assembled destination (open("MEM" + "ORY" + ".md", "a")) and open reached through an indirection both still pass. Fixing them needs the path resolved rather than matched, which is a different kind of analysis than this rule pack does. They are recorded in the threat profile's honest residual and deliberately not asserted: a test that asserts the gate does not catch something turns a weakness into expected behaviour, and whoever improves the rule would have to delete a passing test to do it.

Worth noting those same two shapes also pass NVIDIA/SkillSpector#893, for the same reason, which is the feedback I left on that PR.

🤖 Generated with Claude Code

…itived

Found by building the boundary fixtures from NVIDIA/SkillSpector#879 against
our own gate rather than only against theirs. Two defects in
dev-exec-writes-agent-memory, pulling in opposite directions.

False negatives. The pattern required the sink token before the path, which
is the JS shape: appendFileSync("MEMORY.md", ...). Python puts the path first
and the mode second, so the `open` alternative consumed straight past the
filename and could not match a Python write in any mode. pathlib was absent.
So was CLAUDE.md, the most-loaded memory file for the agent this gate exists
to protect. Measured misses, all now blocked:

  open("MEMORY.md", "a")              was exit 0, now exit 1
  open("MEMORY.md", "w")              was exit 0, now exit 1
  Path("MEMORY.md").write_text(...)   was exit 0, now exit 1
  appendFileSync("CLAUDE.md", ...)    was exit 0, now exit 1

Assertion 7 had been passing on a TypeScript-only fixture, which is how a
language-shaped hole stays invisible. One fixture per language now.

A false positive. The shell-redirect alternative was `>>?` plus up to 60
characters of anything, and a single `>` is the second character of a JS arrow
function, so any `=>` within 60 characters of "MEMORY.md" reported a write.
readFileSync("MEMORY.md") inside beforeAll(() => {...}) blocked. Confirmed
pre-existing against the published image before touching the rule. Reading
agent memory is not the threat; writing to it is.

The rule is three branches now, split by where the write intent sits relative
to the destination. Eleven shapes checked locally against the compiled
regexes, then the full set re-run in-container: five previously-missed shapes
block, seven controls stay quiet, and gate-proof passes 12 for 12.

Also corrects a sentence I put in the walkthrough yesterday. Step 6 claimed
the gate "fires on all four" rows. It fires on two: one miss is the prose
control with no write, the other was the CLAUDE.md bug above.

Two shapes still pass and are recorded as residuals, not assertions: an
assembled destination, and open reached through an indirection. Both need the
path resolved rather than matched. Asserting them would turn a weakness into
expected behaviour and make the next improvement delete a passing test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@garymike
garymike merged commit 6b882e0 into main Oct 21, 2026
10 checks passed
@garymike
garymike deleted the fix/agent-memory-write-rule-coverage branch October 21, 2026 17:17
garymike added a commit that referenced this pull request Oct 21, 2026
…ublished (#88)

Fourth time today the five reusable-workflow digest pins went stale, so the
manual step gets a control.

#73 made those pins deliberate on purpose: a rebuild must not silently change
what every caller scans with, after betterleaks 1.6.1 -> 1.8.1 in #64 turned
green repos red overnight. The cost of "deliberate" is that nothing moves them
for you, and nothing reminded anyone either. #80 bumped eight tools and #81
bumped SkillSpector, both published, neither moved the pins, and callers kept
getting the previous toolchain while tools.lock said otherwise. #82 was the
cleanup. Every toolbox/** change since has put them behind again, including
#87 and #86.

Each build job now records the digest it published, and a new `pins` job
compares those against the pin in the matching reusable workflow.

Two design points that are the whole reason this works:

It is a SEPARATE job, after base and domain, so publishing is never blocked.
It cannot be blocked: the digest to pin does not exist until the publish has
happened, so failing the publish on a stale pin would be circular. The images
ship, and the run still goes red, so the follow-up is not optional. Same
contract as tool-drift.yml: red means a pin is behind, not that anything is
broken.

It reads steps.build.outputs.digest rather than the registry. Comparing
against the `latest` index digest false-reds on images whose index carries
attestations, which is exactly why I pulled the equivalent check back out of
check-tool-updates.sh in #82 rather than ship a flaky gate. The build already
holds both halves, so no lookup and no ambiguity.

mcp-review-toolbox is built here but no reusable workflow pins it, so it is
reported as unpinned rather than counted as a mismatch.

Verified by running the comparison standalone against three scenarios:
all pins matching exits 0; two behind exits 1 with both named in ::error::
and marked in the job summary; and no digest files at all exits 1 saying the
pins were NOT verified, rather than passing quietly, which is the failure mode
that let this go unnoticed in the first place. actionlint clean, zizmor clean.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
garymike added a commit that referenced this pull request Oct 21, 2026
…hed (#89)

The first refresh prompted by the check rather than by remembering. #88 added
a `pins` job to build-toolbox.yml; its own merge triggered a rebuild, the job
ran, and it went red naming all five stale pins with the digest each should
move to. A true positive on first firing, because #86 and #87 had already left
them behind.

  security-toolbox-base   d2f1d6b33d9c -> 9be060794ecb
  gha-toolbox             7e6558f3b46c -> 4cef3443acbd
  sast-toolbox            6bf6d9aafd5a -> edfd4db8752a
  iac-toolbox             bc4198b93fcc -> 08e1338df20c
  skill-audit-toolbox     c10bc17412a2 -> f5f99b96668f

Digests taken from the job's own annotations, which report
steps.build.outputs.digest from the run that published them. No local pull and
no registry lookup, so neither the index-vs-manifest ambiguity from #82 nor a
hand-copied digest can get in.

Verified behind the new pins: base ships trufflehog 3.99.3, betterleaks 1.9.0,
osv-scanner 2.7.0 and syft 1.55.0, with pip 26.2.2 vendoring certifi
2026.10.19, so #87's CVE fix is actually in what callers now get.
skill-audit ships SkillSpector v2.13.0.

One thing the check cannot currently survive: #86's build failed on a
cosign/Rekor 409, "an equivalent entry already exists in the transparency
log", because toolbox/gha was unchanged so gha-toolbox rebuilt to a byte
identical digest that was already signed. A failed domain arm skips the pins
job rather than running it. Benign cause, but it means a reproducible rebuild
can silence the check. Noted, not fixed here.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
garymike added a commit that referenced this pull request Oct 21, 2026
…#90)

Two faults, one cause, found because #86's build went red on both at once.

A rebuild whose inputs have not changed produces a byte-identical digest, and
Rekor rejects a second equivalent entry for it with 409
createLogEntryConflict. That is not a signing failure: the entry it conflicts
with IS this digest's signature, already in the log from the earlier run.
toolbox/gha was untouched by #86, gha-toolbox rebuilt to the same bytes, and
the sign step failed.

The sign step now matches createLogEntryConflict specifically and treats it as
already-signed, with a notice explaining why. Any other cosign failure still
fails: verified against four stubbed outcomes, where success and the 409 exit
0, and a missing OIDC token and a registry 403 both exit 1.

The second fault is worse, and the 409 is what exposed it. A red domain arm
skipped the pins job entirely, because needs: [base, domain] will not run on a
partial build. So a reproducible rebuild could silence the very check added in
#88 to watch the pins, while five images published with nobody comparing
anything. It now runs with if: !cancelled().

That raises the question the first version did not have to answer: what to do
about an image that never published. Nothing to compare, so it is reported as
"published no digest in this run, so its pin was NOT checked" rather than
dropped. Four greens must not read as a clean sweep when six images were
expected. Verified on a simulated partial build: the four that published are
confirmed current, and the two that did not are named as unchecked.

The zero-digest case still fails outright rather than warning. On a totally
failed build that is redundant with the build's own red, but the alternative
is a job that reports success having compared nothing, which is the failure
shape that let the pins drift unnoticed four times in the first place.

actionlint clean, zizmor clean.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant