[Marc-cn]

Implements the run-bundle flow from #677: a researcher records an experiment once; anyone can verify it, and rerun it. Spec: specs/046-amber-run-bundle/spec.md.

Commands

	who	what
darnit-amber run -- ./experiment.sh	researcher	runs the Amber audit and the command under Witness inside a clean checkout; stores a bundle: repo + commit, command and exit status, an allowlisted environment, the outputs the command produced and their contents. The statement is signed (DSSE) with a key generated once, timestamped by Sigstore's TSA
darnit-amber claim <id> --file --regex --threshold	researcher or reviewer	states the paper's number; darnit measures it from the stored output (digest-checked) and writes a derived bundle, signed by the claimer, with the verdict
darnit-amber verify <id> --pubkey k.pub / --expect-identity --expect-issuer	anyone	statement signature, envelopes bound to the statement, outputs vs Witness products and stored bytes, timestamps on every signature, signer as the caller names it; Rekor inclusion when published
darnit-amber rerun <id> …	anyone	verifies first, shows the command, asks; fresh clone at the recorded commit, same command; value and verdict first vs rerun, output drift

Verdicts: REPRODUCED only when every claim is, NOT_REPRODUCED when any is, else NOT_COMPARABLE (--not-comparable "reason" for hardware-bound numbers: recorded, not judged; ordering operators need two numbers). Claims are measured only from files Witness recorded as products of the experiment, never from the checkout. An entry file (--claims entry.yaml) gives the same information up front; packages/darnit-amber/examples/ has three.

Signing, time and what is public

Nothing leaves the machine by default except a hash.

	proves	public
default	the statement and both Witness envelopes are signed with the researcher's key and existed at the time in the RFC 3161 countersignature	nothing (the TSA sees a hash)
--identity	who: a ten-minute Fulcio certificate from a GitHub/Google login, no key to manage; verify requires the expected identity and issuer	the login, in Sigstore's certificate log
--publish	anyone can confirm it from Rekor without the researcher's cooperation; the bundle is stored before publishing	the envelopes and signer; outputs stay in the store

Bundles record no host name, user name, credentials or paths of the researcher; clone URLs lose their userinfo. Witness keeps the command's stdout/stderr (README says so).

Tested end to end

	result
SPF, AEADProtocols, GigaDORAM (USENIX Sec '23/'24)	run, claim, verify, rerun; verdicts as in the papers; S3 store via moto
--identity	real GitHub login from the VM; tests/darnit_amber/fixtures/fcaecc71….intoto.json
--publish	Rekor entries 3691869540, 3691869544, 3691869564 (statement, audit, experiment); tests/darnit_amber/fixtures/c3e2fda5….intoto.json

Both fixtures were made under env -i with a throwaway HOME and verify offline in the test suite, so the Sigstore paths are covered in CI without a login or network.

What's in the PR

• run.py, claim.py, rerun.py: run, Witness capture, statement signing, claims, verify, rerun
• identity.py, publish.py: Fulcio and Rekor
• s3_store.py: S3-compatible AttestationStore (entry point s3; boto3 is the darnit-amber[s3] extra)
• cli.py: darnit-amber run | claim | verify | rerun
• README, spec 046, CHANGELOG
• 61 tests in test_run_bundle.py; full suite 5270 passed on the VM

Review

A cold read (2026-10-30) found three ways verify could report verified without having checked what it claimed; fixed with regression tests. The 2026-11-01 review found eight more (the predicate was unsigned, claims measured from committed files, string comparisons, unchecked stored outputs, secrets in the environment, a test writing a real key, rerun executing unverified bundles, --pubkey ignored for certificates) and a should-fix list; all addressed, each with a test, see the comment of 2026-11-04. Follow-ups in #700.

Not in this PR

Pinned darnit image (#677 step B), registry for artifact images, corpus index, darnit-amber list, MCP tools, claim --identity. Stacked on #532 (this branch carries its commit); to be split into smaller PRs once #532 lands.

Assisted by Claude Code

[kusari-inspector]

Kusari Analysis Results:

✅ No Flagged Issues Detected
All values appear to be within acceptable risk parameters.

Both analyses recommend PROCEED. The code analysis found no code issues, exposed secrets, or workflow issues. The dependency findings are limited to test fixture requirements.txt files, which lowers urgency. The only new package, darnit-amber 0.1.1, is the project's own package, though it has no scorecard data. Numpy 1.26.4 is flagged end of life but has no advisories. The unpinned numpy in tests/darnit_amber/fixtures/compound_repo/requirements.txt lists 8 older advisories (e.g. CVE-2019-6446, CVE-2021-41495, CVE-2017-12852). If the unpinned entry is not deliberate, pin it with pip install numpy==1.26.4. Click 8.1.7 has one advisory, CVE-2026-7246, which requires local access, high privileges and user interaction (EPSS 0.009). It has NO_FIX available, so it is unactionable and does not change the recommendation. No KEV entries, blocked packages, or license concerns were found. Combined, the risk profile is low and the PR is safe to merge.

Note

View full detailed analysis result for more information on the output and the checks that were run.

---

@kusari-inspector rerun - Trigger a re-analysis of this PR
@kusari-inspector feedback [your message] - Send feedback to our AI and team
See Kusari's documentation for setup and configuration.
Commit: bf56017, performed at: 2026-10-30T17:13:17Z

Found this helpful? Give it a 👍 or 👎 reaction!

[Marc-cn]

@kusari-inspector feedback The flagged pins (requests 2.31.0, numpy 1.0) are test fixtures for the requirements pin classifier; they are old on purpose and are never installed. They come from the #532 renamnot from this PR's commits.

[kusari-inspector]

Thank you for your feedback! 📝 Your message has been submitted for review.

[kusari-inspector]

Kusari PR Analysis rerun based on - 079fdf8 performed at: 2026-10-28T14:59:58Z - link to updated analysis

[kusari-inspector]

Kusari PR Analysis rerun based on - e20bebb performed at: 2026-10-28T15:06:34Z - link to updated analysis

[kusari-inspector]

Kusari PR Analysis rerun based on - 3690844 performed at: 2026-10-30T15:29:10Z - link to updated analysis

[kusari-inspector]

Kusari PR Analysis rerun based on - b18a2d9 performed at: 2026-10-30T15:46:47Z - link to updated analysis

[kusari-inspector]

Kusari PR Analysis rerun based on - 073b9cc performed at: 2026-10-30T15:48:26Z - link to updated analysis

[kusari-inspector]

Kusari PR Analysis rerun based on - 43d3e9e performed at: 2026-10-30T15:56:11Z - link to updated analysis

[kusari-inspector]

Kusari PR Analysis rerun based on - bf56017 performed at: 2026-10-30T17:13:31Z - link to updated analysis

[mlieberman85]

Requesting changes. The framework/implementation split holds, and the tests never trigger a Sigstore login. But verify reports verified: true for content nothing signs, and I got false REPRODUCED results several ways. I know this is a draft PR, so take this as early feedback. I used a local key with --no-timestamp, with no keyless signing, TSA or Rekor. Items that use the PR's own fixtures verify them with --offline. Where I patched code or injected a test root to reach a path, the item says so.

Blocking

B1. verified: true covers the Witness envelopes, not the predicate.
verify (rerun.py:181-240) checks the envelopes and their timestamps, and binds the id, the command, the signer fingerprint and the output digests. It never checks claims, verdict, audit.results, environment, entry, signing.identity, or the subject's name and commit against anything signed. The bundle id is content-addressed, but anyone can compute a new id over edited content that reuses the researcher's envelopes. On the PR's own fixtures I added a REPRODUCED claim, set verdict: REPRODUCED, set every audit result to PASS and recomputed the id:

• key fixture: verify <new id> --pubkey tests/darnit_amber/fixtures/published-signing.pub --offline gave exit 0, verified: true.
• identity fixture, with signing.identity also rewritten to another address: exit 0, verified: true, identity_verified: true. The forged address is what signing.identity shows, while the certificate subject beside it is still the real one, and nothing compares them.

claim writes derived bundles nobody signs, and verify treats them the same way. Fix: sign the amber/run/v1 statement itself (DSSE, same signer, timestamped; claim signs its derived statement as the claimer). Or have verify recompute every derived field from signed inputs and report anything it can't recompute as unverified.

B2. Claims are measured from files the experiment didn't produce.
measure_claim (run.py:229-260, :474-475) reads repo / measure.file from the checkout whether or not Witness recorded it as a product. A repo with a committed results/table.txt (speedup: 2.4), whose experiment only wrote logs/run.log ("benchmark binary not found; skipping"), got table3-speedup: measured 2.4, REPRODUCED and overall REPRODUCED. That is exactly the case of artifacts that ship precomputed results. Fix: measure only captured products, from the captured bytes, as claim.py does.

B3. Non-numeric comparisons.
_coerce (run.py:222-226) falls back to strings. Measured 9 ms against > 10 gave REPRODUCED ("measured 9 ms > 10"). >= 10 crashed with TypeError: unsupported operand type(s) for -: 'str' and 'int' after both Witness steps had run, and nothing was stored. < and > also ignore tolerance. Fix: ordering operators require two numbers, otherwise NOT_COMPARABLE with a reason, and measure_claim never raises.

B4. Stored outputs are never checked against their digest, so a false claim needs no JSON editing.
claim.py:33-34 measures from read_blob without comparing it to out["sha256"], and verify never hashes stored blobs. On a copy of the PR's key fixture, whose verdict is NOT_COMPARABLE, I wrote score: 99 to <store>/<id>/outputs/result.txt and ran claim <id> --id x --file result.txt --regex 'score: ([0-9]+)' --threshold '>= 50'. It gave REPRODUCED, and verify on the new id with the published key gave exit 0. With the S3 store, anyone with bucket write access can do this. Fix: hash on every read.

B5. Secrets go into the bundle.
run.py:324 passes the full environment, minus SIGSTORE_ID_TOKEN, to Witness. Witness masked AWS_SECRET_ACCESS_KEY, GITHUB_TOKEN, ANTHROPIC_API_KEY and MY_DB_PASSWORD, but stored a fake RESEARCH_SERVICE_CREDENTIAL verbatim in both envelopes, along with hostname, username, HOME and PWD. Masking also doesn't reach output: an experiment that echoes $MY_DB_PASSWORD stores the value verbatim in the command-run attestation, while the environment attestor masks it. A clone URL with a token in it goes verbatim into subject.name and entry.artifact.clone (run with a fake token through git config url.<path>.insteadOf). #700 lists the environment, hostname and user as non-blocking, but bundles exist to be shared, so I'd block on it. Fix: strip userinfo from URLs, and give the environment attestor an allowlist, since a blocklist misses names it doesn't know. Witness passes its environment to the child, so don't narrow what the experiment itself sees. Document that stdout and stderr are recorded.

B6. A test writes a real signing key into the developer's data directory.
test_default_key_is_generated_once_with_tight_permissions (tests/darnit_amber/test_run_bundle.py:256-268) sets XDG_DATA_HOME, which user_data_root() ignores on macOS (platform_paths.py:80). The test fails, after default_key() has already written signing.key/signing.pub under ~/Library/Application Support/darnit/amber/. Any later darnit-amber run on that machine signs with it. Linux CI won't catch this. Fix: monkeypatch user_data_root, ideally in the autouse _isolate_operator_config in tests/conftest.py.

B7. rerun verifies nothing and runs the bundle's command on the host.
A bundle with no envelopes, an edited command and a made-up id still executes, with the full environment, no container and no confirmation. A command of cat "$HOME/Library/Application Support/darnit/amber/signing.key" > o.txt put the default private key into the new bundle's stored outputs (I confirmed the hash, in a throwaway HOME). Fix: verify first, print the command and require consent until #677's container lands, and say so in the README.

B8. --pubkey is ignored for certificate-bearing envelopes.
verify 97e67cac... --out tests/darnit_amber/fixtures --pubkey <unrelated key> --offline gives exit 0, verified: true, with no signer_matches_statement (rerun.py:163-164, :225). The key fixture rejects the same key. There's also no way to name an expected identity: with a test root I injected as _TRUST_ROOT, a self-made attacker identity verified. So anyone pinning a researcher's key or identity accepts a bundle from any Fulcio identity. Fix: honor --pubkey, and require the expected identity and issuer, as cosign does.

Should fix

• Dirty tree recorded as a clean commit. The subject gitCommit is HEAD even with a modified tree; only the git status inside the envelope shows it. Refuse a dirty tree, or record it and don't call the result reproducible from the commit.
• measure.file isn't constrained. /etc/hosts and a ../-relative path to it were both read and recorded. run.py uses safe_rel for outputs but not for measure.file.
• environment.darnit is null in the e2e bundle. _version("darnit") looks up the wrong name; the distribution is darnit-core.
• run on a local path leaves amber-audit.json in the checkout. It would also overwrite a tracked file of that name.
• rerun of a local-path bundle uses the current checkout, not the recorded commit. I ran at one commit, committed a change and reran: the new code ran. README:64 says "same commit, same command, fresh clone".
• The overall verdict may be too lenient. One REPRODUCED plus NOT_COMPARABLE gives REPRODUCED, and test_run_bundle.py:130 pins that, so it looks deliberate. The README says the worst claim decides but doesn't rank NOT_COMPARABLE. A claim that couldn't be measured ("output not produced", "regex did not match") isn't reproduced, so I'd count it against the verdict and change that test.
• A failing experiment produces no bundle. A command that exits 3 returns exit 1 with an empty store, so a crash can't be recorded as a failed reproduction.
• --publish failure loses the run. With publish_envelope patched to raise: exit 1 and an empty store, despite the comment at run.py:488-491. Store first, keeping in mind that publication is inside the statement and feeds the id.
• The timestamp check uses all() over the stamps that exist. On the key fixture with one envelope's stamp stripped and the id recomputed: exit 0. With both stripped: exit 2.
• Error handling. cli.py:153 catches only RunError, so a missing --claims, a bad regex and the B3 TypeError all print tracebacks.

Scope and process

• 129 files: #532's rename (114 of them), #677 steps A-C, claims, an S3 store, Fulcio and Rekor. I'd land #532 first, then split out the S3 store and --publish/--identity.
• New hard dependencies for every darnit-mcp install: boto3, sigstore (through the private sigstore._internal), pyopenssl, rfc3161-client. At least the S3 store should be an extra.
• It differs from #677's decisions: no container (1), key signing by default rather than keyless (3), and an S3 store rather than GHCR (4). You flagged the bucket on #677; the other two should be decided there as well. There's no new spec under specs/ for the run bundle, and no CHANGELOG entry for it.
• run, verify, claim and rerun are CLI only, with no MCP tools. Worth a design discussion, since the product is MCP plus agent.
• Five commits carry Co-Authored-By: Claude where AI_POLICY.md asks for Assisted-by (the project's CLAUDE.md rules it out), and 16 lack Signed-off-by, so DCO fails. The PR conflicts with main (CHANGELOG.md, test_plan_contract.py), and no CI workflow has run on this head, only DCO and Kusari Inspector.
• Both new fixtures contain your home path, and the identity fixture also contains your email. Flagging it in case you didn't mean to commit them.

Tests on ad3fa54, with HOME pointed at a temp directory: 5322 passed, 11 skipped, 1 failed (B6). ruff and validate_sync pass.

---

Drafted with AI assistance (Claude), reviewed and filed by me. Measurements quoted were executed and verified.

[Marc-cn]

Thanks for the review,every finding reproduced, and all of them are addressed on 3f509be (rebased on current main through #532's new head). The 21 earlier commits are rewritten with Assisted-by and my sign-off, same trees; the fixes are three commits on top, plus regenerated fixtures and uv.lock.

Blocking:

• B1: the statement is now signed (witness sign, same signer and TSA) as statement_envelope; verify requires the signed payload to equal the canonical statement byte for byte, and the certificate's subject/issuer to equal signing. A claim bundle is signed by the claimer (claim_signer). Your forgery (edit, recompute id) now fails statement_signed.
• B2: claims are measured only from bytes Witness recorded as products; measure.file goes through safe_rel. The committed results/table.txt case is a test.
• B3: </<=/>/>= need two numbers, else NOT_COMPARABLE with the reason; tolerance applies to all of them; measurement never raises.
• B4: every read of a stored output checks its digest (claim refuses, verify reports stored_outputs_match).
• B5: clone URLs lose userinfo; the environment is an allowlist recorded in the statement and Witness's environment attestor is not used (so no hostname/username/HOME/PWD either); README says stdout/stderr are recorded.
• B6: tests/conftest.py replaces darnit_amber.run.user_data_root for every test.
• B7: rerun verifies first and executes nothing from a bundle that does not verify, prints the command and needs --yes (or a terminal answer); README says it runs on the host.
• B8: --pubkey must be the key in the certificate; --expect-identity and --expect-issuer are required for an identity bundle (cosign style).

Should fix: dirty checkouts refused (--allow-dirty records no commit); measure.file confined; darnit-core looked up; audit JSON written outside the checkout; rerun clones a local path at the recorded commit; verdict is REPRODUCED only when every claim is (test changed); a failing experiment still gets a bundle (the command runs in a wrapper that records its exit status, and verify recomputes the wrapper from the recorded command, since Witness writes nothing on a non-zero exit); the bundle is stored before any Rekor call and publication moved beside the statement; every signature needs a trusted timestamp; the CLI reports errors instead of tracebacks.

Process: spec at specs/046-amber-run-bundle/spec.md, CHANGELOG entry, boto3 is now the darnit-amber[s3] extra (sigstore/pyopenssl/rfc3161-client stay: verify needs the trusted root for any bundle). Fixtures regenerated under env -i with a throwaway HOME: no path, host or user in them; the email in the identity fixture is the Fulcio subject and is meant to be there. Full suite 5270 passed, ruff and validate_sync clean, on the VM.

Not done here, as you suggested: the split into PRs once #532 lands (S3 store, --identity, --publish), MCP tools, and the three #677 decisions (container, key vs keyless default, S3 vs GHCR). I'll take those to #677.