diff --git a/.gitignore b/.gitignore index 45a037d..766aa1c 100644 --- a/.gitignore +++ b/.gitignore @@ -299,4 +299,6 @@ Testing/ *.fatbin # Specifics -gpu-proof.json \ No newline at end of file +gpu-proof.json +# Local working notes (never committed) +docs/open-tasks/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 978fd3f..4e232ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,57 @@ All notable changes to pytest-gpu-proof are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/); versions follow [SemVer](https://semver.org/) (pre-1.0: minor bumps may break). +## [0.4.1] — 2026-10-12 + +### Fixed + +- Shard entries record `dirty` (the recording tree's state); merge ORs shard + flags into the merged `repo.dirty`, carry-forward refuses a dirty shard, and + verification refuses any dirty shard unless `allow_dirty` is set. +- `gpu-proof merge --carry-from` refuses a source receipt recorded with a dirty + working tree. The merged `repo.dirty` is OR-ed over the fresh inputs only, so a + carried shard from a dirty recording was laundered into a clean-looking merged + receipt (found by GRiD's split driver, 2026-10-10). Carried shards now record + `carried.original_dirty`, and verification refuses a carried shard recorded + dirty unless `allow_dirty` is set. + +- The expected-skips mismatch names the delta (unexpected skips and stale + baseline entries) instead of only reporting that the sets differ. +- The receipt's `environment.gpu_info` falls back to the same torch probe the + `gpu_required` marker uses when nvidia-smi is absent or fails (`source` + records which probe answered), so a torch-only box no longer records + `gpu_info: null` and then fails a `require_gpu` policy. + +- Verification refuses a receipt whose session started more than five + minutes before its commit existed (committer time): a cheap replay and + forgery guard that needs no nonce. + +- `mode: ci-gpu` is now bound to the signing key: a ci-gpu receipt verifies + only when the policy lists `allowed_key_fingerprints` and the receipt was + signed by one of them (previously the mode was an unverified label). +- Without an explicit `carried_max_age_days`, a carried shard's original run + may not be older than the receipt's `max_age_days` (was a fixed 30 days). + +### Added + +- `gpu-proof merge --verify-inputs`: authenticate every input shard against + its signer's current GitHub keys before merging (unsigned or tampered inputs + are refused). +- `gpu-proof status PATH...`: one-line summaries of receipt files or + directories (commit, dirty, signed, outcome, counts, shards, carried). +- `gpu-proof durations RECEIPT... --out map.json`: fold per-test durations + into a JSON map for shard bin-packing. +- Command-line equivalents for the toml-only options: + `--gpu-proof-no-gpu-uuid`, `--gpu-proof-require-gpu`, + `--gpu-proof-max-age-days N`. +- `gpu-proof plan --receipt OLD [--repo .] [--json]`: a dry run of the + carry-forward conditions against the current tree, reporting per shard + whether it could be carried or must be re-run (and why). Drivers that plan + partial re-runs no longer reimplement the checks. +- `record_gpu_uuid` (`[tool.gpu_proof]`, default `true`): set `false` to omit each GPU's + nvidia-smi UUID from the receipt environment while keeping name, driver, memory and + compute capability. Verification does not read the UUID. + ## [0.4.0] — 2026-08-17 ### Added diff --git a/README.md b/README.md index 5c1d48c..9007f63 100644 --- a/README.md +++ b/README.md @@ -92,15 +92,23 @@ fingerprint scope, no selected tests, or write failure makes pytest fail and leaves no stale receipt behind. `--gpu-proof-best-effort` is an explicit development-only opt-out. -## Four interfaces +## Five interfaces -The project exposes four small interfaces: +The project exposes five small interfaces: 1. **Markers** select receipt tests. -2. **`gpu_proof_check`** compares a reference callable with a candidate. +2. **`gpu_proof_check`** compares a reference callable with a candidate + (optional; every current user selects tests with the marker alone). 3. **`gpu-proof verify`** validates a receipt and repository policy. 4. **`gpu-proof merge`** combines separately executed shards and optionally carries unchanged shards forward. +5. **`gpu-proof plan`** reports, before a partial re-run, which shards of an + old receipt could still be carried and which must be re-run. + +Two maintenance helpers round it out: `gpu-proof status PATH...` summarizes +receipt files (commit, dirty, signed, outcome, counts, shards), and +`gpu-proof durations RECEIPT... --out map.json` folds per-test durations into +a map that shard bin-packers can calibrate from. ### Markers @@ -163,6 +171,7 @@ fingerprint_excluded_paths = ["gpu-proof.json"] required_marker = "gpu_proof" max_age_days = 30 require_gpu = true +record_gpu_uuid = false # keep device name/driver/memory, omit the card's UUID ``` By default, the fingerprint covers every tracked file in the repository, @@ -270,6 +279,15 @@ runtime environments, duplicate tests, and duplicate shard names. It records input provenance but does not verify each input signature; the merger signs for the union. See [sharding and carry-forward](docs/sharding.md). +Carry-forward (`gpu-proof merge ... --carry-from last-green.json`) grafts an +absent old shard only when its receipt commit is an ancestor of the current +one, both the receipt and the shard were recorded with a clean tree, its +narrow fingerprint recomputes identically, and it introduces no duplicate +test. `gpu-proof plan --receipt last-green.json` prints the same verdicts +without merging. Generated inputs named through `fingerprint_extra_paths` +should be gitignored: an untracked generated file makes the recording tree +dirty, and dirty recordings are never carried. + ## Receipt schema New receipts use schema `"3"`. Signer identity is inside the signed payload, diff --git a/docs/ci_gpu_mode.md b/docs/ci_gpu_mode.md index 8824147..ee13c1e 100644 --- a/docs/ci_gpu_mode.md +++ b/docs/ci_gpu_mode.md @@ -2,7 +2,10 @@ CI-GPU mode uses the same receipt and verifier but records `mode: ci-gpu`. Choose it when GPU tests must run on a controlled runner rather than a -developer machine. +developer machine. The verifier binds the mode to the runner's signing key: +a `ci-gpu` receipt verifies only when the repository policy lists +`allowed_key_fingerprints` and the receipt was signed by one of them, so the +label cannot be claimed by an arbitrary developer key. ## Dedicated signing identity diff --git a/docs/policy.md b/docs/policy.md index f69c5b3..46b6e14 100644 --- a/docs/policy.md +++ b/docs/policy.md @@ -53,7 +53,7 @@ schema 3. Repositories that have finished migrating should pin | `allowed_key_fingerprints` | SSH SHA-256 fingerprints accepted in restricted mode | | `min_schema` | Minimum acceptable receipt schema version (1, 2, or 3) | | `max_age_days` | Maximum age of the merged/session receipt | -| `require_mode` | Required receipt mode: `local` or `ci-gpu` | +| `require_mode` | Required receipt mode: `local` or `ci-gpu` (a `ci-gpu` receipt additionally requires `allowed_key_fingerprints` and must be signed by a listed key) | | `allow_dirty` | Permit dirty recording or verification trees | | `required_fingerprint_paths` | Exact tracked path list | | `required_fingerprint_extra_paths` | Exact generated/ignored path list | @@ -61,7 +61,7 @@ schema 3. Repositories that have finished migrating should pin | `required_test_manifest` | Repository-relative file containing exact node IDs | | `required_shard_fingerprints` | Exact paths/extras for every named shard | | `allow_carried` | Permit carry-forward shards | -| `carried_max_age_days` | Maximum age of each carried shard's original run | +| `carried_max_age_days` | Maximum age of each carried shard's original run (default: `max_age_days`, capped at 30) | Example shard scope: diff --git a/docs/security_model.md b/docs/security_model.md index ebe2e20..566c4de 100644 --- a/docs/security_model.md +++ b/docs/security_model.md @@ -75,7 +75,9 @@ escaping, or unmerged inputs fail closed. Schema 3 rejects dirty recording and verification trees by default. A receipt may verify at its exact commit or a descendant only when the source manifest still matches. Freshness limits replay duration but no server-issued nonce is -currently used. +currently used; a session that started more than five minutes before its +commit's committer time is refused, so a receipt cannot be back-dated onto a +commit that did not yet exist. `allow_dirty: true`, long age limits, `--allow-skipped`, and `--allow-unsigned` weaken guarantees. Unsigned mode authenticates no signer, @@ -98,8 +100,10 @@ each input signature. The merger's signature attests to the union. A review workflow that needs independent shard authentication should verify every input before merging. -Carry-forward means a test passed at an ancestor commit and its declared shard -inputs are byte-identical now. It does not establish that omitted dependencies +Carry-forward means a test passed at an ancestor commit, with a clean working +tree, and its declared shard inputs are byte-identical now. A receipt recorded +dirty is never a carry source: the merged `repo.dirty` is computed from fresh +inputs only, so carrying from it would launder the dirty flag. It does not establish that omitted dependencies or the old environment remain equivalent. Carried shards are rejected unless policy opts in and bounds their original age. diff --git a/docs/sharding.md b/docs/sharding.md index 5e63942..a3150c0 100644 --- a/docs/sharding.md +++ b/docs/sharding.md @@ -38,8 +38,10 @@ session failed. The merged receipt records input source names, counts, times, and recorded signers, then signs the result with the merger's key. Input signatures are -provenance, not independently verified during offline merge. The merger -attests to the union. +provenance, not verified during an offline merge, unless `--verify-inputs` is +passed: then every input must verify against its signer's current GitHub keys +(unsigned inputs are refused) before the union is formed. The merger attests +to the union either way. ## Carry forward unchanged shards @@ -55,10 +57,22 @@ An absent old shard is carried only when: 1. its receipt commit is the current commit or an ancestor; 2. its stored narrow manifest—including explicit extra paths—recomputes identically at the current tree; -3. it does not duplicate a fresh node ID; -4. every claimed node ID exists in the old tests list. +3. its receipt was recorded with a clean working tree (`repo.dirty` false): a + dirty recording may have exercised uncommitted bytes outside the shard's + narrow manifest, and the merged `repo.dirty` covers fresh inputs only; +4. it does not duplicate a fresh node ID; +5. every claimed node ID exists in the old tests list. + +Every shard entry records `dirty` (the recording tree's state), merge ORs +shard flags into the merged `repo.dirty`, carry-forward refuses a dirty shard, +each carried shard records `carried.original_dirty`, and verification refuses +any dirty shard unless `allow_dirty` is set. Freshly rerun shards always win. Changed or malformed scopes require a rerun. +`gpu-proof plan --receipt old.json --repo .` applies the same conditions as a +dry run and reports, per shard, `carryable` or `stale` with the reason (add +`--json` for drivers), so a partial re-run can be planned without reimplementing +the checks. Verification rejects carried shards by default: diff --git a/pyproject.toml b/pyproject.toml index d70c630..252b10d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "pytest-gpu-proof" -version = "0.4.0" +version = "0.4.1" description = "pytest plugin for GPU equivalence testing with signed receipts verified via GitHub SSH keys" readme = "README.md" requires-python = ">=3.11" diff --git a/src/pytest_gpu_proof/__init__.py b/src/pytest_gpu_proof/__init__.py index 6a9beea..3d26edf 100644 --- a/src/pytest_gpu_proof/__init__.py +++ b/src/pytest_gpu_proof/__init__.py @@ -1 +1 @@ -__version__ = "0.4.0" +__version__ = "0.4.1" diff --git a/src/pytest_gpu_proof/cli.py b/src/pytest_gpu_proof/cli.py index 804a6f5..948fab9 100644 --- a/src/pytest_gpu_proof/cli.py +++ b/src/pytest_gpu_proof/cli.py @@ -106,9 +106,116 @@ def main(): "allow_carried (default: rejected).") mp.add_argument("--repo", default=".", metavar="PATH", help="Repo root for carry-from fingerprint/ancestry checks") + mp.add_argument("--verify-inputs", action="store_true", default=False, + help="Authenticate every input shard's signature against the " + "signer's current GitHub keys before merging (unsigned inputs " + "are refused)") + + # status + sp = subparsers.add_parser( + "status", + help="Summarize receipt files (commit, dirty, outcome, tests, shards, signed)", + ) + sp.add_argument("paths", nargs="+", metavar="PATH", + help="Receipt files or directories (every *.json inside)") + sp.add_argument("--json", action="store_true", default=False) + + # durations + dp = subparsers.add_parser( + "durations", + help="Fold per-test durations from receipts into a JSON map (bin-packing calibration)", + ) + dp.add_argument("receipts", nargs="+", metavar="RECEIPT") + dp.add_argument("--out", required=True, metavar="PATH", + help="JSON map node_id -> seconds; merged into an existing file") + + # plan + pp = subparsers.add_parser( + "plan", + help="Dry run of carry-forward: which shards of an old receipt could be " + "carried against the current tree and which must be re-run", + ) + pp.add_argument("--receipt", required=True, metavar="PATH", + help="Older sharded receipt (the one you would --carry-from)") + pp.add_argument("--repo", default=".", metavar="PATH", + help="Repository root for fingerprint/ancestry checks (default: .)") + pp.add_argument("--json", action="store_true", default=False, + help="Print the plan as JSON instead of one line per shard") args = parser.parse_args() + if args.command == "plan": + import json + + from .merge import MergeError, load_receipt, plan_carry + + try: + report = plan_carry(load_receipt(args.receipt), repo_root=args.repo) + except MergeError as e: + print(f"gpu-proof plan: {e}", file=sys.stderr) + sys.exit(1) + if args.json: + print(json.dumps(report, indent=2, sort_keys=True)) + else: + r = report["receipt"] + print(f"receipt {str(r.get('commit_sha'))[:12]} vs HEAD {str(r.get('head'))[:12]}: " + f"{'carryable' if r['carryable'] else 'NOT carryable'} ({r['reason']})") + for s in report["shards"]: + print(f" {s['verdict']:9s} {s['name']} ({s['node_count']} tests): {s['reason']}") + n = sum(1 for s in report["shards"] if s["verdict"] == "carryable") + print(f"{n}/{len(report['shards'])} shard(s) carryable") + sys.exit(0) + + if args.command == "status": + import json + import os + + from .merge import receipt_status + + files = [] + for p in args.paths: + if os.path.isdir(p): + files += sorted(os.path.join(p, f) for f in os.listdir(p) if f.endswith(".json")) + else: + files.append(p) + rows = [receipt_status(f) for f in files] + if args.json: + print(json.dumps(rows, indent=2, sort_keys=True)) + else: + for r in rows: + if "error" in r: + print(f"{r['file']}: {r['error']}") + continue + print(f"{r['file']}: schema {r['schema']} @ {str(r['commit_sha'])[:12]}" + f"{' DIRTY' if r['dirty'] else ''}{'' if r['signed'] else ' UNSIGNED'}" + f" {r['outcome'] or '-'} tests={r['tests']} failed={r['failed']} " + f"skipped={r['skipped']} shards={len(r['shards'])} carried={r['carried']} " + f"ended={r['ended_at']}") + sys.exit(0) + + if args.command == "durations": + import json + + from .merge import MergeError, collect_durations, load_receipt + + try: + receipts = [load_receipt(p) for p in args.receipts] + except MergeError as e: + print(f"gpu-proof durations: {e}", file=sys.stderr) + sys.exit(1) + existing = {} + try: + with open(args.out) as f: + existing = json.load(f) + except (OSError, ValueError): + existing = {} + merged = collect_durations(receipts, existing if isinstance(existing, dict) else {}) + with open(args.out, "w") as f: + json.dump(merged, f, indent=2, sort_keys=True) + f.write("\n") + print(f"durations: {len(merged)} node(s) -> {args.out}") + sys.exit(0) + if args.command == "merge": from .merge import MergeError, merge_receipts @@ -120,6 +227,7 @@ def main(): unsigned=args.unsigned, carry_from=args.carry_from, repo_root=args.repo, + verify_inputs=args.verify_inputs, ) except MergeError as e: print(f"gpu-proof merge: {e}", file=sys.stderr) diff --git a/src/pytest_gpu_proof/config.py b/src/pytest_gpu_proof/config.py index c2212c7..d9e96df 100644 --- a/src/pytest_gpu_proof/config.py +++ b/src/pytest_gpu_proof/config.py @@ -21,6 +21,9 @@ class GpuProofConfig: github_username: Optional[str] = None max_age_days: int = 30 require_gpu: bool = False + # Record each device's nvidia-smi UUID in the receipt environment. Off keeps the + # device name, driver, memory and compute capability but not the card identity. + record_gpu_uuid: bool = True # Sharded emission: a declared shard name + its narrow # fingerprint paths (None -> the global fingerprint_paths). shard_name: Optional[str] = None @@ -92,7 +95,7 @@ def path_list(opt_name, toml_key, default=None): return [p.strip() for p in raw.split(",") if p.strip()] return [str(p) for p in raw] - max_age = toml_cfg.get("max_age_days") + max_age = resolve("--gpu-proof-max-age-days", "max_age_days", None) max_age_days = int(max_age) if max_age is not None else 30 return GpuProofConfig( @@ -117,7 +120,11 @@ def path_list(opt_name, toml_key, default=None): ), github_username=resolve("--gpu-proof-github-user", "github_username", None), max_age_days=max_age_days, - require_gpu=bool(toml_cfg.get("require_gpu", False)), + # store_true flags can only turn a setting ON (or, for the UUID, OFF). + require_gpu=bool(opt("--gpu-proof-require-gpu", False) + or toml_cfg.get("require_gpu", False)), + record_gpu_uuid=bool(toml_cfg.get("record_gpu_uuid", True)) + and not bool(opt("--gpu-proof-no-gpu-uuid", False)), shard_name=resolve("--gpu-proof-shard", "shard_name", None), shard_fingerprint_paths=shard_paths, shard_fingerprint_extra_paths=( diff --git a/src/pytest_gpu_proof/gitutils.py b/src/pytest_gpu_proof/gitutils.py index 78f56d7..564f98f 100644 --- a/src/pytest_gpu_proof/gitutils.py +++ b/src/pytest_gpu_proof/gitutils.py @@ -49,6 +49,15 @@ def get_branch(root: str = ".") -> Optional[str]: return _git("rev-parse", "--abbrev-ref", "HEAD", root=root) +def get_commit_time(sha: str, root: str = ".") -> Optional[int]: + """Committer time (Unix seconds) of ``sha``, or None when git cannot answer.""" + value = _git("show", "-s", "--format=%ct", sha, root=root) + try: + return int(value) if value else None + except ValueError: + return None + + def is_dirty( root: str = ".", *, diff --git a/src/pytest_gpu_proof/merge.py b/src/pytest_gpu_proof/merge.py index 7a2343f..fa10ff9 100644 --- a/src/pytest_gpu_proof/merge.py +++ b/src/pytest_gpu_proof/merge.py @@ -119,7 +119,8 @@ def merge_payloads(receipts: List[dict], sources: List[str]) -> dict: # the first shard. `dirty` is OR-ed — one dirty shard makes the merged # attestation dirty, and the verifier's dirty policy then applies honestly. merged["repo"] = dict(receipts[0].get("repo", {})) - merged["repo"]["dirty"] = any(r.get("repo", {}).get("dirty") for r in receipts) + merged["repo"]["dirty"] = any(r.get("repo", {}).get("dirty") for r in receipts) or any( + shard.get("dirty") for r in receipts for shard in (r.get("shards") or [])) # environment: fields asserted identical above; gpu_info from the first # shard that has one (a CPU-only shard shouldn't erase the GPU record). merged["environment"] = dict(receipts[0].get("environment", {})) @@ -181,7 +182,12 @@ def carry_forward(payload: dict, old_receipt: dict, old_source: str, 1. the old receipt is schema '2' and its commit is an ANCESTOR of the fresh payload's commit (same history, older point); 2. the shard's narrow fingerprint recomputes IDENTICAL against the - current tree — the inputs that shard proved are unchanged at HEAD. + current tree — the inputs that shard proved are unchanged at HEAD; + 3. the old receipt was recorded with a CLEAN tree (its `repo.dirty` is + false) — a dirty recording may have exercised uncommitted bytes + outside the shard's narrow fingerprint, and the merged `repo.dirty` + covers fresh inputs only. The carried metadata records + ``original_dirty`` so the verifier can refuse it independently. Shard signatures are provenance (merge is offline); the re-signed merged receipt is the attestation, and the verifier re-checks every carried @@ -199,6 +205,16 @@ def carry_forward(payload: dict, old_receipt: dict, old_source: str, raise MergeError( f"--carry-from {old_source}: its commit {old_sha[:12]} is not an " f"ancestor of {new_sha[:12]} — different history, cannot carry.") + # A receipt recorded with a dirty tree attests to unknown bytes: its shards + # ran against uncommitted edits that may lie OUTSIDE their narrow + # fingerprints, so a digest match at HEAD proves nothing. The merged + # receipt's `repo.dirty` is OR-ed over the FRESH inputs only, so carrying + # such a shard would launder it into a clean-looking descendant. Refuse. + if old_receipt.get("repo", {}).get("dirty"): + raise MergeError( + f"--carry-from {old_source}: that receipt was recorded with a dirty " + f"working tree; its shards cannot be carried — re-run them.") + merged_dirty = bool(payload.get("repo", {}).get("dirty")) fresh_names = {s.get("name") for s in payload.get("shards") or []} fresh_ids = {t.get("node_id") for t in payload.get("tests", [])} @@ -208,6 +224,10 @@ def carry_forward(payload: dict, old_receipt: dict, old_source: str, nm = shard.get("name") if nm in fresh_names: continue # freshly re-run — the new result wins + if shard.get("dirty"): + raise MergeError( + f"--carry-from {old_source}: shard {nm!r} was recorded with a dirty " + f"working tree; it cannot be carried — re-run it.") sfp = shard.get("fingerprint", {}) try: snow = recompute_fingerprint(sfp, root=repo_root) @@ -230,6 +250,7 @@ def carry_forward(payload: dict, old_receipt: dict, old_source: str, grafted["carried"] = { "from": old_source.rsplit("/", 1)[-1], "original_commit_sha": old_sha, + "original_dirty": bool(shard.get("dirty", False)), "original_ended_at": old_receipt.get("session", {}).get("ended_at"), "original_signer": ( (old_receipt.get("signer") or {}).get("github_user") @@ -246,6 +267,7 @@ def carry_forward(payload: dict, old_receipt: dict, old_source: str, fresh_ids.add(nid) carried_count += 1 payload["session"]["node_ids"] = [t["node_id"] for t in payload["tests"]] + payload["repo"]["dirty"] = merged_dirty # carried shards are clean by construction payload["schema_version"] = str(payload.get("schema_version")) if not carried_count: print(f"[gpu-proof] merge: nothing to carry from {old_source} " @@ -253,6 +275,108 @@ def carry_forward(payload: dict, old_receipt: dict, old_source: str, return payload +def verify_input_signatures(receipts: List[dict], sources: List[str]) -> List[str]: + """Authenticate every input shard against GitHub keys BEFORE merging + (opt-in: ``gpu-proof merge --verify-inputs``). Unsigned inputs are refused; + returns the verified signer per input. Uses the verifier's own signature + path so an input shard cannot satisfy a weaker check than a receipt.""" + from .verify import VerificationError, _verify_signature + + signers = [] + for src, r in zip(sources, receipts): + if not r.get("signature"): + raise MergeError(f"--verify-inputs: {src} is unsigned") + try: + username, _fp, _schema = _verify_signature(r, str(r.get("schema_version")), None, {}) + except VerificationError as exc: + raise MergeError(f"--verify-inputs: {src}: {exc}") from exc + signers.append(username) + return signers + + +def receipt_status(path: str) -> dict: + """One-line summary fields for ``gpu-proof status`` (never raises on a + malformed file: it is reported as such).""" + try: + r = load_receipt(path) + except MergeError as exc: + return {"file": os.path.basename(path), "error": str(exc)} + repo = r.get("repo") or {} + tests = r.get("tests") or [] + shards = r.get("shards") or [] + return { + "file": os.path.basename(path), + "schema": r.get("schema_version"), + "commit_sha": repo.get("commit_sha"), + "dirty": bool(repo.get("dirty")) or any(s.get("dirty") for s in shards), + "signed": bool(r.get("signature")), + "outcome": (r.get("session") or {}).get("outcome"), + "ended_at": (r.get("session") or {}).get("ended_at"), + "tests": len(tests), + "failed": sum(1 for t in tests if t.get("outcome") not in ("passed", "skipped")), + "skipped": sum(1 for t in tests if t.get("outcome") == "skipped"), + "shards": [s.get("name") for s in shards], + "carried": sum(1 for s in shards if s.get("carried")), + } + + +def collect_durations(receipts: List[dict], existing: Optional[dict] = None) -> dict: + """Fold per-node ``duration_s`` from receipts into ``existing`` (later + receipts win) — the calibration input for drivers that bin-pack shards.""" + out = dict(existing or {}) + for r in receipts: + for t in r.get("tests") or []: + nid, dur = t.get("node_id"), t.get("duration_s") + if isinstance(nid, str) and isinstance(dur, (int, float)): + out[nid] = float(dur) + return out + + +def plan_carry(old_receipt: dict, repo_root: str = ".") -> dict: + """Dry run of :py:func:`carry_forward`'s soundness conditions against the + CURRENT tree, without grafting: which shards of ``old_receipt`` could be + carried as-is and which must be re-run, with a reason each. Drivers that + plan a partial re-run (which shards to execute fresh) use this instead of + reimplementing the checks.""" + from .fingerprint import FingerprintError, recompute_fingerprint + from .gitutils import get_commit_sha + + report: dict = {"receipt": {}, "shards": []} + schema = old_receipt.get("schema_version") + old_sha = (old_receipt.get("repo") or {}).get("commit_sha") + head = get_commit_sha(repo_root) + receipt_ok, why = True, "ok" + if schema not in ("2", "3") or not old_receipt.get("shards"): + receipt_ok, why = False, "not a sharded schema-2/3 receipt" + elif not old_sha or not head: + receipt_ok, why = False, "missing commit SHA" + elif old_sha != head and not _git_is_ancestor(repo_root, old_sha, head): + receipt_ok, why = False, f"commit {old_sha[:12]} is not an ancestor of HEAD {head[:12]}" + elif (old_receipt.get("repo") or {}).get("dirty"): + receipt_ok, why = False, "recorded with a dirty working tree" + report["receipt"] = {"commit_sha": old_sha, "head": head, "carryable": receipt_ok, "reason": why} + for shard in old_receipt.get("shards") or []: + nm = shard.get("name") + if not receipt_ok: + verdict, reason = "stale", why + elif shard.get("dirty"): + verdict, reason = "stale", "recorded with a dirty working tree" + else: + sfp = shard.get("fingerprint", {}) + try: + now = recompute_fingerprint(sfp, root=repo_root) + except FingerprintError as exc: + verdict, reason = "stale", f"fingerprint invalid: {exc}" + else: + if now["digest"] == sfp.get("digest"): + verdict, reason = "carryable", "fingerprint unchanged" + else: + verdict, reason = "stale", "fingerprint changed" + report["shards"].append({"name": nm, "verdict": verdict, "reason": reason, + "node_count": len(shard.get("node_ids") or [])}) + return report + + def merge_receipts( paths: List[str], out: str, @@ -262,6 +386,7 @@ def merge_receipts( unsigned: bool = False, carry_from: Optional[str] = None, repo_root: str = ".", + verify_inputs: bool = False, ) -> dict: """Merge receipts at ``paths`` and write the re-signed result to ``out``. @@ -273,6 +398,9 @@ def merge_receipts( see :py:func:`carry_forward` for the soundness conditions. """ receipts = [load_receipt(p) for p in paths] + if verify_inputs: + for src, who in zip(paths, verify_input_signatures(receipts, list(paths))): + print(f"[gpu-proof] merge: input {src} signed by @{who}") payload = merge_payloads(receipts, list(paths)) if carry_from: payload = carry_forward(payload, load_receipt(carry_from), carry_from, diff --git a/src/pytest_gpu_proof/plugin.py b/src/pytest_gpu_proof/plugin.py index 7d86222..f89e6b4 100644 --- a/src/pytest_gpu_proof/plugin.py +++ b/src/pytest_gpu_proof/plugin.py @@ -327,6 +327,28 @@ def pytest_addoption(parser): metavar="USERNAME", help="GitHub username of the signer (default: auto-detect from git remote)", ) + group.addoption( + "--gpu-proof-no-gpu-uuid", + action="store_true", + default=False, + help="Omit each GPU's nvidia-smi UUID from the receipt environment " + "(same as record_gpu_uuid = false in [tool.gpu_proof])", + ) + group.addoption( + "--gpu-proof-require-gpu", + action="store_true", + default=False, + help="Record the repository's require_gpu intent (same as require_gpu = true " + "in [tool.gpu_proof]; the verifier reads it from the receipt's repo config)", + ) + group.addoption( + "--gpu-proof-max-age-days", + type=int, + default=None, + metavar="N", + help="Freshness horizon recorded for this run (same as max_age_days in " + "[tool.gpu_proof])", + ) group.addoption( "--gpu-proof-best-effort", action="store_true", diff --git a/src/pytest_gpu_proof/receipt.py b/src/pytest_gpu_proof/receipt.py index 63fd7d0..fbc4012 100644 --- a/src/pytest_gpu_proof/receipt.py +++ b/src/pytest_gpu_proof/receipt.py @@ -28,7 +28,35 @@ def _utcnow() -> str: return datetime.datetime.now(datetime.UTC).strftime("%Y-%m-%dT%H:%M:%SZ") -def _gpu_info() -> Optional[Dict[str, Any]]: +def _gpu_info_from_torch() -> Optional[Dict[str, Any]]: + """nvidia-smi-free fallback: the same torch probe `gpu_required` uses, so a + box that RUNS GPU tests never records `gpu_info: null` (which would then + fail a `require_gpu` policy for the wrong reason).""" + try: + import torch # type: ignore + except ImportError: + return None + try: + if not torch.cuda.is_available(): + return None + devices = [] + for index in range(torch.cuda.device_count()): + props = torch.cuda.get_device_properties(index) + devices.append({ + "index": str(index), + "uuid": None, + "name": props.name, + "driver_version": None, + "memory": f"{props.total_memory // (1024 * 1024)} MiB", + "compute_capability": f"{props.major}.{props.minor}", + }) + except Exception: # a broken CUDA runtime must not sink the receipt + return None + return {"devices": devices, "cuda_visible_devices": os.environ.get("CUDA_VISIBLE_DEVICES"), + "source": "torch"} + + +def _gpu_info(include_uuid: bool = True) -> Optional[Dict[str, Any]]: try: result = subprocess.run( [ @@ -41,16 +69,16 @@ def _gpu_info() -> Optional[Dict[str, Any]]: timeout=10, ) except (FileNotFoundError, subprocess.TimeoutExpired): - return None + return _gpu_info_from_torch() if result.returncode != 0 or not result.stdout.strip(): - return None + return _gpu_info_from_torch() devices = [] for line in result.stdout.splitlines(): parts = [part.strip() for part in line.split(",")] devices.append( { "index": parts[0] if len(parts) > 0 else None, - "uuid": parts[1] if len(parts) > 1 else None, + "uuid": parts[1] if include_uuid and len(parts) > 1 else None, "name": parts[2] if len(parts) > 2 else None, "driver_version": parts[3] if len(parts) > 3 else None, "memory": parts[4] if len(parts) > 4 else None, @@ -60,10 +88,11 @@ def _gpu_info() -> Optional[Dict[str, Any]]: return { "devices": devices, "cuda_visible_devices": os.environ.get("CUDA_VISIBLE_DEVICES"), + "source": "nvidia-smi", } -def _env_info() -> Dict[str, Any]: +def _env_info(record_gpu_uuid: bool = True) -> Dict[str, Any]: import pytest from pytest_gpu_proof import __version__ @@ -73,7 +102,7 @@ def _env_info() -> Dict[str, Any]: "platform": platform.platform(), "pytest_version": pytest.__version__, "plugin_version": __version__, - "gpu_info": _gpu_info(), + "gpu_info": _gpu_info(include_uuid=record_gpu_uuid), } @@ -160,7 +189,7 @@ def build_receipt_payload( "pytest_args": list(config.invocation_args), }, "tests": test_results, - "environment": _env_info(), + "environment": _env_info(getattr(config, "record_gpu_uuid", True)), } if config.shard_name: @@ -184,6 +213,10 @@ def build_receipt_payload( "started_at": started_at, "ended_at": ended_at, "carried": None, + # The recording tree's state travels WITH the shard: a merge ORs + # these into the merged repo.dirty, so a dirty shard can never be + # laundered by being merged or carried next to clean ones. + "dirty": payload["repo"]["dirty"], } ] return payload diff --git a/src/pytest_gpu_proof/verify.py b/src/pytest_gpu_proof/verify.py index c0f4287..a9b1611 100644 --- a/src/pytest_gpu_proof/verify.py +++ b/src/pytest_gpu_proof/verify.py @@ -11,7 +11,7 @@ from typing import Optional from .fingerprint import FingerprintError, recompute_fingerprint -from .gitutils import GitError, get_commit_sha, is_dirty, require_repository +from .gitutils import GitError, get_commit_sha, get_commit_time, is_dirty, require_repository from .signers.base import VerifierError as _VerifierError from .signers.ed25519 import ( _public_key_fingerprint, @@ -380,6 +380,17 @@ def _verify( raise VerificationError( f"receipt mode {receipt.get('mode')!r} does not match required mode {required_mode!r}" ) + # ci-gpu is a claim that a controlled runner recorded the receipt. The only + # thing the verifier can bind that claim to is the signing key, so a ci-gpu + # receipt must be signed by a key the repository policy explicitly lists. + if receipt.get("mode") == "ci-gpu": + allowed_keys = set(policy.get("allowed_key_fingerprints", [])) + if not allowed_keys: + raise VerificationError( + "ci-gpu receipts require a policy allowed_key_fingerprints list " + "naming the runner's signing key") + if signer[1] not in allowed_keys: + raise VerificationError("ci-gpu receipt is not signed by an allowed runner key") stored_fp = receipt["fingerprint"] if ( @@ -430,8 +441,16 @@ def _verify( if not allow_dirty and (repo.get("dirty") or current_dirty): raise VerificationError("repository policy requires clean recording and verification trees") + max_days = ( + max_age_days_override + if max_age_days_override is not None + else policy.get("max_age_days", toml.get("max_age_days", 30)) + ) + if type(max_days) is not int or max_days < 0: + raise VerificationError("max_age_days must be a non-negative integer") + if schema in {"2", "3"} and "shards" in receipt: - _verify_shards(receipt, root, policy) + _verify_shards(receipt, root, policy, max_days) failed = [test["node_id"] for test in tests if test.get("outcome") not in {"passed", "skipped"}] if schema == "3" and session.get("outcome") != "passed": @@ -453,7 +472,17 @@ def _verify( else: expected = None if expected is not None and skipped != expected: - raise VerificationError("recorded skip set does not exactly match the expected baseline") + unexpected = sorted(skipped - expected) + stale = sorted(expected - skipped) + detail = [] + if unexpected: + detail.append(f"{len(unexpected)} unexpected skip(s): {', '.join(unexpected[:5])}" + + (" …" if len(unexpected) > 5 else "")) + if stale: + detail.append(f"{len(stale)} baselined test(s) now run (stale baseline): " + f"{', '.join(stale[:5])}" + (" …" if len(stale) > 5 else "")) + raise VerificationError( + "recorded skip set does not exactly match the expected baseline — " + "; ".join(detail)) if skipped and expected is None and not allow_skipped: raise VerificationError(f"{len(skipped)} marked test(s) were skipped") @@ -468,24 +497,27 @@ def _verify( if require_gpu and not (receipt.get("environment") or {}).get("gpu_info"): raise VerificationError("repository policy requires recorded GPU information") - max_days = ( - max_age_days_override - if max_age_days_override is not None - else policy.get("max_age_days", toml.get("max_age_days", 30)) - ) - if type(max_days) is not int or max_days < 0: - raise VerificationError("max_age_days must be a non-negative integer") ended = _parse_time(session.get("ended_at"), "session.ended_at") age = datetime.datetime.now(datetime.UTC) - ended if age < datetime.timedelta(minutes=-5): raise VerificationError("receipt timestamp is in the future") if age > datetime.timedelta(days=max_days): raise VerificationError(f"receipt is older than the {max_days}-day policy") + # A session cannot have started before the commit it attests to existed + # (5 min of clock skew allowed). Cheap replay/forgery guard without a nonce. + committed = get_commit_time(stored_sha, root) + if committed is not None: + started = _parse_time(session.get("started_at"), "session.started_at") + commit_dt = datetime.datetime.fromtimestamp(committed, datetime.UTC) + if started < commit_dt - datetime.timedelta(minutes=5): + raise VerificationError( + "receipt session started before its commit existed " + f"({session.get('started_at')} < commit time {commit_dt:%Y-%m-%dT%H:%M:%SZ})") print(f"[gpu-proof] All {len(tests) - len(skipped)} executed test(s) passed") print("[gpu-proof] Receipt verified successfully.") -def _verify_shards(receipt: dict, root: str, policy: dict) -> None: +def _verify_shards(receipt: dict, root: str, policy: dict, max_days: int = 30) -> None: shards = receipt.get("shards") if not isinstance(shards, list) or not shards: raise VerificationError("sharded receipt has no shards") @@ -529,15 +561,26 @@ def _verify_shards(receipt: dict, root: str, policy: dict) -> None: raise VerificationError(f"shard {name!r}: {exc}") from exc if current["digest"] != fingerprint.get("digest"): raise VerificationError(f"shard {name!r} fingerprint does not match") + if shard.get("dirty") and not policy.get("allow_dirty", False): + raise VerificationError( + f"shard {name!r} was recorded with a dirty working tree") carried = shard.get("carried") if carried: if not policy.get("allow_carried", False): raise VerificationError(f"shard {name!r} is carried but policy rejects carry-forward") if not isinstance(carried, dict): raise VerificationError(f"shard {name!r} carried metadata is invalid") + # A shard carried from a dirty recording (merge >= 0.4.1 refuses + # to create one; an older or hand-edited receipt could still say + # so) is dirty evidence; the merged repo.dirty does not cover it. + if carried.get("original_dirty") and not policy.get("allow_dirty", False): + raise VerificationError( + f"carried shard {name!r} was recorded with a dirty working tree") original = _parse_time(carried.get("original_ended_at"), f"shard {name}.original_ended_at") age = datetime.datetime.now(datetime.UTC) - original - limit = policy.get("carried_max_age_days", 30) + # A carried shard's original run is never allowed to be older than + # the receipt itself may be, unless policy says so explicitly. + limit = policy.get("carried_max_age_days", min(30, max_days)) if type(limit) is not int or limit < 0: raise VerificationError("carried_max_age_days must be a non-negative integer") if age < datetime.timedelta(minutes=-5) or age > datetime.timedelta(days=limit): diff --git a/tests/test_config.py b/tests/test_config.py index a0a3579..b17f9b6 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -21,6 +21,7 @@ def test_dataclass_defaults_are_safe(): first.fingerprint_paths.append("x") assert second.fingerprint_paths == ["."] assert second.best_effort is False + assert second.record_gpu_uuid is True def test_load_toml_defaults_absent_invalid_and_non_table(tmp_path): @@ -104,3 +105,19 @@ def test_explicit_empty_fingerprint_exclusions(tmp_path): "[tool.gpu_proof]\nfingerprint_excluded_paths = []\n" ) assert load_config(FakeConfig(tmp_path)).fingerprint_excluded_paths == [] + + +def test_record_gpu_uuid_from_toml(tmp_path): + (tmp_path / "pyproject.toml").write_text("[tool.gpu_proof]\nrecord_gpu_uuid = false\n") + config = load_config( + FakeConfig( + tmp_path, + { + "--gpu-proof-fingerprint-paths": None, + "--gpu-proof-shard-fingerprint-paths": None, + "--gpu-proof-fail-on-skip": False, + "--gpu-proof-best-effort": False, + }, + ) + ) + assert config.record_gpu_uuid is False diff --git a/tests/test_merge.py b/tests/test_merge.py index 77f8d56..31ba751 100644 --- a/tests/test_merge.py +++ b/tests/test_merge.py @@ -66,9 +66,9 @@ def _fake_verify(data, signature, username): def test_happy_merge_verifies(tmp_path, tmp_git_repo, signer_with_key): signer, public_key, key_path = signer_with_key a = _shard(tmp_path, tmp_git_repo, signer, "a.json", - [_result("tests/test_add.py::test_add")], started_ago=10, ended_ago=8) + [_result("tests/test_add.py::test_add")], started_ago=4, ended_ago=3) b = _shard(tmp_path, tmp_git_repo, signer, "b.json", - [_result("tests/test_add.py::test_sub")], started_ago=5, ended_ago=3) + [_result("tests/test_add.py::test_sub")], started_ago=2, ended_ago=1) out = tmp_path / "merged.json" merged = merge_receipts([str(a), str(b)], str(out), key_path=key_path) diff --git a/tests/test_receipt.py b/tests/test_receipt.py index 1864787..0a1f89a 100644 --- a/tests/test_receipt.py +++ b/tests/test_receipt.py @@ -93,6 +93,9 @@ def test_environment_and_gpu_capture(monkeypatch): assert info["cuda_visible_devices"] == "1" assert _env_info()["gpu_info"] == info assert _utcnow().endswith("Z") + anonymous = _env_info(record_gpu_uuid=False)["gpu_info"] + assert all(d["uuid"] is None for d in anonymous["devices"]) + assert [d["name"] for d in anonymous["devices"]] == [d["name"] for d in info["devices"]] @pytest.mark.parametrize( @@ -298,3 +301,41 @@ def test_signer_resolution_failure(monkeypatch, mock_config): monkeypatch.setattr("pytest_gpu_proof.receipt.get_github_username", lambda root: None) with pytest.raises(ValueError, match="cannot determine"): _resolve_github_username(mock_config, None, ".") + + +def test_gpu_info_falls_back_to_torch_when_nvidia_smi_is_absent(monkeypatch): + """A3 (2026-10-12): the environment block uses the same torch probe as the + gpu_required marker, so a torch-only box never records gpu_info: null.""" + import sys + import types + from pytest_gpu_proof import receipt as receipt_mod + + def no_smi(*args, **kwargs): + raise FileNotFoundError("nvidia-smi") + monkeypatch.setattr(receipt_mod.subprocess, "run", no_smi) + props = types.SimpleNamespace(name="Fake GPU", total_memory=8 * 1024 ** 3, major=12, minor=0) + fake_cuda = types.SimpleNamespace(is_available=lambda: True, device_count=lambda: 1, + get_device_properties=lambda i: props) + monkeypatch.setitem(sys.modules, "torch", types.SimpleNamespace(cuda=fake_cuda)) + info = receipt_mod._gpu_info() + assert info["source"] == "torch" + assert info["devices"] == [{"index": "0", "uuid": None, "name": "Fake GPU", "driver_version": None, + "memory": "8192 MiB", "compute_capability": "12.0"}] + # no CUDA -> None; a broken runtime -> None; no torch -> None + fake_cuda.is_available = lambda: False + assert receipt_mod._gpu_info() is None + fake_cuda.is_available = lambda: True + fake_cuda.device_count = lambda: (_ for _ in ()).throw(RuntimeError("boom")) + assert receipt_mod._gpu_info() is None + monkeypatch.setitem(sys.modules, "torch", None) + assert receipt_mod._gpu_info() is None + + +def test_gpu_info_falls_back_to_torch_on_nvidia_smi_failure(monkeypatch): + import sys + import types + from pytest_gpu_proof import receipt as receipt_mod + monkeypatch.setattr(receipt_mod.subprocess, "run", + lambda *a, **k: types.SimpleNamespace(returncode=1, stdout="")) + monkeypatch.setitem(sys.modules, "torch", None) + assert receipt_mod._gpu_info() is None diff --git a/tests/test_sharding.py b/tests/test_sharding.py index 9a592fa..4e0cb69 100644 --- a/tests/test_sharding.py +++ b/tests/test_sharding.py @@ -286,6 +286,12 @@ def test_schema1_with_shards_block_rejected(tmp_path, tmp_git_repo, signer_with_ def test_carry_preserves_and_checks_extra_paths(tmp_path, tmp_git_repo, signer_with_key): signer, _, _ = signer_with_key + # Generated inputs live outside version control (gitignored): the recording + # must still be CLEAN, or carry-forward refuses it (dirty receipts attest + # to unknown bytes). Explicit extra paths are hashed regardless of tracking. + (tmp_git_repo / ".gitignore").write_text("generated.bin\n") + _git(tmp_git_repo, "add", ".gitignore") + _git(tmp_git_repo, "commit", "-m", "ignore generated input") generated = tmp_git_repo / "generated.bin" generated.write_bytes(b"stable") old = _shard_receipt( @@ -293,6 +299,7 @@ def test_carry_preserves_and_checks_extra_paths(tmp_path, tmp_git_repo, signer_w [_result("t::extra")], shard_extra_paths=["generated.bin"], ) old_r = json.loads(old.read_text()) + assert old_r["repo"]["dirty"] is False fresh = _shard_receipt( tmp_path, tmp_git_repo, signer, "fresh.json", "fresh", ["src"], [_result("t::fresh")], @@ -466,3 +473,225 @@ def test_direct_shard_validation_rejects_bad_carried_limit( str(tmp_git_repo), {"allow_carried": True, "carried_max_age_days": "bad"}, ) + + +def test_carry_refuses_dirty_source_receipt(tmp_path, tmp_git_repo, signer_with_key): + """A receipt recorded with a dirty tree attests to unknown bytes (edits may + lie outside a shard's narrow fingerprint) and the merged repo.dirty is OR-ed + over FRESH inputs only — carrying from it would launder the dirty flag.""" + signer, _, _ = signer_with_key + old, fresh = _carry_setup(tmp_path, tmp_git_repo, signer) + old_r = json.loads(old.read_text()) + old_r["repo"]["dirty"] = True + payload = merge_payloads([json.loads(fresh.read_text())], ["fresh.json"]) + with pytest.raises(MergeError, match="dirty working tree"): + carry_forward(payload, old_r, "old", str(tmp_git_repo)) + # a clean source records original_dirty=False on every grafted shard + old_r["repo"]["dirty"] = False + merged = carry_forward( + merge_payloads([json.loads(fresh.read_text())], ["fresh.json"]), + old_r, "old", str(tmp_git_repo)) + grafted = [s for s in merged["shards"] if s.get("carried")] + assert grafted and all(s["carried"]["original_dirty"] is False for s in grafted) + + +def test_verify_refuses_carried_shard_recorded_dirty(tmp_path, tmp_git_repo, signer_with_key): + """Defense in depth: a merged receipt whose carried shard says + original_dirty (an older merger, or a hand-edited receipt) is refused + unless the policy allows dirty evidence.""" + signer, public_key, key_path = signer_with_key + old, fresh = _carry_setup(tmp_path, tmp_git_repo, signer) + out = tmp_path / "merged.json" + merge_receipts([str(fresh)], str(out), key_path=key_path, + carry_from=str(old), repo_root=str(tmp_git_repo)) + merged = json.loads(out.read_text()) + for shard in merged["shards"]: + if shard.get("carried"): + shard["carried"]["original_dirty"] = True + merged.pop("signature", None) + merged.pop("signer", None) + write_receipt(finalize_receipt(merged, signer), str(out)) + os.chdir(tmp_git_repo) + with _mock_github_keys(public_key): + with pytest.raises(VerificationError, match="recorded with a dirty working tree"): + _verify(str(out), _policy(tmp_path, allow_carried=True), + str(tmp_git_repo), "testuser", None) + _verify(str(out), _policy(tmp_path, allow_carried=True, allow_dirty=True), + str(tmp_git_repo), "testuser", None) + + +def test_shard_records_its_own_dirty_flag_and_merge_ors_it(tmp_path, tmp_git_repo, signer_with_key): + """A1 (2026-10-12): the recording tree's dirty state travels with the shard; + merging a dirty shard next to clean ones makes the merged receipt dirty + even when the merger only looks at shard entries.""" + signer, _, _ = signer_with_key + clean = _shard_receipt(tmp_path, tmp_git_repo, signer, "clean.json", "clean", ["src"], [_result("t::c")]) + assert json.loads(clean.read_text())["shards"][0]["dirty"] is False + (tmp_git_repo / "src" / "scratch.py").write_text("x = 1\n") # untracked -> dirty + dirty = _shard_receipt(tmp_path, tmp_git_repo, signer, "dirty.json", "dirty", ["src"], [_result("t::d")]) + (tmp_git_repo / "src" / "scratch.py").unlink() + d = json.loads(dirty.read_text()) + assert d["repo"]["dirty"] is True and d["shards"][0]["dirty"] is True + # a hand-cleaned top-level flag cannot hide a dirty shard + d["repo"]["dirty"] = False + merged = merge_payloads([json.loads(clean.read_text()), d], ["clean.json", "dirty.json"]) + assert merged["repo"]["dirty"] is True + + +def test_carry_refuses_a_dirty_shard_inside_a_clean_receipt(tmp_path, tmp_git_repo, signer_with_key): + signer, _, _ = signer_with_key + old, fresh = _carry_setup(tmp_path, tmp_git_repo, signer) + old_r = json.loads(old.read_text()) + for shard in old_r["shards"]: + shard["dirty"] = True + payload = merge_payloads([json.loads(fresh.read_text())], ["fresh.json"]) + with pytest.raises(MergeError, match="shard 'modB' was recorded with a dirty"): + carry_forward(payload, old_r, "old", str(tmp_git_repo)) + + +def test_verify_refuses_dirty_shard_unless_policy_allows(tmp_path, tmp_git_repo, signer_with_key): + signer, public_key, key_path = signer_with_key + a = _shard_receipt(tmp_path, tmp_git_repo, signer, "a.json", "modA", ["src"], [_result("t::a")]) + out = tmp_path / "merged.json" + merge_receipts([str(a)], str(out), key_path=key_path, repo_root=str(tmp_git_repo)) + merged = json.loads(out.read_text()) + merged["shards"][0]["dirty"] = True + merged.pop("signature", None); merged.pop("signer", None) + write_receipt(finalize_receipt(merged, signer), str(out)) + os.chdir(tmp_git_repo) + with _mock_github_keys(public_key): + with pytest.raises(VerificationError, match="shard 'modA' was recorded with a dirty"): + _verify(str(out), _policy(tmp_path), str(tmp_git_repo), "testuser", None) + _verify(str(out), _policy(tmp_path, allow_dirty=True), str(tmp_git_repo), "testuser", None) + + +def test_plan_carry_reports_each_shard(tmp_path, tmp_git_repo, signer_with_key, capsys): + """B2 (2026-10-12): `gpu-proof plan` = carry_forward's conditions as a report.""" + from pytest_gpu_proof.cli import main as cli_main + from pytest_gpu_proof.merge import plan_carry + signer, _, _ = signer_with_key + old, _fresh = _carry_setup(tmp_path, tmp_git_repo, signer, drift_b=True) + old_r = json.loads(old.read_text()) + report = plan_carry(old_r, str(tmp_git_repo)) + assert report["receipt"]["carryable"] is True + verdicts = {s["name"]: s["verdict"] for s in report["shards"]} + assert verdicts == {"modA": "carryable", "modB": "stale"} + # dirty shard, dirty receipt, non-ancestor, bad fingerprint, monolithic + old_r["shards"][0]["dirty"] = True + assert plan_carry(old_r, str(tmp_git_repo))["shards"][0]["reason"] == "recorded with a dirty working tree" + old_r["shards"][0]["dirty"] = False + old_r["shards"][0]["fingerprint"] = {"algorithm": "bogus", "included_paths": ["src"]} + assert "fingerprint invalid" in plan_carry(old_r, str(tmp_git_repo))["shards"][0]["reason"] + old_r["repo"]["dirty"] = True + rep = plan_carry(old_r, str(tmp_git_repo)) + assert rep["receipt"]["carryable"] is False and all(s["verdict"] == "stale" for s in rep["shards"]) + old_r["repo"]["dirty"] = False + old_r["repo"]["commit_sha"] = "0" * 40 + assert "not an ancestor" in plan_carry(old_r, str(tmp_git_repo))["receipt"]["reason"] + old_r["repo"]["commit_sha"] = None + assert plan_carry(old_r, str(tmp_git_repo))["receipt"]["reason"] == "missing commit SHA" + assert plan_carry({"schema_version": "1", "tests": []}, str(tmp_git_repo))["receipt"]["reason"].startswith("not a sharded") + # CLI: text and json forms, unreadable receipt + import sys as _sys + argv = _sys.argv + try: + _sys.argv = ["gpu-proof", "plan", "--receipt", str(old), "--repo", str(tmp_git_repo)] + with pytest.raises(SystemExit) as e: + cli_main() + assert e.value.code == 0 + out = capsys.readouterr().out + assert "carryable modA" in out and "stale modB" in out and "1/2 shard(s) carryable" in out + _sys.argv = ["gpu-proof", "plan", "--receipt", str(old), "--repo", str(tmp_git_repo), "--json"] + with pytest.raises(SystemExit): + cli_main() + assert json.loads(capsys.readouterr().out)["shards"][1]["verdict"] == "stale" + _sys.argv = ["gpu-proof", "plan", "--receipt", str(tmp_path / "missing.json")] + with pytest.raises(SystemExit) as e: + cli_main() + assert e.value.code == 1 + finally: + _sys.argv = argv + + +def test_carried_age_is_clamped_to_the_receipt_age_policy(tmp_path, tmp_git_repo, signer_with_key): + """A5 (2026-10-12): without an explicit carried_max_age_days, a carried + shard may not be older than max_age_days (previously a fixed 30).""" + signer, public_key, key_path = signer_with_key + old, fresh = _carry_setup(tmp_path, tmp_git_repo, signer, old_days_ago=10) + out = tmp_path / "merged.json" + merge_receipts([str(fresh)], str(out), key_path=key_path, + carry_from=str(old), repo_root=str(tmp_git_repo)) + os.chdir(tmp_git_repo) + with _mock_github_keys(public_key): + with pytest.raises(VerificationError, match="outside its age policy"): + _verify(str(out), _policy(tmp_path, allow_carried=True, max_age_days=5), + str(tmp_git_repo), "testuser", None) + _verify(str(out), _policy(tmp_path, allow_carried=True, max_age_days=5, + carried_max_age_days=20), str(tmp_git_repo), "testuser", None) + _verify(str(out), _policy(tmp_path, allow_carried=True, max_age_days=15), + str(tmp_git_repo), "testuser", None) + + +def test_merge_verify_inputs_authenticates_every_shard(tmp_path, tmp_git_repo, signer_with_key, capsys): + """A2 (2026-10-12): --verify-inputs refuses unsigned or unverifiable inputs + and names the signer of each verified one.""" + from pytest_gpu_proof.merge import verify_input_signatures + signer, public_key, key_path = signer_with_key + a = _shard_receipt(tmp_path, tmp_git_repo, signer, "a.json", "modA", ["src"], [_result("t::a")]) + b = _shard_receipt(tmp_path, tmp_git_repo, signer, "b.json", "modB", ["tests"], [_result("t::b")]) + out = tmp_path / "merged.json" + with _mock_github_keys(public_key): + merge_receipts([str(a), str(b)], str(out), key_path=key_path, + repo_root=str(tmp_git_repo), verify_inputs=True) + assert capsys.readouterr().out.count("signed by @testuser") == 2 + bad = json.loads(b.read_text()) + bad["tests"][0]["duration_s"] = 9.0 # tamper -> signature no longer matches + with pytest.raises(MergeError, match="verify-inputs: b.json: signature does not match"): + verify_input_signatures([json.loads(a.read_text()), bad], ["a.json", "b.json"]) + bad["signature"] = None + with pytest.raises(MergeError, match="b.json is unsigned"): + verify_input_signatures([bad], ["b.json"]) + + +def test_status_and_durations_subcommands(tmp_path, tmp_git_repo, signer_with_key, capsys): + """B3/B5 (2026-10-12): status summarizes receipt files; durations folds + per-test seconds into a JSON map (later receipts win).""" + from pytest_gpu_proof.cli import main as cli_main + from pytest_gpu_proof.merge import collect_durations, receipt_status + signer, _, _ = signer_with_key + rdir = tmp_path / "receipts"; rdir.mkdir() + a = _shard_receipt(rdir, tmp_git_repo, signer, "a.json", "modA", ["src"], [_result("t::a")]) + (rdir / "junk.json").write_text("{}") + st = receipt_status(str(a)) + assert st["shards"] == ["modA"] and st["tests"] == 1 and st["signed"] and not st["dirty"] + assert "error" in receipt_status(str(rdir / "junk.json")) + assert collect_durations([{"tests": [{"node_id": "x", "duration_s": 1}, {"node_id": "bad"}]}], + {"x": 0.5, "y": 2.0}) == {"x": 1.0, "y": 2.0} + import sys as _sys + argv = _sys.argv + try: + _sys.argv = ["gpu-proof", "status", str(rdir)] + with pytest.raises(SystemExit) as e: + cli_main() + out = capsys.readouterr().out + assert e.value.code == 0 and "a.json: schema 3 @" in out and "junk.json: " in out + _sys.argv = ["gpu-proof", "status", str(a), "--json"] + with pytest.raises(SystemExit): + cli_main() + assert json.loads(capsys.readouterr().out)[0]["file"] == "a.json" + dmap = tmp_path / "durations.json" + dmap.write_text("not json") + _sys.argv = ["gpu-proof", "durations", str(a), "--out", str(dmap)] + with pytest.raises(SystemExit) as e: + cli_main() + assert e.value.code == 0 and json.loads(dmap.read_text()) == {"t::a": 0.01} + dmap.write_text("[]") + with pytest.raises(SystemExit): + cli_main() + assert json.loads(dmap.read_text()) == {"t::a": 0.01} + _sys.argv = ["gpu-proof", "durations", str(tmp_path / "missing.json"), "--out", str(dmap)] + with pytest.raises(SystemExit) as e: + cli_main() + assert e.value.code == 1 + finally: + _sys.argv = argv diff --git a/tests/test_verifier.py b/tests/test_verifier.py index 6d9614c..b5a519e 100644 --- a/tests/test_verifier.py +++ b/tests/test_verifier.py @@ -863,3 +863,94 @@ def test_dirty_query_error_is_rejected( monkeypatch.setattr("pytest_gpu_proof.verify.is_dirty", lambda *a, **k: (_ for _ in ()).throw(GitError("status broke"))) with _mock_github_keys(public_key), pytest.raises(VerificationError, match="status broke"): _verify(str(path), None, str(tmp_git_repo), "testuser", None) + + +def test_expected_skips_failure_names_the_delta(tmp_path, tmp_git_repo, signer_with_key): + """B4 (2026-10-12): the mismatch message lists unexpected skips and stale + baseline entries instead of only saying the sets differ.""" + signer, public_key = signer_with_key + path = _make_receipt(tmp_path, tmp_git_repo, signer, results=_mixed_results()) + baseline = tmp_path / "expected_skips.txt" + baseline.write_text("tests/test_add.py::test_never_skipped\n") + with _mock_github_keys(public_key): + with pytest.raises(VerificationError) as info: + _verify(str(path), None, str(tmp_git_repo), "testuser", None, + expected_skips_path=str(baseline)) + msg = str(info.value) + assert "1 unexpected skip(s): tests/test_add.py::test_skipped" in msg + assert "stale baseline" in msg and "test_never_skipped" in msg + # unexpected only (empty baseline): no stale clause + baseline.write_text("# nothing expected\n") + with _mock_github_keys(public_key): + with pytest.raises(VerificationError) as info: + _verify(str(path), None, str(tmp_git_repo), "testuser", None, + expected_skips_path=str(baseline)) + assert "unexpected skip(s)" in str(info.value) and "stale" not in str(info.value) + + +def test_verify_refuses_session_that_predates_its_commit(tmp_path, tmp_git_repo, signer_with_key, monkeypatch): + """A4 (2026-10-12): a session cannot start before the commit it attests to + existed (5 min skew allowed); git failing to date the commit is not an error.""" + signer, public_key = signer_with_key + + def backdate(payload): + payload["session"]["started_at"] = _utcstamp(days_ago=2) + payload["session"]["ended_at"] = _utcstamp(days_ago=2) + path = _make_receipt(tmp_path, tmp_git_repo, signer, mutate=backdate) + with _mock_github_keys(public_key): + with pytest.raises(VerificationError, match="started before its commit existed"): + _verify(str(path), None, str(tmp_git_repo), "testuser", None) + from pytest_gpu_proof import verify as verify_mod + monkeypatch.setattr(verify_mod, "get_commit_time", lambda sha, root: None) + _verify(str(path), None, str(tmp_git_repo), "testuser", None) + + +def test_get_commit_time_handles_bad_output(tmp_git_repo, monkeypatch): + from pytest_gpu_proof import gitutils + assert isinstance(gitutils.get_commit_time("HEAD", str(tmp_git_repo)), int) + assert gitutils.get_commit_time("0" * 40, str(tmp_git_repo)) is None + monkeypatch.setattr(gitutils, "_git", lambda *a, **k: "not-a-number") + assert gitutils.get_commit_time("HEAD", str(tmp_git_repo)) is None + + +def test_ci_gpu_receipts_require_an_allowed_runner_key(tmp_path, tmp_git_repo, signer_with_key): + """C1 (2026-10-12): mode ci-gpu is bound to the signing key — the policy + must list the runner's key fingerprint and the receipt must be signed by it.""" + signer, public_key = signer_with_key + + def ci_mode(payload): + payload["mode"] = "ci-gpu" + path = _make_receipt(tmp_path, tmp_git_repo, signer, mutate=ci_mode) + policy = tmp_path / "policy.json" + with _mock_github_keys(public_key): + policy.write_text(json.dumps({"require_mode": "ci-gpu"})) + with pytest.raises(VerificationError, match="require a policy allowed_key_fingerprints"): + _verify(str(path), str(policy), str(tmp_git_repo), "testuser", None) + policy.write_text(json.dumps({"signer_mode": "restricted", + "allowed_key_fingerprints": ["SHA256:someone-else"]})) + with pytest.raises(VerificationError, match="not allowed by repository policy"): + _verify(str(path), str(policy), str(tmp_git_repo), "testuser", None) + policy.write_text(json.dumps({"allowed_key_fingerprints": [signer.key_fingerprint()]})) + _verify(str(path), str(policy), str(tmp_git_repo), "testuser", None) + # open mode with the right key listed but the receipt signed by another key + from pytest_gpu_proof import verify as verify_mod + other = tmp_path / "other" + other.write_text(json.dumps({"allowed_key_fingerprints": ["SHA256:x"]})) + with pytest.raises(VerificationError, match="not signed by an allowed runner key"): + _verify(str(path), str(other), str(tmp_git_repo), "testuser", None) + + +def test_load_config_cli_flags_for_toml_only_options(tmp_path): + """C3 (2026-10-12): record_gpu_uuid / require_gpu / max_age_days are reachable + from the command line, with the same precedence as the other options.""" + from pytest_gpu_proof.config import load_config + from tests.test_config import FakeConfig + (tmp_path / "pyproject.toml").write_text( + "[tool.gpu_proof]\nmax_age_days = 7\nrecord_gpu_uuid = true\nrequire_gpu = false\n") + cfg = load_config(FakeConfig(tmp_path, {"--gpu-proof-enable": True, + "--gpu-proof-no-gpu-uuid": True, + "--gpu-proof-require-gpu": True, + "--gpu-proof-max-age-days": 3})) + assert cfg.record_gpu_uuid is False and cfg.require_gpu is True and cfg.max_age_days == 3 + cfg = load_config(FakeConfig(tmp_path, {"--gpu-proof-enable": True})) + assert cfg.record_gpu_uuid is True and cfg.require_gpu is False and cfg.max_age_days == 7