Skip to content

feat: publish deterministic artifact diff report - #92

Merged
stacknil merged 4 commits into
mainfrom
stacknil/artifact-diff-json-report
Aug 24, 2026
Merged

feat: publish deterministic artifact diff report#92
stacknil merged 4 commits into
mainfrom
stacknil/artifact-diff-json-report

Conversation

@stacknil

@stacknil stacknil commented Aug 23, 2026

Copy link
Copy Markdown
Owner

Summary

  • add deterministic artifact-contract-diff/v1 serialization for the shared report model
  • add --json-out with output-root isolation and atomic replacement
  • publish a strict Draft 2020-12 schema with status/reason/snapshot cross-field constraints
  • document report semantics, privacy boundaries, and rollback behavior

This is layer 3 of the split replacement for #89 and is stacked on #91.

Design decision

Keep compare_artifact_trees() as the sole semantic core. Human output and JSON are projections of the same immutable report. JSON output contains stable relative identities and bounded summaries only—no roots, bodies, timestamps, or ambient metadata. The destination resolves outside both input trees and is replaced atomically only after a complete synchronized write.

Main risk

A destination-path validation or write failure must never modify an input artifact or destroy a previous report. Root-containment, symlink-root, comparison-failure preservation, simulated os.replace failure, temporary-file cleanup, and deterministic-byte tests cover this boundary. The strict schema may also reject hand-authored payloads that contradict the core invariants; that is intentional.

Compatibility impact

Additive optional CLI flag and new artifact-contract-diff/v1 schema. Existing human CLI behavior, regeneration authority, committed artifacts, evidence schemas, and release labels remain unchanged. Consumers that adopt the JSON report should validate against the versioned schema.

Rollback path

Revert be7c323, f775814, 9b38243, then 77872f6. #91 remains an independently usable human comparator with structured reason classification and no machine-readable output contract.

Validation

  • .venv\Scripts\python.exe -m pytest -q tests/test_artifact_contract_diff_core.py tests/test_artifact_contract_diff_structured.py tests/test_artifact_contract_diff_json_report.py tests/test_artifact_regeneration_check.py — 37 passed, 1 skipped
  • .venv\Scripts\python.exe -m pytest -q tests/test_artifact_contract_diff_json_report.py tests/test_markdown_links.py tests/test_reviewer_docs.py — 31 passed
  • Draft 2020-12 schema self-check — passed
  • .venv\Scripts\python.exe scripts/check_release_contract.py — 29 strict artifacts matched, 6 visual artifacts regenerated, 8 existing schema tests passed, full suite 249 passed / 1 skipped
  • real data/processed JSON report generated twice — identical SHA-256, schema-valid summary (14 total / 8 comparable / 6 presence-only), no local path, smoke report removed
  • git diff --check — passed
  • privacy scan of added lines — no local usernames, OneDrive/absolute machine paths, credentials, or school identifiers

Review state

Draft for the required schema compatibility and delayed review. Relative to #91 this PR changes 6 files and contains 382 substantive behavior/schema lines (125 Python + 257 schema), below the 500-line split gate.

Closes #85.

@stacknil

Copy link
Copy Markdown
Owner Author

Post-CI diff review completed against be7c323 (remote head matches the reviewed local head).

  • Design decision: compare_artifact_trees() remains the only semantic core; JSON is a deterministic projection with a versioned Draft 2020-12 schema and no roots, bodies, timestamps, or ambient metadata.
  • Main risk: output-path or atomic-write failures must not mutate either input tree or destroy a previous report. Containment, symlink-root, comparison-failure, simulated replace-failure, and cleanup tests cover this boundary.
  • Compatibility impact: additive optional --json-out plus artifact-contract-diff/v1. Existing artifacts, evidence schemas, regeneration authority, human output, and release labels remain unchanged.
  • Rollback path: revert be7c323, f775814, 9b38243, then 77872f6; feat: classify structured artifact changes #91 remains independently usable without machine-readable output.

Reviewed relative scope is 6 files and 382 substantive behavior/schema lines (125 Python + 257 schema). All four remote checks are green; no bot or reviewer comments are outstanding.

External clean-clone reproduction also passed at this exact head: fresh venv install, 29 strict artifacts, 6 visual snapshots, 8 existing schema tests, full suite 249 passed / 1 skipped, and a schema-valid real report with SHA-256 5BD27339A4F7A3884C315CC09A9B3B7E10EEF51B54B44A1A30F884D53BEC2B80. The clone and smoke report were removed after verification.

Keeping the PR draft for delayed schema compatibility review.

@stacknil
stacknil changed the base branch from stacknil/artifact-diff-structured-summary to main August 24, 2026 09:57
@stacknil

Copy link
Copy Markdown
Owner Author

Delayed JSON/schema compatibility review completed on 2026-08-24 against exact head be7c323, after retargeting this PR from the merged #91 branch to main. The relative scope remains the intended six-file projection layer.

The remaining prototype review findings are covered here: output paths resolving inside either input tree fail closed before comparison; report writes are same-directory temporary files plus fsync and atomic replace; comparison/write failures preserve an existing report; recognized digest/path metadata is validated before projection; constructor and Draft 2020-12 schema reject representable status/reason/snapshot contradictions; report payloads are detached; README examples are shell-safe; field semantics and resource limits are documented.

Fresh validation:

  • .\.venv\Scripts\python.exe -m pytest -q tests/test_artifact_contract_diff_json_report.py -> 9 passed (independent schema compatibility matrix).

  • Full artifact-diff targeted matrix -> 37 passed, 1 skipped.

  • .\.venv\Scripts\python.exe scripts\check_release_contract.py -> 29 strict matches, 6 visual smoke artifacts, 8 existing schema tests, and 249 passed / 1 skipped overall.

  • Two real data/processed reports were byte-identical with SHA-256 5BD27339A4F7A3884C315CC09A9B3B7E10EEF51B54B44A1A30F884D53BEC2B80; each was schema-valid with 14 total / 8 comparable / 6 presence-only and no checkout path, timestamp, body, or forbidden key.

  • git diff --check origin/main...be7c323 and changed-line privacy/secret scan passed.

  • All four remote checks remain green; no external bot/reviewer feedback is outstanding.

  • Design decision: JSON remains a deterministic projection of the same invariant-checked comparison core; it has no independent matching, acceptance, or update policy.

  • Main risk: JSON Schema cannot express every dynamic summary-count equation, so arithmetic reconciliation remains authoritative in the sole core constructor. Output-path validation also retains the documented local-filesystem TOCTOU boundary.

  • Compatibility impact: additive optional --json-out and artifact-contract-diff/v1; existing human output, committed artifacts, evidence schemas, runtime dependencies, and release labels remain unchanged.

  • Rollback path: revert be7c323, f775814, 9b38243, then 77872f6; merged feat: classify structured artifact changes #91 remains independently usable without machine-readable output.

The overnight schema/security review and independent compatibility requirements are satisfied. Marking the final layer ready for merge; no release or tag is part of this change.

@stacknil
stacknil marked this pull request as ready for review August 24, 2026 10:02
@stacknil
stacknil merged commit 8393757 into main Aug 24, 2026
4 checks passed
@stacknil
stacknil deleted the stacknil/artifact-diff-json-report branch August 24, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add structured artifact-contract diff reporting

1 participant