[ChristianPavilonis]

Summary

• Scaffold hardened Axum application Dockerfiles and .dockerignore files using the application's committed lockfile and declared Rust toolchain, a paired Rust builder version/digest, a digest-pinned Dockerfile frontend, BuildKit dependency caches, CA trust, numeric non-root identity and explicit container binding.
• Include Axum: validate multi-platform containers and provide an application publishing workflow #399 preparation: native amd64/arm64 container CI, verified OCI artifact handoff, and an inactive, copyable application-owned GHCR publishing recipe that preserves tested image bytes.
• Include Axum: document Docker, Kubernetes, and managed-cloud deployments #400 preparation: Docker/Compose guidance and packaging smoke, plus internal Kubernetes and AWS deployment designs. Runnable Kubernetes manifests and authorized cloud execution are not delivered here.
• Keep framework packaging scans report-only and application-release verification strict. No public application image, external registry publication, production deployment, runtime API change or vulnerability waiver is included.

Scope and issue relationships

Part of #391. Related to #398, #399, #400 and #410, with provider implementation tracked separately in #414. All four issues remain open for their outstanding acceptance. This PR delivers packaging preparation and the related CI/documentation preparation; it does not redefine the issues' completion criteria.

The current combined PR scope and custom OCI verifier are deliberately retained after review. The publishing recipe also retains its combined publication/attestation job. Its whole-job OIDC signing-authority risk is explicitly accepted and documented; credentials and OIDC are not available to the native build jobs.

Issue	Delivered preparation	Acceptance still outstanding
#398	Hardened Docker scaffolding, portable-consumer instructions and constrained packaging checks	Accepted-runtime required-config failure, durable restart/recovery, readiness and bounded SIGTERM/draining
#399	Native architecture CI, tested-artifact handoff/index checks and application-owned GHCR recipe	Authorized protected GHCR publication/attestation verification and accepted-runtime final-image cases
#400	Docker/Compose guide and no-store smoke, internal Kubernetes design and selected ECS/Fargate target	Production Docker/Compose cases, runnable kind/Envoy Gateway manifests and checks, and authorized ECS/Fargate deployment
#410	AWS reference design linking independent config/secrets/KV selection and acceptance requirements	Provider integration in #414 and separately authorized real IAM/resource/version/DynamoDB acceptance

Applications own their release artifacts, registry approval, credentials, configuration, secrets, storage policy and deployment. Trusted Server's images and releases remain downstream. Resolved review threads do not establish production acceptance or authorize merging unfinished issue criteria.

Main changes

• Conditional root container templates and generator regression tests. GNU binaries build on the pinned Bookworm Rust builder and run on the compatible shell-free gcr.io/distroless/cc-debian13:nonroot runtime. Runtime updates are pulled and tested; publication and rollback use the exact tested output digests.
• Disposable container-probe preparation stages framework sources deliberately. It labels its derived fixture lock, records both seed and prepared lock hashes, rejects changed registry name/version/source tuples, and keeps the separate application-owned committed consumer-lock contract intact.
• Native packaging checks cover context exclusions, ELF/native libraries, CA trust, UID/GID 10001, read-only root, dropped capabilities, mount configuration, published HTTP, controlled positive/negative TLS, and clean no-store Compose stop/start/teardown. STOPSIGNAL SIGINT uses the existing graceful path, not bounded SIGTERM acceptance.
• OCI verification checks manifests, blobs, inputs, source/run/attempt identity and scans; preserves optional supplied platform variants; and records BuildKit-resolved image materials in retained provenance and release acceptance records.
• The application-owned publishing recipe independently reconstructs committed inputs before registry login, selects exact same-run immutable artifacts, preserves child/index digests, and verifies expected attestation subjects and workflow identity. SCAN_EXCEPTIONS does not enable an automatic waiver path in this sample workflow.
• cargo-audit 0.22.2 is installed from a checksum-pinned source archive with its packaged lockfile. Rust advisory collection and framework verification remain report-only. Strict application verification rejects Rust vulnerabilities and unsound warnings as well as High/Critical image findings; no automatic Rust advisory waiver path is provided.
• Axum-only checks no longer trigger unrelated deployment smokes. Recipe actionlint, action-pin, Zizmor and ShellCheck coverage moves with the Axum workflow, the helper suite keeps one owner, and docs build in the existing docs-format job.
• Internal designs are dated and excluded from public navigation: docs/superpowers/specs/2026-10-14-deploying-axum-kubernetes.md and docs/superpowers/plans/2026-10-14-aws-reference-deployment.md.

Review follow-through at 6ef7f999

• Fixed explicit smoke failures and config/teardown diagnostics on Bash 3.2; added source-backed assertion regressions and direct source-archive checksum comparison.
• Moved Axum static validation into an independent job, with handoff depending on static validation and native builds. Removed cross-owner workflow assertions and unused deployment Node setup. Deployment workflow changes relative to the PR baseline are now only a reciprocal pin-maintenance comment.
• Restored maintained blob/main guide links. Strict application verification now rejects Rust vulnerabilities and unsound warnings. Both policies require unfiltered warning-category selection. Framework collection/verification remains report-only; OS exceptions cannot waive Rust findings, and the Tokio update remains separate.

Validation of these changes:

• All 130 helper tests with a freshly prepared context, zero skips.
• Bash 3.2 focused shell/deployment/advisory checks, including security rejection, visible diagnostics and failed checksum handling.
• Native amd64 generated-context/ELF/HTTP/TLS and Compose stop/start/teardown smokes under Bash 3.2, using the exact retained image from run 41481693509, not a rebuilt release candidate.
• The retained arm64 bundle is rejected by strict verification on its unsound warning and accepted by report-only verification.
• Scoped CLI tests and strict all-feature Clippy, Rust formatting, docs format/lint/build, Bash syntax, ShellCheck, actionlint, recipe action pins and Zizmor.
• All 660 deployment contracts and existing Rust build-cache action contracts.
• Current-head hosted checks are running, including native amd64/arm64 and handoff, at run 42162617461.

Native macOS userland, protected GHCR/OIDC publication and cloud/runtime production acceptance were not exercised. Two read-only workers checked shell portability and CI/policy behavior. The policy review found an informational-warning filter gap; it was fixed and covered before committing. No repository protection settings, dependency upgrade or Rust waiver path was added.

Earlier validation

Local checks at implementation commit 827317aa6eb0797c62d8f1469d089b7a9df0b3c1, including unchanged relevant inputs tested immediately before committing:

• Workspace/all-target tests and strict all-feature Clippy.
• Rust formatting, native adapter feature compilation, Spin WASM compilation and locked app-demo tests.
• All 114 helper tests passed with a freshly prepared context, with no skipped prepared-context checks.
• Docs formatting, lint and build; Bash syntax, ShellCheck, actionlint, action-reference checks and Zizmor.
• All 660 deploy-action contracts passed with fixture commit signing disabled and LC_ALL=C, matching the test's deterministic sorting contract.
• Native amd64 final-image build, constrained HTTP/TLS execution, clean Compose stop/start/teardown, and exact OCI registry-copy/Docker-pull identity passed under SCAN_POLICY=strict.
• That exact amd64 image reported zero High/Critical OS findings and zero cargo-audit vulnerabilities, with no exceptions or filtering. Its runtime child is sha256:cfc14c62b697225036d242674494d4b10e80c5a9525aa8d42c81489c1234aac5. This is point-in-time evidence, not a permanent vulnerability-free claim. The retained hosted inventories also include an unsound Tokio warning, distinct from the zero-vulnerability count. Current strict verification rejects that warning.
• Hosted native amd64/arm64 and handoff subsequently passed at 827317aa, run 41481693509. These earlier results do not substitute for current-head verification above.

The local handoff used the pinned BuildKit image and local Buildx 0.38.1. Hosted workflows pin Buildx 0.37.0. Both must retain the observed resolved-material/config-digest evidence and exact tested child identities. No QEMU execution substitutes for native arm64 acceptance.

Remaining acceptance outside this preparation

• Integrate accepted runtime/lifecycle/configuration contracts and exercise required-input failure before readiness, mounted-state persistence/recovery, competing-writer rejection, and bounded SIGTERM/draining through the actual candidate images on both native architectures.
• Execute separately authorized protected, app-owned GHCR publication and attestation verification without rebuilding tested artifacts. No registry publication or release waiver has been authorized by this review cleanup.
• Deliver and exercise plain Kubernetes manifests with kind and Envoy Gateway, including probes, storage, HTTPS/proxy trust, shutdown, upgrade and rollback. Local packaging does not qualify EKS or AWS services.
• Integrate accepted AppConfig Agent, Secrets Manager and DynamoDB providers, then obtain separately authorized real resource/IAM/version/table evidence. Ordinary KV proof does not certify Trusted Server identity/CAS. Proposed ECR transfer remains unimplemented and untested.
• Exercise stateless ECS/Fargate only with explicit account/resource/credential/budget/cleanup approval.

Packaging preparation is not production certification. No end-client-delivery guarantee, multi-writer embedded-store support, Kubernetes/EKS acceptance, Fargate acceptance, real AWS provider acceptance or identity/CAS certification is claimed.

Use a concrete final squash subject such as Scaffold and verify hardened Axum application containers; earlier implementation-progress commit subjects are not release-status evidence.

[prk-Jr]

🔧 Please resolve the integration conflicts against current main and validate the resulting revision before merging.

The reviewed PR head is 493644242f5839afbe5d37a032e08c13a381dc04; current main was checked at bc4aeb850c74ba982bdb4ad45309520b2ebb07b5. The merge check identifies conflicts in crates/edgezero-cli/src/generator.rs, .github/workflows/deploy-action.yml, and the location of docs/specs/aws-reference-deployment.md.

When resolving the generator conflict, keep the generated Rust version and the Docker builder digest aligned with main's toolchain upgrade. This PR introduces GENERATED_RUST_VERSION = "1.95.0" and a pinned 1.95 builder, while current main uses 1.98.1. Update the toolchain declaration and builder digest together; changing the image tag alone leaves the old digest selected. Rerun the affected checks on the integrated revision, including container preparation and native image execution.

No additional confirmed code defects were found in this review pass across the 29 changed files and relevant consumers.

Validation at the reviewed PR head: workspace tests passed (1,424 passed, one ignored), formatting and strict all-feature Clippy passed, native feature compilation and Spin WASM compilation passed, container helper tests passed including the prepared-fixture checks, and docs lint/format/build passed. Workflow and shell static checks passed, with a different local actionlint version from CI. These results do not validate a resolved merge with current main. Real Docker execution could not be repeated because the local daemon was unavailable; no current-head hosted checks were listed.

😃 The publisher independently recomputes committed source identities before registry login, verifies both architecture bundles, and prevents report-only scans from authorizing GHCR publication.

[aram356]

PR Review

Summary

Adds a hardened Axum Dockerfile scaffold, native amd64/arm64 container CI with an OCI artifact handoff, an app-owned GHCR publishing recipe, and Docker/Kubernetes/AWS deployment docs. The verification tooling is careful, but the generated image ignores SIGTERM, its runtime base means the shipped strict release gate rejects every build, and several integration hazards with current main merge without a textual conflict. This review adds to the existing conflict request rather than repeating it.

I built the fixture image from this head locally (native arm64) and measured its stop behavior and scan results; the numbers are in the inline comments.

Findings

Blocking

• 🔧 The container ignores SIGTERM, so every stop ends in SIGKILL (crates/edgezero-cli/src/templates/root/Dockerfile.hbs:64, inline)
• 🔧 The runtime base makes the shipped strict release gate unpassable (crates/edgezero-cli/src/templates/root/Dockerfile.hbs:52, inline)
• 🔧 New stackpop links after the repository transfer; the generated README's docs link returns 404 (crates/edgezero-cli/src/templates/root/README.md.hbs:57, inline)
• 🔧 The toolchain test breaks on the required Rust 1.98.1 bump (crates/edgezero-cli/src/generator.rs:1141, inline)
• 🔧 Merging would close #398, #399 and #400: GitHub lists all three as closing references for this PR, but the description says #399 and #400 must stay open and that #398's production acceptance is still outstanding. The description also calls the PR a draft, but it is not marked as one. Remove the #399/#400 links from the Development sidebar, decide explicitly whether this merge should close #398, and convert the PR to draft if it is not ready to merge.
• 🔧 The spec relocation breaks guide links (in addition to the existing conflict request): on main, docs/specs moved under docs/superpowers/, which docs/.vitepress/config.mts excludes with srcExclude: ['superpowers/**']. Once aws-reference-deployment.md moves there, the relative links at docs/guide/deploying-axum-containers.md:150 and docs/guide/deploying-axum-kubernetes.md:160,240 point at pages the site no longer builds, and VitePress fails the build on dead links by default. Link to the file on GitHub instead, or drop those links. Also note that no hosted CI has run on 49364424: GitHub does not run pull_request workflows while a PR has merge conflicts, so the integrated revision will be the first to exercise the new workflows on this head.

Non-blocking

• ♻️ Split this PR: 29 files and +4,325 lines combine about 340 lines of scaffold change (#398), about 3,000 lines of CI, verification and publishing tooling with tests (#399), and about 1,000 lines of deployment and AWS design docs (#400/#410). Each part has different reviewers, risk and acceptance criteria. Suggested split: (1) the Dockerfile, .dockerignore and README templates, generator tests and the Axum guide section; (2) native CI, the handoff helpers and the publishing recipe; (3) the design docs under docs/superpowers/specs/.
• ♻️ Keep point-in-time status out of committed docs: several pages record review-time state that will be wrong soon after merge. Examples: dated scan counts and a specific CI run link (docs/guide/publishing-axum-containers.md:208-226); this PR's own heads and CI status (docs/specs/aws-reference-deployment.md:24-36); a developer's local, uncommitted worktree (issue-410-native-store-providers, docs/specs/aws-reference-deployment.md:32 and :215); and issue-ownership notes in the Axum guide's "Packaging verification and limits" section. That belongs in the PR description and the issues. The docs should state the contract and its current limits.
• ♻️ Keep the Kubernetes draft out of the published navigation (docs/.vitepress/config.mts:61, inline)
• ♻️ Replace the hand-written TOML parser (scripts/axum_container_toml.mjs:3, inline)
• ♻️ One implementation of the context identity, not three (examples/container-publishing/release.yml:46, inline)
• ♻️ Every source edit reinstalls cmake and rebuilds all dependencies (crates/edgezero-cli/src/templates/root/Dockerfile.hbs:40, inline)
• 🤔 The vulnerability gate can't see the Rust dependencies (crates/edgezero-cli/src/templates/root/Dockerfile.hbs:46, inline)
• 🤔 The container CI resolves a fresh dependency graph on every run (scripts/prepare_generated_axum_container.sh:22, inline)
• 🤔 Downstream-specific build steps in the framework guide (docs/guide/adapters/axum.md:305, inline)
• ⛏ Non-hermetic unit test fails on hosts without readelf (scripts/tests/test_axum_container_artifacts.test.mjs:325, inline)
• ⛏ Stale Python references after the Node migration (scripts/axum_container_helpers.mjs:87, inline)
• ⛏ Prerequisite lists disagree (scripts/test_generated_axum_container.sh:7, inline)
• ⛏ Absolute symlink targets never resolve (scripts/axum_container_helpers.mjs:171, inline)
• ⛏ Duplicate BuildKit pin for an unused builder (.github/workflows/axum-container.yml:104, inline)
• ⛏ Docs-only edits run both native container builds (.github/workflows/axum-container.yml:18, inline)
• ⛏ Drop bzip2/xz decompression of artifact input (examples/container-publishing/verify_artifacts.mjs:189, inline)

CI Status

Local runs at 49364424:

• fmt: PASS
• clippy (--workspace --all-targets --all-features -- -D warnings): PASS
• tests (cargo test --workspace --all-targets): PASS
• Node helper tests: 50 pass, 1 fail on macOS (non-hermetic; see inline), 3 skipped by default. The 3 prepared-fixture tests pass with AXUM_PREPARED_CONTEXT set.
• shellcheck, actionlint and zizmor on the new workflows and scripts: clean
• Fixture docker build (native arm64): PASS
• Hosted CI on this head: none has run (blocked by merge conflicts)

[dhruv8sh]

PR Review

Summary

Solid, security-conscious Axum container packaging, but the branch is stale against main (missing #366 and #419), conflicts, and reintroduces stackpop references. Rebase and the items below are needed before merge.

😃 Praise

• Strong hardening: non-root UID 10001, --read-only, cap-drop=ALL, no-new-privileges; digest-pinned base images and actions with persist-credentials: false; toolchain verified before rustup; strict scan policy and post-push digest verification for GHCR; Dockerfile/.dockerignore generated only for the Axum adapter, with good unit tests (mock-rustup toolchain check, digit-leading app names).

Findings

Blocking

• 🔧 Stale base / merge conflict: branch lacks #366 (Rust 1.98.1, Node 24.20.0) and #419 (IABTechLab transfer); git merge-tree reports a conflict. GENERATED_RUST_VERSION = "1.95.0" in generator.rs while main emits rust 1.98.1. Rebase, bump to 1.98.1, re-resolve the rust:1.98.1-bookworm@sha256:… digest in Dockerfile.hbs, and update the hard-coded "1.95.0" in the new tests.
• 🔧 stackpop references reintroduced (inline on the README template). Also in docs/guide/publishing-axum-containers.md (git dependency examples), issue/PR links in the Kubernetes guide and docs/specs/aws-reference-deployment.md, and the org.opencontainers.image.source fallback stackpop/edgezero in examples/container-publishing/release.yml. Switch to IABTechLab/edgezero / iabtechlab.github.io/edgezero.
• 🔧 Hard-coded Node 24.12.0 in workflows (inline).
• ❓ Draft docs in main nav (inline).

Non-blocking

• 🤔 Bespoke tooling scope: scripts/axum_container_toml.mjs is a 182-line hand-rolled TOML parser and examples/container-publishing/verify_artifacts.mjs is 524 lines of OCI/tar verification; with release.yml and publish.sh that is a lot of security-sensitive code for the framework to own as an "example". Consider a small TOML dep (e.g. smol-toml) for tests, or splitting the publishing recipe into its own PR.
• 🤔 Missing # syntax= directive for heredoc RUN (inline).
• ⛏ No dependency layer caching in the Dockerfile (inline).
• ⛏ rmSync without force (inline).
• 📝 Gate toolchain: clippy on 1.98.1 fails in untouched edgezero-macros (inline_trait_bounds) because the branch still pins 1.95.0; resolved by the rebase.

CI Status

• fmt: PASS
• clippy: PASS on branch pin 1.95.0; FAIL on 1.98.1 (pre-existing lint in untouched crate, fixed by rebase)
• tests: PASS
• Docker/OCI workflows and Node helper tests: not run locally (require Docker/buildx)

[prk-Jr]

🔧 Review: request changes

Reviewed all 33 changed files at 1f244a7c1b78dafd5bc9291ad86c385360636f72, against 87456c708ffe9edf86b3a0f8b06dfd15d60b07b5. One existing merge-metadata blocker remains; one new, nonblocking Compose smoke finding is attached inline. No additional blocking defect was confirmed in scaffolding, toolchain validation, container hardening, TLS checks, publishing permissions, or deployment contracts.

🔧 P2 — Merging would prematurely close unfinished issues (existing finding, still unresolved)

GitHub currently lists #398, #399, and #400 as closing references. The PR explicitly says #399/#400 must remain open, while #398 still has outstanding production acceptance. Merging would therefore close work the PR acknowledges is incomplete.

Remove #399/#400 from the closing associations and retain #398 until its remaining acceptance is complete, or explicitly separate the delivered packaging slice. The PR also remains marked ready for review despite its description calling it draft.

😃 The artifact handoff is carefully designed. The publisher independently recomputes committed source identities before registry login, verifies both architecture bundles and their metadata, and preserves tested child digests. Report-only scans cannot authorize GHCR publication.

Validation: Rust 1.98.1 workspace tests passed (1,553 passed, one ignored), together with formatting, strict all-feature Clippy, native feature compilation, and Spin WASM compilation. The helper suite passed 77 tests, followed by all three prepared-context checks passing. Local Node was 24.12.0; hosted checks used the declared 24.20.0. ShellCheck, actionlint, and diff checks passed. Hosted native amd64/arm64 container and handoff checks passed; their merge-result file tree exactly matches the reviewed head.

Local Docker execution was unavailable because the daemon was stopped. Protected GHCR publication, attestations, Kubernetes, and AWS acceptance remain unexecuted; the docs explicitly defer those claims. Green packaging checks do not establish release security approval: current hosted logs retain one High OS finding and five Rust advisories.

[dhruv8sh]

PR Review

Summary

Container scaffolding, native multi-arch CI and the handoff verification are solid and hosted CI is green, but there are two correctness gaps (builder digest/version drift, vacuous lockfile check) and an OIDC scoping question to resolve before merge.

Findings

Blocking

• 🔧 Rust version and builder digest can drift (Dockerfile.hbs:4)
• 🔧 Lockfile check is vacuous (prepare_generated_axum_container.sh:23)
• ❓ OIDC token scoped to the whole publisher job (release.yml:121)

Non-blocking

• 🤔 Scope: ~470 lines of draft AWS/Kubernetes design (not #398) plus a 539-line hand-rolled tar/OCI verifier. Suggest splitting the design docs into their own PR under #391/#400, and weighing whether Skopeo/oras verification could replace some of the custom parsing.
• ⛏ Pinning consistency: setup-node comment is only # v6; the recipe uses Node 24.12.0 vs the repo's 24.20.0; cargo install cargo-audit has no checksum while gh is checksum-verified.
• ⛏ Docs hygiene: new docs/superpowers/ files are undated (others use YYYY-MM-DD-); the K8s spec uses VitePress ::: callouts but is excluded from the site, so it renders raw on GitHub (use > [!WARNING]); the guide links blob/main/... URLs that 404 until merge; the PR body cites docs/specs/aws-reference-deployment.md but the file is under docs/superpowers/plans/.
• ⛏ Head commit message ("Implemented all 18 accepted plan items… Changes remain uncommitted…") should be squashed/reworded.
• 📝 The runtime base distroless/cc-debian12:nonroot is intentionally floating; consequently the attested base_images record a tag, not runtime bytes. Consider recording the resolved digest from BuildKit metadata.

CI Status

• fmt: PASS
• clippy: PASS
• tests: PASS (hosted CI on 1f244a7; local Node script suite 77 pass / 3 skip)

[ChristianPavilonis]

Review reconciliation at d7c4ee9

I checked all 39 review threads and all five nonempty submitted review summaries against the current branch. No pending review was exposed by the API. This pass changes GitHub discussion state only, not source code or release policy.

Replied and resolved

• Toolchain test fixture: 66c73e9 makes the test use its synthetic Rust pin rather than inheriting the production 1.98.1 pin. The focused regression passed.
• Duplicate BuildKit pin: 4ea53d6 removes the unused image setting from both workflows. Only the handoff helper creates the pinned builder.
• Compose teardown: 4ea53d6 checks teardown before PASS, preserves failures and diagnostics, and adds failure-path regressions.

There are now 29 resolved threads and 10 open threads. Previously resolved threads already had explanatory replies, so I did not duplicate them or dismiss any reviewer's change request.

The review-summary integration request is also addressed. Main was merged in 66c73e9, production Rust is 1.98.1, Node is aligned to 24.20.0, and GitHub reports MERGEABLE. The cited point-in-time documentation status was removed, internal drafts were taken out of published navigation, relocated documentation links were repaired, and the Kubernetes draft now uses GitHub-compatible warning syntax. These do not clear the separate future builder-digest maintenance or merge-acceptance requests below.

Remaining inline requests

Request	Status and proposed follow-through
Runtime base / strict release gate	Partial. 1f244a7 switched to distroless, removed runtime apt tooling, moved TLS diagnostics to a separate image, and adapted the smokes. Current native scans still report one fixable High OS finding; the Rust inventory reports five advisories. Keep this open rather than equating report-only packaging success with release approval. No waiver or weaker gate is assumed.
Source-edit rebuilds and dependency caching	Two related threads. cmake installation already precedes source COPY, but Cargo cache mounts are still absent. Batch the remaining build-cache improvement.
Dockerfile frontend directive	Still absent. Add the reviewed frontend pin and appropriate regression.
Rust version / builder digest drift	Still open. Pair the maintained version/digest and make the upgrade instructions explicitly require updating the digest. The current integration fix alone does not prevent future drift.
Derived fixture lock	Partial. Hosted prepared-context tests already reject registry versions outside the seed lock. The preparation script itself still reconciles before its locked check. Make the derived-lock identity and what that check proves explicit, preserving the separate committed consumer-lock contract.
Whole-job OIDC authority	Requires an explicit security choice. Separate copying from signing authority, or retain the combined job and explicitly accept that any compromised step in it can request OIDC. Merely delaying registry login does not isolate OIDC.
Over-broad deploy workflow triggers	Still open. Move or gate Axum-only checks without losing existing actionlint, pin and Zizmor coverage for the inactive release recipe.
arm64 variant	Still open. A focused Node reproduction confirms that description and index construction discard an explicitly supplied v8 variant. Preserve optional supplied metadata without inventing a variant for images that omit it.
SCAN_EXCEPTIONS plumbing	Partial. The sample performs strict verification and record construction before publication, so the alleged publish-then-record failure is not its current execution order. The standalone record command supports --exceptions. Clarify that exporting SCAN_EXCEPTIONS does not enable exceptions in the sample's direct verifier calls, preserving its documented no-automatic-waiver policy.

Remaining summary-only requests

• GitHub still lists Axum: scaffold hardened production application Dockerfiles #398, Axum: validate multi-platform containers and provide an application publishing workflow #399 and Axum: document Docker, Kubernetes, and managed-cloud deployments #400 as closing references. Remove Axum: validate multi-platform containers and provide an application publishing workflow #399/Axum: document Docker, Kubernetes, and managed-cloud deployments #400 closing associations to honor the stated contract. Axum: scaffold hardened production application Dockerfiles #398 still needs its production acceptance before closure, unless an owner explicitly approves a narrower delivery slice. The PR is also ready for review while its body calls it draft.
• The split/scope suggestions remain unanswered. Separating scaffold, publishing and design work changes this PR's scope; replacing the custom OCI verifier is not a mechanical cleanup. Neither choice is assumed approved.
• Batch the remaining documentation/history hygiene: dated design filenames with updated links, review-time GitHub links that currently target files absent from main, stale paths/CI/conflict/scan statements in the PR body, and a concrete final squash/reworded commit message.
• The latest review's nonblocking supply-chain follow-ups remain: an explicit cargo-audit tool checksum pin, distinct from its existing version-pinned locked Cargo install, and recording the resolved runtime-base digest rather than only the floating tag. Exact released output digests are already preserved; resolved-base attribution is a separate improvement.

Evidence and limits

• Local helper suite: 88 passed, zero failed, three prepared-context checks skipped.
• Focused Rust toolchain regression: one passed.
• Current-head hosted container run: all 91 helper checks passed, including prepared context; native amd64/arm64 HTTP/TLS/Compose checks and handoff passed.
• The hosted checkout was merge 316a4e2. Its Git tree exactly equals d7c4ee9, so that execution evidence applies to the current source tree. All 33 reported PR checks are successful.
• Protected GHCR publication/attestation and production Kubernetes/AWS/runtime acceptance were not newly exercised. Packaging success does not clear their outstanding acceptance requirements.

[prk-Jr]

PR Review

Summary

Follow-up review at d7c4ee9e4a9db1d547272b3f13a6177f10098dba (base 87456c708ffe9edf86b3a0f8b06dfd15d60b07b5), covering the changes since my previous review at 1f244a7c. 4ea53d66 and d7c4ee9e do what they set out to do, and I found no new code defects in this delta. One earlier blocker is still open: it concerns PR metadata, not code.

Findings

Blocking

• 🔧 Merging would still close unfinished issues (existing finding, still unresolved): GitHub currently lists #398, #399 and #400 as closing references. The PR description says #399 and #400 must stay open and that #398 still has production acceptance outstanding (required-input startup failure, durable restart, readiness and bounded SIGTERM). Once the code threads are settled, merging would close all three automatically. Fix: remove #399 and #400 from the Development sidebar. Either change the #398 link to a non-closing reference or have an owner explicitly approve closing it on the packaging slice. The description also calls the PR a draft, but it is still marked ready for review. This is a metadata change only.

😃 Praise

• The Compose smoke test now checks compose down before reporting PASS. It keeps the primary failure status if cleanup also fails and preserves diagnostics. I made the success-path teardown ignore failures, and both new teardown tests failed, so the regression is covered.
• Artifact selection now reads every page of results, and a duplicate accepted artifact that only appears on a later page is rejected rather than silently selected.

Checked in this delta, no findings

• node-version-file: .tool-versions with an exact # v6.5.0 pin comment. The standalone recipe's 24.20.0 pin is now kept in sync with .tool-versions by a test.
• gh api --paginate --slurp | jq: the pinned publisher gh (2.102.0) supports --slurp, and the default Actions bash shell uses pipefail, so an API failure fails the step.
• Registry readiness now fails the job if the registry never starts. Buildx setup uses driver: docker, and only the handoff helper creates the pinned BuildKit builder.
• The runtime-tar check now rejects .git at the root and in nested paths while still allowing .gitignore/.gitkeep. The GHCR preflight fails with a clear message, before any network call, when inline credentials are missing.

The other reviewers' open threads are not repeated here.

CI Status

• Hosted CI at d7c4ee9e: all 33 checks passed, including native amd64/arm64 container execution and the handoff (run 41463920215, Node 24.20.0).
• Local helper suite: 88 passed, 0 failed, 3 prepared-context tests skipped. Local Node was 24.12.0, not the declared 24.20.0.
• Not rerun locally: Rust fmt/clippy/tests (no Rust changes since my last review; hosted CI covers them) and Docker execution. GHCR publication/attestation, Kubernetes and AWS acceptance remain unexercised, as the PR states.

[ChristianPavilonis]

Review follow-through completed

Pushed 91cb3e6 and 827317a. Every one of the 39 review threads is now resolved, with an explanatory reply. The 26 threads already resolved before this audit retained their existing fix explanations; this audit resolved three previously fixed threads, then replied to and resolved the ten remaining threads after implementing or recording the agreed response.

The owner's decisions also apply to the summary-only PR split/custom-verifier request and later scope request: retain this PR's current scope and verifier. The combined OIDC publisher is retained with its whole-job signing-authority risk explicitly accepted in the guide and thread reply. Review-summary text has no separate thread-resolution control; these decisions are recorded here and in the updated PR description.

The remaining fixes include:

• Paired generator Rust/builder pins, explicit digest-upgrade instructions, the pinned Dockerfile frontend and Cargo cache mounts.
• Standalone derived-lock checks and seed/prepared hash labeling, optional platform-variant preservation, BuildKit-resolved base provenance and checksum-pinned cargo-audit source installation.
• Narrower CI ownership/triggers with retained static/security checks and docs-build coverage.
• Dated internal design paths, pushed commit-permalink guide links, exception-policy clarification and current PR status/evidence.
• A compatible Debian 13 distroless runtime and three targeted patch-level lockfile updates that clear the previously recorded image/advisory findings in the tested local candidate.

I removed the closing associations for #398, #399 and #400 and verified that all remain open. This preserves their outstanding acceptance criteria rather than closing unfinished work. The PR body now agrees with its ready-for-review metadata and explicitly distinguishes delivered preparation from production acceptance. Concrete new commit subjects and the proposed final squash subject replace progress prose as the description of this delivery; old tested revision hashes were preserved.

Local validation at 827317a passed:

• All 114 helper tests with a fresh prepared context, with no skips.
• Workspace/all-target tests, strict all-feature Clippy, formatting, native feature checks, Spin WASM compilation and locked app-demo tests.
• Docs formatting/lint/build, Bash syntax, ShellCheck, actionlint, action pin checks, Zizmor and all 660 deploy-action contracts. The action contracts require C locale and fixture commit signing disabled in this local environment.
• The full native amd64 final-image build, constrained HTTP/TLS and clean Compose checks, followed by separate strict import, preserved the exact runtime digest. Its strict scan had zero High/Critical OS findings and its lock-bound Rust inventory had zero advisories, without exceptions.
• Actual retained BuildKit metadata includes the resolved runtime digest and config binding. Both pushed design permalinks were verified through the GitHub contents API.

Current-head hosted checks, including native arm64 and the two-platform handoff, are still running. These are not claimed as passed. Protected GHCR/OIDC publication and production Kubernetes/AWS/runtime acceptance were not exercised or waived. Resolving review discussions does not dismiss reviewer verdicts, grant release approval or complete those acceptance criteria.

[prk-Jr]

PR Review

Summary

Re-review at 827317aa6eb0797c62d8f1469d089b7a9df0b3c1 (base 87456c708ffe9edf86b3a0f8b06dfd15d60b07b5), covering 91cb3e65 and 827317aa. My earlier blocker is resolved: GitHub no longer lists any closing issues, and the description now says #398, #399, #400 and #410 remain open. I found no new defects. The two inline comments are non-blocking. Approving, which replaces my previous change request.

😃 Praise

• The remaining review threads were closed with real mechanisms, not just replies: the paired Rust version/builder digest constant, the derived-lock registry subset check, the platform-bound runtime material in the signed release record, and the checksum-verified cargo-audit source install. I disabled the subset check as a test, and both of its regression tests failed, so the check is covered.
• The cleared scan comes from the Debian 13 runtime and the patch-level lockfile updates, not from a waiver. The hosted native logs at this head show 0 High and 0 Critical OS findings and 0 Rust advisories on both architectures.

Verified in this delta

• The Dockerfile renders the builder digest from the same constant as RUST_VERSION. The frontend is pinned, and the cache mounts still copy the binary out within the same RUN. .dockerignore excludes **/target.
• The cargo-audit 0.22.2 checksum matches the checksum crates.io publishes for that version.
• The moved trigger paths cover everything removed from deploy-action.yml. The release recipe keeps actionlint, the pin gate, Zizmor and ShellCheck.
• The derived-lock check applies only to the framework fixture. The app-owned recipe still uses prepare_inputs.mjs and its committed lock.
• An optional image variant is carried from the config into the receipt and the index, with a check that they match.

Findings

Non-blocking

• 🤔 Guide links are pinned to pre-merge commits (inline, docs/guide/adapters/axum.md:342; also docs/guide/deploying-axum-containers.md:132,150)
• ⛏ Duplicated validation tool versions; the static checks gate the native builds (inline, .github/workflows/axum-container.yml:51)

CI Status

• Hosted CI at 827317aa: all 33 checks passed, including native amd64/arm64 and the handoff (run 41481693509).
• Local: fmt passed. cargo test -p edgezero-cli passed. Clippy (-p edgezero-cli --all-targets --all-features -D warnings) passed on the pinned Rust 1.98.1. The container helper suite had 111 passed, 0 failed and 3 prepared-context tests skipped, on local Node 24.12.0.
• Not exercised: local Docker, GHCR/OIDC publication, and Kubernetes/AWS acceptance, which the PR explicitly leaves outstanding.

[aram356]

PR Review

Summary

Re-review at 827317aa. I checked all 22 findings from my previous review against this head: 21 are fixed in code, and the PR split was declined as an owner decision, which I'm not raising again. The runtime now passes the strict image gate and stops cleanly, and the integration problems with main are resolved. Two new blocking problems remain: the smoke scripts' security assertions can't fail under macOS's default bash, and two tests assert workflow files whose changes never trigger them. prk-Jr's two open threads from yesterday (commit permalinks, duplicated tool pins on the critical path) are not repeated here.

Previous findings, verified

• Stop signal: on a native arm64 build of the fixture, docker stop now takes 1 s and exits 0 (previously 20 s, exit 137); the Compose example's stop takes 1 s and exits 0 (previously 31 s, exit 137).
• Runtime base: the arm64 image is 64 MB (previously 177 MB) and has no shell. The hosted bundles from run 41481693509 report 0 High/Critical OS findings on both architectures.
• Floating runtime provenance: both hosted bundles record the resolved cc-debian13:nonroot digest in their BuildKit materials, and the handoff verified them.
• Confirmed in code: IABTechLab links, the toolchain test's synthetic pin, no closing-issue associations, internal designs out of the published nav, smol-toml instead of the hand-written parser, the shared context hash, cmake before COPY plus cache mounts, the seeded fixture lockfile, the generic asset-stage example, the stubbed readelf test, the Python references, absolute symlink resolution, the single BuildKit pin, docs paths out of the container triggers, and the removal of bzip2/xz.

Findings

Blocking

• 🔧 On macOS's default bash, the smoke scripts' security checks can't fail (scripts/test_axum_container_deployment.sh:39-40, plus four lines in scripts/test_generated_axum_container.sh; inline)
• 🔧 Tests assert workflow files whose changes never trigger them (scripts/tests/test_axum_container_publishing.test.mjs:252 and :232; inline)

Non-blocking

• 🤔 Strict release policy doesn't cover the Rust advisory inventory (examples/container-publishing/verify_artifacts.mjs:307; inline)
• ⛏ Leftover changes in deploy-action.yml (.github/workflows/deploy-action.yml:107; inline)
• ⛏ GNU-only checksum check (scripts/test_axum_container_handoff.sh:103; inline)

📌 Out of Scope

• Main's Cargo.lock also pins tokio 1.53.1, which RUSTSEC-2026-0386 (AsyncFd::new and related constructors violate I/O safety) marks unsound; it is patched in 1.53.3. The fixture lock inherits it from there. Bump it in a separate change.

CI Status

Local runs at 827317aa:

• fmt: PASS
• clippy (--workspace --all-targets --all-features -- -D warnings): PASS
• tests (cargo test --workspace --all-targets): PASS (1,554 passed, 1 ignored)
• feature check (--features "fastly cloudflare spin") and Spin wasm32-wasip2 check: PASS
• Node helper tests on macOS (Node 24.20.0, /bin/bash 3.2): 103 pass, 8 fail (explained inline), 3 skipped. The 3 prepared-context tests pass with AXUM_PREPARED_CONTEXT set.
• shellcheck, actionlint and zizmor on the new and changed workflows and scripts: clean
• Native arm64 fixture build, HTTP, docker stop and Compose stop: PASS
• Hosted CI at this head: all checks pass, including native amd64/arm64 and the handoff

[ChristianPavilonis]

Review follow-through at 6ef7f99

Implemented the approved responses in 6ef7f99, replied to all seven newly open threads, and resolved them. Earlier resolved threads and their explanations were preserved.

• The Bash 3.2 security assertions and visible config/teardown diagnostics are fixed, with regressions and actual native amd64 HTTP/TLS/Compose execution under Bash 3.2 against the retained tested image.
• Axum tests no longer enforce unrelated deployment/docs workflow text. Static recipe/security validation runs independently, and handoff needs both static checks and native builds. The deployment workflow differs from the PR baseline only by the pin-maintenance comment.
• Guide links now follow main. Source archive verification compares the computed hash directly before extraction/install.
• Strict application verification blocks Rust vulnerabilities and unsound warnings. Collection and framework verification remain report-only. Filtered warning-category settings are rejected under both policies. OS exceptions do not waive Rust findings; no Rust waiver mechanism or dependency upgrade was added.

The review summary's Tokio warning is real: the retained arm64 bundle now fails strict verification on that warning and still passes report-only. Zero vulnerabilities is not zero advisory warnings. The Tokio update remains separate, and the PR description now makes that distinction.

Local validation passed: all 130 helper tests with fresh prepared context and no skips; Bash 3.2 focused regressions; native amd64 HTTP/TLS/Compose; CLI tests and strict all-feature Clippy; Rust formatting; docs formatting/lint/build; Bash syntax, ShellCheck, actionlint, recipe action pins and Zizmor; all 660 deployment contracts and the existing Rust build-cache action contracts. Two read-only workers reviewed the changes; their one policy-filter finding was fixed and covered before committing.

Current-head hosted checks are running at https://github.com/IABTechLab/edgezero/actions/runs/42162617461. Preparation passed and both native jobs have started independently of static validation, confirming the intended scheduling change. No hosted result is claimed before completion.

This does not dismiss reviewer verdicts, close outstanding acceptance issues, approve registry publication, change repository protection, or establish native macOS/GHCR/OIDC/Kubernetes/AWS production acceptance.

[prk-Jr]

PR Review

Summary

Full re-review at 89d50968f4c958b1229230e63f8856e90ec83aba (base cf2c57b1), covering all 37 changed files rather than only the delta since my approval at 827317aa. The Rust generator, the Dockerfile template, the workflows and the verifier's OCI/scan handling hold up. Two small defects contradict claims this PR makes, so I'm moving from approve to request changes. Both fixes are a line or two.

😃 Praise

• The verifier is careful: digest regexes are anchored, tar members are checked against an allowlist of names, blob reads use O_NOFOLLOW, exceptions must match exactly, and strict mode can't be bypassed through Trivy severity casing or a filtered cargo-audit report.
• Native build jobs get no credentials or OIDC, every action is pinned to a full SHA, and no ${{ }} expression is interpolated into a run: block. The builder digest and Rust version come from one paired constant.

Findings

Blocking

• 🔧 Bash 3.2 cleanup aborts when the smoke fails before any container exists (scripts/test_generated_axum_container.sh:52; inline)
• 🔧 Lowercase or indented FROM lines escape the pinned-base check (examples/container-publishing/prepare_inputs.mjs:42, verify_artifacts.mjs:450; inline)

Non-blocking

• 🤔 Compose-check diagnostics are lost in CI (scripts/test_axum_container_deployment.sh:28; inline)
• 📝 The docs could say SIGTERM is ignored outright. The app runs as PID 1 and installs only tokio::signal::ctrl_c() (dev_server.rs:314), so SIGTERM does nothing until the runtime's kill timeout. STOPSIGNAL SIGINT is right for Docker and Compose, but docker kill -s TERM or an orchestrator stop-signal override gets no reaction, then a SIGKILL. One sentence in axum.md and deploying-axum-containers.md would make that explicit.
• 🌱 Hardening (low; producer-controlled or fails closed):
  • hashContext hashes name\0bytes\0 with no length prefix, so two different contexts can produce the same hash, and the native build never re-hashes the extracted context before docker buildx build. The separate git-archive hash still binds the source.
  • readTar accumulates PAX metadata without a total limit and re-spreads the object on every header, so time and memory grow without bound. extract/runtime-tar don't checksum first.
  • The derived-lock subset check (axum_container_helpers.mjs:104) compares only registry+ sources, so an added git+/sparse+ package passes.
  • Every "unsafe archive members" test case also sets link, so removing the traversal checks would still pass.
  • A CRLF .tool-versions fails the toolchain check with a misleading "exact X.Y.Z" message.

CI Status

• Hosted at 89d50968: all 33 checks pass, including native amd64/arm64 and the handoff (run 42185282051).
• Local on Rust 1.98.1: fmt passed, cargo test -p edgezero-cli passed (227), clippy (-p edgezero-cli --all-targets --all-features -D warnings) passed. Helper suite: 127 passed, 0 failed, 3 prepared-context tests skipped, on local Node 24.12.0. actionlint, ShellCheck and the action-pin check were clean, and the docs build succeeded.
• Not exercised: local Docker build, GHCR/OIDC publication, Kubernetes/AWS.

[prk-Jr]

🔧 Cleanup aborts on Bash 3.2 when CONTAINERS is empty

CONTAINERS only gets entries from line 129 onward. If anything fails before that, the EXIT trap reaches this loop with an empty array under set -u. That includes the context probe catching a .dockerignore regression, a failed docker build, a platform mismatch, or the ELF/user checks. macOS /bin/bash 3.2 then aborts with CONTAINERS[@]: unbound variable. The rest of cleanup never runs: the probe and app images are not removed, nothing reaches --evidence-dir (no logs, no exit-status), and the "diagnostics retained" message never prints. The script still exits non-zero, so the check fails correctly, but the diagnostics this PR says are fixed on Bash 3.2 are lost.

Reproduced with /bin/bash 3.2.57: set -u; A=(); trap 'for c in "${A[@]}"; do :; done; echo done' EXIT; false prints the unbound-variable error and never reaches done. Hosted Linux uses bash 5, so CI can't catch it.

Fix:

Suggested change

	for container in "${CONTAINERS[@]}"; do
	for container in ${CONTAINERS[@]+"${CONTAINERS[@]}"}; do

Ideally, also add a Bash 3.2 test that runs cleanup with an empty array.

[prk-Jr]

🔧 Lowercase or indented FROM lines are not recorded, so unpinned bases pass check_inputs

check_inputs (verify_artifacts.mjs:337) only accepts base_images that are pinned @sha256 or the allowed distroless runtime tag. But /^FROM (\S+)/gm only matches an uppercase FROM at column 0, and Dockerfile keywords are case-insensitive and may be indented. With from debian:latest AS tools or FROM debian:latest in an app's Dockerfile, base_images lists only the pinned bases, so the release record says every base is pinned when one isn't. resolved_base_images only requires a material for the runtime tag, so it doesn't catch the extra base either.

The opposite also happens: FROM --platform=$BUILDPLATFORM rust@sha256:… records --platform=$BUILDPLATFORM and is rejected, which blocks a common multi-stage pattern.

Fix: apply this here and at the same regex in verify_artifacts.mjs:450 (the two must stay identical for release-record):

Suggested change

	base_images: [...files.get('Dockerfile').toString('utf8').matchAll(/^FROM (\S+)/gm)].map(m => m[1]),
	base_images: [...files.get('Dockerfile').toString('utf8').matchAll(/^\s*FROM\s+(?:--\S+\s+)*(\S+)/gim)].map(m => m[1]),

Optionally, also require every pkg:docker/ material in resolved_base_images to match a declared base.

[prk-Jr]

🤔 Compose diagnostics stay in the runner's temp directory

On failure these logs go only to $WORK, a mktemp directory. The handoff doesn't pass an evidence location, and the failure upload in axum-container.yml only collects accepted/*.log and accepted/smoke. So a Compose stop/restriction failure in CI shows just the one-line message. Consider taking an optional output directory, copying $WORK into the uploaded $OUTPUT, or printing the compose logs to stderr in this branch. (Inferred from the code paths; not run.)