diff --git a/AGENTS.md b/AGENTS.md index 284b9cc..538ac8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,6 +24,7 @@ docker run --rm -v "$PWD/strategy.pine:/in/strategy.pine:ro" -v "$PWD/ohlcv.csv: ## Layout - `docker/` — `Dockerfile`, the sixteen vendored engine harness files with `HARNESS-SHA256SUMS`, `GUARD-POLICY-SETS.pin`, `SOURCE-SETS.tsv` (Debian source-set registry), `hpo-toolchain.args`, the HPO dispatcher, license sums. + The image also carries the engine's selected-admit helper at `/opt/pineforge/bin/pineforge-selected-admit` and its manifest at `/opt/pineforge/share/pineforge/pineforge-selected-admit.manifest.json`, two files taken from the same verified engine release tarball as the library and headers. - `docs/release-chain.md` — the release-chain contract: candidate and readiness, the `pineforge-release` ready event, consumer receipts, rollback. - `facts/` — the public facts store (`facts.json`, `facts.schema.json`), `requirements-validation.txt`, `receipt-exclusions.json`, `repo-descriptions.json`; its README.md says how it is validated. - `scripts/` — the logic the workflows run: `release_pair.py` (version order, pairing), `release_pipeline.py` (candidate, finalize, dispatch, reconcile), `release_receipts.py`, `release_sources.py`, `release_smoke.py`, `facts_*.py`, `sync-harness.sh`. @@ -93,6 +94,7 @@ What a Pine script computes is a parity probe, not a CI test; probes live in the - Append-only: recovery "never force-pushes or rewrites the tag"; "A release's source assets are never deleted while its image remains on ghcr."; `scripts/release_rollback.py` restores recorded aliases and deletes nothing. - Secrets: "Credentials are a single org GitHub App (`PINEFORGE_APP_ID` / `PINEFORGE_APP_PRIVATE_KEY`); no personal access tokens." "Private targets are configured in the protected `release-automation` environment, not published in the workflow or this diagram". "Private targets never enter matrix outputs, log messages or public receipts." - "The guard policy value is read from `docker/GUARD-POLICY-SETS.pin`, computed privately and never re-derived here". "Every harness file in the image must be byte for byte the engine's file at the ref that scripts/sync-harness.sh synced from" (`docker/Dockerfile`): never hand-edit the sixteen vendored files or `docker/HARNESS-SHA256SUMS`. + The harness check stays a sixteen-name check; a separate exact helper check verifies the two tarball assets, their paths, root ownership, modes, source sha256 values and manifest, and image smoke rechecks the installed files and canonical `--describe` output. - `facts/facts.json` is machine-written: "do not hand-edit quantities". "PR code never receives App credentials"; `authz` refuses `pull_request_target`. - Every `uses:` in `.github` is a lowercase 40-hex commit SHA, a local path without a ref, or a sha256-pinned Docker image (`test_release_action_pins.py`); major bumps of release-chain actions are "reviewed and merged by hand", never auto-merged (`.github/dependabot.yml`). Add no `timeout-minutes` to a job (`python-test.yml` says a checker refuses new ones). - `README.md`, `docs/release-chain.md` and `LICENSE` are inputs of the release tests, and several tests pin their exact text (the README pair rule is pinned by SHA-256): change that text only together with its test. diff --git a/CLAUDE.md b/CLAUDE.md index 48c450c..ebcecac 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -24,6 +24,7 @@ docker run --rm -v "$PWD/strategy.pine:/in/strategy.pine:ro" -v "$PWD/ohlcv.csv: ## Layout - `docker/` — `Dockerfile`, the sixteen vendored engine harness files with `HARNESS-SHA256SUMS`, `GUARD-POLICY-SETS.pin`, `SOURCE-SETS.tsv` (Debian source-set registry), `hpo-toolchain.args`, the HPO dispatcher, license sums. + The image also carries the engine's selected-admit helper at `/opt/pineforge/bin/pineforge-selected-admit` and its manifest at `/opt/pineforge/share/pineforge/pineforge-selected-admit.manifest.json`, two files taken from the same verified engine release tarball as the library and headers. - `docs/release-chain.md` — the release-chain contract: candidate and readiness, the `pineforge-release` ready event, consumer receipts, rollback. - `facts/` — the public facts store (`facts.json`, `facts.schema.json`), `requirements-validation.txt`, `receipt-exclusions.json`, `repo-descriptions.json`; its README.md says how it is validated. - `scripts/` — the logic the workflows run: `release_pair.py` (version order, pairing), `release_pipeline.py` (candidate, finalize, dispatch, reconcile), `release_receipts.py`, `release_sources.py`, `release_smoke.py`, `facts_*.py`, `sync-harness.sh`. @@ -93,6 +94,7 @@ What a Pine script computes is a parity probe, not a CI test; probes live in the - Append-only: recovery "never force-pushes or rewrites the tag"; "A release's source assets are never deleted while its image remains on ghcr."; `scripts/release_rollback.py` restores recorded aliases and deletes nothing. - Secrets: "Credentials are a single org GitHub App (`PINEFORGE_APP_ID` / `PINEFORGE_APP_PRIVATE_KEY`); no personal access tokens." "Private targets are configured in the protected `release-automation` environment, not published in the workflow or this diagram". "Private targets never enter matrix outputs, log messages or public receipts." - "The guard policy value is read from `docker/GUARD-POLICY-SETS.pin`, computed privately and never re-derived here". "Every harness file in the image must be byte for byte the engine's file at the ref that scripts/sync-harness.sh synced from" (`docker/Dockerfile`): never hand-edit the sixteen vendored files or `docker/HARNESS-SHA256SUMS`. + The harness check stays a sixteen-name check; a separate exact helper check verifies the two tarball assets, their paths, root ownership, modes, source sha256 values and manifest, and image smoke rechecks the installed files and canonical `--describe` output. - `facts/facts.json` is machine-written: "do not hand-edit quantities". "PR code never receives App credentials"; `authz` refuses `pull_request_target`. - Every `uses:` in `.github` is a lowercase 40-hex commit SHA, a local path without a ref, or a sha256-pinned Docker image (`test_release_action_pins.py`); major bumps of release-chain actions are "reviewed and merged by hand", never auto-merged (`.github/dependabot.yml`). Add no `timeout-minutes` to a job (`python-test.yml` says a checker refuses new ones). - `README.md`, `docs/release-chain.md` and `LICENSE` are inputs of the release tests, and several tests pin their exact text (the README pair rule is pinned by SHA-256): change that text only together with its test. diff --git a/README.md b/README.md index 7eb13f0..e9d325d 100644 --- a/README.md +++ b/README.md @@ -402,6 +402,19 @@ and checks their installed bytes against the sums. It also refuses to build when `docker/run_json.py`'s `EXPECTED_PF_ABI` differs from `PF_ABI_VERSION` in the bundled engine headers. +The image also carries two selected-admit files from the same sha256-verified +engine release tarball as the library and headers: +`/opt/pineforge/bin/pineforge-selected-admit` (mode 0755) and +`/opt/pineforge/share/pineforge/pineforge-selected-admit.manifest.json` (mode 0644). +The harness check stays a sixteen-name check. A separate exact helper check +refuses extra entries in `bin/`, `share/` or `share/pineforge/`, requires both +files to be regular, non-symlink files owned by root:root, and compares their +installed sha256 values with a two-file record made from the verified extracted +tarball. It also checks the manifest's helper name, path, hash and size. Image +smoke rechecks the layout, metadata and manifest on each architecture and requires +`pineforge-selected-admit --describe` to exit zero and print the canonical +manifest with only `helper.bytes` and `helper.sha256` removed. + The image includes the sink-to-JSON converter beside `pf_sink.py`. Invoke the module with Python; `/opt/pineforge/bin` is not added to `PATH`, so no command wrappers are installed. Use a retained, completed sink and its captured stdout @@ -453,6 +466,7 @@ with a writable mount to retain compiled artifacts between containers. With no first argument `hpo`, the dispatcher passes every argument unchanged to the engine's existing `/opt/pineforge/bin/entrypoint.sh`. The sixteen engine harness files and their byte and ABI checks remain in place. +The HPO target inherits the same selected-admit helper, manifest and exact checks. For `--target hpo`, `HPO_VERSION` is a required build argument with no default, recorded in the `io.pineforge.hpo.version` label and `PINEFORGE_HPO_VERSION`. diff --git a/docker/Dockerfile b/docker/Dockerfile index 517c0d8..ab95611 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -76,7 +76,11 @@ RUN --mount=type=secret,id=github_token set -eux; \ tar -xzf "${base}.tar.gz"; \ mkdir -p /opt/pineforge; \ cp -r "/tmp/${base}/lib" /opt/pineforge/lib; \ - cp -r "/tmp/${base}/include" /opt/pineforge/include + cp -r "/tmp/${base}/include" /opt/pineforge/include; \ + mkdir -p /opt/pineforge/bin /opt/pineforge/share/pineforge /opt/pineforge-asset-sums; \ + install -m 0755 "/tmp/${base}/bin/pineforge-selected-admit" /opt/pineforge/bin/pineforge-selected-admit; \ + install -m 0644 "/tmp/${base}/share/pineforge/pineforge-selected-admit.manifest.json" /opt/pineforge/share/pineforge/pineforge-selected-admit.manifest.json; \ + ( cd "/tmp/${base}" && sha256sum bin/pineforge-selected-admit share/pineforge/pineforge-selected-admit.manifest.json ) > /opt/pineforge-asset-sums/SHA256SUMS # --- Stage 3: final = toolchain + engine lib + codegen + harness -------------- FROM debian:trixie-slim AS release @@ -163,25 +167,50 @@ COPY docker/run_phase_transport.py /opt/pineforge/bin/run_phase_transport.py COPY docker/run_quota.py /opt/pineforge/bin/run_quota.py COPY docker/selected_window_plan.py /opt/pineforge/bin/selected_window_plan.py COPY docker/selected_window_report.py /opt/pineforge/bin/selected_window_report.py +COPY --from=engine /opt/pineforge/bin/pineforge-selected-admit /opt/pineforge/bin/pineforge-selected-admit +COPY --from=engine /opt/pineforge/share/pineforge/pineforge-selected-admit.manifest.json /opt/pineforge/share/pineforge/pineforge-selected-admit.manifest.json COPY docker/HARNESS-SHA256SUMS /opt/pineforge/bin/HARNESS-SHA256SUMS RUN chmod +x /opt/pineforge/bin/entrypoint.sh /opt/pineforge/bin/run_json.py # Every harness file in the image must be byte for byte the engine's file at the ref that # scripts/sync-harness.sh synced from: the script recorded the sha256 of each fetched file in # HARNESS-SHA256SUMS and the build recomputes them here, on the installed files. The sums file -# must name exactly the sixteen files above, in sorted order, and nothing else may sit in bin/; +# must name exactly the sixteen files above, in sorted order. The helper is checked by the next RUN; # a missing, extra, changed or empty entry fails the build. RUN set -eu; \ cd /opt/pineforge/bin; \ want="bind_compiled_inventory.py entrypoint.sh pf_bars.py pf_sink.py pineforge_report_deadlines.py pineforge_report_supervise.py pineforge_sink_finalize.py pineforge_sink_to_json.py quota_columns.py request_feed_inventory.py run_execution_observer.py run_json.py run_phase_transport.py run_quota.py selected_window_plan.py selected_window_report.py"; \ listed="$(sed 's/^[0-9a-f]\{64\} //' HARNESS-SHA256SUMS | tr '\n' ' ' | sed 's/ $//')"; \ - present="$(LC_ALL=C ls -A | grep -v '^HARNESS-SHA256SUMS$' | tr '\n' ' ' | sed 's/ $//')"; \ + present="$(LC_ALL=C ls -A | grep -v -e '^HARNESS-SHA256SUMS$' -e '^pineforge-selected-admit$' | tr '\n' ' ' | sed 's/ $//')"; \ if [ "$listed" != "$want" ] || [ "$present" != "$want" ]; then \ echo "ERROR: HARNESS-SHA256SUMS lists [${listed}] and bin/ holds [${present}]; both must be exactly [${want}]" >&2; exit 1; \ fi; \ sha256sum -c --strict HARNESS-SHA256SUMS; \ echo "harness: the sixteen files match HARNESS-SHA256SUMS" +# The helper and manifest must match the two files in the verified engine release. +# The asset record is mounted from the engine stage and stays out of the image. +RUN --mount=type=bind,from=engine,source=/opt/pineforge-asset-sums,target=/tmp/pineforge-asset-sums,ro set -eu; \ + cd /opt/pineforge; \ + asset_sums=/tmp/pineforge-asset-sums/SHA256SUMS; \ + want="HARNESS-SHA256SUMS bind_compiled_inventory.py entrypoint.sh pf_bars.py pf_sink.py pineforge-selected-admit pineforge_report_deadlines.py pineforge_report_supervise.py pineforge_sink_finalize.py pineforge_sink_to_json.py quota_columns.py request_feed_inventory.py run_execution_observer.py run_json.py run_phase_transport.py run_quota.py selected_window_plan.py selected_window_report.py"; \ + if [ "$(LC_ALL=C ls -A bin)" != "$(printf '%s\n' $want)" ]; then echo "ERROR: helper bin/ must hold exactly [${want}]" >&2; exit 1; fi; \ + if [ "$(LC_ALL=C ls -A share)" != pineforge ]; then echo "ERROR: helper share/ must hold exactly [pineforge]" >&2; exit 1; fi; \ + if [ "$(LC_ALL=C ls -A share/pineforge)" != pineforge-selected-admit.manifest.json ]; then echo "ERROR: helper share/pineforge/ must hold exactly [pineforge-selected-admit.manifest.json]" >&2; exit 1; fi; \ + for path in bin/pineforge-selected-admit share/pineforge/pineforge-selected-admit.manifest.json; do \ + if ! test -f "$path" || ! test ! -L "$path"; then echo "ERROR: helper file ${path} must be a regular file without symlinks" >&2; exit 1; fi; \ + if [ "$(stat -c %u:%g "$path")" != 0:0 ]; then echo "ERROR: helper file ${path} must be owned by root:root" >&2; exit 1; fi; \ + done; \ + if [ "$(stat -c %a bin/pineforge-selected-admit)" != 755 ]; then echo "ERROR: helper bin/pineforge-selected-admit must have mode 755" >&2; exit 1; fi; \ + if [ "$(stat -c %a share/pineforge/pineforge-selected-admit.manifest.json)" != 644 ]; then echo "ERROR: helper share/pineforge/pineforge-selected-admit.manifest.json must have mode 644" >&2; exit 1; fi; \ + if ! test -f "$asset_sums" || ! test ! -L "$asset_sums"; then echo "ERROR: helper asset record must be a regular file without symlinks" >&2; exit 1; fi; \ + want_sums="$(printf '%s\n' bin/pineforge-selected-admit share/pineforge/pineforge-selected-admit.manifest.json)"; \ + listed="$(sed 's/^[0-9a-f]\{64\} //' "$asset_sums")"; \ + if [ "$listed" != "$want_sums" ] || [ "$(sed -n '$=' "$asset_sums")" != 2 ]; then echo "ERROR: helper asset record must name exactly [bin/pineforge-selected-admit share/pineforge/pineforge-selected-admit.manifest.json]" >&2; exit 1; fi; \ + if ! sha256sum -c --strict "$asset_sums"; then echo "ERROR: helper files differ from the engine release asset" >&2; exit 1; fi; \ + if ! python3 -I -c 'import hashlib,json,pathlib,sys; data=pathlib.Path("bin/pineforge-selected-admit").read_bytes(); manifest=json.loads(pathlib.Path("share/pineforge/pineforge-selected-admit.manifest.json").read_bytes()); helper=manifest["helper"]; helper.get("name")=="pineforge-selected-admit" or sys.exit("ERROR: helper manifest name must be pineforge-selected-admit"); helper.get("path")=="bin/pineforge-selected-admit" or sys.exit("ERROR: helper manifest path must be bin/pineforge-selected-admit"); helper.get("sha256")==hashlib.sha256(data).hexdigest() or sys.exit("ERROR: helper manifest sha256 differs from the helper"); helper.get("bytes")==str(len(data)) or sys.exit("ERROR: helper manifest bytes differs from the helper")'; then echo "ERROR: helper manifest validation failed" >&2; exit 1; fi; \ + echo "helper: the two selected-admit files match the engine release asset" + # The harness (run_json.py) mirrors the engine's C structs; it must be the copy # from the pinned engine tag. Fail the build if its ABI differs from the header # in the bundled tarball (0.1.23/0.1.24 shipped an ABI-2 harness on ABI-3 libs). @@ -416,7 +445,11 @@ COPY --from=hpo-wheel /opt/hpo /opt/pineforge/hpo COPY --from=hpo-wheel /opt/hpo-licenses /opt/pineforge/licenses COPY --from=hpo-native /tmp/hpo-build/bin/pineforge-hpo-native /opt/pineforge/pineforge-hpo-native COPY --from=hpo-native /opt/hpo-native-licenses /opt/pineforge/licenses/pineforge-hpo/THIRD_PARTY_LICENSES -# Hub-owned additions live outside the engine's sixteen-file bin/ directory. +# The inherited image carries sixteen vendored harness files plus the engine helper +# at /opt/pineforge/bin/pineforge-selected-admit and its manifest at +# /opt/pineforge/share/pineforge/pineforge-selected-admit.manifest.json, both from +# the same verified engine release tarball. The harness check stays sixteen-name; +# the helper check is exact. The dispatcher and licence additions live outside bin/. COPY docker/entrypoint-dispatch.sh /opt/pineforge/entrypoint-dispatch.sh COPY docker/LICENSE-SHA256SUMS /opt/pineforge/licenses/LICENSE-SHA256SUMS RUN chmod +x /opt/pineforge/entrypoint-dispatch.sh /opt/pineforge/pineforge-hpo-native diff --git a/scripts/release_smoke.py b/scripts/release_smoke.py index 5c0abee..b83c04f 100644 --- a/scripts/release_smoke.py +++ b/scripts/release_smoke.py @@ -82,6 +82,49 @@ def verify(): ''' +def selected_admit_runtime_script(): + """Verify the helper's exact layout, manifest and canonical description inside the image.""" + return r'''import hashlib,json,pathlib,stat,subprocess,sys +root=pathlib.Path("/opt/pineforge") +def require(condition,message): + if not condition: sys.exit("ERROR: "+message) +def inventory(directory,expected): + try: + present=sorted(path.name for path in (root/directory).iterdir()) + except OSError: + sys.exit("ERROR: helper "+directory+"/ inventory is unavailable") + require(present==expected,"helper "+directory+"/ must hold exactly ["+" ".join(expected)+"]") +inventory("bin","HARNESS-SHA256SUMS bind_compiled_inventory.py entrypoint.sh pf_bars.py pf_sink.py pineforge-selected-admit pineforge_report_deadlines.py pineforge_report_supervise.py pineforge_sink_finalize.py pineforge_sink_to_json.py quota_columns.py request_feed_inventory.py run_execution_observer.py run_json.py run_phase_transport.py run_quota.py selected_window_plan.py selected_window_report.py".split()) +inventory("share",["pineforge"]) +inventory("share/pineforge",["pineforge-selected-admit.manifest.json"]) +for name,mode in (("bin/pineforge-selected-admit",0o755),("share/pineforge/pineforge-selected-admit.manifest.json",0o644)): + path=root/name + require(not path.is_symlink() and path.is_file(),"helper file "+name+" must be a regular file without symlinks") + info=path.stat() + require((info.st_uid,info.st_gid)==(0,0),"helper file "+name+" must be owned by root:root") + require(stat.S_IMODE(info.st_mode)==mode,"helper "+name+" must have mode "+format(mode,"o")) +binary=root/"bin/pineforge-selected-admit" +data=binary.read_bytes() +try: + manifest=json.loads((root/"share/pineforge/pineforge-selected-admit.manifest.json").read_bytes()) + helper=manifest["helper"] + require(isinstance(helper,dict),"helper manifest helper must be an object") +except (OSError,ValueError,KeyError,TypeError): + sys.exit("ERROR: helper manifest validation failed") +require(helper.get("name")=="pineforge-selected-admit","helper manifest name must be pineforge-selected-admit") +require(helper.get("path")=="bin/pineforge-selected-admit","helper manifest path must be bin/pineforge-selected-admit") +require(helper.get("sha256")==hashlib.sha256(data).hexdigest(),"helper manifest sha256 differs from the helper") +require(helper.get("bytes")==str(len(data)),"helper manifest bytes differs from the helper") +del helper["bytes"] +del helper["sha256"] +expected=json.dumps(manifest,ensure_ascii=True,separators=(",",":")).encode("ascii")+b"\n" +described=subprocess.run([str(binary),"--describe"],stdin=subprocess.DEVNULL,capture_output=True,timeout=60) +require(described.returncode==0,"helper --describe must exit zero") +require(described.stdout==expected,"helper --describe differs from the canonical manifest projection") +print("helper: installed files and --describe match the manifest") +''' + + def inspect_index(digest): from release_pipeline import Registry raw = Registry().raw(digest) @@ -145,6 +188,9 @@ def verify_runtime(image, versions, hub_commit, release_version, inventories): if hpo: check += " && cd /opt/pineforge/licenses && sha256sum -c --strict LICENSE-SHA256SUMS" command(*base, "--entrypoint", "bash", ref, "-euc", check) + # The build compares both files with the verified engine tarball. + # Bindings omit those asset hashes; recheck the runtime files and description here. + command(*base, "--entrypoint", "/usr/bin/python3", ref, "-I", "-c", selected_admit_runtime_script()) checked.append((arch, base, ref, hpo, observed)) # Both installed child inventories must verify before either smoke starts. for arch, base, ref, hpo, observed in checked: diff --git a/tests/test_release_asset_fetch.py b/tests/test_release_asset_fetch.py index 4475afd..fec2262 100644 --- a/tests/test_release_asset_fetch.py +++ b/tests/test_release_asset_fetch.py @@ -13,6 +13,7 @@ * https://github.com -> the loopback server's /browser root; * https://api.github.com -> its /api root, if present in the actual RUN; * cd /tmp and the quoted /tmp/ sources -> ASSET_FIXTURE_TMP; +* /opt/pineforge-asset-sums -> ASSET_FIXTURE_RECORD; * /opt/pineforge -> ASSET_FIXTURE_INSTALL. * /tmp/fetch_release_asset.py -> the actual COPY source in this repository; * /run/secrets/github_token -> a file containing only synthetic read-token bytes. @@ -44,6 +45,7 @@ from pathlib import Path import re import shutil +import stat import subprocess import tarfile import tempfile @@ -72,20 +74,30 @@ class _Payload: token: str -def _payload(version: str, arch: str, seed: bytes) -> _Payload: +def _payload(version: str, arch: str, seed: bytes, *, + omit: tuple[str, ...] = ()) -> _Payload: """Produce independent, nonempty archive inputs, not a product-derived hash.""" triple = {"amd64": "linux-x86_64", "arm64": "linux-aarch64"}[arch] base = f"pineforge-v{version}-{triple}" members = { "lib/libpineforge.a": b"fixture static library\0" + seed, "include/pineforge.h": b"/* loopback fixture */\n" + seed.hex().encode() + b"\n", + "bin/pineforge-selected-admit": b"fixture selected-admit helper\0" + seed, + "share/pineforge/pineforge-selected-admit.manifest.json": ( + json.dumps({"fixture_seed": seed.hex()}).encode() + b"\n" + ), } + members = {relative: data for relative, data in members.items() if relative not in omit} output = io.BytesIO() with tarfile.open(fileobj=output, mode="w:gz") as archive: for relative, data in members.items(): entry = tarfile.TarInfo(f"{base}/{relative}") entry.size = len(data) entry.mtime = 0 + if relative == "bin/pineforge-selected-admit": + entry.mode = 0o600 + elif relative == "share/pineforge/pineforge-selected-admit.manifest.json": + entry.mode = 0o666 archive.addfile(entry, io.BytesIO(data)) supplied = output.getvalue() expected = hashlib.sha256(supplied).hexdigest() @@ -350,6 +362,7 @@ def _environment(self, root: Path, payload: _Payload, retries: int, delay: int): "ASSET_FIXTURE_TOKEN_FILE": str(root / "github_token"), "ASSET_FIXTURE_TMP": str(root / "download"), "ASSET_FIXTURE_INSTALL": str(root / "installed"), + "ASSET_FIXTURE_RECORD": str(root / "asset-sums"), } def _run(self, label: str, root: Path, release: _LocalRelease, *, @@ -363,6 +376,7 @@ def _run(self, label: str, root: Path, release: _LocalRelease, *, script = script.replace("/run/secrets/github_token", '"${ASSET_FIXTURE_TOKEN_FILE}"') script = script.replace("cd /tmp;", 'cd "${ASSET_FIXTURE_TMP}";') script = script.replace('"/tmp/', '"${ASSET_FIXTURE_TMP}/') + script = script.replace("/opt/pineforge-asset-sums", '"${ASSET_FIXTURE_RECORD}"') script = script.replace("/opt/pineforge", '"${ASSET_FIXTURE_INSTALL}"') if "/tmp" in script or "/opt/pineforge" in script: raise RuntimeError("fixture setup: unredirected engine destination") @@ -453,11 +467,23 @@ def _assert_bytes_and_consumed(self, root: Path, payload: _Payload, payload.checksum) self.assertTrue(payload.base + ".tar.gz: OK" in observed.completed.stdout, "actual sha256sum did not report the supplied archive as OK") - self.assertEqual(observed.commands.count("sha256sum"), 1) + self.assertEqual(observed.commands.count("sha256sum"), 2) self.assertIn("tar", observed.commands, "actual extraction was not executed") self.assertLess(observed.commands.index("sha256sum"), observed.commands.index("tar")) for relative, supplied in payload.members.items(): self.assertEqual((root / "installed" / relative).read_bytes(), supplied) + helper_modes = ( + ("bin/pineforge-selected-admit", 0o755), + ("share/pineforge/pineforge-selected-admit.manifest.json", 0o644), + ) + for relative, mode in helper_modes: + installed = root / "installed" / relative + self.assertEqual(stat.S_IMODE(installed.stat().st_mode), mode) + expected_record = "".join( + f"{hashlib.sha256(payload.members[relative]).hexdigest()} {relative}\n" + for relative, _ in helper_modes + ).encode("ascii") + self.assertEqual((root / "asset-sums" / "SHA256SUMS").read_bytes(), expected_record) self._assert_known_routes(observed) def test_healthy_fixture_downloads_verifies_and_consumes_supplied_bytes(self): @@ -476,6 +502,57 @@ def test_healthy_fixture_downloads_verifies_and_consumes_supplied_bytes(self): self.assertEqual([request.body_bytes for request in transfers], [len(payload.archive), len(payload.checksum)]) + def test_an_archive_without_either_helper_asset_is_refused(self): + for omitted in ( + "bin/pineforge-selected-admit", + "share/pineforge/pineforge-selected-admit.manifest.json", + ): + with self.subTest(omitted=omitted): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + payload = _payload("1.8.7", "amd64", b"missing helper asset\0") + with _LocalRelease(payload) as release: + observed = self._run("missing-helper-asset-control", root, release, + retries=0, delay=1) + self._assert_bytes_and_consumed(root, payload, observed) + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + payload = _payload("1.8.7", "amd64", b"missing helper asset\0", + omit=(omitted,)) + with _LocalRelease(payload) as release: + observed = self._run("missing-helper-asset", root, release, + retries=0, delay=1) + self._assert_known_routes(observed) + self.assertNotEqual(observed.completed.returncode, 0, observed.proof()) + self.assertIn(payload.base + ".tar.gz: OK", observed.completed.stdout) + self.assertIn("tar", observed.commands) + self.assertIn("sha256sum", observed.commands) + self.assertLess(observed.commands.index("sha256sum"), + observed.commands.index("tar")) + stderr_lines = observed.completed.stderr.splitlines() + source = root / "download" / payload.base / omitted + self.assertIn( + f"install: cannot stat '{source}': No such file or directory", + stderr_lines, + ) + trace_lines = [line for line in stderr_lines if re.match(r"^\++ ", line)] + mode = "0755" if omitted == "bin/pineforge-selected-admit" else "0644" + self.assertEqual( + trace_lines[-1], + f"+ install -m {mode} {source} {root / 'installed' / omitted}", + "execution did not stop at the omitted member's install", + ) + for relative in ("lib/libpineforge.a", "include/pineforge.h"): + self.assertEqual((root / "installed" / relative).read_bytes(), + payload.members[relative]) + if omitted != "bin/pineforge-selected-admit": + self.assertEqual( + (root / "installed" / "bin/pineforge-selected-admit").read_bytes(), + payload.members["bin/pineforge-selected-admit"], + ) + self.assertFalse((root / "asset-sums" / "SHA256SUMS").exists()) + self.assertFalse((root / "installed" / omitted).exists()) + def test_a_real_curl_recovers_after_configured_transient_retries(self): cases = ( ("1.9.2", "amd64", b"two gateway timeouts\0" * 9, diff --git a/tests/test_release_hpo_image.py b/tests/test_release_hpo_image.py index 18be654..332f5f8 100644 --- a/tests/test_release_hpo_image.py +++ b/tests/test_release_hpo_image.py @@ -477,6 +477,50 @@ def _restore_original_doc_cleanup(text: str) -> str: return "".join(lines) +def _without_selected_admit(text: str) -> str: + """Remove only the exact helper additions before checking the historical pin.""" + copies = ( + "COPY --from=engine /opt/pineforge/bin/pineforge-selected-admit /opt/pineforge/bin/pineforge-selected-admit\n" + "COPY --from=engine /opt/pineforge/share/pineforge/pineforge-selected-admit.manifest.json /opt/pineforge/share/pineforge/pineforge-selected-admit.manifest.json\n" + ) + check = r'''# The helper and manifest must match the two files in the verified engine release. +# The asset record is mounted from the engine stage and stays out of the image. +RUN --mount=type=bind,from=engine,source=/opt/pineforge-asset-sums,target=/tmp/pineforge-asset-sums,ro set -eu; \ + cd /opt/pineforge; \ + asset_sums=/tmp/pineforge-asset-sums/SHA256SUMS; \ + want="HARNESS-SHA256SUMS bind_compiled_inventory.py entrypoint.sh pf_bars.py pf_sink.py pineforge-selected-admit pineforge_report_deadlines.py pineforge_report_supervise.py pineforge_sink_finalize.py pineforge_sink_to_json.py quota_columns.py request_feed_inventory.py run_execution_observer.py run_json.py run_phase_transport.py run_quota.py selected_window_plan.py selected_window_report.py"; \ + if [ "$(LC_ALL=C ls -A bin)" != "$(printf '%s\n' $want)" ]; then echo "ERROR: helper bin/ must hold exactly [${want}]" >&2; exit 1; fi; \ + if [ "$(LC_ALL=C ls -A share)" != pineforge ]; then echo "ERROR: helper share/ must hold exactly [pineforge]" >&2; exit 1; fi; \ + if [ "$(LC_ALL=C ls -A share/pineforge)" != pineforge-selected-admit.manifest.json ]; then echo "ERROR: helper share/pineforge/ must hold exactly [pineforge-selected-admit.manifest.json]" >&2; exit 1; fi; \ + for path in bin/pineforge-selected-admit share/pineforge/pineforge-selected-admit.manifest.json; do \ + if ! test -f "$path" || ! test ! -L "$path"; then echo "ERROR: helper file ${path} must be a regular file without symlinks" >&2; exit 1; fi; \ + if [ "$(stat -c %u:%g "$path")" != 0:0 ]; then echo "ERROR: helper file ${path} must be owned by root:root" >&2; exit 1; fi; \ + done; \ + if [ "$(stat -c %a bin/pineforge-selected-admit)" != 755 ]; then echo "ERROR: helper bin/pineforge-selected-admit must have mode 755" >&2; exit 1; fi; \ + if [ "$(stat -c %a share/pineforge/pineforge-selected-admit.manifest.json)" != 644 ]; then echo "ERROR: helper share/pineforge/pineforge-selected-admit.manifest.json must have mode 644" >&2; exit 1; fi; \ + if ! test -f "$asset_sums" || ! test ! -L "$asset_sums"; then echo "ERROR: helper asset record must be a regular file without symlinks" >&2; exit 1; fi; \ + want_sums="$(printf '%s\n' bin/pineforge-selected-admit share/pineforge/pineforge-selected-admit.manifest.json)"; \ + listed="$(sed 's/^[0-9a-f]\{64\} //' "$asset_sums")"; \ + if [ "$listed" != "$want_sums" ] || [ "$(sed -n '$=' "$asset_sums")" != 2 ]; then echo "ERROR: helper asset record must name exactly [bin/pineforge-selected-admit share/pineforge/pineforge-selected-admit.manifest.json]" >&2; exit 1; fi; \ + if ! sha256sum -c --strict "$asset_sums"; then echo "ERROR: helper files differ from the engine release asset" >&2; exit 1; fi; \ + if ! python3 -I -c 'import hashlib,json,pathlib,sys; data=pathlib.Path("bin/pineforge-selected-admit").read_bytes(); manifest=json.loads(pathlib.Path("share/pineforge/pineforge-selected-admit.manifest.json").read_bytes()); helper=manifest["helper"]; helper.get("name")=="pineforge-selected-admit" or sys.exit("ERROR: helper manifest name must be pineforge-selected-admit"); helper.get("path")=="bin/pineforge-selected-admit" or sys.exit("ERROR: helper manifest path must be bin/pineforge-selected-admit"); helper.get("sha256")==hashlib.sha256(data).hexdigest() or sys.exit("ERROR: helper manifest sha256 differs from the helper"); helper.get("bytes")==str(len(data)) or sys.exit("ERROR: helper manifest bytes differs from the helper")'; then echo "ERROR: helper manifest validation failed" >&2; exit 1; fi; \ + echo "helper: the two selected-admit files match the engine release asset"''' + "\n\n" + changes = ( + (copies, ""), (check, ""), + ("# must name exactly the sixteen files above, in sorted order. The helper is checked by the next RUN;\n", + "# must name exactly the sixteen files above, in sorted order, and nothing else may sit in bin/;\n"), + ("grep -v -e '^HARNESS-SHA256SUMS$' -e '^pineforge-selected-admit$'", + "grep -v '^HARNESS-SHA256SUMS$'"), + ) + for current, original in changes: + if text.count(current) != 1: + raise ValueError("expected exactly one approved selected-admit runtime fragment") + text = text.replace(current, original, 1) + if "pineforge-selected-admit" in text: + raise ValueError("expected exactly one approved selected-admit runtime fragment") + return text + + def _restore_original_harness_manifest(text: str) -> str: """Undo exactly the harness list additions for the original runtime pin.""" changes = ( @@ -913,13 +957,14 @@ def problems(text, target=None): def test_release_stage_matches_original_runtime_after_notice_and_harness_changes(self): text = _without_guard_policy(_without_sources_notice(_without_stage_alias(self.named["release"]))) text = _without_parity_carrier(text) + text = _without_selected_admit(text) text = _restore_bookworm_runtime_base(text) text = _restore_original_doc_cleanup(text) text = _restore_original_harness_manifest(text) self.assertEqual(hashlib.sha256(text.encode("utf-8")).hexdigest(), self.ORIGINAL_RUNTIME_STAGE_SHA256, "after restoring the one /usr/share/doc token and removing the marked " - "notice, exact trixie FROM line and approved harness and parity changes, release must remain " + "notice, exact trixie FROM line and approved harness, helper and parity changes, release must remain " "byte for byte the original runtime stage") def test_runtime_base_normalization_refuses_other_from_changes(self): @@ -936,8 +981,31 @@ def test_runtime_base_normalization_refuses_other_from_changes(self): with self.assertRaisesRegex(ValueError, "approved trixie runtime FROM line"): _restore_bookworm_runtime_base(changed) + def test_helper_normalization_refuses_changed_or_duplicated_runtime_fragments(self): + text = _without_stage_alias(self.named["release"]) + stripped = _without_selected_admit(text) + self.assertNotIn("pineforge-selected-admit", stripped) + helper_copy = ("COPY --from=engine /opt/pineforge/bin/pineforge-selected-admit " + "/opt/pineforge/bin/pineforge-selected-admit\n") + for anchor, replacement in ( + (helper_copy, ""), + (helper_copy, helper_copy * 2), + (helper_copy, helper_copy.replace("--from=engine", "--from=codegen")), + ("target=/tmp/pineforge-asset-sums,ro", "target=/tmp/pineforge-asset-sums"), + ('if ! sha256sum -c --strict "$asset_sums";', "if ! true;"), + ('"$(stat -c %u:%g "$path")" != 0:0', '"$(stat -c %u:%g "$path")" != 1:1'), + ("bin/pineforge-selected-admit)\" != 755", "bin/pineforge-selected-admit)\" != 777"), + (' echo "helper: the two selected-admit files match the engine release asset"\n', + ' echo "helper: the two selected-admit files match the engine release asset"; true\n')): + with self.subTest(fragment=anchor): + self.assertEqual(text.count(anchor), 1) + altered = text.replace(anchor, replacement, 1) + self.assertNotEqual(altered, text) + with self.assertRaisesRegex(ValueError, "approved selected-admit runtime fragment"): + _without_selected_admit(altered) + def test_original_harness_normalization_requires_all_added_modules(self): - text = _without_sources_notice(_without_stage_alias(self.named["release"])) + text = _without_selected_admit(_without_sources_notice(_without_stage_alias(self.named["release"]))) for name, copy_line in ( ("pf_bars.py", "COPY docker/pf_bars.py /opt/pineforge/bin/pf_bars.py\n"), ("pf_sink.py", "COPY docker/pf_sink.py /opt/pineforge/bin/pf_sink.py\n"), @@ -981,6 +1049,7 @@ def test_notice_and_harness_changes_cannot_hide_other_instructions(self): self.assertNotEqual(text, stripped) stripped = _without_guard_policy(stripped) stripped = _without_parity_carrier(stripped) + stripped = _without_selected_admit(stripped) stripped = _restore_bookworm_runtime_base(stripped) stripped = _restore_original_doc_cleanup(stripped) stripped = _restore_original_harness_manifest(stripped) @@ -1015,7 +1084,8 @@ def test_notice_and_harness_changes_cannot_hide_other_instructions(self): self.assertNotEqual(altered, text) normalized = _restore_original_harness_manifest( _restore_original_doc_cleanup(_restore_bookworm_runtime_base( - _without_parity_carrier(_without_guard_policy(_without_sources_notice(altered)))))) + _without_selected_admit( + _without_parity_carrier(_without_guard_policy(_without_sources_notice(altered))))))) self.assertNotEqual(hashlib.sha256(normalized.encode("utf-8")).hexdigest(), self.ORIGINAL_RUNTIME_STAGE_SHA256) # Extra commands or instructions inside the carrier must be refused, @@ -1348,7 +1418,7 @@ class DockerfileHarnessTest(unittest.TestCase): cd /opt/pineforge/bin; \ want="bind_compiled_inventory.py entrypoint.sh pf_bars.py pf_sink.py pineforge_report_deadlines.py pineforge_report_supervise.py pineforge_sink_finalize.py pineforge_sink_to_json.py quota_columns.py request_feed_inventory.py run_execution_observer.py run_json.py run_phase_transport.py run_quota.py selected_window_plan.py selected_window_report.py"; \ listed="$(sed 's/^[0-9a-f]\{64\} //' HARNESS-SHA256SUMS | tr '\n' ' ' | sed 's/ $//')"; \ - present="$(LC_ALL=C ls -A | grep -v '^HARNESS-SHA256SUMS$' | tr '\n' ' ' | sed 's/ $//')"; \ + present="$(LC_ALL=C ls -A | grep -v -e '^HARNESS-SHA256SUMS$' -e '^pineforge-selected-admit$' | tr '\n' ' ' | sed 's/ $//')"; \ if [ "$listed" != "$want" ] || [ "$present" != "$want" ]; then \ echo "ERROR: HARNESS-SHA256SUMS lists [${listed}] and bin/ holds [${present}]; both must be exactly [${want}]" >&2; exit 1; \ fi; \ diff --git a/tests/test_release_runtime_smoke.py b/tests/test_release_runtime_smoke.py index a0ebfae..77a0c65 100644 --- a/tests/test_release_runtime_smoke.py +++ b/tests/test_release_runtime_smoke.py @@ -1,7 +1,12 @@ #!/usr/bin/env python3 """CI/cloud checks for nonempty image smoke results and per-child execution.""" import copy +import hashlib +import json +import os +import subprocess import sys +import tempfile import unittest from pathlib import Path from unittest.mock import patch @@ -11,6 +16,7 @@ sys.path.insert(0, str(ROOT / "tests")) import release_contract as contract import release_smoke as smoke +import test_release_workflows as workflow_tests from test_release_binding import fixture_manifest @@ -47,6 +53,19 @@ def test_both_child_digests_run_offline_real_fixture_commands(self): self.assertEqual(proof[arch]["hpo_trials"], 9) ref = contract.IMAGE + "@" + self.m["image"][arch + "_digest"] self.assertTrue(any(ref in call and "linux/" + arch in call and "hpo" in call for call in self.calls)) + helper_calls = [call for call in self.calls + if ref in call and "--entrypoint" in call + and call[call.index("--entrypoint") + 1] == "/usr/bin/python3"] + self.assertEqual(len(helper_calls), 1, "each child must run the helper runtime check") + self.assertEqual(helper_calls[0][-3:-1], ("-I", "-c")) + self.assertEqual(helper_calls[0][-1], smoke.selected_admit_runtime_script()) + harness_check = next(index for index, call in enumerate(self.calls) + if ref in call and "--entrypoint" in call + and call[call.index("--entrypoint") + 1] == "bash") + self.assertLess(harness_check, self.calls.index(helper_calls[0])) + backtest = next(index for index, call in enumerate(self.calls) + if ref in call and call[:2] == ("docker", "run") and "--entrypoint" not in call) + self.assertLess(self.calls.index(helper_calls[0]), backtest) for call in self.calls: if call[:2] == ("docker", "run"): self.assertIn("--network=none", call) @@ -75,5 +94,128 @@ def test_index_requires_unique_distinct_architectures_and_matching_original_byte smoke.inspect_index("sha256:" + contract.sha256(raw)) +class HelperRuntimeSmokeTests(unittest.TestCase): + """Execute the same in-container Python source on an isolated helper layout.""" + + SUCCESS = "helper: installed files and --describe match the manifest" + + def _run(self, prefix: Path) -> subprocess.CompletedProcess: + self.assertTrue(hasattr(smoke, "selected_admit_runtime_script"), + "release smoke must provide the selected-admit runtime check") + source = smoke.selected_admit_runtime_script() + self.assertEqual(source.count('"/opt/pineforge"'), 1) + source = source.replace('"/opt/pineforge"', repr(str(prefix))) + # Root ownership cannot be created or changed by ordinary test users. + # Normalize only fixture ownership after a real stat; keep file types, + # modes and bytes real. The actual image command has no such adapter. + # This exact path list refuses every protected path and all env files. + adapter = ( + "import os,pathlib,sys\n" + f"fixture_paths={[str(prefix / name) for name in workflow_tests.SELECTED_ADMIT_PATHS]!r}\n" + f"fixture_owner={(os.geteuid(), os.getegid())!r}\n" + "real_stat=pathlib.Path.stat\n" + "def fixture_stat(path,*args,**kwargs):\n" + " if str(path) not in fixture_paths: sys.exit('refused fixture stat path: '+str(path))\n" + " print('stat follows: '+str(path),file=sys.stderr,flush=True)\n" + " info=real_stat(path,*args,**kwargs)\n" + " if (info.st_uid,info.st_gid)==fixture_owner:\n" + " values=list(info)\n" + " values[4]=values[5]=0\n" + " return os.stat_result(values)\n" + " return info\n" + "pathlib.Path.stat=fixture_stat\n" + ) + return subprocess.run([sys.executable, "-I", "-c", adapter + source], + cwd=prefix.parent, env={"PATH": os.environ.get("PATH", os.defpath)}, + stdin=subprocess.DEVNULL, capture_output=True, text=True, timeout=60) + + def _passes(self, prefix: Path) -> None: + done = self._run(prefix) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertEqual(done.stdout.splitlines(), [self.SUCCESS]) + for name in workflow_tests.SELECTED_ADMIT_PATHS: + self.assertIn("stat follows: " + str(prefix / name), done.stderr.splitlines()) + + def _fails(self, prefix: Path, message: str) -> None: + done = self._run(prefix) + self.assertNotEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn("ERROR: " + message, done.stderr.splitlines()) + self.assertNotIn(self.SUCCESS, done.stdout) + + def _bind_helper_bytes(self, prefix: Path) -> None: + helper, path = (prefix / name for name in workflow_tests.SELECTED_ADMIT_PATHS) + manifest = json.loads(path.read_bytes()) + manifest["helper"]["bytes"] = str(helper.stat().st_size) + manifest["helper"]["sha256"] = hashlib.sha256(helper.read_bytes()).hexdigest() + path.write_text(json.dumps(manifest, ensure_ascii=True, separators=(",", ":")) + "\n", + encoding="ascii") + + def test_a_one_byte_description_difference_is_refused(self): + with tempfile.TemporaryDirectory() as tmp: + prefix, _ = workflow_tests.selected_admit_layout(Path(tmp)) + self._passes(prefix) + helper = prefix / workflow_tests.SELECTED_ADMIT_PATHS[0] + before = helper.read_bytes() + after = before.replace(b'"version_full":"1.0.0"', b'"version_full":"1.0.1"', 1) + self.assertEqual(len(before), len(after)) + self.assertEqual(sum(a != b for a, b in zip(before, after)), 1) + helper.write_bytes(after) + self._bind_helper_bytes(prefix) + self._fails(prefix, "helper --describe differs from the canonical manifest projection") + + def test_a_helper_that_exits_nonzero_is_refused(self): + with tempfile.TemporaryDirectory() as tmp: + prefix, _ = workflow_tests.selected_admit_layout(Path(tmp)) + self._passes(prefix) + (prefix / workflow_tests.SELECTED_ADMIT_PATHS[0]).write_bytes(b"#!/bin/sh\nexit 7\n") + self._bind_helper_bytes(prefix) + self._fails(prefix, "helper --describe must exit zero") + + def test_an_incorrect_manifest_helper_hash_is_refused(self): + with tempfile.TemporaryDirectory() as tmp: + prefix, _ = workflow_tests.selected_admit_layout(Path(tmp)) + self._passes(prefix) + path = prefix / workflow_tests.SELECTED_ADMIT_PATHS[1] + manifest = json.loads(path.read_bytes()) + manifest["helper"]["sha256"] = "0" * 64 + path.write_text(json.dumps(manifest, ensure_ascii=True, separators=(",", ":")) + "\n", + encoding="ascii") + self._fails(prefix, "helper manifest sha256 differs from the helper") + + def test_runtime_rejects_extra_files_and_directories_in_each_exact_inventory(self): + for name, directory, message in ( + ("bin/pineforge-selected-admit.bak", False, workflow_tests.HelperBuildCheckTest.BIN_ERROR), + ("bin/.x", False, workflow_tests.HelperBuildCheckTest.BIN_ERROR), + ("share/another-file", False, workflow_tests.HelperBuildCheckTest.SHARE_ERROR), + ("share/another-directory", True, workflow_tests.HelperBuildCheckTest.SHARE_ERROR), + ("share/pineforge/another-file", False, workflow_tests.HelperBuildCheckTest.MANIFEST_ERROR)): + with self.subTest(path=name), tempfile.TemporaryDirectory() as tmp: + prefix, _ = workflow_tests.selected_admit_layout(Path(tmp)) + self._passes(prefix) + path = prefix / name + if directory: + path.mkdir() + else: + path.write_bytes(b"extra\n") + self._fails(prefix, message.removeprefix("ERROR: ")) + + def test_runtime_rejects_wrong_modes_and_symlinks_for_both_assets(self): + for name, mode in zip(workflow_tests.SELECTED_ADMIT_PATHS, (0o755, 0o644)): + for fault in ("mode", "symlink"): + with self.subTest(asset=name, fault=fault), tempfile.TemporaryDirectory() as tmp: + prefix, _ = workflow_tests.selected_admit_layout(Path(tmp)) + self._passes(prefix) + path = prefix / name + if fault == "mode": + path.chmod(0o777) + message = "helper " + name + " must have mode " + format(mode, "o") + else: + target = Path(tmp) / "symlink-target" + path.rename(target) + path.symlink_to(target) + message = "helper file " + name + " must be a regular file without symlinks" + self._fails(prefix, message) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_release_workflows.py b/tests/test_release_workflows.py index d1277b9..f72d9c0 100644 --- a/tests/test_release_workflows.py +++ b/tests/test_release_workflows.py @@ -2540,6 +2540,7 @@ def test_the_build_checks_the_installed_files_against_the_recorded_sums(self): self.assertTrue(block.startswith("RUN set -eu; \\\n"), block[:40]) self.assertIn("cd /opt/pineforge/bin; \\\n", block) self.assertIn('want="' + " ".join(self.HARNESS) + '"; \\\n', block) + self.assertIn("grep -v -e '^HARNESS-SHA256SUMS$' -e '^pineforge-selected-admit$'", block) self.assertIn('[ "$listed" != "$want" ] || [ "$present" != "$want" ]', block) self.assertIn("exit 1;", block) self.assertNotIn("|| true", block) @@ -2653,14 +2654,16 @@ def _layout(self, root: Path) -> Path: bin_dir.mkdir() for name in self.HARNESS + (self.SUMS,): shutil.copyfile(REPO / "docker" / name, bin_dir / name) + (bin_dir / "pineforge-selected-admit").write_bytes(b"#!/bin/sh\nexit 0\n") return bin_dir def _names(self, bin_dir: Path) -> tuple[list[str], list[str]]: """The two name lists the check compares, read off the layout: the names in its sums file, in the - file's order, and the names in its directory, in C order, without the sums file.""" + file's order, and the names in its directory, in C order, without the sums file or helper.""" lines = (bin_dir / self.SUMS).read_bytes().decode("utf-8").splitlines() listed = [line.split(" ", 1)[1] for line in lines] - present = sorted(entry.name for entry in bin_dir.iterdir() if entry.name != self.SUMS) + present = sorted(entry.name for entry in bin_dir.iterdir() + if entry.name not in (self.SUMS, "pineforge-selected-admit")) return listed, present def _edit_sums(self, bin_dir: Path, edit) -> None: @@ -2767,6 +2770,333 @@ def wrong_hash(entries): self._fails_on_hash(bin_dir, "run_json.py") +SELECTED_ADMIT_PATHS = ( + "bin/pineforge-selected-admit", + "share/pineforge/pineforge-selected-admit.manifest.json", +) + + +def selected_admit_asset_sums(prefix: Path, sums: Path) -> None: + sums.write_text("".join(hashlib.sha256((prefix / name).read_bytes()).hexdigest() + + " " + name + "\n" for name in SELECTED_ADMIT_PATHS), + encoding="ascii") + + +def selected_admit_layout(root: Path) -> tuple[Path, Path]: + """A complete harness and two helper assets, with an ordered manifest in the engine format.""" + prefix = root / "prefix" + (prefix / "bin").mkdir(parents=True) + (prefix / "share/pineforge").mkdir(parents=True) + for name in HarnessSyncTest.HARNESS + (HarnessSyncTest.SUMS,): + shutil.copyfile(REPO / "docker" / name, prefix / "bin" / name) + projection = { + "schema": "pineforge-selected-capacity-helper-manifest/v1", + "helper": {"name": "pineforge-selected-admit", "path": SELECTED_ADMIT_PATHS[0]}, + "engine": {"version_full": "1.0.0", "version": "1.0.0", "git_sha": "0" * 40, "dirty": False}, + "descriptor": {"request_schema": "pineforge-selected-capacity-request/v1", + "result_schema": "pineforge-selected-capacity-result/v1", "spec_sha256": "1" * 64}, + "failure_catalog": {"codes": ["selected_bytes_limit_exceeded"], "catalog_sha256": "2" * 64}, + "stages": ["source", "selected"], + } + line = json.dumps(projection, ensure_ascii=True, separators=(",", ":")) + helper = prefix / SELECTED_ADMIT_PATHS[0] + helper.write_text("#!/bin/sh\n" + 'if [ "$#" -ne 1 ] || [ "$1" != --describe ]; then exit 2; fi\n' + "printf '%s\\n' " + shlex.quote(line) + "\n", encoding="ascii") + helper.chmod(0o755) + manifest = copy.deepcopy(projection) + manifest["helper"]["bytes"] = str(helper.stat().st_size) + manifest["helper"]["sha256"] = hashlib.sha256(helper.read_bytes()).hexdigest() + installed_manifest = prefix / SELECTED_ADMIT_PATHS[1] + installed_manifest.write_text(json.dumps(manifest, ensure_ascii=True, separators=(",", ":")) + "\n", + encoding="ascii") + installed_manifest.chmod(0o644) + sums = root / "asset-sums" / "SHA256SUMS" + sums.parent.mkdir() + selected_admit_asset_sums(prefix, sums) + return prefix, sums + + +class HelperBuildCheckTest(unittest.TestCase): + """Run the real mounted helper RUN; each defect follows a passing control.""" + + CHECK = "helper: the two selected-admit files match the engine release asset" + MOUNT = ("--mount=type=bind,from=engine,source=/opt/pineforge-asset-sums," + "target=/tmp/pineforge-asset-sums,ro ") + BIN_NAMES = tuple(sorted(HarnessSyncTest.HARNESS + + (HarnessSyncTest.SUMS, "pineforge-selected-admit"))) + BIN_ERROR = "ERROR: helper bin/ must hold exactly [" + " ".join(BIN_NAMES) + "]" + SHARE_ERROR = "ERROR: helper share/ must hold exactly [pineforge]" + MANIFEST_ERROR = ("ERROR: helper share/pineforge/ must hold exactly " + "[pineforge-selected-admit.manifest.json]") + RECORD_ERROR = ("ERROR: helper asset record must name exactly " + "[bin/pineforge-selected-admit share/pineforge/pineforge-selected-admit.manifest.json]") + HASH_ERROR = "ERROR: helper files differ from the engine release asset" + + @classmethod + def setUpClass(cls): + cls.dockerfile = (REPO / "docker/Dockerfile").read_text(encoding="utf-8") + tools = ("bash", "ls", "sed", "sha256sum", "stat", "python3") + found = {tool: shutil.which(tool) for tool in tools} + missing = [tool for tool in tools if found[tool] is None] + if missing: + raise AssertionError("the Dockerfile's helper check requires " + ", ".join(missing)) + cls.bash, cls.stat, cls.python = (found[name] for name in ("bash", "stat", "python3")) + cls.path = os.environ.get("PATH", os.defpath) + + @classmethod + def _script(cls, dockerfile: str, prefix: Path, sums: Path) -> str: + if cls.CHECK not in dockerfile: + raise AssertionError("docker/Dockerfile holds no RUN for the two helper release assets") + block = _run_block(dockerfile, cls.CHECK) + leading = "RUN " + cls.MOUNT + if not block.startswith(leading): + raise AssertionError("the helper RUN must mount the engine asset record read-only") + joined = block[len(leading):].replace("\\\n", "") + if "\n" in joined: + raise AssertionError("the helper RUN must be one complete logical line") + for image_path in ("/opt/pineforge", "/tmp/pineforge-asset-sums"): + if joined.count(image_path) != 1: + raise AssertionError("the helper RUN must name its fixed root exactly once: " + image_path) + return (joined.replace("/opt/pineforge", shlex.quote(str(prefix))) + .replace("/tmp/pineforge-asset-sums", shlex.quote(str(sums.parent)))) + + def _run(self, prefix: Path, sums: Path) -> subprocess.CompletedProcess: + script = self._script(self.dockerfile, prefix, sums) + tools = prefix.parent / "tools" + tools.mkdir(exist_ok=True) + record = prefix.parent / "stat-calls" + record.write_text("", encoding="ascii") + # A different real owner cannot be arranged without root. Normalize + # only this fixture owner's uid:gid, after evaluating GNU stat on the + # actual file. The real build and image smoke enforce root ownership. + # Exact relative names also refuse protected paths and all env files. + adapter = ( + f"#!{self.python} -I\n" + "import pathlib,subprocess,sys\n" + f"allowed={SELECTED_ADMIT_PATHS!r}\n" + "args=sys.argv[1:]\n" + "if len(args)!=3 or args[0]!='-c' or args[1] not in ('%u:%g','%a') or args[2] not in allowed:\n" + " sys.exit('refused stat fixture path: '+repr(args))\n" + f"if pathlib.Path.cwd()!=pathlib.Path({str(prefix)!r}): sys.exit('refused stat fixture directory')\n" + "print('stat follows: '+args[2],file=sys.stderr,flush=True)\n" + f"done=subprocess.run([{self.stat!r},*args],capture_output=True)\n" + "sys.stderr.buffer.write(done.stderr)\n" + "if done.returncode: sys.exit(done.returncode)\n" + "value=done.stdout.decode('ascii').strip()\n" + f"with pathlib.Path({str(record)!r}).open('a',encoding='ascii') as log:\n" + " log.write(args[1]+'\\t'+args[2]+'\\t'+value+'\\n')\n" + f"if args[1]=='%u:%g' and value=={f'{os.geteuid()}:{os.getegid()}'!r}: value='0:0'\n" + "print(value)\n" + ) + (tools / "stat").write_text(adapter, encoding="utf-8") + (tools / "stat").chmod(0o755) + return subprocess.run([self.bash, "--noprofile", "--norc", "-c", script], + cwd=prefix.parent, env={"PATH": str(tools) + os.pathsep + self.path}, + stdin=subprocess.DEVNULL, capture_output=True, text=True, timeout=60) + + def _passes(self, prefix: Path, sums: Path) -> None: + done = self._run(prefix, sums) + self.assertEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertEqual(done.stdout.splitlines(), + [name + ": OK" for name in SELECTED_ADMIT_PATHS] + [self.CHECK]) + calls = [line.split("\t") for line in (prefix.parent / "stat-calls").read_text().splitlines()] + self.assertEqual([(fmt, name) for fmt, name, _ in calls], + [("%u:%g", name) for name in SELECTED_ADMIT_PATHS] + + [("%a", name) for name in SELECTED_ADMIT_PATHS]) + self.assertEqual([value for fmt, _, value in calls if fmt == "%u:%g"], + [f"{os.geteuid()}:{os.getegid()}"] * 2) + + def _fails(self, prefix: Path, sums: Path, message: str) -> subprocess.CompletedProcess: + done = self._run(prefix, sums) + self.assertNotEqual(done.returncode, 0, done.stdout + done.stderr) + self.assertIn(message, done.stderr.splitlines()) + self.assertNotIn(self.CHECK, done.stdout) + return done + + def test_the_two_assets_pass_and_both_ownership_expressions_are_evaluated(self): + with tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + self.assertIn('if [ "$(stat -c %u:%g "$path")" != 0:0 ]', self.dockerfile) + + def test_a_missing_helper_is_refused(self): + with tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + (prefix / SELECTED_ADMIT_PATHS[0]).rename(Path(tmp) / "removed-helper") + self._fails(prefix, sums, self.BIN_ERROR) + + def test_a_missing_manifest_is_refused(self): + with tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + (prefix / SELECTED_ADMIT_PATHS[1]).rename(Path(tmp) / "removed-manifest") + self._fails(prefix, sums, self.MANIFEST_ERROR) + + def test_changed_asset_bytes_are_refused_against_the_source_record(self): + for name in SELECTED_ADMIT_PATHS: + with self.subTest(asset=name), tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + path = prefix / name + path.write_bytes(path.read_bytes() + b"\n") + done = self._fails(prefix, sums, self.HASH_ERROR) + self.assertIn(name + ": FAILED", done.stdout.splitlines()) + + def test_a_third_name_in_the_source_record_is_refused(self): + with tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + digest = hashlib.sha256((prefix / SELECTED_ADMIT_PATHS[0]).read_bytes()).hexdigest() + sums.write_text(sums.read_text() + digest + " bin/another-file\n", encoding="ascii") + self._fails(prefix, sums, self.RECORD_ERROR) + + def test_either_missing_source_record_name_is_refused(self): + for missing in range(2): + with self.subTest(missing=SELECTED_ADMIT_PATHS[missing]), tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + lines = sums.read_text().splitlines(keepends=True) + sums.write_text("".join(line for index, line in enumerate(lines) if index != missing), + encoding="ascii") + self._fails(prefix, sums, self.RECORD_ERROR) + + def test_helper_modes_other_than_755_are_refused(self): + for mode in (0o644, 0o775, 0o777): + with self.subTest(mode=oct(mode)), tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + (prefix / SELECTED_ADMIT_PATHS[0]).chmod(mode) + self._fails(prefix, sums, "ERROR: helper bin/pineforge-selected-admit must have mode 755") + + def test_manifest_modes_other_than_644_are_refused(self): + for mode in (0o755, 0o664): + with self.subTest(mode=oct(mode)), tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + (prefix / SELECTED_ADMIT_PATHS[1]).chmod(mode) + self._fails(prefix, sums, "ERROR: helper " + SELECTED_ADMIT_PATHS[1] + " must have mode 644") + + def test_either_asset_as_a_symlink_is_refused(self): + for name in SELECTED_ADMIT_PATHS: + with self.subTest(asset=name), tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + path = prefix / name + target = Path(tmp) / "symlink-target" + path.rename(target) + path.symlink_to(target) + self._fails(prefix, sums, + "ERROR: helper file " + name + " must be a regular file without symlinks") + + def test_the_helper_only_at_another_path_is_refused(self): + for wrong in ("sbin/pineforge-selected-admit", "bin/sub/pineforge-selected-admit", + "pineforge-selected-admit"): + with self.subTest(path=wrong), tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + destination = prefix / wrong + destination.parent.mkdir(parents=True, exist_ok=True) + (prefix / SELECTED_ADMIT_PATHS[0]).rename(destination) + self._fails(prefix, sums, self.BIN_ERROR) + + def test_a_third_bin_file_is_refused_for_unrelated_backup_and_hidden_names(self): + for name in ("another-file", "pineforge-selected-admit.bak", ".x"): + with self.subTest(name=name), tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + (prefix / "bin" / name).write_bytes(b"extra\n") + self._fails(prefix, sums, self.BIN_ERROR) + + def test_the_manifest_only_at_another_path_is_refused(self): + for wrong, message in ( + ("share/pineforge-selected-admit.manifest.json", self.SHARE_ERROR), + ("share/pineforge/sub/pineforge-selected-admit.manifest.json", self.MANIFEST_ERROR), + ("bin/pineforge-selected-admit.manifest.json", self.BIN_ERROR), + ("pineforge-selected-admit.manifest.json", self.MANIFEST_ERROR)): + with self.subTest(path=wrong), tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + destination = prefix / wrong + destination.parent.mkdir(parents=True, exist_ok=True) + (prefix / SELECTED_ADMIT_PATHS[1]).rename(destination) + self._fails(prefix, sums, message) + + def test_an_extra_file_beside_the_manifest_is_refused(self): + with tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + (prefix / "share/pineforge/another-file").write_bytes(b"extra\n") + self._fails(prefix, sums, self.MANIFEST_ERROR) + + def test_an_extra_directory_under_share_is_refused(self): + with tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + (prefix / "share/another-directory").mkdir() + self._fails(prefix, sums, self.SHARE_ERROR) + + def test_an_extra_file_directly_under_share_is_refused(self): + with tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + (prefix / "share/another-file").write_bytes(b"extra\n") + self._fails(prefix, sums, self.SHARE_ERROR) + + def test_manifest_fields_must_identify_the_installed_helper(self): + for field, value, message in ( + ("sha256", "0" * 64, "ERROR: helper manifest sha256 differs from the helper"), + ("path", "sbin/pineforge-selected-admit", + "ERROR: helper manifest path must be bin/pineforge-selected-admit"), + ("name", "another-helper", "ERROR: helper manifest name must be pineforge-selected-admit"), + ("bytes", 1, "ERROR: helper manifest bytes differs from the helper")): + with self.subTest(field=field), tempfile.TemporaryDirectory() as tmp: + prefix, sums = selected_admit_layout(Path(tmp)) + self._passes(prefix, sums) + path = prefix / SELECTED_ADMIT_PATHS[1] + manifest = json.loads(path.read_bytes()) + manifest["helper"][field] = value + path.write_text(json.dumps(manifest, ensure_ascii=True, separators=(",", ":")) + "\n", + encoding="ascii") + # Model a source manifest with this one semantic defect, so + # source-byte equality passes before manifest validation runs. + selected_admit_asset_sums(prefix, sums) + self._fails(prefix, sums, message) + + +class HelperAssetSourceTest(unittest.TestCase): + def test_exact_asset_installs_copies_and_record_come_from_the_verified_tree(self): + dockerfile = (REPO / "docker/Dockerfile").read_text(encoding="utf-8") + engine = re.search(r"^FROM \S+ AS engine\n.*?(?=^FROM |\Z)", dockerfile, re.M | re.S).group(0) + extraction = _run_block(engine, 'sha256sum -c "${base}.tar.gz.sha256"') + self.assertIn("set -eux;", extraction) + self.assertLess(extraction.index('sha256sum -c "${base}.tar.gz.sha256"'), + extraction.index('tar -xzf "${base}.tar.gz"')) + copies = [line for line in dockerfile.splitlines() + if line.startswith("COPY ") and "pineforge-selected-admit" in line] + self.assertEqual(copies, ["COPY --from=engine /opt/pineforge/" + name + " /opt/pineforge/" + name + for name in SELECTED_ADMIT_PATHS]) + for name, mode in zip(SELECTED_ADMIT_PATHS, ("0755", "0644")): + install = f'install -m {mode} "/tmp/${{base}}/{name}" /opt/pineforge/{name};' + self.assertIn(install, extraction) + self.assertLess(extraction.index('tar -xzf "${base}.tar.gz"'), extraction.index(install)) + record = ('( cd "/tmp/${base}" && sha256sum ' + " ".join(SELECTED_ADMIT_PATHS) + + " ) > /opt/pineforge-asset-sums/SHA256SUMS") + self.assertIn(record, extraction) + check = _run_block(dockerfile, HelperBuildCheckTest.CHECK) + self.assertTrue(check.startswith("RUN " + HelperBuildCheckTest.MOUNT)) + self.assertIn('want="' + " ".join(HelperBuildCheckTest.BIN_NAMES) + '";', check) + allowlists = [line for line in check.splitlines() + if line.strip().startswith(("want=", "want_sums=", "for path in "))] + for line in copies + allowlists: + with self.subTest(line=line): + self.assertFalse(any(char in line for char in "*?["), line) + self.assertNotIn("pineforge-asset-sums", "\n".join( + line for line in dockerfile.splitlines() if line.startswith("COPY "))) + self.assertEqual(check.count('sha256sum -c --strict "$asset_sums"'), 1) + + class PythonTestGateTest(unittest.TestCase): def test_release_rules_run_in_their_own_job(self): text = (WORKFLOWS / "python-test.yml").read_text(encoding="utf-8")