Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 61 additions & 2 deletions docs/release-chain.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,28 @@ pending and creates/updates a neutral incident. All other failures are errors.
through its existing evidence/PR process; this hub never fabricates evidence or
receives private registry credentials.

For a candidate without a stored manifest, finalization stages the original
outer npm tarball, codegen PyPI sdist and both Linux engine tarballs. Filenames
and versions come from the candidate pins and selected producer URLs. It checks
the npm registry's mandatory SHA-512 integrity, PyPI's version-specific sdist
SHA-256 and the candidate's sdist digest, and each engine archive's mandatory
companion `.sha256` with its filename binding. Refetched producer commits must
equal the candidate. Any supplied GitHub asset digest must also agree.
All producer verification and facts binding finish before the first upload.
Public npm/PyPI downloads remain anonymous and use the existing fixed origins;
the hub token is used only for the authenticated GitHub API and upload origins.

The separate binary writer attaches those four compressed files without
unpacking/repacking them. It reads each upload back and checks its size and
SHA-256, including when GitHub supplies no digest. Identical existing bytes are
reused; conflicting bytes are never overwritten. A partial upload leaves the
draft and its attachments for retry, with no manifest or authorized promotion.
If an interrupted upload leaves an attachment listed in any state other than
`uploaded`, the writer refuses it as invalid hub asset metadata and finalization
fails closed: no manifest, promotion or delivery, and nothing is overwritten or
deleted. Later runs fail the same way until an operator removes that attachment
from the draft.
Missing producer checksums stay pending; corruption fails finalization.
Only then does finalization attach `release-manifest.json`, assign the fixed
version image tag, move eligible aliases, publish the GitHub release, and fan
out. Alias decisions and delivery reservations share job-level `release-finalize`
Expand All @@ -160,7 +182,7 @@ The previous alias digests are preserved before the first promotion.
## Exact ready event contract

The `offline`, `hosted`, `application`, `web`, `data` and `license` event type remains
**`pineforge-release`**, with unchanged payload bytes. Its ten payload fields are
**`pineforge-release`**. Its ten payload fields remain
exactly section 3.2 of the design: `schema`, `event_id`, `release_version`,
`prerelease`, `hub_commit`, `versions`, `image`, `facts`, `manifest`, `run_id`.
The `compute` role uses **`release-pair`**, described below.
Expand All @@ -173,12 +195,49 @@ Fixtures for consumer implementations:
- [Strict validator](../scripts/release_contract.py)

Fixture identities are synthetic, never live release evidence. The manifest is
UTF-8 JSON with sorted keys, compact separators and one final LF. It contains
UTF-8 JSON with sorted keys, compact separators and one final LF. Its legacy form contains
the eight fields above other than `event_id` and `manifest`. Both omitted fields
contain its SHA-256; excluding them avoids a circular hash. `run_id` is the
original candidate build run, a JSON integer. Replays fetch the existing bytes;
attempt IDs occur only in receipts. HPO is explicitly null before the cutoff.

New manifests also contain exactly one additional root field, `assets`: an
array sorted by unique `name`, with exactly four entries: one npm outer tarball,
one codegen sdist and engine Linux x86_64 and aarch64 tarballs. Engine names are
`pineforge-v<versions.engine.version>-linux-<arch>.tar.gz`; npm and sdist names
are the selected producer URL basenames. Each entry has exactly these fields:

| Asset field | Type and meaning |
|---|---|
| `name` | Safe original producer basename, starting with an ASCII letter or digit and otherwise containing only letters, digits, `_`, `.`, `-`; audit namespaces `release-`, `consumer-`, `facts-`, `delivery-` are reserved. |
| `size` | Positive JSON integer, at most 9007199254740991, never a boolean; length of the original compressed bytes. |
| `sha256` | Exactly 64 lowercase hexadecimal characters, computed after producer-authority verification. |
| `source` | Selected HTTPS producer download URL with the same basename: the fixed engine repository/tag on `github.com`, the `@pineforge/codegen-pyodide/-/` package path on `registry.npmjs.org`, or `files.pythonhosted.org` for the codegen sdist. No credentials, query, fragment, traversal or encoded path is accepted. |

Consumers fetch mirrors by `name` from the matching hub release and verify
`size` and `sha256`. `source` records provenance and grants no permission to
fetch an arbitrary URL.

Readers accept exactly the legacy eight root keys or that set plus validated
`assets`, retaining the existing schema and nested fields. The ready payload
projects only the original eight fields and adds `event_id` and `manifest`;
`assets` is obtained from the manifest, never an eleventh dispatch field. Both
hashes cover the entire original raw manifest, including `assets`. Readers
that enforce eight manifest keys, echo every manifest field into the payload,
or drop unknown keys before hashing need this update. Readers that ignore the
extension must still hash the original bytes and compare all ten payload fields.
Consumer implementations and release-record readers must confirm compatibility
before consuming newly extended releases.

Once a manifest is stored, finalization checks its mirrored hub attachments
against every descriptor before promotion or delivery planning, without resolving
producers or newer facts again. A missing or corrupt attachment blocks replay.
Legacy manifests remain valid and are never extended or rewritten on replay;
their raw bytes, event IDs and original build run IDs remain unchanged. Receipt
fields, signatures, evidence hashes and stored receipt bytes are unchanged.
Producer binaries are release attachments; consumer receipt ZIPs still contain
only `release-receipt.json`.

`image.repository` must equal EXACTLY one of two values, the image path before
and after the GitHub organization rename:
`ghcr.io/pineforge-4pass/pineforge-release` or `ghcr.io/tunaxis/pineforge-release`.
Expand Down
45 changes: 40 additions & 5 deletions scripts/release_artifacts.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
import release_pair as pair
import parity_bundle as parity
from release_contract import (COMMIT, Invalid, Pending, canonical, decode,
match, require, sha256)
match, require, sha256, validate_assets)

PRODUCERS = {"engine": "pineforge-4pass/pineforge-engine",
"codegen": "pineforge-4pass/pineforge-codegen-oss",
Expand Down Expand Up @@ -223,6 +223,12 @@ def source_matches(client, component, commit, sources, native=None):
require(expected_native == {name: git_blob(raw) for name, raw in native.items()}, "HPO native sdist differs from tagged source")


def attachment(raw, source):
"""Describe the original compressed bytes, only after producer verification."""
return ({"name": urllib.parse.urlsplit(source).path.rsplit("/", 1)[-1],
"size": len(raw), "sha256": sha256(raw), "source": source}, raw)


def parity_source_proof(client, version, bundle_root):
"""Compare the freshly synced local bundle with a second tagged-source fetch."""
_, commit = release(client, "engine", version)
Expand All @@ -236,7 +242,7 @@ def parity_source_proof(client, version, bundle_root):
return parity.verify_bundle(bundle_root, source)


def producer(client, component, version):
def producer(client, component, version, *, attachments=None):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — producer()

fans out to 10 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

"""Return manifest pin + byte inventory, after checking even the event's own artifacts."""
v = pair.parse(pair.hpo_version(version, to_semver=True) if component == "hpo" else version)
obj, commit = release(client, component, str(v))
Expand All @@ -252,9 +258,15 @@ def producer(client, component, version):
for triple in triples:
name = f"pineforge-v{v}-{triple}.tar.gz"
raw = asset(client, obj, name, repo)
checksum = asset(client, obj, name + ".sha256", repo).decode("ascii").split()
require(len(checksum) == 2 and checksum[0] == sha256(raw) and checksum[1].lstrip("*") == name,
checksum_raw = asset(client, obj, name + ".sha256", repo)
try:
checksum = checksum_raw.decode("ascii").split()
except UnicodeError:
raise Invalid("engine tarball checksum differs") from None
require(len(checksum) == 2 and checksum[0] == sha256(raw) and checksum[1] in (name, "*" + name),
"engine tarball checksum differs")
if attachments is not None and triple.startswith("linux-"):
attachments.append(attachment(raw, f"https://github.com/{repo}/releases/download/{obj['tag_name']}/{name}"))
# Pre-tag/reconcile callers prepare their source proof separately.
# Publish bindings independently prove and inventory their local bundle.
return pin, inventory
Expand Down Expand Up @@ -295,10 +307,12 @@ def producer(client, component, version):
pin.update(wheel_sha256=sha256(wheel), sdist_sha256=sha256(sdist))
source_matches(client, component, commit, sources, native)
inventory.update({name: sha256(raw) for name, raw in sources.items()})
if component == "codegen" and attachments is not None:
attachments.append(attachment(sdist, sdists[0]["url"]))
return pin, inventory


def npm(client, pin, inventory):
def npm(client, pin, inventory, *, attachments=None):
v = pin["version"]
obj = decode(client.download("https://registry.npmjs.org/@pineforge%2fcodegen-pyodide/" + v, {"registry.npmjs.org"}))
require(obj["name"] == "@pineforge/codegen-pyodide" and obj["version"] == v, "unexpected npm identity")
Expand All @@ -318,9 +332,30 @@ def npm(client, pin, inventory):
require({name: sha256(data) for name, data in sources.items()} == inventory, "npm and PyPI codegen sources differ")
if obj.get("gitHead"):
require(obj["gitHead"] == pin["commit"], "npm git commit differs")
if attachments is not None:
attachments.append(attachment(raw, obj["dist"]["tarball"]))
return {**pin, "npm_version": v, "npm_integrity": integrity}


def verified_assets(client, pins, inventory):
"""Stage every mirror from candidate pins; never upload or resolve latest."""
staged = []
for role in ("engine", "codegen"):
resolved, _ = producer(client, role, pins[role]["version"], attachments=staged)
expected = pins[role]
if role == "engine":
# The candidate already binds parity to the built image. Refetch
# the archive identity without dropping that proof from ready pins.
expected = {key: value for key, value in expected.items() if key != "parity"}
# Codegen includes the original candidate's sdist digest. A newly
# published API digest cannot authorize replacement of the built bytes.
require(resolved == expected, role + " producer identity differs from candidate")
ready = {**pins, "codegen": npm(client, pins["codegen"], inventory, attachments=staged)}
staged.sort(key=lambda item: item[0]["name"])
validate_assets([descriptor for descriptor, _ in staged], ready)
return ready, staged


def command(*args, input=None):
result = subprocess.run(args, input=input, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False)
require(result.returncode == 0, "release command failed: " + args[0])
Expand Down
61 changes: 59 additions & 2 deletions scripts/release_contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import hashlib
import json
import re
from urllib.parse import urlsplit

import release_pair as pair
from release_sources import RELEASE_TAG
Expand Down Expand Up @@ -96,6 +97,58 @@ def version(value):
return v


def asset_descriptor(obj, versions):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — asset_descriptor()

high coupling complexity (Ca·Ce = 15).

Grounded coupling-delta finding (deterministic), not an LLM guess.

"""Validate one producer filename and its provenance; return its required kind."""
keys(obj, ("name", "size", "sha256", "source"))
name = match(r"[A-Za-z0-9][A-Za-z0-9_.-]*", obj["name"])
require(not name.startswith(("release-", "consumer-", "facts-", "delivery-")),
"reserved attachment name")
positive(obj["size"])
match(SHA, obj["sha256"])
source = obj["source"]
require(isinstance(source, str) and source.isascii()
and not any(c.isspace() or ord(c) < 32 or ord(c) == 127 for c in source),
"unsafe attachment source")
try:
parsed = urlsplit(source)
except ValueError:
raise Invalid("unsafe attachment source") from None
require(parsed.scheme == "https" and parsed.netloc in
("github.com", "registry.npmjs.org", "files.pythonhosted.org")
and not parsed.query and not parsed.fragment
and source == "https://" + parsed.netloc + parsed.path,
"unexpected attachment origin")
parts = parsed.path.split("/")
require(parts[0] == "" and all(part not in ("", ".", "..") for part in parts[1:])
and re.fullmatch(r"/[A-Za-z0-9_@./-]+", parsed.path) is not None
and parts[-1] == name, "unsafe attachment path")
if parsed.netloc == "registry.npmjs.org":
require(parsed.path == "/@pineforge/codegen-pyodide/-/" + name
and name.endswith((".tgz", ".tar.gz")), "unexpected npm attachment")
return "npm"
if parsed.netloc == "files.pythonhosted.org":
require(name.endswith(".tar.gz"), "unexpected sdist attachment")
return "sdist"
engine = versions["engine"]
for arch in ("x86_64", "aarch64"):
expected = f"pineforge-v{engine['version']}-linux-{arch}.tar.gz"
if name == expected:
require(source == f"https://github.com/pineforge-4pass/pineforge-engine/releases/download/{engine['tag']}/{name}",
"unexpected engine attachment source")
return arch
raise Invalid("unexpected engine attachment")


def validate_assets(assets, versions):
require(type(assets) is list and len(assets) == 4, "incomplete attachment inventory")
kinds = [asset_descriptor(obj, versions) for obj in assets]
names = [obj["name"] for obj in assets]
require(len(set(names)) == len(names) and names == sorted(names),
"attachment names must be unique and sorted")
require(set(kinds) == {"npm", "sdist", "x86_64", "aarch64"}, "unexpected attachment inventory")
return assets


def parity_capable(release):
# parity_bundle uses the byte primitives above; import the cutoff lazily
# so the shared byte contract does not introduce an import cycle.
Expand Down Expand Up @@ -155,7 +208,8 @@ def parity_child_digests(image):


def validate(manifest):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — validate()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

keys(manifest, MANIFEST_KEYS)
require(type(manifest) is dict and set(manifest) in (MANIFEST_KEYS, MANIFEST_KEYS | {"assets"}),
"unexpected object fields")
require(manifest["schema"] == SCHEMA, "unsupported release schema")
v = version(manifest["release_version"])
require(type(manifest["prerelease"]) is bool and manifest["prerelease"] == v.prerelease,
Expand Down Expand Up @@ -206,14 +260,17 @@ def validate(manifest):
match(COMMIT, facts["commit"])
match(SHA, facts["sha256"])
require(facts["path"] == "facts/facts.json" and facts["release_key"] == str(v), "unexpected facts identity")
if "assets" in manifest:
validate_assets(manifest["assets"], versions)
return manifest


def payload(raw):
manifest = validate(decode(raw))
require(canonical(manifest) == raw, "noncanonical manifest bytes")
digest = sha256(raw)
return {**manifest, "event_id": "sha256:" + digest, "manifest": {"asset": ASSET, "sha256": digest}}
return {**{key: manifest[key] for key in MANIFEST_KEYS},
"event_id": "sha256:" + digest, "manifest": {"asset": ASSET, "sha256": digest}}


def validate_candidate(obj):
Expand Down
Loading
Loading