From 294a15e7613bc91bb5d209eea0d2828ec2cdea57 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Tue, 20 Oct 2026 02:48:45 +0300 Subject: [PATCH] feat(sponsord): install sponsord and its onramp from decdn/decdn releases decdn/sponsord was merged into the decdn/decdn workspace and archived; one decdn release now ships decdn-node, decdn, sponsord and sponsord-onramp with one SHA256SUMS, the same archive and image names and the same signing keys. Repoint the sponsord and sponsord_onramp roles (release base and index, source repo, unit Documentation=), vendor decdn's KEYS as the sponsord role's decdn-release-KEYS.asc (a byte copy, as the onramp's) and let the onramp verify against its own copy. Keep sponsord_version / sponsord_onramp_version as variables but require them to equal decdn_node_version (test-scripts, upstream-drift), and make the decdn bump the only bump: bump_decdn.py moves the three lines, the sponsord commented examples and Compose's two sponsord image digests; open-decdn-bump.sh verifies all three digest files and copies a changed KEYS into every role. upstream-drift checks the sponsord digests against the decdn tag. Docs, Compose, cloud-init, monitoring runbook URLs and the molecule fixtures' repo strings follow. The lines stay at 0.2: the bump PR after decdn v0.3.0, the first release that carries the sponsord archives, moves everything to 0.3. Co-Authored-By: Claude Fable 5.1 --- .claude/skills/grafana-dashboards/SKILL.md | 2 +- .github/workflows/upstream-drift.yml | 42 ++--- AGENTS.md | 80 +++++----- SECURITY.md | 8 +- ansible/README.md | 4 +- ansible/galaxy/publisher/README.md | 4 +- .../inventory/group_vars/sponsord_hosts.yml | 9 +- .../group_vars/sponsord_onramp_hosts.yml | 2 +- .../cloud-init-publisher/molecule.yml | 5 +- .../molecule/cloud-init-publisher/prepare.yml | 8 +- .../molecule/sponsord-install/side_effect.yml | 2 +- .../files/fixture/Cargo.toml | 4 +- .../files/fixture/rust-toolchain.toml | 2 +- .../sponsord-onramp-source/molecule.yml | 2 +- .../sponsord-onramp-source/side_effect.yml | 2 +- .../molecule/validation-sponsord/converge.yml | 8 +- ansible/roles/decdn_node/README.md | 49 +++--- ansible/roles/decdn_node/defaults/main.yml | 3 +- ansible/roles/sponsord/README.md | 67 +++----- ansible/roles/sponsord/defaults/main.yml | 28 ++-- ...elease-KEYS.asc => decdn-release-KEYS.asc} | 11 +- ansible/roles/sponsord/tasks/install.yml | 4 +- ansible/roles/sponsord/tasks/main.yml | 6 +- .../sponsord/templates/sponsord.service.j2 | 2 +- ansible/roles/sponsord/vars/main/arch.yml | 4 +- ansible/roles/sponsord_onramp/README.md | 21 +-- .../roles/sponsord_onramp/defaults/main.yml | 33 ++-- .../roles/sponsord_onramp/tasks/install.yml | 6 +- ansible/roles/sponsord_onramp/tasks/main.yml | 6 +- .../templates/sponsord-onramp.service.j2 | 2 +- .../roles/sponsord_onramp/vars/main/arch.yml | 4 +- cloud-init/README.md | 10 +- cloud-init/tests/lint.py | 4 +- cloud-init/user-data-publisher.yaml | 11 +- compose/.env.example | 5 +- compose/README.md | 26 ++- docs/publishers.md | 6 +- .../inventory/group_vars/sponsord_hosts.yml | 9 +- .../group_vars/sponsord_onramp_hosts.yml | 2 +- monitoring/README.md | 4 +- monitoring/sponsord/dashboard-sponsord.json | 2 +- monitoring/sponsord/prometheus-alerts.yml | 20 +-- scripts/bump_decdn.py | 86 +++++++--- scripts/open-decdn-bump.sh | 81 ++++++---- scripts/tests/test_bump_decdn.py | 151 ++++++++++++------ tests/scripts-test.sh | 31 ++-- 46 files changed, 497 insertions(+), 381 deletions(-) rename ansible/roles/sponsord/files/{sponsord-release-KEYS.asc => decdn-release-KEYS.asc} (90%) diff --git a/.claude/skills/grafana-dashboards/SKILL.md b/.claude/skills/grafana-dashboards/SKILL.md index a159126..45fefb0 100644 --- a/.claude/skills/grafana-dashboards/SKILL.md +++ b/.claude/skills/grafana-dashboards/SKILL.md @@ -47,7 +47,7 @@ the iroh relay's in `monitoring/iroh-relay/`): - `monitoring/sponsord/` — the onboarding sponsor's own pair: `dashboard-sponsord.json` (uid `decdn-sponsord`) and `prometheus-alerts.yml` (group `sponsord`, `job="sponsord"`). The chart does not render this directory (sponsord has no Kubernetes path). Its - `sponsord_*` names come from upstream `decdn/sponsord` `crates/sponsord/src/metrics.rs` + `sponsord_*` names come from upstream decdn/decdn's `crates/sponsord/src/metrics.rs` (check them with the recipe after the node one below). Its logs are plain text, not JSON: select `{unit="sponsord.service"}` and filter on the `level` label the `grafana_alloy` role parses. The rules have promtool unit tests in diff --git a/.github/workflows/upstream-drift.yml b/.github/workflows/upstream-drift.yml index 053b61e..499630b 100644 --- a/.github/workflows/upstream-drift.yml +++ b/.github/workflows/upstream-drift.yml @@ -1,9 +1,10 @@ --- name: Upstream drift -# The repo pins one decdn/decdn and one decdn/sponsord release line (decdn_node_version, -# sponsord_version: "X.Y", whose newest finished vX.Y.N the roles install) and carries -# copies of what that release publishes. This job resolves each line as the roles do +# The repo pins one decdn/decdn release line (decdn_node_version = sponsord_version = +# sponsord_onramp_version: "X.Y", whose newest finished vX.Y.N the roles install; one +# release ships the node, the CLI, sponsord and the onramp) and carries copies of what +# that release publishes. This job resolves the line as the roles do # (tasks/resolve-release.yml) and checks, weekly: # - the generated mirrors of decdn/decdn at that release's tag: # roles/decdn_node/vars/main/networks.yml (contract addresses) and its subsets in @@ -13,11 +14,11 @@ name: Upstream drift # - the hand-copied release facts: the onramp's decdn tag and SHA256SUMS digest # (compose/sponsord-onramp.env.example) and the Compose image digests # (compose/.env.example), against that release's assets; -# - that each repo's newest vX.Y.Z tag is still in the pinned line. +# - that decdn's newest vX.Y.Z tag is still in the pinned line. # It is a scheduled early warning, not a PR gate: an upstream release must not turn # unrelated PRs red. Fix drift by re-running the generator it names; act on a new -# release with the bump checklist (roles/decdn_node/README.md, roles/sponsord/README.md); -# scripts/open-decdn-bump.sh opens the decdn bump's PR. +# release with the bump checklist (roles/decdn_node/README.md); +# scripts/open-decdn-bump.sh opens the bump's PR. on: schedule: - cron: '17 6 * * 1' # Mondays 06:17 UTC @@ -40,9 +41,10 @@ jobs: shell: bash steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - # The pins, read from the role defaults (plain scalars, so no YAML parser), and - # each line resolved as the roles resolve it: the highest vX.Y.N that is not a + # The pin, read from the role defaults (plain scalars, so no YAML parser), and + # the line resolved as the roles resolve it: the highest vX.Y.N that is not a # draft or a prerelease and has a SHA256SUMS.asc asset. An exact pin is itself. + # The sponsord roles must pin the same line (make test-scripts checks too). - name: Read and resolve the pinned releases id: pins env: @@ -62,18 +64,21 @@ jobs: } decdn="$(pin decdn_node decdn_node_version)" sponsord="$(pin sponsord sponsord_version)" - if [ -z "$decdn" ] || [ -z "$sponsord" ]; then + onramp="$(pin sponsord_onramp sponsord_onramp_version)" + if [ -z "$decdn" ] || [ -z "$sponsord" ] || [ -z "$onramp" ]; then echo "::error::a role has no pinned version" exit 1 fi + if [ "$sponsord" != "$decdn" ] || [ "$onramp" != "$decdn" ]; then + echo "::error::the roles pin different lines (decdn_node $decdn, sponsord $sponsord, sponsord_onramp $onramp); one decdn/decdn release ships all of them" + exit 1 + fi decdn_tag="$(resolve decdn/decdn "$decdn")" - sponsord_tag="$(resolve decdn/sponsord "$sponsord")" - if [ -z "$decdn_tag" ] || [ -z "$sponsord_tag" ]; then - echo "::error::no finished release in decdn/decdn $decdn or decdn/sponsord $sponsord" + if [ -z "$decdn_tag" ]; then + echo "::error::no finished release in decdn/decdn $decdn" exit 1 fi - { echo "decdn=$decdn"; echo "sponsord=$sponsord" - echo "decdn_tag=$decdn_tag"; echo "sponsord_tag=$sponsord_tag"; } >> "$GITHUB_OUTPUT" + { echo "decdn=$decdn"; echo "decdn_tag=$decdn_tag"; } >> "$GITHUB_OUTPUT" # Public repo, the resolved tag. The sync script reads HEAD through git. - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -90,9 +95,7 @@ jobs: - name: Compare the mirrors and the pins with upstream env: DECDN_PIN: ${{ steps.pins.outputs.decdn }} - SPONSORD_PIN: ${{ steps.pins.outputs.sponsord }} DECDN_TAG: ${{ steps.pins.outputs.decdn_tag }} - SPONSORD_TAG: ${{ steps.pins.outputs.sponsord_tag }} run: | rc=0 check() { @@ -165,16 +168,15 @@ jobs: echo "$repo $tag: $var should be $got, compose/.env.example has $want" return 1 } - echo "### Upstream drift (decdn/decdn $DECDN_PIN: $DECDN_TAG, $(git -C upstream rev-parse --short HEAD); decdn/sponsord $SPONSORD_PIN: $SPONSORD_TAG)" >> "$GITHUB_STEP_SUMMARY" + echo "### Upstream drift (decdn/decdn $DECDN_PIN: $DECDN_TAG, $(git -C upstream rev-parse --short HEAD))" >> "$GITHUB_STEP_SUMMARY" check "network profiles" python3 scripts/sync-network-profiles.py upstream --ref HEAD --check check "config schema keys" schema_check check "decdn/decdn release line" release_check decdn/decdn "$DECDN_PIN" - check "decdn/sponsord release line" release_check decdn/sponsord "$SPONSORD_PIN" check "Compose onramp decdn pin" sums_check decdn/decdn "$DECDN_TAG" check "Compose decdn-node digest" digest_check decdn/decdn "$DECDN_TAG" DECDN_IMAGE_DIGEST \ decdn-node-image-digest.txt - check "Compose sponsord digest" digest_check decdn/sponsord "$SPONSORD_TAG" SPONSORD_IMAGE_DIGEST \ + check "Compose sponsord digest" digest_check decdn/decdn "$DECDN_TAG" SPONSORD_IMAGE_DIGEST \ sponsord-image-digest.txt - check "Compose sponsord-onramp digest" digest_check decdn/sponsord "$SPONSORD_TAG" \ + check "Compose sponsord-onramp digest" digest_check decdn/decdn "$DECDN_TAG" \ SPONSORD_ONRAMP_IMAGE_DIGEST sponsord-onramp-image-digest.txt exit "$rc" diff --git a/AGENTS.md b/AGENTS.md index 279c87d..4a7be94 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -131,7 +131,7 @@ scripts/ # upstream-mirror generators, the compose Alloy config g and `ansible/molecule/schema/files/schema-keys.txt` (`gen-schema-keys.py`), both generated from the tag of the newest decdn/decdn release in the line `decdn_node_version` pins. The weekly `upstream-drift` workflow flags staleness -against that tag, and a new upstream release line of either repo. These are the only protocol facts (contract addresses) the +against that tag, and a new upstream release line. These are the only protocol facts (contract addresses) the repo carries, and `networks.yml` carries its upstream commit. **Generated from this repo — regenerate, never hand-edit:** `compose/alloy/config.alloy` @@ -141,37 +141,37 @@ runtime (`scripts/render-compose-alloy.sh`, which runs the `compose`-tagged task test-scripts` fails while the committed file differs from a fresh render, and `make lint-alloy` validates and runs it. -**Pinned upstream releases.** One decdn/decdn release line (`decdn_node_version`, today -`0.2`) and one decdn/sponsord release line (`sponsord_version` = `sponsord_onramp_version`, -today `0.2`; decdn/sponsord tags its whole workspace `vX.Y.Z`, there are no per-crate -tags) are pinned across every path. A line, `X.Y`, is what the role defaults, the -onramp's installer pin (`sponsord_onramp_decdn_release`, `vX.Y`) and the chart's -`appVersion` name: the roles install its newest finished `vX.Y.N` on every run -(`tasks/resolve-release.yml`, copied into all three roles: GitHub's release list, then -the same signed `SHA256SUMS` and exact `--version` check against the resolved version; -the onramp reads the installers' digest from that release's verified `SHA256SUMS`), -and the chart's default image tag is the line's floating tag, pulled `Always`. An -exact `X.Y.Z` in inventory (or `image.tag`/`image.digest`) holds a host on one release. -So an upstream patch release needs no release of the collections or the chart; a new -line does. What cannot resolve a line names one release of it and moves on every -upstream release, repo-only: `compose/.env.example`'s image digests, the onramp's -installer pin in `compose/sponsord-onramp.env.example` (tag + `SHA256SUMS` digest), the -chart's example digest, the commented exact-pin examples, and the generated mirrors -(which a patch is not expected to change; when one does, the bump PR says so and the -collections need a release). cloud-init takes the role defaults at `DEVOPS_REF`. Docs -name the pin (`decdn_node_version`, `sponsord_version`), never its number: besides this -paragraph, only the commented examples and the comments beside a pinned value name a -number, and they move with the pin (`bump_decdn.py` moves decdn's; the sponsord -checklist lists sponsord's). `make test-scripts` checks the pins agree with each other, -the Compose example (a release of the line), `appVersion`, the chart's -`artifacthub.io/images` tag and this paragraph, and that the vendored KEYS hold the -keys SECURITY.md publishes; the weekly `upstream-drift` job checks the exact copies and -the mirrors against each line's newest release and flags a newer upstream line. Bump -with the checklists in `roles/decdn_node/README.md` and `roles/sponsord/README.md`; -`scripts/open-decdn-bump.sh` opens the decdn bump's PR (decdn's `publish-crates.sh` -runs it after each release). Both upstream repos are public and also publish their -binaries as crates (`decdn-node`, `decdn-cli`, `sponsord`, `sponsord-onramp`); the -roles do not install from crates.io (no maintainer signature). +**Pinned upstream releases.** One decdn/decdn release line is pinned across every +path, by three role defaults that must agree (`decdn_node_version` = `sponsord_version` += `sponsord_onramp_version`, today `0.2`): one decdn/decdn release (`vX.Y.Z`, the whole +workspace, no per-crate tags) ships `decdn-node`, `decdn`, `sponsord` and +`sponsord-onramp`, their four images and one `SHA256SUMS`. A line, `X.Y`, is what the +role defaults, the onramp's installer pin (`sponsord_onramp_decdn_release`, `vX.Y`) and +the chart's `appVersion` name: the roles install its newest finished `vX.Y.N` on every +run (`tasks/resolve-release.yml`, copied into all three roles: GitHub's release list, +then the same signed `SHA256SUMS` and exact `--version` check against the resolved +version; the onramp reads the installers' digest from that release's verified +`SHA256SUMS`), and the chart's default image tag is the line's floating tag, pulled +`Always`. An exact `X.Y.Z` in inventory (or `image.tag`/`image.digest`) holds a host on +one release. So an upstream patch release needs no release of the collections or the +chart; a new line does. What cannot resolve a line names one release of it and moves +on every upstream release, repo-only: `compose/.env.example`'s three image digests, the +onramp's installer pin in `compose/sponsord-onramp.env.example` (tag + `SHA256SUMS` +digest), the chart's example digest, the commented exact-pin examples, and the +generated mirrors (which a patch is not expected to change; when one does, the bump PR +says so and the collections need a release). cloud-init takes the role defaults at +`DEVOPS_REF`. Docs name the pin (`decdn_node_version`, `sponsord_version`), never its +number: besides this paragraph, only the commented examples and the comments beside a +pinned value name a number, and they move with the pin (`bump_decdn.py` moves all of +them). `make test-scripts` checks the pins agree with each other, the Compose example (a +release of the line), `appVersion`, the chart's `artifacthub.io/images` tag and this +paragraph, and that the vendored KEYS (three byte copies of decdn's) hold the keys +SECURITY.md publishes; the weekly `upstream-drift` job checks the exact copies and the +mirrors against the line's newest release and flags a newer upstream line. Bump with +the checklist in `roles/decdn_node/README.md`; `scripts/open-decdn-bump.sh` opens the +bump's PR (decdn's `publish-crates.sh` runs it after each release). Upstream is public +and also publishes the binaries as crates (`decdn-node`, `decdn-cli`, `sponsord`, +`sponsord-onramp`); the roles do not install from crates.io (no maintainer signature). **Monitoring assets are maintained here, by hand.** `monitoring/` holds the deCDN Grafana dashboards and Prometheus alert rules for every deploy path; upstream ships none. @@ -183,7 +183,7 @@ node's `/metrics` and upstream's `adr/appendix-observability.md` registry (a `pl row emits nothing). Nothing in CI checks the names, so a typo renders `(no data)` or a rule that never fires. See `.claude/skills/grafana-dashboards/SKILL.md`. sponsord's pair lives in `monitoring/sponsord/`, which the chart does not render; its `sponsord_*` -names must be ones upstream `decdn/sponsord` `crates/sponsord/src/metrics.rs` exports, +names must be ones upstream decdn/decdn's `crates/sponsord/src/metrics.rs` exports, and its rules have promtool unit tests (`monitoring/sponsord/prometheus-alerts_test.yml`, run by `make lint-helm`), as the node's `DecdnNodeDown` does (`monitoring/decdn-node/prometheus-alerts_test.yml`, which the chart's `.helmignore` @@ -308,13 +308,14 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). Alloy (every component healthy, the targets per profile), since `alloy validate` neither parses OTTL nor evaluates `sys.env`. -- **`ansible/roles/sponsord`** — the deCDN onboarding sponsor (`decdn/sponsord`): the +- **`ansible/roles/sponsord`** — the deCDN onboarding sponsor (decdn/decdn's `sponsord` crate): the treasury wallet's signer for capped capabilities plus the PaymentPool keeper. It is independent of the node, so `playbooks/sponsord.yml` (`make deploy-sponsord`, also imported by `publisher.yml`) targets its own `sponsord_hosts` group, standalone or co-located with a node. - - **Install:** a GPG-verified `v*` decdn/sponsord release by default - (`sponsord_version`: the newest of its line, or an exact release), a host-side + - **Install:** a GPG-verified `v*` decdn/decdn release by default + (`sponsord_version`: the newest of its line, the node's line, or an exact + release; the same release ships both daemons and the node), a host-side `source` build, or a manual binary. The `--version` backstop checks the version in `release` mode only (the version has a default, so other modes would fail against it), as in the other two roles. @@ -362,8 +363,9 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). `make test-scripts`); the role checks no group name, so collection users keep their own groups, and its token gate and `/healthz` fail without a daemon. - **Install / unit / gate:** the sponsord role's patterns, copied: a - GPG-verified `v*` decdn/sponsord release (default), `source` or manual (the - KEYS and the source installer are the sponsord role's files, via `role_path`), `DynamicUser` with the token and the Turnstile secret (inventory + GPG-verified `v*` decdn/decdn release (default), `source` or manual (the + source installer is the sponsord role's file, via `role_path`; the KEYS its own + copy of decdn's), `DynamicUser` with the token and the Turnstile secret (inventory or operator, with `secret.env`'s provenance record and guards) as `LoadCredential=`, `PartOf=sponsord.service`, a restart-inputs record, and a fatal `/healthz` gate on a loopback listener. @@ -656,7 +658,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). - **The allow-lists are deliberate friction.** A new key, mount or inline env var in `compose.yaml` fails the lint until it is added to the matching list in `compose/tests/*.jq`, and the PR should say why. Env var names follow upstream - `decdn/sponsord` (`crates/*/src/config.rs`); re-check them when bumping images. + decdn/decdn (`crates/sponsord*/src/config.rs`); re-check them when bumping images. - **CI's Compose is older than a dev box's.** The runner's release mishandled `config --no-env-resolution` (v2.33 lacks it), which is why the inline-env render uses `/dev/null` env files instead. Before relying on a newer `config` flag, try diff --git a/SECURITY.md b/SECURITY.md index cf8d864..e9392d9 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -23,10 +23,10 @@ In `release` install mode, and while `decdn_verify_release_signature` keeps its default of `true`, the `decdn_node` role checks every downloaded tarball against the release's GPG-signed `SHA256SUMS`, using the maintainer keys vendored at `ansible/roles/decdn_node/files/decdn-release-KEYS.asc`. The `sponsord` and -`sponsord_onramp` roles do the same with -`ansible/roles/sponsord/files/sponsord-release-KEYS.asc`. Both files are copies of -upstream's `KEYS` (decdn/decdn and decdn/sponsord), which hold the same maintainer -keys, and a good signature from any of them is accepted: +`sponsord_onramp` roles do the same with their own `files/decdn-release-KEYS.asc` +(one decdn/decdn release ships all four binaries). The three files are byte copies of +upstream's `KEYS` (`make test-scripts` checks), which holds the maintainer keys, and a +good signature from any of them is accepted: ``` Ant Somers diff --git a/ansible/README.md b/ansible/README.md index 7f6be87..cc27ac1 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -186,8 +186,8 @@ ignored on the first converge. `decdn_node_version` defaults to the decdn/decdn release line this repo pins (the role's `defaults/main.yml`, `"X.Y"`): each run installs the line's newest finished release, so a patch release arrives with the next deploy. Set an exact - `"X.Y.Z"` to hold a host on one release. The sponsord roles pin a decdn/sponsord - line the same way (`sponsord_version`). + `"X.Y.Z"` to hold a host on one release. The sponsord roles pin the same + decdn/decdn line the same way (`sponsord_version`, `sponsord_onramp_version`). - **`source`** clones `decdn_node_source_repo` (default `decdn/decdn` on GitHub) at `decdn_node_source_ref` (any tag, branch or SHA) **on the node** and builds it there as an unprivileged `decdn-build` user, in a fresh environment per commit, with a diff --git a/ansible/galaxy/publisher/README.md b/ansible/galaxy/publisher/README.md index bc866a4..91abfad 100644 --- a/ansible/galaxy/publisher/README.md +++ b/ansible/galaxy/publisher/README.md @@ -12,8 +12,8 @@ them and `decdn.node`'s (see [Usage](#usage)): | Role | Purpose | |------|---------| -| `decdn.publisher.sponsord` | The `sponsord` onboarding sponsor (treasury signer + PaymentPool keeper), standalone or beside a node — local binary, a host-side source build or a GPG-verified decdn/sponsord `v*` release (pinned by `sponsord_version`), `DynamicUser` unit with the API token and treasury wallet as systemd credentials, loopback-only API, `/healthz` deploy gate. `sponsord_network` sets the chain and PaymentPool from upstream's manifest. Restarts refuse while a pool top-up is held; `tasks_from: backup` / `decommission` for day 2. | -| `decdn.publisher.sponsord_onramp` | `sponsord-onramp`, sponsord's public side (Turnstile gate, installers, capability API), on the daemon's host — local binary, a host-side source build or a GPG-verified decdn/sponsord `v*` release (pinned by `sponsord_onramp_version`, kept equal to `sponsord_version`), `DynamicUser` unit with the daemon token and Turnstile secret as systemd credentials, loopback listener behind Caddy (distro package, ACME TLS, admin API off; or your own proxy), `/healthz` deploy gate; `tasks_from: decommission`. | +| `decdn.publisher.sponsord` | The `sponsord` onboarding sponsor (treasury signer + PaymentPool keeper), standalone or beside a node — local binary, a host-side source build or a GPG-verified decdn/decdn `v*` release (pinned by `sponsord_version`, the same line as the node's), `DynamicUser` unit with the API token and treasury wallet as systemd credentials, loopback-only API, `/healthz` deploy gate. `sponsord_network` sets the chain and PaymentPool from upstream's manifest. Restarts refuse while a pool top-up is held; `tasks_from: backup` / `decommission` for day 2. | +| `decdn.publisher.sponsord_onramp` | `sponsord-onramp`, sponsord's public side (Turnstile gate, installers, capability API), on the daemon's host — local binary, a host-side source build or a GPG-verified decdn/decdn `v*` release (pinned by `sponsord_onramp_version`, kept equal to `sponsord_version`), `DynamicUser` unit with the daemon token and Turnstile secret as systemd credentials, loopback listener behind Caddy (distro package, ACME TLS, admin API off; or your own proxy), `/healthz` deploy gate; `tasks_from: decommission`. | | `decdn.publisher.iroh_relay` | A self-hosted [iroh relay](https://github.com/n0-computer/iroh) (`iroh-relay`) for deCDN peers that cannot hole-punch — sha256-pinned upstream release (or a local binary), its own Let's Encrypt TLS, QUIC address discovery, `DynamicUser` unit holding only `CAP_NET_BIND_SERVICE`, loopback metrics, a deploy gate on every listener, a refusal of ports another process holds; access limited to an allowlist of endpoint IDs by default (`iroh_relay_access`; `tasks_from: node-ids` reads nodes' IDs); `tasks_from: decommission`. Point nodes at it with `decdn_relay_urls`. | | `decdn.publisher.iroh_dns_server` | A self-hosted [iroh DNS server](https://github.com/n0-computer/iroh) (`iroh-dns-server`): the pkarr relay deCDN nodes publish their address records to and the DNS server peers resolve them through, in place of n0's `dns.iroh.link` — sha256-pinned upstream release (or a local binary), its own Let's Encrypt TLS, DNS on one bind address (clear of systemd-resolved's stub), `DynamicUser` unit holding only `CAP_NET_BIND_SERVICE`, loopback metrics, a deploy gate on every listener, `/healthz`'s version and an SOA answer over udp and tcp, a refusal of ports another process holds; `tasks_from: decommission`. Point nodes at it with `decdn_discovery_pkarr_url` and `decdn_discovery_dns_origin`. | diff --git a/ansible/inventory/group_vars/sponsord_hosts.yml b/ansible/inventory/group_vars/sponsord_hosts.yml index b59380d..00626d2 100644 --- a/ansible/inventory/group_vars/sponsord_hosts.yml +++ b/ansible/inventory/group_vars/sponsord_hosts.yml @@ -11,15 +11,16 @@ # The API token is generated on the host at /etc/sponsord/api-token. # --- Binary ------------------------------------------------------------------------ -# The default method, "release", installs the newest release of the decdn/sponsord -# line the role pins (sponsord_version "X.Y"), on each deploy; set an exact one only -# to hold the host on it, together with sponsord_onramp_version: +# The default method, "release", installs the newest release of the decdn/decdn line +# the role pins (sponsord_version "X.Y", the decdn_node role's line: one release ships +# the node, sponsord and the onramp), on each deploy; set an exact one only to hold +# the host on it, together with sponsord_onramp_version: # sponsord_version: "0.2.0" # Or build on the host from a git ref (tag, branch or SHA; sponsord_version is not # checked in this mode): # sponsord_install_method: source # sponsord_source_ref: v0.2.0 -# Or copy a control-machine build: a decdn/sponsord checkout's target dir +# Or copy a control-machine build: a decdn/decdn checkout's target dir # (`cargo build --release -p sponsord`; the role checks the ELF matches the host). # sponsord_install_method: manual # sponsord_release_target_dir: /path/to/sponsord/target/release diff --git a/ansible/inventory/group_vars/sponsord_onramp_hosts.yml b/ansible/inventory/group_vars/sponsord_onramp_hosts.yml index 14b6387..f24cfde 100644 --- a/ansible/inventory/group_vars/sponsord_onramp_hosts.yml +++ b/ansible/inventory/group_vars/sponsord_onramp_hosts.yml @@ -17,7 +17,7 @@ # sponsord_onramp_install_method: source # sponsord_onramp_source_ref: v0.2.0 # Or manual, which follows sponsord_release_target_dir (`cargo build --release` in -# a decdn/sponsord checkout builds both binaries): +# a decdn/decdn checkout builds both binaries): # sponsord_onramp_install_method: manual # --- Public address ---------------------------------------------------------------- diff --git a/ansible/molecule/cloud-init-publisher/molecule.yml b/ansible/molecule/cloud-init-publisher/molecule.yml index 48e0c90..80b7329 100644 --- a/ansible/molecule/cloud-init-publisher/molecule.yml +++ b/ansible/molecule/cloud-init-publisher/molecule.yml @@ -6,8 +6,9 @@ # covers what differs: # prepare cloud-init and the repo (as ../cloud-init), shared mount propagation # (as ../sponsord), and a signed loopback mirror holding a decdn/decdn -# release of the node stubs and a decdn/sponsord release of the shared -# sponsor stubs, each under its own base. +# release of the node stubs and a second release of the shared sponsor +# stubs, each under its own base (upstream ships all four binaries in one +# release; the stubs are separate fixtures, so the mirror keeps two). # converge seeds the template (placeholders filled, an http origin backend; # release bases, keyrings and Caddy's `tls internal` pointed at local # ones) and boots. Asserts the host stopped at the secret gate with the diff --git a/ansible/molecule/cloud-init-publisher/prepare.yml b/ansible/molecule/cloud-init-publisher/prepare.yml index 5326ca0..92b56e9 100644 --- a/ansible/molecule/cloud-init-publisher/prepare.yml +++ b/ansible/molecule/cloud-init-publisher/prepare.yml @@ -11,10 +11,10 @@ vars: stub_version: "0.0.0-molecule-stub" stubs: "{{ lookup('ansible.builtin.env', 'MOLECULE_PROJECT_DIRECTORY') }}/molecule" - # Two repositories' releases, so each gets its own base (converge.yml) and release - # list: decdn/decdn's node and CLI (../cloud-init's stub), and - # decdn/sponsord's one release holding both archives under one SHA256SUMS, as it - # releases its workspace. + # Two mirrored releases, each with its own base (converge.yml) and release list: + # one holding the node and CLI stubs (../cloud-init's), one holding both sponsor + # stubs under one SHA256SUMS. Upstream ships all four from one decdn/decdn + # release; the stubs are separate fixtures, so the mirror keeps two bases. mirror_releases: - dir: "decdn/v{{ stub_version }}" bins: diff --git a/ansible/molecule/sponsord-install/side_effect.yml b/ansible/molecule/sponsord-install/side_effect.yml index 1c78a02..6ad7f8f 100644 --- a/ansible/molecule/sponsord-install/side_effect.yml +++ b/ansible/molecule/sponsord-install/side_effect.yml @@ -48,7 +48,7 @@ - name: Seed a source build stamp ansible.builtin.copy: dest: /usr/local/lib/sponsord/installed-source - content: "https://github.com/decdn/sponsord.git@0000000000000000000000000000000000000000\n" + content: "https://github.com/decdn/decdn.git@0000000000000000000000000000000000000000\n" mode: "0644" - name: Install the pinned release diff --git a/ansible/molecule/sponsord-onramp-source/files/fixture/Cargo.toml b/ansible/molecule/sponsord-onramp-source/files/fixture/Cargo.toml index 1380ad5..bf898cd 100644 --- a/ansible/molecule/sponsord-onramp-source/files/fixture/Cargo.toml +++ b/ansible/molecule/sponsord-onramp-source/files/fixture/Cargo.toml @@ -1,5 +1,5 @@ -# Test fixture for molecule/sponsord-onramp-source: a workspace shaped like -# decdn/sponsord (packages and binaries sponsord and sponsord-onramp), with no +# Test fixture for molecule/sponsord-onramp-source: a workspace shaped like decdn/decdn's +# sponsord crates (packages and binaries sponsord and sponsord-onramp), with no # dependencies so it compiles in seconds. Each binary execs the matching Python stub # (molecule/sponsord, molecule/sponsord-onramp), so both roles run end to end on # source-built binaries. diff --git a/ansible/molecule/sponsord-onramp-source/files/fixture/rust-toolchain.toml b/ansible/molecule/sponsord-onramp-source/files/fixture/rust-toolchain.toml index 397dc7e..0865d02 100644 --- a/ansible/molecule/sponsord-onramp-source/files/fixture/rust-toolchain.toml +++ b/ansible/molecule/sponsord-onramp-source/files/fixture/rust-toolchain.toml @@ -1,4 +1,4 @@ -# The same channel decdn/sponsord's rust-toolchain.toml pins (see +# The same channel decdn/decdn's rust-toolchain.toml pins (see # ../../../source-build/files/fixture/rust-toolchain.toml). [toolchain] channel = "1.95.0" diff --git a/ansible/molecule/sponsord-onramp-source/molecule.yml b/ansible/molecule/sponsord-onramp-source/molecule.yml index a81a361..9050e18 100644 --- a/ansible/molecule/sponsord-onramp-source/molecule.yml +++ b/ansible/molecule/sponsord-onramp-source/molecule.yml @@ -3,7 +3,7 @@ # side by side: the same host shape, inventory and converge (that scenario's # prepare, converge and verify, by path), then side_effect.yml here switches both # roles to install_method "source" against files/fixture (a dependency-free -# workspace shaped like decdn/sponsord whose binaries exec that scenario's stubs), +# workspace shaped like decdn/decdn's sponsord crates whose binaries exec that scenario's stubs), # checks both are built in the sandbox, stamped and serving, and switches back to # "manual". It needs network access to static.rust-lang.org (rustup and the # toolchain), on both platforms. diff --git a/ansible/molecule/sponsord-onramp-source/side_effect.yml b/ansible/molecule/sponsord-onramp-source/side_effect.yml index 40d575c..fb90dee 100644 --- a/ansible/molecule/sponsord-onramp-source/side_effect.yml +++ b/ansible/molecule/sponsord-onramp-source/side_effect.yml @@ -1,6 +1,6 @@ --- # "source" for both roles on the host ../sponsord-onramp converged: each builds its -# own clone of one fixture repo shaped like decdn/sponsord, sharing one build user and +# own clone of one fixture repo shaped like decdn/decdn's sponsord crates, sharing one build user and # toolchain, installs and stamps its binary, and passes its /healthz gate; "manual" # afterwards clears both stamps. The host is left on the manual stubs for verify. # diff --git a/ansible/molecule/validation-sponsord/converge.yml b/ansible/molecule/validation-sponsord/converge.yml index e824fbc..9f36953 100644 --- a/ansible/molecule/validation-sponsord/converge.yml +++ b/ansible/molecule/validation-sponsord/converge.yml @@ -1305,13 +1305,13 @@ tag: sponsord-source-repo method: source # kics-scan ignore-line - repo: "https://token@github.com/decdn/sponsord.git" + repo: "https://token@github.com/decdn/decdn.git" task: Refuse credentials in the source repo URL - role: sponsord tag: sponsord-source-repo method: source # kics-scan ignore-line - repo: "https://github.com/decdn/sponsord.git?access_token=x" + repo: "https://github.com/decdn/decdn.git?access_token=x" task: Refuse credentials in the source repo URL - {role: sponsord, tag: sponsord-source-jobs, method: source, jobs: "0", task: Validate sponsord_source_build_jobs} @@ -1323,14 +1323,14 @@ tag: onramp-source-repo method: source # kics-scan ignore-line - repo: "https://user:token@github.com/decdn/sponsord.git" + repo: "https://user:token@github.com/decdn/decdn.git" task: Refuse credentials in the source repo URL # Accepted: the scp form passes the credential check and fails at the # next assert (the bad jobs value) instead. - role: sponsord_onramp tag: onramp-source-repo-scp-ok method: source - repo: "git@github.com:decdn/sponsord.git" + repo: "git@github.com:decdn/decdn.git" jobs: "x" task: Validate sponsord_onramp_source_build_jobs - {role: sponsord_onramp, tag: onramp-source-jobs, method: source, jobs: "-1", diff --git a/ansible/roles/decdn_node/README.md b/ansible/roles/decdn_node/README.md index 8da036f..1e51a94 100644 --- a/ansible/roles/decdn_node/README.md +++ b/ansible/roles/decdn_node/README.md @@ -730,11 +730,12 @@ and drop its line from `/etc/cron.deny` and `/etc/at.deny`. ## Bumping the decdn release -The role defaults, the onramp's installer pin and the chart's `appVersion` name a -release line, so a decdn/decdn release in that line reaches deployments on its own. -What cannot resolve a line names one release, and moves on every upstream release: -Compose's image digest and onramp installer pin, the chart's example digest, the -commented exact-pin examples, and the generated mirrors. `scripts/open-decdn-bump.sh +The role defaults (this role's and the two sponsord roles': one decdn/decdn release +ships the node, the CLI, sponsord and the onramp), the onramp's installer pin and the +chart's `appVersion` name a release line, so a decdn/decdn release in that line reaches +deployments on its own. What cannot resolve a line names one release, and moves on +every upstream release: Compose's three image digests and onramp installer pin, the +chart's example digest, the commented exact-pin examples, and the generated mirrors. `scripts/open-decdn-bump.sh [vX.Y.Z]` opens that PR: decdn's `publish-crates.sh` runs it at the end of each decdn release, and a maintainer runs it by hand from a decdn/devops clone (with no tag, it takes decdn's latest release). It works in a worktree at `origin/main`, takes the @@ -755,7 +756,8 @@ release, or a new line is out. The steps: -1. The line: `decdn_node_version` in `defaults/main.yml`, and +1. The line: `decdn_node_version` in `defaults/main.yml`, `sponsord_version` in + `roles/sponsord/defaults/main.yml`, and `sponsord_onramp_version` plus `sponsord_onramp_decdn_release` (`vX.Y`) in `roles/sponsord_onramp/defaults/main.yml`. 2. The generated mirrors, from a decdn checkout at the new tag: `scripts/sync-network-profiles.py --ref v` and @@ -769,19 +771,23 @@ The steps: `ONRAMP_DECDN_SUMS_SHA256` (`sha256sum SHA256SUMS` once `SHA256SUMS.asc` verifies) in `compose/sponsord-onramp.env.example`. The role reads both from the release itself. -4. `DECDN_IMAGE_DIGEST` in `compose/.env.example` (and the example digest in +4. `DECDN_IMAGE_DIGEST`, `SPONSORD_IMAGE_DIGEST` and `SPONSORD_ONRAMP_IMAGE_DIGEST` in + `compose/.env.example` (and the node's example digest in `charts/decdn-node/values.yaml`), from the release's signed - `decdn-node-image-digest.txt` (`image-digest.txt` in v0.0.1). + `decdn-node-image-digest.txt` (`image-digest.txt` in v0.0.1), + `sponsord-image-digest.txt` and `sponsord-onramp-image-digest.txt`. 5. The line again: `appVersion` in `charts/decdn-node/Chart.yaml` and the image tag in its `artifacthub.io/images` annotation. CI's `helm` job then validates the chart against the line's newest `decdn` (`scripts/fetch-decdn-cli.sh`, the same resolution and signature check as this role's). -6. `files/decdn-release-KEYS.asc`, if upstream's `KEYS` changed (and the sponsord and - sponsord_onramp copies, and `SECURITY.md`'s fingerprints). -7. The commented exact-pin examples (`decdn_node_version`, `decdn_node_source_ref` - and `sponsord_onramp_decdn_release`, in the inventory templates, their - `examples/collection-fleet` copies and `cloud-init/user-data-publisher.yaml`), and - for a new line AGENTS.md's "Pinned upstream releases". Other docs name the pin, +6. `files/decdn-release-KEYS.asc`, if upstream's `KEYS` changed (the sponsord and + sponsord_onramp roles' byte copies with it, and `SECURITY.md`'s fingerprints). +7. The commented exact-pin examples (`decdn_node_version`, `decdn_node_source_ref`, + `sponsord_version`, `sponsord_source_ref`, `sponsord_onramp_version`, + `sponsord_onramp_source_ref` and `sponsord_onramp_decdn_release`, in the inventory + templates, their `examples/collection-fleet` copies and + `cloud-init/user-data-publisher.yaml`), and for a new line AGENTS.md's "Pinned + upstream releases". Other docs name the pin, never its number. The changelog entry is the PR title (`feat`, `fix` or `chore`, not `ci` or `build`, which the changelogs leave out; RELEASING.md). 8. The monitoring assets, by hand (nothing in CI checks the node's metric names): @@ -789,12 +795,13 @@ The steps: list in `crates/node/src/metrics.rs` between the two tags, fix any renamed or removed `decdn_*` name in `monitoring/decdn-node/`, add a new inbound failure reason to both "Unattributed stream failures" panels, and chart or alert on new - series worth it (`.claude/skills/grafana-dashboards/SKILL.md`). Republish the - Grafana Cloud copies after the merge. + series worth it (`.claude/skills/grafana-dashboards/SKILL.md`). The same for + sponsord's `monitoring/sponsord/` against `crates/sponsord/src/metrics.rs`, and the + env var names Compose and the sponsord roles render against + `crates/sponsord*/src/config.rs`. Republish the Grafana Cloud copies after the merge. -`make test-scripts` fails while the role, the onramp's decdn line, the chart's -`appVersion` and its `artifacthub.io/images` image or AGENTS.md's pinned line +`make test-scripts` fails while the three roles' lines, the onramp's decdn line, the +chart's `appVersion` and its `artifacthub.io/images` image or AGENTS.md's pinned line disagree, while Compose's onramp pin is not a release of the line, or while the KEYS -and SECURITY.md disagree; the weekly `upstream-drift` job checks the exact copies -against the line's newest release. A decdn/sponsord release is bumped the same way -([sponsord's README](../sponsord/README.md#bumping-the-sponsord-release)). +copies and SECURITY.md disagree; the weekly `upstream-drift` job checks the exact +copies against the line's newest release. diff --git a/ansible/roles/decdn_node/defaults/main.yml b/ansible/roles/decdn_node/defaults/main.yml index 76582bc..146eb79 100644 --- a/ansible/roles/decdn_node/defaults/main.yml +++ b/ansible/roles/decdn_node/defaults/main.yml @@ -62,7 +62,8 @@ decdn_rustup_sha256: # decdn_node_release_index on every run, so a new patch is installed and the daemon # restarted by the next run), or an exact "X.Y.Z" pins that one release. The # default is the line this repo is synced against; a new line also moves the -# generated mirrors, the onramp's decdn pin and the chart's appVersion (README.md, +# generated mirrors, the sponsord roles' pins (the same release ships sponsord and +# its onramp), the onramp's decdn pin and the chart's appVersion (README.md, # "Bumping the decdn release"). decdn_node_version: "0.2" # Release target triple. Derived from the host's gathered architecture (x86_64 or diff --git a/ansible/roles/sponsord/README.md b/ansible/roles/sponsord/README.md index ebb5c4c..1e2e9ff 100644 --- a/ansible/roles/sponsord/README.md +++ b/ansible/roles/sponsord/README.md @@ -1,6 +1,7 @@ # Role: `sponsord` -Deploys [sponsord](https://github.com/decdn/sponsord), the deCDN onboarding sponsor. +Deploys [sponsord](https://github.com/decdn/decdn/tree/main/crates/sponsord), the deCDN +onboarding sponsor ([docs](https://github.com/decdn/decdn/tree/main/docs/sponsord)). It holds the **treasury wallet**, which owns one `PaymentPool` pool, and does two things: @@ -20,16 +21,16 @@ gate, installers, behind a TLS reverse proxy), is the - **Installs the binary.** - `release` (the default) downloads `sponsord--.tar.gz` from the - `v` release of `decdn/sponsord` (one tag releases the whole workspace) - and verifies it against the GPG-signed `SHA256SUMS`, using - `files/sponsord-release-KEYS.asc`, a copy of upstream's `KEYS`. + `v` release of `decdn/decdn` (one tag releases the whole workspace: the + node, the CLI, sponsord and the onramp) and verifies it against the GPG-signed + `SHA256SUMS`, using `files/decdn-release-KEYS.asc`, a copy of upstream's `KEYS`. `sponsord_version` defaults to the release line this repo pins (`"X.Y"`): every run takes the newest finished `vX.Y.N` from `sponsord_release_index` (GitHub's release list, as the decdn_node role does; see its README), so a patch release is installed by the next deploy. An exact `"X.Y.Z"` holds the host on one release. `sponsord_release_base` (and `sponsord_release_index`) point it at a mirror. - - `source` clones `sponsord_source_repo` (default `decdn/sponsord` on GitHub) at + - `source` clones `sponsord_source_repo` (default `decdn/decdn` on GitHub) at `sponsord_source_ref` (any tag, branch or SHA) on the host and runs `cargo build --release --locked -p sponsord` as the unprivileged `decdn-build` user, exactly as the `decdn_node` role's `source` mode does (see its README): a sha256-pinned @@ -38,9 +39,9 @@ gate, installers, behind a TLS reverse proxy), is the unchanged commit is not rebuilt, and a root-side install that refuses symlinks. `sponsord_version` is not checked in this mode; set the ref to a release tag (`v`) to build exactly that release. The build user, its - home and the toolchain are shared with the other roles on the host. cargo also fetches upstream's - `decdn/decdn` git dependencies, pinned by `Cargo.lock`. - - `manual` copies a binary built on the control machine, from a `decdn/sponsord` + home and the toolchain are shared with the other roles on the host. The clone is + the whole decdn workspace; `-p sponsord` builds only what the daemon needs. + - `manual` copies a binary built on the control machine, from a `decdn/decdn` checkout or the published crate (`cargo install --locked sponsord@`, which carries no maintainer signature), with `sponsord_manual_bin_src`, or `sponsord_release_target_dir` with an ELF/arch check. @@ -129,10 +130,10 @@ See [`defaults/main.yml`](defaults/main.yml) for the full list with comments. | Variable | Default | Notes | |----------|---------|-------| | `sponsord_install_method` | `release` | `release`, `source` or `manual` | -| `sponsord_source_repo` / `sponsord_source_ref` | `decdn/sponsord` on GitHub / `""` | `source` only. The ref is required: a tag, branch or SHA. | +| `sponsord_source_repo` / `sponsord_source_ref` | `decdn/decdn` on GitHub / `""` | `source` only. The ref is required: a tag, branch or SHA. | | `sponsord_source_build_jobs` | `""` | `source` only: `CARGO_BUILD_JOBS` (`""` = one per CPU). | | `sponsord_manual_bin_src` / `sponsord_release_target_dir` | `""` | `manual` only. Set one. | -| `sponsord_version` | the pinned release line | `release` mode: a `decdn/sponsord` release line (`"X.Y"`, its newest finished release) or an exact `"X.Y.Z"`, quoted; see [`defaults/main.yml`](defaults/main.yml). Equal to `sponsord_onramp_version`. | +| `sponsord_version` | the pinned release line | `release` mode: a `decdn/decdn` release line (`"X.Y"`, its newest finished release) or an exact `"X.Y.Z"`, quoted; see [`defaults/main.yml`](defaults/main.yml). Equal to `sponsord_onramp_version` and to the decdn_node role's `decdn_node_version`. | | `sponsord_network` | `""` | Network profile: `arbitrum-sepolia` supplies chain id and PaymentPool. | | `sponsord_chain_id`, `sponsord_payment_pool_address` | from profile | Explicit values win over the profile. | | `sponsord_pool_id` | `""` | **Required.** 0x + 64 hex. `""` only on the wallet-creation run. | @@ -237,35 +238,17 @@ export it, so watch the address yourself. (no control machine), in release mode, beside the onramp. - `molecule/validation-sponsord` holds the negative cases. -## Bumping the sponsord release - -decdn/sponsord releases `sponsord` and `sponsord-onramp` from one `vX.Y.Z` tag. Both -roles pin its release line, so a patch release in the line reaches deployments on -their next run; only Compose names one release. When upstream cuts a new release (the -weekly `upstream-drift` workflow fails on it): - -- **A patch release (a `vX.Y.N` in the pinned line `X.Y`)** is a repo-only change: step 2, - plus steps 3 and 4 if they apply. -- **A new line (a release outside `X.Y`)** is every step, and `decdn.publisher` needs a release after - it (RELEASING.md). - -1. The line: `sponsord_version` here and `sponsord_onramp_version` in - `roles/sponsord_onramp/defaults/main.yml`, to the same `"X.Y"`. -2. `SPONSORD_IMAGE_DIGEST` and `SPONSORD_ONRAMP_IMAGE_DIGEST` in - `compose/.env.example`, from the release's signed `sponsord-image-digest.txt` - and `sponsord-onramp-image-digest.txt`. -3. `files/sponsord-release-KEYS.asc`, if upstream's `KEYS` changed (and the - decdn_node copy, and `SECURITY.md`'s fingerprints). -4. Re-check the env var names the Compose file and the role render against the new - `crates/*/src/config.rs` (AGENTS.md, `compose/`). -5. The commented examples that name an exact version (`sponsord_version`, - `sponsord_source_ref`, `sponsord_onramp_version` and `sponsord_onramp_source_ref`, - in the inventory templates and their `examples/collection-fleet` copies) and - AGENTS.md's "Pinned upstream releases". Other docs name the pin, never its - number. The changelog entry is the PR title (`feat`, `fix` or `chore`, not `ci` or - `build`, which the changelogs leave out; RELEASING.md). - -`make test-scripts` fails while the two lines, or AGENTS.md's pinned line, disagree; -the weekly `upstream-drift` job checks the Compose digests against the line's newest -release. A decdn/decdn release is bumped the same way -([decdn_node's README](../decdn_node/README.md#bumping-the-decdn-release)). +## Bumping the release + +`sponsord` and `sponsord-onramp` ship from decdn/decdn's release (one `vX.Y.Z` tag for +the whole workspace), so there is no sponsord bump of its own: `sponsord_version` and +`sponsord_onramp_version` are the decdn_node role's `decdn_node_version`, and the decdn +bump moves all three together with the two Compose image digests +([decdn_node's README](../decdn_node/README.md#bumping-the-decdn-release); +`scripts/open-decdn-bump.sh` opens the PR). `make test-scripts` fails while the three +lines, or AGENTS.md's pinned line, disagree; the weekly `upstream-drift` job checks the +Compose digests against the line's newest release. What that bump leaves to a human, +for these two roles: re-check the env var names the Compose file and the roles render +against the new `crates/sponsord/src/config.rs` and `crates/sponsord-onramp/src/config.rs` +(AGENTS.md, `compose/`), and the `sponsord_*` series `monitoring/sponsord/` names against +`crates/sponsord/src/metrics.rs`. diff --git a/ansible/roles/sponsord/defaults/main.yml b/ansible/roles/sponsord/defaults/main.yml index 5f6bbcf..bbd8a68 100644 --- a/ansible/roles/sponsord/defaults/main.yml +++ b/ansible/roles/sponsord/defaults/main.yml @@ -7,12 +7,12 @@ # # Per-host values belong in host_vars// (non-secret in main.yml, the RPC URL # — if the role should author it — in the git-ignored secret.yml). Upstream: -# github.com/decdn/sponsord, docs/operator.md. +# github.com/decdn/decdn (crates/sponsord), docs/sponsord/operator.md. # --- Install method ----------------------------------------------------------- # "release" (default): download the pinned v GitHub Release tarball and -# verify it against the GPG-signed SHA256SUMS. decdn/sponsord releases its whole -# workspace (sponsord, sponsord-onramp) from one vX.Y.Z tag. +# verify it against the GPG-signed SHA256SUMS. One decdn/decdn release (vX.Y.Z) +# ships the node, the CLI, sponsord and sponsord-onramp. # "source": clone sponsord_source_repo at sponsord_source_ref on the host and # `cargo build -p sponsord` there as an unprivileged build user (see "Build from # source" below). @@ -29,7 +29,7 @@ sponsord_release_target_dir: "" # The repo and the ref to build: any tag, branch or commit SHA (REQUIRED in # "source" mode). A branch is re-resolved every run, so a moved branch rebuilds; # an unchanged commit does not. The URL must not embed credentials. -sponsord_source_repo: "https://github.com/decdn/sponsord.git" +sponsord_source_repo: "https://github.com/decdn/decdn.git" sponsord_source_ref: "" # CARGO_BUILD_JOBS for the build ("" = one per CPU); lower it if the host runs # out of memory compiling. @@ -49,26 +49,28 @@ decdn_rustup_sha256: aarch64-unknown-linux-gnu: 15f6e4ce9f583b929c996c91562bad6d4454f3281de858b02cdfdef615fac433 # --- Release to install ("release" mode) ---------------------------------------- -# The decdn/sponsord release to install: a release line, "X.Y", installs the newest +# The decdn/decdn release to install: a release line, "X.Y", installs the newest # finished vX.Y.N release (looked up in sponsord_release_index on every run, so a new # patch is installed and the daemon restarted by the next run), or an exact "X.Y.Z" -# pins that one release. Keep it equal to sponsord_onramp_version: both come from -# one workspace release (README.md, "Bumping the sponsord release"; make -# test-scripts checks the defaults agree). +# pins that one release. Keep it equal to sponsord_onramp_version and to the +# decdn_node role's decdn_node_version: one decdn/decdn release ships all four +# binaries (roles/decdn_node/README.md, "Bumping the decdn release"; make +# test-scripts checks the defaults agree, and scripts/bump_decdn.py moves them together). sponsord_version: "0.2" # Release target triple, derived from the host's gathered architecture (see # vars/main/arch.yml). Set it explicitly only for a play with gather_facts: false. sponsord_target: "{{ _sponsord_arch_targets.get(ansible_facts.get('architecture', ''), '') }}" -sponsord_release_base: "https://github.com/decdn/sponsord/releases/download" +sponsord_release_base: "https://github.com/decdn/decdn/releases/download" # Where a release line is resolved: a JSON list in the shape of GitHub's -# GET /repos/decdn/sponsord/releases. Unauthenticated, GitHub allows 60 requests an +# GET /repos/decdn/decdn/releases. Unauthenticated, GitHub allows 60 requests an # hour per address; pin an exact sponsord_version, or serve this list from a mirror. -sponsord_release_index: "https://api.github.com/repos/decdn/sponsord/releases?per_page=100" +sponsord_release_index: "https://api.github.com/repos/decdn/decdn/releases?per_page=100" # Verify SHA256SUMS.asc against sponsord_release_keyring before trusting the # manifest. Set false only for an air-gapped mirror that strips the signature. sponsord_verify_release_signature: true -# Defaults to the copy of upstream's KEYS vendored with this role. -sponsord_release_keyring: sponsord-release-KEYS.asc +# Defaults to the copy of decdn/decdn's KEYS vendored with this role (a byte copy of +# the decdn_node role's; make test-scripts keeps them identical). +sponsord_release_keyring: decdn-release-KEYS.asc # --- Chain ---------------------------------------------------------------------- # Network profile. Set sponsord_network to a chain upstream's `decdn config init diff --git a/ansible/roles/sponsord/files/sponsord-release-KEYS.asc b/ansible/roles/sponsord/files/decdn-release-KEYS.asc similarity index 90% rename from ansible/roles/sponsord/files/sponsord-release-KEYS.asc rename to ansible/roles/sponsord/files/decdn-release-KEYS.asc index 047854b..b7a6133 100644 --- a/ansible/roles/sponsord/files/sponsord-release-KEYS.asc +++ b/ansible/roles/sponsord/files/decdn-release-KEYS.asc @@ -1,10 +1,9 @@ -Public keys trusted to sign sponsord releases. Import with `gpg --import KEYS`. -The same maintainer keys sign decdn's releases (decdn/decdn's KEYS). +Public keys trusted to sign deCDN releases. Import with `gpg --import KEYS`. -A release tag, the SHA256SUMS manifest, and each image's SBOM and digest file -are each signed by one maintainer's personal key. A good signature from ANY -key in this file is authentic — see SECURITY.md. Adding or removing a -maintainer is a change to this file. +A release tag, the SHA256SUMS manifest, and each container image's SBOM +and digest file are each signed by one maintainer's personal key. A good signature from ANY key in this +file is authentic — see SECURITY.md. Adding or removing a maintainer is a +change to this file. Ant Somers DA75 1570 6F18 73D2 74D8 A369 9E11 A9FF D62D AADB diff --git a/ansible/roles/sponsord/tasks/install.yml b/ansible/roles/sponsord/tasks/install.yml index a4f09cf..4c293fe 100644 --- a/ansible/roles/sponsord/tasks/install.yml +++ b/ansible/roles/sponsord/tasks/install.yml @@ -1,7 +1,7 @@ --- # Install the sponsord binary. Three methods, selected by sponsord_install_method: # "release" (default): download the tarball from the GitHub Release of -# sponsord_version (one tag releases the decdn/sponsord workspace): an exact +# sponsord_version (one decdn/decdn tag releases the whole workspace): an exact # version, or the newest release of a line "X.Y" (tasks/resolve-release.yml). # Verify it against the GPG-signed SHA256SUMS manifest, then extract. # Idempotent and upgrade-aware via a version stamp. @@ -149,7 +149,7 @@ if grep -qE '^\[GNUPG:\] (EXPKEYSIG|REVKEYSIG|EXPSIG)' gpg-status.txt; then echo "SHA256SUMS carries a good signature from an EXPIRED or REVOKED" >&2 echo "release key. Refusing to install. Re-sync" >&2 - echo "roles/sponsord/files/sponsord-release-KEYS.asc from upstream KEYS," >&2 + echo "roles/sponsord/files/decdn-release-KEYS.asc from upstream KEYS," >&2 echo "or verify this release out of band." >&2 exit 1 fi diff --git a/ansible/roles/sponsord/tasks/main.yml b/ansible/roles/sponsord/tasks/main.yml index 2c0f96b..e3b7bc9 100644 --- a/ansible/roles/sponsord/tasks/main.yml +++ b/ansible/roles/sponsord/tasks/main.yml @@ -24,7 +24,7 @@ - sponsord_version is match('^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(\.(0|[1-9][0-9]*)(-[0-9A-Za-z.-]+)?)?$') fail_msg: >- sponsord_install_method "release" (the default) requires sponsord_version: a - release line ("X.Y", the newest vX.Y.N release of decdn/sponsord, looked up in + release line ("X.Y", the newest vX.Y.N release of decdn/decdn, looked up in sponsord_release_index) or an exact version ("X.Y.Z", the vX.Y.Z release), as a quoted string without the "v" (got "{{ sponsord_version }}"). Downloads come from sponsord_release_base. To run something unreleased, set @@ -135,7 +135,7 @@ fail_msg: >- sponsord_install_method "manual" needs sponsord_manual_bin_src (a control-machine path to the sponsord binary) or sponsord_release_target_dir - (the control-machine target/release dir of a decdn/sponsord checkout). + (the control-machine target/release dir of a decdn/decdn checkout). # Internal facts, never a set_fact on sponsord_manual_bin_src itself: an # include_role `vars:` value would outrank it, and a derived path would leak @@ -175,7 +175,7 @@ No sponsord binary under sponsord_release_target_dir ({{ sponsord_release_target_dir }}) or its cross output dir ({{ (_sponsord_target_dir | dirname) ~ '/' ~ sponsord_target ~ '/release' }}). - Build it in a decdn/sponsord checkout — `cargo build --release -p sponsord`, + Build it in a decdn/decdn checkout — `cargo build --release -p sponsord`, or cross-compiling for the host, `cross build --release -p sponsord --target {{ sponsord_target }}`. when: _sponsord_use_dir | bool diff --git a/ansible/roles/sponsord/templates/sponsord.service.j2 b/ansible/roles/sponsord/templates/sponsord.service.j2 index 5406505..5278729 100644 --- a/ansible/roles/sponsord/templates/sponsord.service.j2 +++ b/ansible/roles/sponsord/templates/sponsord.service.j2 @@ -9,7 +9,7 @@ # does reach it as an environment variable, from the 0600 secret.env. [Unit] Description=sponsord: capability signing and pool top-ups -Documentation=https://github.com/decdn/sponsord +Documentation=https://github.com/decdn/decdn/blob/main/docs/sponsord/operator.md After=network-online.target Wants=network-online.target diff --git a/ansible/roles/sponsord/vars/main/arch.yml b/ansible/roles/sponsord/vars/main/arch.yml index db87007..3a45a7b 100644 --- a/ansible/roles/sponsord/vars/main/arch.yml +++ b/ansible/roles/sponsord/vars/main/arch.yml @@ -1,7 +1,7 @@ --- # Host architecture (ansible_facts['architecture']) -> the upstream release target -# triple. These are the two Linux targets decdn/sponsord's release builds (LINUX in -# .github/scripts/release_plan.py); +# triple. These are the two Linux targets decdn/decdn's release builds sponsord for +# (the `builds:` matrix in .github/workflows/release.yml); # the defaults derive sponsord_target from this map and tasks/main.yml asserts the # result is one of its values. Role vars, not defaults: the supported set is a fact # about upstream's release matrix, not an operator knob. diff --git a/ansible/roles/sponsord_onramp/README.md b/ansible/roles/sponsord_onramp/README.md index 9b2bfc8..fd220f0 100644 --- a/ansible/roles/sponsord_onramp/README.md +++ b/ansible/roles/sponsord_onramp/README.md @@ -1,6 +1,6 @@ # Role: `sponsord_onramp` -Deploys `sponsord-onramp` from [decdn/sponsord](https://github.com/decdn/sponsord), +Deploys `sponsord-onramp` from [decdn/decdn](https://github.com/decdn/decdn/tree/main/crates/sponsord-onramp), the public side of the deCDN onboarding sponsor. It serves: - the gate page people pass (Cloudflare Turnstile) before the sponsor pays for @@ -22,17 +22,18 @@ is unreachable. - **Installs the binary**, exactly as the `sponsord` role does: - `release` (the default) downloads `sponsord-onramp--.tar.gz` - from the `v` release of `decdn/sponsord` (`sponsord_onramp_version`, - the same release line, or exact release, as the daemon's `sponsord_version`; a - line resolves to its newest finished release on every run, from - `sponsord_onramp_release_index`) and verifies it against the GPG-signed - `SHA256SUMS`, with the KEYS file vendored in the `sponsord` role. + from the `v` release of `decdn/decdn` (`sponsord_onramp_version`, + the same release line, or exact release, as the daemon's `sponsord_version` and + the node's `decdn_node_version`; a line resolves to its newest finished release + on every run, from `sponsord_onramp_release_index`) and verifies it against the + GPG-signed `SHA256SUMS`, with the decdn KEYS vendored in this role (the same + file the installers' decdn release is checked against). `sponsord_onramp_release_base` points it at a mirror. - `source` builds `sponsord_onramp_source_ref` of `sponsord_onramp_source_repo` on the host (`cargo build -p sponsord-onramp`) as the `sponsord` role's `source` mode does, from its own clone, so the two can pin different refs. Each builds - from scratch, so a co-located host compiles the sponsord workspace twice. - - `manual` copies a binary built in a `decdn/sponsord` checkout (or installed + from scratch, so a co-located host compiles the shared crates twice. + - `manual` copies a binary built in a `decdn/decdn` checkout (or installed with `cargo install --locked sponsord-onramp@`, unsigned). `sponsord_onramp_release_target_dir` defaults to `sponsord_release_target_dir`, so one `cargo build --release` covers both. @@ -121,10 +122,10 @@ See [`defaults/main.yml`](defaults/main.yml) for the full list with comments. | Variable | Default | Notes | |----------|---------|-------| | `sponsord_onramp_install_method` | `release` | `release`, `source` or `manual` | -| `sponsord_onramp_source_repo` / `_source_ref` | `decdn/sponsord` on GitHub / `""` | `source` only. The ref is required: a tag, branch or SHA. | +| `sponsord_onramp_source_repo` / `_source_ref` | `decdn/decdn` on GitHub / `""` | `source` only. The ref is required: a tag, branch or SHA. | | `sponsord_onramp_source_build_jobs` | `""` | `source` only: `CARGO_BUILD_JOBS` (`""` = one per CPU). | | `sponsord_onramp_manual_bin_src` / `_release_target_dir` | `""` / `sponsord_release_target_dir` | `manual` only. | -| `sponsord_onramp_version` | the pinned release line | `release` mode: a `decdn/sponsord` release line (`"X.Y"`, its newest finished release) or an exact `"X.Y.Z"`, quoted; see [`defaults/main.yml`](defaults/main.yml). Keep it equal to `sponsord_version`: overriding one does not move the other. | +| `sponsord_onramp_version` | the pinned release line | `release` mode: a `decdn/decdn` release line (`"X.Y"`, its newest finished release) or an exact `"X.Y.Z"`, quoted; see [`defaults/main.yml`](defaults/main.yml). Keep it equal to `sponsord_version` (and the node's `decdn_node_version`): overriding one does not move the other. | | `sponsord_onramp_network` | `sponsord_network` | Supplies CapacityBond and SlashJudge. Must match the daemon's network. | | `sponsord_onramp_capacity_bond_address`, `_slash_judge_address` | from profile | Explicit values win. SlashJudge may be `""`. | | `sponsord_onramp_rpc_url` | `""` | **Required. Public**: served to every user. No credentials, query or API key. | diff --git a/ansible/roles/sponsord_onramp/defaults/main.yml b/ansible/roles/sponsord_onramp/defaults/main.yml index f546f66..e35b7e3 100644 --- a/ansible/roles/sponsord_onramp/defaults/main.yml +++ b/ansible/roles/sponsord_onramp/defaults/main.yml @@ -7,12 +7,13 @@ # # The onramp listens on loopback. Caddy, installed by this role unless you turn it # off, terminates TLS for sponsord_onramp_domain and is the only public listener -# (AGENTS.md hard rule 2). Upstream: github.com/decdn/sponsord, docs/operator.md. +# (AGENTS.md hard rule 2). Upstream: github.com/decdn/decdn (crates/sponsord-onramp), +# docs/sponsord/operator.md. # --- Install method ----------------------------------------------------------- # "release" (default): download the pinned v GitHub Release tarball and -# verify it against the GPG-signed SHA256SUMS. decdn/sponsord releases its whole -# workspace (sponsord, sponsord-onramp) from one vX.Y.Z tag. +# verify it against the GPG-signed SHA256SUMS. One decdn/decdn release (vX.Y.Z) +# ships the node, the CLI, sponsord and sponsord-onramp. # "source": clone sponsord_onramp_source_repo at sponsord_onramp_source_ref on the host and # `cargo build -p sponsord-onramp` there as an unprivileged build user (see "Build from # source" below). @@ -20,7 +21,7 @@ sponsord_onramp_install_method: release # "release" | "source" | "manual" sponsord_onramp_manual_bin_src: "" # manual: control-machine path to the binary # Alternative to sponsord_onramp_manual_bin_src: the control-machine `target/release` -# dir of a decdn/sponsord checkout (falls back to target//release). The role +# dir of a decdn/decdn checkout (falls back to target//release). The role # checks the binary is an ELF for the host's architecture before shipping it. sponsord_onramp_release_target_dir: "{{ sponsord_release_target_dir | default('') }}" @@ -28,7 +29,7 @@ sponsord_onramp_release_target_dir: "{{ sponsord_release_target_dir | default('' # The repo and the ref to build: any tag, branch or commit SHA (REQUIRED in # "source" mode). A branch is re-resolved every run, so a moved branch rebuilds; # an unchanged commit does not. The URL must not embed credentials. -sponsord_onramp_source_repo: "https://github.com/decdn/sponsord.git" +sponsord_onramp_source_repo: "https://github.com/decdn/decdn.git" sponsord_onramp_source_ref: "" # CARGO_BUILD_JOBS for the build ("" = one per CPU); lower it if the host runs # out of memory compiling. @@ -48,20 +49,21 @@ decdn_rustup_sha256: aarch64-unknown-linux-gnu: 15f6e4ce9f583b929c996c91562bad6d4454f3281de858b02cdfdef615fac433 # --- Release to install ("release" mode) ---------------------------------------- -# The decdn/sponsord release to install, a line ("X.Y", the newest finished vX.Y.N) -# or an exact "X.Y.Z": the same one as the sponsord role's sponsord_version. +# The decdn/decdn release to install, a line ("X.Y", the newest finished vX.Y.N) +# or an exact "X.Y.Z": the same one as the sponsord role's sponsord_version (and +# the decdn_node role's decdn_node_version: one release ships all four binaries). sponsord_onramp_version: "0.2" # Derived from the host's gathered architecture (vars/main/arch.yml). sponsord_onramp_target: "{{ _sponsord_onramp_arch_targets.get(ansible_facts.get('architecture', ''), '') }}" -sponsord_onramp_release_base: "https://github.com/decdn/sponsord/releases/download" -# Where a release line is resolved (GitHub's GET /repos/decdn/sponsord/releases +sponsord_onramp_release_base: "https://github.com/decdn/decdn/releases/download" +# Where a release line is resolved (GitHub's GET /repos/decdn/decdn/releases # shape; 60 unauthenticated requests an hour per address). -sponsord_onramp_release_index: "https://api.github.com/repos/decdn/sponsord/releases?per_page=100" +sponsord_onramp_release_index: "https://api.github.com/repos/decdn/decdn/releases?per_page=100" sponsord_onramp_verify_release_signature: true -# Upstream signs every decdn/sponsord release with the keys in its one KEYS file, -# vendored once, in the sponsord role. Both roles ship side by side (this repo and -# the decdn.publisher collection), so the path holds in either. -sponsord_onramp_release_keyring: "{{ role_path }}/../sponsord/files/sponsord-release-KEYS.asc" +# decdn/decdn's KEYS, vendored with this role: the same file the installers' decdn +# release is checked against (sponsord_onramp_decdn_keyring below), since one +# release ships both. +sponsord_onramp_release_keyring: decdn-release-KEYS.asc # --- Chain ---------------------------------------------------------------------- # The onramp takes the chain id and the PaymentPool from the daemon (/v1/info). It @@ -140,7 +142,8 @@ sponsord_onramp_gate_template_src: "" # --- Release the installers install -------------------------------------------- # The decdn/decdn release the installers download `decdn` from. A release line, -# "vX.Y" (the default follows the decdn_node role's), serves the newest finished +# "vX.Y" (the default follows the decdn_node role's, and so the release the onramp +# itself comes from), serves the newest finished # vX.Y.N, looked up in sponsord_onramp_decdn_release_index on every run, with the # SHA-256 of its SHA256SUMS read from sponsord_onramp_decdn_release_base and # verified against sponsord_onramp_decdn_keyring first. An exact tag, "vX.Y.Z", diff --git a/ansible/roles/sponsord_onramp/tasks/install.yml b/ansible/roles/sponsord_onramp/tasks/install.yml index 1ca6ba5..6245e78 100644 --- a/ansible/roles/sponsord_onramp/tasks/install.yml +++ b/ansible/roles/sponsord_onramp/tasks/install.yml @@ -1,8 +1,8 @@ --- # Install the sponsord-onramp binary. Three methods, selected by sponsord_onramp_install_method: # "release" (default): download the tarball from the GitHub Release of -# sponsord_onramp_version (the decdn/sponsord workspace release that also carries -# sponsord): an exact version, or the newest release of a line "X.Y" +# sponsord_onramp_version (the decdn/decdn release that also carries sponsord +# and the node): an exact version, or the newest release of a line "X.Y" # (tasks/resolve-release.yml). Verify it against the GPG-signed SHA256SUMS # manifest, then extract. # Idempotent and upgrade-aware via a version stamp. @@ -158,7 +158,7 @@ if grep -qE '^\[GNUPG:\] (EXPKEYSIG|REVKEYSIG|EXPSIG)' gpg-status.txt; then echo "SHA256SUMS carries a good signature from an EXPIRED or REVOKED" >&2 echo "release key. Refusing to install. Re-sync" >&2 - echo "roles/sponsord/files/sponsord-release-KEYS.asc from upstream KEYS," >&2 + echo "roles/sponsord_onramp/files/decdn-release-KEYS.asc from upstream KEYS," >&2 echo "or verify this release out of band." >&2 exit 1 fi diff --git a/ansible/roles/sponsord_onramp/tasks/main.yml b/ansible/roles/sponsord_onramp/tasks/main.yml index 0f2fa90..73eea61 100644 --- a/ansible/roles/sponsord_onramp/tasks/main.yml +++ b/ansible/roles/sponsord_onramp/tasks/main.yml @@ -28,7 +28,7 @@ fail_msg: >- sponsord_onramp_install_method "release" (the default) requires sponsord_onramp_version: a release line ("X.Y", the newest vX.Y.N release of - decdn/sponsord, looked up in sponsord_onramp_release_index) or an exact version + decdn/decdn, looked up in sponsord_onramp_release_index) or an exact version ("X.Y.Z", the vX.Y.Z release), as a quoted string without the "v" (got "{{ sponsord_onramp_version }}"), the same as sponsord_version. Downloads come from sponsord_onramp_release_base. To run something unreleased, set sponsord_onramp_install_method "source" with a @@ -138,7 +138,7 @@ sponsord_onramp_install_method "manual" needs sponsord_onramp_manual_bin_src (a control-machine path to the sponsord-onramp binary) or sponsord_onramp_release_target_dir (the control-machine target/release dir of - a decdn/sponsord checkout; it defaults to sponsord_release_target_dir). + a decdn/decdn checkout; it defaults to sponsord_release_target_dir). - name: Decide whether target-dir resolution applies ansible.builtin.set_fact: @@ -174,7 +174,7 @@ No sponsord-onramp binary under sponsord_onramp_release_target_dir ({{ sponsord_onramp_release_target_dir }}) or its cross output dir ({{ (_sponsord_onramp_target_dir | dirname) ~ '/' ~ sponsord_onramp_target ~ '/release' }}). - Build it in a decdn/sponsord checkout — `cargo build --release -p + Build it in a decdn/decdn checkout — `cargo build --release -p sponsord-onramp`, or cross-compiling for the host, `cross build --release -p sponsord-onramp --target {{ sponsord_onramp_target }}`. when: _sponsord_onramp_use_dir | bool diff --git a/ansible/roles/sponsord_onramp/templates/sponsord-onramp.service.j2 b/ansible/roles/sponsord_onramp/templates/sponsord-onramp.service.j2 index 137bdbc..4a25265 100644 --- a/ansible/roles/sponsord_onramp/templates/sponsord-onramp.service.j2 +++ b/ansible/roles/sponsord_onramp/templates/sponsord-onramp.service.j2 @@ -8,7 +8,7 @@ # and hands them to this unit as credentials. [Unit] Description=sponsord-onramp: the public onramp in front of sponsord -Documentation=https://github.com/decdn/sponsord +Documentation=https://github.com/decdn/decdn/blob/main/docs/sponsord/operator.md After=network-online.target sponsord.service Wants=network-online.target sponsord.service # The onramp reads the daemon's limits once, at start; restart it with the daemon. diff --git a/ansible/roles/sponsord_onramp/vars/main/arch.yml b/ansible/roles/sponsord_onramp/vars/main/arch.yml index 0b4387b..199d0db 100644 --- a/ansible/roles/sponsord_onramp/vars/main/arch.yml +++ b/ansible/roles/sponsord_onramp/vars/main/arch.yml @@ -1,7 +1,7 @@ --- # Host architecture (ansible_facts['architecture']) -> the upstream release target -# triple. These are the two Linux targets decdn/sponsord's release builds for -# sponsord-onramp (LINUX in .github/scripts/release_plan.py); the defaults derive +# triple. These are the two Linux targets decdn/decdn's release builds +# sponsord-onramp for (the `builds:` matrix in .github/workflows/release.yml); the defaults derive # sponsord_onramp_target from this map and tasks/main.yml asserts the result is one # of its values. Role vars, not defaults: the supported set is a fact about # upstream's release matrix, not an operator knob. diff --git a/cloud-init/README.md b/cloud-init/README.md index 9b508e5..70cb477 100644 --- a/cloud-init/README.md +++ b/cloud-init/README.md @@ -38,8 +38,8 @@ for the secrets of every group the host is in. > (`release` mode). The `manual` mode would install binaries that nothing verified, > and the Ansible path's `source` mode builds whatever its git ref points at (and > takes many minutes on first boot), so the lint refuses both. The release is the -> newest of the release line the roles pin at `DEVOPS_REF` (`decdn_node_version` for -> decdn/decdn, `sponsord_version` for the sponsor), looked up on every run; set +> newest of the release line the roles pin at `DEVOPS_REF` (`decdn_node_version`; +> the sponsor's `sponsord_version` is the same line), looked up on every run; set > `decdn_node_version` in the user-data to another line, or to an exact `"X.Y.Z"`. > `decdn_node_release_base` points the download at a mirror that serves > `v/{decdn-node,decdn}--.tar.gz`, `SHA256SUMS` and @@ -206,9 +206,9 @@ Until then it serves its bytes as a cache does. > **Releases only.** The sponsor's services install only from GPG-verified release tarballs > (`release` mode), and the lint refuses `manual` and `source`. Both come from one -> decdn/sponsord release (`v`), the newest of the line the roles pin at -> `DEVOPS_REF` unless the user-data sets `sponsord_version` and -> `sponsord_onramp_version`. +> decdn/decdn release (`v`, the same release as the node's), the newest of +> the line the roles pin at `DEVOPS_REF` unless the user-data sets `sponsord_version` +> and `sponsord_onramp_version`. > `sponsord_release_base` and `sponsord_onramp_release_base` point at a mirror; the > signature is still checked against the KEYS vendored in the `sponsord` role. diff --git a/cloud-init/tests/lint.py b/cloud-init/tests/lint.py index 4b8dda2..fbf1365 100755 --- a/cloud-init/tests/lint.py +++ b/cloud-init/tests/lint.py @@ -390,14 +390,14 @@ def filled(v): or PUBLIC_RPC_URL_BAD_CHARS.search(url)): violation("inventory: sponsord_onramp_rpc_url must be a public http(s) URL with no userinfo, query " "or fragment (the onramp serves it to every user; a keyed URL belongs to sponsord)") - # decdn/sponsord ships both daemons in one release, so a user-data that moves one + # One decdn/decdn release ships both daemons, so a user-data that moves one # version off the role default must move the other with it. if "sponsord_onramp_hosts" in groups: sv = values_of(inventory, "sponsord_version") ov = values_of(inventory, "sponsord_onramp_version") if (sv or ov) and not (len(sv) == len(ov) == 1 and str(sv[0]) == str(ov[0])): violation("inventory: set sponsord_version and sponsord_onramp_version together, once each and " - "equal (one decdn/sponsord release ships both)") + "equal (one decdn/decdn release ships both)") releases = values_of(inventory, ONRAMP_PINS[0]) sums = values_of(inventory, ONRAMP_PINS[1]) line = len(releases) == 1 and isinstance(releases[0], str) and ONRAMP_LINE.fullmatch(releases[0]) diff --git a/cloud-init/user-data-publisher.yaml b/cloud-init/user-data-publisher.yaml index 999c023..7fbc8f7 100644 --- a/cloud-init/user-data-publisher.yaml +++ b/cloud-init/user-data-publisher.yaml @@ -106,15 +106,16 @@ write_files: ansible_connection: local ansible_python_interpreter: /usr/bin/python3 vars: - # sponsord, installed from the GPG-verified release tarball. The release is - # the newest of the sponsord role's default line at DEVOPS_REF + # sponsord, installed from the GPG-verified release tarball of decdn/decdn + # (one release ships the node, the CLI, sponsord and the onramp). The release + # is the newest of the sponsord role's default line at DEVOPS_REF # (sponsord_version); set sponsord_version here, and sponsord_onramp_version # to the same, to another line or an exact "X.Y.Z". sponsord_install_method: release # Optional: a mirror that serves v/sponsord--.tar.gz, # SHA256SUMS and SHA256SUMS.asc. The signature is still checked against the # vendored KEYS. - # sponsord_release_base: "https://mirror.example/decdn/sponsord/releases/download" + # sponsord_release_base: "https://mirror.example/decdn/decdn/releases/download" sponsord_network: arbitrum-sepolia # MUST-EDIT: the pool the treasury owns, 0x + 64 hex, printed by `decdn pool open`. sponsord_pool_id: "CHANGE_ME" @@ -125,10 +126,10 @@ write_files: localhost: vars: # sponsord-onramp, installed from the GPG-verified release tarball of the - # same decdn/sponsord release as sponsord (sponsord_onramp_version). + # same decdn/decdn release as sponsord (sponsord_onramp_version). sponsord_onramp_install_method: release # Optional: a mirror, as above, for v/sponsord-onramp--.tar.gz. - # sponsord_onramp_release_base: "https://mirror.example/decdn/sponsord/releases/download" + # sponsord_onramp_release_base: "https://mirror.example/decdn/decdn/releases/download" # Caddy terminates TLS with a certificate from ACME. The domain's A/AAAA records # must point at this VM, and tcp/80 and tcp/443 must reach it. sponsord_onramp_proxy: caddy diff --git a/compose/.env.example b/compose/.env.example index e6ee76c..4e089f3 100644 --- a/compose/.env.example +++ b/compose/.env.example @@ -42,8 +42,9 @@ DECDN_GID=999 DECDN_ENV_FILE=/etc/decdn/decdn.env # --- sponsord and onramp ---------------------------------------------------------- -# The newest release of the decdn/sponsord line sponsord_version pins: one release -# for both images +# From the same decdn/decdn release as the node's digest above: one release ships +# all three images (sponsord_version = decdn_node_version). Each from the release's +# signed sponsord-image-digest.txt / sponsord-onramp-image-digest.txt. SPONSORD_IMAGE_DIGEST=sha256:8f042a0b2cbad8ee50430d7f7933e3e43b82b84a16370101923a43a2e21edb1f # SPONSORD_IMAGE_REPO=ghcr.io/decdn/sponsord SPONSORD_ONRAMP_IMAGE_DIGEST=sha256:c27bd9542c147c879303ed2e08c086a4aaa918ef2fe0c773e957f9648301f52a diff --git a/compose/README.md b/compose/README.md index 5e2ec02..7394fe4 100644 --- a/compose/README.md +++ b/compose/README.md @@ -68,9 +68,10 @@ onramp's domain. Nothing in it is secret. It is the only place Compose's inputs from: the wrapper always names the project (`decdn`) and refuses to run while the shell sets a `COMPOSE_*` variable or one `compose.yaml` reads, which Compose would otherwise take over `.env`, so its checks see what Compose runs. `init` -starts it from `.env.example`, which carries the digests of the newest releases of -the lines the Ansible roles pin (`decdn_node_version` and `sponsord_version`), each -copied from that release's signed digest file. Compose pins by digest only, so it +starts it from `.env.example`, which carries the three digests of the newest release +of the line the Ansible roles pin (`decdn_node_version` = `sponsord_version`: one +decdn/decdn release ships every image), each copied from that release's signed digest +file. Compose pins by digest only, so it names one release where the roles follow the line; [Operate](#operate) says how to take a newer one. @@ -739,19 +740,12 @@ distribution than the image's may not start in it. Build with `cross`, as the up release does. The image must carry the `decdn` CLI too: the wrapper and the node's healthcheck use it. -The sponsord images work the same way. `decdn/sponsord`'s `deploy/Dockerfile` builds -either one from source: - -```bash -sudo docker build -f /deploy/Dockerfile --build-arg BIN=sponsord \ - -t 127.0.0.1:5000/sponsord:dev -sudo docker build -f /deploy/Dockerfile --build-arg BIN=sponsord-onramp \ - -t 127.0.0.1:5000/sponsord-onramp:dev -sudo docker push 127.0.0.1:5000/sponsord:dev && sudo docker push 127.0.0.1:5000/sponsord-onramp:dev -``` - -Then set `SPONSORD_IMAGE_REPO` / `SPONSORD_IMAGE_DIGEST` and -`SPONSORD_ONRAMP_IMAGE_REPO` / `SPONSORD_ONRAMP_IMAGE_DIGEST` in `.env`. +The sponsord images work the same way: they are the `sponsord` and `sponsord-onramp` +stages of decdn/decdn's root `Dockerfile`, built from the same release archives as the +node image (`docker build --target sponsord`; upstream's `RELEASING.md` has the +`dist//` layout the stages copy from). Push both to the local registry, then set +`SPONSORD_IMAGE_REPO` / `SPONSORD_IMAGE_DIGEST` and `SPONSORD_ONRAMP_IMAGE_REPO` / +`SPONSORD_ONRAMP_IMAGE_DIGEST` in `.env`. ## Without the wrapper diff --git a/docs/publishers.md b/docs/publishers.md index 4a8636b..a55459c 100644 --- a/docs/publishers.md +++ b/docs/publishers.md @@ -112,9 +112,9 @@ details: [`iroh_relay`](../ansible/roles/iroh_relay/README.md), ## Day 2 -Upgrades: the roles pin a release line for decdn/decdn (`decdn_node_version`) and for -decdn/sponsord (`sponsord_version` = `sponsord_onramp_version`), and each deploy installs -the line's newest release; the onramp also serves its installers the newest decdn of +Upgrades: the roles pin one decdn/decdn release line (`decdn_node_version` = +`sponsord_version` = `sponsord_onramp_version`: one release ships the node, sponsord and +the onramp), and each deploy installs the line's newest release; the onramp also serves its installers the newest decdn of the line (`sponsord_onramp_decdn_release`). Set an exact `"X.Y.Z"` (for the installers, a `vX.Y.Z` tag with its `SHA256SUMS` digest) to hold a host on one release. Origin nodes on the chart and Compose follow the node operators' notes diff --git a/examples/collection-fleet/inventory/group_vars/sponsord_hosts.yml b/examples/collection-fleet/inventory/group_vars/sponsord_hosts.yml index b59380d..00626d2 100644 --- a/examples/collection-fleet/inventory/group_vars/sponsord_hosts.yml +++ b/examples/collection-fleet/inventory/group_vars/sponsord_hosts.yml @@ -11,15 +11,16 @@ # The API token is generated on the host at /etc/sponsord/api-token. # --- Binary ------------------------------------------------------------------------ -# The default method, "release", installs the newest release of the decdn/sponsord -# line the role pins (sponsord_version "X.Y"), on each deploy; set an exact one only -# to hold the host on it, together with sponsord_onramp_version: +# The default method, "release", installs the newest release of the decdn/decdn line +# the role pins (sponsord_version "X.Y", the decdn_node role's line: one release ships +# the node, sponsord and the onramp), on each deploy; set an exact one only to hold +# the host on it, together with sponsord_onramp_version: # sponsord_version: "0.2.0" # Or build on the host from a git ref (tag, branch or SHA; sponsord_version is not # checked in this mode): # sponsord_install_method: source # sponsord_source_ref: v0.2.0 -# Or copy a control-machine build: a decdn/sponsord checkout's target dir +# Or copy a control-machine build: a decdn/decdn checkout's target dir # (`cargo build --release -p sponsord`; the role checks the ELF matches the host). # sponsord_install_method: manual # sponsord_release_target_dir: /path/to/sponsord/target/release diff --git a/examples/collection-fleet/inventory/group_vars/sponsord_onramp_hosts.yml b/examples/collection-fleet/inventory/group_vars/sponsord_onramp_hosts.yml index 14b6387..f24cfde 100644 --- a/examples/collection-fleet/inventory/group_vars/sponsord_onramp_hosts.yml +++ b/examples/collection-fleet/inventory/group_vars/sponsord_onramp_hosts.yml @@ -17,7 +17,7 @@ # sponsord_onramp_install_method: source # sponsord_onramp_source_ref: v0.2.0 # Or manual, which follows sponsord_release_target_dir (`cargo build --release` in -# a decdn/sponsord checkout builds both binaries): +# a decdn/decdn checkout builds both binaries): # sponsord_onramp_install_method: manual # --- Public address ---------------------------------------------------------------- diff --git a/monitoring/README.md b/monitoring/README.md index a4dc2e1..f3604bd 100644 --- a/monitoring/README.md +++ b/monitoring/README.md @@ -90,7 +90,7 @@ on both. So: ## sponsord -`sponsord/` holds the same pair for the deCDN onboarding sponsor (`decdn/sponsord`), +`sponsord/` holds the same pair for the deCDN onboarding sponsor (decdn/decdn's `sponsord` crate), which runs on the Ansible and Compose paths only. The chart never renders it: its `files/monitoring` symlink reaches `decdn-node/` alone, and `make lint-helm` checks that the packaged chart carries nothing else. @@ -98,7 +98,7 @@ the packaged chart carries nothing else. | File | What it is | |------|------------| | `sponsord/dashboard-sponsord.json` | `uid: decdn-sponsord`: pool balance, the top-up hold, keeper reads and top-ups, capabilities issued, errors by code, the daemon's and the onramp's logs. | -| `sponsord/prometheus-alerts.yml` | Rule group `sponsord`: down, keeper failing, a stale pool read (or none since start), a held top-up (and one held over 2 h), an empty pool, sponsord's own request errors. `runbook_url` points at upstream's `docs/operator.md`, "Monitor". | +| `sponsord/prometheus-alerts.yml` | Rule group `sponsord`: down, keeper failing, a stale pool read (or none since start), a held top-up (and one held over 2 h), an empty pool, sponsord's own request errors. `runbook_url` points at upstream's `docs/sponsord/operator.md`, "Monitor". | They select `job="sponsord"`, which the `grafana_alloy` role (and Compose's `alloy` profile) stamps on sponsord's diff --git a/monitoring/sponsord/dashboard-sponsord.json b/monitoring/sponsord/dashboard-sponsord.json index 1e59c74..d27e473 100644 --- a/monitoring/sponsord/dashboard-sponsord.json +++ b/monitoring/sponsord/dashboard-sponsord.json @@ -30,7 +30,7 @@ } ] }, - "description": "The deCDN onboarding sponsor (decdn/sponsord): pool balance, keeper, top-up hold, issuance and errors, logs. Metrics job=\"sponsord\" as the grafana_alloy role ships them.", + "description": "The deCDN onboarding sponsor (decdn/decdn's sponsord): pool balance, keeper, top-up hold, issuance and errors, logs. Metrics job=\"sponsord\" as the grafana_alloy role ships them.", "editable": true, "graphTooltip": 1, "links": [], diff --git a/monitoring/sponsord/prometheus-alerts.yml b/monitoring/sponsord/prometheus-alerts.yml index 1250a6a..858513a 100644 --- a/monitoring/sponsord/prometheus-alerts.yml +++ b/monitoring/sponsord/prometheus-alerts.yml @@ -1,9 +1,9 @@ # sponsord reference Prometheus alerting rules # -# For the deCDN onboarding sponsor (decdn/sponsord), whose /metrics the -# grafana_alloy role ships as job="sponsord". The series and what upstream asks -# operators to alert on are in upstream docs/operator.md, "Monitor"; the names -# are crates/sponsord/src/metrics.rs. Thresholds are operator-onboarding +# For the deCDN onboarding sponsor (decdn/decdn's `sponsord` crate), whose /metrics +# the grafana_alloy role ships as job="sponsord". The series and what upstream asks +# operators to alert on are in upstream docs/sponsord/operator.md, "Monitor"; the +# names are crates/sponsord/src/metrics.rs. Thresholds are operator-onboarding # defaults, not normative. # # NOT covered: the treasury wallet's own USDC and gas balance, which pays every @@ -65,7 +65,7 @@ groups: component: sponsor severity: warning annotations: - runbook_url: "https://github.com/decdn/sponsord/blob/main/docs/operator.md#5-monitor" + runbook_url: "https://github.com/decdn/decdn/blob/main/docs/sponsord/operator.md#5-monitor" summary: "sponsord pool keeper failing on {{ $labels.instance }}" description: | The keeper failed a chain read or a top-up in the last two @@ -88,7 +88,7 @@ groups: component: sponsor severity: warning annotations: - runbook_url: "https://github.com/decdn/sponsord/blob/main/docs/operator.md#5-monitor" + runbook_url: "https://github.com/decdn/decdn/blob/main/docs/sponsord/operator.md#5-monitor" summary: "sponsord has not read its pool balance for 3h on {{ $labels.instance }}" description: | sponsord_pool_last_check_unix has not advanced for three @@ -103,7 +103,7 @@ groups: component: sponsor severity: warning annotations: - runbook_url: "https://github.com/decdn/sponsord/blob/main/docs/operator.md#5-monitor" + runbook_url: "https://github.com/decdn/decdn/blob/main/docs/sponsord/operator.md#5-monitor" summary: "sponsord has not read its pool balance since it started on {{ $labels.instance }}" description: | sponsord_pool_last_check_unix is still 0 three hours after @@ -119,7 +119,7 @@ groups: component: sponsor severity: warning annotations: - runbook_url: "https://github.com/decdn/sponsord/blob/main/docs/operator.md#5-monitor" + runbook_url: "https://github.com/decdn/decdn/blob/main/docs/sponsord/operator.md#5-monitor" summary: "sponsord is holding pool top-ups on {{ $labels.instance }}" description: | The keeper broadcast a top-up and could not read its receipt, @@ -142,7 +142,7 @@ groups: component: sponsor severity: critical annotations: - runbook_url: "https://github.com/decdn/sponsord/blob/main/docs/operator.md#5-monitor" + runbook_url: "https://github.com/decdn/decdn/blob/main/docs/sponsord/operator.md#5-monitor" summary: "sponsord top-ups held for over 2h on {{ $labels.instance }}" description: | A top-up has sat unconfirmed for more than two hours, so the @@ -162,7 +162,7 @@ groups: component: sponsor severity: critical annotations: - runbook_url: "https://github.com/decdn/sponsord/blob/main/docs/operator.md#5-monitor" + runbook_url: "https://github.com/decdn/decdn/blob/main/docs/sponsord/operator.md#5-monitor" summary: "sponsord pool empty on {{ $labels.instance }}" description: | The pool's remaining balance was 0 at the keeper's last read. diff --git a/scripts/bump_decdn.py b/scripts/bump_decdn.py index ad866b8..559f149 100755 --- a/scripts/bump_decdn.py +++ b/scripts/bump_decdn.py @@ -25,8 +25,10 @@ they come from a checkout of the tag, not from this repo. Of the text that names the version, only what is keyed to a pin moves: the comments beside the pinned values, the commented examples, keyed by a variable name, and AGENTS.md. Docs name the pin, -never its number: decdn/sponsord carries version numbers too, so no rewrite here -could tell which pin a bare "0.0.2" names. +never its number, so no rewrite here has to tell which pin a bare "0.0.2" names. +One decdn/decdn release ships sponsord and its onramp with the node, so the sponsord +roles' lines (sponsord_version, sponsord_onramp_version), their commented examples +and Compose's two sponsord image digests move here too. Every edit is anchored on a pattern that must match exactly once, and the result is read back, so a reshaped file fails the command instead of reporting a bump. Files @@ -54,6 +56,7 @@ LINE_RE = r"[0-9]+\.[0-9]+" NODE_DEFAULTS = "ansible/roles/decdn_node/defaults/main.yml" +SPONSORD_DEFAULTS = "ansible/roles/sponsord/defaults/main.yml" ONRAMP_DEFAULTS = "ansible/roles/sponsord_onramp/defaults/main.yml" ONRAMP_ENV = "compose/sponsord-onramp.env.example" COMPOSE_ENV = "compose/.env.example" @@ -71,13 +74,31 @@ for before, after in ((r'^# decdn_node_version: *"', '"'), (r"^# +decdn_node_source_ref: *v", "$")) ), *( - (f"{tree}/group_vars/sponsord_onramp_hosts.yml", r"^# sponsord_onramp_decdn_release: *v", "$") + (f"{tree}/group_vars/sponsord_hosts.yml", before, after) for tree in ("ansible/inventory", "examples/collection-fleet/inventory") + for before, after in ((r'^# sponsord_version: *"', '"'), (r"^# sponsord_source_ref: *v", "$")) + ), + *( + (f"{tree}/group_vars/sponsord_onramp_hosts.yml", before, after) + for tree in ("ansible/inventory", "examples/collection-fleet/inventory") + for before, after in ( + (r'^# sponsord_onramp_version: *"', '"'), + (r"^# sponsord_onramp_source_ref: *v", "$"), + (r"^# sponsord_onramp_decdn_release: *v", "$"), + ) ), ("cloud-init/user-data-publisher.yaml", r'^ *# sponsord_onramp_decdn_release: *"v', '"'), ] -# The one place prose names the pinned line (its line may wrap inside the phrase). -AGENTS = ("AGENTS.md", r"`decdn_node_version`, today\s+`", "`") +# The one place prose names the pinned line (its line may wrap inside the phrase): the +# three role pins in one parenthesis, the number after the last. +AGENTS = ("AGENTS.md", r"`sponsord_onramp_version`, today\s+`", "`") +# Compose's image digests: the env key and the flag that carries the new one (each +# from the release's -image-digest.txt, which open-decdn-bump.sh verifies). +IMAGE_DIGESTS = ( + ("DECDN_IMAGE_DIGEST", "--image-digest"), + ("SPONSORD_IMAGE_DIGEST", "--sponsord-image-digest"), + ("SPONSORD_ONRAMP_IMAGE_DIGEST", "--onramp-image-digest"), +) class ShapeError(Exception): @@ -114,6 +135,14 @@ def pinned(root: Path) -> str: return current +# The sponsord roles' pins, which the same release ships: (file, variable). +LINE_PINS = ( + (NODE_DEFAULTS, "decdn_node_version"), + (SPONSORD_DEFAULTS, "sponsord_version"), + (ONRAMP_DEFAULTS, "sponsord_onramp_version"), +) + + def synced(root: Path) -> str: """The release the exact copies name: the one Compose's image digest is of.""" m = re.findall(rf"^# decdn/decdn v({SEMVER})\nDECDN_IMAGE_DIGEST=", read(root, COMPOSE_ENV), re.M) @@ -185,33 +214,45 @@ def replace_everywhere(root: Path, old: str, new: str) -> list[str]: return changed -def rewrite(root: Path, version: str, sums: str, digest: str) -> list[str]: +def rewrite(root: Path, version: str, sums: str, digests: dict[str, str]) -> list[str]: + """Move every pin to ; maps each IMAGE_DIGESTS env key to its new value.""" if not VERSION.match(version): raise ShapeError(f"{version} is not X.Y.Z") if not HEX64.match(sums): raise ShapeError(f"--sums-sha256 {sums!r} is not 64 lowercase hex") - if not (digest.startswith("sha256:") and HEX64.match(digest[len("sha256:"):])): - raise ShapeError(f"--image-digest {digest!r} is not sha256:<64 lowercase hex>") + if set(digests) != {key for key, _ in IMAGE_DIGESTS}: + raise ShapeError(f"digests given for {sorted(digests)}, expected {[key for key, _ in IMAGE_DIGESTS]}") + for key, flag in IMAGE_DIGESTS: + digest = digests[key] + if not (digest.startswith("sha256:") and HEX64.match(digest[len("sha256:"):])): + raise ShapeError(f"{flag} {digest!r} is not sha256:<64 lowercase hex>") line = line_of(version) pinned(root) # a line, or the shape is not one this script moves + compose_env = read(root, COMPOSE_ENV) old_sums = env_value(read(root, ONRAMP_ENV), "ONRAMP_DECDN_SUMS_SHA256", ONRAMP_ENV) - old_digest = env_value(read(root, COMPOSE_ENV), "DECDN_IMAGE_DIGEST", COMPOSE_ENV) - for name, old in (("ONRAMP_DECDN_SUMS_SHA256", old_sums), ("DECDN_IMAGE_DIGEST", old_digest)): + old_digests = {key: env_value(compose_env, key, COMPOSE_ENV) for key, _ in IMAGE_DIGESTS} + for name, old in (("ONRAMP_DECDN_SUMS_SHA256", old_sums), *old_digests.items()): bare = old.removeprefix("sha256:") if not HEX64.match(bare): raise ShapeError(f"the current {name} {old!r} is not a digest") + # Three distinct images: a digest shared between two env keys would be replaced + # twice, so a copy of the pin could end up as the wrong image. + if len(set(old_digests.values())) != len(old_digests): + raise ShapeError(f"{COMPOSE_ENV}: two image digests are equal") changed: set[str] = set() if old_sums != sums: changed.update(replace_everywhere(root, old_sums, sums)) - if old_digest != digest: - changed.update(replace_everywhere(root, old_digest.removeprefix("sha256:"), digest.removeprefix("sha256:"))) + for key, _ in IMAGE_DIGESTS: + if old_digests[key] != digests[key]: + changed.update(replace_everywhere( + root, old_digests[key].removeprefix("sha256:"), digests[key].removeprefix("sha256:"))) # (file, pattern, replacement): the version-bearing lines. The line copies first. agents_file, agents_before, agents_after = AGENTS edits = [ - (NODE_DEFAULTS, r'^(decdn_node_version: *")[^"]*(")', rf"\g<1>{line}\g<2>"), + *((rel, rf'^({var}: *")[^"]*(")', rf"\g<1>{line}\g<2>") for rel, var in LINE_PINS), (ONRAMP_DEFAULTS, rf"^(sponsord_onramp_decdn_release: *)v{LINE_RE}( )", rf"\g<1>v{line}\g<2>"), (CHART, r'^(appVersion: *")[^"]*(")', rf"\g<1>{line}\g<2>"), (CHART, rf"(image: ghcr\.io/decdn/decdn-node:){LINE_RE}$", rf"\g<1>{line}"), @@ -229,24 +270,26 @@ def rewrite(root: Path, version: str, sums: str, digest: str) -> list[str]: (root / rel).write_text(new) changed.add(rel) - verify(root, version, sums, digest) + verify(root, version, sums, digests) return sorted(changed) -def verify(root: Path, version: str, sums: str, digest: str) -> None: +def verify(root: Path, version: str, sums: str, digests: dict[str, str]) -> None: """Read every pin back; a mismatch is a file this script does not understand.""" line = line_of(version) onramp = read(root, ONRAMP_DEFAULTS) onramp_env = read(root, ONRAMP_ENV) + compose_env = read(root, COMPOSE_ENV) chart = read(root, CHART) want = [ (NODE_DEFAULTS, "decdn_node_version", pinned(root), line), + *((rel, var, scalar(read(root, rel), var, rel), line) for rel, var in LINE_PINS[1:]), (ONRAMP_DEFAULTS, "sponsord_onramp_decdn_release", scalar(onramp, "sponsord_onramp_decdn_release", ONRAMP_DEFAULTS), f"v{line}"), (ONRAMP_ENV, "ONRAMP_DECDN_RELEASE", env_value(onramp_env, "ONRAMP_DECDN_RELEASE", ONRAMP_ENV), f"v{version}"), (ONRAMP_ENV, "ONRAMP_DECDN_SUMS_SHA256", env_value(onramp_env, "ONRAMP_DECDN_SUMS_SHA256", ONRAMP_ENV), sums), - (COMPOSE_ENV, "DECDN_IMAGE_DIGEST", env_value(read(root, COMPOSE_ENV), "DECDN_IMAGE_DIGEST", COMPOSE_ENV), digest), + *((COMPOSE_ENV, key, env_value(compose_env, key, COMPOSE_ENV), digests[key]) for key, _ in IMAGE_DIGESTS), (COMPOSE_ENV, "the release above DECDN_IMAGE_DIGEST", synced(root), version), (CHART, "appVersion", scalar(chart, "appVersion", CHART), line), ] @@ -260,8 +303,9 @@ def verify(root: Path, version: str, sums: str, digest: str) -> None: if images != [line]: raise ShapeError(f"{CHART}: artifacthub.io/images names {images}, expected [{line!r}]") example = re.findall(rf"e\.g\. for v{re.escape(version)}\n *# *(sha256:[0-9a-f]{{64}})", read(root, CHART_VALUES)) - if example != [digest]: - raise ShapeError(f"{CHART_VALUES}: the example digest for v{version} is {example}, expected [{digest!r}]") + if example != [digests["DECDN_IMAGE_DIGEST"]]: + raise ShapeError(f"{CHART_VALUES}: the example digest for v{version} is {example}, " + f"expected [{digests['DECDN_IMAGE_DIGEST']!r}]") for rel, before, after in EXAMPLES: found = re.findall(rf"{before}({SEMVER}){after}", read(root, rel), re.M) if found != [version]: @@ -283,7 +327,8 @@ def main() -> None: rw = sub.add_parser("rewrite") rw.add_argument("version") rw.add_argument("--sums-sha256", required=True) - rw.add_argument("--image-digest", required=True) + for key, flag in IMAGE_DIGESTS: + rw.add_argument(flag, required=True, dest=key, metavar="sha256:HEX", help=f"the release's {key}") args = ap.parse_args() try: @@ -296,7 +341,8 @@ def main() -> None: elif args.cmd == "kind": print(kind(args.root, args.tag)) elif args.cmd == "rewrite": - for rel in rewrite(args.root, args.version, args.sums_sha256, args.image_digest): + digests = {key: getattr(args, key) for key, _ in IMAGE_DIGESTS} + for rel in rewrite(args.root, args.version, args.sums_sha256, digests): print(f"wrote {rel}") except ShapeError as e: print(f"bump_decdn: {e}", file=sys.stderr) diff --git a/scripts/open-decdn-bump.sh b/scripts/open-decdn-bump.sh index 408097c..4dd6a98 100755 --- a/scripts/open-decdn-bump.sh +++ b/scripts/open-decdn-bump.sh @@ -20,13 +20,14 @@ # exact release: Compose, the examples and the generated mirrors, a repo-only # change. A release that starts a new line moves the line as well, and the # collections and the chart need a release after it. -# Release facts (the SHA256SUMS digest, the decdn-node image digest) come from -# the release's assets once their signatures verify against the vendored +# Release facts (the SHA256SUMS digest, the three image digests: one decdn/decdn +# release ships the node, the CLI, sponsord and the onramp) come from the +# release's assets once their signatures verify against the vendored # decdn-release-KEYS.asc; the generated mirrors and KEYS come from a clone of the # tag, whose signature is checked the same way. What needs a human is listed in # the PR body: a schema-key change (node.toml.j2 and the chart's values.config -# follow it by hand), a KEYS change (the sponsord copy and SECURITY.md follow -# it), and any generator that failed. +# follow it by hand), a KEYS change (SECURITY.md's fingerprints follow it), and +# any generator that failed. # # Nothing happens when the pin is already at that release or a later one, or # when a PR from bump/decdn- exists in any state: a closed bump PR is never @@ -48,7 +49,13 @@ DEVOPS=decdn/devops # location. SCRIPTS=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) KEYS=ansible/roles/decdn_node/files/decdn-release-KEYS.asc +# The sponsord roles' byte copies of it (make test-scripts keeps all three equal). +KEYS_COPIES=(ansible/roles/sponsord/files/decdn-release-KEYS.asc + ansible/roles/sponsord_onramp/files/decdn-release-KEYS.asc) SCHEMA_KEYS=ansible/molecule/schema/files/schema-keys.txt +# The images whose digests Compose pins, by Dockerfile stage (= the digest file's +# name and the image's path under ghcr.io/decdn/), with bump_decdn.py's flag. +IMAGES=(decdn-node:--image-digest sponsord:--sponsord-image-digest sponsord-onramp:--onramp-image-digest) die() { echo "error: $*" >&2; exit 1; } usage() { sed -n '5p' "${BASH_SOURCE[0]}" | sed 's/^# //' >&2; exit 2; } @@ -155,31 +162,38 @@ mkdir -m 700 "$GNUPG" gpg --homedir "$GNUPG" --batch --quiet --import "$KEYS" 2>/dev/null || die "$KEYS is not a valid OpenPGP keyring" ASSETS="$WORK/assets" -gh release download "$TAG" -R "$DECDN" -D "$ASSETS" \ - -p SHA256SUMS -p SHA256SUMS.asc \ - -p decdn-node-image-digest.txt -p decdn-node-image-digest.txt.asc || - die "could not download $TAG's SHA256SUMS and decdn-node-image-digest.txt (and their .asc)" +patterns=(-p SHA256SUMS -p SHA256SUMS.asc) +for img in "${IMAGES[@]}"; do + patterns+=(-p "${img%%:*}-image-digest.txt" -p "${img%%:*}-image-digest.txt.asc") +done +gh release download "$TAG" -R "$DECDN" -D "$ASSETS" "${patterns[@]}" || + die "could not download $TAG's SHA256SUMS and *-image-digest.txt (and their .asc)" # A signature by a key the vendored KEYS does not hold fails here, including # upstream's next key after a rotation: vendor the new KEYS by hand (README step # 6) and run this again. -for f in SHA256SUMS decdn-node-image-digest.txt; do +for f in SHA256SUMS "${IMAGES[@]/%:*/-image-digest.txt}"; do + [[ -f "$ASSETS/$f" && -f "$ASSETS/$f.asc" ]] || die "$TAG has no $f (and its .asc); no PR opened" gpg --homedir "$GNUPG" --batch --verify "$ASSETS/$f.asc" "$ASSETS/$f" 2>"$WORK/gpg.log" || { cat "$WORK/gpg.log" >&2 die "$TAG's $f does not verify against $KEYS" } done -# A signature says a maintainer signed the file, not for which release: neither -# file is versioned by its name. SHA256SUMS lists archives named for the -# version, and the digest file is one line naming the decdn-node image, so a -# signed file copied from another release, or the CLI image's digest file, -# stops here. -grep -qE "^[0-9a-f]{64} decdn-node-${VERSION//./\\.}-" "$ASSETS/SHA256SUMS" || - die "$TAG's SHA256SUMS lists no decdn-node-$VERSION archive" +# A signature says a maintainer signed the file, not for which release: no file +# is versioned by its name. SHA256SUMS lists archives named for the version, and +# each digest file is one line naming its image, so a signed file copied from +# another release, or the CLI image's digest file under another name, stops here. +DIGEST_FLAGS=() +for img in "${IMAGES[@]}"; do + stage=${img%%:*} + grep -qE "^[0-9a-f]{64} ${stage}-${VERSION//./\\.}-" "$ASSETS/SHA256SUMS" || + die "$TAG's SHA256SUMS lists no $stage-$VERSION archive" + line=$(<"$ASSETS/$stage-image-digest.txt") + [[ "$line" =~ ^ghcr\.io/decdn/${stage}@(sha256:[0-9a-f]{64})$ ]] || + die "$TAG's $stage-image-digest.txt is not one ghcr.io/decdn/$stage@sha256: line" + DIGEST_FLAGS+=("${img#*:}" "${BASH_REMATCH[1]}") + [[ "$stage" != decdn-node ]] || DIGEST=${BASH_REMATCH[1]} +done SUMS=$(sha256sum "$ASSETS/SHA256SUMS" | cut -d' ' -f1) -line=$(<"$ASSETS/decdn-node-image-digest.txt") -[[ "$line" =~ ^ghcr\.io/decdn/decdn-node@(sha256:[0-9a-f]{64})$ ]] || - die "$TAG's decdn-node-image-digest.txt is not one ghcr.io/decdn/decdn-node@sha256: line" -DIGEST=${BASH_REMATCH[1]} UPSTREAM="$WORK/decdn" git -c advice.detachedHead=false clone --quiet --depth 1 --branch "$TAG" \ @@ -204,7 +218,7 @@ need() { NEEDS+=("$item") } -bump rewrite "$VERSION" --sums-sha256 "$SUMS" --image-digest "$DIGEST" >/dev/null || +bump rewrite "$VERSION" --sums-sha256 "$SUMS" "${DIGEST_FLAGS[@]}" >/dev/null || die "could not move the pins to $TAG; no PR opened" # Step 2: the generated mirrors, from the tag. --check first: each mirror records the @@ -239,13 +253,12 @@ if [[ "$KIND" == patch ]] && { [[ -n "$MIRRORS_MOVED" ]] || ! git diff --quiet - \`decdn.publisher\` after merging (RELEASING.md)." fi -# Step 6: KEYS is a byte copy of upstream's. +# Step 6: KEYS and the sponsord roles' copies are byte copies of upstream's. if ! cmp -s "$UPSTREAM/KEYS" "$KEYS"; then cp "$UPSTREAM/KEYS" "$KEYS" - cp "$UPSTREAM/KEYS" ansible/roles/sponsord_onramp/files/decdn-release-KEYS.asc - need "decdn's \`KEYS\` changed and is copied to \`$KEYS\` and the onramp's - \`ansible/roles/sponsord_onramp/files/decdn-release-KEYS.asc\`: update the sponsord copy - (\`ansible/roles/sponsord/files/sponsord-release-KEYS.asc\`) and SECURITY.md's fingerprints." + for copy in "${KEYS_COPIES[@]}"; do cp "$UPSTREAM/KEYS" "$copy"; done + need "decdn's \`KEYS\` changed and is copied to \`$KEYS\` and the sponsord roles' + \`files/decdn-release-KEYS.asc\`: update SECURITY.md's fingerprints." fi # ---- PR body ------------------------------------------------------------- @@ -257,16 +270,16 @@ BODY="$WORK/body.md" { if [[ "$KIND" == patch ]]; then echo "Moves what names one decdn/decdn release to [${TAG}](https://github.com/${DECDN}/releases/tag/${TAG}) from \`v${CURRENT}\`:" - echo "Compose's image digest and onramp installer pin, the chart's example digest, the exact-pin" - echo "examples and the generated mirrors. The roles and the chart pin the line \`${LINE}\` and" + echo "Compose's three image digests and onramp installer pin, the chart's example digest, the" + echo "exact-pin examples and the generated mirrors. The roles and the chart pin the line \`${LINE}\` and" echo "pick ${TAG} up on their own." else echo "Moves the decdn/decdn pin to the line \`${TAG%.*}\` and [${TAG}](https://github.com/${DECDN}/releases/tag/${TAG}), from \`${LINE}\` and \`v${CURRENT}\`." fi echo echo "- decdn changes: https://github.com/${DECDN}/compare/v${CURRENT}...${TAG}" - echo "- \`SHA256SUMS\` (\`${SUMS}\`) and the \`decdn-node\` image digest (\`${DIGEST}\`)" - echo " come from the release's assets, verified against \`${KEYS}\`." + echo "- \`SHA256SUMS\` (\`${SUMS}\`) and the image digests (\`decdn-node\`: \`${DIGEST}\`; \`sponsord\`" + echo " and \`sponsord-onramp\` likewise) come from the release's assets, verified against \`${KEYS}\`." echo if ((${#NEEDS[@]})); then echo "**Needs work:**" @@ -304,14 +317,14 @@ git add -u || die "git add failed; no PR opened" # --no-verify: CI is the gate on this PR. if [[ "$KIND" == patch ]]; then SUMMARY="Move what names one decdn/decdn release from v${CURRENT} to ${TAG} -(${UPSTREAM_SHA}): the Compose image digest and onramp installer pin, the +(${UPSTREAM_SHA}): the Compose image digests and onramp installer pin, the chart's example digest, the inventory and cloud-init exact-pin examples and the generated mirrors. The roles and the chart pin the line ${LINE}." else SUMMARY="Move the decdn/decdn pin from the line ${LINE} (v${CURRENT}) to ${TAG} -(${UPSTREAM_SHA}): the role default, the onramp's installer line, the chart's -appVersion and Artifact Hub image, AGENTS.md's pinned line, and what names one -release (Compose, the examples, the generated mirrors)." +(${UPSTREAM_SHA}): the three role defaults, the onramp's installer line, the +chart's appVersion and Artifact Hub image, AGENTS.md's pinned line, and what +names one release (Compose, the examples, the generated mirrors)." fi git commit --quiet --no-verify -F - < str: @@ -63,7 +72,8 @@ def setUp(self) -> None: self.line = bd.pinned(self.root) self.version = bd.synced(self.root) self.sums = bd.env_value(self.text(bd.ONRAMP_ENV), "ONRAMP_DECDN_SUMS_SHA256", bd.ONRAMP_ENV) - self.digest = bd.env_value(self.text(bd.COMPOSE_ENV), "DECDN_IMAGE_DIGEST", bd.COMPOSE_ENV) + self.digests = {key: bd.env_value(self.text(bd.COMPOSE_ENV), key, bd.COMPOSE_ENV) for key in KEYS} + self.digest = self.digests["DECDN_IMAGE_DIGEST"] def tearDown(self) -> None: self._tmp.cleanup() @@ -83,6 +93,11 @@ class Compare(Fixture): def test_the_synced_release_is_in_the_line(self) -> None: self.assertEqual(self.version.rsplit(".", 1)[0], self.line) + def test_the_three_roles_pin_one_line(self) -> None: + for rel, var in bd.LINE_PINS: + with self.subTest(var=var): + self.assertEqual(bd.scalar(self.text(rel), var, rel), self.line) + def test_relations(self) -> None: major, minor, _ = (int(p) for p in self.version.split(".")) self.assertEqual(bd.compare(self.root, f"v{self.version}"), "same") @@ -116,97 +131,115 @@ def test_refuses_an_exact_role_pin(self) -> None: class Rewrite(Fixture): - def assert_exact_copies(self, version: str, sums: str, digest: str) -> None: + def assert_exact_copies(self, version: str, sums: str, digests: dict[str, str]) -> None: self.assertIn(f"ONRAMP_DECDN_RELEASE=v{version}\n", self.text(bd.ONRAMP_ENV)) self.assertIn(f"last moved:\n# decdn/decdn v{version}.\n", self.text(bd.ONRAMP_ENV)) self.assertIn(f"ONRAMP_DECDN_SUMS_SHA256={sums}\n", self.text(bd.ONRAMP_ENV)) - self.assertIn(f"# decdn/decdn v{version}\nDECDN_IMAGE_DIGEST={digest}\n", self.text(bd.COMPOSE_ENV)) - self.assertIn(f"e.g. for v{version}\n # {digest}\n", self.text(bd.CHART_VALUES)) - for tree in ("ansible/inventory", "examples/collection-fleet/inventory"): + compose_env = self.text(bd.COMPOSE_ENV) + self.assertIn(f"# decdn/decdn v{version}\nDECDN_IMAGE_DIGEST={digests['DECDN_IMAGE_DIGEST']}\n", compose_env) + for key in KEYS: + self.assertIn(f"\n{key}={digests[key]}\n", compose_env) + self.assertIn(f"e.g. for v{version}\n # {digests['DECDN_IMAGE_DIGEST']}\n", self.text(bd.CHART_VALUES)) + for tree in TREES: for host in ("decdn-node-1", "origin-1"): host_vars = self.text(f"{tree}/host_vars/{host}/main.yml") self.assertIn(f'\n# decdn_node_version: "{version}"\n', host_vars) self.assertIn(f" decdn_node_source_ref: v{version}\n", host_vars) - self.assertIn(f"\n# sponsord_onramp_decdn_release: v{version}\n", - self.text(f"{tree}/group_vars/sponsord_onramp_hosts.yml")) + sponsord = self.text(f"{tree}/group_vars/sponsord_hosts.yml") + self.assertIn(f'\n# sponsord_version: "{version}"\n', sponsord) + self.assertIn(f"\n# sponsord_source_ref: v{version}\n", sponsord) + onramp = self.text(f"{tree}/group_vars/sponsord_onramp_hosts.yml") + self.assertIn(f'\n# sponsord_onramp_version: "{version}"\n', onramp) + self.assertIn(f"\n# sponsord_onramp_source_ref: v{version}\n", onramp) + self.assertIn(f"\n# sponsord_onramp_decdn_release: v{version}\n", onramp) self.assertIn(f'# sponsord_onramp_decdn_release: "v{version}"\n', self.text("cloud-init/user-data-publisher.yaml")) def assert_line_copies(self, line: str) -> None: self.assertEqual(bd.pinned(self.root), line) + for rel, var in bd.LINE_PINS: + self.assertIn(f'\n{var}: "{line}"\n', self.text(rel)) self.assertIn(f"sponsord_onramp_decdn_release: v{line} ", self.text(bd.ONRAMP_DEFAULTS)) self.assertIn(f'appVersion: "{line}"', self.text(bd.CHART)) self.assertIn(f"image: ghcr.io/decdn/decdn-node:{line}\n", self.text(bd.CHART)) - self.assertIn(f"`decdn_node_version`, today `{line}`)", flat(self.text("AGENTS.md"))) + self.assertIn(f"`sponsord_onramp_version`, today `{line}`)", flat(self.text("AGENTS.md"))) def test_a_new_line_moves_every_pin(self) -> None: - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) self.assert_line_copies("9.8") - self.assert_exact_copies("9.8.7", SUMS, DIGEST) + self.assert_exact_copies("9.8.7", SUMS, DIGESTS) self.assertEqual(bd.synced(self.root), "9.8.7") def test_a_patch_leaves_the_line(self) -> None: - line_files = (bd.NODE_DEFAULTS, bd.ONRAMP_DEFAULTS, bd.CHART, "AGENTS.md") - changed = bd.rewrite(self.root, self.next_patch(), SUMS, DIGEST) + line_files = (bd.NODE_DEFAULTS, bd.SPONSORD_DEFAULTS, bd.ONRAMP_DEFAULTS, bd.CHART, "AGENTS.md") + changed = bd.rewrite(self.root, self.next_patch(), SUMS, DIGESTS) for rel in line_files: with self.subTest(rel=rel): self.assertNotIn(rel, changed) self.assertEqual(self.text(rel), self.original[rel]) self.assert_line_copies(self.line) - self.assert_exact_copies(self.next_patch(), SUMS, DIGEST) + self.assert_exact_copies(self.next_patch(), SUMS, DIGESTS) def test_round_trip_is_byte_identical(self) -> None: - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) - bd.rewrite(self.root, self.version, self.sums, self.digest) + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) + bd.rewrite(self.root, self.version, self.sums, self.digests) for rel in FILES: with self.subTest(rel=rel): self.assertEqual(self.text(rel), self.original[rel]) - def test_leaves_the_sponsord_pin(self) -> None: - onramp = self.text(bd.ONRAMP_DEFAULTS) - sponsord = bd.scalar(onramp, "sponsord_onramp_version", bd.ONRAMP_DEFAULTS) - agents = self.text("AGENTS.md") - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) - self.assertEqual(bd.scalar(self.text(bd.ONRAMP_DEFAULTS), "sponsord_onramp_version", - bd.ONRAMP_DEFAULTS), sponsord) - # AGENTS.md names both pins in one sentence; only decdn's moves. - sponsord_today = f"`sponsord_onramp_version`, today `{sponsord}`;" - self.assertIn(sponsord_today, flat(agents)) - self.assertIn(sponsord_today, flat(self.text("AGENTS.md"))) + def test_moves_the_sponsord_pins_with_the_node(self) -> None: + # One decdn/decdn release ships sponsord and its onramp: their lines and + # Compose's two sponsord digests move with the node's, and no copy of an old + # sponsord digest survives in a tracked file. + old = [self.digests[k].removeprefix("sha256:") for k in ("SPONSORD_IMAGE_DIGEST", "SPONSORD_ONRAMP_IMAGE_DIGEST")] + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) + self.assertEqual(bd.scalar(self.text(bd.SPONSORD_DEFAULTS), "sponsord_version", bd.SPONSORD_DEFAULTS), "9.8") + self.assertEqual(bd.scalar(self.text(bd.ONRAMP_DEFAULTS), "sponsord_onramp_version", bd.ONRAMP_DEFAULTS), "9.8") + for rel in FILES: + for digest in old: + self.assertNotIn(digest, self.text(rel), rel) def test_moves_agents_md_wrapped_either_way(self) -> None: original = self.text("AGENTS.md") for sep in (" ", "\n"): with self.subTest(sep=sep): - text, n = re.subn(r"(`decdn_node_version`, today)\s+`", rf"\g<1>{sep}`", original) + text, n = re.subn(r"(`sponsord_onramp_version`, today)\s+`", rf"\g<1>{sep}`", original) self.assertEqual(n, 1) self.write("AGENTS.md", text) - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) - self.assertIn(f"`decdn_node_version`, today{sep}`9.8`", self.text("AGENTS.md")) - bd.rewrite(self.root, self.version, self.sums, self.digest) + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) + self.assertIn(f"`sponsord_onramp_version`, today{sep}`9.8`", self.text("AGENTS.md")) + bd.rewrite(self.root, self.version, self.sums, self.digests) def test_leaves_the_changelogs(self) -> None: - image = self.digest.removeprefix("sha256:") + images = [d.removeprefix("sha256:") for d in self.digests.values()] for rel in CHANGELOGS: - self.write(rel, self.text(rel) + f"\n{self.sums}\n{image}\n") + self.write(rel, self.text(rel) + "\n" + "\n".join([self.sums, *images]) + "\n") subprocess.run(["git", "-C", str(self.root), "add", "-A"], check=True) - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) for rel in CHANGELOGS: self.assertIn(self.sums, self.text(rel)) - self.assertIn(image, self.text(rel)) + for image in images: + self.assertIn(image, self.text(rel)) def test_refuses_a_reshaped_file(self) -> None: self.write(bd.COMPOSE_ENV, self.text(bd.COMPOSE_ENV).replace(f"# decdn/decdn v{self.version}\n", "")) with self.assertRaisesRegex(bd.ShapeError, "compose/.env.example: 0 matches"): - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) def test_refuses_a_second_pin_line(self) -> None: self.write(bd.CHART, self.text(bd.CHART) + f'appVersion: "{self.line}"\n') with self.assertRaisesRegex(bd.ShapeError, "2 matches"): - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) def test_refuses_bad_arguments(self) -> None: - for args in (("9.8", SUMS, DIGEST), ("9.8.7", "A" * 64, DIGEST), ("9.8.7", SUMS, "b" * 64)): + cases = [ + ("9.8", SUMS, DIGESTS), + ("9.8.7", "A" * 64, DIGESTS), + ("9.8.7", SUMS, {**DIGESTS, "DECDN_IMAGE_DIGEST": "b" * 64}), + ("9.8.7", SUMS, {**DIGESTS, "SPONSORD_IMAGE_DIGEST": "sha256:" + "C" * 64}), + ("9.8.7", SUMS, {k: v for k, v in DIGESTS.items() if k != "SPONSORD_ONRAMP_IMAGE_DIGEST"}), + ] + for args in cases: with self.subTest(args=args), self.assertRaises(bd.ShapeError): bd.rewrite(self.root, *args) @@ -215,45 +248,61 @@ def test_refuses_a_digest_beside_the_line(self) -> None: self.write(bd.ONRAMP_DEFAULTS, self.text(bd.ONRAMP_DEFAULTS).replace( 'sponsord_onramp_decdn_sums_sha256: "" ', f'sponsord_onramp_decdn_sums_sha256: {self.sums} ')) with self.assertRaisesRegex(bd.ShapeError, "sponsord_onramp_decdn_sums_sha256"): - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) def test_refuses_a_second_artifacthub_image(self) -> None: # Not a version the anchored edit moves, so only the read-back sees it. self.write(bd.CHART, self.text(bd.CHART) + "# image: ghcr.io/decdn/decdn-node:latest\n") with self.assertRaisesRegex(bd.ShapeError, "artifacthub.io/images"): - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) def test_refuses_a_drifted_chart_example(self) -> None: bare = self.digest.removeprefix("sha256:") self.write(bd.CHART_VALUES, self.text(bd.CHART_VALUES).replace(bare, "d" * 64)) with self.assertRaisesRegex(bd.ShapeError, "example digest"): - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) def test_refuses_an_empty_current_digest_without_writing(self) -> None: # Replacing "" would rewrite every tracked file. + for key in KEYS: + with self.subTest(key=key): + self.write(bd.COMPOSE_ENV, self.original[bd.COMPOSE_ENV].replace( + f"{key}={self.digests[key]}", f"{key}=")) + before = {rel: self.text(rel) for rel in FILES} + with self.assertRaisesRegex(bd.ShapeError, "is not a digest"): + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) + for rel in FILES: + self.assertEqual(self.text(rel), before[rel]) + + def test_refuses_two_images_on_one_digest_without_writing(self) -> None: + # Replacing the shared value would move both copies to one image's new digest. self.write(bd.COMPOSE_ENV, self.text(bd.COMPOSE_ENV).replace( - f"DECDN_IMAGE_DIGEST={self.digest}", "DECDN_IMAGE_DIGEST=")) + f"SPONSORD_ONRAMP_IMAGE_DIGEST={self.digests['SPONSORD_ONRAMP_IMAGE_DIGEST']}", + f"SPONSORD_ONRAMP_IMAGE_DIGEST={self.digests['SPONSORD_IMAGE_DIGEST']}")) before = {rel: self.text(rel) for rel in FILES} - with self.assertRaisesRegex(bd.ShapeError, "is not a digest"): - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) + with self.assertRaisesRegex(bd.ShapeError, "two image digests are equal"): + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) for rel in FILES: self.assertEqual(self.text(rel), before[rel]) def test_skips_binary_and_refuses_undecodable_files(self) -> None: (self.root / "logo.bin").write_bytes(b"\0" + self.sums.encode()) subprocess.run(["git", "-C", str(self.root), "add", "-A"], check=True) - bd.rewrite(self.root, "9.8.7", SUMS, DIGEST) + bd.rewrite(self.root, "9.8.7", SUMS, DIGESTS) self.assertEqual((self.root / "logo.bin").read_bytes(), b"\0" + self.sums.encode()) (self.root / "notes.txt").write_bytes(b"\xff" + SUMS.encode()) subprocess.run(["git", "-C", str(self.root), "add", "-A"], check=True) with self.assertRaisesRegex(bd.ShapeError, "notes.txt: not UTF-8"): - bd.rewrite(self.root, "1.2.3", "e" * 64, DIGEST) + bd.rewrite(self.root, "1.2.3", "e" * 64, DIGESTS) class Cli(Fixture): """open-decdn-bump.sh reads stdout and the exit code; a refusal must exit 1.""" + REWRITE = ["rewrite", "9.8.7", "--sums-sha256", SUMS, + *(arg for key, flag in bd.IMAGE_DIGESTS for arg in (flag, DIGESTS[key]))] + def run_cli(self, *args: str) -> subprocess.CompletedProcess[str]: return subprocess.run( [sys.executable, str(HERE.parent / "bump_decdn.py"), "--root", str(self.root), *args], @@ -268,9 +317,15 @@ def test_pinned_synced_compare_and_kind_print_one_word(self) -> None: r = self.run_cli("kind", "v99.0.0") self.assertEqual((r.returncode, r.stdout), (0, "line\n")) + def test_rewrite_prints_the_files_it_wrote(self) -> None: + r = self.run_cli(*self.REWRITE) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn(f"wrote {bd.SPONSORD_DEFAULTS}\n", r.stdout) + self.assertIn(f"wrote {bd.COMPOSE_ENV}\n", r.stdout) + def test_a_refusal_exits_1(self) -> None: self.write(bd.COMPOSE_ENV, self.text(bd.COMPOSE_ENV).replace(f"# decdn/decdn v{self.version}\n", "")) - r = self.run_cli("rewrite", "9.8.7", "--sums-sha256", SUMS, "--image-digest", DIGEST) + r = self.run_cli(*self.REWRITE) self.assertEqual(r.returncode, 1) self.assertTrue(r.stderr.startswith("bump_decdn: "), r.stderr) self.assertEqual(self.run_cli("compare", "v1.2").returncode, 1) @@ -279,6 +334,8 @@ def test_a_refusal_exits_1(self) -> None: def test_usage_exits_2(self) -> None: self.assertEqual(self.run_cli().returncode, 2) self.assertEqual(self.run_cli("rewrite", "9.8.7").returncode, 2) + # Every image's digest is required: a bump that forgot one must not run. + self.assertEqual(self.run_cli(*self.REWRITE[:-2]).returncode, 2) if __name__ == "__main__": diff --git a/tests/scripts-test.sh b/tests/scripts-test.sh index f00e6de..9a3d02c 100755 --- a/tests/scripts-test.sh +++ b/tests/scripts-test.sh @@ -1857,16 +1857,17 @@ cmp -s "$repo/ansible/roles/decdn_node/files/install-build-output.py" \ "$repo/ansible/roles/sponsord/files/install-build-output.py" \ || fail "decdn_node and sponsord files/install-build-output.py differ" # The release-line resolver is copied into every role that installs a release, and -# the onramp carries its own copy of decdn's KEYS (it checks the installers' decdn -# release; decdn_node ships in the other collection). +# so is decdn's KEYS: one decdn/decdn release ships the node, the CLI, sponsord and +# the onramp (the sponsord roles ship in the other collection, so each carries its +# own copy). for r in sponsord sponsord_onramp; do cmp -s "$repo/ansible/roles/decdn_node/tasks/resolve-release.yml" \ "$repo/ansible/roles/$r/tasks/resolve-release.yml" \ || fail "decdn_node and $r tasks/resolve-release.yml differ" + cmp -s "$repo/ansible/roles/decdn_node/files/decdn-release-KEYS.asc" \ + "$repo/ansible/roles/$r/files/decdn-release-KEYS.asc" \ + || fail "decdn_node and $r files/decdn-release-KEYS.asc differ" done -cmp -s "$repo/ansible/roles/decdn_node/files/decdn-release-KEYS.asc" \ - "$repo/ansible/roles/sponsord_onramp/files/decdn-release-KEYS.asc" \ - || fail "decdn_node and sponsord_onramp files/decdn-release-KEYS.asc differ" # decdn_backup_local_dir and sponsord_backup_local_dir: one expression (backups/ beside # the repo's playbooks/, or beside the project's ansible.cfg from a collection), the # key's own name aside. Only decdn_node's runs in a molecule scenario. @@ -1898,10 +1899,11 @@ done pass "the decommission defaults and the confirmation's service list agree across all five roles" # --- the release pins agree ---------------------------------------------------------- -# decdn/sponsord releases sponsord and sponsord-onramp from one tag, and the onramp's -# installers should hand users the decdn the node role deploys. The pins are copied -# into compose/sponsord-onramp.env.example and the chart's appVersion; the chart -# check is here, not in lint-helm, so a PR that bumps only the role still runs it. +# One decdn/decdn release ships decdn-node, decdn, sponsord and sponsord-onramp, so +# the three roles pin one line, and the onramp's installers should hand users the +# decdn the node role deploys. The pins are copied into +# compose/sponsord-onramp.env.example and the chart's appVersion; the chart check is +# here, not in lint-helm, so a PR that bumps only the role still runs it. pin() { # : a scalar default, quotes stripped sed -nE "s/^$2: *\"?([^\" #]*)\"?.*/\1/p" "$repo/ansible/roles/$1/defaults/main.yml" } @@ -1914,7 +1916,9 @@ for p in "decdn_node decdn_node_version" "sponsord sponsord_version" "sponsord_o [[ "$(pin $p)" =~ $pin_re ]] || fail "${p#* }: the default in roles/${p% *}/defaults/main.yml is not X.Y or X.Y.Z" done [ "$(pin sponsord sponsord_version)" = "$(pin sponsord_onramp sponsord_onramp_version)" ] \ - || fail "sponsord_version and sponsord_onramp_version differ (one decdn/sponsord release ships both)" + || fail "sponsord_version and sponsord_onramp_version differ (one decdn/decdn release ships both)" +[ "$(pin sponsord sponsord_version)" = "$(pin decdn_node decdn_node_version)" ] \ + || fail "sponsord_version and decdn_node_version differ (one decdn/decdn release ships sponsord and the node)" [ "$(pin sponsord_onramp sponsord_onramp_decdn_release)" = "v$(pin decdn_node decdn_node_version)" ] \ || fail "sponsord_onramp_decdn_release is not v" # Compose has no resolver: its env example names an exact release and digest, which @@ -1979,9 +1983,8 @@ elif [[ -n ${CI:-} ]]; then else skipped+=("artifacthub.io/images check (needs yq)") fi -# Both vendored KEYS hold the same maintainer keys (they differ in header text only), -# and those are the fingerprints SECURITY.md publishes, so swapping a key takes a -# visible edit there too. +# The vendored KEYS (byte copies, checked above) hold the fingerprints SECURITY.md +# publishes, so swapping a key takes a visible edit there too. command -v gpg >/dev/null || fail "gpg is required (the vendored KEYS checks)" fprs() { # : its primary-key fingerprints, sorted local out @@ -1990,8 +1993,6 @@ fprs() { # : its primary-key fingerprints, sorted } want_fprs="$(fprs "$repo/ansible/roles/decdn_node/files/decdn-release-KEYS.asc")" [ -n "$want_fprs" ] || fail "decdn_node/files/decdn-release-KEYS.asc: no keys" -[ "$(fprs "$repo/ansible/roles/sponsord/files/sponsord-release-KEYS.asc")" = "$want_fprs" ] \ - || fail "the decdn and sponsord release KEYS hold different keys" [ "$(sed -nE 's/^Fingerprint: *//p' "$repo/SECURITY.md" | tr -d ' ' | sort)" = "$want_fprs" ] \ || fail "SECURITY.md's Fingerprint: lines differ from the vendored KEYS" pass "release pins agree across roles, Compose (incl. its iroh images), the chart and its artifacthub.io/images, and AGENTS.md; the vendored KEYS match SECURITY.md"