Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions .github/workflows/facts-fanout.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ on:

permissions:
contents: read
actions: read

# This signal serializes independently of descriptions and release automation.
concurrency:
Expand Down Expand Up @@ -53,10 +54,16 @@ jobs:
python3 scripts/facts_fanout.py prepare --facts facts/facts.json --dry-run "$DRY_RUN" \
--output-dir "$RUNNER_TEMP/fanout-source" --github-output "$GITHUB_OUTPUT" \
--receipt "$RUNNER_TEMP/fanout-audit/prepare.json"
- name: Require full main proof for the facts snapshot
id: main-proof
env:
GH_TOKEN: ${{ github.token }}
WAIT_SECONDS: ${{ github.event_name == 'push' && '1800' || '0' }}
run: python3 scripts/release_main_proof.py facts-source --source-dir "$RUNNER_TEMP/fanout-source" --wait-seconds "$WAIT_SECONDS"
- name: Mint consumer token
id: app
if: >-
steps.source.outcome == 'success' && steps.source.outputs.mint_allowed == 'true' &&
steps.source.outcome == 'success' && steps.source.outputs.mint_allowed == 'true' && steps.main-proof.outcome == 'success' &&
github.ref == 'refs/heads/main' && github.ref_protected &&
(github.event_name == 'push' || github.event_name == 'workflow_dispatch' || github.event_name == 'schedule') &&
vars.FACTS_WEB_DISPATCH_ENABLED == 'true'
Expand All @@ -72,7 +79,7 @@ jobs:
permission-contents: write
github-api-url: https://api.github.com
- name: Send the validated signal and record its status
if: ${{ !cancelled() && steps.source.outcome == 'success' && steps.source.outputs.mint_allowed == 'true' }}
if: ${{ !cancelled() && steps.source.outcome == 'success' && steps.source.outputs.mint_allowed == 'true' && steps.main-proof.outcome == 'success' }}
env:
FACTS_READ_TOKEN: ${{ github.token }}
FACTS_WEB_REPOSITORY: ${{ secrets.FACTS_WEB_REPOSITORY }}
Expand Down Expand Up @@ -122,10 +129,16 @@ jobs:
python3 scripts/facts_fanout.py prepare --facts facts/facts.json --dry-run "$DRY_RUN" \
--output-dir "$RUNNER_TEMP/fanout-source" --github-output "$GITHUB_OUTPUT" \
--receipt "$RUNNER_TEMP/fanout-audit/prepare.json"
- name: Require full main proof for the facts snapshot
id: main-proof
env:
GH_TOKEN: ${{ github.token }}
WAIT_SECONDS: ${{ github.event_name == 'push' && '1800' || '0' }}
run: python3 scripts/release_main_proof.py facts-source --source-dir "$RUNNER_TEMP/fanout-source" --wait-seconds "$WAIT_SECONDS"
- name: Mint consumer token
id: app
if: >-
steps.source.outcome == 'success' && steps.source.outputs.mint_allowed == 'true' &&
steps.source.outcome == 'success' && steps.source.outputs.mint_allowed == 'true' && steps.main-proof.outcome == 'success' &&
github.ref == 'refs/heads/main' && github.ref_protected &&
(github.event_name == 'push' || github.event_name == 'workflow_dispatch' || github.event_name == 'schedule') &&
vars.FACTS_LICENSE_DISPATCH_ENABLED == 'true'
Expand All @@ -141,7 +154,7 @@ jobs:
permission-contents: write
github-api-url: https://api.github.com
- name: Send the validated signal and record its status
if: ${{ !cancelled() && steps.source.outcome == 'success' && steps.source.outputs.mint_allowed == 'true' }}
if: ${{ !cancelled() && steps.source.outcome == 'success' && steps.source.outputs.mint_allowed == 'true' && steps.main-proof.outcome == 'success' }}
env:
FACTS_READ_TOKEN: ${{ github.token }}
FACTS_WEB_REPOSITORY: pineforge-license
Expand Down
9 changes: 8 additions & 1 deletion .github/workflows/handle-upstream.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ concurrency:

permissions:
contents: read # the App token does the push
actions: read # exact release-commit main-push proof
issues: write # neutral failure incidents only

jobs:
Expand Down Expand Up @@ -337,6 +338,7 @@ jobs:
- name: Bump VERSION, commit, tag
if: steps.decide.outputs.mode == 'bump'
env:
MAIN_PROOF_TOKEN: ${{ github.token }}
E: ${{ steps.decide.outputs.engine }}
C: ${{ steps.decide.outputs.codegen }}
H: ${{ steps.decide.outputs.hpo }}
Expand All @@ -356,6 +358,7 @@ jobs:
# release commit (VERSION = the pair version). Tag that commit.
commit="$(git log HEAD --format=%H -n 1 --fixed-strings --grep="chore(release): v${RELEASE} — ${components}")"
[ -n "$commit" ] || { echo "::error::VERSION is ${RELEASE} but main has no release commit for it; refusing to tag"; exit 1; }
python3 scripts/release_main_proof.py commit --sha "$commit" --wait-seconds 1800
git tag -a "v${RELEASE}" "$commit" -m "$(tag_message "$RELEASE")"
git push origin "v${RELEASE}"
line="pushed v${RELEASE} at ${commit} (${components}); after the tag exists on main and handle-upstream is green, a user with write access must run: gh workflow run publish.yml --repo pineforge-4pass/pineforge-release --ref main -f tag=v${RELEASE}"
Expand Down Expand Up @@ -394,8 +397,12 @@ jobs:
git add VERSION ${harness}
git diff --cached --quiet && { echo "::error::nothing staged — refusing empty release"; exit 1; }
git commit -m "chore(release): v${next} — ${components}"
git tag -a "v${next}" -m "$(tag_message "$next")"
commit="$(git rev-parse HEAD)"
# Push the FINAL release commit first so its own full main-push CI
# and facts run can execute. Base/PR proof cannot authorize its tag.
git push origin HEAD:main
python3 scripts/release_main_proof.py commit --sha "$commit" --wait-seconds 1800
git tag -a "v${next}" "$commit" -m "$(tag_message "$next")"
git push origin "v${next}"
line="pushed v${next} (${components}); after the tag exists on main and handle-upstream is green, a user with write access must run: gh workflow run publish.yml --repo pineforge-4pass/pineforge-release --ref main -f tag=v${next}"
echo "::notice title=pineforge publish needed::${line}"
Expand Down
16 changes: 16 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ jobs:
PF_CI_LABELS: '["self-hosted","pf-ci","pf-ci-x64"]'
permissions:
contents: write # GitHub lists draft candidates only with push access
actions: read
outputs:
exists: ${{ steps.candidate.outputs.exists }}
tag: ${{ needs.tag-guard.outputs.tag }}
Expand All @@ -104,6 +105,12 @@ jobs:
with:
ref: ${{ needs.tag-guard.outputs.sha }}
persist-credentials: false
- name: Require full main proof before candidate reuse
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ needs.tag-guard.outputs.tag }}
RELEASE_SHA: ${{ needs.tag-guard.outputs.sha }}
run: python3 scripts/release_main_proof.py tag --tag "$TAG" --sha "$RELEASE_SHA"
- name: Reuse an already verified candidate
id: candidate
env:
Expand All @@ -122,6 +129,7 @@ jobs:
permissions:
contents: write # append verified candidate to its draft release
packages: write # push to GHCR
actions: read
outputs:
prerelease: ${{ steps.pair.outputs.prerelease }}
steps:
Expand All @@ -142,6 +150,13 @@ jobs:
python-version: '3.12'
- run: python3 -m pip install -r facts/requirements-validation.txt

- name: Require full main proof before image publication
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ env.RELEASE_TAG }}
RELEASE_SHA: ${{ needs.tag-guard.outputs.sha }}
run: python3 scripts/release_main_proof.py tag --tag "$TAG" --sha "$RELEASE_SHA"

- name: Assert tag matches VERSION
env:
TAG: ${{ env.RELEASE_TAG }}
Expand Down Expand Up @@ -690,6 +705,7 @@ jobs:
contents: write
packages: write
issues: write
actions: read

failure-report:
needs: [tag-guard, prepare, publish, notify-consumers]
Expand Down
11 changes: 10 additions & 1 deletion .github/workflows/release-finalize.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ permissions:

jobs:
finalize:
if: github.ref == 'refs/heads/main'
# Job-level concurrency also applies when invoked through workflow_call.
# Serialize alias decisions and persistent delivery reservations together.
concurrency:
Expand All @@ -35,6 +36,7 @@ jobs:
contents: write
packages: write
issues: write
actions: read
outputs:
deliveries: ${{ steps.delivery-output.outputs.deliveries }}
compute_configuration_error: ${{ steps.ready.outputs.compute_configuration_error }}
Expand Down Expand Up @@ -75,7 +77,7 @@ jobs:
RELEASE_APPLICATION_MAX_AGE_MINUTES: ${{ vars.RELEASE_APPLICATION_MAX_AGE_MINUTES || '1440' }}
run: |
set -euo pipefail
python3 scripts/release_pipeline.py finalize --tag "$TAG" --output "$RUNNER_TEMP/deliveries.json"
python3 scripts/release_main_proof.py finalize --tag "$TAG" --output "$RUNNER_TEMP/deliveries.json"
- name: Publish delivery matrix
id: delivery-output
if: ${{ always() }}
Expand Down Expand Up @@ -127,6 +129,7 @@ jobs:
permissions:
contents: write # append neutral delivery audit assets in this hub only
issues: write
actions: read
strategy:
fail-fast: false
matrix:
Expand All @@ -145,6 +148,12 @@ jobs:
with:
ref: main
persist-credentials: false
- name: Require full main proof before release delivery
id: main-proof
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ matrix.tag }}
run: python3 scripts/release_main_proof.py delivery --tag "$TAG"
- name: Require one configured consumer repository
shell: bash
env:
Expand Down
7 changes: 7 additions & 0 deletions .github/workflows/release-reconcile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ on:

permissions:
contents: read
actions: read
issues: write

concurrency:
Expand All @@ -15,6 +16,7 @@ concurrency:

jobs:
producers:
if: github.ref == 'refs/heads/main'
# pineforge-release runs on GitHub-hosted runners; standing exception for this repository
runs-on: ubuntu-latest
environment: release-automation
Expand All @@ -23,6 +25,11 @@ jobs:
with:
ref: main
persist-credentials: false
- name: Require full main proof before producer reconciliation
id: main-proof
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release_main_proof.py commit --sha "$(git rev-parse HEAD)"
- name: Mint hub-only dispatch token
id: app
uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
Expand Down
7 changes: 7 additions & 0 deletions .github/workflows/release-redeliver.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ jobs:
permissions:
contents: write # append one neutral redelivery audit asset in this hub
issues: write
actions: read
steps:
- name: Stop automatic apt upgrades (until the release image is rebaked)
shell: bash
Expand Down Expand Up @@ -86,6 +87,12 @@ jobs:
RELEASE_COMPUTE_FIRST_PAIR: ${{ vars.RELEASE_COMPUTE_FIRST_PAIR }}
run: |
python3 scripts/release_pipeline.py redeliver-check --tag "$TAG" --consumer "$CONSUMER"
- name: Require full main proof before release redelivery
id: main-proof
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ inputs.tag }}
run: python3 scripts/release_main_proof.py delivery --tag "$TAG"
- name: Require one configured consumer repository
id: target
shell: bash
Expand Down
117 changes: 117 additions & 0 deletions docs/release-main-proof.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Main-commit proof before publication

Release publication requires the hub's own complete CI and facts validation
for the exact release commit on `main`. A green pull request, merge queue,
producer workflow, other commit, or similarly named check cannot authorize it.
The existing tag ancestry, producer artifacts, runtime smoke, measured facts,
immutable manifest, and consumer policies continue to apply.

`scripts/release_main_proof.py` reads GitHub's public hub repository, workflow,
run, and job identities. It resolves each workflow by its file path and numeric
ID, selects its newest `push` run for `main` at the requested 40-character SHA,
and reads that run's current attempt. Both workflows and every job below must
be `completed` with conclusion `success`:

| Workflow | Complete expected job inventory |
| --- | --- |
| `python-test.yml` | `authz`, `changes`, `fingerprint-canonical`, `release-rules`, `description-renderer`, `release-ready-replay`, `release/ready` |
| `facts-validate.yml` | `authz`, `validate` |

The reader checks the repository and head repository identities, workflow
path, event, branch, SHA, main ancestry, run ID, attempt, and each job's source
identity. It reads paginated results and refuses incomplete or ambiguous
inventories. Missing, red, running, cancelled, skipped, neutral, or malformed
proof refuses publication. An older successful run cannot hide a newer failed
or unfinished run. There is no success override or supplied proof-file input.
The unit fixtures model endpoint responses; production always reads GitHub.
Jobs are fetched through the selected attempt's endpoint; when GitHub includes
the optional per-job attempt field, it must match that endpoint's attempt.

## Publication boundaries

| Route | Required proof |
| --- | --- |
| Fresh upstream release | Push the final VERSION and harness commit to `main`, prove that exact SHA, then create and push its annotated tag. |
| Retag recovery | Prove the existing final release commit before creating its missing tag. |
| Candidate reuse and image build | Re-resolve the existing tag, compare it with the SHA admitted by `tag-guard`, and prove that SHA before reuse or image publication. |
| Every finalizer trigger | Guard each selected candidate's tag and `hub_commit`; prove the facts commit actually read before writing its manifest or publishing aliases and readiness. |
| Replaying an existing manifest | Prove its release commit and its immutable bound facts commit before replaying publication. |
| Normal consumer delivery and redelivery | Prove the published manifest's release and facts commits before minting a consumer token. |
| Facts web and license signals | Prove the commit in the prepared facts payload, with its facts digest bound to that snapshot, before minting or sending. |
| Producer reconciliation | Prove the checked-out main SHA before replaying producer events; the handler separately proves the release commit it creates. |

`finalize` in the new helper calls the existing `release_pipeline.finalize_all`
with guarded candidate and manifest storage. This covers reusable, manual, scheduled,
and facts-push finalization, including an empty-tag candidate sweep. It avoids a
preflight over one candidate list followed by publication from a newly selected
list. Facts may arrive in a later main commit than the release commit, so each
of those exact SHAs requires its own full CI and facts proof. The manifest's
bound facts commit is proved before its immutable write, ahead of every alias,
publication and readiness step; replay proves the existing manifest's bound
commit too. The latest facts read used only for a freshness report needs no
publication proof.

When candidate proof is refused for a row that already has `release-ready.json`,
the finalizer leaves that row as historical data: it performs no alias,
publication or delivery work for it, and continues the sweep. An unprovable row
without readiness remains pending or failed as appropriate. Historical rows
read only to compare versions do not acquire publication authority.

The handler and facts-fanout push jobs wait up to 1,800 seconds for missing or
unfinished runs, without dispatching or rerunning CI. The `commit` and
`facts-source` commands accept `--wait-seconds` (default zero, maximum 3,600).
Failed proof stops immediately; an expired wait still refuses publication.
The finalizer reports unfinished proof as readiness pending and retries at its
next normal wake-up. Other routes refuse unavailable proof at their normal
wake-up; their existing recovery triggers remain responsible for subsequent
attempts.

## Recovering a release commit with failed CI

The handler pushes the VERSION bump before proving that exact release commit.
If its CI is red, no tag is created and VERSION can remain ahead of the landed
pair. Sending the same event again still checks the original release commit;
a later green main commit cannot authorize its missing tag.

A maintainer must manually revert the untagged VERSION bump on `main` to the
landed release version, then send the upstream event again. Use the normal
reviewed main change process and resolve the CI failure before replaying the
event. The handler can then create a fresh release commit, whose own complete
main-push CI and facts proof must pass before tagging. Leave existing tags and
release assets intact. If proof was only pending, retry the event after that
original commit's proof passes; the retag route still selects that commit.

The workflows grant their hub token `actions: read` for proof reads; consumer
tokens retain their existing scopes. Proof snapshots are cached only within
one operation, never persisted as a reusable authorization. GitHub reads and
publication are separate API operations; existing immutable tags and assets
remain part of the release contract.

## Focused controls

Run these in an authorized test environment, one module at a time:

```sh
python3 -m py_compile scripts/release_main_proof.py tests/test_release_main_proof.py tests/test_release_workflows.py tests/test_release_binding_workflows.py
python3 -m unittest discover -s tests -p 'test_release_main_proof.py' -v
python3 -m unittest discover -s tests -p 'test_release_workflows.py' -v
python3 -m unittest discover -s tests -p 'test_release_binding_workflows.py' -v
python3 -m unittest discover -s tests -p 'test_facts_fanout.py' -v
python3 -m unittest discover -s tests -p 'test_release_redeliver.py' -v
python3 tools/release-dry-run/dry_run.py --ready
```

The new controls cover valid identities and full-green acceptance, each absent
workflow and job, failed or unfinished proof, incorrect SHA/event/repository,
skipped proof, stale attempts, pagination, finalizer publication refusal, and
the owned workflow boundaries. Existing workflow text pins and shell replay
bridges pin the guarded finalizer entrypoint and proof-reader calls. The shell
replays serialize GitHub responses and execute the real reader for tag,
candidate, image, and redelivery boundaries. Their fixture clock expires an
unfinished handler wait without replacing its decision. Removed, softened,
or late guards have negative controls alongside the full-green controls.

Source changes and offline controls do not establish live release readiness.
The normal pull-request CI, review and merge route still applies, followed by
the required CI on the final main release commit and the normal publication
and consumer checks.
Loading
Loading