Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 64 additions & 9 deletions .github/workflows/handle-upstream.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ name: handle-upstream
# artifacts, and the release version becomes the pair version.
# Hpo joins at the single cutoff in scripts/release_pair.py: then an event
# probes both partners and the tag records all three versions.
# The handler pushes the tag with the App token. Publication is a separate
# operator dispatch of publish.yml on main with that tag as input (build the
# The handler pushes the tag with the App token, then automatically dispatches
# publish.yml on main with its own tag output when policy allows (build the
# combined image -> fan out to the MCPs and the app). Idempotent, with
# half-failed-release recovery, a downgrade guard, and serialized concurrency.

Expand Down Expand Up @@ -314,15 +314,19 @@ jobs:
printf '%s\n' "$line" >> "$GITHUB_STEP_SUMMARY"

- name: Re-publish (recover a half-failed prior release)
id: republish
if: steps.decide.outputs.mode == 'republish'
env:
TAG: v${{ steps.decide.outputs.cur_ver }}
run: |
set -euo pipefail
line="versions unchanged but image missing for ${TAG}; after the tag exists on main and handle-upstream is green, a user with write access must run: gh workflow run publish.yml --repo pineforge-4pass/pineforge-release --ref main -f tag=${TAG}"
echo "::warning title=pineforge publish needed::${line}"
# Republish assumes the tag: a missing tag fails this job on purpose
# at this idempotent push, so the existing producer alert fires.
git push origin "$TAG"
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
line="versions unchanged but image missing for ${TAG}; handle-upstream automatically dispatches publish.yml when allowed by recovery policy"
echo "::notice title=pineforge publish automatic::${line}"
printf '%s\n' "$line" >> "$GITHUB_STEP_SUMMARY"
exit 0

# Engine events already prepare/prove their source before the decision.
# A codegen/HPO event that completes the pair must prove the pinned
Expand All @@ -335,6 +339,7 @@ jobs:
run: python3 scripts/release_pipeline.py producer engine "$E"

- name: Bump VERSION, commit, tag
id: bump
if: steps.decide.outputs.mode == 'bump'
env:
E: ${{ steps.decide.outputs.engine }}
Expand All @@ -358,8 +363,9 @@ jobs:
[ -n "$commit" ] || { echo "::error::VERSION is ${RELEASE} but main has no release commit for it; refusing to tag"; exit 1; }
git tag -a "v${RELEASE}" "$commit" -m "$(tag_message "$RELEASE")"
git push origin "v${RELEASE}"
line="pushed v${RELEASE} at ${commit} (${components}); after the tag exists on main and handle-upstream is green, a user with write access must run: gh workflow run publish.yml --repo pineforge-4pass/pineforge-release --ref main -f tag=v${RELEASE}"
echo "::notice title=pineforge publish needed::${line}"
echo "tag=v${RELEASE}" >> "$GITHUB_OUTPUT"
line="pushed v${RELEASE} at ${commit} (${components}); handle-upstream automatically dispatches publish.yml when allowed by recovery policy"
echo "::notice title=pineforge publish automatic::${line}"
printf '%s\n' "$line" >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
Expand Down Expand Up @@ -397,10 +403,59 @@ jobs:
git tag -a "v${next}" -m "$(tag_message "$next")"
git push origin HEAD:main
git push origin "v${next}"
line="pushed v${next} (${components}); after the tag exists on main and handle-upstream is green, a user with write access must run: gh workflow run publish.yml --repo pineforge-4pass/pineforge-release --ref main -f tag=v${next}"
echo "::notice title=pineforge publish needed::${line}"
echo "tag=v${next}" >> "$GITHUB_OUTPUT"
line="pushed v${next} (${components}); handle-upstream automatically dispatches publish.yml when allowed by recovery policy"
echo "::notice title=pineforge publish automatic::${line}"
printf '%s\n' "$line" >> "$GITHUB_STEP_SUMMARY"

- name: Check publication recovery policy
id: publish-policy
if: steps.decide.outputs.mode == 'bump' || steps.decide.outputs.mode == 'republish'
env:
GH_TOKEN: ${{ steps.app.outputs.token }}
TAG: ${{ steps.bump.outputs.tag || steps.republish.outputs.tag }}
RECOVERY: ${{ steps.decide.outputs.mode == 'republish' || steps.decide.outputs.retag == 'true' }}
run: |
set -euo pipefail
policy="$(python3 scripts/release_pipeline.py publish-policy --tag "$TAG" --recovery "$RECOVERY")"
printf '%s\n' "$policy" >> "$GITHUB_OUTPUT"
if printf '%s\n' "$policy" | grep -qx 'allowed=false'; then
latest="$(printf '%s\n' "$policy" | sed -n 's/^latest_ready=//p')"
line="automatic publication skipped for ${TAG}: recovery is older than latest ready ${latest}; manual recovery: gh workflow run publish.yml --repo pineforge-4pass/pineforge-release --ref main -f tag=${TAG}"
echo "::notice title=pineforge publish skipped::${line}"
printf '%s\n' "$line" >> "$GITHUB_STEP_SUMMARY"
fi

- name: Mint hub Actions token
id: publish-app
if: steps.publish-policy.outputs.allowed == 'true'
uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
with:
app-id: ${{ secrets.PINEFORGE_APP_ID }}
private-key: ${{ secrets.PINEFORGE_APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: pineforge-release
permission-actions: write
github-api-url: https://api.github.com

- name: Dispatch publication
if: steps.publish-policy.outputs.allowed == 'true'
env:
GH_TOKEN: ${{ steps.publish-app.outputs.token }}
TAG: ${{ steps.bump.outputs.tag || steps.republish.outputs.tag }}
run: |
set -euo pipefail
if gh workflow run publish.yml --repo pineforge-4pass/pineforge-release --ref main -f tag="$TAG"; then
line="automatically dispatched publish.yml on main for ${TAG}"
echo "::notice title=pineforge publish dispatched::${line}"
printf '%s\n' "$line" >> "$GITHUB_STEP_SUMMARY"
else
line="automatic publication dispatch failed for ${TAG}; manual recovery: gh workflow run publish.yml --repo pineforge-4pass/pineforge-release --ref main -f tag=${TAG}"
echo "::error title=pineforge publish dispatch failed::${line}"
printf '%s\n' "$line" >> "$GITHUB_STEP_SUMMARY"
exit 1
fi

- name: Report failed upstream handling
if: ${{ failure() && !cancelled() }}
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release-reconcile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
repositories: pineforge-release
permission-contents: write
github-api-url: https://api.github.com
- name: Verify artifacts and replay missed stable producer events
- name: Replay missed stable producer events and report stalled tagged publication
env:
GH_TOKEN: ${{ steps.app.outputs.token }}
RELEASE_REPORT_OWNER: ${{ vars.RELEASE_REPORT_OWNER }}
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ is the release tag, with its `v`: `v1.0.0`).

```
{engine, codegen, HPO} releases → existing producer events → handle-upstream.yml
→ lockstep VERSION + tag push (App) → operator dispatch on main with tag input
→ lockstep VERSION + tag push (App) → automatic dispatch on main with tag input
→ publish.yml: guard tag shape, existence and ancestry → build verified SHA
→ release-finalize.yml: packages + measured release facts + immutable manifest
→ pineforge-release → consumer PR → own CI + E2E → merge/deploy → served receipt
Expand All @@ -168,7 +168,7 @@ is the release tag, with its `v`: `v1.0.0`).
- `handle-upstream.yml` — receives `repository_dispatch` from engine / codegen-oss / HPO
(`{version, prerelease, run_id}`), applies the pairing rule, sets `VERSION`,
commits + pushes the tag (idempotent, with half-failed-release recovery and a
downgrade guard). Publication then needs the operator command below.
downgrade guard). Publication dispatches automatically when recovery policy allows.
- `publish.yml` — only by dispatch on `main` with a required `tag` input:
first checks the release tag's shape, existence and reachability from main.
It checks out the verified commit SHA, checks the pair, waits for upstream
Expand All @@ -182,10 +182,10 @@ is the release tag, with its `v`: `v1.0.0`).
this diagram; each consumer gets an App token scoped to its configured target.
A prerelease reaches the application by hand-off, with no repository dispatch
and no App token. Stable consumers without verified completion stay pending.
- `release-reconcile.yml` validates and repairs missed stable producer events;
- `release-reconcile.yml` repairs missed stable producer events and reports tagged publication gaps after a grace period;
`release-receipts.yml` validates consumer evidence and repairs dropped callbacks.

After the tag exists on main and `handle-upstream` is green, a user with write access runs publication or replay:
Bump/retag/republish dispatch publication automatically when allowed; for older-version recovery or a failed dispatch, a user with write access runs:
`gh workflow run publish.yml --repo pineforge-4pass/pineforge-release --ref main -f tag=<vX.Y.Z or vX.Y.Z-rc.N>`.
The first `tag-guard` job requires a main-ref dispatch, validates the complete
release tag pattern, fetches tags and main with full history, and requires the
Expand Down
58 changes: 45 additions & 13 deletions docs/release-chain.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ verified commit SHA, with no persisted credentials, so moving a tag between
jobs cannot change the tree built. A failed or skipped guard prevents
finalization; failure reporting first refuses to check out or run repository
code unless the guard succeeded.
The publication job also requires the tag to equal `v` plus the checked-out
`VERSION`, reads component pins from the annotated tag message (with the existing
pair-version or legacy fallback), and applies the pairing rule before building.
The reusable `release-finalize.yml` call inherits the caller's main ref.
Tag pushes, release events and `workflow_run` do not start publication.

Expand All @@ -69,32 +72,61 @@ own published artifacts are verified before the existing partner/pair decision.
Incomplete lockstep waits. Invalid flags, source mismatches, authentication,
rate limits and transport failures fail visibly.

`release-reconcile.yml` wakes hourly or manually. It paginates the
`release-reconcile.yml` is configured to wake hourly or manually. It paginates the
three fixed producer repositories, excludes drafts/prereleases/non-product tags
and versions older than hub VERSION. It checks whether the desired stable
version already has a hub Git tag before downloading any artifacts. Only a
version without that tag has its artifacts verified and the **same producer
events** sent back to the existing handler. The handler pushes the hub tag;
after the tag exists on main and `handle-upstream` is green, a user with write
access dispatches `publish.yml` on `main` with that tag using the command below.
events** sent back to the existing handler. After pushing the hub tag successfully,
the handler automatically dispatches `publish.yml` on `main` in bump (including
retag) and republish modes when policy allows; noop and wait never dispatch.
The input tag comes only from the same job's successful tag-step output, never
the event payload. Republish first idempotently pushes its assumed existing tag;
a missing tag fails the job and triggers the producer incident.
Finalization then continues even while the release is a draft awaiting prerequisites.
Older missed versions are superseded by the newest desired stable version;
they are never replayed as downgrades after it. Duplicate notifications reuse
the handler's existing decision. A stored candidate makes an unchanged-version
decision a no-op even while the fixed image tag is absent. Nothing calls a
producer's version-bump job.

Recovery of an existing hub tag emits an operator notice and never force-pushes
or rewrites the tag. Publication and replay use the same operator command,
run by a user with write access after the tag exists on main and
`handle-upstream` is green:
The post-push `publish-policy` check reuses the finalizer's `finalized_version(store)`
and semver ordering. A normal bump is allowed. Retag/republish recovery older
than the latest ready release succeeds with a notice and step summary, without
minting the dispatch token or dispatching. Equal versions are allowed because
recovery restores the image of the latest ready version; newer versions are
allowed too. An unreadable release store fails the policy check.

The existing App token keeps Contents write only. A second mint requests
Actions write only, limited to `pineforge-release`, immediately before dispatch.
The second token's blast radius is "dispatch, cancel or re-run ANY workflow of this hub"; the control is the pinned step list (the mint is immediately followed by the dispatch step, the failure reporter is last), which `tests/test_release_workflows.py` freezes.
The job keeps the `release-automation` environment; required reviewers there
can still pause the chain. The owner must confirm the App's Actions write
permission on the hub and the environment's reviewers.

A failed dispatch fails the job and fires the existing producer incident;
the pushed tag remains and a rerun is safe. Older-version recovery and failed
dispatch notices retain this manual command for a user with write access:
`gh workflow run publish.yml --repo pineforge-4pass/pineforge-release --ref main -f tag=<vX.Y.Z or vX.Y.Z-rc.N>`.
Use the actual existing tag as `tag`. Pushing a tag alone publishes nothing.
Both a new tag push and retag recovery leave publication to the operator;
an unchanged version with an image or verified candidate keeps its existing no-op.
Recovery never force-pushes or rewrites the tag; an unchanged version with an
image or verified candidate keeps its existing no-op.
Tags predating this workflow need a reviewed migration; the hub does not invent
new manifests for historical images without candidate verification.

For an existing hub tag, reconcile reports a deduplicated incident if no
candidate draft and no fixed version image exist after the grace period in
`PUBLISH_START_GRACE_MINUTES`. The clock is the annotated tagger date.
Lightweight tags, unreadable tag metadata, and store or registry errors fail
closed. This tagged-publication check never dispatches publication.

Live activation caveat: when last read, `release-reconcile`, `release-finalize`,
`release-receipts`, `facts-fanout` and `facts-descriptions` were
`disabled_manually`; the reconcile incident safety net and their scheduled/push
triggers remain inactive until the owner activates them. The reason for those
settings and whether human stable publication was an intentional public-image
gate remain owner decisions; this change does not alter live settings.

Exception on record: on 2026-10-25 the `v1.7.0` tag was re-pointed from `73d2a1eed8393d46cae0fa8d1d785ac2919d5a10` to `ab95811bc18dc7a593ce27f864e340bd270fef79` (the licence-sums fix, #121) by the maintainer's decision after a full pre-proof; the published image's revision label is `ab95811bc18dc7a593ce27f864e340bd270fef79`.

## Candidate and readiness
Expand Down Expand Up @@ -249,8 +281,8 @@ The sender reuses the existing App token mint, scoped to one target with
Contents write. The existing App must be installed on the private compute
repository with the App's full permission set, to be confirmed in the App's
settings before the owner decides **O-1**. App permissions apply to every
repository in the installation: the existing App already holds Actions write
for the hub, and installing it on compute grants that and every other App
repository in the installation: the documented App permission set includes Actions write
for the hub (the owner must confirm it live), and installing it on compute grants that and every other App
permission there. The mint uses `owner: github.repository_owner`, so compute
joins the same owner's installation; each mint's requested permissions limit
that token rather than the App installation.
Expand Down Expand Up @@ -1352,7 +1384,7 @@ use `ubuntu-latest` by the same standing exception, including publication workfl
| Name / setting | Scope | Purpose |
|---|---|---|
| `PINEFORGE_APP_ID`, `PINEFORGE_APP_PRIVATE_KEY` (existing) | Hub `release-automation` environment | Short-lived, one-target installation tokens; no PAT fallback. |
| App installation permissions | The App's full permission set on every installed repository, including the private compute repository | Confirm the full set in the App's settings before the owner decides O-1. The App already holds Actions write for the hub; installing it on compute grants that and every other App permission there. The mint uses `owner: github.repository_owner` and joins the same owner's installation. Hub tag pushes and producer-event reconciliation; consumer dispatch and completion evidence reads. Publication and replay use the write-access operator's login. Each mint asks only for the permissions needed by that job: hub Contents write; receipt consumers Contents read/write, Actions read and Pull requests read; delivery-only compute uses only the existing Contents-write dispatch mint. Consumer PR writers in other repos separately need Pull requests write without branch bypass. |
| App installation permissions | The App's full permission set on every installed repository, including the private compute repository | Confirm the full set in the App's settings before the owner decides O-1, including the documented Actions write for the hub; installing it on compute grants that and every other App permission there. The mint uses `owner: github.repository_owner` and joins the same owner's installation. Hub tag pushes and producer-event reconciliation; consumer dispatch and completion evidence reads. Automatic publication uses a separate hub mint with Actions write only; manual recovery uses a write-access operator's login. Each mint asks only for the permissions needed by that job: hub Contents write or the separate hub Actions write; receipt consumers Contents read/write, Actions read and Pull requests read; delivery-only compute uses only the existing Contents-write dispatch mint. Consumer PR writers in other repos separately need Pull requests write without branch bypass. |
| `RELEASE_HOSTED_MCP_REPOSITORY`, `RELEASE_APPLICATION_REPOSITORY`, `FACTS_WEB_REPOSITORY` (existing) | Hub `release-automation` environment secrets | Private short repository names for neutral hosted/application/web targets. Public offline/data/license names are fixed in workflows. |
| `RELEASE_COMPUTE_REPOSITORY` (new secret) | Hub `release-automation` environment secrets | Private short name of the compute target repository; the application target likewise lives in `RELEASE_APPLICATION_REPOSITORY`. Both reuse the existing one-target App mint. No values belong in source or public audit assets. |
| `RELEASE_RECEIPT_POLICY` (new secret) | Hub repository Actions secret (read by the receipt verify job, which runs in the `release-automation` environment; do not also create it in the environment, it would shadow this one) | JSON map of each enabled receipt role to its trusted receipt workflow/artifact/base and full CI/E2E/deploy job inventories, plus optional application-only `post_merge`. Compute needs no entry. Keep private workflow/job details out of public source. |
Expand Down
Loading
Loading