Skip to content

Publish release benchmark assets before releases become immutable #225

Description

@acgetchell

Summary

The Release Benchmarks workflow listens for release: published and later runs
gh release upload ... --clobber. Repository releases are immutable once
published, so the workflow cannot attach its durable Criterion archive at that
point.

The v0.4.5 release is immutable and has no release assets. Its workflow run
stopped earlier at the benchmark job timeout, but after that runtime failure is
fixed, the current upload step will still be unable to mutate the published
release.

Current State

  • .github/workflows/release-benchmarks.yml checks out the published tag, saves
    and packages Criterion baselines, then uploads them to the already published
    release with --clobber.
  • docs/RELEASING.md instructs maintainers to publish the GitHub release first
    and then verify the workflow-created durable asset.
  • The temporary Actions artifact is retained for 30 days and is explicitly not
    a substitute for the release asset.

Proposed Changes

  1. Create the tagged GitHub release as a draft before running release
    benchmarks.
  2. Trigger the release benchmark workflow for an explicit stable semver tag,
    and fail closed unless that tag resolves to the expected mutable draft.
  3. Check out and benchmark that exact tag, package the complete baseline, and
    upload it to the draft release.
  4. Publish the draft only after the durable asset upload succeeds.
  5. Define safe rerun behavior without relying on post-publication --clobber,
    while preserving concurrency and least-privilege permissions.
  6. Update release and benchmarking documentation, plus workflow validation and
    any repository-owned tests, to match the new sequence.

Benefits

  • Every immutable release is published with its durable benchmark evidence
    already attached.
  • Failed benchmark or upload runs leave a recoverable draft instead of a public
    release with missing assets.
  • The documented verification step reflects behavior GitHub permits.

Acceptance Criteria

  • The workflow rejects missing, prerelease, published, or otherwise
    unsuitable release targets before benchmarking.
  • The workflow uploads the expected
    la-stack-$TAG-criterion-baseline.tar.gz asset while the release is a
    mutable draft.
  • The release becomes public only after the asset is present.
  • Reruns have explicit, safe behavior and never require mutating an
    immutable release.
  • docs/RELEASING.md and docs/BENCHMARKING.md describe the same
    draft-run-publish sequence.
  • GitHub Actions syntax, security checks, repository tooling tests, and the
    applicable local CI gate pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingperformancePerformance related issuesreleaseRelease preparation and packagingreproducibilityReproducibility and artifact packagingtestingIssues that come up in testing

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions