Skip to content

fix: handle stop during startup with safe state transitions and hard-exit - #639

Open
jumski wants to merge 2 commits into
portable-worker-release-followupsfrom
pr-633-review-fixes
Open

fix: handle stop during startup with safe state transitions and hard-exit#639
jumski wants to merge 2 commits into
portable-worker-release-followupsfrom
pr-633-review-fixes

Conversation

@jumski

@jumski jumski commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

This PR fixes a race condition where calling stop() on a worker during its startup phase could trigger an invalid state transition, since the lifecycle was being moved to Stopping before startup had a chance to complete its own transitions.

Changes

Core worker shutdown sequencing

Worker.stop() now aborts the worker immediately but defers the transitionToStopping() call until any in-flight startupPromise has settled. This prevents an illegal Starting → Stopping transition while still ensuring the abort signal prevents any batch processing from starting.

The Created → Stopping transition is now explicitly allowed in WorkerState, enabling workers that are stopped before they ever start to reach Stopped cleanly.

Lifecycle hardening

The workerRow guard in acknowledgeStop() has been removed from both WorkerLifecycle and FlowWorkerLifecycle. A worker stopped before its first HTTP request never has a workerRow, and the guard was incorrectly blocking a valid shutdown path.

SupabasePlatformAdapter stop sequencing

stopWorker() is now idempotent — concurrent callers share a single performStopWorker() promise so SQL closes exactly once. The stop sequence now:

  1. Awaits any in-flight worker replacement before proceeding
  2. Marks the worker stopped in the database
  3. Calls worker.stop()
  4. Closes the SQL connection

markWorkerStopped has been moved out of the shutdown handler and into performStopWorker() so it runs in the correct order relative to worker.stop() and sql.end(). SQL close errors are logged but do not suppress an earlier operation error.

The abortController is no longer reset during worker replacement once a stop has been initiated, preventing a replacement from making the platform signal live again mid-shutdown.

ProcessPlatformAdapter early exit during hung startup

A startupCompleted flag is tracked. If a signal arrives before startup finishes, the adapter now hard-exits immediately with code 0 rather than waiting on a potentially hung bootstrap. The docs are updated to reflect this behavior.

processDeps optional offSignal

process.off is no longer required for getProcessDeps() to succeed. offSignal is exposed as undefined when process.off is absent, allowing environments that only implement process.on to work correctly.


Fixes #635
Fixes #636
Fixes #634

@changeset-bot

changeset-bot Bot commented Aug 18, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 943b2dd

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@nx-cloud

nx-cloud Bot commented Aug 18, 2026

Copy link
Copy Markdown

View your CI Pipeline Execution ↗ for commit 943b2dd

Command Status Duration Result
nx run edge-worker:test:integration ✅ Succeeded 4m 29s View ↗
nx affected -t verify-exports --base=origin/mai... ✅ Succeeded 3s View ↗
nx affected -t build --configuration=production... ✅ Succeeded 3s View ↗
nx affected -t lint typecheck test --parallel -... ✅ Succeeded 25s View ↗
nx run edge-worker:e2e:portable-runtimes ✅ Succeeded 57s View ↗
nx run client:e2e ✅ Succeeded 1m 15s View ↗
nx run cli:e2e ✅ Succeeded 4s View ↗
nx run edge-worker:e2e ✅ Succeeded 48s View ↗
Additional runs (2) ✅ Succeeded ... View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗


☁️ Nx Cloud last updated this comment at 2026-08-18 13:29:17 UTC

Agent added 2 commits August 18, 2026 13:10
- Serialize Worker.stop() behind startup and allow Created -> Stopping cancellation
- Add idempotent Supabase stop gate: awaits in-flight worker replacement, marks after startup settles, keeps the shutdown signal aborted, and never lets a sql close failure mask the operation error
- Exit immediately with code 0 on the first process signal during startup and honor optional process.off
- Document startup-signal behavior for Node/Bun process workers

Fixes #635
Fixes #636
Fixes #634
…failures

Review follow-up for PR 633:

- Supabase shutdown runs under a 5s deadline, drains the worker before
  marking, force-closes SQL at the deadline, and reports dual peer
  failures via AggregateError without masking SQL close failures
- HTTP startup waits for in-flight replacements; stopPromise permanently
  closes startup admission
- Worker main loop retries heartbeat/batch failures with capped
  exponential backoff (100ms-5s, abort-aware, reset on success)
- StepTaskPoller rethrows poll failures so Worker can distinguish them
  from empty polls
- Process adapter preserves startup/drain/marking errors over cleanup
  failures

Adds a draft news article for the 0.15.0 release (hidden on pgflow.dev
until draft: false).
@jumski
jumski force-pushed the pr-633-review-fixes branch from 530448c to 943b2dd Compare August 18, 2026 13:11
@jumski
jumski force-pushed the portable-worker-release-followups branch from 621f55e to 75cf582 Compare August 18, 2026 13:11
@github-actions

Copy link
Copy Markdown
Contributor

🔍 Preview Deployment: Website

Deployment successful!

🔗 Preview URL: https://pr-639.pgflow.pages.dev

📝 Details:

  • Branch: pr-633-review-fixes
  • Commit: 34c7517f666d685ab41c11e5c14a50bb4cad8634
  • View Logs

_Last updated: _

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant