Skip to content

Retry native worker attachment after lost database connections - #472

Closed
ghirish-shopify wants to merge 1 commit into
TangleML:masterfrom
ghirish-shopify:codex/retry-native-worker-attachment
Closed

ghirish-shopify wants to merge 1 commit into
TangleML:masterfrom
ghirish-shopify:codex/retry-native-worker-attachment

Conversation

@ghirish-shopify

Copy link
Copy Markdown

A native worker can be submitted successfully and then lose its Tangle attachment when the database connection drops. The queued sweep currently marks that execution SYSTEM_ERROR and skips its descendants, even while the worker continues running.

Snapshot ORM inputs before submission and retry only the database attachment in fresh sessions after an invalidated connection, with three attempts and bounded backoff. The launcher is called once. If commit succeeded but its acknowledgement was lost, verify the same execution, cache key, artifact URIs, and initial native-handle fingerprint before accepting the existing attachment. Conflicting attachments are rejected and cancellation flags are preserved.

Also allow a launcher to return an already-terminal handle. It is attached as pending, and the ordinary refresh path validates/registers outputs and updates descendants. This is needed when recovering a delayed native submit response.

Validation: full backend suite (676 passed), followed by the expanded focused suite (35 passed). Fault injection covers disconnect before read, before commit, and after commit; early completion with artifact registration and downstream release; terminal failure; bounded retries; cancellation; and conflicting/duplicate attachment. Terminal-handle regression cases fail on the prior code. Black and git diff --check pass.

Limits: retries recover transient connection loss within the current orchestrator process. They do not provide durable recovery if the process crashes before recording the native handle, or if the database stays unavailable beyond the retry budget. No schema, launcher API, or production dependency pin changes. Downstream Oasis must adopt the merged backend commit before this protects its running orchestrator.

@ghirish-shopify
ghirish-shopify requested a review from a team November 17, 2026 21:02
@ghirish-shopify
ghirish-shopify marked this pull request as draft November 17, 2026 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant