Repository navigation
Retry native worker attachment after lost database connections - #472
Closed
ghirish-shopify wants to merge 1 commit into
Closed
ghirish-shopify wants to merge 1 commit into
ghirish-shopify wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A native worker can be submitted successfully and then lose its Tangle attachment when the database connection drops. The queued sweep currently marks that execution
SYSTEM_ERRORand skips its descendants, even while the worker continues running.Snapshot ORM inputs before submission and retry only the database attachment in fresh sessions after an invalidated connection, with three attempts and bounded backoff. The launcher is called once. If commit succeeded but its acknowledgement was lost, verify the same execution, cache key, artifact URIs, and initial native-handle fingerprint before accepting the existing attachment. Conflicting attachments are rejected and cancellation flags are preserved.
Also allow a launcher to return an already-terminal handle. It is attached as pending, and the ordinary refresh path validates/registers outputs and updates descendants. This is needed when recovering a delayed native submit response.
Validation: full backend suite (676 passed), followed by the expanded focused suite (35 passed). Fault injection covers disconnect before read, before commit, and after commit; early completion with artifact registration and downstream release; terminal failure; bounded retries; cancellation; and conflicting/duplicate attachment. Terminal-handle regression cases fail on the prior code. Black and
git diff --checkpass.Limits: retries recover transient connection loss within the current orchestrator process. They do not provide durable recovery if the process crashes before recording the native handle, or if the database stays unavailable beyond the retry budget. No schema, launcher API, or production dependency pin changes. Downstream Oasis must adopt the merged backend commit before this protects its running orchestrator.