Repository navigation
fix(resilience): cut silent streams before server detach timeout and clarify OPERATION_ABANDONED - #238
Open
dborowitz wants to merge 1 commit into
Open
fix(resilience): cut silent streams before server detach timeout and clarify OPERATION_ABANDONED#238dborowitz wants to merge 1 commit into
dborowitz wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces handling for INVALID_HANDLE.OPERATION_ABANDONED errors in the Managed Spark Connect client. It adds a handler to enrich these exceptions with recovery guidance, emits telemetry events, and ensures progress bars are closed appropriately when an operation is abandoned. Additionally, it updates the default WebSocket silence limit to 150 seconds to better align with Spark's detached timeout behavior and improves error message formatting to handle multiple paragraphs and JVM stack traces. I have included a suggestion to make the _error_condition helper more robust by checking for string attributes in addition to callable methods.
…clarify OPERATION_ABANDONED Long-running Spark Connect queries can fail during ReattachExecute with [INVALID_HANDLE.OPERATION_ABANDONED] when a backend tunnel or stream behind an open WebSocket stalls silently, or when an operation stays detached (due to a client disconnect or paused iterator consumption) longer than the server's spark.connect.execute.manager.detachedTimeout (5 minutes by default): - Lower default MANAGED_SPARK_CONNECT_WS_SILENCE_LIMIT_S from 300 to 150: Spark Connect sends progress updates while tasks run and rotates reattachable streams every 120 s (senderMaxStreamDuration=2m) even when idle. Because forward_bytes checks silence on each 60 s WebSocket recv() poll, the previous 300 s limit waited 300-360 s before cutting a silent stream, past Spark's 5-minute detachedTimeout. Lowering the default to 150 s (> /usr/local/google/home/dborowitz/.gemini/config/120 s stream rotation) cuts wedged streams within 150-210 s, leaving time to reconnect and reattach before a 5-minute server timeout expires. - Enrich INVALID_HANDLE.OPERATION_ABANDONED without re-running operations: add install_operation_abandoned_handler in client/reattach_patch.py and install it on every ManagedSparkSession client across _execute_and_fetch_as_iterator, _execute_and_fetch, and _execute. Never auto-replay abandoned operations; emit reattach.operation_abandoned (operation_id, plan_kind, command_kind); close open progress bars at actual completed task counts; isolate hook/enrichment errors so the original exception is never masked; and append Managed Spark recovery guidance onto the raised PySpark exception. - Preserve multi-paragraph messages in %sparksql / %%sparksql error formatting: split formatted Spark SQL errors into prose paragraphs prior to any SQL/plan context or JVM stack trace so appended Note paragraphs are rendered rather than truncated at the first blank line.
dborowitz
force-pushed
the
dborowitz-operation-abandoned
branch
from
November 16, 2026 22:35
edc0cec to
e604681
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Long-running Spark Connect queries can fail during
ReattachExecutewith[INVALID_HANDLE.OPERATION_ABANDONED]when a backend tunnel or stream behind an open WebSocket stalls silently, or when an operation stays detached (due to a client disconnect or paused iterator consumption) longer than the server'sspark.connect.execute.manager.detachedTimeout(5 minutes by default):MANAGED_SPARK_CONNECT_WS_SILENCE_LIMIT_Sfrom300to150: Spark Connect sends progress updates while tasks run and rotates reattachable streams every 120 s (senderMaxStreamDuration=2m) even when idle. Becauseforward_byteschecks silence on each 60 s WebSocketrecv()poll, the previous 300 s limit waited 300–360 s before cutting a silent stream—past Spark's 5-minutedetachedTimeout. Lowering the default to 150 s (> 120 s stream rotation) cuts wedged streams within 150–210 s, leaving time to reconnect and reattach before a 5-minute server timeout expires.INVALID_HANDLE.OPERATION_ABANDONEDwithout re-running operations: Addinstall_operation_abandoned_handlerinclient/reattach_patch.pyand install it on everyManagedSparkSessionclient across_execute_and_fetch_as_iterator,_execute_and_fetch, and_execute. Abandoned operations are never auto-replayed (since write/DDL/DML operations may already have committed); instead the hook emitsreattach.operation_abandoned(operation_id,plan_kind,command_kind), closes any open progress bar at its actual completed task count, isolates hook/enrichment errors so the original exception is never masked, and appends Managed Spark recovery guidance onto the raised PySpark exception.%sparksql/%%sparksqlerror formatting: Split formatted Spark SQL error messages into prose paragraphs prior to any SQL/plan context or JVM stack trace so appendedNote:paragraphs are rendered instead of truncated at the first blank line.