Skip to content

fix(resilience): cut silent streams before server detach timeout and clarify OPERATION_ABANDONED - #238

Open
dborowitz wants to merge 1 commit into
mainfrom
dborowitz-operation-abandoned
Open

dborowitz wants to merge 1 commit into
mainfrom
dborowitz-operation-abandoned

Conversation

@dborowitz

Copy link
Copy Markdown
Contributor

Long-running Spark Connect queries can fail during ReattachExecute with [INVALID_HANDLE.OPERATION_ABANDONED] when a backend tunnel or stream behind an open WebSocket stalls silently, or when an operation stays detached (due to a client disconnect or paused iterator consumption) longer than the server's spark.connect.execute.manager.detachedTimeout (5 minutes by default):

  • Lower default MANAGED_SPARK_CONNECT_WS_SILENCE_LIMIT_S from 300 to 150: Spark Connect sends progress updates while tasks run and rotates reattachable streams every 120 s (senderMaxStreamDuration=2m) even when idle. Because forward_bytes checks silence on each 60 s WebSocket recv() poll, the previous 300 s limit waited 300–360 s before cutting a silent stream—past Spark's 5-minute detachedTimeout. Lowering the default to 150 s (> 120 s stream rotation) cuts wedged streams within 150–210 s, leaving time to reconnect and reattach before a 5-minute server timeout expires.
  • Enrich INVALID_HANDLE.OPERATION_ABANDONED without re-running operations: Add install_operation_abandoned_handler in client/reattach_patch.py and install it on every ManagedSparkSession client across _execute_and_fetch_as_iterator, _execute_and_fetch, and _execute. Abandoned operations are never auto-replayed (since write/DDL/DML operations may already have committed); instead the hook emits reattach.operation_abandoned (operation_id, plan_kind, command_kind), closes any open progress bar at its actual completed task count, isolates hook/enrichment errors so the original exception is never masked, and appends Managed Spark recovery guidance onto the raised PySpark exception.
  • Preserve multi-paragraph messages in %sparksql / %%sparksql error formatting: Split formatted Spark SQL error messages into prose paragraphs prior to any SQL/plan context or JVM stack trace so appended Note: paragraphs are rendered instead of truncated at the first blank line.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces handling for INVALID_HANDLE.OPERATION_ABANDONED errors in the Managed Spark Connect client. It adds a handler to enrich these exceptions with recovery guidance, emits telemetry events, and ensures progress bars are closed appropriately when an operation is abandoned. Additionally, it updates the default WebSocket silence limit to 150 seconds to better align with Spark's detached timeout behavior and improves error message formatting to handle multiple paragraphs and JVM stack traces. I have included a suggestion to make the _error_condition helper more robust by checking for string attributes in addition to callable methods.

…clarify OPERATION_ABANDONED

Long-running Spark Connect queries can fail during ReattachExecute with
[INVALID_HANDLE.OPERATION_ABANDONED] when a backend tunnel or stream behind an
open WebSocket stalls silently, or when an operation stays detached (due to a
client disconnect or paused iterator consumption) longer than the server's
spark.connect.execute.manager.detachedTimeout (5 minutes by default):

- Lower default MANAGED_SPARK_CONNECT_WS_SILENCE_LIMIT_S from 300 to 150:
  Spark Connect sends progress updates while tasks run and rotates reattachable
  streams every 120 s (senderMaxStreamDuration=2m) even when idle. Because
  forward_bytes checks silence on each 60 s WebSocket recv() poll, the previous
  300 s limit waited 300-360 s before cutting a silent stream, past Spark's
  5-minute detachedTimeout. Lowering the default to 150 s (> /usr/local/google/home/dborowitz/.gemini/config/120 s stream
  rotation) cuts wedged streams within 150-210 s, leaving time to reconnect and
  reattach before a 5-minute server timeout expires.
- Enrich INVALID_HANDLE.OPERATION_ABANDONED without re-running operations:
  add install_operation_abandoned_handler in client/reattach_patch.py and
  install it on every ManagedSparkSession client across
  _execute_and_fetch_as_iterator, _execute_and_fetch, and _execute. Never
  auto-replay abandoned operations; emit reattach.operation_abandoned
  (operation_id, plan_kind, command_kind); close open progress bars at actual
  completed task counts; isolate hook/enrichment errors so the original
  exception is never masked; and append Managed Spark recovery guidance onto
  the raised PySpark exception.
- Preserve multi-paragraph messages in %sparksql / %%sparksql error formatting:
  split formatted Spark SQL errors into prose paragraphs prior to any SQL/plan
  context or JVM stack trace so appended Note paragraphs are rendered rather
  than truncated at the first blank line.
@dborowitz
dborowitz force-pushed the dborowitz-operation-abandoned branch from edc0cec to e604681 Compare November 16, 2026 22:35
@dborowitz
dborowitz requested review from medb and tim-u November 16, 2026 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant