Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ These environment variables change how the client behaves.
| Variable | Default | Effect |
|----------|---------|--------|
| `MANAGED_SPARK_CONNECT_REPLAY_ON_POSITION_LOST` | `1` | When a read query (`collect`, `toPandas`, `count`, ...) fails with `INVALID_CURSOR.POSITION_NOT_AVAILABLE` because the server no longer buffers the result the client has to resume from, re-run it once under a new operation id and log a `WARNING`. Commands (writes, DDL) are never re-run, and neither are queries whose plan contains a UDF, UDTF, `mapInPandas`, `mapInArrow` or `applyInPandas`, or that come from `foreach` or `foreachPartition`, since re-running them re-runs any side effects of that code. UDFs only called by name (registered with `spark.udf.register` and called from SQL) aren't detected. Set to `0` to disable, e.g. for queries that are expensive or not deterministic, or that call side-effecting UDFs by name. Read when the session is created. |
| `MANAGED_SPARK_CONNECT_WS_SILENCE_LIMIT_S` | `300` | Cut the proxy WebSocket connection if a query's result stream (a reattachable `ExecutePlan` or `ReattachExecute`) has been waiting this many seconds for its next response and nothing at all was received, so a wedged stream fails and pyspark reattaches instead of hanging. By default the server ends these streams every 2 minutes even when it has nothing to send, so that much silence means the connection is broken. Other RPCs, such as a long `AnalyzePlan`, and result streams that nobody is reading from don't count. The check runs on each 60 s WebSocket receive poll, so the cut happens between this limit and the limit plus 60 s. Set to `0` to disable. |
| `MANAGED_SPARK_CONNECT_WS_SILENCE_LIMIT_S` | `150` | Cut the proxy WebSocket connection if a query's result stream (a reattachable `ExecutePlan` or `ReattachExecute`) has been waiting this many seconds for its next response and nothing at all was received, so a wedged stream behind an open WebSocket fails and pyspark reattaches instead of hanging or hitting `INVALID_HANDLE.OPERATION_ABANDONED`. By default the server sends progress updates while tasks run and ends these streams every 2 minutes (`spark.connect.execute.reattachable.senderMaxStreamDuration`) even when it has nothing to send; keep custom values above `senderMaxStreamDuration` so idle streams are not cut prematurely. Other RPCs, such as a long `AnalyzePlan`, and result streams that nobody is reading from don't count. The check runs on each 60 s WebSocket receive poll, so the cut happens between this limit and the limit plus 60 s (150–210 s by default, before Spark's 5-minute default `detachedTimeout`). Set to `0` to disable. |
| `MANAGED_SPARK_CONNECT_LIVENESS_TTL_SECONDS` | unset | Before an RPC, the client checks that the Dataproc session is still alive. Once a check finds it `ACTIVE`, that answer is reused for this many seconds: unset or negative reuses it until an RPC fails, `0` checks before every RPC. A failed RPC always triggers a fresh check. The session only counts as gone when Dataproc reports it not found or in a terminal state; when the check itself fails (for example during a control-plane outage), the RPC proceeds. An RPC on a gone session raises `SessionNotActiveError` (a `RuntimeError`, from `google.cloud.managed_spark_connect.exceptions`) saying what Dataproc reported, e.g. `FAILED` and why. |
| `MANAGED_SPARK_CONNECT_SESSION_START_RETRIES` | `1` | `getOrCreate()` makes a new session's first RPC itself, and if that fails and Dataproc reports the session gone (failed or terminated before it could be used), creates another session in its place, this many times, logging a `WARNING`. Only for generated session IDs: a session created with `sessionId()` raises `SessionNotActiveError` instead. `0` disables retrying. |
| `GRPC_VERBOSITY` | `ERROR` | gRPC's own variable. If it is unset when this package is imported, and gRPC hasn't been imported yet, the package sets it to `ERROR` for the whole process to silence gRPC's informational fork messages. Set it yourself to override. |
Expand Down Expand Up @@ -192,7 +192,8 @@ result conversions — as timed *spans*. It is off by default; opt in via

Without instrumentation the session still hooks a few pyspark client methods
on its own client (operation ids and links, progress bars, the
`INVALID_CURSOR.POSITION_NOT_AVAILABLE` replay; see
`INVALID_CURSOR.POSITION_NOT_AVAILABLE` replay, and
`INVALID_HANDLE.OPERATION_ABANDONED` error guidance; see
[Configuration](#configuration-environment-variables)) and coalesces the
`ReleaseExecute` RPCs of its result streams.

Expand Down
9 changes: 6 additions & 3 deletions docs/instrumentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,10 @@ JsonlFileSink(path, *, append=True) InMemorySink() LoggingSink(logger=None,
`wrap_client=False` installs no instrumentation wrappers on the pyspark client
(no `client.*` spans). The session's own client hooks are installed either way:
they set operation ids, display operation links, close a query's progress
bar when it ends and re-run a read query once on
`INVALID_CURSOR.POSITION_NOT_AVAILABLE` (see `reattach.position_lost` below).
bar when it ends, re-run a read query once on
`INVALID_CURSOR.POSITION_NOT_AVAILABLE` (see `reattach.position_lost` below),
and enrich `INVALID_HANDLE.OPERATION_ABANDONED` errors without re-running the
operation (see `reattach.operation_abandoned` below).
Every session also coalesces the `ReleaseExecute(until)` RPCs of its result
streams, through a process-wide patch of pyspark's
`ExecutePlanResponseReattachableIterator._release_until` that leaves the
Expand Down Expand Up @@ -142,11 +144,12 @@ sessions; a session without an `Instrumentation` pays nothing for them.
| `auth.token_refreshed` | The Application Default Credentials were actually refreshed while opening a websocket (inside the `auth.refresh` span) | `expiry_wall_ns` (epoch ns of the new token's expiry, `null` if unknown), `had_token` (a token existed before the refresh) |
| `progress.bar_open` | The execution progress handler created a progress bar for an operation (first progress message with tasks that is not already final) | `operation_id`, `total_tasks` |
| `progress.bar_update` | Every later progress message for an open bar, except the final one | `operation_id`, `completed_tasks`, `total_tasks` |
| `progress.bar_close` | The bar was closed and unregistered: by the handler on the final (`done=True`) message, or, if that never came, when the query's `_execute_and_fetch` returned or raised | `operation_id`, `completed_tasks`, `total_tasks`, `reason=done\|query_end\|replay` (`replay`: the query hit `INVALID_CURSOR.POSITION_NOT_AVAILABLE` and is re-run under a new operation id) |
| `progress.bar_close` | The bar was closed and unregistered: by the handler on the final (`done=True`) message, or, if that never came, when the execution hook (`_execute_and_fetch_as_iterator`, `_execute_and_fetch`, or `_execute`) returned or raised | `operation_id`, `completed_tasks`, `total_tasks`, `reason=done\|query_end\|replay` (`replay`: the query hit `INVALID_CURSOR.POSITION_NOT_AVAILABLE` and is re-run under a new operation id) |
| `guard.result` | The session liveness guard ran (before an RPC, or after a failed one) and either had to ask the control plane or found the session inactive. A cached "active" verdict, the normal case for every RPC, emits nothing | `mode=pre_rpc\|on_error` (`direct` when called outside the channel), `cached` (answered from the liveness cache, no `GetSession` RPC), `active` (the verdict; `null` when the check raised), `error_type` (only when the check raised) |
| `grpc.retry` | pyspark's retry loop is about to retry an RPC, after the policy's backoff. The failed RPC is the `grpc.*` span that ended just before, on the same `thread` | `attempt` (1 = first retry within that loop), `reason=error\|reattach` (`reattach`: the server ended the stream and pyspark reattaches), `grpc_code`, `error_type` |
| `session.start_failed` | A session `getOrCreate()` just created never became usable: Dataproc reported it failed or terminated, either in the `CreateSession` result, while waiting for its endpoint, or after its first RPC failed (see `session.check_new`) | `detail` (what Dataproc reported, e.g. `Managed Spark Session sc-123 is FAILED: <reason>`), `attempt` (1 for the first session), `retry` (whether another session is created in its place: only for generated session IDs, up to `MANAGED_SPARK_CONNECT_SESSION_START_RETRIES` times) |
| `reattach.position_lost` | A query's `_execute_and_fetch` (`collect`, `toPandas`, `count`, ...) failed with `INVALID_CURSOR.POSITION_NOT_AVAILABLE`: the server no longer buffers the response the client has to reattach from. A replay also logs a `WARNING` | `operation_id`, `replay` (whether the query is re-run once under a new operation id: only root (read) plans without inline UDFs or UDTFs, and not with `MANAGED_SPARK_CONNECT_REPLAY_ON_POSITION_LOST=0`; if the re-run fails the same way, a second event has its `operation_id` and `replay=false`), `new_operation_id` (only when `replay=true`) |
| `reattach.operation_abandoned` | An execution (`_execute_and_fetch_as_iterator`, `_execute_and_fetch`, or `_execute`) failed with `INVALID_HANDLE.OPERATION_ABANDONED`: the operation stayed detached (for example due to lost connectivity or paused iterator consumption) longer than `spark.connect.execute.manager.detachedTimeout` and was removed by the server. Never re-run automatically; recovery guidance is appended onto the raised PySpark exception | `operation_id`, `plan_kind=root\|command` (when present on the `ExecutePlanRequest`), `command_kind` (only when `plan_kind=command` and set, e.g. `sql_command`, `write_operation`) |

Notes: `progress.*` events require the progress handler to be active,
which it only is in interactive environments.
Expand Down
12 changes: 11 additions & 1 deletion google/cloud/managed_spark_connect/client/proxy.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,18 @@

# Cut a connection whose server stays silent this long while a result stream
# is waiting for its next response (see forward_bytes). 0 disables.
#
# While WebSocket keepalive catches a dead first-hop peer in ~80 s, a backend
# tunnel or HTTP/2 stream behind an open WebSocket frontend can stall silently
# while PONGs keep flowing. Spark Connect sends progress updates while tasks
# run and rotates reattachable streams every 120 s (senderMaxStreamDuration=2m)
# even when idle, while detached operations are evicted after
# spark.connect.execute.manager.detachedTimeout (5m by default on unmodified
# servers). Because recv() polls every WS_RECV_TIMEOUT_S (60 s), 150 s (> 120 s
# stream rotation) typically cuts a silent stream within 150-210 s, leaving
# time to reconnect and send ReattachExecute before a 5m server timeout expires.
_SILENCE_LIMIT_ENV = "MANAGED_SPARK_CONNECT_WS_SILENCE_LIMIT_S"
_DEFAULT_SILENCE_LIMIT_S = 300.0
_DEFAULT_SILENCE_LIMIT_S = 150.0


def clear_credentials_cache():
Expand Down
167 changes: 161 additions & 6 deletions google/cloud/managed_spark_connect/client/reattach_patch.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Client-side recovery from ``INVALID_CURSOR.POSITION_NOT_AVAILABLE``.
"""Client-side reattach recovery, abandoned-handle guidance, and release coalescing.

When a transport blip breaks an ``ExecutePlan`` stream, pyspark reattaches
from the last response it returned. If the server no longer buffers that
Expand All @@ -20,14 +20,25 @@
``DefaultPolicy`` does not retry, and the whole query fails.

:func:`install_replay_on_position_lost` hooks ``client._execute_and_fetch``
``client._execute_and_fetch`` (``collect``, ``toPandas``, ``count``, ...):
(``collect``, ``toPandas``, ``count``, ...):
when a root (read) plan fails that way, the query is re-run once under a
fresh ``operation_id``. Command plans, and plans with inline user code
(UDFs, ``mapInPandas``, ``foreach``, ...) whose side effects would run
again, are never re-run. Every such failure emits
``reattach.position_lost``, and a replay also logs a ``WARNING``.
``MANAGED_SPARK_CONNECT_REPLAY_ON_POSITION_LOST=0`` turns the replay off.

:func:`install_operation_abandoned_handler` hooks the client's execution
methods (``_execute_and_fetch_as_iterator``, ``_execute_and_fetch`` and
``_execute``) for ``[INVALID_HANDLE.OPERATION_ABANDONED]``, which the server
raises when an operation stays detached (for example because connectivity was
lost or an iterator consumer paused reading) longer than
``spark.connect.execute.manager.detachedTimeout`` and is removed by the
server. Operations are never re-run automatically after abandonment; instead
the hook emits ``reattach.operation_abandoned``, closes any open progress bar
for the operation, and appends actionable Managed Spark recovery guidance onto
the PySpark exception in place.

:func:`install_release_coalescing` makes the reattachable iterators of one
client send at most one ``ReleaseExecute(until)`` at a time. pyspark sends
one per response, each on its release thread pool, so a fast stream behind a
Expand Down Expand Up @@ -56,13 +67,68 @@

ENV_REPLAY_ON_POSITION_LOST = "MANAGED_SPARK_CONNECT_REPLAY_ON_POSITION_LOST"
POSITION_LOST_CONDITION = "INVALID_CURSOR.POSITION_NOT_AVAILABLE"
OPERATION_ABANDONED_CONDITION = "INVALID_HANDLE.OPERATION_ABANDONED"

_OPERATION_ABANDONED_HANDLED_ATTR = "_msc_operation_abandoned_handled"
_OPERATION_ABANDONED_NOTE = (
"Note: The client lost connectivity to (or paused reading results from) "
"the Managed Spark session for longer than "
"'spark.connect.execute.manager.detachedTimeout', so the server removed "
"the detached operation (which may or may not have completed). The "
"session is still usable; for write, DDL, or DML statements, check "
"whether changes were committed before re-running. To tolerate longer "
"client disconnects on new sessions, set "
".config('spark.connect.execute.manager.detachedTimeout', '30m') (or a "
"longer duration) when creating the session."
)


def _error_condition(exc: BaseException) -> Optional[str]:
try:
cond = getattr(exc, "getCondition", None) or getattr(
exc, "getErrorClass", None
)
except Exception:
return None
if isinstance(cond, str):
return cond or None
if callable(cond):
try:
value = cond()
except Exception:
return None
return value if isinstance(value, str) and value else None
return None


def _is_position_not_available(exc: BaseException) -> bool:
cond = getattr(exc, "getCondition", None) or getattr(
exc, "getErrorClass", None
)
return callable(cond) and cond() == POSITION_LOST_CONDITION
return _error_condition(exc) == POSITION_LOST_CONDITION


def _is_operation_abandoned(exc: BaseException) -> bool:
return _error_condition(exc) == OPERATION_ABANDONED_CONDITION


def _enrich_operation_abandoned_error(exc: BaseException) -> None:
"""Append Managed Spark recovery guidance onto ``exc`` in place."""
try:
msg = getattr(exc, "_message", None)
if isinstance(msg, str) and _OPERATION_ABANDONED_NOTE not in msg:
sep = "\n\n" if msg else ""
exc._message = f"{msg}{sep}{_OPERATION_ABANDONED_NOTE}"
if (
isinstance(exc.args, tuple)
and len(exc.args) == 1
and isinstance(exc.args[0], str)
and _OPERATION_ABANDONED_NOTE not in exc.args[0]
):
exc.args = (exc._message,)
except Exception:
logger.debug(
"Failed to enrich %s exception message",
OPERATION_ABANDONED_CONDITION,
exc_info=True,
)


# Inline user code in a plan: Python, Scala and Java UDFs (including the
Expand Down Expand Up @@ -197,6 +263,95 @@ def _execute_and_fetch(req, *args, **kwargs):
install_client_hooks(client, {"_execute_and_fetch": replay_hook})


def _record_and_enrich_abandoned(
exc: BaseException,
req: Any,
recorder: Any,
close_progress_bar: Optional[Callable[..., None]],
) -> None:
if not _is_operation_abandoned(exc):
return
if getattr(exc, _OPERATION_ABANDONED_HANDLED_ATTR, False):
return
try:
setattr(exc, _OPERATION_ABANDONED_HANDLED_ATTR, True)
except Exception: # pragma: no cover - defensive for custom exceptions
pass
operation_id = getattr(req, "operation_id", None) or ""
try:
attrs = {"operation_id": operation_id}
plan_kind, command_kind = _plan_kinds(req)
if plan_kind is not None:
attrs["plan_kind"] = plan_kind
if command_kind is not None:
attrs["command_kind"] = command_kind
recorder.event("reattach.operation_abandoned", **attrs)
except Exception:
logger.debug(
"Failed to record reattach.operation_abandoned event",
exc_info=True,
)
_enrich_operation_abandoned_error(exc)
if close_progress_bar is not None:
try:
close_progress_bar(operation_id or None)
except Exception:
logger.debug(
"Failed to close progress bar for abandoned operation %s",
operation_id,
exc_info=True,
)


def install_operation_abandoned_handler(
client: Any,
recorder: Any = NOOP_RECORDER,
close_progress_bar: Optional[Callable[..., None]] = None,
) -> None:
"""Emit ``reattach.operation_abandoned`` and enrich abandoned-handle errors.

Hooks whichever of ``_execute_and_fetch_as_iterator``,
``_execute_and_fetch`` and ``_execute`` exist on ``client`` so all query,
SQL command (including DDL/DML) and ``DataFrameWriter`` paths are covered
without re-running the operation or duplicating telemetry/guidance when
``_execute_and_fetch`` delegates to ``_execute_and_fetch_as_iterator``.
"""

def make_sync_hook(inner):
def wrapped(req, *args, **kwargs):
try:
return inner(req, *args, **kwargs)
except Exception as exc:
_record_and_enrich_abandoned(
exc, req, recorder, close_progress_bar
)
raise

return wrapped

def make_iterator_hook(inner):
def wrapped(req, *args, **kwargs):
try:
yield from inner(req, *args, **kwargs)
except Exception as exc:
_record_and_enrich_abandoned(
exc, req, recorder, close_progress_bar
)
raise

return wrapped

hooks = {}
if hasattr(client, "_execute_and_fetch_as_iterator"):
hooks["_execute_and_fetch_as_iterator"] = make_iterator_hook
if hasattr(client, "_execute_and_fetch"):
hooks["_execute_and_fetch"] = make_sync_hook
if hasattr(client, "_execute"):
hooks["_execute"] = make_sync_hook
if hooks:
install_client_hooks(client, hooks)


# Stubs of the clients install_release_coalescing() was called for. An
# iterator coalesces only if it talks through one of them.
_COALESCED_STUBS: "weakref.WeakSet[Any]" = weakref.WeakSet()
Expand Down
Loading
Loading