Skip to content

feat: integrate ManagedSparkSession with Spark Connect cell monitoring - #237

Open
ptch314 wants to merge 1 commit into
mainfrom
feat/spark-connect-cell-monitoring
Open

ptch314 wants to merge 1 commit into
mainfrom
feat/spark-connect-cell-monitoring

Conversation

@ptch314

@ptch314 ptch314 commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Integrates ManagedSparkSession and interactive IPython extras with managed-spark-cell-monitoring's Spark Connect GetStatus poller via a thin lifecycle bridge:

  • Interactive Extras (_ipython.py): Auto-loads managed_spark_cell_monitoring.spark_connect (%cellmonitor) when installed in IPython notebooks, respecting MANAGED_SPARK_CONNECT_ENABLE_EXTRAS and %config ManagedSparkConnect.enable_extras.
  • Client Concurrency (client/core.py): Adds "/GetStatus" to _UNGUARDED_METHOD_SUFFIXES so background GetStatus RPCs can poll concurrently while an ExecutePlan stream is active on the channel.
  • Thin Session Lifecycle Bridge (_cell_monitoring.py, session.py):
    • Lazily resolves managed_spark_cell_monitoring.spark_connect.create_session_poller(...) with zero overhead or extra imports when managed-spark-cell-monitoring is not installed or inactive.
    • Wraps _execute and _execute_and_fetch_as_iterator across full stream lifetimes, classifying completion outcomes (SUCCEEDED, FAILED, CANCELLED) for finish_operation(...).
    • Suppresses duplicate per-operation links and tqdm progress bars while cell monitoring is active for an operation, restoring them automatically via session fallback if the server does not serve CellMonitorTelemetry.

Companion PRs in GoogleCloudDataproc/managed-spark-cell-monitoring:

Testing

  • pyink . --check
  • vermin --no-tips --violations --eval-annotations --target=3.10- google tests setup.py
  • pytest tests/unit/test_cell_monitoring.py tests/unit/test_init.py and full unit test suite
  • python -m build --no-isolation & twine check --strict dist/*

Wires optional notebook cell monitoring into `ManagedSparkSession` and
IPython interactive extras via a lightweight lifecycle bridge:

- `google.cloud.managed_spark_connect._ipython`: auto-loads
  `managed_spark_cell_monitoring.spark_connect` (`%cellmonitor`) alongside
  installed interactive extras when available.
- `google.cloud.managed_spark_connect.client.core`: adds `/GetStatus` to
  `_UNGUARDED_METHOD_SUFFIXES` so background `GetStatus` polling runs
  concurrently with in-flight `ExecutePlan` streams.
- `google.cloud.managed_spark_connect._cell_monitoring` & `session`:
  - Lazily delegates `begin_operation`, `finish_operation(..., outcome=...)`
    (`SUCCEEDED`, `FAILED`, `CANCELLED`), and task progress suppression to
    `managed_spark_cell_monitoring.spark_connect.create_session_poller(...)`
    with zero imports or overhead when the extension is inactive.
  - Restores per-operation Console links and legacy `tqdm` progress bars
    automatically if the session falls back.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a thin lifecycle bridge (SessionCellMonitorBridge) to integrate with managed_spark_cell_monitoring.spark_connect for cell monitoring in Managed Spark Connect sessions. Key changes include tracking operation outcomes, suppressing progress bars, and handling fallbacks. The review feedback highlights three main improvement opportunities: wrapping the exception classification logic in classify_exception_outcome to prevent secondary exceptions from masking the original error, avoiding exception classification in _execute when the operation is not monitored, and ensuring the poller is closed during garbage collection while using getattr instead of direct __dict__ access.

Comment on lines +24 to +57
def classify_exception_outcome(exc: BaseException) -> str:
"""Classifies an `ExecutePlan` exception into `CANCELLED` or `FAILED`."""
if isinstance(exc, (KeyboardInterrupt, GeneratorExit)):
return "CANCELLED"

seen: set[int] = set()
cur: Optional[BaseException] = exc
while cur is not None and id(cur) not in seen:
seen.add(id(cur))
code_fn = getattr(cur, "code", None)
if callable(code_fn):
try:
code = code_fn()
if getattr(code, "name", None) == "CANCELLED":
return "CANCELLED"
except Exception:
pass
grpc_code = getattr(cur, "grpc_status_code", None)
if getattr(grpc_code, "name", None) == "CANCELLED":
return "CANCELLED"
err_cls_fn = getattr(cur, "getErrorClass", None)
if callable(err_cls_fn):
try:
err_cls = err_cls_fn() or ""
if "CANCELED" in err_cls or "CANCELLED" in err_cls:
return "CANCELLED"
except Exception:
pass
msg = str(cur)
if "OPERATION_CANCELED" in msg or "JOB_CANCELLED" in msg:
return "CANCELLED"
cur = cur.__cause__ or cur.__context__

return "FAILED"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The exception classification logic in classify_exception_outcome accesses various attributes and methods (like code(), grpc_status_code, getErrorClass(), and str(cur)) on the exception chain. If any of these custom exceptions have buggy implementations or descriptors that raise exceptions, classify_exception_outcome will raise an unhandled exception. This will propagate and mask the original exception, which is a critical robustness issue. Wrapping the entire classification logic in a try...except BaseException: block ensures that the original exception is never masked and "FAILED" is safely returned as a fallback.

def classify_exception_outcome(exc: BaseException) -> str:
    """Classifies an `ExecutePlan` exception into `CANCELLED` or `FAILED`."""
    try:
        if isinstance(exc, (KeyboardInterrupt, GeneratorExit)):
            return "CANCELLED"

        seen: set[int] = set()
        cur: Optional[BaseException] = exc
        while cur is not None and id(cur) not in seen:
            seen.add(id(cur))
            code_fn = getattr(cur, "code", None)
            if callable(code_fn):
                try:
                    code = code_fn()
                    if getattr(code, "name", None) == "CANCELLED":
                        return "CANCELLED"
                except Exception:
                    pass
            try:
                grpc_code = getattr(cur, "grpc_status_code", None)
                if getattr(grpc_code, "name", None) == "CANCELLED":
                    return "CANCELLED"
            except Exception:
                pass
            err_cls_fn = getattr(cur, "getErrorClass", None)
            if callable(err_cls_fn):
                try:
                    err_cls = err_cls_fn() or ""
                    if "CANCELED" in err_cls or "CANCELLED" in err_cls:
                        return "CANCELLED"
                except Exception:
                    pass
            try:
                msg = str(cur)
                if "OPERATION_CANCELED" in msg or "JOB_CANCELLED" in msg:
                    return "CANCELLED"
            except Exception:
                pass
            try:
                cur = cur.__cause__ or cur.__context__
            except Exception:
                break
    except BaseException:
        pass

    return "FAILED"

Comment on lines +2638 to +2640
poller = self.__dict__.get("_cell_monitor_poller")
if poller is not None:
poller.close()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

While closing the poller in _close_client is correct, we should also ensure the poller is closed when the session is garbage collected (i.e., in __del__). Currently, __del__ only calls super().__del__() if _client_closed is False, which does not trigger _close_client or close the poller. This can lead to background thread/resource leaks if a session goes out of scope without being explicitly stopped. Additionally, using getattr(self, "_cell_monitor_poller", None) is more idiomatic than direct __dict__ access.

Suggested change
poller = self.__dict__.get("_cell_monitor_poller")
if poller is not None:
poller.close()
poller = getattr(self, "_cell_monitor_poller", None)
if poller is not None:
poller.close()

Comment on lines 2087 to 2108
def execute_hook(inner):
def _execute(req, *args, **kwargs):
monitored = False
if not self._sql_lazy_transformation(req):
self._display_operation_link(req.operation_id)
return inner(req, *args, **kwargs)
monitored = self._cell_monitor_poller.begin_operation(
req.operation_id
)
if not monitored:
self._display_operation_link(req.operation_id)
outcome = "SUCCEEDED"
try:
return inner(req, *args, **kwargs)
except BaseException as exc:
outcome = _cell_monitoring.classify_exception_outcome(exc)
raise
finally:
if monitored:
self._cell_monitor_poller.finish_operation(
req.operation_id, outcome=outcome
)

return _execute

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In _execute, _cell_monitoring.classify_exception_outcome(exc) is called unconditionally when an exception is raised, even if monitored is False. This is inefficient and introduces unnecessary risk of secondary exceptions masking the original error on unmonitored operations. We should only classify the exception if monitored is True, matching the pattern used in _execute_and_fetch_as_iterator.

        def execute_hook(inner):
            def _execute(req, *args, **kwargs):
                monitored = False
                if not self._sql_lazy_transformation(req):
                    monitored = self._cell_monitor_poller.begin_operation(
                        req.operation_id
                    )
                    if not monitored:
                        self._display_operation_link(req.operation_id)
                outcome = "SUCCEEDED"
                try:
                    return inner(req, *args, **kwargs)
                except BaseException as exc:
                    if monitored:
                        outcome = _cell_monitoring.classify_exception_outcome(exc)
                    raise
                finally:
                    if monitored:
                        self._cell_monitor_poller.finish_operation(
                            req.operation_id, outcome=outcome
                        )

            return _execute

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant