Repository navigation
Conversation
Wires optional notebook cell monitoring into `ManagedSparkSession` and
IPython interactive extras via a lightweight lifecycle bridge:
- `google.cloud.managed_spark_connect._ipython`: auto-loads
`managed_spark_cell_monitoring.spark_connect` (`%cellmonitor`) alongside
installed interactive extras when available.
- `google.cloud.managed_spark_connect.client.core`: adds `/GetStatus` to
`_UNGUARDED_METHOD_SUFFIXES` so background `GetStatus` polling runs
concurrently with in-flight `ExecutePlan` streams.
- `google.cloud.managed_spark_connect._cell_monitoring` & `session`:
- Lazily delegates `begin_operation`, `finish_operation(..., outcome=...)`
(`SUCCEEDED`, `FAILED`, `CANCELLED`), and task progress suppression to
`managed_spark_cell_monitoring.spark_connect.create_session_poller(...)`
with zero imports or overhead when the extension is inactive.
- Restores per-operation Console links and legacy `tqdm` progress bars
automatically if the session falls back.
There was a problem hiding this comment.
Code Review
This pull request introduces a thin lifecycle bridge (SessionCellMonitorBridge) to integrate with managed_spark_cell_monitoring.spark_connect for cell monitoring in Managed Spark Connect sessions. Key changes include tracking operation outcomes, suppressing progress bars, and handling fallbacks. The review feedback highlights three main improvement opportunities: wrapping the exception classification logic in classify_exception_outcome to prevent secondary exceptions from masking the original error, avoiding exception classification in _execute when the operation is not monitored, and ensuring the poller is closed during garbage collection while using getattr instead of direct __dict__ access.
| def classify_exception_outcome(exc: BaseException) -> str: | ||
| """Classifies an `ExecutePlan` exception into `CANCELLED` or `FAILED`.""" | ||
| if isinstance(exc, (KeyboardInterrupt, GeneratorExit)): | ||
| return "CANCELLED" | ||
|
|
||
| seen: set[int] = set() | ||
| cur: Optional[BaseException] = exc | ||
| while cur is not None and id(cur) not in seen: | ||
| seen.add(id(cur)) | ||
| code_fn = getattr(cur, "code", None) | ||
| if callable(code_fn): | ||
| try: | ||
| code = code_fn() | ||
| if getattr(code, "name", None) == "CANCELLED": | ||
| return "CANCELLED" | ||
| except Exception: | ||
| pass | ||
| grpc_code = getattr(cur, "grpc_status_code", None) | ||
| if getattr(grpc_code, "name", None) == "CANCELLED": | ||
| return "CANCELLED" | ||
| err_cls_fn = getattr(cur, "getErrorClass", None) | ||
| if callable(err_cls_fn): | ||
| try: | ||
| err_cls = err_cls_fn() or "" | ||
| if "CANCELED" in err_cls or "CANCELLED" in err_cls: | ||
| return "CANCELLED" | ||
| except Exception: | ||
| pass | ||
| msg = str(cur) | ||
| if "OPERATION_CANCELED" in msg or "JOB_CANCELLED" in msg: | ||
| return "CANCELLED" | ||
| cur = cur.__cause__ or cur.__context__ | ||
|
|
||
| return "FAILED" |
There was a problem hiding this comment.
The exception classification logic in classify_exception_outcome accesses various attributes and methods (like code(), grpc_status_code, getErrorClass(), and str(cur)) on the exception chain. If any of these custom exceptions have buggy implementations or descriptors that raise exceptions, classify_exception_outcome will raise an unhandled exception. This will propagate and mask the original exception, which is a critical robustness issue. Wrapping the entire classification logic in a try...except BaseException: block ensures that the original exception is never masked and "FAILED" is safely returned as a fallback.
def classify_exception_outcome(exc: BaseException) -> str:
"""Classifies an `ExecutePlan` exception into `CANCELLED` or `FAILED`."""
try:
if isinstance(exc, (KeyboardInterrupt, GeneratorExit)):
return "CANCELLED"
seen: set[int] = set()
cur: Optional[BaseException] = exc
while cur is not None and id(cur) not in seen:
seen.add(id(cur))
code_fn = getattr(cur, "code", None)
if callable(code_fn):
try:
code = code_fn()
if getattr(code, "name", None) == "CANCELLED":
return "CANCELLED"
except Exception:
pass
try:
grpc_code = getattr(cur, "grpc_status_code", None)
if getattr(grpc_code, "name", None) == "CANCELLED":
return "CANCELLED"
except Exception:
pass
err_cls_fn = getattr(cur, "getErrorClass", None)
if callable(err_cls_fn):
try:
err_cls = err_cls_fn() or ""
if "CANCELED" in err_cls or "CANCELLED" in err_cls:
return "CANCELLED"
except Exception:
pass
try:
msg = str(cur)
if "OPERATION_CANCELED" in msg or "JOB_CANCELLED" in msg:
return "CANCELLED"
except Exception:
pass
try:
cur = cur.__cause__ or cur.__context__
except Exception:
break
except BaseException:
pass
return "FAILED"| poller = self.__dict__.get("_cell_monitor_poller") | ||
| if poller is not None: | ||
| poller.close() |
There was a problem hiding this comment.
While closing the poller in _close_client is correct, we should also ensure the poller is closed when the session is garbage collected (i.e., in __del__). Currently, __del__ only calls super().__del__() if _client_closed is False, which does not trigger _close_client or close the poller. This can lead to background thread/resource leaks if a session goes out of scope without being explicitly stopped. Additionally, using getattr(self, "_cell_monitor_poller", None) is more idiomatic than direct __dict__ access.
| poller = self.__dict__.get("_cell_monitor_poller") | |
| if poller is not None: | |
| poller.close() | |
| poller = getattr(self, "_cell_monitor_poller", None) | |
| if poller is not None: | |
| poller.close() |
| def execute_hook(inner): | ||
| def _execute(req, *args, **kwargs): | ||
| monitored = False | ||
| if not self._sql_lazy_transformation(req): | ||
| self._display_operation_link(req.operation_id) | ||
| return inner(req, *args, **kwargs) | ||
| monitored = self._cell_monitor_poller.begin_operation( | ||
| req.operation_id | ||
| ) | ||
| if not monitored: | ||
| self._display_operation_link(req.operation_id) | ||
| outcome = "SUCCEEDED" | ||
| try: | ||
| return inner(req, *args, **kwargs) | ||
| except BaseException as exc: | ||
| outcome = _cell_monitoring.classify_exception_outcome(exc) | ||
| raise | ||
| finally: | ||
| if monitored: | ||
| self._cell_monitor_poller.finish_operation( | ||
| req.operation_id, outcome=outcome | ||
| ) | ||
|
|
||
| return _execute |
There was a problem hiding this comment.
In _execute, _cell_monitoring.classify_exception_outcome(exc) is called unconditionally when an exception is raised, even if monitored is False. This is inefficient and introduces unnecessary risk of secondary exceptions masking the original error on unmonitored operations. We should only classify the exception if monitored is True, matching the pattern used in _execute_and_fetch_as_iterator.
def execute_hook(inner):
def _execute(req, *args, **kwargs):
monitored = False
if not self._sql_lazy_transformation(req):
monitored = self._cell_monitor_poller.begin_operation(
req.operation_id
)
if not monitored:
self._display_operation_link(req.operation_id)
outcome = "SUCCEEDED"
try:
return inner(req, *args, **kwargs)
except BaseException as exc:
if monitored:
outcome = _cell_monitoring.classify_exception_outcome(exc)
raise
finally:
if monitored:
self._cell_monitor_poller.finish_operation(
req.operation_id, outcome=outcome
)
return _execute
Summary
Integrates
ManagedSparkSessionand interactive IPython extras withmanaged-spark-cell-monitoring's Spark ConnectGetStatuspoller via a thin lifecycle bridge:_ipython.py): Auto-loadsmanaged_spark_cell_monitoring.spark_connect(%cellmonitor) when installed in IPython notebooks, respectingMANAGED_SPARK_CONNECT_ENABLE_EXTRASand%config ManagedSparkConnect.enable_extras.client/core.py): Adds"/GetStatus"to_UNGUARDED_METHOD_SUFFIXESso backgroundGetStatusRPCs can poll concurrently while anExecutePlanstream is active on the channel._cell_monitoring.py,session.py):managed_spark_cell_monitoring.spark_connect.create_session_poller(...)with zero overhead or extra imports whenmanaged-spark-cell-monitoringis not installed or inactive._executeand_execute_and_fetch_as_iteratoracross full stream lifetimes, classifying completion outcomes (SUCCEEDED,FAILED,CANCELLED) forfinish_operation(...).tqdmprogress bars while cell monitoring is active for an operation, restoring them automatically via session fallback if the server does not serveCellMonitorTelemetry.Companion PRs in
GoogleCloudDataproc/managed-spark-cell-monitoring:GetStatus+cell_monitor.protostubs)GetStatuspoller & AnyWidget snapshot mode, stacked on fix: unit test expected error message #54)Testing
pyink . --checkvermin --no-tips --violations --eval-annotations --target=3.10- google tests setup.pypytest tests/unit/test_cell_monitoring.py tests/unit/test_init.pyand full unit test suitepython -m build --no-isolation&twine check --strict dist/*