Repository navigation
feat(api_core): add method-level terminal error logging in _GapicCallable - #19012
chalmerlowe wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces structured error and cancellation logging for GAPIC method calls in both synchronous and asynchronous contexts. It adds _emit_method_error_log and _report_method_error to enrich OpenTelemetry spans and log terminal RPC failures or cancellations, along with comprehensive tests. The review feedback suggests wrapping _report_method_error in a try...except block to ensure telemetry errors fail open without disrupting the main RPC flow, and logging any exceptions as warnings. Additionally, it recommends avoiding the mutation of the passed error_attrs dictionary in _emit_method_error_log to prevent unintended side-effects.
| error_logging_enabled = client_logging.is_error_logging_enabled( | ||
| self._client_options | ||
| ) | ||
| if span is None and not error_logging_enabled: | ||
| return | ||
|
|
||
| error_attrs = _observability.build_telemetry_error_attributes( | ||
| exc, include_exception=error_logging_enabled | ||
| ) | ||
| status_code = error_attrs.get("rpc.response.status_code", "") | ||
| is_cancelled = status_code in ("CANCELLED", "CancelledError") or isinstance( | ||
| exc, asyncio.CancelledError | ||
| ) | ||
| if is_cancelled: | ||
| error_attrs["rpc.response.status_code"] = "CANCELLED" | ||
|
|
||
| # Enrich active Tier 3 method span | ||
| if span is not None: | ||
| try: | ||
| for k, v in error_attrs.items(): | ||
| if not k.startswith("exception."): | ||
| span.set_attribute(k, v) | ||
| # Intentionally catch broad Exception so telemetry span enrichment | ||
| # fails open and never disrupts the caller's RPC flow. | ||
| except Exception: | ||
| pass | ||
|
|
||
| # Emit L2/L3 Terminal Method Error Log | ||
| if error_logging_enabled: | ||
| self._emit_method_error_log( | ||
| exc, span, error_attrs, is_cancelled, logger=logger | ||
| ) |
There was a problem hiding this comment.
If client_logging.is_error_logging_enabled or _observability.build_telemetry_error_attributes raises an exception, it will propagate out of _report_method_error and disrupt the caller's RPC flow. Wrapping the entire body of _report_method_error in a try...except Exception block ensures that telemetry and logging errors always fail open. To avoid silently masking underlying issues, log the exception as a warning instead of silently passing.
try:
error_logging_enabled = client_logging.is_error_logging_enabled(
self._client_options
)
if span is None and not error_logging_enabled:
return
error_attrs = _observability.build_telemetry_error_attributes(
exc, include_exception=error_logging_enabled
)
status_code = error_attrs.get("rpc.response.status_code", "")
is_cancelled = status_code in ("CANCELLED", "CancelledError") or isinstance(
exc, asyncio.CancelledError
)
if is_cancelled:
error_attrs["rpc.response.status_code"] = "CANCELLED"
# Enrich active Tier 3 method span
if span is not None:
for k, v in error_attrs.items():
if not k.startswith("exception."):
span.set_attribute(k, v)
# Emit L2/L3 Terminal Method Error Log
if error_logging_enabled:
self._emit_method_error_log(
exc, span, error_attrs, is_cancelled, logger=logger
)
except Exception as e:
logger.warning("Failed to report method error: %s", e)References
- Avoid broad
except Exception:blocks that silently returnNone. Instead, log the exception (e.g., usinglogger.warning) to aid in debugging and prevent masking underlying issues.
There was a problem hiding this comment.
Thanks for the review!
- Isolating
span.set_attributein its owntry...except Exception: passblock (separate from_emit_method_error_log'stry...except Exception: passblock) is intentional so that a failing customSpanimplementation cannot skip terminal error log emission (added a clarifying comment in ca5d711). - Keeping
is_error_logging_enabledoutside thetry...exceptblock preserves the fail-loudFeatureGatingErrorcontract if an unsupportedlogger_provideroption is passed. - Using
except Exception: passrather thanlogger.warningmatches the fail-open observability contract across_observability.pyso internal telemetry errors never emit unexpected warnings into user logs.
| if is_cancelled: | ||
| error_attrs.pop("exception.stacktrace", None) | ||
| extra_payload: dict[str, Any] = { | ||
| "rpcName": self._method_name, | ||
| "rpc.system.name": self._rpc_system, | ||
| **error_attrs, | ||
| **_observability.extract_trace_context(span), | ||
| } |
There was a problem hiding this comment.
Mutating the error_attrs dictionary passed as an argument can lead to unexpected side-effects if the caller or other methods reuse it. It is safer to construct extra_payload first and then pop "exception.stacktrace" from extra_payload directly.
| if is_cancelled: | |
| error_attrs.pop("exception.stacktrace", None) | |
| extra_payload: dict[str, Any] = { | |
| "rpcName": self._method_name, | |
| "rpc.system.name": self._rpc_system, | |
| **error_attrs, | |
| **_observability.extract_trace_context(span), | |
| } | |
| extra_payload: dict[str, Any] = { | |
| "rpcName": self._method_name, | |
| "rpc.system.name": self._rpc_system, | |
| **error_attrs, | |
| **_observability.extract_trace_context(span), | |
| } | |
| if is_cancelled: | |
| extra_payload.pop("exception.stacktrace", None) |
There was a problem hiding this comment.
Fixed in ca5d711 — updated _emit_method_error_log to pop "exception.stacktrace" from extra_payload rather than mutating error_attrs in-place.
6389b5c to
ca5d711
Compare
Summary
Adds Tier 3 (method-level) terminal error and cancellation logging in
_GapicCallablewhen experimental Python error logging (GOOGLE_SDK_EXPERIMENTAL_PYTHON_LOGGING_ENABLED) is enabled:_GapicCallable._emit_method_error_logand_GapicCallable._report_method_errorto emit structuredERRORlogs on terminal method failures (includingErrorInfometadata,rpcName,rpc.system.name, trace correlation fields, andexception.stacktrace) andDEBUGlogs without stack traces on RPC or Future/Task cancellation (exceptions.Cancelled,concurrent.futures.CancelledError,asyncio.CancelledError)._GapicCallable._trace_spanto delegate span error enrichment and method error logging toself._report_method_error(exc, span)for both synchronous and asynchronous GAPIC callables while failing open on telemetry or logger errors.tests/unit/gapic/test_method.pyandtests/asyncio/gapic/test_method_async.py.Bug: b/479827845
Verification
nox -s unit-3.14 format lintinpackages/google-api-core.