Skip to content

feat(api_core): add method-level terminal error logging in _GapicCallable - #19012

Draft
chalmerlowe wants to merge 1 commit into
feat/otel-logging-m3-s4-http-attempt-loggingfrom
feat/otel-logging-m3-s5-method-error-logging
Draft

chalmerlowe wants to merge 1 commit into
feat/otel-logging-m3-s4-http-attempt-loggingfrom
feat/otel-logging-m3-s5-method-error-logging

Conversation

@chalmerlowe

Copy link
Copy Markdown
Contributor

Summary

Adds Tier 3 (method-level) terminal error and cancellation logging in _GapicCallable when experimental Python error logging (GOOGLE_SDK_EXPERIMENTAL_PYTHON_LOGGING_ENABLED) is enabled:

  • Adds _GapicCallable._emit_method_error_log and _GapicCallable._report_method_error to emit structured ERROR logs on terminal method failures (including ErrorInfo metadata, rpcName, rpc.system.name, trace correlation fields, and exception.stacktrace) and DEBUG logs without stack traces on RPC or Future/Task cancellation (exceptions.Cancelled, concurrent.futures.CancelledError, asyncio.CancelledError).
  • Updates _GapicCallable._trace_span to delegate span error enrichment and method error logging to self._report_method_error(exc, span) for both synchronous and asynchronous GAPIC callables while failing open on telemetry or logger errors.
  • Adds unit tests in tests/unit/gapic/test_method.py and tests/asyncio/gapic/test_method_async.py.

Bug: b/479827845

Verification

  • Verified locally with nox -s unit-3.14 format lint in packages/google-api-core.
  • Relying on GitHub Actions CI for full multi-version and cross-environment verification.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces structured error and cancellation logging for GAPIC method calls in both synchronous and asynchronous contexts. It adds _emit_method_error_log and _report_method_error to enrich OpenTelemetry spans and log terminal RPC failures or cancellations, along with comprehensive tests. The review feedback suggests wrapping _report_method_error in a try...except block to ensure telemetry errors fail open without disrupting the main RPC flow, and logging any exceptions as warnings. Additionally, it recommends avoiding the mutation of the passed error_attrs dictionary in _emit_method_error_log to prevent unintended side-effects.

Comment on lines +333 to +364
error_logging_enabled = client_logging.is_error_logging_enabled(
self._client_options
)
if span is None and not error_logging_enabled:
return

error_attrs = _observability.build_telemetry_error_attributes(
exc, include_exception=error_logging_enabled
)
status_code = error_attrs.get("rpc.response.status_code", "")
is_cancelled = status_code in ("CANCELLED", "CancelledError") or isinstance(
exc, asyncio.CancelledError
)
if is_cancelled:
error_attrs["rpc.response.status_code"] = "CANCELLED"

# Enrich active Tier 3 method span
if span is not None:
try:
for k, v in error_attrs.items():
if not k.startswith("exception."):
span.set_attribute(k, v)
# Intentionally catch broad Exception so telemetry span enrichment
# fails open and never disrupts the caller's RPC flow.
except Exception:
pass

# Emit L2/L3 Terminal Method Error Log
if error_logging_enabled:
self._emit_method_error_log(
exc, span, error_attrs, is_cancelled, logger=logger
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If client_logging.is_error_logging_enabled or _observability.build_telemetry_error_attributes raises an exception, it will propagate out of _report_method_error and disrupt the caller's RPC flow. Wrapping the entire body of _report_method_error in a try...except Exception block ensures that telemetry and logging errors always fail open. To avoid silently masking underlying issues, log the exception as a warning instead of silently passing.

        try:
            error_logging_enabled = client_logging.is_error_logging_enabled(
                self._client_options
            )
            if span is None and not error_logging_enabled:
                return

            error_attrs = _observability.build_telemetry_error_attributes(
                exc, include_exception=error_logging_enabled
            )
            status_code = error_attrs.get("rpc.response.status_code", "")
            is_cancelled = status_code in ("CANCELLED", "CancelledError") or isinstance(
                exc, asyncio.CancelledError
            )
            if is_cancelled:
                error_attrs["rpc.response.status_code"] = "CANCELLED"

            # Enrich active Tier 3 method span
            if span is not None:
                for k, v in error_attrs.items():
                    if not k.startswith("exception."):
                        span.set_attribute(k, v)

            # Emit L2/L3 Terminal Method Error Log
            if error_logging_enabled:
                self._emit_method_error_log(
                    exc, span, error_attrs, is_cancelled, logger=logger
                )
        except Exception as e:
            logger.warning("Failed to report method error: %s", e)
References
  1. Avoid broad except Exception: blocks that silently return None. Instead, log the exception (e.g., using logger.warning) to aid in debugging and prevent masking underlying issues.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review!

  1. Isolating span.set_attribute in its own try...except Exception: pass block (separate from _emit_method_error_log's try...except Exception: pass block) is intentional so that a failing custom Span implementation cannot skip terminal error log emission (added a clarifying comment in ca5d711).
  2. Keeping is_error_logging_enabled outside the try...except block preserves the fail-loud FeatureGatingError contract if an unsupported logger_provider option is passed.
  3. Using except Exception: pass rather than logger.warning matches the fail-open observability contract across _observability.py so internal telemetry errors never emit unexpected warnings into user logs.

Comment on lines +287 to +294
if is_cancelled:
error_attrs.pop("exception.stacktrace", None)
extra_payload: dict[str, Any] = {
"rpcName": self._method_name,
"rpc.system.name": self._rpc_system,
**error_attrs,
**_observability.extract_trace_context(span),
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Mutating the error_attrs dictionary passed as an argument can lead to unexpected side-effects if the caller or other methods reuse it. It is safer to construct extra_payload first and then pop "exception.stacktrace" from extra_payload directly.

Suggested change
if is_cancelled:
error_attrs.pop("exception.stacktrace", None)
extra_payload: dict[str, Any] = {
"rpcName": self._method_name,
"rpc.system.name": self._rpc_system,
**error_attrs,
**_observability.extract_trace_context(span),
}
extra_payload: dict[str, Any] = {
"rpcName": self._method_name,
"rpc.system.name": self._rpc_system,
**error_attrs,
**_observability.extract_trace_context(span),
}
if is_cancelled:
extra_payload.pop("exception.stacktrace", None)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ca5d711 — updated _emit_method_error_log to pop "exception.stacktrace" from extra_payload rather than mutating error_attrs in-place.

@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m3-s5-method-error-logging branch from 6389b5c to ca5d711 Compare November 6, 2026 16:51

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant