Skip to content

fix(api_core): normalize observability log attributes and canonical status codes - #19144

Draft
chalmerlowe wants to merge 1 commit into
feat/otel-logging-m3-s6-http-error-infofrom
feat/otel-logging-m3-s7-grpc-rpc-method
Draft

chalmerlowe wants to merge 1 commit into
feat/otel-logging-m3-s6-http-error-infofrom
feat/otel-logging-m3-s7-grpc-rpc-method

Conversation

@chalmerlowe

@chalmerlowe chalmerlowe commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Context

End-to-end operational error logging verification surfaced six attribute, stacktrace, and status-code normalization issues in google-api-core:

  1. gRPC rpc.method leading slash: _build_attempt_log_extra emitted gRPC rpc.method with a leading slash (for example, "/google.showcase.v1beta1.Echo/Echo") instead of delegating to extract_rpc_identity(), and _make_grpc_client_request_hook did not guard against slash-only targets ("/").
  2. Null httpRequest.status on HTTP timeouts / transport errors: When an HTTP transport attempt failed without receiving a server response (status_code is None), _build_attempt_log_extra emitted "status": null inside httpRequest instead of omitting "status".
  3. Prefixed status.message on Async HTTP L3 error logs: format_http_response_error wraps server error messages with "<METHOD> <URL>: <message>". Sync HTTP recovered the clean server message via response.json(), whereas Async HTTP (aiohttp._Response) lacks a synchronous .json() method and fell back to the URL-prefixed exc.message.
  4. Legacy "rpcName" key on L3 logs: _GapicCallable._emit_method_error_log emitted "rpcName" instead of the standard OpenTelemetry "rpc.method" attribute used by T3 spans, T4 spans, and L4 gRPC logs.
  5. Non-canonical / Python-class rpc.response.status_code:
    • _extract_status_code fell back to exc.__class__.__name__ (emitting "ReadTimeout" / "TimeoutError" / "ConnectionError" as rpc.response.status_code on client-side failures with no server response) instead of returning None so callers omit rpc.response.status_code.
    • HTTP-only GoogleAPICallError classes without grpc_status_code (such as BadRequest / HTTP 400) fell through to http.HTTPStatus(status).name ("BAD_REQUEST" instead of canonical google.rpc.Code "INVALID_ARGUMENT").
  6. Frame-less exception.stacktrace on Async gRPC L4 logs: _AsyncGrpcAttemptLoggingInterceptor._build_async_rpc_error instantiates grpc.aio.AioRpcError snapshots without raising them (exc.__traceback__ is None), causing traceback.format_exception(exc) to emit a single exception summary line with no "Traceback (most recent call last):" header or stack frames.

Bug: b/479827845

Solution

  • packages/google-api-core/google/api_core/exceptions.py:
    • Add canonical AIP-193 HTTP-to-google.rpc.Code mapping (_HTTP_STATUS_TO_CANONICAL_CODE, mapping 400 -> "INVALID_ARGUMENT", 401 -> "UNAUTHENTICATED", 403 -> "PERMISSION_DENIED", 404 -> "NOT_FOUND", 409 -> "ABORTED", 429 -> "RESOURCE_EXHAUSTED", 499 -> "CANCELLED", 500 -> "INTERNAL", 501 -> "UNIMPLEMENTED", 503 -> "UNAVAILABLE", 504 -> "DEADLINE_EXCEEDED").
    • Return None (or explicit default) from _extract_status_code when an exception carries no RPC or HTTP status code instead of falling back to exc.__class__.__name__.
    • Preserve the raw server error message on GoogleAPICallError._http_error_message in format_http_response_error before prepending "<METHOD> <URL>: ".
  • packages/google-api-core/google/api_core/_observability.py:
    • Omit "rpc.response.status_code" in build_telemetry_error_attributes and _build_attempt_log_extra when no canonical RPC/HTTP status code exists (preserving error.type as the exception class name and preserving the human-readable attempt log (status=...) label).
    • Omit "status" from httpRequest in _build_attempt_log_extra when status_code is None.
    • Delegate gRPC rpc.method normalization in _build_attempt_log_extra and _make_grpc_client_request_hook to extract_rpc_identity().
    • Fall back to target_exc._http_error_message in _extract_error_message for async HTTP responses.
    • Synthesize a call-site Python traceback via traceback.format_stack()[:-1] and traceback.format_exception_only(...) in build_telemetry_error_attributes when include_exception=True and exc.__traceback__ is None.
  • packages/google-api-core/google/api_core/gapic_v1/method.py:
    • Emit normalized "rpc.method" (instead of "rpcName") on L3 method error logs and omit "rpc.method" when method_name is unset or slash-only.
    • Detect concurrent.futures.CancelledError explicitly alongside asyncio.CancelledError so cancellation logs continue setting rpc.response.status_code = "CANCELLED".
  • Unit Tests:
    • Updated test_exceptions.py, test_observability.py, and test_method.py with full coverage across all new mappings and boundary cases.

Verification

  • Verified packages/google-api-core locally in Docker with nox -s format lint mypy unit-3.10 unit-3.15 cover (100% line and branch coverage).
  • Relying on GitHub Actions CI for full multi-version verification.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the observability module to use extract_rpc_identity for normalizing RPC method names, preventing empty or slash-only names from being incorrectly recorded. It also adds unit tests to cover various edge cases for method normalization. The review feedback suggests using index-based access extract_rpc_identity(span_name)[0] instead of tuple unpacking to make the code more resilient to future signature changes.

if span_name.startswith("/") and hasattr(span, "update_name"):
span.update_name(clean_method_name)
span.set_attribute("rpc.method", clean_method_name)
clean_method_name, _, _ = extract_rpc_identity(span_name)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using index-based access extract_rpc_identity(span_name)[0] is more robust than strict tuple unpacking. This avoids relying on the exact tuple length of 3, making the code resilient to future signature changes in extract_rpc_identity, and aligns with how it is used in _build_attempt_log_extra on line 544.

Suggested change
clean_method_name, _, _ = extract_rpc_identity(span_name)
clean_method_name = extract_rpc_identity(span_name)[0]

@chalmerlowe chalmerlowe changed the title fix(api_core): normalize gRPC rpc.method via extract_rpc_identity fix(api_core): normalize rpc.method, httpRequest.status, and async HTTP status.message in observability logs Nov 19, 2026
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m3-s7-grpc-rpc-method branch from 591ef2b to facba85 Compare November 19, 2026 20:23
@chalmerlowe chalmerlowe changed the title fix(api_core): normalize rpc.method, httpRequest.status, and async HTTP status.message in observability logs fix(api_core): normalize observability log attributes and canonical status codes Nov 19, 2026
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m3-s7-grpc-rpc-method branch from facba85 to 210b44b Compare November 19, 2026 21:17
…tatus codes

Normalize gRPC/L3 rpc.method via extract_rpc_identity() (replacing L3 rpcName), omit null httpRequest.status and non-canonical client-timeout rpc.response.status_code, map HTTP error codes to canonical google.rpc.Code names, and preserve clean server error messages for async HTTP L3 status.message.

Bug: b/479827845
@chalmerlowe
chalmerlowe force-pushed the feat/otel-logging-m3-s7-grpc-rpc-method branch from 210b44b to b233ccb Compare November 20, 2026 09:41

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant