Repository navigation
fix(api_core): normalize observability log attributes and canonical status codes - #19144
chalmerlowe wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the observability module to use extract_rpc_identity for normalizing RPC method names, preventing empty or slash-only names from being incorrectly recorded. It also adds unit tests to cover various edge cases for method normalization. The review feedback suggests using index-based access extract_rpc_identity(span_name)[0] instead of tuple unpacking to make the code more resilient to future signature changes.
| if span_name.startswith("/") and hasattr(span, "update_name"): | ||
| span.update_name(clean_method_name) | ||
| span.set_attribute("rpc.method", clean_method_name) | ||
| clean_method_name, _, _ = extract_rpc_identity(span_name) |
There was a problem hiding this comment.
Using index-based access extract_rpc_identity(span_name)[0] is more robust than strict tuple unpacking. This avoids relying on the exact tuple length of 3, making the code resilient to future signature changes in extract_rpc_identity, and aligns with how it is used in _build_attempt_log_extra on line 544.
| clean_method_name, _, _ = extract_rpc_identity(span_name) | |
| clean_method_name = extract_rpc_identity(span_name)[0] |
591ef2b to
facba85
Compare
facba85 to
210b44b
Compare
…tatus codes Normalize gRPC/L3 rpc.method via extract_rpc_identity() (replacing L3 rpcName), omit null httpRequest.status and non-canonical client-timeout rpc.response.status_code, map HTTP error codes to canonical google.rpc.Code names, and preserve clean server error messages for async HTTP L3 status.message. Bug: b/479827845
210b44b to
b233ccb
Compare
Problem / Context
End-to-end operational error logging verification surfaced six attribute, stacktrace, and status-code normalization issues in
google-api-core:rpc.methodleading slash:_build_attempt_log_extraemitted gRPCrpc.methodwith a leading slash (for example,"/google.showcase.v1beta1.Echo/Echo") instead of delegating toextract_rpc_identity(), and_make_grpc_client_request_hookdid not guard against slash-only targets ("/").httpRequest.statuson HTTP timeouts / transport errors: When an HTTP transport attempt failed without receiving a server response (status_code is None),_build_attempt_log_extraemitted"status": nullinsidehttpRequestinstead of omitting"status".status.messageon Async HTTP L3 error logs:format_http_response_errorwraps server error messages with"<METHOD> <URL>: <message>". Sync HTTP recovered the clean server message viaresponse.json(), whereas Async HTTP (aiohttp._Response) lacks a synchronous.json()method and fell back to the URL-prefixedexc.message."rpcName"key on L3 logs:_GapicCallable._emit_method_error_logemitted"rpcName"instead of the standard OpenTelemetry"rpc.method"attribute used byT3spans,T4spans, andL4gRPC logs.rpc.response.status_code:_extract_status_codefell back toexc.__class__.__name__(emitting"ReadTimeout"/"TimeoutError"/"ConnectionError"asrpc.response.status_codeon client-side failures with no server response) instead of returningNoneso callers omitrpc.response.status_code.GoogleAPICallErrorclasses withoutgrpc_status_code(such asBadRequest/ HTTP400) fell through tohttp.HTTPStatus(status).name("BAD_REQUEST"instead of canonicalgoogle.rpc.Code"INVALID_ARGUMENT").exception.stacktraceon Async gRPC L4 logs:_AsyncGrpcAttemptLoggingInterceptor._build_async_rpc_errorinstantiatesgrpc.aio.AioRpcErrorsnapshots without raising them (exc.__traceback__ is None), causingtraceback.format_exception(exc)to emit a single exception summary line with no"Traceback (most recent call last):"header or stack frames.Bug: b/479827845
Solution
packages/google-api-core/google/api_core/exceptions.py:google.rpc.Codemapping (_HTTP_STATUS_TO_CANONICAL_CODE, mapping400 -> "INVALID_ARGUMENT",401 -> "UNAUTHENTICATED",403 -> "PERMISSION_DENIED",404 -> "NOT_FOUND",409 -> "ABORTED",429 -> "RESOURCE_EXHAUSTED",499 -> "CANCELLED",500 -> "INTERNAL",501 -> "UNIMPLEMENTED",503 -> "UNAVAILABLE",504 -> "DEADLINE_EXCEEDED").None(or explicitdefault) from_extract_status_codewhen an exception carries no RPC or HTTP status code instead of falling back toexc.__class__.__name__.GoogleAPICallError._http_error_messageinformat_http_response_errorbefore prepending"<METHOD> <URL>: ".packages/google-api-core/google/api_core/_observability.py:"rpc.response.status_code"inbuild_telemetry_error_attributesand_build_attempt_log_extrawhen no canonical RPC/HTTP status code exists (preservingerror.typeas the exception class name and preserving the human-readable attempt log(status=...)label)."status"fromhttpRequestin_build_attempt_log_extrawhenstatus_code is None.rpc.methodnormalization in_build_attempt_log_extraand_make_grpc_client_request_hooktoextract_rpc_identity().target_exc._http_error_messagein_extract_error_messagefor async HTTP responses.traceback.format_stack()[:-1]andtraceback.format_exception_only(...)inbuild_telemetry_error_attributeswheninclude_exception=Trueandexc.__traceback__ is None.packages/google-api-core/google/api_core/gapic_v1/method.py:"rpc.method"(instead of"rpcName") on L3 method error logs and omit"rpc.method"whenmethod_nameis unset or slash-only.concurrent.futures.CancelledErrorexplicitly alongsideasyncio.CancelledErrorso cancellation logs continue settingrpc.response.status_code = "CANCELLED".test_exceptions.py,test_observability.py, andtest_method.pywith full coverage across all new mappings and boundary cases.Verification
packages/google-api-corelocally in Docker withnox -s format lint mypy unit-3.10 unit-3.15 cover(100%line and branch coverage).