Can't enable logging of prompt inputs and response outputs with Agent Identity

[gwendal-lecren]

Running into 401 when we enable the logging of prompt inputs and response outputs with Agent Identity.

Libraries:

google-adk==2.4.0
google-cloud-aiplatform[agent_engines,adk]==1.161.0
# OTEL config (cf https://docs.cloud.google.com/stackdriver/docs/instrumentation/ai-agent-adk)
opentelemetry-exporter-gcp-logging==1.12.0a0
opentelemetry-exporter-otlp-proto-grpc==1.42.1
opentelemetry-instrumentation-google-genai==1.0b0
opentelemetry-instrumentation-sqlite3==0.63b1
opentelemetry-instrumentation-vertexai==0.62.1

According to the support:

Why the 401 occurs with gen_ai_latest_experimental

1. When you set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, AdkApp's default telemetry builder is forced to route logs over the network (direct API calls to Cloud Logging) instead of using the standard stdout routing (structured_json_file=sys.stdout). This is a necessary workaround because the runtime currently has parsing issues with the new GenAI semantic conventions when written to stdout.
2. While AdkApp now configures the tracing exporter to use mTLS, the default builder does not apply mTLS configuration to the CloudLoggingExporter when it runs in network mode. Under Agent Identity (which enforces certificate-bound tokens), these unencrypted network calls to logging.googleapis.com are rejected with a 401.

It seems the default telemetry builder from the AdkApp needs to manage mTLS properly.

Environment details

• OS type and version: Agent Runtime
• Python version: 3.14
• google-cloud-aiplatform version: 1.161.0

Steps to reproduce

1. Deploy an agent with agent identity
2. Follow this documentation to setup tracing: https://docs.cloud.google.com/stackdriver/docs/instrumentation/ai-agent-adk and that one: https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/tracing#write-traces

Stack trace

{
  "textPayload": "Error while writing to Cloud Logging\nTraceback (most recent call last):\n  File \"/code/.venv/lib/python3.14/site-packages/google/api_core/grpc_helpers.py\", line 55, in error_remapped_callable\n    return callable_(*args, **kwargs)\n  File \"/code/.venv/lib/python3.14/site-packages/grpc/_interceptor.py\", line 276, in __call__\n    response, ignored_call = self._with_call(\n                             ~~~~~~~~~~~~~~~^\n        request,\n        ^^^^^^^^\n    ...<4 lines>...\n        compression=compression,\n        ^^^^^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File \"/code/.venv/lib/python3.14/site-packages/grpc/_interceptor.py\", line 331, in _with_call\n    return call.result(), call\n           ~~~~~~~~~~~^^\n  File \"/code/.venv/lib/python3.14/site-packages/grpc/_channel.py\", line 438, in result\n    raise self\n  File \"/code/.venv/lib/python3.14/site-packages/grpc/_interceptor.py\", line 314, in continuation\n    response, call = self._thunk(new_method).with_call(\n                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^\n        request,\n        ^^^^^^^^\n    ...<4 lines>...\n        compression=new_compression,\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File \"/code/.venv/lib/python3.14/site-packages/grpc/_channel.py\", line 1173, in with_call\n    return _end_unary_response_blocking(state, call, True, None)\n  File \"/code/.venv/lib/python3.14/site-packages/grpc/_channel.py\", line 990, in _end_unary_response_blocking\n    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ngrpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAUTHENTICATED\n\tdetails = \"Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\"\n\tdebug_error_string = \"UNAUTHENTICATED:Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\"\n>\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/code/.venv/lib/python3.14/site-packages/opentelemetry/exporter/cloud_logging/__init__.py\", line 463, in _write_log_entries_to_client\n    client.write_log_entries(\n    ~~~~~~~~~~~~~~~~~~~~~~~~^\n        WriteLogEntriesRequest(entries=batch, partial_success=True)\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File \"/code/.venv/lib/python3.14/site-packages/google/cloud/logging_v2/services/logging_service_v2/client.py\", line 1027, in write_log_entries\n    response = rpc(\n        request,\n    ...<2 lines>...\n        metadata=metadata,\n    )\n  File \"/code/.venv/lib/python3.14/site-packages/google/api_core/gapic_v1/method.py\", line 128, in __call__\n    return wrapped_func(*args, **kwargs)\n  File \"/code/.venv/lib/python3.14/site-packages/google/api_core/retry/retry_unary.py\", line 294, in retry_wrapped_func\n    return retry_target(\n        target,\n    ...<3 lines>...\n        on_error=on_error,\n    )\n  File \"/code/.venv/lib/python3.14/site-packages/google/api_core/retry/retry_unary.py\", line 156, in retry_target\n    next_sleep = _retry_error_helper(\n        exc,\n    ...<6 lines>...\n        timeout,\n    )\n  File \"/code/.venv/lib/python3.14/site-packages/google/api_core/retry/retry_base.py\", line 216, in _retry_error_helper\n    raise final_exc from source_exc\n  File \"/code/.venv/lib/python3.14/site-packages/google/api_core/retry/retry_unary.py\", line 147, in retry_target\n    result = target()\n  File \"/code/.venv/lib/python3.14/site-packages/google/api_core/timeout.py\", line 130, in func_with_timeout\n    return func(*args, **kwargs)\n  File \"/code/.venv/lib/python3.14/site-packages/google/api_core/grpc_helpers.py\", line 57, in error_remapped_callable\n    raise exceptions.from_grpc_error(exc) from exc\ngoogle.api_core.exceptions.Unauthenticated: 401 Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
  "insertId": "1l5xwnrfb5zh2l",
  "resource": {
    "type": "aiplatform.googleapis.com/ReasoningEngine",
    "labels": {
      "resource_container": "XXX",
      "reasoning_engine_id": "XXX",
      "location": "europe-west1"
    }
  },
  "timestamp": "2026-07-15T13:53:17.483151Z",
  "severity": "ERROR",
  "labels": {
    "python_logger": "root"
  },
  "logName": "projects/XXX/logs/agent-logs",
  "sourceLocation": {
    "file": "/code/.venv/lib/python3.14/site-packages/opentelemetry/exporter/cloud_logging/__init__.py",
    "line": "468",
    "function": "_write_log_entries_to_client"
  },
  "receiveTimestamp": "2026-07-15T13:53:17.523967005Z"
}