fix: route MCP calls to the .mtls.googleapis.com endpoint for Agent Identity

[ShresthSamyak]

Summary

MCP tools obtained via AgentRegistry.get_mcp_toolset(...) fail with
401 UNAUTHENTICATED when the app uses Agent Identity, unless
GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=False is set (which
works around it by unbinding the token). This routes Google-hosted MCP servers
to the mutual-TLS endpoint so the channel-bound token is accepted.

Related to #6365 and follow-up to #6370 (which fixes the same class of bug on
the A2A path).

Root cause

MCPSessionManager already negotiates an mTLS transport for Google MCP
endpoints (the Successfully configured mTLS using AsyncAuthorizedSession log
confirms the client certificate and channel-bound token are set up). However,
get_mcp_toolset builds StreamableHTTPConnectionParams(url=endpoint_uri) with
the endpoint exactly as registered — a standard *.googleapis.com host. A
channel-bound (Agent Identity) token is only honored on the corresponding
*.mtls.googleapis.com endpoint, so presenting it to the standard endpoint
still returns 401 even though mTLS was negotiated.

This is the same issue fixed for A2A in #6370, and mirrors the endpoint
resolution added for Application Integration tools in 37ca6fb.

Change

agent_registry.py — in get_mcp_toolset, when a client certificate is
configured (_use_client_cert_effective()), resolve the MCP endpoint through
the existing _mtls_utils.effective_googleapis_endpoint helper so it uses the
*.mtls.googleapis.com variant. It is a no-op for non-Google hosts,
already-mTLS hosts, and when GOOGLE_API_USE_MTLS_ENDPOINT=never, so Service
Account and non-Google flows are unaffected.

Testing plan

Unit tests (pytest), all passing:

• test_get_mcp_toolset_uses_mtls_endpoint_for_google — the connection targets
  the .mtls.googleapis.com endpoint when a client cert is configured.
• test_get_mcp_toolset_keeps_plain_endpoint_without_client_cert — the plain
  endpoint is kept when no client cert is configured.
• test_get_mcp_toolset_keeps_non_google_endpoint — non-Google endpoints are
  left unchanged.

Live validation against a deployed Agent Runtime + Agent Identity setup with a
registered BigQuery MCP server is welcome; the reporter in #6370 observed the
401 on this exact path.

[gnanirahulnutakki]

I found one endpoint-selection edge case in the new routing. The focused Agent Registry and mTLS utility suites pass locally (80 tests); the current red unit-test matrix is failing on unrelated google-antigravity/protobuf version skew. Details and a suggested policy matrix are inline.

[gnanirahulnutakki]

Endpoint selection needs the resolved certificate/transport state, not only the cert-use policy. _use_client_cert_effective() says whether certificate use is enabled; it does not prove that a certificate is available. I reproduced this with GOOGLE_API_USE_CLIENT_CERTIFICATE=true, GOOGLE_API_USE_MTLS_ENDPOINT=auto, and mtls.has_default_client_cert_source() == false: AgentRegistry._base_url correctly remains https://agentregistry.googleapis.com/v1, but the MCP connection becomes https://us-central1-aiplatform.mtls.googleapis.com/mcp. MCPSessionManager then falls back to its plain client when mTLS configuration returns no transport, leaving a non-mTLS client pointed at the mTLS host.

The opposite edge also misses the documented policy: with GOOGLE_API_USE_MTLS_ENDPOINT=always and client-certificate use disabled, this gate never calls the resolver and leaves the regular host. AIP-4114 defines auto as mTLS only when a certificate is available, while always selects the mTLS endpoint independently.

Could this preserve the full matrix—always rewrite, auto rewrite only after mTLS actually negotiates, and never retain the original URL—and add regression cases for auto/no-cert and always/cert-disabled? The A2A follow-up in #6370 already keeps its original endpoint when transport negotiation returns None, which looks like the useful precedent here.

[gnanirahulnutakki]

Verified on c6d04107: the MCP path now shares the base-URL policy, auto is gated on the resolved certificate source, and the regression matrix covers auto with/without a cert plus always, never, and non-Google endpoints. The full focused Agent Registry suite passes locally (52 tests). This addresses the original finding—thank you for the thorough update.

[ShresthSamyak]

Thanks for the careful review, @gnanirahulnutakki — you're right on both counts, and the base-URL policy is the correct reference. I gated on _use_client_cert_effective() (the cert-use policy) instead of cert availability, which mishandled exactly the two cases you called out.

I've pushed a fix that routes the MCP endpoint through the same policy the registry base URL already uses:

• Extracted the base-URL decision into _should_use_mtls_endpoint(client_cert_source) and made _get_agent_registry_base_url delegate to it (so the two can't drift).
• AgentRegistry.__init__ now retains the resolved client_cert_source, and get_mcp_toolset gates the endpoint rewrite on _should_use_mtls_endpoint(self._client_cert_source).

That gives the full matrix:

• always → mTLS endpoint unconditionally (your cert-disabled case now rewrites);
• auto → mTLS endpoint only when a certificate is actually available (client_cert_source is not None), so auto + no cert now correctly stays on the standard host instead of pointing a plain client (after the mTLS fallback) at the mTLS host;
• never → original URL.

Added regression tests for all four Google cases (auto+cert, auto+no-cert, always+no-cert, never+cert) plus the non-Google no-op.

On the A2A precedent — agreed it's the right model; #6370 keys off actual transport negotiation returning None. Here I keyed off the resolved client_cert_source so MCP endpoint selection stays consistent with _get_agent_registry_base_url in the same class, which matches your reproduction (has_default_client_cert_source() == false → no rewrite). Happy to align them further if you'd prefer.

(The red unit-test matrix is unrelated — the integrations/antigravity_agent sample is failing on a google-antigravity Protobuf gencode/runtime version skew on current main; the Agent Registry and mTLS suites pass.)

[gnanirahulnutakki]

The updated endpoint policy now matches the full always / auto / never matrix. I checked the author delta at c6d04107, ran the complete focused Agent Registry suite (52 passed), and verified Ruff, isort, and diff checks. One non-blocking pyink formatting nit remains inline; I found no functional concerns.

[gnanirahulnutakki]

Non-blocking formatter nit: the repository-pinned pyink==25.12 reports this signature and rewrites it onto one line (def test_get_mcp_toolset_never_keeps_plain_endpoint_with_cert(self, registry):). The rest of the focused style checks pass.

[ShresthSamyak]

Thanks @gnanirahulnutakki — and glad the endpoint policy matrix looks right now. Good catch on the formatting nit; I ran the pinned pyink==25.12 and it collapsed that signature onto one line as you noted. Pushed in 2f0043e (pyink and isort now clean on the file).

[gnanirahulnutakki]

Rechecked the formatter follow-up at 2f0043ee. It is exactly the pyink normalization noted in the prior review; the two changed files now pass Pyink, isort, Ruff, and diff checks, and the full focused Agent Registry suite remains green (52 passed). No remaining issues from my review—thank you for the quick update.

[HonzaKopecky]

@ShresthSamyak I can confirm the fix works for both Google BigQuery and Google Drive MCP servers! Just tested it directly in Agent Runtime. Thanks a lot for all the work you put in!

[ShresthSamyak]

That's great to hear, @HonzaKopecky — thank you for testing it directly in Agent Runtime, and for confirming it across both the BigQuery and Google Drive MCP servers. Really appreciate you following through on the verification; it's good to have it validated on real deployments without the token-sharing workaround.