Skip to content

feat: Enable Bound Token for Agentic Identities - #13873

Merged
macastelaz merged 41 commits into
googleapis:agentic-identities-bound-tokenfrom
macastelaz:pr-13169-fixes
Oct 3, 2026
Merged

macastelaz merged 41 commits into
googleapis:agentic-identities-bound-tokenfrom
macastelaz:pr-13169-fixes

Conversation

@macastelaz

@macastelaz macastelaz commented Jul 23, 2026 •

Copy link
Copy Markdown
Contributor

This PR introduces a feature which enables the auth library to acquire bound access-tokens and bound id-tokens in Agentic Environments.

  1. We detect certs in default paths and check if they match the SPIFFE format for agents.

  2. If 1. is a yes then we call the MDS endpoint in a POST request with the certificate in the body.

Note this PR was based on #13169


Manual Testing & End-to-End Verification

We verified this feature end-to-end across both a Live Cloud Run Agent Identity environment (testing against the live Google Metadata Server, Security Token Service, and Vertex AI with the Java Agent Development Kit (com.google.adk:google-adk:1.9.0)) and a 10-Scenario Local Mock MDS Simulation Harness (testing exact HTTP request payloads, true cryptographic certificate/key rotation, combined bundle private-key stripping, well-known directory discovery, non-agent SPIFFE fallback, environment variable precedence, non-atomic rotation retries, and asynchronous container startup polling).

1. Live Cloud Run Agent Identity Verification (<PROJECT_ID>, us-central1)

We deployed a containerized Java test application built against this branch (google-auth-library-oauth2-http:1.50.0-SNAPSHOT @ 27776f6d62a) + Java ADK (com.google.adk:google-adk:1.9.0) to Cloud Run with Agent Identity enabled (--functional-type=agent --identity-type=agent-identity).

Execution A: Default Bound Token Acquisition (GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN unset / default true)

  • Cloud Run Job Executions: agent-bound-token-java-test-cjrgp (latest run with Step 4D) & agent-bound-token-java-test-dd9gm
  • Workload Certificate Discovery:
    • Detected /var/run/secrets/workload-spiffe-credentials/credentials.json
    • Extracted Leaf Certificate SAN URI: spiffe://agents.global.org-<ORG_NUMBER>.system.id.goog/resources/run/projects/<PROJECT_NUMBER>/locations/us-central1/jobs/agent-bound-token-java-test
    • Leaf Certificate SHA-256 (x5t#S256): 5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8
  • Bound Access Token (ComputeEngineCredentials):
    • Successfully acquired bound OAuth2 Access Token via POST to Metadata Server (ya29.d.c0AZ4bNp...).
  • Bound ID Token (IdTokenCredentials) & Cryptographic Binding Verification (RFC 8705 § 3.1):
    • Acquired bound ID Token for target audience https://example-target-service.run.app.
    • Decoded JWT payload inline (textPayload) and verified live Google STS embedded the cnf (Confirmation) claim containing the SHA-256 thumbprint (x5t#S256) of the workload's leaf X.509 certificate:
      "cnf": {
        "x5t#S256": "5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8"
      }
    • Verified an exact byte-for-byte match ([PASS] JWT x5t#S256 thumbprint EXACTLY matches local leaf certificate SHA-256!).
  • Java ADK (com.google.adk:google-adk:1.9.0) + google-genai & mTLS Proof-of-Possession Verification (Steps 4A, 4B, 4C, 4D):
    • Step 4A (ADK HttpClientFactory Non-mTLS Transport): Inspected ADK's shared OkHttpClient (sun.security.ssl.SSLSocketFactoryImpl, no client cert). Calling Google APIs over this non-mTLS channel with the bound token is rejected at the auth layer with HTTP 401 UNAUTHENTICATED.
    • Step 4B (Live ADK LlmAgent + Gemini Turn on Vertex AI): Invoking InMemoryRunner.runAsync(...) with Gemini (gemini-2.5-flash) fails on turn 1 with com.google.genai.errors.ClientException: 401 . Request had invalid authentication credentials, confirming the known limitation where google-genai sends bound tokens over a non-mTLS channel.
    • Step 4C (True mTLS Contrast Test — Matching Workload Certificate): Presenting the exact same bound access token over a true mTLS OkHttpClient (configured with /var/run/secrets/workload-spiffe-credentials/certificates.pem + private_key.pem, x5t#S256 = 5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8) to https://cloudresourcemanager.mtls.googleapis.com/v1/projects/<PROJECT_ID> passes authentication (HTTP 403 PERMISSION_DENIED IAM check instead of 401 UNAUTHENTICATED), proving Google API Frontend verified the token binding against the TLS client certificate handshake.
    • Step 4D (Sender-Constraining / Proof-of-Possession Negative Test — Mismatched Client Certificate): Presenting the exact same bound access token to https://cloudresourcemanager.mtls.googleapis.com/v1/projects/<PROJECT_ID> over an mTLS OkHttpClient configured with a different X.509 client certificate (x5t#S256 = -57ZjYVm89oczfdO02Hf3Sz-FaYQIVH1DD3c9zaBIKA) is rejected at the auth layer with HTTP 401 UNAUTHENTICATED, confirming that Google API Frontend enforces cryptographic thumbprint matching (cnf.x5t#S256 == SHA256(TLS client cert)).

Execution B: Opt-Out Unbound Token Acquisition (GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false)

  • Cloud Run Job Execution: agent-bound-token-java-test-t7b98
  • Updated job environment variable --set-env-vars="GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false,GOOGLE_CLOUD_LOCATION=global,GOOGLE_GENAI_USE_VERTEXAI=true".
  • Verified ComputeEngineCredentials and IdTokenCredentials fell back to standard HTTP GET requests against MDS and issued standard unbound tokens (decoded JWT payload confirmed absence of the cnf claim).
  • Java ADK Live Agent Turn (Step 4B): With GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false, the unbound token works over google-genai's non-mTLS channel and the live ADK LlmAgent + Gemini turn SUCCEEDS:
    [ADK Event] author=bound-token-verify-agent, content=Hello! Yes, as an agent designed to test bound token behavior with the Java ADK, I am expected to receive and process bound tokens.

2. Local End-to-End Simulation & Wire Verification (10 Scenarios)

To verify internal wire-level, discovery, rotation, environment-variable, and error-handling behavior between the client and MDS, we executed our local simulation suite (LocalSimulationRunner.java) spinning up a local Mock MDS (HttpServer) across 10 end-to-end scenarios:

  1. Bound Access Token (ComputeEngineCredentials + GOOGLE_API_CERTIFICATE_CONFIG): Verified HTTP POST to /computeMetadata/v1/instance/service-accounts/default/token?scopes=https://www.googleapis.com/auth/cloud-platform, verified JSON body {"certificate_chain": "-----BEGIN CERTIFICATE-----\n..."} (serialized as a single PEM string), and verified no extra fields are included in the JSON payload.
  2. Bound ID Token (IdTokenCredentials + GOOGLE_API_CERTIFICATE_CONFIG): Verified HTTP POST to /computeMetadata/v1/instance/service-accounts/default/identity?audience=https://target.run.app (with audience passed as a URL query parameter) and JSON body {"certificate_chain": "-----BEGIN CERTIFICATE-----\n..."}.
  3. True Live Cryptographic Certificate & Key Rotation on Disk (Gen-1 $\rightarrow$ Gen-2): Generated a new X.509 SPIFFE certificate and matching 2048-bit RSA key pair on disk, updated file mtime, and verified AgentIdentityUtils.getAgentIdentityCertInfo() invalidated its cache, verified the new key pair, and transmitted the rotated Gen-2 certificate chain (!req3.body.equals(req1.body)).
  4. Opt-Out Override (GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false): Verified setting GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false switches requests back to HTTP GET with an empty request body.
  5. Well-Known Directory Combined Bundle (credentialbundle.pem) + Private Key Stripping: Unset GOOGLE_API_CERTIFICATE_CONFIG, wrote a combined credentialbundle.pem containing both -----BEGIN CERTIFICATE----- and -----BEGIN PRIVATE KEY----- in the well-known directory, and verified that AgentIdentityUtils resolved certPath == keyPath, verified the key pair, and stripped the PRIVATE KEY block from the transmitted POST payload.
  6. Well-Known Directory Separate Files (certificates.pem + private_key.pem): Unset GOOGLE_API_CERTIFICATE_CONFIG, removed credentialbundle.pem, placed separate certificates.pem and private_key.pem in the well-known directory, and verified bound HTTP POST acquisition.
  7. Non-Agent SPIFFE Trust Domain Fallback (shouldRequestBoundToken == false): Configured a valid X.509 certificate with a standard GKE Workload Identity SAN (spiffe://my-standard-gke-project.svc.id.goog/ns/default/sa/my-ksa); verified AgentIdentityUtils did not throw, cached shouldRequestBoundToken = false, and fell back to standard HTTP GET.
  8. Environment Variable Precedence & GOOGLE_API_USE_CLIENT_CERTIFICATE Matrix:
    • 8a: Legacy GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false (with GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN unset) $\rightarrow$ falls back to HTTP GET.
    • 8b: Modern GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=true overrides legacy GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false $\rightarrow$ sends bound HTTP POST.
    • 8c: GOOGLE_API_USE_CLIENT_CERTIFICATE=false with valid agent certs on disk $\rightarrow$ disables bound token (HTTP GET) without startup polling.
    • 8d: GOOGLE_API_USE_CLIENT_CERTIFICATE=true with missing cert files $\rightarrow$ fails closed with IOException after retries instead of silently downgrading to an unbound token.
  9. Non-Atomic Rotation / Transient Key-Pair Mismatch Recovery (CERT_KEY_MATCH_RETRIES): Wrote a new Gen-3 non-prod SPIFFE certificate (spiffe://agents-nonprod.global.org-54321.system.id.goog/...) first while delaying the matching private_key.pem update on a background thread; verified loadAndVerifyCredentials() retried cleanly via CERT_KEY_MATCH_RETRIES and transmitted Gen-3.
  10. Asynchronous Container Startup Credential Delivery Polling (TOTAL_POLL_CYCLES): Started with an empty well-known directory on initial startup (GOOGLE_API_USE_CLIENT_CERTIFICATE=true), delivered certificates.pem + private_key.pem asynchronously from a background thread after ~180ms, and verified the initial refreshAccessToken() polled until the files arrived and succeeded with a bound HTTP POST.

3. Reproducible Test Artifacts & Execution Logs (gpaste - Internal Corp Access Only)

Artifact / Log Description Link
Combined Verification Summary & Logs Complete report & console outputs from Live Cloud Run (Bound cjrgp & Opt-Out t7b98) + Java ADK + Local 10-Scenario Simulation https://paste.googleplex.com/4671448266964992
CloudRunAgentVerifyApp.java Live Cloud Run + Java ADK verification app (validates SPIFFE cert, ADC access token, ID token cnf.x5t#S256 match, and ADK Steps 4A/4B/4C/4D) https://paste.googleplex.com/5431546950057984
LocalSimulationRunner.java Self-contained local E2E test harness with Mock MDS (HttpServer) testing all 10 scenarios https://paste.googleplex.com/4541159570014208
Dockerfile Multi-stage Docker build resolving Java ADK 1.9.0 + google-genai and overlaying our local PR JAR https://paste.googleplex.com/6569302979903488
deploy_cloud_run_job.sh Turnkey script to build via Cloud Build, deploy Cloud Run Job with --identity-type=agent-identity, and execute https://paste.googleplex.com/4908078735163392
build_and_run_local.sh Turnkey script to compile and run the local simulation https://paste.googleplex.com/6449866457350144
Raw Cloud Logging — Bound Run (cjrgp) Plain-text Cloud Run logs from gcloud logging read for default Bound Token execution (with inline decoded JWT & ADK Steps 4A/4B/4C/4D) https://paste.googleplex.com/5404054864396288
Raw Cloud Logging — Opt-Out Run (t7b98) Plain-text Cloud Run logs from gcloud logging read for GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false execution (showing live ADK Gemini response) https://paste.googleplex.com/5990735250325504

Quick Reproduction Steps

To replicate the live Cloud Run test in any GCP project with Agent Identity enabled:

# 1. Place CloudRunAgentVerifyApp.java, LocalSimulationRunner.java, Dockerfile, and deploy_cloud_run_job.sh
#    in a directory alongside your local google-cloud-java checkout.
chmod +x deploy_cloud_run_job.sh build_and_run_local.sh

# 2. Run the local 10-scenario simulation suite:
./build_and_run_local.sh

# 3. Deploy and execute the Cloud Run Job with Agent Identity enabled (Default Bound Token mode):
./deploy_cloud_run_job.sh <PROJECT_ID> us-central1

# 4. To test opt-out behavior (unblocks ADK non-mTLS Gemini calls):
gcloud alpha run jobs update agent-bound-token-java-test \
  --project=<PROJECT_ID> --region=us-central1 \
  --set-env-vars="GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false,GOOGLE_CLOUD_LOCATION=global,GOOGLE_GENAI_USE_VERTEXAI=true"
gcloud alpha run jobs execute agent-bound-token-java-test \
  --project=<PROJECT_ID> --region=us-central1 --wait

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants