Skip to content

feat(server): authenticate postgres with azure workload identity - #5635

Merged
TaylorMutch merged 5 commits into
mainfrom
feat/5560-azure-postgres-workload-identity/TaylorMutch
Nov 17, 2026
Merged

TaylorMutch merged 5 commits into
mainfrom
feat/5560-azure-postgres-workload-identity/TaylorMutch

Conversation

@TaylorMutch

@TaylorMutch TaylorMutch commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Allow gateways on AKS to authenticate to Azure PostgreSQL using their projected workload identity instead of a database password. Refresh cached Entra credentials before opening new connections in both PostgreSQL pools while preserving healthy sessions, verified TLS, and the existing default authentication behavior.

Related Issue

Closes #5560

Changes

  • Add database_auth = "azure-workload-identity", --db-auth, and OPENSHELL_DB_AUTH with CLI/environment/TOML precedence.
  • Exchange the explicitly configured projected identity for PostgreSQL-scoped tokens, cache by reported expiry, and reread the projected file on refresh. Fail with bounded, redacted diagnostics and no alternate credential chain.
  • Guard data and advisory-lock pool acquisitions; disable Azure-mode background prefill and lifetime-based session rotation so physical connections cannot bypass credential refresh.
  • Document gateway-only federation, password-free sslmode=verify-full URLs, non-admin role/schema grants, and operating guidance.

Release Notes

AKS deployments can opt into Entra workload identity for external Azure PostgreSQL without storing a database password.

Testing

  • 392 persistence/configuration unit tests, 43 disposable-PostgreSQL tests, strict Clippy, Rust formatting, and documentation validation pass.

  • Add cache concurrency, assertion rotation, expiry, redaction/failure recovery, URL validation, configuration precedence, and real-PostgreSQL reconnect coverage.

  • Deploy matching Linux AMD64 feature images through Helm on AKS with two gateways, Entra-only PostgreSQL 16, a non-admin mapped role, and verified TLS 1.3 database sessions.

  • Run Kubernetes E2E start_refreshes_stale_bootstrap_image; validate sandbox create/exec/stop/start/delete, credential isolation, and workspace persistence.

  • Terminate all 15 gateway database sessions; verify both data/lock pools reconnect on both replicas with entirely new backend PIDs and no gateway restart or manual credential replacement.

  • Roll both gateways; verify fresh replicas authenticate and recover the existing sandbox and workspace.

  • Complete a live access-token-expiry and projected-ServiceAccount-token-rotation soak. The connection-termination test used a still-valid cached token; mocked expiry tests do not replace this soak.

  • Restart the real Azure PostgreSQL server; both gateways and the sandbox recover with unchanged Pod UIDs/restart counts and a preserved workspace marker. HA failover is not tested on this single-server experiment.

  • Address the busy-pool review regression: the pre-fix test fails; saturated, spare-capacity/HTTP-503, stalled-issuer, and bounded-wait cases pass afterward. The same test passes as an AKS Job using real initial Entra identity and real verified-TLS Azure PostgreSQL, with cache expiry and issuer failure injected inside the isolated test client.

  • Redeploy gateway 0.1.9-dev.48+g8a5bef038; rerun Kubernetes runtime reconciliation E2E and sandbox execution/credential-isolation checks.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Configuration docs and related operating skill updated

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

drew commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Reviewed at 99164a7f3eddd464db6a2fd3eb1a99a0268be231. Found one correctness issue:

Busy pools unnecessarily depend on Entra availability — azure_workload_identity.rs:286–295

Previously, an acquisition waited for a checked-out session to return. In Azure mode, when no session is immediately idle, the wrapper refreshes credentials before entering that wait. Near token expiry, an Entra outage therefore causes requests to fail even when healthy database sessions could serve them. This affects both data and advisory-lock pools.

Reproduced against disposable PostgreSQL using the PR's acquisition logic and a one-connection pool:

  1. Acquire the only connection with a valid cached token.
  2. Expire the cached token and make the token endpoint unavailable.
  3. Start a second acquisition while holding the healthy connection.
  4. The second acquisition fails on token exchange before the connection returns; the held connection still passes its health check.
  5. Return the held connection. The next acquisition succeeds without contacting Entra.

Refresh should gate opening a physical connection while allowing callers to wait for reusable sessions. Please add coverage for an expired token, unavailable Entra, and a busy healthy pool.

The overall design covers both pools, shares cached credentials, and preserves verified TLS. CI checks pass; the PR description still lists live expiry/rotation and database-restart validation as incomplete. I did not rerun the full repository suite.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

SQL I/O timeouts now return UNAVAILABLE with reason DATABASE_UNAVAILABLE instead of INTERNAL, including under default authentication.

  • Impact: Existing SDK/RPC handlers can no longer distinguish in-flight SQL timeouts from the documented connection-acquisition failure using the status and reason.
  • Action: For v0.1.9, scope the new unavailable classification to Azure acquisition failures and preserve the existing status mapping for in-flight SQL I/O timeouts.
  • PR: feat(server): authenticate postgres with azure workload identity #5635
Review details
  • Execution: complete
  • Candidate: PR #5635 (b984a62cf581c3d64ee34dceafbe887497ae0c12)
  • Baseline: e7d2dc722aec74e1bfb3f58f9444a9eaa9d34a0c (e7d2dc722aec74e1bfb3f58f9444a9eaa9d34a0c)
  • Reviewed: rust-sdk, python-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch

Copy link
Copy Markdown
Collaborator Author

Fixed in 8a5bef0. Session reuse now progresses independently of credential acquisition. At capacity with an expired cache, callers wait for a returned session without contacting Entra; with spare capacity, a failed or stalled refresh does not preempt a reusable session. New physical connections still require valid credentials, and the normal acquisition deadline remains bounded.

Validation:

  • Reproduced the reported failure before the fix. The regression now covers a saturated pool, spare capacity with HTTP 503, a stalled issuer, and deadline failure while preserving the held healthy session.
  • All 43 PostgreSQL tests, 392 persistence/configuration tests, strict Clippy, formatting, and docs checks pass.
  • The same regression passes in an AKS Job against real Azure PostgreSQL with verified TLS and real initial gateway workload-identity authentication. Cache expiry and issuer failure are injected only inside that test client; returned connections retain their backend PID.
  • Deployed gateway 0.1.9-dev.48+g8a5bef038, reran Kubernetes runtime reconciliation E2E, and restarted the actual Azure PostgreSQL server. Both gateway Pod UIDs/restart counts and the sandbox Pod UID remained unchanged; authenticated operations recovered and the workspace marker survived.

The natural access-token-expiry/actual projected-token-rotation soak remains outstanding. Temporary test resources are removed, and both updated gateways are Ready.

drew
drew previously approved these changes Nov 17, 2026
@TaylorMutch TaylorMutch added the test:e2e Requires end-to-end coverage label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for f8a5a2a. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch
TaylorMutch added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit d66d338 Nov 17, 2026
160 of 161 checks passed
@TaylorMutch
TaylorMutch deleted the feat/5560-azure-postgres-workload-identity/TaylorMutch branch November 17, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Azure PostgreSQL Entra workload identity authentication with automatic token renewal

2 participants