Repository navigation
feat(server): authenticate postgres with azure workload identity - #5635
Conversation
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-5635.docs.buildwithfern.com/openshell |
|
Reviewed at Busy pools unnecessarily depend on Entra availability — azure_workload_identity.rs:286–295 Previously, an acquisition waited for a checked-out session to return. In Azure mode, when no session is immediately idle, the wrapper refreshes credentials before entering that wait. Near token expiry, an Entra outage therefore causes requests to fail even when healthy database sessions could serve them. This affects both data and advisory-lock pools. Reproduced against disposable PostgreSQL using the PR's acquisition logic and a one-connection pool:
Refresh should gate opening a physical connection while allowing callers to wait for reusable sessions. Please add coverage for an expired token, unavailable Entra, and a busy healthy pool. The overall design covers both pools, shares cached credentials, and preserves verified TLS. CI checks pass; the PR description still lists live expiry/rotation and database-restart validation as incomplete. I did not rerun the full repository suite. |
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Compatibility review (advisory)New or updated breaking changes; see the full report for all findings. 1. api-behavior — No explicit risk acceptance foundSQL I/O timeouts now return UNAVAILABLE with reason DATABASE_UNAVAILABLE instead of INTERNAL, including under default authentication.
Review details
Advisory review; publication gates are unchanged. |
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
|
Fixed in 8a5bef0. Session reuse now progresses independently of credential acquisition. At capacity with an expired cache, callers wait for a returned session without contacting Entra; with spare capacity, a failed or stalled refresh does not preempt a reusable session. New physical connections still require valid credentials, and the normal acquisition deadline remains bounded. Validation:
The natural access-token-expiry/actual projected-token-rotation soak remains outstanding. Temporary test resources are removed, and both updated gateways are Ready. |
|
Label |
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Summary
Allow gateways on AKS to authenticate to Azure PostgreSQL using their projected workload identity instead of a database password. Refresh cached Entra credentials before opening new connections in both PostgreSQL pools while preserving healthy sessions, verified TLS, and the existing default authentication behavior.
Related Issue
Closes #5560
Changes
database_auth = "azure-workload-identity",--db-auth, andOPENSHELL_DB_AUTHwith CLI/environment/TOML precedence.sslmode=verify-fullURLs, non-admin role/schema grants, and operating guidance.Release Notes
AKS deployments can opt into Entra workload identity for external Azure PostgreSQL without storing a database password.
Testing
392 persistence/configuration unit tests, 43 disposable-PostgreSQL tests, strict Clippy, Rust formatting, and documentation validation pass.
Add cache concurrency, assertion rotation, expiry, redaction/failure recovery, URL validation, configuration precedence, and real-PostgreSQL reconnect coverage.
Deploy matching Linux AMD64 feature images through Helm on AKS with two gateways, Entra-only PostgreSQL 16, a non-admin mapped role, and verified TLS 1.3 database sessions.
Run Kubernetes E2E
start_refreshes_stale_bootstrap_image; validate sandbox create/exec/stop/start/delete, credential isolation, and workspace persistence.Terminate all 15 gateway database sessions; verify both data/lock pools reconnect on both replicas with entirely new backend PIDs and no gateway restart or manual credential replacement.
Roll both gateways; verify fresh replicas authenticate and recover the existing sandbox and workspace.
Complete a live access-token-expiry and projected-ServiceAccount-token-rotation soak. The connection-termination test used a still-valid cached token; mocked expiry tests do not replace this soak.
Restart the real Azure PostgreSQL server; both gateways and the sandbox recover with unchanged Pod UIDs/restart counts and a preserved workspace marker. HA failover is not tested on this single-server experiment.
Address the busy-pool review regression: the pre-fix test fails; saturated, spare-capacity/HTTP-503, stalled-issuer, and bounded-wait cases pass afterward. The same test passes as an AKS Job using real initial Entra identity and real verified-TLS Azure PostgreSQL, with cache expiry and issuer failure injected inside the isolated test client.
Redeploy gateway
0.1.9-dev.48+g8a5bef038; rerun Kubernetes runtime reconciliation E2E and sandbox execution/credential-isolation checks.Checklist