Repository navigation
[Health Deid] Replace fixed sleep with retry/backoff in test-resources-post.ps1 - #49625
Closed
Josiah Vinson (jovinson-ms) wants to merge 2 commits into
Closed
Josiah Vinson (jovinson-ms) wants to merge 2 commits into
Josiah Vinson (jovinson-ms) wants to merge 2 commits into
Conversation
…s-post.ps1 Role assignment and network ACL propagation can take minutes after deployment, causing intermittent 403 AuthorizationFailure on upload. Retry container access and each blob upload with exponential backoff (5s initial, 60s cap, 5 min budget) and fail with a clear error. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d9b0092-996f-4265-b2da-1effbf3945c6
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
The retry loop proves the test principal's RBAC and the network rule, but not the DeID managed identity's role assignment. Restore the previous 30s wait as a minimum measured from the network rule update; time spent on retries and uploads counts toward it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d9b0092-996f-4265-b2da-1effbf3945c6
Member
Author
|
Copilot on behalf of Josiah Vinson (@jovinson-ms): combined into #49627. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
sdk/healthdataaiservices/azure-health-deidentification/test-resources-post.ps1opens the storage network ACL, then waits a fixed 30 seconds before uploading the test data intests/data/example_patient_1. The Storage Blob Data Contributor role assignments and the ACL change can take longer than that to take effect, so live pipeline deployments sometimes fail with 403AuthorizationFailure. The storage cmdlets also reported errors without stopping the script, so a failed upload could be missed until tests ran.Fix
Invoke-WithRetryhelper. It retries with exponential backoff (starts at 5s, doubles, caps at 60s) for up to about 5 minutes. If all attempts fail, it throws an error that names the operation and the last error, and points to role assignment or ACL propagation as the likely cause.Start-Sleep -Seconds 30before the upload with a 30-second minimum wait at the end, measured from the network rule update. Time spent on retries and uploads counts toward it. Our own successful access proves the test principal's RBAC and the network rule, but not the DeID service managed identity's role assignment, so the wait still gives that assignment time to propagate.Get-AzStorageContainer(now used as a readiness check) and eachSet-AzStorageBlobContentupload run inside the retry helper with-ErrorAction Stop.Set-AzStorageAccount(the FIXME that opens the ACL) also uses-ErrorAction Stop.Get-ChildItem -File, so only files are uploaded, never directories.Only this test-infrastructure script changes. The Bicep template and the shipped package are unchanged, so there's no CHANGELOG entry.
Validation
main).Start-Sleepmocked. It succeeded after two failures (waits of 5s and 10s), threw the expected error when the time budget ran out, capped waits at 60s, and didn't wait when the first attempt succeeded.Start-Sleepmocked. All 3 files were uploaded in each case. The final wait was 30s when the work was instant, 18s when it took about 12s, and skipped when it took more than 30s. Variables from the upload loop were correctly visible inside the retried block.New-TestResources.ps1using a personal account and the user's own (non-CI) credentials, then removed the resources withRemove-TestResources.ps1:AzureCliCredentialerrors ("Failed to invoke the Azure CLI"). All 5 failed tests passed when re-run. The service's job history showed a job created about 3 minutes after deployment had succeeded, so the DeID managed identity already had storage access.Connect-AzAccountlocally to select the subscription; that step isn't part of this change.Created by Copilot on behalf of Josiah Vinson (Josiah Vinson (@jovinson-ms))