Skip to content

[Health Deid] Replace fixed sleep with retry/backoff in test-resources-post.ps1 - #49625

Closed
Josiah Vinson (jovinson-ms) wants to merge 2 commits into
Azure:mainfrom
jovinson-ms:jovinson-ms-python-test-resources-readiness-probe
Closed

Josiah Vinson (jovinson-ms) wants to merge 2 commits into
Azure:mainfrom
jovinson-ms:jovinson-ms-python-test-resources-readiness-probe

Conversation

@jovinson-ms

@jovinson-ms Josiah Vinson (jovinson-ms) commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Problem

sdk/healthdataaiservices/azure-health-deidentification/test-resources-post.ps1 opens the storage network ACL, then waits a fixed 30 seconds before uploading the test data in tests/data/example_patient_1. The Storage Blob Data Contributor role assignments and the ACL change can take longer than that to take effect, so live pipeline deployments sometimes fail with 403 AuthorizationFailure. The storage cmdlets also reported errors without stopping the script, so a failed upload could be missed until tests ran.

Fix

  • Added an Invoke-WithRetry helper. It retries with exponential backoff (starts at 5s, doubles, caps at 60s) for up to about 5 minutes. If all attempts fail, it throws an error that names the operation and the last error, and points to role assignment or ACL propagation as the likely cause.
  • Replaced the fixed Start-Sleep -Seconds 30 before the upload with a 30-second minimum wait at the end, measured from the network rule update. Time spent on retries and uploads counts toward it. Our own successful access proves the test principal's RBAC and the network rule, but not the DeID service managed identity's role assignment, so the wait still gives that assignment time to propagate.
  • Get-AzStorageContainer (now used as a readiness check) and each Set-AzStorageBlobContent upload run inside the retry helper with -ErrorAction Stop. Set-AzStorageAccount (the FIXME that opens the ACL) also uses -ErrorAction Stop.
  • The upload now uses Get-ChildItem -File, so only files are uploaded, never directories.

Only this test-infrastructure script changes. The Bicep template and the shipped package are unchanged, so there's no CHANGELOG entry.

Validation

  • The script parses with no PowerShell AST errors (checked after rebasing on current main).
  • Ran the helper on its own with Start-Sleep mocked. It succeeded after two failures (waits of 5s and 10s), threw the expected error when the time budget ran out, capped waits at 60s, and didn't wait when the first attempt succeeded.
  • Ran the whole post script with the Az cmdlets and Start-Sleep mocked. All 3 files were uploaded in each case. The final wait was 30s when the work was instant, 18s when it took about 12s, and skipped when it took more than 30s. Variables from the upload loop were correctly visible inside the retried block.
  • Deployed live once with New-TestResources.ps1 using a personal account and the user's own (non-CI) credentials, then removed the resources with Remove-TestResources.ps1:
    • The post script succeeded, and the container check and all 3 uploads worked on the first attempt. This live run was before the 30-second minimum wait was added.
    • Live pytest: the first run had 8 passes and 5 failures. Logs showed local AzureCliCredential errors ("Failed to invoke the Azure CLI"). All 5 failed tests passed when re-run. The service's job history showed a job created about 3 minutes after deployment had succeeded, so the DeID managed identity already had storage access.
  • Limitations: the live run hit no propagation delay, so only the mocked tests exercised the retry path. Validation hasn't run in the CI pipeline with the test service principal. Before deploying, I had to re-run Connect-AzAccount locally to select the subscription; that step isn't part of this change.

Created by Copilot on behalf of Josiah Vinson (Josiah Vinson (@jovinson-ms))

…s-post.ps1

Role assignment and network ACL propagation can take minutes after
deployment, causing intermittent 403 AuthorizationFailure on upload.
Retry container access and each blob upload with exponential backoff
(5s initial, 60s cap, 5 min budget) and fail with a clear error.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0d9b0092-996f-4265-b2da-1effbf3945c6
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

The retry loop proves the test principal's RBAC and the network rule, but
not the DeID managed identity's role assignment. Restore the previous 30s
wait as a minimum measured from the network rule update; time spent on
retries and uploads counts toward it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0d9b0092-996f-4265-b2da-1effbf3945c6
@jovinson-ms

Copy link
Copy Markdown
Member Author

Copilot on behalf of Josiah Vinson (@jovinson-ms): combined into #49627.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants