Skip to content

[DeID] Surface job errors in live test failures; random live job names - #49626

Closed
Josiah Vinson (jovinson-ms) wants to merge 2 commits into
Azure:mainfrom
jovinson-ms:jovinson-ms-deid-live-test-diagnostics
Closed

Josiah Vinson (jovinson-ms) wants to merge 2 commits into
Azure:mainfrom
jovinson-ms:jovinson-ms-deid-live-test-diagnostics

Conversation

@jovinson-ms

Copy link
Copy Markdown
Member

Problem

Live runs of azure-health-deidentification (test_create_wait_finish, test_create_wait_finish_async, test_list_job_docs_pagination, deidentify_documents_async.py) kept failing with HttpResponseError: Operation returned an invalid status 'OK'. That message is how azure-core LRO polling reports a job that reached Failed. It hid the actual job and document errors, so the underlying service issue went undiagnosed for weeks.

Two related problems:

  • Fixed live job names: conftest only regenerated the uniquifier when recording. Live CI sets AZURE_SKIP_LIVE_RECORDING=True, so every live run reused the checked-in uniquifier.conf value. Any live runs that share resources would then try to create jobs with the same names.
  • Running after result(timeout=120): a past build failed because the job was still Running after this call. This is not an SDK bug. LROPoller.result(timeout) returns the current resource once the timeout expires, and the generated polling config and DeID's terminal states are correct. The tests just never checked whether the poller had finished.

Fix

  • Test helper: tests/deid_base_test_case.py adds wait_for_job / wait_for_job_async.
    • They wait up to 600s and fail explicitly if the job hasn't finished.
    • On HttpResponseError / OperationFailed they fetch the job and fail with its status, job.error, summary counts, and up to 20 non-succeeded documents with their errors.
    • Fetching diagnostics never hides the original exception.
  • Batch tests: the three batch tests use the helper. Their per-document asserts now include the document location and error.
  • tests/conftest.py: every live run gets a random uniquifier; AZURE_TEST_KEEP_UNIQUIFIER opts out. It is written to uniquifier.conf only when recording, so playback stays deterministic.
  • Samples: deidentify_documents and list_job_documents, sync and async, print Job <name> ended with status <status>: <error> and re-raise on failure. The async samples also close their credential.

Validation

  • Playback: pytest tests passed 13/13 after rebasing on current main, and black --check -l 120 made no changes.
  • Live: run once before the rebase against temporary resources deployed with New-TestResources.ps1 and torn down afterwards. The package did not change on main between that run and the rebase.
    • Tests: 13/13 passed. test_exception_throws hit a transient Azure CLI credential failure on the first attempt and passed on rerun.
    • The live runs used fresh random uniquifiers.
    • The 4 changed samples passed. A deliberately empty input prefix showed the new failure output (the job error, exit code 1).
    • Live auth used AZURE_TEST_USE_CLI_AUTH=true locally; that setting is not part of this commit.
  • Not addressed: the samples still call result(timeout=60/120), matching the README. If the service is slow they can return a job that is still running.

Note

A sibling PR (stale codegen + lint fix for this package) also edits tests/conftest.py, so a merge conflict is possible depending on which PR merges first.

Created by Copilot on behalf of Josiah Vinson (Josiah Vinson (@jovinson-ms))

- Add wait_for_job/wait_for_job_async test helpers that report job status,
  error, summary, and unsuccessful documents instead of azure-core's generic
  'Operation returned an invalid status OK', and fail explicitly when the job
  is not terminal after the timeout (now 600s).
- Generate a fresh uniquifier for every live run (including
  AZURE_SKIP_LIVE_RECORDING); only persist it when recording.
- Samples print job.error before re-raising on job failure.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 40cdcc81-accc-4b2e-af24-fd79c6480740
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 40cdcc81-accc-4b2e-af24-fd79c6480740
@jovinson-ms

Copy link
Copy Markdown
Member Author

Copilot on behalf of Josiah Vinson (@jovinson-ms): combined into #49627.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants