Skip to content

[durabletask] Add recorded cross-region failover tests - #10529

Open
Bernd Verst (berndverst) wants to merge 2 commits into
Azure:mainfrom
berndverst:durabletask-failover-recordings
Open

Bernd Verst (berndverst) wants to merge 2 commits into
Azure:mainfrom
berndverst:durabletask-failover-recordings

Conversation

@berndverst

@berndverst Bernd Verst (berndverst) commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

🤖 PR Validation — ️✔️ All clear

Breaking Changes Suppressions
️✔️ None ️✔️ None

This checklist is used to make sure that common guidelines for a pull request are followed.

Related command

az durabletask scheduler failover-instance

Test-only follow-up to #10482.

  • Add a genuine, sanitized recording with 99 HTTP interactions for an eastasia primary and a southeastasia partner, both Dedicated capacity 1.
  • Enable the failover lifecycle in default offline playback instead of skipping it as live-only.
  • Cover configuration create/show/update/delete, graceful promotion of the secondary, forced failover back to the original primary, and replication-role checks.
  • Use generated LRO polling for cleanup, add cleanup-helper regressions, and update the test documentation.

No extension runtime code, dependencies, version, or extension-index changes are included. Failed diagnostic captures are excluded from this follow-up.

General Guidelines

  • Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
  • Have you run python scripts/ci/test_index.py -q locally? (pip install azdev required)
  • My extension version conforms to the Extension version schema

The existing extension version is unchanged because this PR only adds test coverage and recordings.

For new extensions:

N/A: this is an existing extension.

About Extension Publish

There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

Validation

Commands were run locally using the Azure CLI development virtual environment:

  • python -m azdev test --discover --no-exitfirst azext_durabletask -a --tb=short -q - passed: 69 tests, no skips.
  • python scripts\ci\test_index.py -q - passed: 9 tests run, including 2 checks skipped outside CI.
  • python -m azdev style durabletask --pep8 --pylint - passed: pylint and flake8.
  • python -m azdev linter durabletask --min-severity medium - passed.
  • python -m azdev scan --confidence-level MEDIUM -d src\durabletask -r --include '*.py' '*.yaml' '*.md' '*.rst' '*.json' '*.cfg' --continue-on-failure - passed: no secrets detected.

The live recording also passed using python -m azdev test --series --live test_durabletask_failover_lifecycle_live -a --tb=short --show-capture=no before the test was renamed to test_durabletask_failover_lifecycle_recorded for normal playback. The successful scenario's temporary resource groups were deleted. A separate account-identifier audit of the recording passed; no real subscription, tenant, caller, or credential details are included.

Enable default playback of the eastasia/southeastasia failover lifecycle, verify replica role changes, and cover generated-LRO cleanup handling. Include only test code, the successful sanitized recording, and related documentation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 02:53
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the test-only update enabling recorded failover playback.

No open issues 🎉

What changed in this PR

This test-only follow-up enables offline playback of the Durable Task cross-region failover lifecycle, without changing extension runtime code or dependencies.

Changes:

  • Switch the scenario from live-only to recorded playback using East Asia and Southeast Asia, with replication-role checks after graceful and forced failover.
  • Use generated long-running-operation polling for cleanup and add cleanup-helper regression tests.
  • Update documentation with the recorded scenario’s coverage, regions, and re-recording command.
File Description
src/​durabletask/​README.md Documents recorded failover coverage and re-recording instructions.
src/​durabletask/​azext_durabletask/​tests/​latest/​test_recording_processors.py Adds cleanup tests for polling, missing configurations, and unexpected errors.
src/​durabletask/​azext_durabletask/​tests/​latest/​test_durabletask_scenario.py Enables recorded failover playback, checks role transitions, and updates cleanup.

Review effort: Balanced

@x-engineering-agent

Copy link
Copy Markdown

Hi Bernd Verst (@berndverst),
Please write the description of changes which can be perceived by customers into HISTORY.rst.
If you want to release a new extension version, please update the version in pyproject.toml (or setup.py, if the extension has not migrated yet) as well.

@yonzhan

Copy link
Copy Markdown
Collaborator

Please fix CI issues

Use pytest below 9.2 after azdev setup. Pytest 9.2 adds a NodeId report field that pytest-forked 1.7.5 cannot marshal, causing every Linux forked test to exit with status 3. Verified all 69 durabletask tests with CI-style Linux fork isolation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@berndverst

Copy link
Copy Markdown
Member Author

Yong Zhang (@yonzhan) Fixed the CI runner failure in 569cdbce5.

The Python 3.10–3.14 jobs all used pytest 9.2.0 with pytest-forked 1.7.5. Pytest 9.2 adds a NodeId object to each test report, which pytest-forked attempts to marshal. Serialization fails and the child exits with status 3, even when the test itself passes. I reproduced this on Linux with a single passing unittest and no Azure code.

The shared CI setup now installs pytest<9.2 after azdev setup, retaining fork isolation rather than skipping any tests. No extension logic or recorded responses were changed.

Verification: all 69 durabletask tests passed under Linux with --forked -n 1, using Azure CLI 2.92.0, pytest 9.1.1, pytest-forked 1.7.5, and vcrpy 9.2.0. Repository pre-push checks also passed all 69 tests, lint, and style. The CI template syntax check and medium-confidence secret scan passed.

@berndverst

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 10529 in repo Azure/azure-cli-extensions

@berndverst

Copy link
Copy Markdown
Member Author

Yong Zhang (@yonzhan) The fix is pushed and the Linux forked suite passes. I requested /azp run, but Azure Pipelines reports that I do not have sufficient privileges. Please trigger /azp run on this PR so the Python 3.10–3.14 matrix can validate commit 569cdbce5.

@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants