Skip to content

[Resiliency] Add resiliency extension 1.0.0b1 (preview) for Microsoft.AzureResilienceManagement api-version 2026-10-01 - #10530

Merged
JaysonTaiMicrosoft merged 43 commits into
Azure:mainfrom
krtcodee:resiliency-2026-10-01-aazdev
Nov 16, 2026
Merged

JaysonTaiMicrosoft merged 43 commits into
Azure:mainfrom
krtcodee:resiliency-2026-10-01-aazdev

Conversation

@krtcodee

@krtcodee krtcodee commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

New extension resiliency 1.0.0b1 (preview) providing az resiliency commands for Microsoft.AzureResilienceManagement api-version 2026-10-01 (stable).

Changes

  • src/resiliency/ – new extension (generated aaz/ code, setup.py, HISTORY.rst, README.md with full command reference).
  • Preview metadata: VERSION = 1.0.0b1, Development Status :: 4 - Beta, azext.isPreview: true.
  • src/service_name.json – added az resiliency → "Azure Resiliency".
  • Command examples on 80 commands, generated from swagger x-ms-examples (5 invalid swagger values corrected: drill-run stage Fault → FaultInjection; unsupported --type/--user-assigned-identities replaced by --user-assigned or removed).
  • src/resiliency/linter_exclusions.yml – excludes require_wait_command_if_no_wait for
    resiliency drill drill-run and resiliency recovery-plan recovery-job: their POST actions
    return no provisioningState, so aaz-dev cannot generate a wait command for these groups.

Testing

  • Scenario tests (tests/latest/test_resiliency.py): usage-plan CRUD and enrollment CRUD, each with output assertions.
  • Goals scenario tests (tests/latest/test_resiliency_goals_plans_enrollments.py): usage-plan and enrollment --no-wait + wait; the full goal-assignment lifecycle (create, show, list, update, refresh-goal-resource, goal-resource list/show, update-goal-resource, recommend-capacity, delete) and operation-status show, on a test-created zonal VM, service group, usage plan and enrollment. All 24 goal-assignment, usage-plan and operation-status commands are now covered.
  • Unit test (tests/latest/test_resiliency_custom.py): drill drill-run list-report-download-url returns the URL.
  • Recovery-plan scenario tests (tests/latest/test_recovery_plan_lifecycles.py), four lifecycle tests against a pre-built environment (setup documented in tests/latest/RECOVERY_PLAN_SETUP.md, with setup scripts; a shared user-assigned identity gets its roles once at setup, so the tests make no role-assignment calls):
    • test_recovery_plan_crud_identity_lifecycle: create, list, show, update, delete, identity assign/show/wait/remove.
    • test_recovery_plan_shared_uai_success_lifecycle: check-readiness, validate-for-operation, validate-for-failover, failover, validate-for-reprotect, reprotect, finalize, recovery jobs and resources.
    • test_recovery_plan_shared_uai_resume_lifecycle: recovery-job resume after a manual pre-action pause.
    • test_recovery_plan_shared_uai_retry_lifecycle: recovery-job retry of a failed reprotect job.
    • The recordings cover all 26 recovery-plan commands.
  • Drill scenario tests (tests/latest/test_resiliency_drills.py):
    • Replayed from recordings (15 commands): test_resiliency_drill_management covers drill create/show/list/update/delete/wait, identity show/assign/remove/wait, drill-resource list/show, add-or-update-resource, resync-readiness-check and drill-run list.
    • @live_only() against a drill and drill run prepared in advance (11 commands), because running a drill needs heavy setup and long-running operations. Evidence of one successful local run of each:
      • test_resiliency_drill_prepared_lifecycle_coverage (validate-for-execution, start, end): 1 passed in 117.36s; service readback confirmed Start reached Running and End returned the drill to NotRunning with attestation Success.
      • test_resiliency_drill_prepared_run_coverage (drill-run show, add-note, generate-report, list-report-download-url, drill-run-resource list/show): 1 passed in 113.58s.
      • test_resiliency_drill_run_mark_as_complete (drill-run mark-as-complete): 1 passed in 148.41s.
      • test_resiliency_drill_run_fail_over (drill-run fail-over, without --no-wait): run on fresh NSG/FaultInjection drill runs; the command waited for the operation, exited 0, and the stage then offered MarkAsComplete (sanitized --debug excerpt in the comment below).
      • Live-only tests skip with a reason when RESILIENCY_TEST_PREPARED_* variables are not set or the drill run is not in a state that allows the action.
    • Not covered yet: drill-run resume and drill-run reprotect (need a drill run in a Recovery Orchestration state; reprotect currently fails in the service with RHUBCloudInternalError).
  • Scenario tests recorded live against the service; offline replay passes (10 tests; the 5 live-only drill tests are skipped in replay).
  • azdev linter resiliency --min-severity medium: all rules pass.
  • azdev style resiliency: pylint and flake8 pass.
  • GitHub-style example check (azdev linter EXT --src <branch> --tgt <merge-base> --rules missing_command_example): pass.

Workarounds pending spec/service fixes (2026-10-01)

  • planType is returned by the service but marked create-only in the spec (@visibility(Lifecycle.Create) → x-ms-mutability: ["create"]).

    • CLI: custom.py / commands.py add it back to recovery-plan create/show/list/update output.
    • PowerShell: a README directive marks it readable, which also fixes Update-AzResiliencyRecoveryPlan (GET + PUT previously omitted the required planType).
  • Goal assignments declare final-state-via: location, but the initial response may not carry Location. PowerShell: a README directive switches create/update to original-uri (same approach as Cdn / FrontDoor); this also avoids the PATCH final result coming back as a JSON string.

  • Recovery-plan actions return the final result (GET on Location) as an operationStatuses record with properties as a JSON-encoded string. CLI (custom.py mixin) and PowerShell (custom/*.json.cs AfterFromJson) decode it so jobId (check-readiness, failover, reprotect) and recoveryResourceQualifications (validate-for-failover/reprotect) are shown. No-op once the service returns the spec shape.

  • Goal resources update (updateGoalResources): the service identifies each item by the goal resource's own ARM ID (resources[].id), which the spec marks read-only. CLI (--resources item id, required) and PowerShell (custom/GoalResource.json.cs, custom/UpdateGoalResourceRequest.json.cs) send it and reject items without it before sending; otherwise the operation stays Accepted and locks the goal assignment.

  • CLI recovery-plan update and recovery-plan identity assign/remove: the final result of the PATCH (final-state-via: location) is an operation-status record (properties: null) instead of the recovery plan, so update printed only the operation's id/name and identity assign/remove printed {}. The commands re-read the recovery plan in on_200, after the final response, to show the updated plan / resulting identity (skipped with --no-wait; no-op once the service returns the resource). PowerShell is not affected (Update- is GET + PUT).

  • Action POSTs (drills, drill runs, recovery-plan actions, recovery jobs): the service does not reliably bind the published operation-id header to its operationId parameter (workflow never starts, later POSTs get 409). Both clients also send the same ID as the operationId query parameter: CLI for all 22 commands that send operation-id (approach from [Resiliency] Add Drill operation ID compatibility krtcodee/azure-cli-extensions#3), PowerShell via one pipeline step (custom/Module.OperationIdQuery.cs).

  • Drill add-or-update resources (Drills_AddOrUpdateResources): the service accepts faultProperties.defaultFault, but the spec marks it read-only, so it was never sent. CLI (--resource-lists item fault-properties.default-fault: fault-name, fault-urn, target-resource-id) and PowerShell (DefaultFaultName/DefaultFaultUrn/DefaultFaultTargetResourceId on Update-AzResiliencyDrillResource, custom/IncludeOrUpdateResource.json.cs) send it only when supplied; requests without it are unchanged.

  • Goal-assignment actions return the final result with properties as a JSON-encoded string, so the output was empty. CLI goal-assignment update-goal-resource / recommend-capacity (_DecodeStringPropertiesOn200) and PowerShell Update-AzResiliencyGoalResource / Get-AzResiliencyGoalAssignmentCapacity (custom/GoalAssignmentActionResults.json.cs) decode it. CLI goal-assignment update (PATCH) gets an operation-status record instead of the goal assignment and failed after the update succeeded; it now takes the final state from the goal assignment itself (final-state-via: original-uri), as PowerShell does.

  • operation-status show / Get-AzResiliencyOperationStatus: the service returns HTTP 202 while the operation is still running, but the spec allows only 200, so both failed. CLI (custom.py) and PowerShell (README directive adding 202 with the same body) now show the current status.

  • CLI drill drill-run list-report-download-url: the spec marks downloadUrl as a secret, so the default output hid it and the command printed {}. The command now outputs it (secret_hidden=False); the response schema keeps its secret marker. Covered by a mocked test (tests/latest/test_resiliency_custom.py). PowerShell is not affected (DownloadUrl is a SecureString).

These will be removed once the Resiliency spec/service is fixed and the clients are regenerated.

Other changes

  • PowerShell: removed Set-AzResiliencyRecoveryPlan. It ignored -EnableSystemAssignedIdentity (adding a user-assigned identity dropped the system-assigned one) and duplicates New-/Update-AzResiliencyRecoveryPlan; Azure CLI has no equivalent.
  • PowerShell: removed Set-AzResiliencyGoalAssignment. It sends PUT, which the Goals service treats as create-only (400 RHUBUserErrorInvalidRequestParameter on an existing goal assignment); New- creates and Update-AzResiliencyGoalAssignment (PATCH) updates, matching Azure CLI goal-assignment create/update.
  • Removed operations the service does not support in this release, in both tools: CLI recovery-plan failover-commit, validate-for-failover-commit, test-failover, validate-for-test-failover, test-failover-cleanup, validate-for-test-failover-cleanup, recovery-plan recovery-job cancel; PowerShell Invoke-AzResiliencyRecoveryPlanFailoverCommit, Test-AzResiliencyRecoveryPlanFailoverCommitValidation, Invoke-AzResiliencyRecoveryPlanTestFailover, Test-AzResiliencyRecoveryPlanTestFailoverValidation, Invoke-AzResiliencyRecoveryPlanTestFailoverCleanup, Test-AzResiliencyRecoveryPlanTestFailoverCleanupValidation, Stop-AzResiliencyRecoveryJob. Also removed CLI unified-resilience-item list/show and PowerShell Get-AzResiliencyUnifiedResilienceItem: the service returns 401 "The AAD authentication failed - unauthorized." (plain text, not ARM error JSON) on every call to unifiedResilienceItems, while the same token works on other routes of the same service group (correlation 7c635f46-7647-473a-b40a-189c14c4d771).

Copilot AI balanced review requested due to automatic review settings October 28, 2026 08:08
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@microsoft-github-policy-service microsoft-github-policy-service Bot added the customer-reported Issues that are reported by GitHub users external to the Azure organization. label Oct 28, 2026
@microsoft-github-policy-service

Copy link
Copy Markdown
Contributor

Thank you for your contribution krtcodee! We will review the pull request and get back to you soon.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The command redacts the secret-marked downloadUrl during output deserialization, preventing users from receiving a usable report link even when the service succeeds.

1 open issue
What changed in this PR

Adds the GA resiliency 1.0.0 Azure CLI extension for managing Azure Resilience Management resources using API version 2026-10-01.

Changes:

  • Introduces generated command groups for drills, recovery plans, goal assignments, usage plans, and operation status.
  • Adds extension packaging, GA metadata, command documentation, and targeted linter exclusions.
  • Adds scenario tests and recorded service responses.
File Description
src/​service_name.json Registers the Azure Resiliency service mapping.
src/​resiliency/​setup.py Defines extension packaging and GA version.
src/​resiliency/​setup.cfg Adds a setup configuration placeholder.
src/​resiliency/​README.md Documents installation, usage, and commands.
src/​resiliency/​linter_exclusions.yml Exempts two groups from wait-command requirements.
src/​resiliency/​HISTORY.rst Records the initial release.
src/​resiliency/​azext_resiliency/​tests/​latest/​test_resiliency.py Adds usage-plan, enrollment, and list scenarios.
src/​resiliency/​azext_resiliency/​tests/​latest/​recordings/​test_resiliency_recovery_plan_list.yaml Records recovery-plan list responses.
src/​resiliency/​azext_resiliency/​tests/​latest/​__init__.py Initializes the latest test package.
src/​resiliency/​azext_resiliency/​tests/​__init__.py Initializes the test package.
src/​resiliency/​azext_resiliency/​custom.py Adds custom-command scaffolding.
src/​resiliency/​azext_resiliency/​commands.py Adds the custom command-table hook.
src/​resiliency/​azext_resiliency/​azext_metadata.json Declares GA status and minimum CLI version.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​usage_plan/​enrollment/​_list.py Lists enrollments with pagination.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​usage_plan/​enrollment/​_delete.py Deletes enrollments with asynchronous polling.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​usage_plan/​enrollment/​__init__.py Imports enrollment commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​usage_plan/​enrollment/​__cmd_group.py Registers the enrollment command group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​usage_plan/​_wait.py Waits for usage-plan conditions.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​usage_plan/​_show.py Retrieves a usage plan.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​usage_plan/​_delete.py Deletes a usage plan.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​usage_plan/​__init__.py Imports usage-plan commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​usage_plan/​__cmd_group.py Registers the usage-plan command group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​unified_resilience_item/​__init__.py Imports unified resilience item commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​unified_resilience_item/​__cmd_group.py Registers the unified resilience item group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​recovery_resource/​__init__.py Imports recovery-resource commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​recovery_resource/​__cmd_group.py Registers the recovery-resource group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​recovery_job/​recovery_job_resource/​__init__.py Imports recovery-job resource commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​recovery_job/​recovery_job_resource/​__cmd_group.py Registers the recovery-job resource group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​recovery_job/​_retry.py Retries failed recovery-job resources.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​recovery_job/​__init__.py Imports recovery-job commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​recovery_job/​__cmd_group.py Registers the recovery-job group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​identity/​__init__.py Imports recovery-plan identity commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​identity/​__cmd_group.py Registers recovery-plan identity management.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​_test_failover_cleanup.py Triggers test-failover cleanup.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​_reprotect.py Triggers recovery-plan reprotection.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​_finalize.py Finalizes recovery-plan configuration.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​_failover_commit.py Commits recovery-plan failover.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​_delete.py Deletes a recovery plan.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​_check_readiness.py Triggers recovery readiness checks.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​__init__.py Imports recovery-plan commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​recovery_plan/​__cmd_group.py Registers the recovery-plan group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​operation_status/​_show.py Retrieves asynchronous operation status.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​operation_status/​__init__.py Imports operation-status commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​operation_status/​__cmd_group.py Registers the operation-status group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​goal_assignment/​goal_resource/​_show.py Retrieves a goal resource.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​goal_assignment/​goal_resource/​_list.py Lists goal resources with pagination.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​goal_assignment/​goal_resource/​__init__.py Imports goal-resource commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​goal_assignment/​goal_resource/​__cmd_group.py Registers the goal-resource group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​goal_assignment/​_wait.py Waits for goal-assignment conditions.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​goal_assignment/​_refresh_goal_resource.py Refreshes resources within a goal assignment.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​goal_assignment/​_recommend_capacity.py Requests capacity recommendations.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​goal_assignment/​_delete.py Deletes a goal assignment.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​goal_assignment/​__init__.py Imports goal-assignment commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​goal_assignment/​__cmd_group.py Registers the goal-assignment group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​identity/​__init__.py Imports drill identity commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​identity/​__cmd_group.py Registers drill identity management.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_run/​drill_run_resource/​__init__.py Imports drill-run resource commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_run/​drill_run_resource/​__cmd_group.py Registers the drill-run resource group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_run/​_resume.py Resumes a paused drill-run workflow.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_run/​_reprotect.py Triggers drill-run reprotection.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_run/​_mark_as_complete.py Marks a drill-run stage complete.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_run/​_list_report_download_url.py Retrieves a report download URL.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_run/​_add_note.py Adds notes to a drill run.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_run/​__init__.py Imports drill-run commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_run/​__cmd_group.py Registers the drill-run group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_resource/​__init__.py Imports drill-resource commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​drill_resource/​__cmd_group.py Registers the drill-resource group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​_start.py Starts a drill instance.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​_resync_readiness_check.py Triggers resource and permission drift checks.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​_end.py Ends a drill with attestation details.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​_delete.py Deletes a drill.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​__init__.py Imports drill commands.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​drill/​__cmd_group.py Registers the drill group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​__init__.py Imports the root command group.
src/​resiliency/​azext_resiliency/​aaz/​latest/​resiliency/​__cmd_group.py Registers az resiliency.
src/​resiliency/​azext_resiliency/​aaz/​latest/​__init__.py Initializes the latest generated package.
src/​resiliency/​azext_resiliency/​aaz/​__init__.py Initializes the generated command package.
src/​resiliency/​azext_resiliency/​_params.py Adds the custom argument-loading hook.
src/​resiliency/​azext_resiliency/​_help.py Adds custom-help scaffolding.
src/​resiliency/​azext_resiliency/​__init__.py Connects generated commands to the extension loader.

Review effort: Balanced

pass

def _output(self, *args, **kwargs):
result = self.deserialize_output(self.ctx.vars.instance, client_flatten=True)
@yonzhan

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

@krtcodee krtcodee changed the title [Resiliency] Add resiliency extension 1.0.0 for Microsoft.AzureResili… [Resiliency] Add resiliency extension 1.0.0 for Microsoft.AzureResilienceManagement api-version 2026-10-01 Oct 28, 2026
@JaysonTaiMicrosoft JaysonTaiMicrosoft added the Request X Engineering Agent Ask the agent to triage this issue or review this PR label Oct 29, 2026
@x-engineering-agent

x-engineering-agent Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

X Engineering Agent Pull request review: Comments

krtcodee CI passed, but this is a partial review, not merge clearance. Five Low convention findings remain below; the panel confirmed no additional semantic defects in the inspected material.

Check Status Detail
Upstream CI Passed The handoff reports one successful check, zero failures and no pending checks.
Agent live tests Not run This is a review-only PR.
Regression coverage Not applicable Not applicable to the changed files.
Findings 5 low findings 5 low
Risk Risk 100/100 Critical Critical; owner review required

To address

Tick each item as you address it.

Review details

Upstream CI

The handoff reports one successful check, zero failures and no pending checks.

Test validation

  • Agent live test: Not run; this is a review-only PR. Check author-provided test evidence and upstream CI separately.
  • Regression coverage: Not applicable to the changed files.

Inspection limits: The Reviewer read 32 of 305 inspection chunks. 101 changed files remain unread, partial or unavailable, including generated command paths, scenario tests and recordings; listed unchanged context was not fully inspected. Handwritten overrides and packaging were inspected, but complete request/response and replay coverage was not established.

Suggested author validation: In src/resiliency/azext_resiliency/tests/latest/test_resiliency_custom.py:44-54, rerun the mocked report-URL case with a valid drill-run GUID; also replay test_resiliency_drills on this head. This is a validation suggestion, not a confirmed failing test.

Risk assessment

100/100 · Critical

The Critical rating is driven by security-sensitive behavior, public CLI behavior, dependency or supply chain and 2 additional signal(s).

  • Change scope: 143 changed files, 53370 changed lines (+53370 / -0), including 118 production files.
  • Affected components: resiliency
  • Risk drivers: security-sensitive behavior (+28); public CLI behavior (+18); dependency or supply chain (+18); failure-handling behavior (+12); generated output (+12)
  • Regression evidence: Changed regression tests are included, reducing risk.
  • Confidence: Medium because changed-line patches were available for 90 of 118 production files.
  • Required review: Owning-squad review is required for resiliency before merge.

Ask the Agent

Tick a box to queue an action. Requires write access to this repository.

  • Re-review this PR
  • Run Agent live tests on this PR

X Engineering Agent · Reviewer · React with +1 or -1 to train the agent

@x-engineering-agent x-engineering-agent Bot added the X Engineering Agent Reviewed Agent reviewed this PR label Oct 29, 2026
x-engineering-agent[bot]

This comment was marked as outdated.

@yonzhan

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

@yonzhan
Yong Zhang (yonzhan) requested a review from a team November 9, 2026 21:16
Add the standard license header to 10 recovery-plan test files (azdev verify license) and suppress the too-few-public-methods and unused-argument warnings on DrillRunListReportDownloadUrl, matching the other helper classes in custom.py.
@JaysonTaiMicrosoft

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

x-engineering-agent[bot]

This comment was marked as outdated.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please provide screenshot of the actual live_only test passing using azdev test ... --live

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the dependent resources can take up to 2 hours to create, so reusing a pre-built environment is reasonable. dataprotection, site-recovery and the drill tests in this PR do the same.

That said, these tests are much more complex than other scenario tests in the repo: about 2,300 lines across 14 files for 12 recorded tests. The drill and goal tests cover 50 commands in about 690 lines. Most of the extra code doesn't test the CLI, and it will be hard for anyone else to maintain or re-record.

Could you simplify them along these lines?

  1. Consolidate into 2–3 lifecycle tests. For example:

    • CRUD and identity
    • failover → reprotect → finalize
    • job resume and retry

    Use plain self.cmd(...) steps with checks, addCleanup to delete the plan, and one small job-polling helper.

  2. Remove the test-harness machinery:

    • the ownership ledger
    • evidence capture and the Session.send / cli_ctx.invoke patches
    • the metaclass
    • the offline tests of the helpers (test_recovery_plan_safe_cleanup.py, test_recovery_plan_finalize_wait.py)
  3. Document how to rebuild the environment. List the env vars and the manifest format, and ideally check in a setup script or template.

  4. Avoid per-test role assignments. Ideally, pre-create a user-assigned identity with the needed roles as part of the environment, if the service supports it for CustomRunbook. Recordings contain unrelated identities.** az role assignment list --scope … --assignee-object-id … lists every assignment at the scope and filters on the client. The failover recording alone has nearly 290 principal IDs, almost all unrelated to the test, including users and groups. The CLI already supports --user-assigned and an associated-identity of type UserAssigned.

For reference, dataprotection's restore tests and site-recovery's failover tests follow this simpler pattern.

@naga-nandyala

Copy link
Copy Markdown
Contributor

This needs rework/fixing. Moving to next milestone (Dec 2026)

@krtcodee krtcodee changed the title [Resiliency] Add resiliency extension 1.0.0 for Microsoft.AzureResilienceManagement api-version 2026-10-01 [Resiliency] Add resiliency extension 1.0.0b1 (preview) for Microsoft.AzureResilienceManagement api-version 2026-10-01 Nov 13, 2026
Replace per-operation scaffolding with four native recorded lifecycles, reproducible shared-fixture setup, and isolated Playback coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5d8a0519-e0d0-4ee2-b065-14ff84798018
Refactor RO CLI tests into shared-UAI lifecycle recordings
@JaysonTaiMicrosoft

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

@x-engineering-agent x-engineering-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review comments are inline on the changed lines. See the X Engineering Agent Report.

Comment on lines +388 to +389
properties.set_const("drillType", "Regional", AAZStrType, ".regional", typ_kwargs={"flags": {"required": True}})
properties.set_const("drillType", "Zonal", AAZStrType, ".zonal", typ_kwargs={"flags": {"required": True}})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High Reject simultaneous regional and zonal variants

Both object options are independently accepted. With both supplied, the second set_const overwrites Regional with Zonal regardless of command-line order, silently discarding the regional choice. Add mutual-exclusion validation through the source model or a handwritten override. Check both conflict orders fail before PUT and each single variant produces its own type.

X Engineering Agent · Reviewer · React with +1 or -1 to train the reviewer

Comment on lines +319 to +324
_builder.set_prop("userAssigned", AAZListType, ".mi_user_assigned", typ_kwargs={"flags": {"action": "remove"}})
_builder.set_prop("systemAssigned", AAZStrType, ".mi_system_assigned", typ_kwargs={"flags": {"action": "remove"}})

user_assigned = _builder.get(".userAssigned")
if user_assigned is not None:
user_assigned.set_elements(AAZStrType, ".")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium Removing one identity detaches every user-assigned identity

On supported CLI 2.75.0, the selective removal path passes [A] into core aaz/_field_value.py:458–464, whose singleton branch sends identity.type=None. Removing A from A+B consequently detaches B too. Implement selective removal on affected cores or require a fixed minimum core. Verify removing A preserves B, and explicit remove-all removes both.

X Engineering Agent · Reviewer · React with +1 or -1 to train the reviewer

@JaysonTaiMicrosoft

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

@JaysonTaiMicrosoft
JaysonTaiMicrosoft merged commit 83c85b7 into Azure:main Nov 16, 2026
24 checks passed
@azclibot

Copy link
Copy Markdown
Collaborator

[Release] Update index.json for extension [ resiliency-1.0.0b1 ] : https://dev.azure.com/msazure/One/_build/results?buildId=191410653&view=results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

customer-reported Issues that are reported by GitHub users external to the Azure organization. Request X Engineering Agent Ask the agent to triage this issue or review this PR X Engineering Agent Reviewed Agent reviewed this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.