Skip to content

Refactor RO CLI tests into shared-UAI lifecycle recordings - #7

Merged
krtcodee merged 1 commit into
krtcodee:resiliency-2026-10-01-aazdevfrom
Kushagra-Soni:dev/kushagrasoni/refactor2d257d3f
Nov 13, 2026
Merged

krtcodee merged 1 commit into
krtcodee:resiliency-2026-10-01-aazdevfrom
Kushagra-Soni:dev/kushagrasoni/refactor2d257d3f

Conversation

@Kushagra-Soni

Copy link
Copy Markdown

Summary

Refactor the RecoveryOrchestration CLI tests into the lifecycle-oriented style used by Drills, addressing the feedback in Azure/azure-cli-extensions#10530.

  • Replace 12 per-operation RO tests with four independently replayable native CLI lifecycles.
  • Reduce RO test/scaffolding Python from 2,318 lines across 14 files to one 366-line scenario file. Remove the superseded ownership ledger, metaclass, diagnostic machinery and 42 helper-only tests.
  • Replace 12 old HTTP cassettes with four newly captured standard testsdk/VCR recordings.
  • Use a preconfigured shared user-assigned managed identity for runbook scenarios. Keep identity command mutations on an isolated disposable system-assigned plan.
  • Move scoped RBAC preparation outside recording. New cassettes contain no role-assignment listing responses.
  • Add reproducible fixture/runbook setup scripts and RECOVERY_PLAN_SETUP.md, documenting manifest/environment inputs, permissions, cleanup and precise coverage.

Only RO tests, their setup documentation/scripts and recordings change. No generated commands, product code, Drills/Goals tests or PowerShell changes.

Native scenarios

Replayable test Coverage
test_recovery_plan_crud_identity_lifecycle Create/list/show/update/delete; SAI reassignment, identity show/wait/remove and independent plan readback.
test_recovery_plan_shared_uai_success_lifecycle Explicit shared-UAI runbook association; readiness, ValidateForOperation, ValidateForFailover, Failover, ValidateForReprotect, Reprotect and Ready-state Finalize.
test_recovery_plan_shared_uai_resume_lifecycle Real manual pre-action pause; native Resume on the original Failover job; original job and selected workload complete.
test_recovery_plan_shared_uai_retry_lifecycle Real failed Reprotect job from an intentional-failure runbook; native Retry on that same job adds the next numbered attempt. The unchanged failure script intentionally makes the second attempt fail too.

Retry is independent of Resume because its intentionally failing prerequisites differ from the successful manual-resume flow. No product responses, job states or recordings were fabricated or stitched.

Fixture and cleanup

Live execution uses existing approved service groups and workload resources, an existing Automation account, a published success runbook and a published pure-failure Python3 runbook. One shared UAI has Automation Job Operator and Reader on the account plus Reader on the selected storage resource.

The setup scripts run separately from scenario recording; individual tests do not create or enumerate RBAC grants. Each scenario creates and deletes its own plan, without deleting shared identities, grants, runbooks or service groups. Native deletion/absence and exact shared-UAI/grant persistence were verified. Six fresh owned plans were created sequentially and deleted during validation.

Validation

  • All four scenarios passed genuine native Record and copied isolated Playback.
  • Final full-extension serial Playback: 10 passed, 5 unchanged Drills live-only skips, 0 failures/errors; 15 discovered.
  • Existing flake8 configuration and syntax parsing passed.
  • Frozen-input hashes, copied-source import provenance, socket guards and hook-removal negative controls passed. No OS-firewall isolation claim.
  • Recording audit passed for fixture identifiers, principals, tenants, emails, JWTs, signed monitor credentials and absence of RBAC setup traffic.
  • Pre-publication correctness review identified separate-account failure-fixture normalization; the complete runbook resource ID is now normalized, with focused same-account/separate-account regression checks.

Coverage boundaries

SAI assignment remains reassignment of an existing SAI, not initial None-to-SAI. Identity wait verifies its completion predicate, not a deliberately prolonged transition. Finalize is Ready-state revalidation, not Warning-to-Ready. Retry proves another numbered attempt on the same failed job, not successful recovery or an unverified script-specific failure cause. Regional and additional identity variants are not claimed.

Replace per-operation scaffolding with four native recorded lifecycles, reproducible shared-fixture setup, and isolated Playback coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 5d8a0519-e0d0-4ee2-b065-14ff84798018
@krtcodee
krtcodee merged commit 9f5928f into krtcodee:resiliency-2026-10-01-aazdev Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants