Skip to content

[ManagedCleanroom] Support CGS as secure key release authority for CPK datasets - #10545

Merged
mansoor sarfraz (msarfraz) merged 3 commits into
Azure:mainfrom
imddanish:feature/managedcleanroom-cgs-skr-authority
Nov 12, 2026
Merged

mansoor sarfraz (msarfraz) merged 3 commits into
Azure:mainfrom
imddanish:feature/managedcleanroom-cgs-skr-authority

Conversation

@imddanish

@imddanish imddanish commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

🤖 PR Validation — ❌ Action needed

Breaking Changes Suppressions
❌ 1 ⚠️ 8 added
❌Azure CLI Extensions Breaking Change Test
❌managedcleanroom
rule cmd_name rule_message suggest_message
❌ 1007 - ParaRemove managedcleanroom frontend analytics dataset publish cmd managedcleanroom frontend analytics dataset publish removed parameter kek_maa_url please add back parameter kek_maa_url for cmd managedcleanroom frontend analytics dataset publish
⚠️ 1006 - ParaAdd managedcleanroom frontend analytics dataset publish cmd managedcleanroom frontend analytics dataset publish added parameter kek_authority_endpoint
⚠️ 1006 - ParaAdd managedcleanroom frontend analytics dataset publish cmd managedcleanroom frontend analytics dataset publish added parameter kek_authority_type
⚠️ 1008 - ParaPropAdd managedcleanroom frontend analytics dataset publish cmd managedcleanroom frontend analytics dataset publish update parameter access_mode: added property choices=['read', 'write']
⚠️ 1008 - ParaPropAdd managedcleanroom frontend analytics dataset publish cmd managedcleanroom frontend analytics dataset publish update parameter identity_issuer_url: added property deprecate_info_hide=True
⚠️ 1008 - ParaPropAdd managedcleanroom frontend analytics dataset publish cmd managedcleanroom frontend analytics dataset publish update parameter identity_issuer_url: added property deprecate_info_target=--identity-issuer-url
⚠️ 1009 - ParaPropRemove managedcleanroom frontend analytics skr-policy cmd managedcleanroom frontend analytics skr-policy update parameter dataset_id: removed property required=True

This PR adds warning or check suppressions. Fix the underlying warning where possible and justify each remaining suppression in a code comment or the PR description.

8 added suppressions
File Line Kind Code
src/managedcleanroom/azext_managedcleanroom/analytics_frontend_api/aio/operations/_operations.py 612 pylint pipeline_response: PipelineResponse = await self._client._pipeline.run( # pylint: disable=protected-access
src/managedcleanroom/azext_managedcleanroom/analytics_frontend_api/aio/operations/_operations.py 636 type ignore {}) # type: ignore
src/managedcleanroom/azext_managedcleanroom/analytics_frontend_api/aio/operations/_operations.py 638 type ignore return cast(JSON, deserialized) # type: ignore
src/managedcleanroom/azext_managedcleanroom/analytics_frontend_api/operations/_operations.py 212 pylint def build_collaboration_analytics_contract_skr_policy_get_request( # pylint: disable=name-too-long
src/managedcleanroom/azext_managedcleanroom/analytics_frontend_api/operations/_operations.py 229 type ignore _url: str = _url.format(**path_format_arguments) # type: ignore
src/managedcleanroom/azext_managedcleanroom/analytics_frontend_api/operations/_operations.py 1639 pylint pipeline_response: PipelineResponse = self._client._pipeline.run( # pylint: disable=protected-access
src/managedcleanroom/azext_managedcleanroom/analytics_frontend_api/operations/_operations.py 1663 type ignore {}) # type: ignore
src/managedcleanroom/azext_managedcleanroom/analytics_frontend_api/operations/_operations.py 1665 type ignore return cast(JSON, deserialized) # type: ignore

Note

The service-side change in Azure Clean Rooms (frontend 2026-03-01-preview adds CGS as an SKR authority) is merged. This PR can be merged once approved.

Related command

  • az managedcleanroom frontend analytics dataset publish
  • az managedcleanroom frontend analytics skr-policy

Description

The Azure Clean Rooms frontend (2026-03-01-preview) now supports the Clean Room Governance Service (CGS) as a secure key release (SKR) authority alongside Microsoft Azure Attestation (MAA) for customer-provided-key (CPK) datasets. This PR updates the extension to match:

  • A dataset KEK now carries an authority object: {"type": "CGS"} or {"type": "MAA", "endpoint": "<maa-url>"}. The previous maaUrl field is deprecated by the service but still accepted.
  • A new collaboration-level endpoint, GET /collaborations/{collaborationId}/analytics/skrpolicy, returns the CGS SKR policy. The dataset-level .../datasets/{datasetId}/skrpolicy endpoint is deprecated by the service and kept for MAA-backed datasets.

Previously, dataset publish in CPK mode always required --kek-maa-url, so CGS-backed datasets could not be published from the CLI.

Changes

  • _params.py

    • Adds --kek-authority-type (CGS, MAA) and --kek-authority-endpoint to dataset publish.
    • --kek-maa-url becomes a hidden, deprecated alias of --kek-authority-endpoint. It uses the same c.deprecate(..., hide=True) pattern as --job-id.
    • --dataset-id on skr-policy is now optional.
  • _frontend_custom.py

    • skr-policy: without --dataset-id, calls the new collaboration-level CGS endpoint. With it, behaves as before.
    • dataset publish (CPK): builds kek.authority.
      • CGS: no endpoint; passing one is rejected.
      • MAA: endpoint required.
      • An endpoint on its own (including the old --kek-maa-url) means MAA, so existing scripts keep working.
      • For MAA, maaUrl is still sent alongside authority so frontends without the new field keep working. The new service ignores it when authority is present.
  • analytics_frontend_api/operations/_operations.py + aio/operations/_operations.py: adds analytics_contract_skr_policy_get (sync + async) and the authority object to the dek/kek JSON templates. As in [ManagedCleanroom] Add --subdirectory parameter for dataset publish #9852, only the actual changes from the updated frontend.yaml are included, to avoid unrelated churn from a full regeneration.

  • _help.py: updated skr-policy help with collaboration-level and dataset examples. dataset publish now documents the new parameters and has separate CGS and MAA CPK examples.

  • tests/latest/test_frontend_dataset.py, test_frontend_collaboration.py: 7 new tests:

    • CGS publish
    • endpoint-only publish implies MAA
    • MAA without an endpoint is rejected
    • CGS with an endpoint is rejected
    • missing authority is rejected
    • case-insensitive authority type
    • collaboration-level skr-policy

    Existing CPK and skr-policy tests are updated.

  • HISTORY.rst: added 1.0.0b11 changelog entry.

  • setup.py: bumped extension version 1.0.0b10 → 1.0.0b11.

Review feedback (latest commit)

These fixes are all in az managedcleanroom frontend analytics dataset publish and were checked against the current frontend models:

  • identity.issuerUrl: the frontend now rejects it ([ValidateIsNull] on DatasetIdentity.IssuerUrl). The CLI no longer requires or sends it. --identity-issuer-url stays as a hidden, deprecated flag that is ignored with a warning pointing to az managedcleanroom frontend oidc set-issuer-url.
  • Help values: examples and help now use valid values: Azure_BlobStorage for the storage type, read/write for access mode, and csv/json/parquet for schema format. --access-mode only accepts read/write (case-insensitive).
  • --body conflicts: every flag that only applies without --body is now in the conflict check, including the new authority flags and the DEK/KEK, --schema-format, --allowed-fields and --subdirectory flags. Combining any of them with --body is an error instead of being silently ignored.
  • allowedFields: the frontend requires datasetAccessPolicy.allowedFields, so --allowed-fields is now required. Empty entries are dropped, and at least one field is needed.

Backward compatibility

  • --kek-maa-url keeps working and now prints a deprecation warning.
  • skr-policy -c <id> -d <dataset> is unchanged.
  • --identity-issuer-url keeps being accepted, but is ignored with a deprecation warning, because the frontend rejects issuerUrl.
  • --allowed-fields is now required, and --access-mode only accepts read/write. The frontend already rejected requests without these, so this only moves the error to the CLI.
  • No API version change (2026-03-01-preview).

Breaking change report

  • 1007 ParaRemove kek_maa_url: the tool compares internal parameter names. --kek-maa-url is still accepted as a deprecated alias of --kek-authority-endpoint, the same pattern as --job-id → --run-id, so existing commands keep working with a deprecation warning.
  • Suppressions (8): these are the existing generated-SDK patterns (protected-access, name-too-long, type: ignore) repeated for the new analytics_contract_skr_policy_get operation, matching every other operation in analytics_frontend_api.

Validation

  • ✅ azdev style managedcleanroom: Pylint and Flake8 passed
  • ✅ azdev linter managedcleanroom: no violations
  • ✅ python scripts/ci/test_index.py -q: OK
  • ✅ All 75 unit tests pass
  • ✅ Ran the real az commands against a local mock frontend:
    • both skr-policy forms hit the right URLs
    • CGS, MAA and deprecated --kek-maa-url publish requests send the expected kek bodies
    • the deprecation warnings and validation errors show as expected
  • ✅ The request bodies the CLI builds (SSE, CPK+CGS, CPK+MAA, subdirectory) pass the frontend's DatasetDetails schema

General Guidelines

  • Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
  • Have you run python scripts/ci/test_index.py -q locally? (pip install wheel==0.30.0 required)

About CLI version requirement

  • Does the PR requires a new CLI core version? (No)
  • If yes, do you have a label of CLI Version Required and have you updated azext_metadata.json in your extension to refer to a new CLI core version?

About Extension Publish

  • Have you updated the version of setup.py?
  • Have you updated HISTORY.rst?
  • Note: src/index.json is auto-updated post-merge — not modified in this PR.

…K datasets

Aligns the managedcleanroom frontend commands with the 2026-03-01-preview
frontend spec changes that add CGS (Clean Room Governance Service) as a
secure key release (SKR) authority alongside MAA.

* dataset publish: add --kek-authority-type (CGS, MAA) and
  --kek-authority-endpoint; send kek.authority. Deprecate --kek-maa-url as
  a hidden alias of --kek-authority-endpoint. For MAA, the legacy maaUrl is
  still sent for backward compatibility with older frontends.
* skr-policy: --dataset-id is now optional; when omitted, returns the
  collaboration-level CGS SKR policy from the new
  /collaborations/{id}/analytics/skrpolicy endpoint.
* analytics_frontend_api (sync + async): add
  analytics_contract_skr_policy_get and the authority object in the
  dek/kek JSON templates (substantive delta only).
* Tests, help, HISTORY.rst and version bump 1.0.0b10 -> 1.0.0b11.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@microsoft-github-policy-service microsoft-github-policy-service Bot added the customer-reported Issues that are reported by GitHub users external to the Azure organization. label Nov 2, 2026
@microsoft-github-policy-service

Copy link
Copy Markdown
Contributor

Thank you for your contribution imddanish! We will review the pull request and get back to you soon.

@imddanish
imddanish marked this pull request as ready for review November 2, 2026 08:09
Copilot AI balanced review requested due to automatic review settings November 2, 2026 08:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new KEK authority parameters are omitted from the mutual-exclusion check, allowing them to be silently ignored when combined with --body and potentially retaining an unintended authority.

1 open issue
What changed in this PR

Adds CGS secure key release support for customer-provided-key datasets in the Managed Cleanroom extension, while preserving existing MAA CLI usage. Merge remains dependent on deployment of the service-side change.

Changes:

  • Adds KEK authority options, validation, and backward-compatible MAA payloads.
  • Adds collaboration-level SKR policy retrieval through synchronous and asynchronous clients.
  • Updates help, unit tests, and release metadata for 1.0.0b11.
File Description
src/​managedcleanroom/​setup.py Bumps the extension version to 1.0.0b11.
src/​managedcleanroom/​HISTORY.rst Records CGS support and CLI changes.
src/​managedcleanroom/​azext_managedcleanroom/​tests/​latest/​test_frontend_dataset.py Tests authority payloads and validation.
src/​managedcleanroom/​azext_managedcleanroom/​tests/​latest/​test_frontend_collaboration.py Tests collaboration-level SKR policy routing.
src/​managedcleanroom/​azext_managedcleanroom/​analytics_frontend_api/​operations/​_operations.py Adds synchronous SKR retrieval and updates payload documentation.
src/​managedcleanroom/​azext_managedcleanroom/​analytics_frontend_api/​aio/​operations/​_operations.py Adds asynchronous SKR retrieval and updates payload documentation.
src/​managedcleanroom/​azext_managedcleanroom/​_params.py Registers authority options and the deprecated MAA alias.
src/​managedcleanroom/​azext_managedcleanroom/​_help.py Documents policy scopes and CGS/MAA publishing examples.
src/​managedcleanroom/​azext_managedcleanroom/​_frontend_custom.py Builds authority payloads and routes SKR policy requests.

Review effort: Balanced

@yonzhan

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).

@anantshankar17 Anant (anantshankar17) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining issues:

  1. Parameter-based dataset publishing requires and sends identity.issuerUrl, which the current frontend rejects.
  2. New examples use unsupported AzureStorageAccount / ReadWrite values; these should be Azure_BlobStorage and read or write.
  3. Combining --body with new authority flags silently ignores those flags instead of rejecting the combination.

Md Danish and others added 2 commits November 11, 2026 05:43
* Stop requiring and sending identity.issuerUrl: the frontend rejects it
  ([ValidateIsNull] on DatasetIdentity.IssuerUrl). --identity-issuer-url is
  now a hidden, deprecated flag that is ignored with a warning pointing to
  `az managedcleanroom frontend oidc set-issuer-url`.
* Reject --body combined with any parameter-mode flag, including the new
  --kek-authority-type/--kek-authority-endpoint and the DEK/KEK,
  --schema-format, --allowed-fields and --subdirectory flags, instead of
  silently ignoring them.
* Require --allowed-fields: the frontend requires
  datasetAccessPolicy.allowedFields. Empty entries are dropped and at least
  one field name must be supplied.
* Restrict --access-mode to read/write and fix help examples and docs to
  use valid values: Azure_BlobStorage storage type, read/write access mode,
  csv/json/parquet schema format.
* Tests: update publish tests and add coverage for every --body conflict,
  issuerUrl not being sent, the deprecation warning, and allowed-fields
  validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@imddanish imddanish changed the title [ManagedCleanroom] [Do not merge] Support CGS as secure key release authority for CPK datasets [ManagedCleanroom] Support CGS as secure key release authority for CPK datasets Nov 11, 2026
@imddanish

Copy link
Copy Markdown
Contributor Author

Remaining issues:

  1. Parameter-based dataset publishing requires and sends identity.issuerUrl, which the current frontend rejects.
  2. New examples use unsupported AzureStorageAccount / ReadWrite values; these should be Azure_BlobStorage and read or write.
  3. Combining --body with new authority flags silently ignores those flags instead of rejecting the combination.

Thanks Anant (@anantshankar17), all three are Fixed in the latest commit, plus one related issue I found while verifying against the frontend:

  • identity.issuerUrl: the CLI no longer requires or sends issuerUrl (the frontend now rejects it via [ValidateIsNull] on DatasetIdentity.IssuerUrl). --identity-issuer-url is kept as a hidden, deprecated flag that's ignored with a warning pointing to az managedcleanroom frontend oidc set-issuer-url, so existing scripts don't break.
  • Example values: all dataset publish examples and help now use Azure_BlobStorage and read/write. --access-mode is restricted to read/write in the CLI (case-insensitive). --storage-account-type stays free-form, with the valid values listed in help, so new store types don't need a CLI release. I also corrected --schema-format help, which said "default: Delta"; the frontend accepts csv/json/parquet.
  • --body + flags: every parameter-mode flag is now part of the --body conflict check: the new authority flags, plus the DEK/KEK, --schema-format, --allowed-fields and --subdirectory flags that were also being silently ignored. A test covers each flag.
  • Additional fix, allowedFields: the frontend requires datasetAccessPolicy.allowedFields, but the CLI only sent it when --allowed-fields was given, so parameter-mode publish would still fail. --allowed-fields is now required, empty entries are dropped, and at least one field is needed.

@anantshankar17

Copy link
Copy Markdown

Yong Zhang (@yonzhan) please help complete this PR soon.

@anantshankar17

Anant (anantshankar17) commented Nov 12, 2026 •

Copy link
Copy Markdown

mansoor sarfraz (@msarfraz) / Yong Zhang (@yonzhan): can one of you please help complete this ? We are still waiting for it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@msarfraz
mansoor sarfraz (msarfraz) merged commit 7273727 into Azure:main Nov 12, 2026
24 checks passed
@azclibot

Copy link
Copy Markdown
Collaborator

[Release] Update index.json for extension [ managedcleanroom-1.0.0b11 ] : https://dev.azure.com/msazure/One/_build/results?buildId=190838561&view=results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Attestation Auto-Assign Auto assign by bot customer-reported Issues that are reported by GitHub users external to the Azure organization.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants