Skip to content

fix: cloud control hotswap removes reserved aws: tags - #1940

Merged
aws-cdk-automation merged 3 commits into
mainfrom
fix-tag-hotswap
Sep 9, 2026
Merged

fix: cloud control hotswap removes reserved aws: tags#1940
aws-cdk-automation merged 3 commits into
mainfrom
fix-tag-hotswap

Conversation

@dgandhi62

@dgandhi62 dgandhi62 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Two cli-integ tests on the shared cc-hotswap stack failed on every PR (tests 5 & 6), blocking the merge queue:

  • hotswap deployment supports CloudControl-based resources with attribute resolution
  • hotswap deployment caches template and uses it for subsequent hotswaps

Both died on the --hotswap deploy with:

❌  cdktest-...-cc-hotswap failed to deploy
‣ HotswapFailed: [cloudcontrol] Failed to update <queue-url> (AWS::SQS::Queue)
‣ ValidationException: aws: prefixed tag key names are not allowed for external use.

Only the SQS Queue failed. The CloudWatch Dashboard and Events Rule in the same stack hotswapped successfully. Nothing in this repo seems to have caused this change - leading me to believe this is a service side behavioral change.

Why

The only thing the deploy changed on the Queue was a tag (the test fixture flips DynamicTag between deploys). For a tag change, the hotswap code sent one "replace the whole Tags list" operation:

json

[{ "op": "replace", "path": "/Tags", "value": [
    { "Key": "DynamicTag",     "Value": "new value" },
    { "Key": "DynamoTableArn", "Value": "arn:aws:dynamodb:..." }
]}]

Any tag already on the resource that isn't in this list gets dropped. The issue arose because CFN adds tags themselves on resources (aws:cloudformation:stack-name, stack-id, and logical-id). The underlying change now made it such that us sending the list of tags asked the service to remove the CFN ones as well, which it wouldn't allow because they are aws: prefixed. So it rejected the request.

Fix

We now address individual tags by their index in the resource's current Tags list, so reserved tags are never named by the patch and the service sees no change to them:

[{ "op": "replace", "path": "/Tags/4", "value": { "Key": "DynamicTag", "Value": "new value" } }]

The live ordering is not the template ordering, so the indices cannot be derived from the template. They come from reading current state with Cloud Control GetResource. If it cannot be read, we raise HotswapTagReadFailed

Known assumption

The approach assumes the live tag ordering is stable between the GetResource read and when Cloud Control applies the patch. The passing integ test shows this holds in practice for SQS; it is not proven to be deterministic across calls in general.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions Bot added the p2 label Sep 3, 2026
@dgandhi62
dgandhi62 deployed to no-approval September 3, 2026 18:15 — with GitHub Actions Active
@aws-cdk-automation
aws-cdk-automation requested a review from a team September 3, 2026 18:16
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@dgandhi62 dgandhi62 changed the title fix: fix: cloud control hotswap removes reserved aws: tags Sep 3, 2026
@dgandhi62
dgandhi62 deployed to no-approval September 4, 2026 20:53 — with GitHub Actions Active
A tags-only hotswap emitted a wholesale "replace /Tags", which declares the
resource's complete desired tag set. On a CloudFormation-created resource that
implies deleting the reserved aws:cloudformation tags, which the service rejects
with "aws: prefixed tag key names are not allowed for external use".

Address individual tags by their index in the resource's current Tags list
instead, so reserved tags are never named by the patch. Indices come from a
Cloud Control GetResource read; if that read fails we now raise
HotswapTagReadFailed rather than falling back to the request that fails.

Also re-enables the DynamicTag line in the cc-hotswap integ fixture, which was
temporarily disabled to unblock CI.
A tags-only hotswap emitted a wholesale "replace /Tags", which declares the
resource's complete desired tag set. On a CloudFormation-created resource that
implies deleting the reserved aws:cloudformation tags, which the service rejects
with "aws: prefixed tag key names are not allowed for external use".

Address individual tags by their index in the resource's current Tags list
instead, so reserved tags are never named by the patch. Indices come from a
Cloud Control GetResource read; if that read fails we now raise
HotswapTagReadFailed rather than falling back to the request that fails.

Also re-enables the DynamicTag line in the cc-hotswap integ fixture, which was
temporarily disabled to unblock CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants