Repository navigation
Conversation
Chunk names now contain a hash of the matcher config and use create_before_destroy. A changed config is written to a new set of parameters, the Lambdas are switched to it, and only then is the old set deleted, including chunks orphaned when the config gets smaller. The IAM policies grant the matcher config path once, so they do not change with each config version. Fixes github-aws-runners#5520
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Tier-driven rechunking and unordered upgrade-time IAM updates can still break configuration loading.
2 open issues
What changed in this PR
Addresses #5520 by versioning runner matcher configuration chunks in SSM, aiming to keep webhook and dispatcher configuration readable during updates.
Changes:
- Adds content-based chunk names and create-before-destroy lifecycle.
- Uses stable wildcard IAM permissions for matcher parameters.
- Adds naming regression tests and updates documentation.
| File | Description |
|---|---|
| modules/webhook/webhook.tf | Versions chunks and supplies stable ARN patterns. |
| modules/webhook/README.md | Documents new data sources. |
| modules/webhook/eventbridge/variables.tf | Accepts the matcher ARN pattern. |
| modules/webhook/eventbridge/README.md | Documents the updated configuration input. |
| modules/webhook/eventbridge/dispatcher.tf | Uses wildcard matcher permissions. |
| modules/webhook/direct/webhook.tf | Uses wildcard matcher permissions. |
| modules/webhook/direct/variables.tf | Accepts the matcher ARN pattern. |
| modules/webhook/direct/README.md | Documents the updated configuration input. |
| modules/multi-runner/tests/webhook-matcher-config.tftest.hcl | Tests chunk naming stability and changes. |
| docs/configuration.md | Explains versioned matcher parameter paths. |
Review effort: Balanced
The chunk name hash now covers the parameter tier and the chunk list, so a tier change that changes the chunk count also creates a new parameter set, and an Advanced to Standard downgrade does not recreate a parameter under the same name. The webhook and dispatcher Lambdas now depend on their SSM policy, so on the first upgrade they are not switched to the new names before the policy allows them.
Contributor
Author
|
@Brend-Smits @edersonbrilhante could you check again? |
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
First-upgrade shrinking and failed-apply rollback ordering for orphaned/deposed SSM chunks still need validation against actual SSM/Lambda resources, as checked-in tests assert names and reported ordering checks use a separate terraform_data model.
No open issues 🎉
2 resolved since last review
Review effort: Balanced
OpenTofu cannot read run outputs inside assert blocks, and it sees outputs from a run two steps back as unknown. Pass the previous path in through the run variables, and compare each run with the run right before it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Fixes #5520
Summary
When the runner matcher config needs fewer SSM chunks, the orphaned chunk was destroyed before the webhook and dispatcher Lambdas stopped reading it, so config loading failed during the apply. In-place updates of several chunks could also be read half old and half new, and crossing the 1-chunk boundary renamed chunk 0 with the same ordering problem.
Changes
modules/webhook/webhook.tf: chunk names are nowrunner-matcher-config-<sha256[:12]>-<index>, withcreate_before_destroy. A changed config is written to a new set of parameters, the Lambdas are switched to it, and only then is the old set deleted. This includes chunks orphaned when the config gets smaller. An unchanged config keeps the same names, so there is no replacement on every apply.…/runner-matcher-config*once, instead of each chunk ARN. They no longer change with each config version, so there is no window where a Lambda reads names its policy does not allow yet. The pattern without a dash also covers the old single-chunk name during the first apply after upgrading.docs/configuration.md: the SSM path table shows the new name format.The first apply after upgrading replaces the existing chunks through the same path: new names first, then the Lambdas, then the old names are deleted.
Test plan
modules/multi-runner/tests/webhook-matcher-config.tftest.hcl(mocked apply): chunk names contain the hash and index, an unchanged config keeps the names, and a changed config uses new names. A tier change with the same config uses new names. The first and third runs fail without the change. Passes with bothterraform testandtofu test.terraform graph -type=applyon a local model of the resources (terraform_datawith the samecount, names and lifecycle). Without the change, the Lambda update depends on the destroy of the orphaned chunk. With the change, every chunk destroy, orphaned or replaced, depends on the Lambda update. Checked for shrinking, growing, the same number of chunks, and 1 ↔ 2 chunks.terraform fmt,terraform validatefor the root module,modules/webhookandmodules/multi-runner. Fullmodules/multi-runnertest suite (22 tests).terraform-docsimage used by CI.