Skip to content

fix(webhook): replace matcher config chunks before deleting the old ones - #5521

Open
atsikham wants to merge 6 commits into
github-aws-runners:mainfrom
atsikham:fix/webhook-matcher-config-chunk-replacement
Open

atsikham wants to merge 6 commits into
github-aws-runners:mainfrom
atsikham:fix/webhook-matcher-config-chunk-replacement

Conversation

@atsikham

@atsikham atsikham commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #5520

Summary

When the runner matcher config needs fewer SSM chunks, the orphaned chunk was destroyed before the webhook and dispatcher Lambdas stopped reading it, so config loading failed during the apply. In-place updates of several chunks could also be read half old and half new, and crossing the 1-chunk boundary renamed chunk 0 with the same ordering problem.

Changes

  • modules/webhook/webhook.tf: chunk names are now runner-matcher-config-<sha256[:12]>-<index>, with create_before_destroy. A changed config is written to a new set of parameters, the Lambdas are switched to it, and only then is the old set deleted. This includes chunks orphaned when the config gets smaller. An unchanged config keeps the same names, so there is no replacement on every apply.
  • The IAM policies of the direct webhook and the dispatcher grant …/runner-matcher-config* once, instead of each chunk ARN. They no longer change with each config version, so there is no window where a Lambda reads names its policy does not allow yet. The pattern without a dash also covers the old single-chunk name during the first apply after upgrading.
  • docs/configuration.md: the SSM path table shows the new name format.
  • The name hash also covers the parameter tier and the chunk list. The webhook and dispatcher Lambdas depend on their SSM policy.

The first apply after upgrading replaces the existing chunks through the same path: new names first, then the Lambdas, then the old names are deleted.

Test plan

  • New modules/multi-runner/tests/webhook-matcher-config.tftest.hcl (mocked apply): chunk names contain the hash and index, an unchanged config keeps the names, and a changed config uses new names. A tier change with the same config uses new names. The first and third runs fail without the change. Passes with both terraform test and tofu test.
  • Ordering checked with terraform graph -type=apply on a local model of the resources (terraform_data with the same count, names and lifecycle). Without the change, the Lambda update depends on the destroy of the orphaned chunk. With the change, every chunk destroy, orphaned or replaced, depends on the Lambda update. Checked for shrinking, growing, the same number of chunks, and 1 ↔ 2 chunks.
  • Failed apply in the same model: with a failure before the Lambda update, the Lambda still lists the old names and all old chunks still exist.
  • terraform fmt, terraform validate for the root module, modules/webhook and modules/multi-runner. Full modules/multi-runner test suite (22 tests).
  • READMEs regenerated with the terraform-docs image used by CI.

Chunk names now contain a hash of the matcher config and use
create_before_destroy. A changed config is written to a new set of
parameters, the Lambdas are switched to it, and only then is the old set
deleted, including chunks orphaned when the config gets smaller. The IAM
policies grant the matcher config path once, so they do not change with
each config version.

Fixes github-aws-runners#5520
@atsikham
atsikham requested review from a team as code owners October 14, 2026 11:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Tier-driven rechunking and unordered upgrade-time IAM updates can still break configuration loading.

2 open issues
What changed in this PR

Addresses #5520 by versioning runner matcher configuration chunks in SSM, aiming to keep webhook and dispatcher configuration readable during updates.

Changes:

  • Adds content-based chunk names and create-before-destroy lifecycle.
  • Uses stable wildcard IAM permissions for matcher parameters.
  • Adds naming regression tests and updates documentation.
File Description
modules/​webhook/​webhook.tf Versions chunks and supplies stable ARN patterns.
modules/​webhook/​README.md Documents new data sources.
modules/​webhook/​eventbridge/​variables.tf Accepts the matcher ARN pattern.
modules/​webhook/​eventbridge/​README.md Documents the updated configuration input.
modules/​webhook/​eventbridge/​dispatcher.tf Uses wildcard matcher permissions.
modules/​webhook/​direct/​webhook.tf Uses wildcard matcher permissions.
modules/​webhook/​direct/​variables.tf Accepts the matcher ARN pattern.
modules/​webhook/​direct/​README.md Documents the updated configuration input.
modules/​multi-runner/​tests/​webhook-matcher-config.tftest.hcl Tests chunk naming stability and changes.
docs/​configuration.md Explains versioned matcher parameter paths.

Review effort: Balanced

Comment thread modules/webhook/webhook.tf Resolved
Comment thread modules/webhook/direct/webhook.tf Resolved
atsikham and others added 2 commits October 27, 2026 22:09
The chunk name hash now covers the parameter tier and the chunk list, so a
tier change that changes the chunk count also creates a new parameter set,
and an Advanced to Standard downgrade does not recreate a parameter under
the same name. The webhook and dispatcher Lambdas now depend on their SSM
policy, so on the first upgrade they are not switched to the new names
before the policy allows them.
@atsikham

Copy link
Copy Markdown
Contributor Author

@Brend-Smits @edersonbrilhante could you check again?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

First-upgrade shrinking and failed-apply rollback ordering for orphaned/deposed SSM chunks still need validation against actual SSM/Lambda resources, as checked-in tests assert names and reported ordering checks use a separate terraform_data model.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

OpenTofu cannot read run outputs inside assert blocks, and it sees outputs
from a run two steps back as unknown. Pass the previous path in through
the run variables, and compare each run with the run right before it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(webhook): dispatcher config breaks when the runner matcher config shrinks to fewer SSM chunks

3 participants