Repository navigation
Conversation
Each control-plane lambda references only its role ARN, and its permissions live in separate role policies and attachments with no ordering between them. Terraform therefore updates the function and its policies in parallel. When an apply adds a permission the new code needs (for example SSM read access to an additional GitHub App's parameters), the function can be updated first, and containers that cold-start in that window fail with AccessDenied until the policy lands. We hit this when adding a GitHub App: CloudTrail showed UpdateFunctionConfiguration six seconds before PutRolePolicy, and warm containers that started in between kept failing. Make scale-up, scale-down, pool and ssm-housekeeper depend on every policy and attachment on their own role, alongside the existing log-group dependency. No resource changes; only the apply order changes. Lambdas built with modules/lambda (job-retry, termination-watcher, registration-janitor) attach their policies outside that module, so the same ordering needs a module input and is left for a follow-up.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Each control-plane lambda in
modules/runnersreferences only its role ARN. Its permissions live in separateaws_iam_role_policyandaws_iam_role_policy_attachmentresources with no ordering between them and the function, so Terraform applies the function update and the policy update in parallel.That matters when an apply adds a permission the new configuration needs. In our deployment we added a second GitHub App: the new config pointed the lambdas at the additional App's SSM parameters, and the grant for those parameters arrived in the same apply. CloudTrail showed
UpdateFunctionConfigurationsix seconds beforePutRolePolicy. Containers that cold-started in that window failed with:The role policy was correct once the apply finished.
This PR makes
scale-up,scale-down,poolandssm-housekeeperdepend on every policy and attachment on their own role, alongside the existing log-group dependency. No resource changes; only the apply order changes. Counted resources (VPC execution role, X-Ray, AMI SSM read, job-retry SQS publish, service-linked role) are listed as whole resources, so a disabled one is a no-op dependency.Not covered here: lambdas built with
modules/lambda(job-retry, termination-watcher, registration-janitor). They attach their policies outside that module and read the role from the module'slambdaoutput, which also carries the function, so the samedepends_onfrom the caller would form a cycle. Fixing those needs either arole_policiesinput onmodules/lambda(policies attached inside the module, before the function) or a role-only output. Happy to follow up with whichever the maintainers prefer.Test Plan
terraform init -backend=false && terraform validateinmodules/runners,modules/multi-runner,examples/defaultandexamples/multi-runner: valid, no dependency cycles, same warnings asmain.terraform graphinmodules/runners: each of the four functions now has edges to its role's policies and attachments;mainhas none.terraform fmt -checkandtflintwith the repo's.tflint.hcland.github/lint/tflint.tfvarsonmodules/runnersandmodules/runners/pool: clean.Related Issues
None found.