Skip to content

fix(runners): apply lambda IAM policies before updating the functions - #5587

Open
hugorodro wants to merge 1 commit into
github-aws-runners:mainfrom
hugorodro:fix/lambda-iam-policy-ordering
Open

hugorodro wants to merge 1 commit into
github-aws-runners:mainfrom
hugorodro:fix/lambda-iam-policy-ordering

Conversation

@hugorodro

Copy link
Copy Markdown

Description

Each control-plane lambda in modules/runners references only its role ARN. Its permissions live in separate aws_iam_role_policy and aws_iam_role_policy_attachment resources with no ordering between them and the function, so Terraform applies the function update and the policy update in parallel.

That matters when an apply adds a permission the new configuration needs. In our deployment we added a second GitHub App: the new config pointed the lambdas at the additional App's SSM parameters, and the grant for those parameters arrived in the same apply. CloudTrail showed UpdateFunctionConfiguration six seconds before PutRolePolicy. Containers that cold-started in that window failed with:

not authorized to perform: ssm:GetParameters on resource: arn:aws:ssm:...:parameter/.../app-2/github_app_id because no identity-based policy allows the ssm:GetParameters action

The role policy was correct once the apply finished.

This PR makes scale-up, scale-down, pool and ssm-housekeeper depend on every policy and attachment on their own role, alongside the existing log-group dependency. No resource changes; only the apply order changes. Counted resources (VPC execution role, X-Ray, AMI SSM read, job-retry SQS publish, service-linked role) are listed as whole resources, so a disabled one is a no-op dependency.

Not covered here: lambdas built with modules/lambda (job-retry, termination-watcher, registration-janitor). They attach their policies outside that module and read the role from the module's lambda output, which also carries the function, so the same depends_on from the caller would form a cycle. Fixing those needs either a role_policies input on modules/lambda (policies attached inside the module, before the function) or a role-only output. Happy to follow up with whichever the maintainers prefer.

Test Plan

  • terraform init -backend=false && terraform validate in modules/runners, modules/multi-runner, examples/default and examples/multi-runner: valid, no dependency cycles, same warnings as main.
  • terraform graph in modules/runners: each of the four functions now has edges to its role's policies and attachments; main has none.
  • terraform fmt -check and tflint with the repo's .tflint.hcl and .github/lint/tflint.tfvars on modules/runners and modules/runners/pool: clean.

Related Issues

None found.

Each control-plane lambda references only its role ARN, and its permissions
live in separate role policies and attachments with no ordering between
them. Terraform therefore updates the function and its policies in
parallel. When an apply adds a permission the new code needs (for example
SSM read access to an additional GitHub App's parameters), the function can
be updated first, and containers that cold-start in that window fail with
AccessDenied until the policy lands.

We hit this when adding a GitHub App: CloudTrail showed
UpdateFunctionConfiguration six seconds before PutRolePolicy, and warm
containers that started in between kept failing.

Make scale-up, scale-down, pool and ssm-housekeeper depend on every policy
and attachment on their own role, alongside the existing log-group
dependency. No resource changes; only the apply order changes.

Lambdas built with modules/lambda (job-retry, termination-watcher,
registration-janitor) attach their policies outside that module, so the
same ordering needs a module input and is left for a follow-up.
@hugorodro
hugorodro requested a review from a team as a code owner November 9, 2026 19:54

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant