Skip to content

fix(job-retry): add depends_on on IAM role policy for event source mapping - #5553

Open
tucker-pw wants to merge 1 commit into
github-aws-runners:mainfrom
tucker-pw:fix/job-retry-event-source-mapping-iam-race
Open

tucker-pw wants to merge 1 commit into
github-aws-runners:mainfrom
tucker-pw:fix/job-retry-event-source-mapping-iam-race

Conversation

@tucker-pw

Copy link
Copy Markdown

Description

aws_lambda_event_source_mapping.job_retry has no dependency on aws_iam_role_policy.job_retry, the policy that grants the Lambda execution role sqs:ReceiveMessage on the job-retry queue. Terraform is free to create the two in parallel.

Lambda validates the execution role's SQS permissions synchronously when the event source mapping is created (CreateEventSourceMapping), and IAM changes are eventually consistent. When the mapping is created before the policy has propagated, the apply fails with:

Error: creating Lambda Event Source Mapping (...-job-retry): InvalidParameterValueException:
The function execution role does not have permissions to call ReceiveMessage on SQS

Which mappings fail is down to timing, so a re-apply usually succeeds once the policies exist. Adding an explicit depends_on orders the policy before the mapping and removes the race.

The same pattern is present in both job-retry modules, so both are fixed:

  • modules/runners/job-retry/main.tf
  • modules/orchestration-providers/webhook/job-retry/job-retry.tf

Test Plan

  • terraform graph on modules/orchestration-providers/webhook/job-retry before vs after: the edge aws_lambda_event_source_mapping.job_retry -> aws_iam_role_policy.job_retry is absent before this change and present after, confirming the ordering constraint now exists.
  • terraform test for modules/orchestration-providers/webhook/job-retry passes (5/5) with the change.
  • terraform validate passes for modules/runners/job-retry.
  • terraform fmt -check clean on both files.

The timing race itself is non-deterministic and cannot be reproduced in a unit test; the graph edge above is the deterministic proof that the dependency is now enforced.

Related Issues

@tucker-pw
tucker-pw requested a review from a team as a code owner October 26, 2026 23:16
@edersonbrilhante

Copy link
Copy Markdown
Contributor

@tucker-pw Can you sign your commits?

@tucker-pw tucker-pw closed this Nov 10, 2026
@tucker-pw
tucker-pw force-pushed the fix/job-retry-event-source-mapping-iam-race branch from d95c916 to 38d0cb5 Compare November 10, 2026 18:19
@tucker-pw tucker-pw reopened this Nov 10, 2026
@tucker-pw

Copy link
Copy Markdown
Author

@edersonbrilhante Done, the commit is now signed

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants