[atsikham]

Fixes #5410

Summary

The code that slows down writes to Parameter Store (so they don't hit AWS's rate limit) had two problems:

1. It only started slowing down writes once a single Lambda invocation's own batch of runners reached the store's maxWritesPerSecond number, and it based the delay only on that one invocation's batch size. But Parameter Store's write limit applies to the whole AWS account, not to one invocation: several pools can each run their own scale-up/pool Lambda at the same time, each one under its own threshold, while their writes added together go over the account's limit. And a batch smaller than the threshold got no slowdown at all.
2. The write limit was hardcoded to 40 writes/second. 40 is actually the read limit. AWS documents 3 TPS for PutParameter (10 with higher throughput).

Changes

• The code now always slows down writes once a write limit is set on the store, no matter how big or small this invocation's batch is.
• A new Terraform variable, ssm_parameter_store_max_concurrent_invocations, sets how many scale-up/pool Lambda invocations can realistically be running at the same time across the whole deployment. The delay between writes is divided by that number, so the account-wide write budget is shared across all of them instead of each invocation assuming it has the whole budget to itself.
• A new Terraform variable, ssm_parameter_store_max_writes_per_second, makes the write rate configurable. Default stays 40, so nothing changes for existing setups. Lower it if PutParameter gets throttled.
• Both variables are wired up for the scale-up and pool Lambdas in all three setups: modules/runners, modules/multi-runner, and the webhook orchestration modules (runner-config, webhook, scale-runners, pool). The defaults (1 invocation, 40 writes/sec) match how the code behaved before.
• Multi-runner v1: the same two variables apply to all runner configurations.
• Multi-runner v2: global_config_storage_provider.aws.ssm.parameters.max_concurrent_invocations and max_writes_per_second set the default for all runner configurations. A runner configuration can override either value in its own storage_provider.aws.ssm.parameters.
• docs/rate-limits-and-tuning.md is updated: the Parameter Store section explains both new variables and shows the real AWS limits (before it said reads and writes share 40 TPS). Also fixed a claim in the batch_size table that was no longer true (the slowdown no longer depends on batch size), and updated the advice for large deployments. The Parameter Store section also has a table showing where to set the values in each module.

Test plan

• yarn test — github-runner.test.ts checks the delay math in addDelay (including the small-batch case that used to get no delay, and the sharing-across-invocations behavior), plus an end-to-end test using fake timers through createStartRunnerConfig. runner-config-store.test.ts and storage-providers.test.ts check that the write-rate limit can be set, falls back to 40 if the value is missing or invalid, and is picked up from the environment correctly.
• terraform test in modules/multi-runner (v1 translation, v2 global default, per-runner override), scale-runners, pool and runner-config: the values reach the scale-up and pool Lambda environment.
• The migration-test CI job no longer reports the two environment variables as removed when moving from v1 to v2 inputs.
• yarn lint / yarn format
• tsc --noEmit
• terraform fmt -check / terraform validate (tflint could not be installed in this environment — no network access to its Homebrew source — so the full set of pre-commit checks was not run locally)

[edersonbrilhante]

@atsikham also implement the changes in https://github.com/github-aws-runners/terraform-aws-github-runner/tree/main/modules/orchestration-providers/webhook and https://github.com/github-aws-runners/terraform-aws-github-runner/tree/main/modules/multi-runner

multi-runner is the module to deploy multi runners
orchestration-providers/webhook is the module v2 to configure pool/scaleup for a runner

[atsikham]

@edersonbrilhante digging into the wiring: in modules/runners the two new pacing settings are one flat value shared by scale-up and pool lambda. But multi-runner/orchestration-providers/webhook split lambda config per type with no shared bucket. Duplicate the field under both scale.up and pool, or is there a preferred place for settings shared across lambdas in a runner group?

[edersonbrilhante]

Today we have 2 versions in the multi-runner variables.
v1 -> https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/variables.tf#L94-L288
v2 -> https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/variables.tf#L291-L642

v1 or you configure as new property per runner type or create new variables in variables.tf
v2 you can configure as new property per runner type and in global level common settings.
global ssm config: https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/variables.experimental.ssm.tf

Then you need to update these configs that standardize the structure v1 and v2.
https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/config.experimental.translation.tf
https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/config.experimental.resolved.tf

To pass to pool/scale up finally update the ssm block
https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/orchestration-providers/webhook/variables.tf#L212-L221
https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/orchestration-providers/webhook/scale-runners/variables.tf#L134-L140
https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/orchestration-providers/webhook/pool/variables.tf#L117-L120

For v2, I would set in global level, but allow override per runner type, similar what other configs.

[atsikham]

Today we have 2 versions in the multi-runner variables. v1 -> https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/variables.tf#L94-L288 v2 -> https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/variables.tf#L291-L642

v1 or you configure as new property per runner type or create new variables in variables.tf v2 you can configure as new property per runner type and in global level common settings. global ssm config: https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/variables.experimental.ssm.tf

Then you need to update these configs that standardize the structure v1 and v2. https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/config.experimental.translation.tf https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/multi-runner/config.experimental.resolved.tf

To pass to pool/scale up finally update the ssm block https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/orchestration-providers/webhook/variables.tf#L212-L221 https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/orchestration-providers/webhook/scale-runners/variables.tf#L134-L140 https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/modules/orchestration-providers/webhook/pool/variables.tf#L117-L120

For v2, I would set in global level, but allow override per runner type, similar what other configs.

done, please check again

[Copilot]

🟡 Changes recommended

A duplicate import of createAwsSsmRunnerConfigStore causes a syntax error that prevents the test file from loading and breaks type checking.

2 open issues

• Remove duplicate import to fix test file syntax error · New
• Base throughput guidance on Parameter Store write limits · New

What changed in this PR

This PR addresses #5410 by making runner configuration write pacing account for expected concurrent Lambda invocations, rather than only large batches within one invocation.

Changes:

• Applies pacing to all nonempty batches when the storage provider defines a write limit.
• Adds configurable write-rate and concurrency settings across root, multi-runner, and webhook orchestration modules, including v2 defaults and per-runner overrides.
• Adds pacing and configuration tests and updates tuning documentation.

File	Description
variables.tf	Declares root-level pacing inputs.
README.md	Documents the new root inputs.
modules/​runners/​variables.tf	Declares runner-module pacing inputs.
modules/​runners/​scale-up.tf	Passes pacing settings to scale-up.
modules/​runners/​README.md	Documents runner pacing inputs.
modules/​runners/​pool/​variables.tf	Adds pool configuration fields.
modules/​runners/​pool/​README.md	Documents pool configuration additions.
modules/​runners/​pool/​main.tf	Sets pool Lambda pacing variables.
modules/​runners/​pool.tf	Forwards pacing settings to the pool module.
modules/​runner-config/​variables.tf	Adds storage-provider pacing options.
modules/​runner-config/​tests/​tags.tftest.hcl	Checks scale-up environment propagation.
modules/​runner-config/​README.md	Documents storage-provider pacing options.
modules/​runner-config/​orchestration-provider.tf	Forwards settings to webhook orchestration.
modules/​orchestration-providers/​webhook/​variables.tf	Extends the storage-provider interface.
modules/​orchestration-providers/​webhook/​scale-runners/​variables.tf	Accepts scaling pacing settings.
modules/​orchestration-providers/​webhook/​scale-runners/​tests/​scale-runners.tftest.hcl	Checks scale-up pacing variables.
modules/​orchestration-providers/​webhook/​scale-runners/​scale-up.tf	Configures scale-up pacing variables.
modules/​orchestration-providers/​webhook/​scale-runners/​README.md	Documents scaling interface additions.
modules/​orchestration-providers/​webhook/​README.md	Documents webhook storage settings.
modules/​orchestration-providers/​webhook/​pool/​variables.tf	Accepts pool pacing settings.
modules/​orchestration-providers/​webhook/​pool/​tests/​provider.tftest.hcl	Checks pool pacing variables.
modules/​orchestration-providers/​webhook/​pool/​README.md	Documents pool storage settings.
modules/​orchestration-providers/​webhook/​pool/​pool.tf	Configures pool pacing variables.
modules/​multi-runner/​variables.tf	Adds v1 inputs and v2 per-runner overrides.
modules/​multi-runner/​variables.experimental.storage-provider.tf	Defines v2 global pacing defaults.
modules/​multi-runner/​tests/​config-translation.tftest.hcl	Tests translation, inheritance, and overrides.
modules/​multi-runner/​runners.tf	Passes resolved settings to runners.
modules/​multi-runner/​README.md	Documents multi-runner pacing configuration.
modules/​multi-runner/​config.experimental.translation.tf	Translates v1 pacing inputs.
modules/​multi-runner/​config.experimental.resolved.tf	Resolves per-runner overrides against defaults.
main.tf	Forwards root pacing inputs.
lambdas/​libs/​storage-providers/​storage-providers.ts	Loads the write limit from the environment.
lambdas/​libs/​storage-providers/​storage-providers.test.ts	Tests write-limit forwarding.
lambdas/​libs/​storage-providers/​aws/​ssm/​runner-config-store.ts	Makes the store write limit configurable.
lambdas/​libs/​storage-providers/​aws/​ssm/​runner-config-store.test.ts	Tests write-limit configuration and fallback.
lambdas/​functions/​control-plane/​src/​scale-runners/​github-runner.ts	Paces small batches and scales delays by concurrency.
lambdas/​functions/​control-plane/​src/​scale-runners/​github-runner.test.ts	Adds delay calculation and execution tests.
docs/​rate-limits-and-tuning.md	Updates pacing configuration and tuning guidance.

_{Review effort: Balanced}