Skip to content

fix(pool): count only pool-created instances toward the pool size - #5585

Open
hugorodro wants to merge 1 commit into
github-aws-runners:mainfrom
hugorodro:fix/pool-created-by-filter
Open

hugorodro wants to merge 1 commit into
github-aws-runners:mainfrom
hugorodro:fix/pool-created-by-filter

Conversation

@hugorodro

Copy link
Copy Markdown

Description

In our production deployment the pool lambda consistently under-filled its pool. The cause is in how it counts "still booting" capacity: every running EC2 instance of the pool's cell that is not yet in GitHub's runner list and is younger than the boot window is counted toward the pool size. Instances launched by the scale-up lambda match that test too, but they register as busy and never become idle pool runners, so the pool thinks it is closer to full than it is.

Measured at typical pool ticks against a pool of 30: 8.7 instances per tick were counted as "still booting", and 4.9 of those were scale-up launches.

This PR makes the pool count only instances it created itself:

  • ListRunnerFilters gains an optional createdBy?: RunnerSource.
  • The EC2 provider emits it as a tag:ghr:created_by filter in constructFilters (the tag is already written on every instance).
  • The EC2 pool capability passes createdBy: 'pool-lambda' when listing runners for the pool.

Scale-down is unchanged: it still lists every instance of the environment, so scale-up launches are still reaped as before. The runners_maximum_count cap on the pool top-up is also unchanged; the pool only sees fewer phantom runners and tops up accordingly.

Test Plan

  • New vitest case filters instances on the creating lambda in lambdas/libs/compute-providers/aws/ec2/src/runners.test.ts asserts the tag:ghr:created_by filter is emitted when createdBy is set.
  • lambdas/libs/compute-providers/aws/ec2/src/control-plane/pool.test.ts updated to expect createdBy: 'pool-lambda' on the pool's listRunners call; the existing runners_maximum_count cases still pass.
  • Ran yarn install --immutable && yarn format && yarn lint && yarn test && yarn build in lambdas/ on this branch (rebased on main).

Related Issues

The pool lambda counted every running instance of its cell that was
absent from GitHub's runner list and younger than the boot window as
"still booting" and therefore as pool capacity. Instances launched by
the scale-up lambda match that too, but they register busy and never
become idle pool runners, so the pool under-filled.

Measured in a production deployment against a pool of 30: 8.7 instances
per tick were counted as "still booting", and 4.9 of those were scale-up
launches that register busy.

Add `createdBy` to `ListRunnerFilters`, emit it as a
`tag:ghr:created_by` EC2 filter, and have the pool capability list only
`pool-lambda` instances. Scale-down keeps listing every instance of the
environment; the `runners_maximum_count` cap still applies to the
top-up.
@hugorodro
hugorodro requested a review from a team as a code owner November 9, 2026 19:41
@Brend-Smits
Brend-Smits requested a balanced review from Copilot November 10, 2026 12:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The inventory filter excludes scale-up instances from the maximum-count check, allowing mixed-source pools to launch beyond runners_maximum_count.

1 open issue
What changed in this PR

This PR addresses pool under-filling caused by counting booting scale-up instances as available pool capacity. It adds creation-source filtering to the EC2 provider and applies it to pool inventory queries.

Changes:

  • Add an optional createdBy runner filter and map it to the existing EC2 tag.
  • Restrict pool inventory queries to instances created by pool-lambda.
  • Update tests to verify filter construction and pool query arguments.
File Description
lambdas/​libs/​compute-providers/​core/​index.ts Add the optional creation-source filter.
lambdas/​libs/​compute-providers/​aws/​ec2/​src/​runners.ts Translate the filter into an EC2 tag constraint.
lambdas/​libs/​compute-providers/​aws/​ec2/​src/​runners.test.ts Test creation-source filter construction.
lambdas/​libs/​compute-providers/​aws/​ec2/​src/​control-plane/​pool.ts Restrict inventory to pool-created instances.
lambdas/​libs/​compute-providers/​aws/​ec2/​src/​control-plane/​pool.test.ts Verify the pool supplies the creation-source filter.

Review effort: Balanced

runnerOwner,
runnerType,
statuses: ['running'],
createdBy: 'pool-lambda',

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants