Repository navigation
Conversation
The pool lambda counted every running instance of its cell that was absent from GitHub's runner list and younger than the boot window as "still booting" and therefore as pool capacity. Instances launched by the scale-up lambda match that too, but they register busy and never become idle pool runners, so the pool under-filled. Measured in a production deployment against a pool of 30: 8.7 instances per tick were counted as "still booting", and 4.9 of those were scale-up launches that register busy. Add `createdBy` to `ListRunnerFilters`, emit it as a `tag:ghr:created_by` EC2 filter, and have the pool capability list only `pool-lambda` instances. Scale-down keeps listing every instance of the environment; the `runners_maximum_count` cap still applies to the top-up.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The inventory filter excludes scale-up instances from the maximum-count check, allowing mixed-source pools to launch beyond runners_maximum_count.
1 open issue
What changed in this PR
This PR addresses pool under-filling caused by counting booting scale-up instances as available pool capacity. It adds creation-source filtering to the EC2 provider and applies it to pool inventory queries.
Changes:
- Add an optional
createdByrunner filter and map it to the existing EC2 tag. - Restrict pool inventory queries to instances created by
pool-lambda. - Update tests to verify filter construction and pool query arguments.
| File | Description |
|---|---|
lambdas/libs/compute-providers/core/index.ts |
Add the optional creation-source filter. |
lambdas/libs/compute-providers/aws/ec2/src/runners.ts |
Translate the filter into an EC2 tag constraint. |
lambdas/libs/compute-providers/aws/ec2/src/runners.test.ts |
Test creation-source filter construction. |
lambdas/libs/compute-providers/aws/ec2/src/control-plane/pool.ts |
Restrict inventory to pool-created instances. |
lambdas/libs/compute-providers/aws/ec2/src/control-plane/pool.test.ts |
Verify the pool supplies the creation-source filter. |
Review effort: Balanced
| runnerOwner, | ||
| runnerType, | ||
| statuses: ['running'], | ||
| createdBy: 'pool-lambda', |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description
In our production deployment the pool lambda consistently under-filled its pool. The cause is in how it counts "still booting" capacity: every running EC2 instance of the pool's cell that is not yet in GitHub's runner list and is younger than the boot window is counted toward the pool size. Instances launched by the scale-up lambda match that test too, but they register as busy and never become idle pool runners, so the pool thinks it is closer to full than it is.
Measured at typical pool ticks against a pool of 30: 8.7 instances per tick were counted as "still booting", and 4.9 of those were scale-up launches.
This PR makes the pool count only instances it created itself:
ListRunnerFiltersgains an optionalcreatedBy?: RunnerSource.tag:ghr:created_byfilter inconstructFilters(the tag is already written on every instance).createdBy: 'pool-lambda'when listing runners for the pool.Scale-down is unchanged: it still lists every instance of the environment, so scale-up launches are still reaped as before. The
runners_maximum_countcap on the pool top-up is also unchanged; the pool only sees fewer phantom runners and tops up accordingly.Test Plan
filters instances on the creating lambdainlambdas/libs/compute-providers/aws/ec2/src/runners.test.tsasserts thetag:ghr:created_byfilter is emitted whencreatedByis set.lambdas/libs/compute-providers/aws/ec2/src/control-plane/pool.test.tsupdated to expectcreatedBy: 'pool-lambda'on the pool'slistRunnerscall; the existingrunners_maximum_countcases still pass.yarn install --immutable && yarn format && yarn lint && yarn test && yarn buildinlambdas/on this branch (rebased onmain).Related Issues