Skip to content

fix(lambda): recognise AWS SDK errors that lost their name - #5578

Open
Brend-Smits wants to merge 1 commit into
mainfrom
fix/sdk-error-names-5565-5571
Open

Brend-Smits wants to merge 1 commit into
mainfrom
fix/sdk-error-names-5565-5571

Conversation

@Brend-Smits

@Brend-Smits Brend-Smits commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

In v8.0.0 the lambda bundles contained several copies of @smithy/core and @aws-sdk/core. The SDK registers service exceptions in a TypeRegistry owned by @smithy/core and looks them up from @aws-sdk/core, so with duplicate copies the lookup missed and service failures reached the lambdas as a plain Error: the name is Error and the code is only in Code, __type or the message. Every check keyed on error.name then missed the error. main and v8.1.0 no longer have the duplicates, but nothing in the code or CI protects against them coming back, and the checks only trusted name.

This PR makes those checks tolerant of that shape and adds a guard for the root cause:

A message only counts as a code when it is a single identifier, so ordinary messages such as Request failed with a socket hang up are not turned into failure codes, and unnamed AccessDeniedException, ThrottlingException or ParameterVersionNotFound errors are still treated as failures where they should be.

Test Plan

  • Added unit tests for the lost-name shape in the cache store, the config consumer, and EC2 create (CreateFleet, RunInstances dedicated host, AMI lookup, and the resulting retryable classification through scale_errors), plus unit tests for awsErrorCodes() at 100% coverage. 11 of the new tests failed before the fix and pass now.
  • The installed SDK no longer produces the exact v8.0.0 error shape, so these tests use hand-built errors in that shape. Two aws-ssm-util tests pin the error chain that getParameter really produces (GetParameterError with the SDK error as cause) so the fixtures match what the cache store sees.
  • The dependency guard passes on main and fails on the v8.0.0 lockfile (13 copies of @smithy/core and 10 of @aws-sdk/core reachable from the lambdas).
  • yarn format-check, yarn lint, yarn test (12 projects, uncached) and yarn build all pass in lambdas/.
  • Not deployed to a live environment from this branch.

Related Issues

Fixes #5571. Related to #5565: this adds the guard against duplicate SDK cores that it asks for, and makes the affected checks robust.

When the SDK error classes are not resolved (as with the duplicate
@smithy/core and @aws-sdk/core copies in v8.0.0), a service failure
reaches the lambdas as a plain Error: the name is "Error" and the code
is only in `Code`, `__type` or the message. Every check keyed on
`error.name` then missed it:

- the runner group cache store treated a missing cache parameter as a
  failure, so scale-up terminated the launched instance and retried
  forever instead of resolving the group from GitHub;
- request-level EC2 failures such as UnauthorizedOperation never matched
  scale_errors, so the message was dropped instead of retried;
- the SSM runner config consumer stopped polling on a missing parameter
  and did not retry throttled deletes.

Add awsErrorCodes() in a new aws-error-util lib, so storage-providers
and compute-providers share one way to read the code from the name,
`Code`, `__type` or a bare-code message along the cause chain. It is
used for these checks, and requestFailureCodes() now also records those
codes; existing codes such as aws-name:Error are unchanged.

Add a test that fails when more than one copy of @smithy/core or
@aws-sdk/core is reachable from the AWS clients, so a partial dependency
bump cannot reintroduce the duplicates.

Refs #5565, #5571

Signed-off-by: Brend Smits <brend.smits@philips.com>
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Dependency Review

The following issues were found:
  • ✅ 0 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ⚠️ 2 package(s) with unknown licenses.
See the Details below.

License Issues

lambdas/libs/compute-providers/package.json

PackageVersionLicenseIssue Type
@aws-github-runner/aws-error-utilNullUnknown License

lambdas/libs/storage-providers/package.json

PackageVersionLicenseIssue Type
@aws-github-runner/aws-error-utilNullUnknown License

OpenSSF Scorecard

PackageVersionScoreDetails
npm/@aws-github-runner/aws-error-util UnknownUnknown
npm/@aws-github-runner/aws-error-util UnknownUnknown

Scanned Files

  • lambdas/libs/compute-providers/package.json
  • lambdas/libs/storage-providers/package.json

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the changes addressing unnamed AWS SDK errors.

No open issues 🎉

What changed in this PR

Makes Lambda error handling resilient when AWS SDK errors lose their names, addressing runner-group cache failures in #5571 and missed EC2 retry classifications in #5565.

Changes:

  • Adds a shared helper that extracts AWS error codes from error fields and cause chains.
  • Uses derived codes for SSM cache misses, configuration polling and retries, and EC2 failure classification.
  • Adds regression tests and a dependency guard against duplicate AWS SDK core installations.
File Description
lambdas/​yarn.lock Registers the helper workspace and consumer dependencies.
lambdas/​libs/​storage-providers/​package.json Adds the shared error helper dependency.
lambdas/​libs/​storage-providers/​aws/​ssm/​runner-group-cache-store.ts Recognizes missing cache parameters despite lost error names.
lambdas/​libs/​storage-providers/​aws/​ssm/​runner-group-cache-store.test.ts Tests unnamed cache misses and propagation of other failures.
lambdas/​libs/​storage-providers/​aws/​ssm/​runner-config-consumer.ts Recognizes unnamed missing-parameter errors while polling.
lambdas/​libs/​storage-providers/​aws/​ssm/​runner-config-consumer.test.ts Tests unnamed polling, throttling, and access-error handling.
lambdas/​libs/​storage-providers/​aws/​ssm/​runner-config-consumer-common.ts Uses derived codes to identify retryable errors.
lambdas/​libs/​compute-providers/​package.json Adds the shared error helper dependency.
lambdas/​libs/​compute-providers/​aws/​ec2/​src/​runners.ts Adds derived AWS codes to request failure classifications.
lambdas/​libs/​compute-providers/​aws/​ec2/​src/​runners.test.ts Covers unnamed EC2 and AMI lookup failures, including retry classification.
lambdas/​libs/​aws-ssm-util/​src/​index.test.ts Verifies preservation of SDK error causes and messages.
lambdas/​libs/​aws-powertools-util/​src/​aws-sdk-single-instance.test.ts Guards against multiple reachable SDK core installations.
lambdas/​libs/​aws-error-util/​vitest.config.ts Sets full coverage thresholds for the helper.
lambdas/​libs/​aws-error-util/​tsconfig.json Configures the helper’s TypeScript source scope.
lambdas/​libs/​aws-error-util/​src/​index.ts Extracts error codes with bounded, cycle-safe cause traversal.
lambdas/​libs/​aws-error-util/​src/​index.test.ts Tests code extraction, input filtering, and traversal limits.
lambdas/​libs/​aws-error-util/​package.json Defines the helper workspace and development scripts.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants