Skip to content

feat: add lambda-runtime-invocation-id header - #1159

Open
darklight3it wants to merge 11 commits into
mainfrom
feat/add-runtime-invocation-id-header-support
Open

feat: add lambda-runtime-invocation-id header#1159
darklight3it wants to merge 11 commits into
mainfrom
feat/add-runtime-invocation-id-header-support

Conversation

@darklight3it

@darklight3it darklight3it commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Add Lambda-Runtime-Invocation-Id header support for cross-wiring protection.

The RIC now echoes the invocation ID received from RAPID on /next back on /response and /error, enabling RAPID to detect and reject stale responses from timed-out invocations.

Problem

On Lambda Managed Instances (LMI) and On-Demand (OD), when an invoke times out, the runtime process continues running in the background. If a new invoke arrives with the same requestId, RAPID accepts it. The still-running old invocation eventually posts its response, and RAPID matches it to the new invoke — delivering the wrong response (cross-wiring).

Solution

RAPID sends a unique per-invoke nonce via Lambda-Runtime-Invocation-Id header on /next. The runtime echoes it back on /response and /error. RAPID validates the match before accepting the response.

Backward Compatibility

Fully backward compatible in both directions:

  • If RAPID doesn't send the header → RIC doesn't see it → doesn't echo → no behavior change
  • If RIC doesn't echo it (old version) → RAPID skips validation → no behavior change

Rate-limited malformed-header logging

Added an internal generic RateLimiter for runtime different uses. In this case the request was to rate limit warning caused by malformed Lambda-Runtime-Invocation-Id headers.

The limiter:

  • Uses a monotonic Instant and configurable Duration.
  • Is protected by a standard mutex because the check-and-update operation must be atomic across concurrent Tokio tasks.
  • Is process-local, so a static limiter is shared across warm invocations in the same Lambda execution environment.
  • Resets its timestamp and clears mutex poisoning if the mutex is poisoned, since the limiter state is disposable and must not crash the runtime.
  • Keeps rate-limiting independent from tracing, allowing a future tracing-subscriber integration without coupling the runtime behavior to a specific subscriber.

Malformed invocation-ID headers are ignored rather than converted with from_utf8_lossy, because the value may be echoed back as an HTTP header and must remain a valid ASCII header value. A rate-limited warning is emitted with the configured interval included as structured log metadata.

Testing

  • added unit test
  • added a new dockerized test working with the new RIE.

@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from b60f865 to 048c1e8 Compare August 9, 2026 18:16
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from 4c28d43 to afb6708 Compare August 26, 2026 16:15
@darklight3it
darklight3it marked this pull request as ready for review August 26, 2026 16:18
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from afb6708 to de1f2e4 Compare August 26, 2026 16:23
@darklight3it darklight3it added the enhancement New feature or request label Aug 27, 2026
@darklight3it darklight3it self-assigned this Aug 27, 2026
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch 5 times, most recently from c403f80 to 31c22d2 Compare August 27, 2026 12:49
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from 31c22d2 to 2cabdac Compare August 27, 2026 13:11
Comment thread lambda-runtime/src/types.rs Outdated
Comment thread examples/invocation-id-concurrent/src/main.rs Outdated
Comment thread test/dockerized/scenarios/concurrent_scenarios.py
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from fc33a71 to 1152b7c Compare August 27, 2026 16:04

@jlizen jlizen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core approach is good, but some small tweaks.

Also: currently we log Lambda function timeout! for any 410, but we now will have a 410 if a stale response is rejected. We should tweak the message.

let mut req = build_request().method(Method::POST).uri(uri).body(body)?;

if let Some(id) = self.invocation_id {
req.headers_mut().insert(LAMBDA_RUNTIME_INVOCATION_ID, id.parse()?);

@jlizen jlizen Aug 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will error out and crash the runtime if a malformed invocation id header is sent.

We originally decode with String::from_utf8_lossy(), but that replaces non-utf8 bytes with U+FFFD which is anyway not ascii. So then this parse will fail.

I did a quick check and didn't find any other round trips, this new code is the only place impacted.

I know we control the sender but we should be defensive against malformed inputs anyway. I would suggest a rate-limited log warning if we have bad bytes.

@darklight3it darklight3it Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! I also added a general rate_limiter class. I made it very generic so it can be reused in the runtime.

We can probably open an issue to see if we can use it in other places. And evaluate integration with logging mechanism.

Another thing I would think about is to offer this capabilities to our reexported logging to offer the possibility for the customer to use our own logging and a very convenient rate limiter.

Comment thread examples/invocation-id-concurrent/src/main.rs
Comment thread test/dockerized/scenarios/concurrent_scenarios.py Outdated
Comment thread test/dockerized/scenarios/concurrent_scenarios.py Outdated
Comment thread lambda-runtime/src/requests.rs
Comment thread lambda-runtime/src/layers/api_response.rs
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch 2 times, most recently from 58592a7 to bc1209f Compare August 31, 2026 08:01
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from bc1209f to c09906e Compare August 31, 2026 08:06
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from c09906e to eddf4c5 Compare August 31, 2026 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants