feat: add lambda-runtime-invocation-id header - #1159
Conversation
b60f865 to
048c1e8
Compare
4c28d43 to
afb6708
Compare
afb6708 to
de1f2e4
Compare
c403f80 to
31c22d2
Compare
31c22d2 to
2cabdac
Compare
fc33a71 to
1152b7c
Compare
jlizen
left a comment
There was a problem hiding this comment.
Core approach is good, but some small tweaks.
Also: currently we log Lambda function timeout! for any 410, but we now will have a 410 if a stale response is rejected. We should tweak the message.
| let mut req = build_request().method(Method::POST).uri(uri).body(body)?; | ||
|
|
||
| if let Some(id) = self.invocation_id { | ||
| req.headers_mut().insert(LAMBDA_RUNTIME_INVOCATION_ID, id.parse()?); |
There was a problem hiding this comment.
This will error out and crash the runtime if a malformed invocation id header is sent.
We originally decode with String::from_utf8_lossy(), but that replaces non-utf8 bytes with U+FFFD which is anyway not ascii. So then this parse will fail.
I did a quick check and didn't find any other round trips, this new code is the only place impacted.
I know we control the sender but we should be defensive against malformed inputs anyway. I would suggest a rate-limited log warning if we have bad bytes.
There was a problem hiding this comment.
Done! I also added a general rate_limiter class. I made it very generic so it can be reused in the runtime.
We can probably open an issue to see if we can use it in other places. And evaluate integration with logging mechanism.
Another thing I would think about is to offer this capabilities to our reexported logging to offer the possibility for the customer to use our own logging and a very convenient rate limiter.
58592a7 to
bc1209f
Compare
bc1209f to
c09906e
Compare
c09906e to
eddf4c5
Compare
Summary
Add
Lambda-Runtime-Invocation-Idheader support for cross-wiring protection.The RIC now echoes the invocation ID received from RAPID on
/nextback on/responseand/error, enabling RAPID to detect and reject stale responses from timed-out invocations.Problem
On Lambda Managed Instances (LMI) and On-Demand (OD), when an invoke times out, the runtime process continues running in the background. If a new invoke arrives with the same
requestId, RAPID accepts it. The still-running old invocation eventually posts its response, and RAPID matches it to the new invoke — delivering the wrong response (cross-wiring).Solution
RAPID sends a unique per-invoke nonce via
Lambda-Runtime-Invocation-Idheader on/next. The runtime echoes it back on/responseand/error. RAPID validates the match before accepting the response.Backward Compatibility
Fully backward compatible in both directions:
Rate-limited malformed-header logging
Added an internal generic
RateLimiterfor runtime different uses. In this case the request was to rate limit warning caused by malformedLambda-Runtime-Invocation-Idheaders.The limiter:
Instantand configurableDuration.tracing, allowing a futuretracing-subscriberintegration without coupling the runtime behavior to a specific subscriber.Malformed invocation-ID headers are ignored rather than converted with
from_utf8_lossy, because the value may be echoed back as an HTTP header and must remain a valid ASCII header value. A rate-limited warning is emitted with the configured interval included as structured log metadata.Testing