Skip to content

Add HTTP span tags to the AWS Lambda invocation span - #12226

Open
claponcet wants to merge 11 commits into
masterfrom
clara.poncet/lambda-http-span-tags
Open

Add HTTP span tags to the AWS Lambda invocation span#12226
claponcet wants to merge 11 commits into
masterfrom
clara.poncet/lambda-http-span-tags

Conversation

@claponcet

@claponcet claponcet commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Computes the HTTP span tag set for every HTTP-shaped Lambda trigger from the event the tracer already parses for the WAF, and applies it at span creation. Follow-up to #12202, which did the refactor this builds on.

Tags written by the new LambdaAppSecHandler.applyHttpTags, transcribed from HttpServerDecorator.doOnRequest:

tag source
http.method the event's method (not set for WebSocket, which has none)
http.url URIUtils.buildURL(scheme, host, port, path) — no query string, see below
http.query.string rawQueryString on v2 and Function URL, the rebuilt query elsewhere, when isHttpServerTagQueryString()
http.useragent user-agent header
http.route v1 resource, v2 routeKey minus the method prefix, WebSocket routeKey
http.hostname x-forwarded-host, else the URL host
http.status_code the handler response's top-level statusCode, on the exit path

Supporting changes:

  • LambdaEventParser now extracts host (requestContext.domainName, else the Host header) and route, reads ALB headers from multiValueHeaders when headers is absent, and lowercases request header keys at extraction — as the response side and the Lambda Extension both already do. That last one also fixes a latent miss in extractHeadersWithCookies, whose "cookie" lookup never matched an API Gateway v1 style Cookie.
  • API Gateway v2 and Function URL payloads carry the request line verbatim in rawPath + rawQueryString, so it is used as-is rather than rebuilt from queryStringParameters, which comma-joins repeated keys (?a=1&a=2 would come back as a=1%2C2) and drops percent-encoded path segments. The other triggers expose no such field and keep the rebuilt path.
  • A trailing :port is stripped from the Host header. An ALB on a non-standard listener sends example.com:8080, and the port is tracked separately from x-forwarded-port, so URIUtils.buildURL would otherwise emit it twice. IPv6 literals are left alone.
  • LambdaURIDataAdapter.host() returns the event host instead of null; without it URIUtils.buildURL omits the authority and http.url would be a bare path. The scheme derived from x-forwarded-proto is now lowercased and whitelisted to http/https, and the default port follows the scheme rather than always being 443 — URIUtils.buildURL suppresses a port only for 80/http and 443/https, and compares the scheme exactly.
  • processResponseData sets the status through span.setHttpStatusCode so it serialises as a string, and sets the error flag from getHttpServerErrorStatuses(), matching HttpServerDecorator.doOnResponseStatus and the Extension's error=1 on a 5xx handler response.

Tagging is gated on AppSec being active, as processRequestStart already was. The tags are written onto the AppSec-derived TagContext before mergeContexts, so CoreTracer copies them onto the span at creation and nothing has to retain the parsed event across the customer's handler. No changes outside the datadog.trace.lambda package.

Motivation

AppSec Milestone 1 requires the invocation span to carry HTTP tags. Today the tracer sets none; everything on the final aws.lambda span comes from the Lambda Extension, whose coverage is uneven — no http.route for Function URL, WebSocket or ALB, no http.url for ALB, no query string anywhere, and a http.user_agent key that differs from Tags.HTTP_USER_AGENT. This also makes the tags correct in deployments with no extension in the path, such as a remote agent or the system-tests java_lambda weblog.

Additional Notes

  • For API Gateway REST and HTTP v2 the extension's trigger tags still overwrite http.url, http.method and http.route: its meta.extend runs at runtimeDone, after the tracer's meta is merged. Accepted for now — a follow-up extension change to stop overwriting tracer-supplied keys makes these win with no further tracer work. Until then the span also carries both http.useragent and http.user_agent.
  • Deliberately out of scope: span.kind: server (it changes stats aggregation and entry-span semantics, and the extension already sets it), and the client-IP tag group (http.client_ip, network.client.ip, peer.ipv4/ipv6/port), which would need ClientIpAddressResolver moved from agent-bootstrap to internal-api. Both are tracked as separate follow-ups.
  • The query string is not appended to http.url: QueryObfuscator.processTags obfuscates http.query.string and re-appends it, so doing it here too would double the query.
  • The route is set via Tags.HTTP_ROUTE rather than HttpResourceDecorator.withRoute, which would also rewrite the resource name and break the extension's dd-tracer-serverless-span placeholder filter.

Contributor Checklist

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@claponcet claponcet added type: feature Enhancements and improvements comp: asm waf Application Security Management (WAF) tag: ai generated Largely based on code generated by an AI or LLM labels Aug 17, 2026
@datadog-datadog-prod-us1-2

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.01 s 13.91 s [+0.1%; +1.4%] (maybe worse)
startup:insecure-bank:tracing:Agent 12.85 s 12.95 s [-1.4%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent 17.48 s 17.30 s [+0.1%; +1.9%] (maybe worse)
startup:petclinic:iast:Agent 17.37 s 17.50 s [-1.6%; +0.1%] (no difference)
startup:petclinic:profiling:Agent 17.26 s 16.66 s [-0.6%; +7.8%] (no difference)
startup:petclinic:sca:Agent 17.54 s 17.40 s [-0.2%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 16.52 s 16.70 s [-2.2%; -0.1%] (maybe better)

Commit: 4713302a · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@claponcet

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dc12bd5a6a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread dd-trace-core/src/main/java/datadog/trace/lambda/LambdaAppSecHandler.java Outdated
Comment thread dd-trace-core/src/main/java/datadog/trace/lambda/LambdaEventParser.java Outdated
Comment thread dd-trace-core/src/main/java/datadog/trace/lambda/LambdaEventParser.java Outdated
@claponcet
claponcet marked this pull request as ready for review August 18, 2026 09:10
@claponcet
claponcet requested review from a team as code owners August 18, 2026 09:10
@claponcet
claponcet requested review from jandro996 and mcculls August 18, 2026 09:10

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be7c5bacbf

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread dd-trace-core/src/main/java/datadog/trace/lambda/LambdaEventParser.java Outdated

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

Bracketed IPv6 Host values containing a non-default listener port retain that port, after which URL construction appends x-forwarded-port again. This produces malformed http.url tags for affected ALB invocations.

Open Bits AI session

🤖 Datadog Autotest · Commit be7c5ba · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@dougqh

dougqh commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

🤖 Claude (via automated review)

Non-blocking perf note, not a request for changes on this PR: LambdaEventParser.extractHeaders builds the raw header map via extractStringMap, then allocates a second HashMap just to lowercase the keys — a double allocation + double iteration per invocation.

On most request paths this would be noise, but Lambda invocation handlers run at whatever rate the function is invoked, so small per-call overhead compounds across the customer's traffic in a way a one-off request path doesn't. Worth a quick follow-up to fold the lowercasing into a single pass (e.g. build the lowercase map directly instead of copying), independent of this PR.

@dougqh dougqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted some performance concerns

But given that lambdas probably aren't as hot as conventional Java applications, I don't those are worth blocking on

Case-colliding header names consequently collapse in payload order rather than
in the intermediate map's hash order, matching the Lambda Extension.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@jandro996 jandro996 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of questions from a first pass over this - nothing that should hold up the PR, just curious what you think.

@jandro996 jandro996 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! just some questions

@claponcet
claponcet added this pull request to the merge queue Aug 19, 2026
@dd-octo-sts

dd-octo-sts Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Aug 19, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-08-19 13:38:10 UTC ℹ️ Start processing command /merge


2026-08-19 13:38:16 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).


2026-08-19 14:22:04 UTCMergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for b472c86:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

  • Investigate the failures and when ready, re-add your pull request to the queue!
  • If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
  • Any question, go check the FAQ.
Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 19, 2026
@claponcet
claponcet enabled auto-merge August 19, 2026 15:01
@claponcet
claponcet disabled auto-merge August 19, 2026 15:01
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@claponcet
claponcet requested a review from a team as a code owner August 19, 2026 15:13
@claponcet
claponcet requested review from AlexeyKuznetsov-DD and removed request for a team August 19, 2026 15:13
@claponcet
claponcet enabled auto-merge August 19, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: asm waf Application Security Management (WAF) tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants