Skip to content

feat(observability): correlate HTTP receipts, OCSF and request traces - #5214

Merged
krishicks merged 7 commits into
NVIDIA:mainfrom
afourniernv:codex/relay-state-01-correlation-primitives
Nov 17, 2026
Merged

krishicks merged 7 commits into
NVIDIA:mainfrom
afourniernv:codex/relay-state-01-correlation-primitives

Conversation

@afourniernv

@afourniernv afourniernv commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Connect inspected HTTP responses, supervisor traces and OCSF records using the OpenShell-generated request ID. Keep each control's original audit record instead of combining records into a final summary.

Related Issue

Closes #5393. Tracking: #5210. Supervisor sampling remains separate in #5362.

Changes

  • Correlate all OCSF event classes at emission. Preserve each control's fields, event ID, classification and timing, with an event kind and request-local sequence.
  • Return the protected X-OpenShell-Request-ID header without adding receipt fields to JSON error bodies. Strip upstream receipt headers and trailers consistently; middleware cannot change the receipt.
  • Keep supervisor-owned request/stage spans with explicit outcomes and bounded failure reasons, independent of audit emission. Workload W3C context is an untrusted link, not a parent; inspected requests create no extra denial-only spans.
  • Distinguish handled middleware failures from policy refusals in stage spans, including local 502 responses that were successfully delivered.
  • Preserve request attribution through terminal errors and active policy-reload cancellation. The existing 101 handoff record joins its handshake; protocol-start work and later WebSocket frames do not inherit the HTTP scope.
  • Create forward-proxy correlation from the method and raw headers without adding a strict query parse. Initialize the emission hook at shared proxy startup, covering sandbox and host-proxy deployments.
  • Remove primary-record merging, fallback summaries, late-record suppression, general response-head validation and the unused correlation setter.

The public Rust addition is an emission enrichment hook. No gateway RPC, Protobuf field, configuration option or Relay dependency is added.

Release Notes

Receipts are join keys, not authorization or proof of execution or complete delivery. Records and request ID joins work without exported spans. L4 and uninspected traffic do not provide the protected-receipt guarantee.

Inspected request spans include allowed traffic at INFO level and are outside the denial-span budgets. Query and collector-capacity guidance is documented; this PR does not add supervisor sampling.

Testing

  • Network Nextest: 1,723 passed; seven skipped. A fresh-process regression verifies shared production startup before any fixture can register the emission hook.
  • Full mise run test passed on the final source, including Rust workspace tests, gateway tests with test-support, doctests, Python, TypeScript and tooling tests, with four Rust test threads.
  • All five CI example/test-tooling Nextest suites passed: 94 tests, three skipped. Conformance test-list consistency passed.
  • Full mise run pre-commit passed, including workspace, feature and external-manifest strict Clippy; the commit hook passed again.
  • Docs checks: zero errors; three existing Fern warnings.
  • Deterministic regressions cover explicit-forward malformed queries, successful keep-alive request followed by a parse rejection across all OCSF classes, active/idle/between-request reloads, terminal errors, WebSocket scope separation, receipt trailers, SigV4 and forward-credential stage results, and sampling-independent correlation.
  • Request/response middleware regressions check operational stage errors, valid policy refusals, and successful local error delivery. Request fixtures pin the injected runner before dispatch and verify the actual failure diagnostic.
  • Real shared-proxy exchanges exercise the original FORWARD, plaintext CONNECT and TLS CONNECT terminal-error emitters. WebSocket allow/deny tests verify the handoff receipt, scope isolation and no denied text reaching upstream. No new frame-denial spans are introduced.

Earlier live qualification on 54a11a15d passed eight genuine Hermes/Relay conversations: REST/MCP allow/deny, request/response redaction, response withholding, two-turn recovery, interrupted delivery, authenticated read-only GitHub access, and correlation without supervisor trace export. Verified 207 agent spans, 34 receipts and 308 original per-control records. The current follow-up is covered by the rebuilt Nextest suites above, not a new live qualification.

The explicit-forward query fix has Rust coverage. A separate transparent malformed-query probe disconnected before receipt allocation and is not covered by that fix. WebSocket/trailer and reload boundary cases have Rust coverage, not live Hermes coverage.

The first full test attempt exhausted host disk space in unchanged Vault tests. Clearing the previously approved regenerable compiler cache allowed the unchanged full command to pass.

Checklist

  • Conventional commits and DCO sign-offs.
  • Crate and published observability guidance updated.
  • Maintainer release compatibility decision.

@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@sftimeless

Copy link
Copy Markdown

Hey, thanks for working on this.

The separation between correlation context and authorization identity is important.

I've been testing Keel as an external pre-execution authorization service through OpenShell's Supervisor Middleware, and this connects directly to an issue we encountered during our v0.1.3 qualification.

In that experiment, the adapter could associate RequestContext.request_id with a signed Keel permit, but the corresponding supervisor OCSF event did not carry that identifier. We documented the ALLOW/DENY examples in #4557.

I see this PR introduces OcsfEvent::set_correlation_uid and the OpenTelemetry correlation helpers, while leaving live HTTP enforcement and middleware integration for subsequent work.

For that next integration step, would the intended approach be to propagate the supervisor-generated RequestContext.request_id into metadata.correlation_uid on the actual middleware enforcement events, as proposed in #4557?

That would let an external authorization provider join:

  • Its signed authorization decision and permit ID
  • The middleware request ID
  • OpenShell's observed enforcement result

It would also be useful to preserve that identifier across request and response evaluation, including denials and middleware failures.

I'm not suggesting that correlation alone proves exact-request binding or downstream effects. Those remain separate guarantees.

Happy to help validate the eventual runtime integration against our existing sanitized test cases if helpful.

@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@afourniernv afourniernv changed the title feat(observability): add enforcement correlation primitives feat(observability): correlate HTTP enforcement with traces and OCSF Nov 6, 2026
@krishicks

krishicks commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Written by Claude Code, reviewed by @krishicks

Superseded by this comment. The request spans can stay in this PR; what matters is that no span exists only because of a denial, and that nothing depends on spans being exported. The question below about the request ID join without a sampled span still stands.

A question about span volume. Every inspected request gets openshell.http.request and openshell.policy.evaluate, and the queued layers add more INFO children. Supervisors use the SDK's default parentbased_always_on sampler, because OTEL_TRACES_SAMPLER doesn't reach them, and an unsampled workload traceparent can't suppress these spans. So every inspected request is exported. Inspected denials also move from supervisor.egress.deny, which #4789 bounded with per-destination and process-wide budgets, to request spans with no bound.

We see supervisor traces as a tool for latency and errors, sampled like any other service, rather than a full record of every request. OCSF already provides that record, and request rates belong in metrics. Sampling for supervisors is tracked in #5362, and supervisor metrics in #5364. In the meantime, we'd like these spans not to assume they're always exported. In particular, could you confirm that the request ID join through metadata.correlation_uid and the response header works with no sampled span? Also, when supervisor.egress.deny gives way to request spans, the request spans should stay inside whatever bound #4789 provides until sampling exists.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

Successful HTTP FORWARD requests without an L7 route now emit a fallback HTTP Activity record instead of the existing Allowed/Success record with the requested path and process.

  • Impact: OCSF consumers filtering action=Allowed or attributing requests through actor.process lose successful requests and their original request details.
  • Action: Populate the correlated FORWARD record with the existing action, status, path, process and endpoint fields before response-head emission.
  • PR: feat(observability): correlate HTTP receipts, OCSF and request traces #5214

2. api-behavior — No explicit risk acceptance found

x-openshell-request-id previously passed through HTTP forwarding and allowed middleware mutations; the candidate strips incoming values, replaces upstream values and rejects mutations.

  • Impact: Existing middleware writing or removing this header fails closed by default, while applications using it for end-to-end request correlation lose their identifiers.
  • Action: Keep legacy header handling on v0.1.8 and make protected receipts opt-in, documenting a separate header for application-owned correlation.
  • PR: feat(observability): correlate HTTP receipts, OCSF and request traces #5214

3. api-behavior — No explicit risk acceptance found

Inspected HTTP denials replace bounded supervisor.egress.deny spans with unbounded openshell.http.request and policy spans that can inherit workload traceparent.

Review details
  • Execution: complete
  • Candidate: PR #5214 (37a7bef8c689b4e3ea34b2f839fa2bd70f306472)
  • Baseline: 9321d1632a41cd7337db38ff982164bff9120d08 (9321d1632a41cd7337db38ff982164bff9120d08)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@krishicks krishicks left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Written by Claude Code, reviewed by @krishicks

Reviewed at 7e8f7ae. Each point starts with what a user would see; the cause and file references follow. Line numbers refer to that commit.

Audit records

  1. A SIEM shows requests the supervisor blocked as allowed. A request blocked by a middleware failure (403), admission exhaustion (503), a WebSocket preflight error (502), or a credential preparation failure (502) gets one HTTP Activity record that says Allowed, with no failure status.
    • Cause: emit_primary reclassifies the record only when the outcome is denied (l7/correlation.rs:212-220). The platform_error and upstream_error outcomes keep the allow record (relay.rs:900-904, relay.rs:1823-1855).
  2. Allowed plain-HTTP proxy requests no longer say which process made them. On main, every allowed FORWARD request without L7 inspection is recorded with its path, binary, PID, and cmdline. With this PR, it gets a placeholder record: "HTTP request ended before a response was selected", URL /, no action, no process.
    • Cause: the correlation is created up front (proxy.rs:5921-5926), so the empty slot is emitted at the response head. The real allow record arrives after the relay (proxy.rs:7228) and is dropped (correlation.rs:174).
  3. Requests rejected before policy runs are reported as policy denials. An Upgrade refusal, a malformed request target, or an invalid GraphQL or JSON-RPC body produces "HTTP request blocked after policy admission", URL /, and a span with policy.decision=deny. Policy never ran on these requests, so an operator investigating policy is sent to the wrong place.
    • Cause: these paths emit only a Network Activity record, so the placeholder record is emitted and relabelled as a denial (relay.rs:1310-1335, relay.rs:1550-1665, relay.rs:3344-3348).
  4. Some requests still produce extra HTTP Activity records that can't be joined by request ID, and one produces two records that disagree. This happens to any request with response middleware, a request-middleware denial, or a token-grant refusal. A token-grant refusal gives one record saying Denied and another saying Allowed with a 502. Denials on these paths also open a separate supervisor.egress.deny span next to the request span.
    • Cause: these paths still emit through telemetry::emit_decision without correlation_uid (http_response.rs:698-1251, middleware.rs:1023, token_grant_injection.rs:344). The request ID is already available at the call sites (relay.rs:1810, relay.rs:2655, relay.rs:3138).
  5. When credential resolution or MCP reinspection rejects a request, the record no longer shows which rule matched. The surviving record loses the request path, policy rule, and reason. A problem after the response headers are sent produces no correlated record at all.
    • Cause: emit() replaces the primary record with any later HTTP Activity, and silently drops one that arrives after the head is written (correlation.rs:166-176).

Trust and sampling

  1. A sandboxed workload decides which trace ID appears on supervisor audit records. If it sends a sampled traceparent, the HTTP Activity record's trace.uid is the workload's trace ID. A workload could point its denials at an unrelated trace. Once #5362 adds a sampling ratio, it could also force its request spans to be exported regardless of that ratio.
    • Cause: a sampled workload context becomes the parent of openshell.http.request (propagation.rs:169-174). #4789 documents that workload headers never supply supervisor correlation, and #2640 keeps workload context in a separate field.
    • Linking sampled context the way unsampled context is already linked would keep the connection to the agent's trace without either effect.

Behavior changes beyond correlation

These change what clients see. Could they move to a separate PR, or at least be listed in the PR description?

  1. Agent HTTP clients stop retrying, so a routine connection close becomes a failed request. When an upstream closes an idle keep-alive connection, main closes the client connection. Go, urllib3, and curl retry that automatically. With this PR the client gets a 502 and surfaces the error.
    • Cause: a synthetic 502 on upstream EOF before headers (http_response.rs:100-110).
  2. Some upstream APIs that work today stop working. For example, a server that sends HTTP/1.1 200 without a reason phrase now has its response rejected, and the agent sees the connection close. The audit record still reports a 502.
    • Cause: new strict response-head validation that applies whether or not a request is correlated (http_response.rs:425-440). httparse accepts a missing reason phrase.
  3. Agents get a 500 response where the connection used to close. This happens on a missing credential or a stale credential generation.
    • Cause: relay.rs:1064-1080, proxy.rs:6851-6880.

Spans

  1. Traces don't show where an inspected request's time went. Only policy evaluation appears under openshell.http.request. Middleware calls, credential preparation, and the upstream dial are not children of the request span, and remote middleware receives the connection's trace context instead of the request's.
  2. WebSocket requests show up as a single span lasting the whole session. The root span stays open until the upgraded connection closes.
  3. The span marks an error but not which stage failed. record_error_stage ignores its stage argument (correlation.rs:162).

Tests

Most of the record problems above would have been caught by a test asserting exactly one HTTP Activity per request. The dispatch tests use .find() instead. These paths have no coverage:

  • placeholder records and rejections before policy runs
  • the content of a non-L7 FORWARD record
  • token-grant refusal
  • request-middleware denial
  • upstream EOF before headers
  • a second request on a keep-alive connection, which should get a new ID

Process

The branch conflicts with main.

@krishicks

krishicks commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

This is what the robot drew when I asked for a before/after with this PR:

5214-before 5214-after

@krishicks

krishicks commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Written by Claude Code, reviewed by @krishicks

Superseded by this comment. The request spans don't need to wait for #5362, and the diagram below is out of date.

A follow-up connecting the review with my earlier comment on span volume. The compatibility review's item 3 raises the same concern: span volume. Review point 10 asks for a child span per stage, which adds spans. Nothing bounds that until #5362 adds supervisor sampling, and the #4789 deny-span limiter isn't the right tool, since #5362 removes it.

Could the request spans land with or after #5362? This PR would keep the request ID, correlation_uid, the response header, and the primary record. Those don't depend on spans. That also needs the request ID join to work with no sampled span, which the earlier comment asked you to confirm.

Then the after diagram would look like this:
5214-after-review

Copy link
Copy Markdown
Contributor

Drafted with Codex; reviewed and approved by @HarryMoss.

Hi @afourniernv, I’ve read Kris’s latest review and the proposed split between the correlation core and request spans.

My local #2640 connection-startup prototype is based on #5214 at 7e8f7aec and touches the shared CONNECT/proxy and telemetry paths. I’ll keep the L4 span work local pending #5362.

Once you’ve settled the split, which revision should I track for those shared paths so I can avoid duplicating your fixes?

Context: #2640 (comment)

@afourniernv afourniernv changed the title feat(observability): correlate HTTP enforcement with traces and OCSF feat(observability): join HTTP receipts with OCSF records Nov 10, 2026
@krishicks

krishicks commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Drafted by 🤖, signed off by @krishicks

A correction to my last comment: the request spans don't need to wait for #5362. They can stay in this PR, along with per-stage children (review point 10). The concern is spans that exist only because a denial happened. A request span that records every outcome is the right shape.

Two requirements apply whether or not sampling exists:

  • No span exists only because of a denial. On inspected paths, that means no supervisor.egress.deny span next to the request span for middleware denials, response-middleware blocks, or token-grant refusals (review point 4).
  • Nothing depends on spans being exported. Records, enforcement, and the request ID join have to work the same whether a request's span is sampled or not.

The updated diagram:

5214-after-review

@HarryMoss

HarryMoss commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Drafted by Codex; @HarryMoss reviewed

Thanks @afourniernv for the updates. I reran the two regressions from my earlier review on ad6040b8. Twenty-nine focused existing controls passed; both added regressions still reproduce:
Thanks @afourniernv for the updates. I reran the two regressions from my earlier review on ad6040b8. Thirty-nine focused existing controls passed, including the real FORWARD sampling/export matrix and WebSocket trace-lifetime checks; both added regressions still reproduce:

  • The truncated Content-Length response returns the expected relay error, but its request span finishes as allowed, with no error status.
  • The body-credential refusal returns its 403 receipt, but the companion Network Activity and credential finding still lack metadata.correlation_uid.

I have the small test-only patch ready to share. This is local macOS evidence, rather than a fresh deployed validation.

The compatibility advisory also matches what I see in the successful middleware merge path: routine invocation details are dropped rather than retained on the primary record. Would you prefer retaining those audit details, or documenting that loss as an explicit compatibility decision? That question follows Kris's guidance that audit records and joins work independently of trace export.

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

Successful token grants previously emitted one HTTP Activity record per credential; inspected requests now merge these records into the existing request decision without retaining their messages.

Review details
  • Execution: complete
  • Candidate: PR #5214 (09ee9b53aabd0a7144aa74e8ed9c83aff0cbd97a)
  • Baseline: 622a15705ec15fd816fb6a1aefa4581baa23a3e0 (622a15705ec15fd816fb6a1aefa4581baa23a3e0)
  • Reviewed: rust-sdk, python-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

Closes NVIDIA#5393

Refs NVIDIA#5210

Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv
afourniernv force-pushed the codex/relay-state-01-correlation-primitives branch from 11030e5 to 0f92636 Compare November 16, 2026 01:13
Signed-off-by: Alex Fournier <afournier@nvidia.com>
@krishicks

Copy link
Copy Markdown
Collaborator

Drafted by 🤖, signed off by @krishicks

@afourniernv I'd like to propose a simpler shape for this PR: keep one HTTP Activity record per control rather than merging them into one primary record per request. This changes my earlier position on #5210.

Merging is where the bugs are. Re-reviewing d0bdbe9, most of what remains comes from merging records into one primary: a mid-body truncation recorded as a clean allow, timeouts and transport errors recorded as Denied, granted credential identities lost, and later problems dropped after the first. The compatibility review's findings come from the same place. Each fix so far has added more seed, merge and overlay rules, and every new stage would need its own.

We may prefer the granularity anyway. With a record per control, the records for a request show its path through enforcement: policy Allowed under github-api, pii-redact Modified, content-guard Denied, and nothing after. That shows which control decided, which controls let the request through first (a sign policy may be too permissive), what was modified on the way, and how far the request got. A merged record keeps only the final outcome. OCSF fits this model: metadata.correlation_uid exists "to correlate this OCSF event with other related OCSF events", and the Security Control fields describe the action of one control.

Proposed scope:

  1. Request ID scope and receipt header. Stamp metadata.correlation_uid on every record emitted while handling the request, from a scope set at the emit path the way feat(ocsf): correlate audit events with sampled OpenTelemetry traces #4789 fills trace.uid, so no call site can miss it (this fixes Harry's body-credential case by construction). Add the protected X-OpenShell-Request-ID header using the existing response-head handling. Tag each record with its stage (decision, middleware invocation, token grant) and give it an order, so consumers can count requests by their decision record and read the trail in sequence. Records otherwise keep main's content, timing and classification.
  2. Request and stage spans, in this PR or the next: a supervisor-owned root with a child per stage that records its outcome, the workload traceparent linked, no denial-only spans, and no change to records.

That still delivers #5393: from a response, an agent or operator can find every record about the request, with tracing on or off. It also removes the merge machinery, the record-replacement compatibility concerns, and the new response-head validation.

What it gives up is a single record carrying both the decision and the status the client received. If we find we need that later, we can agree an outcome classification table first and add it as a separate step.

Here's how the features compare:

5214-feature-split

Correlate original control records at emission with a request-scoped ID and sequence. Remove primary-record aggregation and broad response-head validation while retaining protected receipts and request tracing.

Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv

Copy link
Copy Markdown
Contributor Author

@krishicks Agreed. Updated in 7778e71. Records now stay separate and keep their original fields and timing. The shared request ID and sequence connect them. I removed the record-merging code, fallback summaries and general response-header validation, and kept the request/stage spans.

Network and OCSF tests, Clippy and pre-commit pass. The full suite hit the existing VM cleanup failure; its isolated rerun passed. Could you take another look?

@krishicks

Copy link
Copy Markdown
Collaborator

Drafted by 🤖, signed off by @krishicks

Re-reviewed at 7778e71 against the split proposal. This is the shape we asked for: one record per control with main's content, timing and classification, the request ID stamped at emission, and no new response-head validation. That resolves most of the d0bdbe9 items:

  • No more truncation-as-allow or timeouts-as-Denied.
  • Body-credential companions are correlated.
  • Grant identities are back.
  • Late records are no longer dropped.
  • Upstream heads that work on main still work.

What remains, each starting with what a user would see:

Before merge

  1. A forward-proxy request with an undecodable query leaves no audit record. GET http://host/path?q=%ff through the plain-HTTP forward path now drops the connection with no response and no L7 deny record. On main the query was decoded tolerantly, so the 403 and deny record were still produced. Since the agent controls the query string, it can probe denied endpoints without a trace in the audit log.
    • Cause: proxy.rs:5978-5983 builds an L7Request with a strict query parse (rest.rs:383) and ? for every forward request, only to create the correlation. RequestCorrelation::build needs just the method and raw header.
  2. Failures that end a request can't be found by its request ID.
    • "L7 relay error" / "TLS relay error" (proxy.rs:3343+, 3414+) and the policy-reload close are emitted after the relay future returns or is cancelled, so the request scope is already gone.
    • The request span ends incomplete with no reason, so neither the records nor the trace join the failure to the receipt.
    • Emitting these inside the scoped loop, and selecting on staleness inside it, would cover them.
  3. WebSocket frame denials after an upgrade get no span and no trace.uid.
    • On main (via feat(ocsf): correlate audit events with sampled OpenTelemetry traces #4789) they got a deny span.
    • Now finish_http_trace() (relay.rs:2933, 4554, and the forward path) ends the request span, but the scope stays installed for the whole handle_upgrade session. Frame records from websocket.rs therefore go through emit_inspected_decision with an empty span.
    • Every frame also shares the upgrade's request ID tagged decision, which conflicts with the docs' advice to group by correlation_uid when counting policy outcomes.
    • Either clear the scope at the upgrade, so frames are handled like other uninspected egress, or document that frames belong to the request and tag them distinctly.

Worth fixing, not blocking

  1. Some records carry the wrong stage tag.
    • WebSocket middleware invocations (middleware.rs:578-655) are tagged decision, not middleware_invocation.
    • The 101 upgrade record and stale-policy closes also default to decision, as do Detection Findings. The docs then have to warn readers not to count those.
    • Tagging findings and lifecycle records with their own kinds would let "count decision records" mean what it says.
  2. Traces still don't say which stage failed.
    • otel.status_code is declared on stage spans but never recorded (request_trace.rs:101).
    • Most PlatformError rejections pass no reason (e.g. relay.rs:2774, 2812, 4507, proxy.rs:6705, 6745).
    • Only denied is sticky (request_trace.rs:112), so a later Allowed can overwrite failed.
  3. An upstream trailer named X-OpenShell-Request-ID fails the response on the response-middleware path (http_response.rs:2381). The non-middleware path strips it (rest.rs:3753-3760).
  4. Supervisor JSON error bodies gain a request_id field. This is new vs main and not in the description. The response-middleware 403 body (http_response.rs:2419-2437) doesn't have it. Either drop it, since the receipt header carries the ID, or add it everywhere and list it in the release notes.
  5. Cleanup:
    • register_event_enricher runs on every request and ignores the result (correlation.rs:139). If anything else registers first, correlation stops silently. Registering once at supervisor startup and asserting the result would avoid that.
    • OcsfEvent::set_correlation_uid is public but only tests use it.
    • Most correlation: Option<&RequestCorrelation> parameters and _correlated twins now only feed observe_http, which duplicates the explicit record_* calls.

Tests: the keep-alive test filters to HTTP Activity (correlation_dispatch_tests.rs). Nothing asserts that the following carry no request ID:

  • records emitted between requests (parse rejections, relay errors, stale closes)
  • frames after an upgrade

Description and docs:

  • The PR body still describes the primary-record design ("Emit one primary HTTP decision…") and its release notes.
  • ocsf-json-export.mdx still describes the change rather than current behavior ("Keep the legacy span query", "The old deny-span budgets…"). Once released, it would read better describing only what the supervisor does now.

Avoid parsing queries solely for correlation. Carry request attribution through terminal errors and policy reloads, and clear HTTP scope at WebSocket upgrade.

Preserve per-control audit records while reporting bounded stage outcomes and filtering receipt trailers consistently. Remove JSON receipt additions and the unused correlation setter.

Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv

Copy link
Copy Markdown
Contributor Author

@krishicks Updated in 54a11a1. Removed the extra query parse, kept request attribution through errors and policy reloads, and cleared the HTTP scope at WebSocket upgrade. Also fixed event/stage reporting and receipt trailers, and removed the unused setter and JSON receipt fields. Audit records still keep their original fields and timing.

Pre-commit and the full test command pass. The final rebuilt network suite passes 1,710 tests, and eight live Hermes/Relay runs passed, including correlation without supervisor trace export.

The explicit-forward query regression, WebSocket/trailer edges and reload boundaries have Rust coverage. The separate transparent-query behavior is unchanged.

Ready for another look.

…ures

Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv

Copy link
Copy Markdown
Contributor Author

@krishicks Updated in c871af5. Fixed the four Nextest failures by initializing the emission hook in the direct-response fixtures. Also finished stage status/reason handling, removed audit-derived trace state and redundant helpers, and added the keep-alive parse-rejection regression.

The full workspace Nextest rerun passes 9,150 tests with four workers. The independent network suite, five additional test suites, Clippy/pre-commit and docs also pass. The initial unrestricted run hit two timeouts in unchanged VM tests; both passed alone and in the full rerun.

CI is waiting for the mirror to advance from 54a11a1. Could you run /ok to test c871af57862f895aa75aff642cc0ad0eb069f2d4?

@krishicks

krishicks commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Drafted by 🤖, signed off by @krishicks

Re-reviewed at c871af5 against the previous review. All three blocking items are fixed:

  • The forward-path malformed query again gets main's 403 and deny record.
  • Relay errors and policy-reload closes carry the request ID without leaking between requests.
  • The scope is cleared at the WebSocket upgrade, so frame denials get their deny span again.

Stage tags, stage span status and reasons, receipt trailers, JSON bodies and the cleanup items are addressed too. Nothing here blocks merge. The remaining items, each starting with what a user would see:

  1. On the host proxy, records lose the request ID while clients still get the header.
    • The enricher is now registered only in run_networking (run.rs:301), and start_host_proxy (host.rs:85) never calls init_request_correlation(). In 7778e71, per-request registration covered this path.
    • The test fixtures register the enricher themselves, so the tests don't catch it.
    • Calling init_request_correlation() in ProxyHandle::start_with_bind_addr would cover both entry points.
  2. The record saying L7 enforcement has stopped can't be joined to its handshake. "101 Switching Protocols — raw bidirectional relay (L7 enforcement no longer active)" is emitted in handle_upgrade (relay.rs:2328), after finish_http_request() clears the scope. Emitting it before the scope is cleared would keep it with its request.
  3. Forward-path connection errors don't carry the request ID. "Proxy connection error" (proxy.rs:592) still uses egress_emit!, so a captured forward error (proxy.rs:7274) loses its ID. emit_error_event would cover it, matching the relay paths.
  4. On the forward path, a middleware failure is reported as a denial in traces. proxy.rs:6696 records request_middleware_denied for failures, while the relay paths use request_middleware_failed.
  5. A request whose middleware fails closed has a request-middleware stage span that shows OK, while the request span shows failed. The fail-closed result returns Ok(Denied), so the stage span doesn't record the error.
  6. Tests:
    • oversized_response_header_keeps_request_attribution_after_dispatch_returns_error rebuilds the "L7 relay error" record instead of going through proxy.rs:3359. Switching that site back to egress_emit! would go unnoticed.
    • No end-to-end test checks that a denied frame after an upgrade has a deny span and no correlation_uid.
  7. Docs:
    • ocsf-json-export.mdx says kinds "distinguish decisions, findings, connection events and middleware invocations" but doesn't list the literal http_event_kind values.
    • A few phrases still describe the change rather than current behavior: "remain unchanged" (ocsf-json-export.mdx:132, README), "Other egress paths retain the bounded deny-span behavior" (README).
    • middleware_phase is described as retained, but it is new here (main has unmapped.phase).
5214-c871af578

@krishicks

Copy link
Copy Markdown
Collaborator

/ok to test c871af5

Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv

Copy link
Copy Markdown
Contributor Author

@krishicks Fixed the handled-failure reporting gap in 57ec95c. Request and response middleware failures now mark the middleware stage as ERROR even when a local error response was delivered successfully. Valid policy refusals stay OK. HTTP responses and original OCSF records are unchanged.

The full workspace Nextest run passes 9,152 tests; the final network run passes 1,718. The additional suites and Clippy/pre-commit also pass. Request fixtures now pin the injected runner correctly and distinguish actual mixed-hook failures, missing services and valid denials.

Could you run /ok to test 57ec95ce077aa0a933e8ee753c5a7c0918133af0 for the updated head?

Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv

Copy link
Copy Markdown
Contributor Author

@krishicks Addressed the remaining follow-ups in d6784dd: shared host/sandbox proxy startup, forward terminal-error attribution, the 101 handoff receipt, consistent middleware failure reasons and field documentation.

The new regression goes through the actual FORWARD, plaintext CONNECT and TLS CONNECT error emitters. WebSocket tests verify the handshake join, cancellation and frame isolation. One clarification: frame denials are Network Activity records; that path does not create HTTP denial spans today. I preserved that behavior rather than adding new spans.

Full mise run test, pre-commit/strict Clippy and docs checks pass. The independent network Nextest run passes 1,723 tests. The first full run hit disk exhaustion in unchanged Vault tests; clearing the previously approved compiler cache allowed the unchanged command to pass.

Could you run /ok to test d6784dd223d3421727055e5e01f5624500930354 for the updated head?

@krishicks

Copy link
Copy Markdown
Collaborator

/ok to test d6784dd

@krishicks
krishicks added this pull request to the merge queue Nov 17, 2026
Merged via the queue into NVIDIA:main with commit 8a20212 Nov 17, 2026
92 checks passed
@HarryMoss

Copy link
Copy Markdown
Contributor

Thanks @krishicks and @afourniernv — good to see this merged. Following the final rework, I built merged commit 8a20212 and ran a local Linux ARM64 qualification using the production standalone network-proxy role, Collector 0.164.0 and Jaeger 2.22.0.

The final run passed 21 wire scenarios and 195 assertions. All 49 retained OCSF records passed schema validation, and all 12 captured request traces were retrieved from Jaeger, including the spans referenced by the native OTLP log envelopes. Logs were retained through the Collector’s file sink.

The controls covered allowed and denied HTTP requests, HTTP/Finding companions for an authority mismatch, truncated upstream responses, untrusted workload trace context and standalone CONNECT refusals. With trace export disabled, enforcement, request receipts and per-control audit joins remained intact. A brief Collector outage also preserved the wire behaviour; this wasn’t a sustained backpressure or delivery-durability test.

Separately, two diagnostic gaps reproduced in test-only probes on d6784dd on both macOS ARM64 and Linux ARM64. The relevant code is unchanged in the merge:

  • A scripted unavailable grant resolver returns 502 and sends zero bytes upstream, but openshell.credential.prepare exports Unset without a failure reason.
  • An in-memory broken-pipe write through the production live-upload adapter sets its failure flag, but openshell.upstream.send exports Unset without a failure reason. The successful-write control passed.
    Would a small follow-up recording ERROR and a bounded reason on those children be useful? The live relay can still deliver a valid early upstream response, so its overall outcome should remain the relay’s decision. I have test-only reproductions for both cases.

For #2640, I propose adapting onto the merged foundation, preserving HTTP-aware terminal-error attribution and ending CONNECT/transparent startup ownership after the startup audit, before the long-lived relay.

This qualifies the local standalone proxy pipeline, not a full managed sandbox, Tempo or real-provider workflow. The #2640 adaptation hasn’t been tested yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(observability): join inspected HTTP receipts with OCSF records

4 participants