Skip to content

fix(proxy): handle upstream HTTP connection closure - #5335

Open
shiju-nv wants to merge 2 commits into
NVIDIA:mainfrom
shiju-nv:fix/5322-upstream-idle-close/shiju-nv
Open

shiju-nv wants to merge 2 commits into
NVIDIA:mainfrom
shiju-nv:fix/5322-upstream-idle-close/shiju-nv

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Retire HTTP connections promptly when the upstream closes while idle, including JSON-RPC and MCP. For relay-owned upstream failures, return a correlated 502 Bad Gateway with Connection: close before any final response bytes are written to the client; close without appending another status once a final response has started. Apply the same error handling to plain forward HTTP and CONNECT tunnels.

Related Issue

Closes #5322

Scope follows the updated issue criteria and the issue discussion. Automatic reconnect/replay remains outside this fix. Explicit gateway errors apply to every HTTP method, including GET; applications may need to retry a 502 because some clients only retry transport-level failures.

Changes

  • Watch idle upstream connections in all five HTTP request loops, including JSON-RPC/MCP. For a ready allowed request, return a correlated 502 on unsolicited data or reset without forwarding it. Routine idle EOF uses debug logging.
  • Track final response bytes accepted by the client, excluding separately written informational responses and preserving partial-write counts. Preserve original upstream causes and endpoint observations. Distinguish incomplete framing from normal body completion followed by policy or middleware errors.
  • Share correlated gateway errors, upstream shutdown, and failure telemetry with forward HTTP. Record a 502 only after its write succeeds, and end affected WebSocket middleware sessions as upstream failures.
  • Extend existing test matrices for inspected requests, protocol coverage, truncated bodies, resets, and policy reload; add a full forward-handler regression. Existing response-middleware-owned failures retain their own responses and events.

Release Notes

HTTP clients receive idle upstream closure promptly. Relay-owned failures before a final response starts return an explicit gateway error with a request ID.

Testing

  • Sixteen distinct focused regressions passed across retained and corrected receipts: idle closure, queued failures/no replay, reset causes, local SigV4 100 Continue and upstream informational responses, ready unsolicited data/reset, forward-handler 502/activity, policy reload and middleware ownership/streaming.
  • The three existing CI regressions now pass with expectations for typed EOF handling, the correlated 502 and failure record, and outer-terminal tracing after malformed committed response framing.
  • Final cargo clippy --locked -p openshell-supervisor-network --lib --tests -- -D warnings, Rust formatting, whitespace and SPDX checks passed.
  • Full Linux/Kubernetes E2E and the reporter's EKS/Istio/Bun deployment not run locally. Hosted CI owns broad verification.

Validated with Rust 1.95.0 on macOS after rebasing onto 0be63c5b0 and resolving conflicts with current request tracing and streaming middleware.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@EmilienM EmilienM left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another iteration needed: 1 blocking, 4 major, 7 minor (inline, 5 below)

The tunnel REST/GraphQL/passthrough loops are covered and tested, but the JSON-RPC/MCP loop and the forward-proxy entry path don't get the fix, and response_started can still suppress the 502 with nothing delivered to the client.

Outside the diff:

  • Blocking: the PR has no area label, so Review Readiness fails; area:supervisor (or area:runtime) fits.
  • Major: crates/openshell-supervisor-network/src/proxy.rs:6836, the forward-proxy error boundary downcasts the sibling typed relay errors but has no arm for the new UpstreamHttpError, so plain forward requests still close silently with no 502 and the allow-event for an allowed request is now skipped.
  • Major: issue #5322's first acceptance criterion (replaying a request sent on a reused connection when the upstream closed before any response byte) isn't implemented; since the PR closes the issue, either implement it for idempotent methods or update the issue so the descope is recorded.
  • Minor: relay.rs:2355, an upstream failure now ends the websocket middleware session as Cancellation while the sibling arms use UpstreamFailure.
  • Minor: the PR body says the connection behavior is documented, but the second commit removed the docs paragraph; restore a trimmed note or drop the claim.
How this was reviewed
  • Commit e5fe0357, round 1
  • Standard review, because the change is code in crates/openshell-supervisor-network/src/l7/relay.rs, l7/rest.rs, and l7/rest/http_response.rs
  • Passes: architecture, performance, security, scalability, rust, supply-chain, then correctness
  • Independent skeptics checked each candidate finding (2 per blocking or major, 1 per minor); 11 of 11 held up, 2 of them after re-checking the code
  • Repo rules read from main: AGENTS.md, CLAUDE.md, CONTRIBUTING.md, docs/CONTRIBUTING.mdx
  • Not covered: nothing skipped

@shiju-nv shiju-nv added retired:area:supervisor Retired; do not apply. See #5151 for item-specific migration to the current taxonomy. area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. and removed retired:area:supervisor Retired; do not apply. See #5151 for item-specific migration to the current taxonomy. labels Nov 6, 2026
@github-actions
github-actions Bot requested review from TaylorMutch, drew and purp November 6, 2026 17:21
@shiju-nv

shiju-nv commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

All five HTTP loops now watch idle upstream connections, including JSON-RPC/MCP. Forward HTTP uses the same upstream-error handling and preserves allowed-request telemetry. Response commitment tracks successful client writes, so an incomplete buffered header can still produce a 502. Original I/O causes are preserved, and truncated bodies are distinguished from policy or middleware failures after normal body completion.

The shared handler preserves the request ID, shuts down upstream, and records a 502 only after delivery succeeds. Session cleanup uses UpstreamFailure. The tests now cover inspected REST, resets and the forward-handler path. The stale docs claim is removed, and area:runtime is set.

@purp purp added topic:mcp Model Context Protocol support, server/client implementations, and standard integration. topic:networking General network transport, connectivity, and proxy behavior. topic:observability Logs, metrics, traces, telemetry, dashboards, event correlation, and operational data export. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts. labels Nov 15, 2026
@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

Blocked

GitHub reports merge conflicts with main, so Gator cannot proceed with review.

Action required: @shiju-nv, resolve the merge conflicts and push the updated branch. Gator will re-check the process gates and review history on the next cycle.

Gator metadata
  • Head SHA: c54804740ead9cb16bbf6b661a1771bbcdec9b1d
  • Gator payload: 13
  • Next state: gator:blocked
  • Blocked reason: merge_conflicts

@johntmyers johntmyers added the gator:blocked Gator is blocked by process or repository gates label Nov 16, 2026

@FrostGod FrostGod left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some clients, such as Go's net/http, automatically retry an idempotent request (like GET) when a reused connection fails before any response arrives. A 502 counts as a real response, so those clients will now show the error instead of retrying. Is that trade-off intended for GET traffic, or should the 502 apply only to non-idempotent requests?

// Parsing wins a simultaneous close so forwarding can return 502, but
// ready unsolicited data must never become that request's response.
if matches!(request, Ok(Some(_))) {
match upstream.read(&mut byte).now_or_never() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a request is ready and the upstream has sent unsolicited bytes, the client gets a bare close. The request was never forwarded, so could we return a 502 here instead?

}
return Ok(RelayOutcome::Consumed);
return Err(miette::Report::new(UpstreamHttpError {
response_started: informational_header_bytes > 0,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any 1xx response, including the 100 Continue the proxy writes itself for SigV4 (rest.rs:1077), marks the response as started, so a later upstream failure gets no 502. HTTP still allows a final status after a 1xx. Should "started" count only bytes of the final response?

U: AsyncRead + Unpin,
{
let mut byte = [0u8; 1];
let request = tokio::select! {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the upstream closes while the client is still sending request headers, the partial request is dropped with no reply. Is that intended? If so, could we add a short comment saying so?

request
}

fn emit_idle_upstream_close(ctx: &L7EvalContext, result: std::io::Result<usize>) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every normal keep-alive timeout now emits an Informational OCSF "Close" event. With upstreams that time out after 5 seconds, that could be a lot of events. Is that volume OK, or should this be a debug! log?

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv
shiju-nv force-pushed the fix/5322-upstream-idle-close/shiju-nv branch from c548047 to 5e7eaca Compare November 17, 2026 12:54
@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

Relay-owned HTTP failures while awaiting a final response now return 502 Bad Gateway instead of a transport-level close, including for GET.

  • Impact: Existing clients such as Go's net/http stop automatically retrying eligible GET requests on reused connections and instead return the 502 to applications.
  • Action: Preserve transport-level closure for existing clients in v0.1.9, or defer unconditional 502 responses to a minor release with actionable, idempotency-aware retry migration guidance.
  • PR: fix(proxy): handle upstream HTTP connection closure #5335
Review details
  • Execution: complete
  • Candidate: PR #5335 (f70e46876b96a1e384f519eb81c2e59f20340ef8)
  • Baseline: ce58cf8e9b953281352d98a637e41616295b1318 (ce58cf8e9b953281352d98a637e41616295b1318)
  • Reviewed: rust-sdk, python-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

@shiju-nv, I checked your update about the five HTTP loops, forward-proxy handling, response commitment, and failure telemetry against the current patch. The implementation and Fern docs follow the updated issue scope, including explicit 502 responses for GET and no automatic replay. One blocking test update remains: Branch Checks fails three existing response/tracing tests on Linux and macOS because they still expect the previous behavior.

Action required: @shiju-nv, update those regression tests as described inline and push the change so CI can verify the new contract.

Blocking findings:

  • GATOR-5e7eacaa-01: Existing EOF and request-tracing assertions must match the new error and event handling.

Carried findings: None.

Non-blocking suggestions: None.

Gator metadata
  • Validation: Focused networking bug fix implementing issue #5322.
  • Docs: Fern network-rules page explains idle closure, 502 responses, and retry implications.
  • Checks: DCO and issue acceptance pass; required Branch Checks fails three existing Rust tests on Linux and macOS.
  • E2E: Required for network-proxy behavior; test:e2e is absent and E2E execution was skipped. Dispatch follows resolution of blocking review feedback.
  • Head SHA: 5e7eacaaedede863ec39ef619711eb03d5a56293
  • Base SHA: ce58cf8e9b953281352d98a637e41616295b1318
  • Merge base SHA: 0be63c5b0de27692df03ba0e4b3bc212312d3b3c
  • Patch ID: d5ff900a5c311c8b47cf09e6c8c53f7bb967c679
  • Gator payload: 13
  • Review mode: initial
  • Previous reviewed SHA: none
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:in-review

@johntmyers johntmyers added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:blocked Gator is blocked by process or repository gates labels Nov 17, 2026
Signed-off-by: Shiju <shiju@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Nov 17, 2026
@johntmyers

Copy link
Copy Markdown
Collaborator

/ok to test edce16d

@github-actions

Copy link
Copy Markdown

Label test:e2e applied for edce16d. Open Branch E2E Checks, find the run for commit edce16d, and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

PR Review Status

@shiju-nv, I checked your updated EOF and tracing tests against the new error handling. The direct helper now asserts the typed upstream failure, forward HTTP checks the correlated 502 and transport-failure event, and the startup test exercises malformed framing at the outer error boundary. GATOR-5e7eacaa-01 is fixed and its thread is resolved. The independent follow-up review found no new blocking findings.

Required CI is now queued or running on this head, including Branch Checks, Helm Lint, and Branch E2E Checks. Gator will inspect the results next cycle.

Blocking findings: None.

Carried findings: None; GATOR-5e7eacaa-01 resolved by this test update.

Gator metadata
  • Validation: Focused networking fix implementing accepted issue bug(proxy): a reused keep-alive connection whose upstream closed while idle is dropped without a response #5322; author has verified repository write permission.
  • Docs: Fern network-rules page documents idle closure, correlated 502 responses for every method, and retry implications.
  • Checks: DCO, issue acceptance, and Trivy Changes passed; current-head Branch Checks and Helm Lint queued/running.
  • E2E: Applied test:e2e and posted /ok to test with the full freshly read head SHA. The label-help rerun hint raced with fresh mirror dispatch; a new current-head push run started after labeling, with E2E suite bundles and version/build prerequisites active. No completed skipped E2E attempt needs rerunning.
  • Head SHA: edce16d40a12354401a4bfb458435923fa95dcf5
  • Base SHA: ce58cf8e9b953281352d98a637e41616295b1318
  • Merge base SHA: 0be63c5b0de27692df03ba0e4b3bc212312d3b3c
  • Patch ID: 58595b9de989a064bf1906625d62b3bf6944762d
  • Gator payload: 13
  • Review mode: follow_up
  • Previous reviewed SHA: 5e7eacaaedede863ec39ef619711eb03d5a56293
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Nov 17, 2026
@johntmyers johntmyers added gator:blocked Gator is blocked by process or repository gates gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:blocked Gator is blocked by process or repository gates gator:watch-pipeline Gator is monitoring PR CI/CD status labels Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. gator:approval-needed Gator completed review; maintainer approval needed test:e2e Requires end-to-end coverage topic:mcp Model Context Protocol support, server/client implementations, and standard integration. topic:networking General network transport, connectivity, and proxy behavior. topic:observability Logs, metrics, traces, telemetry, dashboards, event correlation, and operational data export. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(proxy): a reused keep-alive connection whose upstream closed while idle is dropped without a response

5 participants