Skip to content

fix(network): refresh expired policy DNS mappings on connect - #5650

Open
drew wants to merge 2 commits into
mainfrom
fix/4674-refresh-expired-dns/drew
Open

drew wants to merge 2 commits into
mainfrom
fix/4674-refresh-expired-dns/drew

Conversation

@drew

@drew drew commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Refresh expired policy DNS mappings when a new transparent connection uses a cached synthetic address. Clients can reconnect after the resolver TTL without performing another DNS lookup, while resolver failures and rejected destinations still fail closed.

Related Issue

Closes #4674

Changes

  • Retain expired mapping identities and reuse the trusted DNS publication and destination-validation path on connect.
  • Reauthorize mapped connections at dispatch and rebuild connectors from the latest validated pins.
  • Retain authored refresh-eligible ports independently of the contracts satisfied by the latest DNS answer.
  • Combine concurrent refreshes for each synthetic address and back off for one second after failure. Start the fresh mapping lease after resolution completes.
  • Preserve process/port authorization and policy-generation checks; policy-generation invalidation remains tracked separately in bug(network): every policy generation change invalidates all policy DNS mappings #5366.
  • Document refresh behavior and the mapping-denial reason, and exercise cached-address expiry in the existing transparent TCP E2E test.

Release Notes

Clients that cache DNS answers can open new connections after a synthetic mapping expires. OpenShell refreshes and validates the destination before connecting.

Testing

  • Networking crate: 1,682 tests passed, 2 ignored.
  • Dispatch regression verifies a real TCP connection to the fresh destination after expiry; partial-mapping regression verifies recovery of a missing port after expiry.
  • Focused concurrency regression passed with a resolver slower than the one-second TTL floor.
  • Clippy for all networking crate targets, Rust formatting, and mise run docs passed.
  • Repository pre-commit checks passed, including workspace and additional test/example Clippy checks, formatting, Markdown, license, Helm, and SDK checks.
  • Docker sandbox E2E local_container_native_tcp_uses_policy_dns_and_fails_closed passed with freshly built supervisor and sandbox images, including a connection to a cached numeric address after 31 seconds.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Relevant implementation and published documentation updated

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

@drew drew added area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. topic:documentation Architecture docs, READMEs, project website updates, and automated documentation generation. topic:networking General network transport, connectivity, and proxy behavior. topic:policy Policy authoring, validation, proving, distribution, enforcement, ergonomics, and hardening. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts. labels Nov 17, 2026
@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for e66f4e3. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

The code review found no blocking issues. Expired mappings refresh through the trusted resolver and destination-validation path, while process, port, endpoint, and policy-generation checks remain in place. The docs explain refresh behavior and failed-refresh denials.

Blocking findings: None.

Carried findings: None.

Gator metadata
  • Validation: Focused network reliability fix implementing accepted issue #4674.
  • Docs: Published network-policy and logging pages updated.
  • Checks: Branch checks remain pending; Helm lint passed and deployment scanning was not applicable.
  • E2E: test:e2e applied; current-head Branch E2E Checks attempt 2 running for the standard suite, including the cached-address expiry regression.
  • Head SHA: e66f4e3ef69851e0ab0ae4e6860632574e020555
  • Base SHA: 624aceea32934f73c350ede3f2185880816a030c
  • Merge base SHA: 2e5589b2180dd432320adda124c4f7bcede119b5
  • Patch ID: 5893900c887c164a917e8afbb8d4fbcd9a9ef6b2
  • Gator payload: 13
  • Review mode: initial
  • Previous reviewed SHA: none
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added the gator:watch-pipeline Gator is monitoring PR CI/CD status label Nov 17, 2026

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed e66f4e3 locally. I found two reproducible correctness issues, detailed inline.

Validation: 71 existing policy-DNS tests passed. Two additional focused regressions reproduced the findings below, including the dispatch case with real TTL expiry. The tests ran in a temporary copy of this exact commit.

@johntmyers johntmyers added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Nov 17, 2026
}

#[async_trait::async_trait]
pub(super) trait MappingRefresher: Send + Sync {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep refresh orchestration in PolicyDnsService and expose one async lookup_for_connect operation that returns the resulting mapping?

Having the store call back into its owner adds MappingRefresher, a weak reference, OnceLock, and a separate enable_refresh step. Callers then have to refresh and look up the mapping separately.

process: { run_as_user: sandbox, run_as_group: sandbox }
";

struct RefreshResolver {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we simplify this fixture? It combines mutable answers, failure injection, call counting, and pause/release controls. This module already has FakeResolver, NxDomainResolver, and BlockingResolver.

}

#[tokio::test]
async fn expired_mapping_refresh_does_not_upgrade_ports_observations_or_policy_generations() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make this a compact refresh-rejection test and give it a name that matches its assertions? It currently checks that these three attempts return an error and make no additional upstream DNS query.

If we move to a service lookup_for_connect operation, assert rejection through that operation and keep the cases clearly labeled. That would cover the behavior callers depend on without tying the test to a separate refresh_if_expired helper.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

The follow-up review found no new blocking issues. Thanks @TaylorMutch: I checked both reported correctness problems against the new code and regressions. Dispatch now reauthorizes and rebuilds connectors from current pins, and refresh eligibility retains authored ports when the latest DNS answer satisfies only some ports.

@shiju-nv, I also checked your requests for service-owned refresh orchestration, simpler resolver fixtures, and a compact rejection test. Those remain open advisory review topics; this commit retains the existing structure. I have left your threads untouched.

Blocking findings: None.

Carried Gator findings: None.

Required branch checks and the standard E2E suite are running on this head. Gator will monitor their results; maintainer review is still required.

Gator metadata
  • Validation: Focused network reliability fix implementing accepted issue #4674.
  • Docs: Previously reviewed published network-policy and logging updates remain accurate; delta adds implementation details to the crate README.
  • Checks: DCO, Helm Lint, and Trivy Changes passed; Branch Checks pending.
  • E2E: test:e2e retained; current-head Branch E2E Checks run 44156946578 is queued with standard-suite jobs running. No rerun needed for this head.
  • Head SHA: 97e8a42fc96be51179c31eb19ed2691bc2cd2100
  • Base SHA: 624aceea32934f73c350ede3f2185880816a030c
  • Merge base SHA: 2e5589b2180dd432320adda124c4f7bcede119b5
  • Patch ID: b10279719bf29e9a09ebfad4b49dfd2b9da29bff
  • Gator payload: 13
  • Review mode: follow_up
  • Previous reviewed SHA: e66f4e3ef69851e0ab0ae4e6860632574e020555
  • Review budget exhausted: no
  • Maintainer decision required: no convergence decision required; open advisory human threads remain for normal review.
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Nov 17, 2026
@johntmyers johntmyers added the gator:approval-needed Gator completed review; maintainer approval needed label Nov 17, 2026

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked 97e8a42. Both previously reported correctness issues are fixed: dispatch reauthorizes and rebuilds connectors from current DNS pins, and refresh eligibility preserves authored ports independently of the latest usable contracts. No new correctness findings. Validation: 72 policy-DNS tests and the real-TCP dispatch regression passed locally; Branch Checks and E2E are green. The open architecture and test-simplification suggestions are advisory.

@johntmyers johntmyers added gator:merge-ready and removed gator:approval-needed Gator completed review; maintainer approval needed labels Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. gator:merge-ready state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage topic:documentation Architecture docs, READMEs, project website updates, and automated documentation generation. topic:networking General network transport, connectivity, and proxy behavior. topic:policy Policy authoring, validation, proving, distribution, enforcement, ergonomics, and hardening. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A policy DNS address stops taking new connections after the trusted resolver's TTL, one second when that TTL is 0

4 participants