Skip to content

test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance - #5665

Draft
matthewgrossman wants to merge 15 commits into
test/5426-port-cli-attach-exec/matthewgrossmanfrom
test/5655-port-network-l7-policy/matthewgrossman
Draft

matthewgrossman wants to merge 15 commits into
test/5426-port-cli-attach-exec/matthewgrossmanfrom
test/5655-port-network-l7-policy/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Port the network, L7, policy-update, and port-forward tests from e2e/rust to testkit, and delete the legacy files. Most tests move into the conformance suite. The IP-literal egress test moves to a new routable-host-ip feature package.

Stacked on #5442. The base is test/5426-port-cli-attach-exec/matthewgrossman, and I'll retarget to main after #5442 merges.

Related Issue

Closes #5655
Part of #5281

GitHub links #5655 only after this PR is retargeted to main, because closing keywords apply only to PRs into the default branch.

Changes

  • testkit: Context::compute_driver, SandboxBuilder::driver_config, Sandbox::logs / logs_until, policy::set, policy::from_yaml, and policy::base_with.
  • host-server support: handler responses, chunked bodies, a raw TcpServer, free_port, and a loopback exchange client.
  • New conformance suites:
    • network: bypass denial, reserved destinations, localhost without proxy env, listener lifetime.
    • network_l7 + network_l7/mcp: REST, GraphQL, JSON-RPC, and MCP.
    • network_egress: reload, ambiguous update, tls: skip, middleware, pipelining.
    • policy_update and port_forward.
  • credential_gating additions: header and body rewrite, WebSocket text rewrite, and static credentials bound to profile endpoints.
  • New feature package tests/suites/features/routable-host-ip: the IP-literal endpoint test (explicit allowed_ips and implicit). CI rows: Docker rootful (deb), Podman rootful (rpm), k3s, and k3s-ha.
  • New tmachine feature config-push: an ubuntu-docker-rootful-config-push conformance row that runs every policy test with config_delivery_mode = "push". It replaces live_policy_update in e2e:rust:push.
  • Removed:
    • The 11 legacy files, plus static_provider_credentials_are_bound_to_profile_endpoints and its helpers from host_gateway_alias.rs. The other two tests in that file stay until PR5.
    • The matching [[test]] entries, PODMAN_CI_TESTS entries, and artifacts.nix follow-up binary.
    • The e2e:websocket-conformance task and the now-unused SandboxGuard::spawn_forward.
  • Docs: conformance README suites table, and TESTING.md (routable-host-ip, config-push env).

Defaults applied

  • Upstream fixtures are in-process Rust servers on the test host, reached at host.openshell.internal. The legacy private-range allowed_ips entries are gone.
  • Client scripts run through Sandbox::exec in sleep infinity sandboxes, using pipes. The legacy tests were also non-TTY, so PTY/pipe parity is unchanged.
  • There is no exec_until helper; Sandbox::poll_exec already covers it.
  • Network tests use policy::base(), python_image(), and the shared Python binaries list.
  • Merges keep every assertion:
    • REST allow and deny are merged.
    • The JSON-RPC audit-mode case is a second endpoint in the JSON-RPC test.
    • MCP is split into three tests, one per protocol version set.
    • The live_policy_update round-trip and from-empty tests are merged into policy_set_versions_and_ignores_repeats.
  • Policy-update tests wait until the active revision is loaded and none is pending or failed. This covers the supervisor's enrichment of the base policy.

Deviations from the plan

  • port_forward is its own module, not part of cli.rs, to avoid conflicts with test(e2e): port CLI exec, attach, and SSH host identity tests to testkit #5442. It starts forward start -d with cli_spawn and waits for the exit code, because the backgrounded ssh keeps the CLI's pipes open. A cleanup hook runs forward stop.
  • The static credential binding test lives in credential_gating.rs. The unbound endpoint is a second port on the test host instead of host.docker.internal, so the 403 holds on every driver. The legacy Podman connect-denied special case is gone.

Rows that kept a container fixture

None. The IP-literal test needs a host address that the supervisor allows as an IP literal. Rootless Podman's address is link-local, which is always blocked. So instead of keeping a container fixture, that test is a feature package that does not run on rootless Podman.

Driver coverage changes

  • Docker and Kubernetes ran all of these tests before and still do, through the conformance rows.
  • Podman previously ran only bypass_detection, forward_proxy_*, host_gateway_alias, live_policy_update, and no_proxy. It now runs all of the ported conformance tests, on rootful and rootless.
  • The VM host lane previously ran only host_gateway_alias from this set. It now runs all of them through openshell-conformance in e2e-vm.sh. That is new coverage. If a newly covered row fails, I'll raise it here before skipping anything.

Testing

  • cargo fmt --check and cargo clippy --locked --all-targets -D warnings for tests/testkit, tests/support, tests/suites/conformance, tests/suites/features
  • cargo test for tests/testkit and tests/support (new host-server unit tests included)
  • tasks/scripts/check-conformance-program.sh (71 tests)
  • cargo check --tests for e2e/rust with docker, podman, kubernetes, vm, and config-push features. cargo clippy on host_gateway_alias is clean.
  • E2E suites (CI)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

…olicy set helpers

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…vers to host-server

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ewrite tests

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
… tests

These tests now run in the conformance suite and the routable-host-ip
feature package.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…suites

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 858dbc6

@matthewgrossman matthewgrossman added test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 858dbc6. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant