Repository navigation
Conversation
…ored spec Suites that check the gateway's TLS boundary need the CA and client certificate testkit already found for the gateway, and suites that check what the gateway persists need the stored sandbox spec. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
… suites Ports the cases from e2e/python that had no Rust coverage into the conformance suite, so they run on every compute driver instead of only the Docker Python lane: - policy_validation: create refuses root, path traversal, a writable / and ambiguous endpoint pinning; MCP endpoints get the default version. - policy_enforcement: Landlock read/write/deny matrix, the policy user, cross-directory renames (Landlock refer), and EPERM/EACCES for connections no rule allows. - provider_credentials: credentials refused outside their profile placement, the Google Cloud metadata server's placeholder token, unknown providers, and credentials kept out of the stored spec. Refs #5435 Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Ports e2e/python/test_security_tls.py: a client certificate from the gateway's CA authenticates a user, a CA-only client reaches only Health, and a foreign certificate or plaintext connection is refused. Refs #5435 Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Ports e2e/python/oidc to e2e/rust/tests/oidc_authz.rs in the Keycloak lane: requests without a token, token scopes, client credentials, the non-member matrix over every workspace-scoped RPC, Platform Admin bypass, workspace user and admin roles, and platform-scoped RPCs. Tokens come from password and client-credentials grants; calls use the Rust SDK's generated client. Refs #5435 Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Every case in e2e/python now maps to Rust coverage or an explicit drop (#5435). The SDK-client suites (exec admission, sandbox API, exec_python, workspace API) run in ai-openshell/sdk-python against published core gateways. - Delete e2e/python and the e2e:python and e2e:oidc-python:docker tasks; e2e no longer depends on python:proto or PYTHONPATH=python. - Move the workload image's Dockerfile to e2e/workload/Dockerfile. The image tag and contents are unchanged; testkit, the Rust harness and tmachine keep using openshell/e2e-python:dev. - Docker E2E: replace the python and oidc-python rows with gateway-mtls, drop the python:proto setup step, and run oidc_authz in the oidc-pkce lanes. Add a gateway-mtls row on the deb tmachine gateway. - Update TESTING.md and the testkit docs. Refs #5435; python/ itself is removed by #5287. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
|
🌿 Preview your docs: https://nvidia-preview-pr-5624.docs.buildwithfern.com/openshell |
|
Label |
elezar
left a comment
There was a problem hiding this comment.
Thanks for the thorough mapping table. The ported coverage doesn't duplicate what's already in conformance (filesystem_policy.rs, credential_gating.rs) or transparent_tcp.rs, and the testkit additions are small. CI is red, though, and I have a few structural questions.
CI failures (k3s and k3s-ha; I haven't checked the other lanes)
policy_enforcement::landlock_enforces_filesystem_policy: the probe callspwd.getpwuid(os.getuid()). On k3s the exec runs as uid 10001, which has no passwd entry (KeyError). The Kubernetes driver's fallback uid isDEFAULT_SANDBOX_UID = 10001, so the exec user may not be the policy'srun_as_user: sandboxthere. Is "exec runs as the policy user" meant to be a conformance contract? If so, the Kubernetes driver may need a look. If not, resolve the name tolerantly and assert only the Landlock behaviour.provider_credentials::google_metadata_serves_placeholder_tokenandprovider_credentials_are_not_stored_in_sandbox_spec:provider create --type google-cloudand--type claude-codefail withprovider profile '…' not found; import a matching profile. They rely on profiles already being on the gateway. The two placement tests in the same file import an inline profile withinstall_provider, which works on every lane. Please do the same here.gateway-mtls::foreign_client_certificate_is_rejectedfails onubuntu-docker-rootful. I haven't looked at the cause.
Structure and direction
e2e/rust/tests/oidc_authz.rsadds about 1.2k lines to the legacye2e/rusttree, on rawopenshell-sdk, behinde2e-oidc-pkce. The direction elsewhere (#4891, #5137) is testkit-basedtests/suites/features/*suites on Keycloak, andkeycloak-provider-refreshalready exists there. Is the legacy location deliberate? Also, e2e/rust clippy doesn't enablee2e-oidc-pkce, so this file isn't linted in CI.- The mTLS refusal tests pass on any connect failure. A wrong port or a down gateway would also pass them. Please assert the specific TLS or gRPC error.
e2e/workload/is next toe2e/gpu/(task namese2e:workload:buildande2e:workloads:build), which is easy to confuse. A README line, or a clearer name, would help. The Dockerfile also still installscloudpickle,google-auth,requestsand a uv venv, which served the removed SDK tests. Trimming it would shrink an image that is loaded into every environment.- #5442 edits
testkit/src/context.rs,conformance/src/lib.rs,e2e-suites.yml,TESTING.mdand the testkit README. Whichever merges second will need a rebase.
On lane cost: only about four of the new conformance tests start sandboxes (about 16–19s each on k3s), and the rest take milliseconds. I don't see a problem there.
@matthewgrossman: could you take a look at points 1 and 4? They concern where the OIDC tests belong (legacy e2e/rust or a testkit features suite) and whether "exec runs as the policy user" is a conformance contract.
Summary
The Python SDK has moved to ai-openshell/sdk-python, and core is removing its copy in
python/(#5287). Core's Python E2E suites still import the SDK frompython/, sopython/can't be removed yet.This PR ports the coverage of those suites to Rust, so core keeps it without depending on Python. It then deletes
e2e/python/. The suites that test SDK client behavior (exec_admission,sandbox_api,sandbox_exec_pythonandworkspace_api) aren't ported. sdk-python already runs copies of them against core gateways.The ported tests cover:
Related Issue
Closes #5435. Unblocks #5287.
Changes
tests/suites/conformancecover policy validation, policy enforcement and provider credentials. They run on every driver lane.tests/suites/features/gateway-mtlssuite covers mTLS.e2e/rust/tests/oidc_authz.rscovers OIDC and workspace authorization. It runs in the existing Keycloak lanes.Gateway::mtls_dir()andSandbox::spec().e2e/python/, its mise tasks and its CI rows. The workload image's Dockerfile moves toe2e/workload/Dockerfile; the Rust tests still use it.Case-by-case mapping
test_policy_validation.pygrpc/policy.rstest_sandbox_policy.pye2e/rust/tests/transparent_tcp.rstest_sandbox_landlock.pyfilesystem_policy.rscovers device paths and path replacementtest_sandbox_venv.pyENVpropagation: driver unit testsPATH(asserts the fixture image, not product behavior);pip/uvinstalls (need live PyPI)test_security_tls.pytest_sandbox_providers.pycredential_gating.rs,provider_auto_create.rs. Attach/detach:provider_readiness.rs. Imported profiles:host_gateway_alias.rs,stable_external_placeholders.rs. Update/merge semantics, the other placement styles, and profile scoping: server and core unit testsgit clone(needs github.com); real google-auth SDK compatibility (needs google-auth in every lane's image)oidc/oidc_auth_test.pyoidc_pkce.rsoidc/workspace_authz_test.pyall_workspaces; platform profile and global policy RPCsGetSandboxLogswith a spoofed workspace (the same scenario is unit-tested ingrpc/policy.rs)exec_admission,sandbox_api,sandbox_exec_python,workspace_apitests/e2eTesting
mise run pre-commitpasses (the pre-commit hook ranrust:formatandrust:linton each commit; I didn't run the full task)Local runs against Docker gateways, before rebasing onto current main:
gateway-mtls: 4/4oidc_authzagainst Keycloak: 7/7After the rebase, I re-ran clippy with
-D warningson the conformance,gateway-mtlsande2e/rust(--features e2e-oidc-pkce) workspaces, plusrust:lockfiles:check. The Podman, Kubernetes, VM and tmachine lanes haven't run locally, so CI covers them.Known limits, unchanged from the Python tests:
e2e/rustclippy doesn't enablee2e-oidc-pkce, so it doesn't lintoidc_pkce.rsoroidc_authz.rs.Checklist