Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions .github/workflows/e2e-docker-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ on:
required: false
type: string
default: >-
[{"suite":"python","cmd":"mise run --no-deps --skip-deps e2e:python","apt_packages":"","python_proto":true,"mcp":false},{"suite":"oidc-python","cmd":"mise run --no-deps --skip-deps e2e:oidc-python:docker","apt_packages":"","python_proto":true,"mcp":false},{"suite":"oidc-pkce-docker","cmd":"mise run --no-deps --skip-deps e2e:oidc-pkce:docker","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"rust-docker","cmd":"mise run --no-deps --skip-deps e2e:rust","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"rust-docker-push","cmd":"mise run --no-deps --skip-deps e2e:rust:push","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"oci-image","cmd":"mise run --no-deps --skip-deps e2e:oci-image","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"additional-ca","cmd":"OPENSHELL_E2E_DOCKER_TEST=additional_ca mise run --no-deps --skip-deps e2e:docker","apt_packages":"openssh-client","python_proto":false,"mcp":false},{"suite":"middleware-http-hooks","cmd":"mise run --no-deps --skip-deps e2e:middleware-http-hooks","apt_packages":"","python_proto":false,"mcp":false},{"suite":"mcp","cmd":"mise run --no-deps --skip-deps e2e:mcp","apt_packages":"","python_proto":false,"mcp":true}]
[{"suite":"oidc-pkce-docker","cmd":"mise run --no-deps --skip-deps e2e:oidc-pkce:docker","apt_packages":"openssh-client","mcp":false},{"suite":"rust-docker","cmd":"mise run --no-deps --skip-deps e2e:rust","apt_packages":"openssh-client","mcp":false},{"suite":"gateway-mtls","cmd":"mise run --no-deps --skip-deps e2e:gateway-mtls","apt_packages":"","mcp":false},{"suite":"rust-docker-push","cmd":"mise run --no-deps --skip-deps e2e:rust:push","apt_packages":"openssh-client","mcp":false},{"suite":"oci-image","cmd":"mise run --no-deps --skip-deps e2e:oci-image","apt_packages":"openssh-client","mcp":false},{"suite":"additional-ca","cmd":"OPENSHELL_E2E_DOCKER_TEST=additional_ca mise run --no-deps --skip-deps e2e:docker","apt_packages":"openssh-client","mcp":false},{"suite":"middleware-http-hooks","cmd":"mise run --no-deps --skip-deps e2e:middleware-http-hooks","apt_packages":"","mcp":false},{"suite":"mcp","cmd":"mise run --no-deps --skip-deps e2e:mcp","apt_packages":"","mcp":true}]

permissions:
actions: read
Expand Down Expand Up @@ -97,10 +97,6 @@ jobs:
- name: Log in to GHCR
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${GITHUB_ACTOR}" --password-stdin

- name: Install Python dependencies and generate protobuf stubs
if: matrix.python_proto
run: uv sync --frozen && mise run --no-deps python:proto

- name: Run tests
env:
OPENSHELL_MCP_CONFORMANCE_CLIENT_IMAGE: ${{ format('openshell-mcp-conformance-client:{0}', inputs.image-tag) }}
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/e2e-suites.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ jobs:
{"environment":"fedora-podman-rootful-cdi","installer":"rpm","family":"features","package":"cdi","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootless-cdi","installer":"rpm","family":"features","package":"cdi","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"ubuntu-docker-rootful-keycloak","installer":"deb","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"ubuntu-docker-rootful","installer":"deb","family":"features","package":"gateway-mtls","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"ubuntu-docker-rootful","installer":"deb","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootful","installer":"rpm","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"},
{"environment":"fedora-podman-rootless","installer":"rpm","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"},
Expand Down
92 changes: 27 additions & 65 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ mise run ci # Everything: lint, compile checks, and tests
crates/*/src/ # Inline #[cfg(test)] modules
crates/*/tests/ # Rust integration tests
python/openshell/ # Python unit tests (*_test.py suffix)
e2e/python/ # Python E2E tests (test_*.py prefix)
e2e/workload/ # Workload image the E2E sandboxes run python3, curl, and git from
e2e/rust/ # Rust CLI E2E tests
tests/testkit/ # Support library for SDK-driven E2E suites
tests/suites/ # E2E suites, grouped by test family
Expand Down Expand Up @@ -345,73 +345,23 @@ can run the suite. The expectations characterize the sandbox's behavior today,
not a specification, and the `echo-socket` check needs `perl` in the workload
image, which the default image has.

### Python E2E (`e2e/python/`)
### E2E workload image (`e2e/workload/`)

`mise run e2e:python` builds `openshell/e2e-python:dev` from
`e2e/python/Dockerfile.workload` and selects it only for the test gateway.
This Noble-based fixture supplies the `sandbox` user, Python tooling, Git,
and a writable `/sandbox/.venv`. Its Python version comes from `.python-version`
so cloudpickle code objects match the test runner. The production workload
`mise run e2e:workload:build` builds `openshell/e2e-python:dev` from
`e2e/workload/Dockerfile`; `testkit::python_image()` and the Rust Docker
harness select it for tests that need Python, Git, curl, openssl, or the
`sandbox` user. This Noble-based fixture also has a writable `/sandbox/.venv`.
Its Python version comes from `.python-version`. The production workload
default remains the unmodified NVIDIA Ubuntu image.

The Rust Docker harness also selects this fixture for tests that need tools
or the named user. Explicit `--from` or `--template` arguments and the default-image
tests retain their own image selection. Docker sandbox and support-container
fixtures use the fixed image name; no workload-image override is needed.
Podman, VM, and Kubernetes retain their existing pinned, pullable fixture and
image setup. Conformance-only runs do not need the Docker fixture.
Build the Docker fixture separately with `mise run e2e:workload:build`.
Explicit `--from` or `--template` arguments and the default-image tests retain
their own image selection. Docker sandbox and support-container fixtures use
the fixed image name; no workload-image override is needed. Podman, VM, and
Kubernetes retain their existing pinned, pullable fixture and image setup.

Tests use the `sandbox` fixture from `conftest.py` to create real sandboxes:

```python
def test_exec_returns_stdout(sandbox):
with sandbox(delete_on_exit=True) as sb:
result = sb.exec(["echo", "hello"])
assert result.exit_code == 0
assert "hello" in result.stdout
```

#### `Sandbox.exec_python`

`exec_python` serializes a Python callable with `cloudpickle`, sends it to the
sandbox, and returns the result. Because cloudpickle serializes module-level
functions by reference (which fails inside the sandbox), use one of these
patterns:

**Closures from factory functions:**

```python
def _make_adder():
def add(a, b):
return a + b
return add

def test_addition(sandbox):
with sandbox(delete_on_exit=True) as sb:
result = sb.exec_python(_make_adder(), args=(2, 3))
assert result.stdout.strip() == "5"
```

**Bound methods on local classes:**

```python
def test_multiply(sandbox):
class Calculator:
def multiply(self, a, b):
return a * b

with sandbox(delete_on_exit=True) as sb:
result = sb.exec_python(Calculator().multiply, args=(6, 7))
assert result.stdout.strip() == "42"
```

#### Shared Fixtures (`e2e/python/conftest.py`)

| Fixture | Scope | Purpose |
|---|---|---|
| `sandbox_client` | session | gRPC client connected to the active gateway |
| `sandbox` | function | Factory returning a `Sandbox` context manager |
Core has no Python E2E suite. The Python SDK's own E2E suite lives in
[ai-openshell/sdk-python](https://github.com/ai-openshell/sdk-python) and runs
against published core gateways.

### Rust CLI E2E (`e2e/rust/`)

Expand Down Expand Up @@ -453,7 +403,8 @@ Suites:
- Common suite (`--features e2e`) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests.
- Conformance (`tests/suites/conformance`) - the testkit `conformance` crate:
lifecycle, the CLI, persistent workspaces, file transfer, the policy
advisor, and credential gating. Driver E2E runs it with
advisor, credential gating, policy validation and enforcement (Landlock and
denied connections), and provider credential placement. Driver E2E runs it with
`openshell-conformance` against the lane's gateway and writes a JUnit report
under `results/`; set `OPENSHELL_CONFORMANCE_BIN` to use a prebuilt program,
as Branch E2E does with the static build it makes once per architecture.
Expand Down Expand Up @@ -650,6 +601,17 @@ rootful and rootless, and the `k3s` and `k3s-ha` installers. The tests run on
the K3s node, so the K3s installers set the chart's `server.hostGatewayIP` to
the node's address.

The `gateway-mtls` feature suite (`tests/suites/features/gateway-mtls`) checks
an mTLS gateway's user boundary: a client certificate from the gateway's CA
authenticates a user, a CA-only client reaches only Health, and a foreign
certificate or plaintext connection is refused. CI runs it against the deb
gateway on Docker rootful and in the Docker E2E workflow. Run it against a
local Docker gateway:

```shell
mise run e2e:gateway-mtls
```

Run the VM-backed Rust CLI e2e suite:

```shell
Expand Down
132 changes: 0 additions & 132 deletions e2e/python/conftest.py

This file was deleted.

2 changes: 0 additions & 2 deletions e2e/python/oidc/__init__.py

This file was deleted.

21 changes: 0 additions & 21 deletions e2e/python/oidc/conftest.py

This file was deleted.

Loading
Loading