Skip to content

test(conformance): run the conformance suite against a gateway that pushes configuration - #5622

Open
pimlock wants to merge 9 commits into
mainfrom
test/conformance-config-push/pimlock
Open

pimlock wants to merge 9 commits into
mainfrom
test/conformance-config-push/pimlock

Conversation

@pimlock

@pimlock pimlock commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Runs the full conformance suite in CI against a gateway with config_delivery_mode = "push", ahead of making push the default in #4498.

Related Issue

Refs #4498. No issue required: test and CI coverage only, no product behavior change.

Changes

  • New tmachine feature environment ubuntu-docker-rootful-config-push: ubuntu-docker-rootful plus playbooks/config-push.yaml, which writes config_delivery_mode = "push" to /etc/openshell-test/gateway-overrides.toml. The deb, rpm and binaries installers already merge that file.
  • One row in .github/workflows/e2e-suites.yml: conformance in the new environment with the deb installer.
  • mise run e2e:testkit:push runs conformance against a local Docker gateway with OPENSHELL_CONFIG_DELIVERY_MODE=push, for hosts without KVM.

Nothing asserts push is in use. A typo in the setting fails gateway startup (unknown fields are rejected), and supervisors built from this tree always support push, so push is off only if the override file stops being merged. A separate evidence suite was tried and dropped as not worth a VM per E2E run.

Scope notes:

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Local results, before merging main:

  • tmachine (aarch64 host), ubuntu-docker-rootful-config-push + deb: conformance 22/22 pass.
  • Local Docker gateway in push mode: conformance 22/22 pass; the gateway log shows all 27 supervisor sessions accepted with config_push=true.

The new CI row runs on amd64 and has not run in CI yet.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

…ushes configuration

Add the ubuntu-docker-rootful-config-push tmachine environment. Its setup
playbook, config-delivery.yaml, writes config_delivery_mode to the gateway
overrides that the deb, rpm and binaries installers apply, and publishes the
mode as OPENSHELL_TEST_CONFIG_DELIVERY_MODE.

A push gateway lets a supervisor without push support keep polling, so the
conformance suite passes in either mode. The new config-push feature suite
fails unless the published mode is push and a new sandbox's supervisor
reports a pushed-configuration session.

E2E Suites runs the conformance family and the config-push suite in the new
environment with the deb installer. mise run e2e:testkit:push runs both
against a local Docker gateway in push mode.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
The supervisor logs a pushed-configuration session only when the gateway
accepted it with push enabled, so the config-push suite no longer also
requires an OPENSHELL_TEST_CONFIG_DELIVERY_MODE variable. The environment's
playbook, now config-push.yaml, sets push without a mode input, since
polling is going away, and no longer writes an [openshell] table that a
second overrides writer could duplicate.

The test retries a failed `openshell logs` call until its timeout.
e2e:testkit:push runs the push check before the conformance suite, so a
conformance failure can't hide whether push was on. The TESTING.md section
moves next to the tmachine runner docs instead of capturing the feature
suite paragraphs that follow the gateway overrides section.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 16, 2026
…teway-push-1d7bb5

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

# Conflicts:
#	.github/workflows/e2e-suites.yml
#	tests/suites/features/Cargo.toml
#	tests/tmachine/config.nix
@github-actions

Copy link
Copy Markdown

@pimlock
pimlock marked this pull request as draft November 16, 2026 21:48
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
A gateway built from this tree rejects unknown config fields and its
supervisor always supports push, so the push conformance row already
runs with push on unless the override file stops being merged. The
separate suite cost a VM per E2E run for that one case.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
… file

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock
pimlock marked this pull request as ready for review November 16, 2026 23:54
@pimlock pimlock added the test:e2e Requires end-to-end coverage label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 002bf18. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant