Skip to content

test(conformance): port workspace lifecycle tests to testkit - #4607

Open
politerealism wants to merge 1 commit into
NVIDIA:mainfrom
politerealism:test/3954-workspace-stories/politerealism
Open

politerealism wants to merge 1 commit into
NVIDIA:mainfrom
politerealism:test/3954-workspace-stories/politerealism

Conversation

@politerealism

@politerealism politerealism commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Move the workspace lifecycle tests from the legacy e2e crate into the testkit conformance suite. Workspace create, scoping and delete behavior isn't driver-specific, so the tests run against any prepared gateway.

Related Issue

Closes #5598. Part of #5281.

Changes

  • Add tests/suites/conformance/src/workspaces.rs with three #[testkit::test] tests, and a workspaces row in the conformance README:
    • workspace_full_crud_lifecycle: create, get, listed as Active, a delete refused while it holds a resource leaves it Terminating, delete when empty, gone afterwards. The original checked seven behaviors in one test, so it's split. The name is kept so filters keep working.
    • template_is_scoped_to_its_workspace (new): a resource is listed in its own workspace and in the all-workspaces view with its owner, and isn't visible in the test's workspace.
    • workspace_terminating_rejects_creates: a refused delete leaves the workspace Terminating. New resources are rejected, existing ones can still be deleted, and retrying the delete then succeeds.
  • Use a sandbox template as the scoped resource. The original used a provider, which needs a provider profile that a freshly provisioned gateway doesn't have.
  • Assert on exit status and listed state, not message text.
  • Add Context::cli_find_listed to testkit: finds an entry in a paginated --output json listing and errors when the listing lacks the collection key. find_listed in conformance/src/cli.rs now uses it, and the testkit README lists it.
  • Delete e2e/rust/tests/workspace_lifecycle.rs, its Cargo target and its entry in the Podman archive exclusion list.

Testing

  • Build, clippy, fmt and the pre-commit hook pass; testkit unit tests pass.
  • The three tests pass on the ubuntu-k3s conformance row in CI (passed before the review changes; rerun pending).
  • Not run locally against a gateway. Not run on macOS.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (not applicable)

@copy-pr-bot

copy-pr-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

This pull request has had no activity for 14 days and is now marked stale.

@github-actions github-actions Bot added the state:stale Inactive item at risk of automatic closure. label Oct 30, 2026
elezar added a commit to politerealism/OpenShell that referenced this pull request Nov 13, 2026
Move e2e/rust workspace_lifecycle into a conformance workspaces suite on
#[testkit::test], split into workspace_full_crud_lifecycle,
template_is_scoped_to_its_workspace and workspace_terminating_rejects_creates.
Use a sandbox template as the scoped resource so no provider profile is
needed, and assert on exit status and listed state rather than message text.
Drop the legacy test, its Cargo target, and its Podman archive exclusion.

Supersedes NVIDIA#4607.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the test/3954-workspace-stories/politerealism branch from a2554c4 to 30f47a9 Compare November 13, 2026 13:12
@github-actions

Copy link
Copy Markdown

Maintainer list change

Gains approval rights:

Confirm every change is intended. Anyone listed here can single-handedly satisfy OpenShell / Maintainer Approval.

elezar pushed a commit to politerealism/OpenShell that referenced this pull request Nov 13, 2026
Move e2e/rust workspace_lifecycle into a conformance workspaces suite on
#[testkit::test], split into workspace_full_crud_lifecycle,
template_is_scoped_to_its_workspace and workspace_terminating_rejects_creates.
Use a sandbox template as the scoped resource so no provider profile is
needed, and assert on exit status and listed state rather than message text.
Drop the legacy test, its Cargo target, and its Podman archive exclusion.

Supersedes NVIDIA#4607.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the test/3954-workspace-stories/politerealism branch from 30f47a9 to d63af4f Compare November 13, 2026 15:19
@elezar
elezar changed the base branch from codex/4119-suite-layout to main November 13, 2026 15:19
@elezar elezar added area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. test:e2e Requires end-to-end coverage and removed state:stale Inactive item at risk of automatic closure. labels Nov 13, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/4607 does not exist yet. A maintainer needs to comment /ok to test d63af4f19e8a14827292aaafbf795aef9f2dbe49 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@elezar

elezar commented Nov 13, 2026

Copy link
Copy Markdown
Member

/ok-to-test d63af4f

/// The entry called `name` in a paginated `--output json` listing, where
/// `collection` is the JSON key that holds the entries. A listing without that
/// key is an error, so a "not listed" check cannot pass on a malformed listing.
async fn find_entry(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same paging loop as find_listed in cli.rs (same PAGE_SIZE and MAX_PAGES too), just made generic. Your version is the better one, since it errors when the collection key is missing instead of treating that as "not listed".

Could we move it into testkit in this PR rather than keep two copies? Something like:

impl Context {
    /// The entry called `name` in a paginated `--output json` listing from
    /// `list` (e.g. `["sandbox", "template", "list", "--all-workspaces"]`),
    /// where `collection` is the JSON key holding the entries.
    pub async fn cli_find_listed(
        &self,
        list: &[&str],
        collection: &str,
        name: &str,
    ) -> Result<Option<serde_json::Value>>
}

Then find_listed in cli.rs becomes cx.cli_find_listed(&["sandbox", "list"], "sandboxes", name) plus reading phase, and both local copies of the constants go away. The next ports that need to check whether something is listed can use it too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in fe47492. It's now Context::cli_find_listed(list, collection, name) in testkit, with your signature. Both workspaces.rs and find_listed in cli.rs use it, so the local copies of PAGE_SIZE/MAX_PAGES are gone. I also added it to the testkit README. find_listed now errors when sandboxes is missing; the CLI always writes that key in --output json, even for an empty list, so wait_unlisted is unaffected.

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks good. I went through both old tests line by line, and every check is still covered. Several are tighter now: the Terminating check targets this workspace, the scoped listings check the owner, and the control before the rejected create is a nice touch. Switching from a provider to a template makes sense given the profile requirement.

Approving, but before merge I'd like the Terminating check after the refused delete in workspace_full_crud_lifecycle and the find_entry move into testkit. The other inline comments are nits. Could you also retitle the PR to "test(conformance): port workspace lifecycle tests to testkit" so the squash commit reads well?

expect_status(cx, &workspace, Some("Active")).await?;

create_template(cx, &workspace, &template).await?;
cx.cli(["workspace", "delete", &workspace])

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old test checked that this delete failed because of "still contains resources". Now any failure passes here, such as a bad flag or an auth error. Could you add expect_status(cx, &workspace, Some("Terminating")).await?; right after it? The gateway marks the workspace Terminating before it checks for resources, so that proves we reached that check without matching message text.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. workspace_full_crud_lifecycle now calls expect_status(cx, &workspace, Some("Terminating")) right after the refused delete.

//! workspace-scoped and covered by the workspace delete guard, and needs no
//! provider profile, credential or image.
//!
//! Known limitation: the CLI has no structured error and no structured

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: CliOutput::report_text() is there for matching error text (file_transfer uses it), so "the CLI has no structured error" isn't quite the reason. I'd point to the README rule about only asserting what the public contract promises. Also, these tests don't create sandboxes, so I don't think the operator-workspace note applies. Maybe drop it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The module doc now points to "What a test may assert" in the conformance README, and the operator-workspace note is gone.

create_template(cx, &workspace, &template).await?;

let scoped = template_owner(cx, &["--workspace", &workspace], &template).await?;
let default = template_owner(cx, &[], &template).await?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: testkit pins the CLI to OPENSHELL_WORKSPACE, so this lists the test's workspace, which is only "default" when that variable is unset. Worth saying "the test's workspace" in the doc comment on template_is_scoped_to_its_workspace and in the error message below.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the doc comment and the error message. I also renamed the variable from default to unscoped so it matches.

@politerealism politerealism changed the title test(conformance): migrate workspace lifecycle tests to workspace stories test(conformance): port workspace lifecycle tests to testkit Nov 17, 2026
Move e2e/rust workspace_lifecycle into a conformance workspaces suite on
#[testkit::test], split into workspace_full_crud_lifecycle,
template_is_scoped_to_its_workspace and workspace_terminating_rejects_creates.
Use a sandbox template as the scoped resource so no provider profile is
needed, and assert on exit status and listed state rather than message text.
A refused workspace delete is checked to leave the workspace Terminating, so
the failure is the resource guard rather than an unrelated error.

Add Context::cli_find_listed to testkit to find an entry in a paginated
--output json listing, erroring when the collection key is missing. Use it
in the workspaces suite and in conformance/src/cli.rs, and list it in the
testkit README.

Drop the legacy test, its Cargo target, and its Podman archive exclusion.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
@politerealism

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I've addressed everything in fe47492 and retitled the PR as you suggested. It's still a single commit. CI needs /ok to test fe474928 to run again.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. test:e2e Requires end-to-end coverage topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(e2e): port workspace lifecycle tests from e2e/rust to conformance

3 participants