Repository navigation
test(conformance): port workspace lifecycle tests to testkit - #4607
politerealism wants to merge 1 commit into
Conversation
f5ff370 to
a2554c4
Compare
|
This pull request has had no activity for 14 days and is now marked stale. |
Move e2e/rust workspace_lifecycle into a conformance workspaces suite on #[testkit::test], split into workspace_full_crud_lifecycle, template_is_scoped_to_its_workspace and workspace_terminating_rejects_creates. Use a sandbox template as the scoped resource so no provider profile is needed, and assert on exit status and listed state rather than message text. Drop the legacy test, its Cargo target, and its Podman archive exclusion. Supersedes NVIDIA#4607. Signed-off-by: Evan Lezar <elezar@nvidia.com>
a2554c4 to
30f47a9
Compare
Maintainer list changeGains approval rights: Confirm every change is intended. Anyone listed here can single-handedly satisfy |
Move e2e/rust workspace_lifecycle into a conformance workspaces suite on #[testkit::test], split into workspace_full_crud_lifecycle, template_is_scoped_to_its_workspace and workspace_terminating_rejects_creates. Use a sandbox template as the scoped resource so no provider profile is needed, and assert on exit status and listed state rather than message text. Drop the legacy test, its Cargo target, and its Podman archive exclusion. Supersedes NVIDIA#4607. Signed-off-by: Evan Lezar <elezar@nvidia.com>
30f47a9 to
d63af4f
Compare
|
Label |
|
/ok-to-test d63af4f |
| /// The entry called `name` in a paginated `--output json` listing, where | ||
| /// `collection` is the JSON key that holds the entries. A listing without that | ||
| /// key is an error, so a "not listed" check cannot pass on a malformed listing. | ||
| async fn find_entry( |
There was a problem hiding this comment.
This is the same paging loop as find_listed in cli.rs (same PAGE_SIZE and MAX_PAGES too), just made generic. Your version is the better one, since it errors when the collection key is missing instead of treating that as "not listed".
Could we move it into testkit in this PR rather than keep two copies? Something like:
impl Context {
/// The entry called `name` in a paginated `--output json` listing from
/// `list` (e.g. `["sandbox", "template", "list", "--all-workspaces"]`),
/// where `collection` is the JSON key holding the entries.
pub async fn cli_find_listed(
&self,
list: &[&str],
collection: &str,
name: &str,
) -> Result<Option<serde_json::Value>>
}Then find_listed in cli.rs becomes cx.cli_find_listed(&["sandbox", "list"], "sandboxes", name) plus reading phase, and both local copies of the constants go away. The next ports that need to check whether something is listed can use it too.
There was a problem hiding this comment.
Done in fe47492. It's now Context::cli_find_listed(list, collection, name) in testkit, with your signature. Both workspaces.rs and find_listed in cli.rs use it, so the local copies of PAGE_SIZE/MAX_PAGES are gone. I also added it to the testkit README. find_listed now errors when sandboxes is missing; the CLI always writes that key in --output json, even for an empty list, so wait_unlisted is unaffected.
matthewgrossman
left a comment
There was a problem hiding this comment.
Thanks, this looks good. I went through both old tests line by line, and every check is still covered. Several are tighter now: the Terminating check targets this workspace, the scoped listings check the owner, and the control before the rejected create is a nice touch. Switching from a provider to a template makes sense given the profile requirement.
Approving, but before merge I'd like the Terminating check after the refused delete in workspace_full_crud_lifecycle and the find_entry move into testkit. The other inline comments are nits. Could you also retitle the PR to "test(conformance): port workspace lifecycle tests to testkit" so the squash commit reads well?
| expect_status(cx, &workspace, Some("Active")).await?; | ||
|
|
||
| create_template(cx, &workspace, &template).await?; | ||
| cx.cli(["workspace", "delete", &workspace]) |
There was a problem hiding this comment.
The old test checked that this delete failed because of "still contains resources". Now any failure passes here, such as a bad flag or an auth error. Could you add expect_status(cx, &workspace, Some("Terminating")).await?; right after it? The gateway marks the workspace Terminating before it checks for resources, so that proves we reached that check without matching message text.
There was a problem hiding this comment.
Done. workspace_full_crud_lifecycle now calls expect_status(cx, &workspace, Some("Terminating")) right after the refused delete.
| //! workspace-scoped and covered by the workspace delete guard, and needs no | ||
| //! provider profile, credential or image. | ||
| //! | ||
| //! Known limitation: the CLI has no structured error and no structured |
There was a problem hiding this comment.
Nit: CliOutput::report_text() is there for matching error text (file_transfer uses it), so "the CLI has no structured error" isn't quite the reason. I'd point to the README rule about only asserting what the public contract promises. Also, these tests don't create sandboxes, so I don't think the operator-workspace note applies. Maybe drop it?
There was a problem hiding this comment.
Fixed. The module doc now points to "What a test may assert" in the conformance README, and the operator-workspace note is gone.
| create_template(cx, &workspace, &template).await?; | ||
|
|
||
| let scoped = template_owner(cx, &["--workspace", &workspace], &template).await?; | ||
| let default = template_owner(cx, &[], &template).await?; |
There was a problem hiding this comment.
Nit: testkit pins the CLI to OPENSHELL_WORKSPACE, so this lists the test's workspace, which is only "default" when that variable is unset. Worth saying "the test's workspace" in the doc comment on template_is_scoped_to_its_workspace and in the error message below.
There was a problem hiding this comment.
Fixed in the doc comment and the error message. I also renamed the variable from default to unscoped so it matches.
Move e2e/rust workspace_lifecycle into a conformance workspaces suite on #[testkit::test], split into workspace_full_crud_lifecycle, template_is_scoped_to_its_workspace and workspace_terminating_rejects_creates. Use a sandbox template as the scoped resource so no provider profile is needed, and assert on exit status and listed state rather than message text. A refused workspace delete is checked to leave the workspace Terminating, so the failure is the resource guard rather than an unrelated error. Add Context::cli_find_listed to testkit to find an entry in a paginated --output json listing, erroring when the collection key is missing. Use it in the workspaces suite and in conformance/src/cli.rs, and list it in the testkit README. Drop the legacy test, its Cargo target, and its Podman archive exclusion. Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: politerealism <burdcat17@gmail.com>
d63af4f to
fe47492
Compare
|
Thanks for the review. I've addressed everything in fe47492 and retitled the PR as you suggested. It's still a single commit. CI needs |
Summary
Move the workspace lifecycle tests from the legacy e2e crate into the testkit conformance suite. Workspace create, scoping and delete behavior isn't driver-specific, so the tests run against any prepared gateway.
Related Issue
Closes #5598. Part of #5281.
Changes
tests/suites/conformance/src/workspaces.rswith three#[testkit::test]tests, and aworkspacesrow in the conformance README:workspace_full_crud_lifecycle: create, get, listed as Active, a delete refused while it holds a resource leaves it Terminating, delete when empty, gone afterwards. The original checked seven behaviors in one test, so it's split. The name is kept so filters keep working.template_is_scoped_to_its_workspace(new): a resource is listed in its own workspace and in the all-workspaces view with its owner, and isn't visible in the test's workspace.workspace_terminating_rejects_creates: a refused delete leaves the workspace Terminating. New resources are rejected, existing ones can still be deleted, and retrying the delete then succeeds.Context::cli_find_listedto testkit: finds an entry in a paginated--output jsonlisting and errors when the listing lacks the collection key.find_listedinconformance/src/cli.rsnow uses it, and the testkit README lists it.e2e/rust/tests/workspace_lifecycle.rs, its Cargo target and its entry in the Podman archive exclusion list.Testing
ubuntu-k3sconformance row in CI (passed before the review changes; rerun pending).Checklist