Repository navigation
Conversation
…rows Program rows relied on the program's 900 s default, so one hung test could use much of the 45-minute run timeout. Derive the per-test timeout from the same constants as the nextest profile (120 s) and pass it with a 10 s grace to the program. Part of #5416. Signed-off-by: Evan Lezar <elezar@nvidia.com>
A hung CLI call only showed nextest's generic timeout, with no command name or output, and the gateway log tail started after the stalled test. - The runner exports the per-test limit as OPENSHELL_TEST_TIMEOUT_SECS. testkit caps its CLI, exec, wait and poll timeouts to the time left, so a stuck step fails with its own error. Cleanup keeps its own timeouts. - A CLI command that times out reports its output so far. A command that is slow, exits non-zero, or is still running is reported while the test runs. - The k3s diagnostics collect gateway logs from the run's start time, with a 30 minute fallback, instead of the last 300 lines. Part of #5333. Signed-off-by: Evan Lezar <elezar@nvidia.com>
Karthik-Chowdary
left a comment
There was a problem hiding this comment.
At 23edc35, the new per-test cap can expire while poll_until is still inside an observation: it caps the duration but only checks it after observe().await returns.
There is an existing caller affected by that mismatch: tests/support/keycloak/src/lib.rs::wait_ready runs curl --max-time 10 inside the observation. Its comment explicitly relies on the old between-attempt deadline checks, but a remaining test budget can now be shorter than that fixed 10-second request.
I reproduced the polling path with that current curl invocation against a loopback HTTP listener that accepted the discovery request and withheld its response. With OPENSHELL_TEST_TIMEOUT_SECS=6 (a 1-second step budget after the margin), the helper was still awaiting curl at 2 seconds rather than returning its own contextual timeout. The ready-result control passed. A local correction that bounds the observation by the remaining deadline and clamps the retry sleep returned the Keycloak realm at ... timeout at about 1 second; the pending-result control retained its last observation.
Could the capped deadline cover the in-flight observation as well as the retry interval, preserving the last completed observation in the error? Otherwise a late readiness attempt can still reach the runner's generic TIMEOUT before the helper reports which wait stalled. This is a polling/diagnostic check, not a Keycloak or full e2e qualification. No change to authentication behavior is being suggested.
…t-stalls/elezar Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> # Conflicts: # tests/runner/src/bundle.rs # tests/testkit/testkit/src/cli.rs # tests/testkit/testkit/src/context.rs
matthewgrossman
left a comment
There was a problem hiding this comment.
#5450 has merged, so this PR now targets main. I merged main into this branch as 900ba57 (a normal merge commit, no force push). The conflict resolutions:
tests/runner/src/bundle.rs:pub const TEST_TIMEOUT_SECS = SLOW_PERIOD_SECS * SLOW_TERMINATE_AFTERandpub const TEST_GRACE_SECS = 10live here, and the generated nextest profile uses them. The test checks the fullslow-timeoutline and the 120 s value.tests/runner/src/execute.rs: passes--timeout/--graceto the program from those constants, next toOPENSHELL_TEST_TIMEOUT_SECS, and drops its local grace constant. The program's 120 s limit that #5450 first carried now lands here, together with the deadline that makes it safe.tests/testkit/testkit/src/cli.rs: keeps this PR's spawn-based handling.tests/testkit/testkit/src/context.rs:crate::budget::release()followed by #5431'sself.handles.cleanup.run(CLEANUP_TIMEOUT).
fmt, clippy and the unit tests pass for tests/runner and tests/testkit, and the conformance suite compiles. Nothing from either side was dropped.
Review notes on the change itself, most important first:
1. The stuck step's error is printed only after cleanup (medium-high). In run.rs run_in, the body's error is only shown by the final panic!, after cx.cleanup(). Cleanup lifts the cap and can wait up to CLEANUP_TIMEOUT (2 min) per sandbox delete. If a step times out at about 115 s and the delete takes more than about 15 s, the runner kills the test first and the output is the generic TIMEOUT again, which is what this PR sets out to fix. Suggest eprintln! the error (or panic message) before cx.cleanup() and keep the final panic. Optionally give cleanup a deadline that fits inside the grace period.
2. Some waits aren't capped (medium).
Sandbox::delete()waits onwait_deleted(.., DELETE_TIMEOUT, ..), 2 min, withoutbudget::cap.Sandbox::wait_absent(), added onmainby #5431, is also uncapped.lifecycle.rspasses it 60 s.- Single gRPC calls have no timeout. In
poll_until, the timeout is only checked afterobservereturns, so a hung call there never times out.
Suggest budget::cap on the first two, and a remaining-time tokio::time::timeout around observe (or a small budget helper for gRPC calls). Until then, the README's "the wait helpers give up" isn't quite true.
3. CLI timeout handling (low-medium). In cli.rs, stdout.finish() and stderr.finish() are awaited one after the other. With a grandchild holding a pipe, that can take 4 s of the 5 s margin. child.kill() only signals the direct child, so anything it started keeps running. Suggest tokio::join! for the captures, and process_group(0) plus a group kill as in program/process.rs.
Smaller items:
- The budget clock starts at
budget::start(), not at process spawn, so the real margin is a little under 5 s. Worth a note in the comment. FLOORgives every capped step at least 1 s, even after the deadline. Fine for tests that stop at the first error.- If
datefails, a stale/var/tmp/openshell-run-startedfrom an earlier run on a reused VM sets the wrong log window. Suggestrm -fbefore writing it. The 64 KiB head is mostly startup logs and may not be worth keeping. tests/suites/conformance/src/lifecycle.rs:36-39still calls the exec timeout "the runner's two-minute kill".- Timed-out CLI errors now include up to 2 KiB of stdout and stderr. A CLI that prints a credential would leak it into test output.
Items 1 and 2 seem worth fixing before merge. The rest can be follow-ups.
poll_until now bounds the observation in flight as well as the retry sleep, and keeps the last completed observation in its timeout error. run_in prints the body's error before cleanup so a slow cleanup cannot hide it behind the runner's generic timeout. Sandbox::delete and wait_absent respect the test budget, and the policy advisor retry loops use poll_until. The runner removes a stale run-started marker before writing a new one. Signed-off-by: Evan Lezar <elezar@nvidia.com>
|
Label |
Stacked on #5450. Review that PR first; this diff is against its branch.
Summary
A hung CLI call in a conformance lane showed only nextest's generic TIMEOUT, with no command name or output, and the gateway log tail began after the stalled test.
OPENSHELL_TEST_TIMEOUT_SECS, from the same constant as the nextest profile and the program's--timeout. testkit caps its CLI, sandbox exec, sandbox wait and poll timeouts to the time left minus 5 s, so a stuck step fails with its own error. Cleanup keeps its own timeouts. Without the variable, nothing is capped.--since-time, written by the runner in the guest), with a 30 minute fallback, keeping the first 64 KiB and last 512 KiB.Related Issue
Part of #5333. Not covered here, so the issue stays open:
junit.xmlis still lost on a run-level timeout; that fits test(runner): follow-ups to running a prebuilt testkit program on runner rows #5416.Testing
cargo fmt --check,cargo clippy --all-targets -- -D warningsandcargo testpass fortests/testkitandtests/runner.collect-k3s.shwas only syntax-checked (bash -n), and no e2e lane was run.