Skip to content

test(runner): generate the nextest profile from shared timeout constants - #5450

Merged
matthewgrossman merged 2 commits into
mainfrom
test/5416-program-row-timeout/elezar
Nov 16, 2026
Merged

matthewgrossman merged 2 commits into
mainfrom
test/5416-program-row-timeout/elezar

Conversation

@elezar

@elezar elezar commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Summary

The runner wrote its nextest profile as a hardcoded string. Generate it from named constants (a 30 s slow period, terminated after 4 periods, with a 10 s grace period) so #5455 can share the per-test limit with testkit and the program rows. The grace period is now explicit instead of relying on nextest's default.

Program rows keep the program's 900 s default here. Passing them the runner's 120 s limit would stop a test before testkit's own step timeouts (up to 10 minutes) can report which step stalled. #5455 caps those timeouts to the time a test has left, so it passes --timeout and --grace to the program in the same change.

Related Issue

Part of #5416. The first item (an explicit per-test timeout for program rows) stays open until #5455 lands. Groundwork for #5333.

Changes

  • Generate the nextest profile from SLOW_PERIOD_SECS, SLOW_TERMINATE_AFTER and GRACE_PERIOD_SECS, and write grace-period explicitly.
  • Add a test for the generated slow-timeout line.
  • Correct the RUN_TIMEOUT comment: program rows are bounded by the program's own run --timeout default.

Testing

  • cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test pass for tests/runner.
  • cargo-nextest 0.9.143, the version the guests install, accepts the generated profile and rejects an invalid grace-period.
  • The full e2e lane was not run. Apart from the explicit grace period, which matches nextest's default, test behavior is unchanged.

…rows

Program rows relied on the program's 900 s default, so one hung test could use much of the 45-minute run timeout. Derive the per-test timeout from the same constants as the nextest profile (120 s) and pass it with a 10 s grace to the program.

Part of #5416.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. cli — No explicit risk acceptance found

openshell-test-runner run --program changes its child invocation from test-program run --junit PATH (900-second default) to test-program run --timeout 120 --grace 10 --junit PATH.

  • Impact: Existing prebuilt testkit suites with tests requiring 120–900 seconds now receive SIGTERM and report timeout failures with exit code 1, without a runner flag to restore their previous budget.
  • Action: Preserve the 900-second --program default in v0.1.8 and make the 120-second limit opt-in through a documented runner flag.
  • PR: test(runner): generate the nextest profile from shared timeout constants #5450
Review details
  • Execution: complete
  • Candidate: PR #5450 (29a08e491d961743824200fac035e389f911d875)
  • Baseline: 7873aaa3b8a3529459034cfce3079077a1a1e67b (7873aaa3b8a3529459034cfce3079077a1a1e67b)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@elezar

elezar commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

Compatibility risk acceptance: The changes to openshell-test-runnner are to an internal-only tool and the API is not considered public.

@elezar elezar added area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 10, 2026
@purp purp added meta:labeler-checkpoint Temporary label used by /label-issue to record a verified classification checkpoint. and removed meta:labeler-checkpoint Temporary label used by /label-issue to record a verified classification checkpoint. labels Nov 15, 2026
Passing the runner's 120 s limit to the program would stop a test before
testkit's own step timeouts (up to 10 minutes) can report which step
stalled, contrary to the program's documented contract. Keep program rows
on the program's default until testkit learns the per-test deadline
(#5333).

Write nextest's grace period into the generated profile from the same
constants, rather than relying on nextest's default.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman changed the title test(runner): pass an explicit per-test timeout and grace to program rows test(runner): generate the nextest profile from shared timeout constants Nov 16, 2026

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the slimmed-down version. I pushed accad2d to this branch, and here is why.

What changed. As first written, this PR passed the runner's 120 s limit to the conformance program, while testkit's own step timeouts are longer (sandbox ready 10 min, CLI 5 min, stop 5 min, policy_advisor readiness 4 min). If this merged without #5455, a slow step would be killed with a generic timeout instead of failing with testkit's error, and the program's --timeout help ("Longer than testkit's own step timeouts, so those report first") would no longer hold. To make this PR safe to merge on its own, accad2d:

  • drops --timeout/--grace from the program command, so program rows keep the 900 s default for now;
  • keeps the nextest profile generated from the shared constants, and writes grace-period = "10s" explicitly instead of relying on nextest's default;
  • removes TEST_TIMEOUT_SECS and TEST_GRACE_SECS for now, since nothing in this PR uses them;
  • updates the RUN_TIMEOUT comment and the test.

I've updated the title and description to match.

Plan for #5455. Once this lands, GitHub retargets #5455 to main. We'll merge main into its branch with a regular merge commit (no force push) and move the program limit there, alongside the deadline it adds:

  • add pub const TEST_TIMEOUT_SECS: u64 = SLOW_PERIOD_SECS * SLOW_TERMINATE_AFTER; back to bundle.rs, and make the grace constant shared;
  • pass --timeout/--grace to the program in execute.rs, next to OPENSHELL_TEST_TIMEOUT_SECS;
  • move the assert_eq!(TEST_TIMEOUT_SECS, 120) check there.

That way program rows get the 120 s limit in the same change that makes testkit's waits fit inside it. #5458 then needs main (or #5455) merged in the same way.

@matthewgrossman
matthewgrossman added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit b62ec53 Nov 16, 2026
92 checks passed
@matthewgrossman
matthewgrossman deleted the test/5416-program-row-timeout/elezar branch November 16, 2026 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants