Skip to content

ci: reduce flaky tests and run Node and extension tests in parallel - #1943

Merged
eitsupi merged 7 commits into
mainfrom
ci/reduce-flaky-tests
Nov 14, 2026
Merged

eitsupi merged 7 commits into
mainfrom
ci/reduce-flaky-tests

Conversation

@eitsupi

@eitsupi eitsupi commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Over the past month, the main workflow's test jobs failed often on single OSes, and macOS took a median of 12 minutes because every suite ran back to back. This PR fixes the most frequent flaky failures that need only test changes, and splits each OS into two parallel jobs.

Flaky test fixes

  • Windows Verify managed jgd preparation (most frequent; every inspected failure is Permission denied on profile-declined/startup.status): the test polled the status file every 10 ms while the child replaced it with file.rename. On Windows, an open reader makes the rename fail, and a reader that races the rename gets Permission denied. The child now publishes a completion marker atomically after profile.R returns, and the parent reads the final status once.
  • Windows attach_sess.R parallel_approve: the follower timed out after 20 seconds, and the test timed out waiting 10 seconds for the owner's install. Both budgets cover the owner's whole source install, which is slow on Windows runners. Raise them to 60 seconds (and the outer child wait to 90 seconds). Passing runs take the same time.
  • Session Terminal Lifecycle (1 !== 0, 4 !== 0): clock.countTimers() counts every fake timer in the extension host, including timers from VS Code and unrelated extension code. A new trackModuleTimers test helper counts only timeouts scheduled from the session and terminal modules and checks that they are cleared.
  • Interactive language server tests on macOS: passing runs sometimes take up to about 12 seconds per lint, against a 20-second budget. A timeout also left the lint fixture in the shared Interactive input, so later tests failed with it. Diagnostic and client-startup waits now allow 45 seconds and report the document text, version, client state and current diagnostics on timeout. The input lint test always clears the input. The bulk session setup, which starts four arf sessions serially, gets its own 180-second timeout.

Faster CI

  • Replace the single per-OS test job with two parallel jobs per OS: Node (OS) runs the R integration checks, sess tests, tmux and Node tests, and Extension (OS) runs the plot lifecycle and extension-host tests. On the latest successful macOS run, the extension-host part alone took about 4 minutes after roughly 2 minutes of setup, compared with 8.5 minutes for the whole job. Setup now runs twice per OS, so total runner minutes go up. main has no required status checks, so the new job names do not block merges.
  • Share the setup through two local composite actions: setup-test-tools (R, pnpm, Node.js and npm dependencies) and prepare-test-runtime (R test dependencies, sess build and install, test compilation and arf). The sess bootstrap checks still run between the two, before R test dependencies are installed.
  • Re-include .github/actions/** in the path filter, so changes to the composite actions alone still run the tests.
  • Let Dependabot scan /.github/actions/* as well, since directory: / covers only workflows and a root action.yml.

Not included

  • The arf adoption interrupt test in the Node suite: fixing it needs product changes (arf 0.7.0's IPC interrupt instead of SIGINT), so it will be a separate PR.
  • Installing arf through mise with a lock file instead of the hand-written download and checksum script: planned together with the arf 0.7.0 PR.
  • Shortening the real 10-second wait in the CSP webview test would need a timeout seam in src/webViewer. It saves only about 10 seconds per run.

Wait for the jgd startup profile child to publish completion before
reading its status file instead of polling a file the child replaces by
rename, which fails on Windows with Permission denied or blocks the
rename. Give the parallel sess setup follower and installer fixture
budgets that cover a slow Windows source install.
Fake timers in the extension host also capture timers from VS Code and
unrelated extension code, so clock.countTimers() intermittently reported
foreign timers. Track timeouts scheduled from the session and terminal
modules and assert that those are cleared.
Wait up to 45 seconds for languageserver diagnostics and client startup,
and report the document text, version, client state and current
diagnostics on timeout instead of a generic notebook execution message.
Always clear the shared Interactive input after the lint test so a
timeout does not cascade into later tests, and give the bulk session
setup, which starts four arf sessions serially, its own timeout.
Each OS ran R integration checks, Node tests and extension-host tests
back to back in one job, which made macOS the bottleneck. Add a suite
matrix dimension so R integration and Node tests run in one job and
extension-host tests in another, after the same R, sess and arf setup.
@eitsupi
eitsupi requested a review from renkun-ken November 14, 2026 01:03
@eitsupi
eitsupi marked this pull request as draft November 14, 2026 01:07
Replace the suite matrix dimension and per-step conditions with
separate test-node and test-extension jobs. Move the shared setup into
two local composite actions: setup-test-tools installs R, pnpm and
Node.js, and prepare-test-runtime installs R test dependencies, builds
and installs sess, compiles the tests and installs arf. sess bootstrap
checks still run between the two, before R test dependencies exist.

Re-include .github/actions in the path filter, and let Dependabot
update actions pinned inside the composite actions.
@eitsupi
eitsupi marked this pull request as ready for review November 14, 2026 01:17

@renkun-ken renkun-ken left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 6d1c0a896cfa4437af8d768b58a499383e0766ed. The job split preserves the bootstrap-before-dependency-install order and existing suite coverage. The Windows completion marker and scoped timer tracking look sound. One issue remains: the new diagnostic wait budgets can exceed the enclosing Mocha test timeout and let shared-input cleanup run during a later test; see the inline comment.

Validation on macOS:

  • Build/test compilation, TypeScript checking, and lint/format checks for the changed TypeScript files passed.
  • R/tests/attach_sess.R and R/tests/prepare-jgd.R passed.
  • All 28 terminal lifecycle tests passed in VS Code 1.145.2. An intentionally leaked pending timeout in a temporary copy was detected by the new tracker.
  • All six affected Interactive language-service tests plus their notebook-creation prerequisite passed with the pinned arf 0.5.3 runtime (7 tests).
  • A deterministic fake-clock probe using the compiled diagnostic wait helper reproduced the 60-second timeout and next-test-before-cleanup ordering.
  • All 11 hosted checks are green on this commit.

…imeout

Tests that wait for languageserver startup or diagnostics now raise their
timeout to cover every sequential 45-second wait. Each wait is also capped
by the running test's deadline minus a cleanup reserve, so a slow wait fails
inside its own test and its cleanup cannot run during a later test that
shares the Interactive input.
@eitsupi
eitsupi requested a review from renkun-ken November 14, 2026 04:53

@renkun-ken renkun-ken left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed d789fe9a2b57b32888bd1cb531a0c2a292243077. The previous P2 finding is addressed: the six affected language-service tests now budget for all sequential waits, and the shared wait helper reads the current test timeout and caps its deadline with a five-second cleanup reserve. I found no additional actionable issues in the update.

Validation:

  • Build/test compilation and lint/format checks passed.
  • All six affected Interactive language-service tests and their notebook-creation prerequisite passed in VS Code 1.145.2 with the pinned arf 0.5.3 runtime (7 tests).
  • Eight deterministic probe checks using the compiled helper passed: the original 35s + 35s case, four diagnostic waits just below 45s each, the enclosing-deadline cap, and the individual diagnostic timeout. In each case, the following test observed completed cleanup.
  • All 11 hosted checks are green on this commit.

Approving the updated PR.

@eitsupi
eitsupi merged commit 41964fd into main Nov 14, 2026
11 checks passed
@eitsupi
eitsupi deleted the ci/reduce-flaky-tests branch November 14, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants