[VisruthSK]

This PR assisted by Claude Code.

Submission Checklist

• Run unit tests
• Declare copyright holder and agree to license (see below)

Summary

Trying to run tests in parallel. Closes #1025.

Copyright and Licensing

Please list the copyright holder for the work you are submitting
(this will be you or your assignee, such as a university or company):
Visruth Srimath Kandali

By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses:

• Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
• Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[VisruthSK]

Need to measure test times, try 4 workers (IIRC default is 2), check which tests should be run first, and rebase once v1.0 is merged

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (v1.0@913494b). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             v1.0    #1325   +/-   ##
=======================================
  Coverage        ?   94.00%           
=======================================
  Files           ?       19           
  Lines           ?     9232           
  Branches        ?        0           
=======================================
  Hits            ?     8679           
  Misses          ?      553           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jgabry]

Nice, giving each worker its own copy of CmdStan is simpler than how I imagined dealing with #1025. I like it. I won't do a thorough review until you're ready for that but a few things I noticed that will affect the test timings you're going to measure:

• Based on https://testthat.r-lib.org/articles/parallel.html#setup, Config/testthat/start-first: test-install.R needs to be Config/testthat/start-first: install

• When copying CmdStan use file.copy(..., copy.date = TRUE). Without that the copies will get new timestamps and I think make will rebuild the precompiled headers in every worker.

• Setting CMDSTAN in setup.R doesn't actually switch the path by itself because cmdstanr caches it when the package is loaded. We need to call set_cmdstan_path() again if not reloading the package. So after the Sys.setenv() I think just do suppressMessages(set_cmdstan_path()) and that should work.

• I think WSL will break. I'm not a WSL expert, but CMDSTAN there is a path like //wsl$/... and copying it into the Windows tempdir will mean there's a Linux build at a Windows path. So I think we need to copy it so it stays inside the distro or we need to keep running the WSL job serially. But I'm not 100% sure about this.

[VisruthSK]

Thanks for reading this already! Will incorporate these, that timestamp thing is a good catch and should cut down on times further hopefully. I want a better measurement system to know if this branch is even faster too.

[jgabry]

Disclaimer: these review comments are a mix of me, Claude, and Codex. I didn't bother to separate the writing out, it's all a bit of a mix.

This is in good shape, and the timings look great!

First some bigger things, then smaller things, then a few questions.

1. Every R CMD check job now has a NOTE: "checking for detritus in the
   temp directory ... NOTE: 'cmdstanr-warmup'". setup.R:32 creates the
   lock directory under dirname(tempdir()) and nothing removes it.
   Easiest is to put the lock file somewhere that
   isn't the check's temp directory, for example inside the
   installation next to the existing make/local.cmdstanr-test-backup.
   Also, release it with withr::defer() instead of the bare
   filelock::unlock() at line 54, so an error in the warm-up doesn't
   leave the other workers waiting.

2. Two workers can write the threads build products into the shared
   installation at the same time. The warm-up (setup.R:39-53) builds
   the plain and threads precompiled headers, but the first threads
   model build also writes src/cmdstan/main_threads.o,
   src/cmdstan/main_threads.d and the header's dependency file
   model_header_threads_17_0.d next to the .gch, none of which the
   install or the header target produces (make/program only includes
   the .d files when it has a model target, and install_cmdstan()
   builds the plain set through the bernoulli example). My install has
   main.o and model_header_17_0.d and nothing for threads.
   test-threads.R, test-build-info.R and test-model-rebuild.R compile
   real threads models against the shared install, so whichever two
   get there first race on those files. With clang the compiler writes
   to a temp file and renames, so on macOS it's just duplicated work.
   But with gcc the output path is written directly, so one worker can read a
   half-written .d or link a truncated .o. CI hasn't hit it. The
   fix fits the existing loop: for STAN_THREADS=true also print
   CMDSTAN_MAIN_O and build it together with the header's .d
   (the header path with .hpp.gch replaced by .d). (The
   plain-header iteration is normally a no-op, since make build
   already makes that PCH.)

3. test-utils.R:177 and :185 delete bin/stansummary and
   bin/diagnose from the shared installation and let
   $cmdstan_summary() rebuild them. Under parallel mode any other
   worker calling $cmdstan_summary() or $cmdstan_diagnose() in that
   window (test-fit-shared.R, test-failed-chains.R,
   test-installation-check.R, test-model-sample.R) sees a missing
   binary or a concurrent make bin/stansummary. These two tests need
   local_private_cmdstan() too. The file-level fit_mcmc still
   works, since the tools resolve cmdstan_path() at call time.

4. The os_is_wsl = function() FALSE mock at test-install.R:1494 and
   the .cmdstanr$WSL <- line at helper-envvars-and-paths.R:185 both
   work around the same thing, and it's a bug in the package rather
   than in the tests. install_cmdstan(wsl = FALSE) sets the WSL flag
   to FALSE before check_install_dir() can return early, so after
   "install_cmdstan() errors if installation already exists"
   (test-install.R:34) the WSL job runs the rest of that file with
   os_is_wsl() FALSE while cmdstan_path() is still //wsl$/....
   That's why the toolchain test only ever passed on the WSL job by
   accident (it took the Rtools branch), why run 38794416338 failed
   there once the private copy reset the flag, and why run 38818811363
   got "Command 'bin/stanc.exe' not found" when the copy was made with
   the flag stale. A WSL user whose install_cmdstan() call returns
   early has the same problem until they call set_cmdstan_path().
   I'll fix that on v1.0. Once it's in, you can merge v1.0, drop line
   185, and replace the mock with skip_if(os_is_wsl()) (the Windows
   jobs cover the Rtools branch, and we don't commit mocks of
   os_is_wsl()).

5. The private copies may not be cleaned up. On macOS, Linux and
   Windows the copy lives in the worker's tempfile() (line 187) and
   relies on R removing the tempdir at exit; on WSL it's unlinked at
   teardown_env() (lines 189-193). But testthat gives a worker about
   one second after teardown and then kills it, so neither the WSL
   unlink of 23k files over the \\wsl$ share nor R's removal of a
   500 MB tempdir is likely to finish. That's from reading testthat's
   queue_teardown(), not observed, so could you check what is left in
   $TMPDIR after one local devtools::test()? If copies do pile up,
   on WSL wsl_tempdir() (R/utils.R) with wsl_safe_path(revert = TRUE) would at least put the copy in the distro's /tmp rather
   than its home directory (it registers no deletion itself), and it
   avoids the dirname() on a //wsl$ path (which turns the leading
   // into \\, see #1316).

6. A failed copy falls back to the shared installation with only a
   warning. private_cmdstan$path is set (line 195) before the copy
   runs, and neither cp (line 220) nor robocopy (line 205, exit codes
   8 and up are failures) is checked (the WSL branch is, through
   processx::run()). If the copy fails, set_cmdstan_path() warns
   "Can't find directory", the path stays on the shared install, and
   the test writes the shared make/local, which is what the helper
   exists to prevent; a partial copy fails later with a makefile
   error instead. Either way the cached path means the worker never
   retries. Set the cache after a successful copy and stop on failure.

7. The warm-up now runs for every NOT_CRAN=true test run against the
   developer's real installation, including test_file() on a pure-R
   file. On Linux that can append -fPIC to their make/local and run
   a full clean-all + build with no message at all (verbose = FALSE, and check_sundials_fpic()'s own messages only print when
   interactive), and rebuild_cmdstan() ignores make's exit status, so
   a failed rebuild is silent too. On every platform it builds the
   threads PCH into the installation if missing. I think that's
   acceptable, since the model-methods tests already did the -fPIC
   rebuild and any threads compile builds that PCH, but it should get a
   comment at the top of setup.R saying what the warm-up does and why
   and maybe a message() before a rebuild would help.

8. Test-coverage.yaml sets TESTTHAT_CPUS: 4 but still installs
   CmdStan without -fPIC, so on that job the first worker's setup
   rebuilds CmdStan while the other three wait on the lock. The same
   cpp_options as in R-CMD-check.yaml would be consistent.

Smaller things:

9. Prose the change has made stale: tests/testthat/README.md:35-36
   still says setup.R cleans .stan files under resources/stan/, and
   nothing in it mentions parallel mode, TESTTHAT_CPUS, the per-worker
   tempdir()/stan copy, the private installation copy, the warm-up,
   or that files must not depend on order. test-model-rebuild.R:5-7
   talks about "the shared copy under resources/stan/". The comment
   above local_cmdstan_make_local() (helper:168-170) still says it
   writes the make/local of the current installation, and its "called
   at the top level of a test file" sentence would now mean the whole
   file runs on the private copy and every file-level model rebuilds
   with "the selected CmdStan changed". local_private_cmdstan()
   itself has no comment. And setup.R:1 still says "CI runners have
   four cores, so run two chains at a time", which with four workers is
   eight chains plus compiles; fine if the timings say so, but the
   comment should say what the arithmetic is now.

10. Some edits are left over from the shared-fixture design and
    change nothing: cmdstan_model(stan_program) to testing_model()
    in test-failed-chains.R:1-6, test-fit-shared.R:693 and :730,
    test-installation-check.R:335 and :342 (where stan_program is
    still used at :346). I'd revert them to keep the diff to the
    parallel change.

11. setup.R can use stan_fixture() for stan_dir (helpers are
    sourced before setup files).

Questions:

a. Commit e5a202e builds the precompiled headers only when the file
is missing. If make ever considers an existing PCH stale (after a
make/local edit, say), the file.exists() guard skips the
warm-up and the workers rebuild it concurrently. What was the
failure that led to that commit?

b. Did you get to the timing measurements? The start-first list has
16 files and nothing in the logs says which are the slow ones.
testthat's SlowReporter would give per-file times.

c. On run 39047720495, macos-15-intel reports PASS 3525 and macos-26
PASS 3524 with the same SKIP and WARN, and the serial runs had 3527
on both. A one-expectation difference between two configurations
that should agree suggests an expectation that depends on worker
state or order. Do you know which?

d. The install test on WSL now also does the rebuild_cmdstan() on an
install under /mnt/c (the Windows tempdir), where the old "clean
and rebuild works" rebuilt the distro-local shared install. That
removes the race on the shared install, which is good, but it may
make test-install.R the WSL critical path. Worth a look at its time
in the WSL log.

[jgabry]

I'll fix that on v1.0. Once it's in, you can merge v1.0, drop line
185, and replace the mock with skip_if(os_is_wsl()) (the Windows
jobs cover the Rtools branch, and we don't commit mocks of
os_is_wsl()).

See #1336

[jgabry]

Also see #1338, not sure if that interacts with this PR. Going to merge that one when I get CI passing. (Update: merged)