BUG: make a failing Monte Carlo worker say so

[thc1006]

Three ways a failing Monte Carlo worker gets away without saying so. One hangs the run, one reports success with results it never wrote, and one hangs it again from a level down.

Stacked on #1181, which is the parallel run being unable to start at all. This branch carries that commit, so the diff here shows it too; the work of this pull request is the two commits on top. Happy to rebase once the other one lands.

Pull request type

• Code changes (bugfix, features)

Checklist

• Tests for the changes have been added (if needed)
• Docs have been reviewed and added / updated
• Lint (ruff check / ruff format --check, pylint) has passed locally
• All tests have passed locally
• CHANGELOG.md — no action needed; an LLM workflow auto-updates it after merge, though see BUG: the changelog workflow stopped running, and CHANGELOG.md is 37 merged pull requests behind #1173 for why it currently does not.

Current behavior

A worker that fails early dies inside its own handler. __sim_producer binds sim_idx and inputs_json inside the simulation loop:

while sim_monitor.keep_simulating():
    sim_idx = sim_monitor.increment() - 1
    inputs_json, outputs_json = "", ""

and its except block reads both. A worker that fails in the seeding above the loop, or in the claim that opens it, arrives there with neither name assigned:

UnboundLocalError: cannot access local variable 'inputs_json'

That happens after mutex.acquire() and before mutex.release(), so a manager lock is left held by a process that no longer exists. The next worker blocks on acquire() and the run waits forever. error_event.set() is the handler's last line, so the parent is never told either.

A worker that is killed is not noticed at all. The parent joins each worker and reads the event. A signal runs no handler, so the event stays clear and the join returns because the process is gone. With a SIGKILL in the second of six simulations across two workers:

simulate() returned normally
asked for 6, rows on disk: 2

I would rather have the hang than that one. A hang is at least visible.

New behavior

Both names are bound before the try, and the message says worker startup when no index was claimed instead of naming one that does not exist. The handler now finishes, releases the mutex and sets the event, so a worker failure surfaces as a RuntimeError naming the error file rather than as a wait.

After the join, _refuse_a_worker_that_did_not_finish reads the exit codes. Anything other than zero is refused. It sits before the event check because the two are disjoint: a worker that reports through its handler exits cleanly, and one that was killed only leaves its exit code behind.

The handler holds the shared mutex while it reports, so a failure in the reporting used to leave the lock held by a process that had already gone. The event is set first and from outside the lock, the lock is released from a finally, and each reporting step is separate so an unwritable log cannot replace the failure being reported. A startup failure writes a row of its own now, since the caller is told to read that file and a traceback a worker printed is not there once its output is redirected.

The lock belongs to the manager and is not released when its holder is killed, so a sibling can block on a lock nobody owns while an unbounded join waits with it. That is the case the exit-code check exists for and the one it could not see. _join_the_workers polls instead, and acts only once a worker has actually ended badly. A run that is merely slow is never bounded: an exit code, not a duration, decides. This is deliberately not a general worker-lifecycle change; it is the smallest thing that makes the check above reachable.

Scope is the parallel producer. __run_in_serial has the same unbound inputs_json and belongs to #1177, whose tests cover it already; I have deliberately not touched it.

Breaking change

• No

A run that used to hang now raises, and a run that used to return with missing rows now raises. Both were already failures.

Additional information

Verification, on a clean tree:

pytest tests/unit                     2194 passed
pytest rocketpy --doctest-modules       48 passed
pytest tests/integration               154 passed
pytest tests/acceptance                 18 passed
ruff check . / ruff format --check .   clean
pylint rocketpy/ tests/ docs/          10.00/10, exit 0

The four tests/unit/test_sensitivity.py failures on my machine are a missing statsmodels and fail the same way on an untouched develop.

Mutations, each leaving a control standing:

undone	goes red	still passes
the two names are not bound before the try	five	the test where an index was claimed first
only inputs_json is bound, which is what the traceback points at	still five	same one
the parent stops reading exit codes	the killed-worker test	the rest
if process.exitcode in place of != 0	the None case	the rest
the reporting is not wrapped in try/finally	the three lock tests	the rest
the join goes back to being unbounded	all eight join tests	the rest
the join acts on how long a worker took rather than on how it ended	exactly one, the slow-run test	the rest

Two rows there are worth pointing at. Fixing only the name in the traceback looks complete and leaves the message raising on the other one. And deciding the join on elapsed time rather than on an exit code passes everything except the test that says a slow run must be left alone, which is the failure mode that would hurt real users most.

Still not covered, and I would rather say so than imply otherwise: a worker killed between mutex.acquire() and the write it is guarding leaves a torn record behind. The run is stopped and reported either way now, but the rows it was midway through are not repaired.

_refuse_a_worker_that_did_not_finish is a module-level function rather than a method because the run paths are driven by stub objects in the tests, where self.__helper() would not resolve.

The Monte Carlo objects in these tests are built on tmp_path rather than retargeted after construction. filename is a plain attribute and the three log paths are settled in __init__, so assigning it leaves a test writing the fixture's own files into the working directory, which is what these were doing.

Merged with #1169, #1170 and #1171's follow-up into a throwaway tree on develop and run there too. The fixed-seed baseline is unchanged across all of it: stochastic_calisto under seed 42 reads mass=14.906007947 radius=0.063501935.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.32%. Comparing base (4263fa9) to head (6fb180f).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1182      +/-   ##
===========================================
+ Coverage    84.57%   85.32%   +0.74%     
===========================================
  Files          131      131              
  Lines        17527    17571      +44     
===========================================
+ Hits         14824    14992     +168     
+ Misses        2703     2579     -124     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Fixes parallel Monte Carlo seed handling and ensures failed workers cannot silently hang or produce incomplete successful runs.

Changes:

• Normalizes SeedSequence values without collapsing worker streams.
• Adds worker failure reporting, exit-code validation, and bounded shutdown polling.
• Adds regression coverage for parallel execution and worker failure modes.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
rocketpy/tools.py	Adds deterministic SeedSequence conversion.
rocketpy/stochastic/stochastic_model.py	Accepts worker seed sequences.
rocketpy/simulation/monte_carlo.py	Improves worker reporting and lifecycle handling.
tests/unit/stochastic/test_seed_types.py	Tests seed compatibility and stream independence.
tests/unit/simulation/test_monte_carlo_parallel_runs.py	Exercises real serial and parallel runs.
tests/unit/simulation/test_monte_carlo_worker_reporting.py	Tests worker failure diagnostics.
tests/unit/simulation/test_monte_carlo_worker_join.py	Tests polling and shutdown behavior.
tests/unit/simulation/test_monte_carlo_worker_exit.py	Tests abnormal worker exit detection.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.