Skip to content

Process runner: stop the owned group when the caller is terminated - #15

Merged
florian6973 merged 1 commit into
mainfrom
fix/kill-process-group-on-timeout
Nov 5, 2026
Merged

florian6973 merged 1 commit into
mainfrom
fix/kill-process-group-on-timeout

Conversation

@florian6973

Copy link
Copy Markdown
Collaborator

Problem

runtime.process.run (used by the MEDS-DEV conformance helpers and orchestration) starts its command in a new
process group and kills the group from an except BaseException handler on timeout or interruption. If the
calling Python process is terminated by SIGTERM or SIGHUP, the handler never runs: Python exits on these
signals without unwinding, so the whole group is left running. Typical causes are an outer timeout around
pytest, a scheduler cancelling the job, or a closed terminal.

Found while running GenHPF MEDS-DEV gates. An outer timeout 5400 (the same value as RUN_TIMEOUT) and
detached background shells terminated pytest. That left three meds-dev-model trees, each holding about 4
cores in upstream training, running for 6–9 hours under init on a shared machine.

Change

  • runtime/process.py: while run() waits, SIGTERM and SIGHUP raise SystemExit(128 + signum), so the
    existing cleanup kills the owned group before the caller exits with the conventional status. This applies only
    on POSIX, only in the main thread, and only where the handler is still the default (an inherited nohup ignore
    is kept). Previous handlers are restored afterwards. SIGKILL of the caller still cannot be handled.
  • tests/test_process.py: test_terminating_the_caller_stops_descendant_work[SIGTERM|SIGHUP] signals a process
    blocked in run() and checks that it exits with 128 + signum and that its grandchild does not keep working.

Tests

  • uv run pytest -q: 153 passed. With the unchanged process.py from main, both new tests fail
    (2 failed, 3 passed in tests/test_process.py).
  • ruff check . and ruff format --check .: clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NJQdbJCWyxx5EULVg4ScX3

run() killed its group on timeout or interruption only from an exception handler, which never runs
when the calling Python process receives SIGTERM or SIGHUP (outer timeout, cancelled job, closed
terminal), leaving descendants under init. Map both signals to SystemExit while waiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJQdbJCWyxx5EULVg4ScX3
@florian6973
florian6973 marked this pull request as ready for review November 5, 2026 03:33
@florian6973
florian6973 merged commit 3c2244a into main Nov 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant