Skip to content

fix: supervise fatal heartbeat storage errors - #348

Open
giscebot wants to merge 1 commit into
mainfrom
fix/issue-345-sqlite-disk-io-error
Open

giscebot wants to merge 1 commit into
mainfrom
fix/issue-345-sqlite-disk-io-error

Conversation

@giscebot

Copy link
Copy Markdown
Collaborator

Summary

  • supervise heartbeat threads through the same failure path as worker threads
  • shut the executor down on non-contention heartbeat storage errors instead of letting the daemon thread die silently
  • preserve the original worker exception rather than masking it with a second heartbeat write failure

Rationale

disk I/O error is not SQLite lock contention and must remain fatal. Previously, an error from the heartbeat daemon was reported by Sentry, but ExecutorPool.run() did not observe it, so the worker pool could continue without health heartbeats. The new supervision path fails closed and lets the service manager restart the executor.

This does not hide or retry physical storage failures. Repeated I/O errors still require an operator to inspect the filesystem/device.

Tests

  • pytest -q tests/test_executor.py -k 'heartbeat or run_raises_when_worker_dies_unexpectedly' — 4 passed
  • pytest -q — 714 passed

Refs #345

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant