Skip to content

fix: alert before sqlite filesystem exhaustion - #350

Open
giscebot wants to merge 2 commits into
mainfrom
fix/issue-344-disk-full
Open

giscebot wants to merge 2 commits into
mainfrom
fix/issue-344-disk-full

Conversation

@giscebot

Copy link
Copy Markdown
Collaborator

Summary

  • measure available capacity on the filesystem that contains the SQLite database
  • alert when either configurable threshold is crossed (default: 5 GiB or 10% available), and fail closed when capacity cannot be measured
  • propagate thresholds through the monitor alert service and document the systemd environment settings

Incident evidence

At 2026-11-11 17:12:28 CET, heartbeat commits failed with database or disk is full; all executor workers then hit disk I/O error and the executor restarted. A read-only check with this branch currently reports only 2.6 GiB / 0.9% available on the database filesystem, so host cleanup remains an urgent operational action. This PR adds the missing early warning; it does not reclaim disk space.

Validation

  • pytest -q -p no:cacheprovider — 720 passed
  • focused monitor tests — 60 passed
  • live read-only monitor check — emitted monitor.database_filesystem_low with 2.6 GiB / 0.9% available

Addresses #343
Addresses #344
Addresses #345
Addresses #346

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant