diff --git a/README.md b/README.md index 0c37de1..cdd8165 100644 --- a/README.md +++ b/README.md @@ -281,7 +281,7 @@ failures on application containers. | Version | Date | Author | Changes | |---------|------------|--------|---------| -| 1.6.1 | Pending | — | **In preparation, not published:** HA-monitoring transient transition diagnostics (zero promoted HA Operator tolerated briefly; ambiguous/safety-critical conditions still reported). See [draft v1.6.1 release notes](docs/operations/RELEASE_NOTES_V1.6.1.md). Additional changes and validation to be recorded before release. | +| 1.6.1 | Pending | — | **In preparation, not published:** HA-monitoring transient transition diagnostics (without hiding safety-critical faults); remove unverified default MariaDB label exception while retaining strict built-in dump-helper handling and optional label policies. See [draft v1.6.1 release notes](docs/operations/RELEASE_NOTES_V1.6.1.md). Additional changes and validation to be recorded before release. | | 1.6.0 | 2026-10-15 | — | Changes since 1.5.4: PRs [#3](https://github.com/Radware/CyberController-Container-Alert/pull/3), [#4](https://github.com/Radware/CyberController-Container-Alert/pull/4), [#5](https://github.com/Radware/CyberController-Container-Alert/pull/5), [#6](https://github.com/Radware/CyberController-Container-Alert/pull/6), [#7](https://github.com/Radware/CyberController-Container-Alert/pull/7), [#8](https://github.com/Radware/CyberController-Container-Alert/pull/8), [#10](https://github.com/Radware/CyberController-Container-Alert/pull/10); corrected restart-loop/recovery lifecycle, planned-stop/exit-137 handling, event ID/name reuse, independent optional HA failover detection, Syslog and notification consistency, one complete versioned release package and wizard, revised informational alert messages, and isolated-only release smoke tests. See [complete v1.6.0 release notes](docs/operations/RELEASE_NOTES_V1.6.0.md). | | 1.5.4 | 2026-09-23 | Rahul Kumar | Versioned Docker image (`watchdog:`) driven by the root `VERSION` file; installer loads the packaged `watchdog.tar`, rejects an archive that does not carry the release tag, aborts on Compose failures, and verifies the running image; restart-loop detection now uses Docker `RestartCount` instead of counting poll observations; `container_health_checks` overrides implemented; fixed `.env` migration on root-owned files; `watchdog-config.yaml` made readable by the non-root container user | | 1.5.3 | 2026-09-16 | Rahul Kumar | Resource Limits Enforced | diff --git a/docs/dev/DEVELOPMENT_AND_RELEASE.md b/docs/dev/DEVELOPMENT_AND_RELEASE.md index e0b039f..c16bb4a 100644 --- a/docs/dev/DEVELOPMENT_AND_RELEASE.md +++ b/docs/dev/DEVELOPMENT_AND_RELEASE.md @@ -291,13 +291,11 @@ Production deployment requires a published, prebuilt release package. ### Narrow CyberController event exceptions +The observed CyberController MariaDB SQL dump syntax-check helper is handled by an **application-specific built-in predicate**: the helper image, non-primary container name, Docker `AutoRemove=True` status, expected SQL dump file mount and exit status 137 must **all** match. Normal primary MariaDB and unrelated containers remain monitored. This exception does not require `ignored_container_events` in YAML. -The intended MariaDB HA syntax-check helper carries the explicit label `com.radware.cybercontroller.role=mariadb-ha-syntax-check`. The configured `ignored_container_events` policy ignores only the designated temporary helper `die` event; normal service container failures remain monitored. +`ignored_container_events` remains a **generic, optional label-based event policy**, but no default MariaDB label is generated. In the October 2026 HA lab run, the expected dump-helper exit reached the built-in predicate instead of matching the previously advertised `com.radware.cybercontroller.role=mariadb-ha-syntax-check` rule. Do not claim that CyberController sets that label without concrete evidence. If a future supported helper does carry a verified label, an operator may deliberately configure a narrow `ignored_container_events` rule. -A narrow compatibility fallback exists for dynamic MariaDB dump helpers: it applies only when the helper image, non-primary container name, Docker `AutoRemove` status, expected SQL dump file mount and exit status 137 **all** match. This is not a general rule to suppress MariaDB failures. Do not alter these narrow exceptions simply to suppress alerts; validate changes with regression tests. - - -These exceptions are not blanket MariaDB or OOM alert suppression. Changes to monitoring exceptions require regression validation. +Installer upgrades preserve existing customer YAML, including any earlier label rule; they never silently delete or rewrite configured exceptions. These protections are not blanket MariaDB/OOM alert suppression. Validate all predicate and configuration changes with non-destructive regression tests and only hosted isolated Docker acceptance. ### Historical design and validation diff --git a/docs/operations/INSTALLATION_AND_OPERATIONS.md b/docs/operations/INSTALLATION_AND_OPERATIONS.md index 8202236..d05c9b7 100644 --- a/docs/operations/INSTALLATION_AND_OPERATIONS.md +++ b/docs/operations/INSTALLATION_AND_OPERATIONS.md @@ -231,10 +231,14 @@ HA traps use the same OID and var-binds with `cwFailureType=ha-failover` and `cw | `auto_health_check.timeout_seconds` | `2` | Probe timeout | | `container_health_checks` | `{}` | Per-container `http` / `exec` probe overrides | | `excluded_containers` | `[]` | Containers excluded from alerts | -| `ignored_container_events` | Radware-labelled temporary check | Narrow exceptions for known transient operations | +| `ignored_container_events` | `[]` (optional; not generated by installer) | Opt-in suppression of Docker events based on **verified** container labels; not required for known MariaDB dump helpers | | `ha_monitor.enabled` | `false` | Optional active-node change alerts | | `ha_monitor.exporter_url` | `http://127.0.0.1:9664/metrics` | Local HA exporter address | +**Temporary MariaDB SQL dump helpers:** The watchdog recognizes the known CyberController syntax-check helper automatically if the container uses the expected `kvision_infra_mariadb` image, is not the primary MariaDB service, has `AutoRemove=True`, mounts `/mnt/cli/tmp/mariadb_dump.sql.gz`, and exits 137. This narrowly scoped built-in exception does **not** depend on `ignored_container_events`. The installer does not generate an assumed MariaDB role label. Operators can still add explicit `ignored_container_events` entries for other verified, intentionally excluded labelled containers; normal application failures should not be excluded. + +**Upgrades:** Existing `watchdog-config.yaml` files remain unchanged, including an older `com.radware.cybercontroller.role: mariadb-ha-syntax-check` rule if present. That old rule is not required for the observed dump-helper path; review it before removing it from a customer configuration rather than changing it automatically. + A custom probe can override discovery for a specific container. Example: ```yaml diff --git a/docs/operations/RELEASE_NOTES_V1.6.1.md b/docs/operations/RELEASE_NOTES_V1.6.1.md index 5d1db5c..a053879 100644 --- a/docs/operations/RELEASE_NOTES_V1.6.1.md +++ b/docs/operations/RELEASE_NOTES_V1.6.1.md @@ -7,12 +7,13 @@ | Area | Planned change | Impact | |---|---|---| | HA monitoring diagnostics | During failover, tolerate up to two consecutive checks with no promoted HA Operator. On the third consecutive such reading, emit a throttled error; conflicting promoted nodes, loss of quorum and exporter failures remain immediate errors. | Reduces expected transient Syslog ERROR noise without hiding sustained or unsafe HA status. No change to HA failover/failback detection, owner-only notifications, or Docker container monitoring. | +| MariaDB helper configuration clarity | Remove the unverified MariaDB syntax-check Docker label rule from Python defaults, installer-generated configuration and the example template. Retain optional explicit label-based exclusions and the existing narrow built-in dump-helper exception. | Avoids documenting a label not observed on the temporary MariaDB helper; no change to the verified built-in exit-137 suppression. Existing customer YAML remains preserved on upgrades, including older rules. | **Additional changes:** To be updated as development PRs are integrated. ## Compatibility and operations -No configuration key, new service, port, endpoint or privilege is required by the initial HA diagnostic change. Default HA polling remains 60 seconds; the existing 2-valid-poll failover confirmation and 5-minute repeated-error rate limit are retained. +No new service, port, endpoint or privilege is required. Default HA polling remains 60 seconds; the existing 2-valid-poll failover confirmation and 5-minute repeated-error rate limit are retained. Newly generated configs omit the optional `ignored_container_events` key; the reference template defaults it to `[]`. The event-label mechanism is still available for explicit, verified labels. The strict built-in MariaDB dump-helper exception stays enabled and unchanged. Installer upgrades **preserve existing customer YAML**, so they do not automatically remove old label entries. ## Validation (pending) diff --git a/install.sh b/install.sh index 3b0f3c4..b2f3b2e 100755 --- a/install.sh +++ b/install.sh @@ -677,14 +677,6 @@ configure_all() { printf "restart_window_minutes: 10\n" printf "unhealthy_cycles_threshold: 3\n" printf "\nexcluded_containers: []\n" - printf "\nignored_container_events:\n" - printf " - label:\n" - printf " com.radware.cybercontroller.role: mariadb-ha-syntax-check\n" - printf " events:\n" - printf " - die\n" - printf "# Temporary MariaDB HA syntax-check containers should carry the label shown above.\n" - printf "# Fallback: suppress kvision_infra_mariadb AutoRemove dump helpers exiting 137\n" - printf "# only when /mnt/cli/tmp/mariadb_dump.sql.gz is mounted; primary MariaDB stays monitored.\n" printf "\nrunbook_base_url: \"\"\n" printf "\nlog_level: INFO\n" printf "log_file: /var/log/watchdog/watchdog.log\n" diff --git a/tests/test_installer_image.py b/tests/test_installer_image.py index 64629a1..005d8d6 100644 --- a/tests/test_installer_image.py +++ b/tests/test_installer_image.py @@ -142,7 +142,13 @@ def test_existing_configuration_is_not_regenerated(self): def test_keep_existing_configuration_does_not_add_ha(self): # Upgrading with the default "keep" answer must preserve active YAML. - existing = "\n".join(["syslog:", " enabled: true", ""]) + existing = "\n".join([ + "syslog:", " enabled: true", + "ignored_container_events:", + " - label:", + " com.radware.cybercontroller.role: mariadb-ha-syntax-check", + " events: [die]", "", + ]) (self.root / "watchdog-config.yaml").write_text(existing) (self.root / ".env").write_text( "\n".join(["DOCKER_GID=999", "WATCHDOG_VERSION=1.5.4", ""]) @@ -170,6 +176,7 @@ def test_generated_config_includes_optional_ha_disabled(self): self.assertEqual(config["ha_monitor"], { "enabled": False, "exporter_url": "http://127.0.0.1:9664/metrics", }) + self.assertNotIn("ignored_container_events", config) self.assertEqual(self.calls(), []) def test_generated_config_can_enable_ha_with_custom_exporter(self): diff --git a/tests/test_mariadb_event_exceptions.py b/tests/test_mariadb_event_exceptions.py new file mode 100644 index 0000000..af0ab86 --- /dev/null +++ b/tests/test_mariadb_event_exceptions.py @@ -0,0 +1,87 @@ +"""Non-destructive tests for built-in MariaDB helper and optional label policies.""" +from pathlib import Path +from types import SimpleNamespace +import tempfile +import unittest + +import yaml +import watchdog + + +ROOT = Path(__file__).resolve().parents[1] +IMAGE = "reg.radware.com:18442/kvision/kvision_infra_mariadb:1.30.0" +MOUNT = "/mnt/cli/tmp/mariadb_dump.sql.gz" + + +def helper(name="zen_hypatia", *, image=IMAGE, auto_remove=True, mount=MOUNT, labels=None): + labels = {} if labels is None else labels + return SimpleNamespace( + name=name, + labels=labels, + image=SimpleNamespace(tags=[]), + attrs={ + "Config": {"Image": image, "Labels": labels}, + "HostConfig": {"AutoRemove": auto_remove}, + "Mounts": [{"Source": mount, "Destination": mount}], + }, + ) + + +class MariaDBEventExceptionsTests(unittest.TestCase): + def test_unlabelled_realistic_helper_uses_builtin_predicate(self): + container = helper() + attrs = {"name": "zen_hypatia", "image": IMAGE, "exitCode": "137"} + self.assertEqual(watchdog.DEFAULT_CONFIG["ignored_container_events"], []) + self.assertFalse(watchdog.is_ignored_container_event( + container, attrs, "die", watchdog.DEFAULT_CONFIG["ignored_container_events"])) + self.assertTrue(watchdog.is_kvision_mariadb_dump_helper_crash(container, attrs, 137)) + + def test_builtin_predicate_does_not_hide_other_failures(self): + attrs = {"name": "zen_hypatia", "image": IMAGE, "exitCode": "137"} + cases = ( + ("primary MariaDB", helper(name="config_kvision-infra-mariadb_1"), + {**attrs, "name": "config_kvision-infra-mariadb_1"}, 137), + ("exit 1", helper(), attrs, 1), + ("exit 0", helper(), attrs, 0), + ("not AutoRemove", helper(auto_remove=False), attrs, 137), + ("wrong mount", helper(mount="/tmp/other.sql.gz"), attrs, 137), + ("wrong image", helper(image="other_image:1.0"), + {**attrs, "image": "other_image:1.0"}, 137), + ("normal unlabeled service", helper(name="ordinary-service", + image="other_image:1.0"), {"name": "ordinary-service"}, 137), + ) + for description, container, event, exit_code in cases: + with self.subTest(case=description): + self.assertFalse(watchdog.is_kvision_mariadb_dump_helper_crash( + container, event, exit_code)) + + def test_explicit_verified_label_rule_still_works_when_configured(self): + rule = [{"label": {"com.example.test.role": "temporary"}, "events": ["die"]}] + container = helper(labels={"com.example.test.role": "temporary"}) + event = {"name": "zen_hypatia"} + self.assertTrue(watchdog.is_ignored_container_event(container, event, "die", rule)) + self.assertFalse(watchdog.is_ignored_container_event(container, event, "oom", rule)) + self.assertFalse(watchdog.is_ignored_container_event(helper(), event, "die", rule)) + self.assertFalse(watchdog.is_ignored_container_event(container, event, "die", [])) + + def test_runtime_defaults_and_reference_template_have_no_implicit_label_rule(self): + template = yaml.safe_load((ROOT / "watchdog-config.yaml.example").read_text()) + self.assertEqual(template["ignored_container_events"], []) + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "watchdog-config.yaml" + path.write_text("log_level: WARNING\n") + self.assertEqual(watchdog.load_config(str(path))["ignored_container_events"], []) + + path.write_text( + "ignored_container_events:\n" + " - label:\n" + " com.example.test.role: temporary\n" + " events: [die]\n" + ) + self.assertEqual(watchdog.load_config(str(path))["ignored_container_events"], + [{"label": {"com.example.test.role": "temporary"}, + "events": ["die"]}]) + + +if __name__ == "__main__": + unittest.main() diff --git a/watchdog-config.yaml.example b/watchdog-config.yaml.example index 1398a02..01915ae 100644 --- a/watchdog-config.yaml.example +++ b/watchdog-config.yaml.example @@ -35,12 +35,12 @@ lifecycle_detection: excluded_containers: [] # - my-temp-container -# Narrow exception for labelled temporary syntax-check containers. -ignored_container_events: - - label: - com.radware.cybercontroller.role: mariadb-ha-syntax-check - events: - - die +# Optional label-based Docker event exclusions (disabled by default). +# Use only for verified labels on intentionally excluded temporary containers. +# The known CyberController MariaDB SQL dump helper is handled automatically +# by the watchdog's strict image, AutoRemove, mount, and exit-137 checks. +# No label rule is needed for that helper. +ignored_container_events: [] # ── Logging ─────────────────────────────────────────────────────────────────── diff --git a/watchdog.py b/watchdog.py index 734367c..53f669e 100644 --- a/watchdog.py +++ b/watchdog.py @@ -97,9 +97,6 @@ def setup_logging(level: str, log_file: str | None, syslog_cfg: dict | None = No # ── Config ──────────────────────────────────────────────────────────────────── -_KVISION_MARIADB_SYNTAX_CHECK_LABEL = "com.radware.cybercontroller.role" -_KVISION_MARIADB_SYNTAX_CHECK_ROLE = "mariadb-ha-syntax-check" - DEFAULT_CONFIG = { "alert_channels": ["slack"], "check_interval_seconds": 60, @@ -111,14 +108,7 @@ def setup_logging(level: str, log_file: str | None, syslog_cfg: dict | None = No "unhealthy_cycles_threshold": 3, "alert_on_recovery": True, "excluded_containers": [], - "ignored_container_events": [ - { - "label": { - _KVISION_MARIADB_SYNTAX_CHECK_LABEL: _KVISION_MARIADB_SYNTAX_CHECK_ROLE, - }, - "events": ["die"], - }, - ], + "ignored_container_events": [], # opt-in label policy; MariaDB dump helper uses a separate strict predicate "log_file": "/var/log/watchdog/watchdog.log", "log_level": "INFO", # Alert state survives restarts so recovery alerts still fire after a reboot.