From 96abe056e8d187d90b252b4f0dd60e07794a6e26 Mon Sep 17 00:00:00 2001 From: rdwr-egore Date: Mon, 19 Oct 2026 18:04:44 +0000 Subject: [PATCH 1/5] release: start v1.6.1 stabilization --- README.md | 1 + VERSION | 2 +- docs/operations/RELEASE_NOTES_V1.6.1.md | 23 +++++++++++++++++++++++ 3 files changed, 25 insertions(+), 1 deletion(-) create mode 100644 docs/operations/RELEASE_NOTES_V1.6.1.md diff --git a/README.md b/README.md index 0228ed3..0c37de1 100644 --- a/README.md +++ b/README.md @@ -281,6 +281,7 @@ failures on application containers. | Version | Date | Author | Changes | |---------|------------|--------|---------| +| 1.6.1 | Pending | — | **In preparation, not published:** HA-monitoring transient transition diagnostics (zero promoted HA Operator tolerated briefly; ambiguous/safety-critical conditions still reported). See [draft v1.6.1 release notes](docs/operations/RELEASE_NOTES_V1.6.1.md). Additional changes and validation to be recorded before release. | | 1.6.0 | 2026-10-15 | — | Changes since 1.5.4: PRs [#3](https://github.com/Radware/CyberController-Container-Alert/pull/3), [#4](https://github.com/Radware/CyberController-Container-Alert/pull/4), [#5](https://github.com/Radware/CyberController-Container-Alert/pull/5), [#6](https://github.com/Radware/CyberController-Container-Alert/pull/6), [#7](https://github.com/Radware/CyberController-Container-Alert/pull/7), [#8](https://github.com/Radware/CyberController-Container-Alert/pull/8), [#10](https://github.com/Radware/CyberController-Container-Alert/pull/10); corrected restart-loop/recovery lifecycle, planned-stop/exit-137 handling, event ID/name reuse, independent optional HA failover detection, Syslog and notification consistency, one complete versioned release package and wizard, revised informational alert messages, and isolated-only release smoke tests. See [complete v1.6.0 release notes](docs/operations/RELEASE_NOTES_V1.6.0.md). | | 1.5.4 | 2026-09-23 | Rahul Kumar | Versioned Docker image (`watchdog:`) driven by the root `VERSION` file; installer loads the packaged `watchdog.tar`, rejects an archive that does not carry the release tag, aborts on Compose failures, and verifies the running image; restart-loop detection now uses Docker `RestartCount` instead of counting poll observations; `container_health_checks` overrides implemented; fixed `.env` migration on root-owned files; `watchdog-config.yaml` made readable by the non-root container user | | 1.5.3 | 2026-09-16 | Rahul Kumar | Resource Limits Enforced | diff --git a/VERSION b/VERSION index dc1e644..9c6d629 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.6.0 +1.6.1 diff --git a/docs/operations/RELEASE_NOTES_V1.6.1.md b/docs/operations/RELEASE_NOTES_V1.6.1.md new file mode 100644 index 0000000..5d1db5c --- /dev/null +++ b/docs/operations/RELEASE_NOTES_V1.6.1.md @@ -0,0 +1,23 @@ +# v1.6.1 — Release Notes (Draft, Not Published) + +**Status: In preparation.** v1.6.1 has not been tagged, packaged, published, or deployed. Further fixes and enhancements may be added to this branch before final review. This document must be finalized with the complete inventory and verified test evidence before publishing. + +## Changes since v1.6.0 + +| Area | Planned change | Impact | +|---|---|---| +| HA monitoring diagnostics | During failover, tolerate up to two consecutive checks with no promoted HA Operator. On the third consecutive such reading, emit a throttled error; conflicting promoted nodes, loss of quorum and exporter failures remain immediate errors. | Reduces expected transient Syslog ERROR noise without hiding sustained or unsafe HA status. No change to HA failover/failback detection, owner-only notifications, or Docker container monitoring. | + +**Additional changes:** To be updated as development PRs are integrated. + +## Compatibility and operations + +No configuration key, new service, port, endpoint or privilege is required by the initial HA diagnostic change. Default HA polling remains 60 seconds; the existing 2-valid-poll failover confirmation and 5-minute repeated-error rate limit are retained. + +## Validation (pending) + +- Unit/static tests: pending development PR and current-head CI evidence. +- Release PR CI and GitHub-hosted isolated Docker smoke: pending. Never run destructive Docker tests on operational/shared hosts. +- Full change inventory, acceptance, package/digest validation and rollback review: pending final release decision. + +**No release action:** Do not tag `v1.6.1` or publish a package until the final reviewed release PR is merged into `main`, all required checks are green, and explicit go-ahead is given. From 79ec82ca59c2444f0207a7a7503bbeb4dd1df9ec Mon Sep 17 00:00:00 2001 From: rdwr-egore Date: Mon, 19 Oct 2026 18:08:32 +0000 Subject: [PATCH 2/5] fix: defer transient HA no-promoted diagnostics --- docs/dev/HA_MONITORING.md | 4 +- .../operations/INSTALLATION_AND_OPERATIONS.md | 2 + ha_monitor.py | 12 +- tests/test_ha_monitor.py | 110 +++++++++++++++++- watchdog.py | 26 ++++- 5 files changed, 146 insertions(+), 8 deletions(-) diff --git a/docs/dev/HA_MONITORING.md b/docs/dev/HA_MONITORING.md index 1b5ce4e..210806c 100644 --- a/docs/dev/HA_MONITORING.md +++ b/docs/dev/HA_MONITORING.md @@ -11,6 +11,8 @@ | Run HA polling in a **separate thread and HTTP session** from Docker polling | Slow or blocked Docker API calls must not delay HA observations; HA exporter failures must not interrupt container monitoring | | Require **quorum, a successful Pacemaker scrape, one local node and exactly one promoted owner** | Missing or conflicting telemetry is not proof of a role change; fail closed instead of inventing a failover | | Require **two consecutive valid observations** of a new active node | Avoid reporting transient/promoted-resource ambiguity while the cluster is transitioning | +| Allow **two consecutive no-promoted-owner observations** before logging an ERROR on the third | A failover may temporarily have no active/promoted operator. Avoid transient Syslog noise, but do not hide persistent loss of an owner | +| Report **multiple promoted owners, exporter failures and loss of quorum immediately** | These are not an ordinary empty-promotion gap and must not be silenced by the grace period | | **Only the newly active node** emits the alert | Watchdogs run on both nodes; duplicate notifications from old and new active nodes would be misleading | | Persist the confirmed active node using the existing state file | Survive watchdog restarts without replaying an old role change or creating a second state store | | Reuse the existing notification channels and SNMP MIB | Treat HA as another monitored event without adding a new alert transport or breaking container notifications | @@ -21,7 +23,7 @@ 2. It reads `ha_cluster_pacemaker_resources{resource="HAOperatorRep-Res",role="promoted",status="active"} == 1` to identify the active owner and `ha_cluster_corosync_member_votes{local="true"}` to identify the local node. The Pacemaker scrape-success and quorum metrics must both be 1. 3. The **first valid reading** establishes a silent baseline. An active-node change must appear in **two valid consecutive checks** before it is confirmed. 4. On a confirmed transition A → B, both nodes can update their saved baseline, but **only the watchdog local to B** issues one `WARNING` `ha-failover` event with previous/new node details. -5. Invalid/unavailable metrics leave the last confirmed baseline unchanged and reset a pending candidate. Exporter errors are logged at most once every five minutes; they never trigger a speculative failover alert. +5. Invalid/unavailable metrics leave the last confirmed baseline unchanged and reset a pending candidate. A valid Pacemaker scrape with quorum but **zero promoted HA Operator** matches is treated as a possible promotion gap: the first two consecutive observations are DEBUG-only; the third consecutive observation generates an ERROR. At the default 60-second interval, the third observation is about two minutes after the first. A valid observation resets the counter. **Multiple promoted owners**, failed Pacemaker scrape/lost quorum, and exporter transport failures generate an immediate ERROR. Repeated identical diagnostic errors are throttled to at most once every five minutes; a different failure reason is logged immediately. These observations never trigger a speculative failover alert. 6. HA state is kept separately from container incidents in `watchdog/state.json`. Restart with monitoring enabled restores that state. Running the watchdog with HA monitoring disabled clears its baseline, so a later re-enable starts silently. 7. During the host reboot grace interval, HA role checks are deferred. HA events are not included in container recovery or reboot-summary logic. diff --git a/docs/operations/INSTALLATION_AND_OPERATIONS.md b/docs/operations/INSTALLATION_AND_OPERATIONS.md index 05ae4bc..8202236 100644 --- a/docs/operations/INSTALLATION_AND_OPERATIONS.md +++ b/docs/operations/INSTALLATION_AND_OPERATIONS.md @@ -258,6 +258,8 @@ ha_monitor: Enable this only for HA deployments. Verify its URL read-only, then restart **only** the watchdog ([Section 4.6](#46-apply-configuration-changes)). The first valid reading sets a baseline without sending an alert. +Starting in v1.6.1, up to two consecutive HA exporter observations with **no promoted HA Operator** are treated as a possible short failover transition and do not generate Syslog ERROR noise. The third consecutive observation is an ERROR (about two minutes after the first with default 60-second polling). Multiple promoted owners, exporter failures and missing quorum remain immediate errors; the watchdog does not automatically remediate Pacemaker or change application containers. + ### 4.5 Environment variables The installer manages most `.env` values. SMTP passwords, Slack webhooks and SNMPv3 passphrases belong here, not in YAML. The SNMPv1/v2c community is configured in YAML, so protect that file too. diff --git a/ha_monitor.py b/ha_monitor.py index c2c52ed..d6b2f45 100644 --- a/ha_monitor.py +++ b/ha_monitor.py @@ -21,6 +21,10 @@ } +class NoPromotedHAOperatorError(ValueError): + """Valid HA scrape/quorum, but no uniquely promoted owner *yet*.""" + + @dataclass(frozen=True) class HAObservation: active_node: str @@ -94,8 +98,12 @@ def parse_ha_metrics(text: str) -> HAObservation: raise ValueError("Pacemaker scrape failed or cluster has no quorum") if len(local) != 1 or not local[0] or len(local[0]) > 255: raise ValueError("cannot identify unique local HA node") - if len(promoted) != 1 or not promoted[0] or len(promoted[0]) > 255: - raise ValueError("cannot identify unique promoted HA Operator") + if not promoted: + raise NoPromotedHAOperatorError("no promoted HA Operator observed") + if len(promoted) > 1: + raise ValueError("multiple promoted HA Operators observed") + if not promoted[0] or len(promoted[0]) > 255: + raise ValueError("invalid promoted HA Operator node") return HAObservation(active_node=promoted[0], local_node=local[0]) diff --git a/tests/test_ha_monitor.py b/tests/test_ha_monitor.py index d84cce7..76d0f23 100644 --- a/tests/test_ha_monitor.py +++ b/tests/test_ha_monitor.py @@ -9,7 +9,7 @@ import watchdog from ha_monitor import ( - HAObservation, HARoleTracker, MAX_METRICS_BYTES, + HAObservation, HARoleTracker, MAX_METRICS_BYTES, NoPromotedHAOperatorError, parse_ha_metrics, read_ha_observation, ) @@ -68,6 +68,22 @@ def test_fail_closed_on_invalid_state(self): with self.subTest(sample=sample[-75:]), self.assertRaises(ValueError): parse_ha_metrics(sample) + def test_zero_promoted_is_distinct_from_multiple_promoted(self): + no_promoted = metrics().replace( + 'role="promoted",resource="HAOperatorRep-Res"', + 'role="unpromoted",resource="HAOperatorRep-Res"') + with self.assertRaisesRegex(NoPromotedHAOperatorError, "no promoted"): + parse_ha_metrics(no_promoted) + + two_promoted = metrics(extras=( + f'ha_cluster_pacemaker_resources{{node="{B}",role="promoted",' + 'resource="HAOperatorRep-Res",status="active"} 1')) + with self.assertRaisesRegex(ValueError, "multiple promoted"): + parse_ha_metrics(two_promoted) + with self.assertRaises(ValueError) as raised: + parse_ha_metrics(two_promoted) + self.assertNotIsInstance(raised.exception, NoPromotedHAOperatorError) + def test_ignores_non_matching_roles_and_untrusted_other_metric(self): sample = metrics(owner=A, local=A, extras= 'ha_cluster_pacemaker_resources{node="x",resource="HAOperatorRep-Res",role="promoted",status="active"} 0\n' @@ -147,6 +163,8 @@ def create_shell(self, state_file): w._ha_enabled = True w._ha_tracker = HARoleTracker(w.state.get_ha_node()) w._ha_error_log_at = 0.0 + w._ha_last_error_reason = None + w._ha_missing_promoted_polls = 0 w._session = Mock() w._boot_time = 123 w._grace_until = 0.0 @@ -187,7 +205,7 @@ def test_ha_exporter_failure_is_error_but_throttled(self): with tempfile.TemporaryDirectory() as td: w = self.create_shell(f"{td}/state.json") with patch("watchdog.read_ha_observation", side_effect=TimeoutError("offline")), \ - patch("watchdog.time.monotonic", side_effect=[1000, 1000, 1100, 1301, 1301]), \ + patch("watchdog.time.monotonic", side_effect=[1000, 1100, 1301]), \ patch("watchdog.log.error") as err: w._poll_ha(w._session) w._poll_ha(w._session) @@ -196,6 +214,94 @@ def test_ha_exporter_failure_is_error_but_throttled(self): self.assertIn("HA check unavailable", err.call_args.args[0]) w._dispatch.assert_not_called() + def test_short_no_promoted_gap_is_quiet_and_does_not_change_state(self): + with tempfile.TemporaryDirectory() as td: + w = self.create_shell(f"{td}/state.json") + w._ha_tracker = HARoleTracker(A) + missing = NoPromotedHAOperatorError("no promoted HA Operator observed") + with patch("watchdog.read_ha_observation", side_effect=[ + missing, missing, HAObservation(A, A)]), \ + patch("watchdog.log.error") as err: + for _ in range(3): + w._poll_ha(w._session) + err.assert_not_called() # no Syslog ERROR on normal short failover gap + w._dispatch.assert_not_called() + self.assertEqual(w._ha_tracker.confirmed, A) + self.assertEqual(w._ha_missing_promoted_polls, 0) + + def test_persistent_no_promoted_gap_reports_after_three_polls_and_throttles(self): + with tempfile.TemporaryDirectory() as td: + w = self.create_shell(f"{td}/state.json") + w._ha_tracker = HARoleTracker(A) + missing = NoPromotedHAOperatorError("no promoted HA Operator observed") + with patch("watchdog.read_ha_observation", side_effect=missing), \ + patch("watchdog.time.monotonic", side_effect=[1120, 1180, 1421]), \ + patch("watchdog.log.error") as err: + w._poll_ha(w._session) + w._poll_ha(w._session) + err.assert_not_called() + w._poll_ha(w._session) # third consecutive no-owner observation + self.assertEqual(err.call_count, 1) + w._poll_ha(w._session) # 60s later, throttled + self.assertEqual(err.call_count, 1) + w._poll_ha(w._session) # >= 300s later, diagnostic repeated + self.assertEqual(err.call_count, 2) + self.assertIn("NoPromotedHAOperatorError", err.call_args.args) + self.assertEqual(w._ha_tracker.confirmed, A) + w._dispatch.assert_not_called() + + def test_good_observation_resets_no_promoted_counter(self): + with tempfile.TemporaryDirectory() as td: + w = self.create_shell(f"{td}/state.json") + w._ha_tracker = HARoleTracker(A) + missing = NoPromotedHAOperatorError("no promoted HA Operator observed") + with patch("watchdog.read_ha_observation", side_effect=[ + missing, HAObservation(A, A), missing, missing]), \ + patch("watchdog.log.error") as err: + for _ in range(4): + w._poll_ha(w._session) + err.assert_not_called() + self.assertEqual(w._ha_missing_promoted_polls, 2) + + def test_critical_ha_faults_are_immediate_even_after_transition_gap(self): + with tempfile.TemporaryDirectory() as td: + w = self.create_shell(f"{td}/state.json") + missing = NoPromotedHAOperatorError("no promoted HA Operator observed") + faults = [ + missing, missing, + ValueError("multiple promoted HA Operators observed"), + TimeoutError("exporter unavailable"), + ValueError("Pacemaker scrape failed or cluster has no quorum"), + ] + with patch("watchdog.read_ha_observation", side_effect=faults), \ + patch("watchdog.time.monotonic", return_value=1000), \ + patch("watchdog.log.error") as err: + for _ in range(2): + w._poll_ha(w._session) + err.assert_not_called() + w._poll_ha(w._session) + self.assertEqual(err.call_count, 1) + w._poll_ha(w._session) + self.assertEqual(err.call_count, 2) + w._poll_ha(w._session) + self.assertEqual(err.call_count, 3) + self.assertEqual(w._ha_missing_promoted_polls, 0) + w._dispatch.assert_not_called() + + def test_failover_after_short_gap_still_requires_two_good_polls(self): + with tempfile.TemporaryDirectory() as td: + w = self.create_shell(f"{td}/state.json") + w._ha_tracker = HARoleTracker(A) + missing = NoPromotedHAOperatorError("no promoted HA Operator observed") + with patch("watchdog.read_ha_observation", side_effect=[ + missing, missing, HAObservation(B, B), HAObservation(B, B)]), \ + patch("watchdog.log.error") as err: + for _ in range(4): + w._poll_ha(w._session) + err.assert_not_called() + w._dispatch.assert_called_once() + self.assertEqual(w.state.get_ha_node(), B) + def test_failed_exporter_does_not_overwrite_state_or_alert(self): with tempfile.TemporaryDirectory() as td: w = self.create_shell(f"{td}/state.json") diff --git a/watchdog.py b/watchdog.py index 9922d84..734367c 100644 --- a/watchdog.py +++ b/watchdog.py @@ -27,11 +27,12 @@ import yaml from lifecycle import DockerLifecycleCorrelator -from ha_monitor import HARoleTracker, read_ha_observation +from ha_monitor import HARoleTracker, NoPromotedHAOperatorError, read_ha_observation # ── Logging setup ───────────────────────────────────────────────────────────── LOG_FORMAT = "%(asctime)s [%(levelname)s] %(name)s: %(message)s" SYSLOG_FORMAT = "watchdog[%(process)d]: %(message)s" +HA_NO_PROMOTED_ERROR_POLLS = 3 # Ignore short promotion gaps, never indefinite loss. class AlertSyslogFilter(logging.Filter): @@ -1422,6 +1423,8 @@ def __init__(self, cfg: dict): self.state.save(self._boot_time) self._ha_tracker = HARoleTracker(self.state.get_ha_node()) self._ha_error_log_at = float("-inf") + self._ha_last_error_reason = None + self._ha_missing_promoted_polls = 0 # ── State restore ───────────────────────────────────────────────────────── def _restore_state(self) -> None: @@ -1882,11 +1885,28 @@ def _poll_ha(self, session: requests.Session) -> None: observation = read_ha_observation(session, url) except Exception as exc: self._ha_tracker.reset_candidate() - if time.monotonic() - self._ha_error_log_at >= 300: - self._ha_error_log_at = time.monotonic() + if isinstance(exc, NoPromotedHAOperatorError): + self._ha_missing_promoted_polls += 1 + if self._ha_missing_promoted_polls < HA_NO_PROMOTED_ERROR_POLLS: + log.debug("HA promotion pending (%d/%d): %s", + self._ha_missing_promoted_polls, + HA_NO_PROMOTED_ERROR_POLLS, exc) + return + else: + # Multiple promoters, lost quorum and exporter failures are not + # normal promotion gaps; report their first occurrence at once. + self._ha_missing_promoted_polls = 0 + + reason = (type(exc).__name__, str(exc)) + now = time.monotonic() + if reason != self._ha_last_error_reason or now - self._ha_error_log_at >= 300: + self._ha_error_log_at = now + self._ha_last_error_reason = reason log.error("HA check unavailable (%s): %s", type(exc).__name__, exc) return + self._ha_missing_promoted_polls = 0 + self._ha_last_error_reason = None changed, change = self._ha_tracker.observe(observation) if changed: self.state.set_ha_node(self._ha_tracker.confirmed) From 8a27641847ad14813804f0ee6a94bafc014c0cef Mon Sep 17 00:00:00 2001 From: rdwr-egore Date: Mon, 19 Oct 2026 21:46:16 +0000 Subject: [PATCH 3/5] fix: remove unverified MariaDB helper label from defaults --- README.md | 2 +- docs/dev/DEVELOPMENT_AND_RELEASE.md | 8 +- .../operations/INSTALLATION_AND_OPERATIONS.md | 6 +- docs/operations/RELEASE_NOTES_V1.6.1.md | 3 +- install.sh | 8 -- tests/test_installer_image.py | 9 +- tests/test_mariadb_event_exceptions.py | 87 +++++++++++++++++++ watchdog-config.yaml.example | 12 +-- watchdog.py | 12 +-- 9 files changed, 113 insertions(+), 34 deletions(-) create mode 100644 tests/test_mariadb_event_exceptions.py diff --git a/README.md b/README.md index 0c37de1..cdd8165 100644 --- a/README.md +++ b/README.md @@ -281,7 +281,7 @@ failures on application containers. | Version | Date | Author | Changes | |---------|------------|--------|---------| -| 1.6.1 | Pending | — | **In preparation, not published:** HA-monitoring transient transition diagnostics (zero promoted HA Operator tolerated briefly; ambiguous/safety-critical conditions still reported). See [draft v1.6.1 release notes](docs/operations/RELEASE_NOTES_V1.6.1.md). Additional changes and validation to be recorded before release. | +| 1.6.1 | Pending | — | **In preparation, not published:** HA-monitoring transient transition diagnostics (without hiding safety-critical faults); remove unverified default MariaDB label exception while retaining strict built-in dump-helper handling and optional label policies. See [draft v1.6.1 release notes](docs/operations/RELEASE_NOTES_V1.6.1.md). Additional changes and validation to be recorded before release. | | 1.6.0 | 2026-10-15 | — | Changes since 1.5.4: PRs [#3](https://github.com/Radware/CyberController-Container-Alert/pull/3), [#4](https://github.com/Radware/CyberController-Container-Alert/pull/4), [#5](https://github.com/Radware/CyberController-Container-Alert/pull/5), [#6](https://github.com/Radware/CyberController-Container-Alert/pull/6), [#7](https://github.com/Radware/CyberController-Container-Alert/pull/7), [#8](https://github.com/Radware/CyberController-Container-Alert/pull/8), [#10](https://github.com/Radware/CyberController-Container-Alert/pull/10); corrected restart-loop/recovery lifecycle, planned-stop/exit-137 handling, event ID/name reuse, independent optional HA failover detection, Syslog and notification consistency, one complete versioned release package and wizard, revised informational alert messages, and isolated-only release smoke tests. See [complete v1.6.0 release notes](docs/operations/RELEASE_NOTES_V1.6.0.md). | | 1.5.4 | 2026-09-23 | Rahul Kumar | Versioned Docker image (`watchdog:`) driven by the root `VERSION` file; installer loads the packaged `watchdog.tar`, rejects an archive that does not carry the release tag, aborts on Compose failures, and verifies the running image; restart-loop detection now uses Docker `RestartCount` instead of counting poll observations; `container_health_checks` overrides implemented; fixed `.env` migration on root-owned files; `watchdog-config.yaml` made readable by the non-root container user | | 1.5.3 | 2026-09-16 | Rahul Kumar | Resource Limits Enforced | diff --git a/docs/dev/DEVELOPMENT_AND_RELEASE.md b/docs/dev/DEVELOPMENT_AND_RELEASE.md index e0b039f..c16bb4a 100644 --- a/docs/dev/DEVELOPMENT_AND_RELEASE.md +++ b/docs/dev/DEVELOPMENT_AND_RELEASE.md @@ -291,13 +291,11 @@ Production deployment requires a published, prebuilt release package. ### Narrow CyberController event exceptions +The observed CyberController MariaDB SQL dump syntax-check helper is handled by an **application-specific built-in predicate**: the helper image, non-primary container name, Docker `AutoRemove=True` status, expected SQL dump file mount and exit status 137 must **all** match. Normal primary MariaDB and unrelated containers remain monitored. This exception does not require `ignored_container_events` in YAML. -The intended MariaDB HA syntax-check helper carries the explicit label `com.radware.cybercontroller.role=mariadb-ha-syntax-check`. The configured `ignored_container_events` policy ignores only the designated temporary helper `die` event; normal service container failures remain monitored. +`ignored_container_events` remains a **generic, optional label-based event policy**, but no default MariaDB label is generated. In the October 2026 HA lab run, the expected dump-helper exit reached the built-in predicate instead of matching the previously advertised `com.radware.cybercontroller.role=mariadb-ha-syntax-check` rule. Do not claim that CyberController sets that label without concrete evidence. If a future supported helper does carry a verified label, an operator may deliberately configure a narrow `ignored_container_events` rule. -A narrow compatibility fallback exists for dynamic MariaDB dump helpers: it applies only when the helper image, non-primary container name, Docker `AutoRemove` status, expected SQL dump file mount and exit status 137 **all** match. This is not a general rule to suppress MariaDB failures. Do not alter these narrow exceptions simply to suppress alerts; validate changes with regression tests. - - -These exceptions are not blanket MariaDB or OOM alert suppression. Changes to monitoring exceptions require regression validation. +Installer upgrades preserve existing customer YAML, including any earlier label rule; they never silently delete or rewrite configured exceptions. These protections are not blanket MariaDB/OOM alert suppression. Validate all predicate and configuration changes with non-destructive regression tests and only hosted isolated Docker acceptance. ### Historical design and validation diff --git a/docs/operations/INSTALLATION_AND_OPERATIONS.md b/docs/operations/INSTALLATION_AND_OPERATIONS.md index 8202236..d05c9b7 100644 --- a/docs/operations/INSTALLATION_AND_OPERATIONS.md +++ b/docs/operations/INSTALLATION_AND_OPERATIONS.md @@ -231,10 +231,14 @@ HA traps use the same OID and var-binds with `cwFailureType=ha-failover` and `cw | `auto_health_check.timeout_seconds` | `2` | Probe timeout | | `container_health_checks` | `{}` | Per-container `http` / `exec` probe overrides | | `excluded_containers` | `[]` | Containers excluded from alerts | -| `ignored_container_events` | Radware-labelled temporary check | Narrow exceptions for known transient operations | +| `ignored_container_events` | `[]` (optional; not generated by installer) | Opt-in suppression of Docker events based on **verified** container labels; not required for known MariaDB dump helpers | | `ha_monitor.enabled` | `false` | Optional active-node change alerts | | `ha_monitor.exporter_url` | `http://127.0.0.1:9664/metrics` | Local HA exporter address | +**Temporary MariaDB SQL dump helpers:** The watchdog recognizes the known CyberController syntax-check helper automatically if the container uses the expected `kvision_infra_mariadb` image, is not the primary MariaDB service, has `AutoRemove=True`, mounts `/mnt/cli/tmp/mariadb_dump.sql.gz`, and exits 137. This narrowly scoped built-in exception does **not** depend on `ignored_container_events`. The installer does not generate an assumed MariaDB role label. Operators can still add explicit `ignored_container_events` entries for other verified, intentionally excluded labelled containers; normal application failures should not be excluded. + +**Upgrades:** Existing `watchdog-config.yaml` files remain unchanged, including an older `com.radware.cybercontroller.role: mariadb-ha-syntax-check` rule if present. That old rule is not required for the observed dump-helper path; review it before removing it from a customer configuration rather than changing it automatically. + A custom probe can override discovery for a specific container. Example: ```yaml diff --git a/docs/operations/RELEASE_NOTES_V1.6.1.md b/docs/operations/RELEASE_NOTES_V1.6.1.md index 5d1db5c..a053879 100644 --- a/docs/operations/RELEASE_NOTES_V1.6.1.md +++ b/docs/operations/RELEASE_NOTES_V1.6.1.md @@ -7,12 +7,13 @@ | Area | Planned change | Impact | |---|---|---| | HA monitoring diagnostics | During failover, tolerate up to two consecutive checks with no promoted HA Operator. On the third consecutive such reading, emit a throttled error; conflicting promoted nodes, loss of quorum and exporter failures remain immediate errors. | Reduces expected transient Syslog ERROR noise without hiding sustained or unsafe HA status. No change to HA failover/failback detection, owner-only notifications, or Docker container monitoring. | +| MariaDB helper configuration clarity | Remove the unverified MariaDB syntax-check Docker label rule from Python defaults, installer-generated configuration and the example template. Retain optional explicit label-based exclusions and the existing narrow built-in dump-helper exception. | Avoids documenting a label not observed on the temporary MariaDB helper; no change to the verified built-in exit-137 suppression. Existing customer YAML remains preserved on upgrades, including older rules. | **Additional changes:** To be updated as development PRs are integrated. ## Compatibility and operations -No configuration key, new service, port, endpoint or privilege is required by the initial HA diagnostic change. Default HA polling remains 60 seconds; the existing 2-valid-poll failover confirmation and 5-minute repeated-error rate limit are retained. +No new service, port, endpoint or privilege is required. Default HA polling remains 60 seconds; the existing 2-valid-poll failover confirmation and 5-minute repeated-error rate limit are retained. Newly generated configs omit the optional `ignored_container_events` key; the reference template defaults it to `[]`. The event-label mechanism is still available for explicit, verified labels. The strict built-in MariaDB dump-helper exception stays enabled and unchanged. Installer upgrades **preserve existing customer YAML**, so they do not automatically remove old label entries. ## Validation (pending) diff --git a/install.sh b/install.sh index 3b0f3c4..b2f3b2e 100755 --- a/install.sh +++ b/install.sh @@ -677,14 +677,6 @@ configure_all() { printf "restart_window_minutes: 10\n" printf "unhealthy_cycles_threshold: 3\n" printf "\nexcluded_containers: []\n" - printf "\nignored_container_events:\n" - printf " - label:\n" - printf " com.radware.cybercontroller.role: mariadb-ha-syntax-check\n" - printf " events:\n" - printf " - die\n" - printf "# Temporary MariaDB HA syntax-check containers should carry the label shown above.\n" - printf "# Fallback: suppress kvision_infra_mariadb AutoRemove dump helpers exiting 137\n" - printf "# only when /mnt/cli/tmp/mariadb_dump.sql.gz is mounted; primary MariaDB stays monitored.\n" printf "\nrunbook_base_url: \"\"\n" printf "\nlog_level: INFO\n" printf "log_file: /var/log/watchdog/watchdog.log\n" diff --git a/tests/test_installer_image.py b/tests/test_installer_image.py index 64629a1..005d8d6 100644 --- a/tests/test_installer_image.py +++ b/tests/test_installer_image.py @@ -142,7 +142,13 @@ def test_existing_configuration_is_not_regenerated(self): def test_keep_existing_configuration_does_not_add_ha(self): # Upgrading with the default "keep" answer must preserve active YAML. - existing = "\n".join(["syslog:", " enabled: true", ""]) + existing = "\n".join([ + "syslog:", " enabled: true", + "ignored_container_events:", + " - label:", + " com.radware.cybercontroller.role: mariadb-ha-syntax-check", + " events: [die]", "", + ]) (self.root / "watchdog-config.yaml").write_text(existing) (self.root / ".env").write_text( "\n".join(["DOCKER_GID=999", "WATCHDOG_VERSION=1.5.4", ""]) @@ -170,6 +176,7 @@ def test_generated_config_includes_optional_ha_disabled(self): self.assertEqual(config["ha_monitor"], { "enabled": False, "exporter_url": "http://127.0.0.1:9664/metrics", }) + self.assertNotIn("ignored_container_events", config) self.assertEqual(self.calls(), []) def test_generated_config_can_enable_ha_with_custom_exporter(self): diff --git a/tests/test_mariadb_event_exceptions.py b/tests/test_mariadb_event_exceptions.py new file mode 100644 index 0000000..af0ab86 --- /dev/null +++ b/tests/test_mariadb_event_exceptions.py @@ -0,0 +1,87 @@ +"""Non-destructive tests for built-in MariaDB helper and optional label policies.""" +from pathlib import Path +from types import SimpleNamespace +import tempfile +import unittest + +import yaml +import watchdog + + +ROOT = Path(__file__).resolve().parents[1] +IMAGE = "reg.radware.com:18442/kvision/kvision_infra_mariadb:1.30.0" +MOUNT = "/mnt/cli/tmp/mariadb_dump.sql.gz" + + +def helper(name="zen_hypatia", *, image=IMAGE, auto_remove=True, mount=MOUNT, labels=None): + labels = {} if labels is None else labels + return SimpleNamespace( + name=name, + labels=labels, + image=SimpleNamespace(tags=[]), + attrs={ + "Config": {"Image": image, "Labels": labels}, + "HostConfig": {"AutoRemove": auto_remove}, + "Mounts": [{"Source": mount, "Destination": mount}], + }, + ) + + +class MariaDBEventExceptionsTests(unittest.TestCase): + def test_unlabelled_realistic_helper_uses_builtin_predicate(self): + container = helper() + attrs = {"name": "zen_hypatia", "image": IMAGE, "exitCode": "137"} + self.assertEqual(watchdog.DEFAULT_CONFIG["ignored_container_events"], []) + self.assertFalse(watchdog.is_ignored_container_event( + container, attrs, "die", watchdog.DEFAULT_CONFIG["ignored_container_events"])) + self.assertTrue(watchdog.is_kvision_mariadb_dump_helper_crash(container, attrs, 137)) + + def test_builtin_predicate_does_not_hide_other_failures(self): + attrs = {"name": "zen_hypatia", "image": IMAGE, "exitCode": "137"} + cases = ( + ("primary MariaDB", helper(name="config_kvision-infra-mariadb_1"), + {**attrs, "name": "config_kvision-infra-mariadb_1"}, 137), + ("exit 1", helper(), attrs, 1), + ("exit 0", helper(), attrs, 0), + ("not AutoRemove", helper(auto_remove=False), attrs, 137), + ("wrong mount", helper(mount="/tmp/other.sql.gz"), attrs, 137), + ("wrong image", helper(image="other_image:1.0"), + {**attrs, "image": "other_image:1.0"}, 137), + ("normal unlabeled service", helper(name="ordinary-service", + image="other_image:1.0"), {"name": "ordinary-service"}, 137), + ) + for description, container, event, exit_code in cases: + with self.subTest(case=description): + self.assertFalse(watchdog.is_kvision_mariadb_dump_helper_crash( + container, event, exit_code)) + + def test_explicit_verified_label_rule_still_works_when_configured(self): + rule = [{"label": {"com.example.test.role": "temporary"}, "events": ["die"]}] + container = helper(labels={"com.example.test.role": "temporary"}) + event = {"name": "zen_hypatia"} + self.assertTrue(watchdog.is_ignored_container_event(container, event, "die", rule)) + self.assertFalse(watchdog.is_ignored_container_event(container, event, "oom", rule)) + self.assertFalse(watchdog.is_ignored_container_event(helper(), event, "die", rule)) + self.assertFalse(watchdog.is_ignored_container_event(container, event, "die", [])) + + def test_runtime_defaults_and_reference_template_have_no_implicit_label_rule(self): + template = yaml.safe_load((ROOT / "watchdog-config.yaml.example").read_text()) + self.assertEqual(template["ignored_container_events"], []) + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary) / "watchdog-config.yaml" + path.write_text("log_level: WARNING\n") + self.assertEqual(watchdog.load_config(str(path))["ignored_container_events"], []) + + path.write_text( + "ignored_container_events:\n" + " - label:\n" + " com.example.test.role: temporary\n" + " events: [die]\n" + ) + self.assertEqual(watchdog.load_config(str(path))["ignored_container_events"], + [{"label": {"com.example.test.role": "temporary"}, + "events": ["die"]}]) + + +if __name__ == "__main__": + unittest.main() diff --git a/watchdog-config.yaml.example b/watchdog-config.yaml.example index 1398a02..01915ae 100644 --- a/watchdog-config.yaml.example +++ b/watchdog-config.yaml.example @@ -35,12 +35,12 @@ lifecycle_detection: excluded_containers: [] # - my-temp-container -# Narrow exception for labelled temporary syntax-check containers. -ignored_container_events: - - label: - com.radware.cybercontroller.role: mariadb-ha-syntax-check - events: - - die +# Optional label-based Docker event exclusions (disabled by default). +# Use only for verified labels on intentionally excluded temporary containers. +# The known CyberController MariaDB SQL dump helper is handled automatically +# by the watchdog's strict image, AutoRemove, mount, and exit-137 checks. +# No label rule is needed for that helper. +ignored_container_events: [] # ── Logging ─────────────────────────────────────────────────────────────────── diff --git a/watchdog.py b/watchdog.py index 734367c..53f669e 100644 --- a/watchdog.py +++ b/watchdog.py @@ -97,9 +97,6 @@ def setup_logging(level: str, log_file: str | None, syslog_cfg: dict | None = No # ── Config ──────────────────────────────────────────────────────────────────── -_KVISION_MARIADB_SYNTAX_CHECK_LABEL = "com.radware.cybercontroller.role" -_KVISION_MARIADB_SYNTAX_CHECK_ROLE = "mariadb-ha-syntax-check" - DEFAULT_CONFIG = { "alert_channels": ["slack"], "check_interval_seconds": 60, @@ -111,14 +108,7 @@ def setup_logging(level: str, log_file: str | None, syslog_cfg: dict | None = No "unhealthy_cycles_threshold": 3, "alert_on_recovery": True, "excluded_containers": [], - "ignored_container_events": [ - { - "label": { - _KVISION_MARIADB_SYNTAX_CHECK_LABEL: _KVISION_MARIADB_SYNTAX_CHECK_ROLE, - }, - "events": ["die"], - }, - ], + "ignored_container_events": [], # opt-in label policy; MariaDB dump helper uses a separate strict predicate "log_file": "/var/log/watchdog/watchdog.log", "log_level": "INFO", # Alert state survives restarts so recovery alerts still fire after a reboot. From 843bfc2d4b894a4e293cbfe939dc3038bc8ed44c Mon Sep 17 00:00:00 2001 From: rdwr-egore Date: Mon, 19 Oct 2026 21:55:51 +0000 Subject: [PATCH 4/5] fix: align v1.6.1 installer YAML with reference settings --- README.md | 2 +- docs/dev/DEVELOPMENT_AND_RELEASE.md | 2 + .../operations/INSTALLATION_AND_OPERATIONS.md | 6 +- docs/operations/RELEASE_NOTES_V1.6.1.md | 3 +- install.sh | 8 ++ tests/test_installer_image.py | 84 ++++++++++++++++++- watchdog-config.yaml.example | 4 +- 7 files changed, 104 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index cdd8165..640e6cb 100644 --- a/README.md +++ b/README.md @@ -281,7 +281,7 @@ failures on application containers. | Version | Date | Author | Changes | |---------|------------|--------|---------| -| 1.6.1 | Pending | — | **In preparation, not published:** HA-monitoring transient transition diagnostics (without hiding safety-critical faults); remove unverified default MariaDB label exception while retaining strict built-in dump-helper handling and optional label policies. See [draft v1.6.1 release notes](docs/operations/RELEASE_NOTES_V1.6.1.md). Additional changes and validation to be recorded before release. | +| 1.6.1 | Pending | — | **In preparation, not published:** HA-monitoring transient transition diagnostics (without hiding safety-critical faults); remove unverified default MariaDB label exception while retaining strict built-in dump-helper handling and optional label policies; complete installer-generated YAML and enforce reference-template parity in tests. See [draft v1.6.1 release notes](docs/operations/RELEASE_NOTES_V1.6.1.md). Additional changes and validation to be recorded before release. | | 1.6.0 | 2026-10-15 | — | Changes since 1.5.4: PRs [#3](https://github.com/Radware/CyberController-Container-Alert/pull/3), [#4](https://github.com/Radware/CyberController-Container-Alert/pull/4), [#5](https://github.com/Radware/CyberController-Container-Alert/pull/5), [#6](https://github.com/Radware/CyberController-Container-Alert/pull/6), [#7](https://github.com/Radware/CyberController-Container-Alert/pull/7), [#8](https://github.com/Radware/CyberController-Container-Alert/pull/8), [#10](https://github.com/Radware/CyberController-Container-Alert/pull/10); corrected restart-loop/recovery lifecycle, planned-stop/exit-137 handling, event ID/name reuse, independent optional HA failover detection, Syslog and notification consistency, one complete versioned release package and wizard, revised informational alert messages, and isolated-only release smoke tests. See [complete v1.6.0 release notes](docs/operations/RELEASE_NOTES_V1.6.0.md). | | 1.5.4 | 2026-09-23 | Rahul Kumar | Versioned Docker image (`watchdog:`) driven by the root `VERSION` file; installer loads the packaged `watchdog.tar`, rejects an archive that does not carry the release tag, aborts on Compose failures, and verifies the running image; restart-loop detection now uses Docker `RestartCount` instead of counting poll observations; `container_health_checks` overrides implemented; fixed `.env` migration on root-owned files; `watchdog-config.yaml` made readable by the non-root container user | | 1.5.3 | 2026-09-16 | Rahul Kumar | Resource Limits Enforced | diff --git a/docs/dev/DEVELOPMENT_AND_RELEASE.md b/docs/dev/DEVELOPMENT_AND_RELEASE.md index c16bb4a..4327040 100644 --- a/docs/dev/DEVELOPMENT_AND_RELEASE.md +++ b/docs/dev/DEVELOPMENT_AND_RELEASE.md @@ -54,6 +54,8 @@ git diff --check The Python test environment needs the repo's test dependencies (including Docker SDK, Requests, PyYAML). Installer/package tests use fake Docker. **Never** run real failure-injection tests or `tests/release-smoke.sh` on operational/shared Docker hosts. +**YAML parity guard:** `tests/test_installer_image.py` runs the installer with a fake Docker executable, parses its generated YAML and the reference `watchdog-config.yaml.example`, and compares **all active nested configuration key paths in both directions**, plus non-wizard default values. Any future setting added to or removed from either file must be reflected in the other. Wizard-specific destinations and SNMPv3-only conditional settings are intentionally not compared for literal values. Preserve existing configuration on upgrade; never force-regenerate customer YAML as part of a parity fix. + **Step 4 — Commit, push, request review.** ```bash diff --git a/docs/operations/INSTALLATION_AND_OPERATIONS.md b/docs/operations/INSTALLATION_AND_OPERATIONS.md index d05c9b7..065654a 100644 --- a/docs/operations/INSTALLATION_AND_OPERATIONS.md +++ b/docs/operations/INSTALLATION_AND_OPERATIONS.md @@ -231,10 +231,14 @@ HA traps use the same OID and var-binds with `cwFailureType=ha-failover` and `cw | `auto_health_check.timeout_seconds` | `2` | Probe timeout | | `container_health_checks` | `{}` | Per-container `http` / `exec` probe overrides | | `excluded_containers` | `[]` | Containers excluded from alerts | -| `ignored_container_events` | `[]` (optional; not generated by installer) | Opt-in suppression of Docker events based on **verified** container labels; not required for known MariaDB dump helpers | +| `ignored_container_events` | `[]` | Installer generates an empty list; opt-in Docker-event exclusions require **verified** container labels. The known MariaDB dump-helper exception does not need a label rule | | `ha_monitor.enabled` | `false` | Optional active-node change alerts | | `ha_monitor.exporter_url` | `http://127.0.0.1:9664/metrics` | Local HA exporter address | +**Generated configuration completeness (v1.6.1):** New installer-generated `watchdog-config.yaml` files now explicitly include every active setting from the reference template, including `lifecycle_detection.enabled: true`, `lifecycle_detection.correlation_window_seconds: 3`, `restart_recovery_stability_seconds: 120`, `alert_on_recovery: true`, and an empty `ignored_container_events` list. Alert destinations and channel-specific options still reflect wizard selections; optional SNMPv3 fields are generated only when SNMPv3 is selected. + +The v1.6.0 installer did not write the first four fields, but the Python application already used those defaults, so their absence did not disable monitoring behavior. **On upgrades, the installer preserves the existing YAML** rather than injecting missing keys. To expose the options on an existing installation, add them manually after backing up the configuration; there is no need to regenerate or overwrite the file merely to retain the original defaults. + **Temporary MariaDB SQL dump helpers:** The watchdog recognizes the known CyberController syntax-check helper automatically if the container uses the expected `kvision_infra_mariadb` image, is not the primary MariaDB service, has `AutoRemove=True`, mounts `/mnt/cli/tmp/mariadb_dump.sql.gz`, and exits 137. This narrowly scoped built-in exception does **not** depend on `ignored_container_events`. The installer does not generate an assumed MariaDB role label. Operators can still add explicit `ignored_container_events` entries for other verified, intentionally excluded labelled containers; normal application failures should not be excluded. **Upgrades:** Existing `watchdog-config.yaml` files remain unchanged, including an older `com.radware.cybercontroller.role: mariadb-ha-syntax-check` rule if present. That old rule is not required for the observed dump-helper path; review it before removing it from a customer configuration rather than changing it automatically. diff --git a/docs/operations/RELEASE_NOTES_V1.6.1.md b/docs/operations/RELEASE_NOTES_V1.6.1.md index a053879..43516f1 100644 --- a/docs/operations/RELEASE_NOTES_V1.6.1.md +++ b/docs/operations/RELEASE_NOTES_V1.6.1.md @@ -8,12 +8,13 @@ |---|---|---| | HA monitoring diagnostics | During failover, tolerate up to two consecutive checks with no promoted HA Operator. On the third consecutive such reading, emit a throttled error; conflicting promoted nodes, loss of quorum and exporter failures remain immediate errors. | Reduces expected transient Syslog ERROR noise without hiding sustained or unsafe HA status. No change to HA failover/failback detection, owner-only notifications, or Docker container monitoring. | | MariaDB helper configuration clarity | Remove the unverified MariaDB syntax-check Docker label rule from Python defaults, installer-generated configuration and the example template. Retain optional explicit label-based exclusions and the existing narrow built-in dump-helper exception. | Avoids documenting a label not observed on the temporary MariaDB helper; no change to the verified built-in exit-137 suppression. Existing customer YAML remains preserved on upgrades, including older rules. | +| Installer configuration parity | Generate the previously omitted `lifecycle_detection` settings, `restart_recovery_stability_seconds`, `alert_on_recovery`, and an explicit empty `ignored_container_events` list. Add `runbook_base_url` to the reference example so all active nested YAML key paths match the installer. Add an automated bidirectional key/default parity test. | New installations expose all documented options. No runtime change to the existing Python defaults; wizard-selected destinations are preserved. Existing customer YAML remains untouched during upgrades. | **Additional changes:** To be updated as development PRs are integrated. ## Compatibility and operations -No new service, port, endpoint or privilege is required. Default HA polling remains 60 seconds; the existing 2-valid-poll failover confirmation and 5-minute repeated-error rate limit are retained. Newly generated configs omit the optional `ignored_container_events` key; the reference template defaults it to `[]`. The event-label mechanism is still available for explicit, verified labels. The strict built-in MariaDB dump-helper exception stays enabled and unchanged. Installer upgrades **preserve existing customer YAML**, so they do not automatically remove old label entries. +No new service, port, endpoint or privilege is required. Default HA polling remains 60 seconds; the existing 2-valid-poll failover confirmation and 5-minute repeated-error rate limit are retained. Newly generated configs explicitly include `ignored_container_events: []` (no rules), matching the reference template, rather than including an unverified MariaDB label rule. The event-label mechanism is still available for explicit, verified labels. The strict built-in MariaDB dump-helper exception stays enabled and unchanged. Installer upgrades **preserve existing customer YAML**, so they do not automatically remove old label entries. ## Validation (pending) diff --git a/install.sh b/install.sh index b2f3b2e..47c7b54 100755 --- a/install.sh +++ b/install.sh @@ -675,8 +675,16 @@ configure_all() { printf "cooldown_minutes: 5\n" printf "restart_threshold: 5\n" printf "restart_window_minutes: 10\n" + printf "restart_recovery_stability_seconds: 120\n" printf "unhealthy_cycles_threshold: 3\n" + printf "alert_on_recovery: true\n" + printf "\n# Correlate Docker-requested stops with delayed die events.\n" + printf "lifecycle_detection:\n" + printf " enabled: true\n" + printf " correlation_window_seconds: 3\n" printf "\nexcluded_containers: []\n" + printf "# Optional verified label-based event exclusions; no default MariaDB label rule.\n" + printf "ignored_container_events: []\n" printf "\nrunbook_base_url: \"\"\n" printf "\nlog_level: INFO\n" printf "log_file: /var/log/watchdog/watchdog.log\n" diff --git a/tests/test_installer_image.py b/tests/test_installer_image.py index 005d8d6..e380b54 100644 --- a/tests/test_installer_image.py +++ b/tests/test_installer_image.py @@ -15,6 +15,17 @@ VERSION = (ROOT / "VERSION").read_text().strip() +def config_key_paths(config: dict, prefix: str = "") -> set[str]: + """All active YAML key paths, including parent objects and empty mappings.""" + found: set[str] = set() + for key, value in config.items(): + path = f"{prefix}.{key}" if prefix else str(key) + found.add(path) + if isinstance(value, dict): + found.update(config_key_paths(value, path)) + return found + + class InstallerImageTests(unittest.TestCase): def setUp(self): self.workspace = tempfile.TemporaryDirectory(prefix="watchdog-installer-fake-") @@ -176,7 +187,78 @@ def test_generated_config_includes_optional_ha_disabled(self): self.assertEqual(config["ha_monitor"], { "enabled": False, "exporter_url": "http://127.0.0.1:9664/metrics", }) - self.assertNotIn("ignored_container_events", config) + self.assertEqual(config["ignored_container_events"], []) + self.assertEqual(self.calls(), []) + + def test_generated_yaml_matches_all_active_reference_keys_and_defaults(self): + # Regression guard: every new reference YAML field must also be + # generated by the wizard. Runs only in the fake-Docker workspace. + body = ('detect_docker_gid() { echo 999; }; ' + '_publish_file() { cp "$1" "$2"; }; ' + 'DC=docker; configure_all') + result = self.run_function( + body, input_data="\n".join(["4", "", "", "", "", "", ""]) + "\n" + ) + self.assertEqual(result.returncode, 0, result.output) + generated = yaml.safe_load((self.root / "watchdog-config.yaml").read_text()) + template = yaml.safe_load((ROOT / "watchdog-config.yaml.example").read_text()) + + # Key parity in BOTH directions, including nested settings, avoids + # silently omitting a new option (or documenting an obsolete option). + self.assertEqual(config_key_paths(generated), config_key_paths(template)) + # Wizard-picked destinations/channels necessarily differ from + # example placeholders; all non-interactive defaults must agree. + non_interactive = ( + "check_interval_seconds", "cooldown_minutes", + "restart_threshold", "restart_window_minutes", + "restart_recovery_stability_seconds", "unhealthy_cycles_threshold", + "alert_on_recovery", "lifecycle_detection", + "excluded_containers", "ignored_container_events", + "log_level", "log_file", "state_file", "boot_grace_seconds", + "runbook_base_url", "auto_health_check", "container_health_checks", + "ha_monitor", + ) + for setting in non_interactive: + with self.subTest(setting=setting): + self.assertEqual(generated[setting], template[setting]) + + self.assertEqual(generated["lifecycle_detection"], + {"enabled": True, "correlation_window_seconds": 3}) + self.assertEqual(generated["restart_recovery_stability_seconds"], 120) + self.assertIs(generated["alert_on_recovery"], True) + self.assertEqual(generated["ignored_container_events"], []) + self.assertEqual(self.calls(), []) + + def test_snmpv3_wizard_retains_reference_settings_and_keeps_secrets_in_env(self): + # Conditional SNMPv3 settings are additional to the active reference + # schema, not a reason to omit any common setting. + body = ('detect_docker_gid() { echo 999; }; ' + '_publish_file() { cp "$1" "$2"; }; ' + 'DC=docker; configure_all') + inputs = [ + "3", "nms.example.invalid", "162", "v3", + "snmpuser", "SHA", "example-auth-value", "AES", + "example-priv-value", "", "", + ] + result = self.run_function(body, input_data=chr(10).join(inputs) + chr(10)) + self.assertEqual(result.returncode, 0, result.output) + config = yaml.safe_load((self.root / "watchdog-config.yaml").read_text()) + template = yaml.safe_load((ROOT / "watchdog-config.yaml.example").read_text()) + self.assertTrue(config_key_paths(template) <= config_key_paths(config)) + self.assertEqual(config["alert_channels"], ["snmp_trap"]) + self.assertEqual(config["snmp_trap"]["version"], "v3") + self.assertEqual(config["snmp_trap"]["v3_username"], "snmpuser") + self.assertEqual(config["snmp_trap"]["v3_auth_key_env"], "SNMP_V3_AUTH_KEY") + self.assertEqual(config["snmp_trap"]["v3_priv_key_env"], "SNMP_V3_PRIV_KEY") + self.assertEqual(config["lifecycle_detection"], template["lifecycle_detection"]) + self.assertEqual(config["ignored_container_events"], []) + self.assertNotIn("example-auth-value", + (self.root / "watchdog-config.yaml").read_text()) + self.assertNotIn("example-priv-value", + (self.root / "watchdog-config.yaml").read_text()) + env = (self.root / ".env").read_text() + self.assertIn("SNMP_V3_AUTH_KEY=example-auth-value", env) + self.assertIn("SNMP_V3_PRIV_KEY=example-priv-value", env) self.assertEqual(self.calls(), []) def test_generated_config_can_enable_ha_with_custom_exporter(self): diff --git a/watchdog-config.yaml.example b/watchdog-config.yaml.example index 01915ae..b60a0c9 100644 --- a/watchdog-config.yaml.example +++ b/watchdog-config.yaml.example @@ -42,7 +42,9 @@ excluded_containers: [] # No label rule is needed for that helper. ignored_container_events: [] - +# Optional runbook URL included in alert details (leave empty when not configured). +runbook_base_url: "" + # ── Logging ─────────────────────────────────────────────────────────────────── log_level: INFO # DEBUG | INFO | WARNING | ERROR (default INFO) log_file: /var/log/watchdog/watchdog.log # bind-mounted to ./watchdog/watchdog.log on host From 134a0cb53dc8a4caf35efc41c0112ad990c559bf Mon Sep 17 00:00:00 2001 From: rdwr-egore Date: Tue, 20 Oct 2026 19:58:35 +0000 Subject: [PATCH 5/5] docs(v1.6.1): add isolated SMTP post-install validation and safe cleanup --- .../operations/INSTALLATION_AND_OPERATIONS.md | 72 ++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/docs/operations/INSTALLATION_AND_OPERATIONS.md b/docs/operations/INSTALLATION_AND_OPERATIONS.md index 065654a..20457d6 100644 --- a/docs/operations/INSTALLATION_AND_OPERATIONS.md +++ b/docs/operations/INSTALLATION_AND_OPERATIONS.md @@ -9,6 +9,7 @@ This guide covers **installation, configuration, day-to-day operations, upgrades - [Get and verify the release package](#21-get-and-verify-the-release-package) - [Run the installer](#22-run-the-installer) 3. [Verify the installation](#3-verify-the-installation) + - [Optional isolated SMTP end-to-end validation](#31-optional-isolated-smtp-end-to-end-validation) 4. [Configuration reference](#4-configuration-reference) - [Configuration files and editing](#41-configuration-files-and-editing) - [Watchdog monitoring settings](#42-watchdog-monitoring-settings) @@ -112,7 +113,7 @@ The watchdog should be `running` and become `healthy`; the image should be `watc | `./watchdog/state.json` | Incident, HA and reboot state; do not edit manually | | `grep 'watchdog_alert' watchdog/watchdog.log` | Find existing alert records | -Check that the selected Syslog, SMTP, SNMP or Slack receiver can be reached and inspect any alerts already received. No artificial container failure is needed. +Check that the selected Syslog, SMTP, SNMP or Slack receiver can be reached and inspect any alerts already received. No artificial failure is needed for the basic checks. Optional real SMTP delivery validation is described in Section 3.1 below. **Optional HA check:** If `ha_monitor.enabled: true`, check the local exporter on **each HA node**: @@ -123,6 +124,75 @@ curl -fsS http://127.0.0.1:9664/metrics Use the configured exporter URL if different. The first successful reading establishes a baseline; it does not send a failover alert. Test a real switchover only during planned HA maintenance. +### 3.1 Optional isolated SMTP end-to-end validation + +This optional post-install test creates **only a disposable container** to exercise a real Docker `die` event, watchdog CRASH detection, and SMTP dispatch. It never modifies or stops CyberController application containers. + +> **PRODUCTION CAUTION:** Run only with approval, after checking free host capacity and notifying the email/on-call recipients. A real **CRITICAL TEST** alert may be sent to **all enabled channels**, not just SMTP. Do not stop, kill, or restart real service containers to test notifications. Wait until the watchdog is healthy and any post-reboot grace period has ended. Do **not** run the test with `--rm`: the watchdog needs time to inspect the stopped container. + +**1. Verify configuration (read-only)** from the watchdog installation directory. Confirm `smtp` appears in `alert_channels`, `smtp.enabled: true`, and the recipient is correct. Never print the secret values from `.env`. + +```bash +grep -A6 '^alert_channels:' watchdog-config.yaml +grep -A12 '^smtp:' watchdog-config.yaml +docker inspect docker-container-watchdog --format '{{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{end}}' +docker logs --tail 30 docker-container-watchdog +``` + +**2. Cause one isolated nonzero exit.** Run all commands in the **same shell**, retaining `NAME` for cleanup. Uses only the installed local watchdog image (no pull), without a network, ports, mounts, or restart policy; resource limits apply only to the disposable container. + +```bash +NAME="ca-smtp-test-$(date +%s)" +IMG="$(docker inspect --format '{{.Config.Image}}' docker-container-watchdog)" +docker run -d \ + --pull=never \ + --name "$NAME" \ + --label watchdog.validation=smtp \ + --network none \ + --read-only \ + --no-healthcheck \ + --restart no \ + --memory 64m \ + --cpus 0.10 \ + --pids-limit 32 \ + --entrypoint python3 \ + "$IMG" -c 'import time,sys; time.sleep(10); sys.exit(42)' + +docker wait "$NAME" # Expect 42 +docker inspect "$NAME" --format 'ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' +``` + +Expected: `ExitCode=42 OOMKilled=false`. An unexpected nonzero exit generates a CRITICAL `crashed` alert after the default three-second stop-correlation window. + +**3. Verify end-to-end SMTP:** Wait for the correlation window, then check the watchdog log and **recipient mailbox**. Expect a subject containing `[CRITICAL] Container CRASHED` and the exact `ca-smtp-test-...` name, plus exit code 42. At INFO log level successful submission produces `SMTP: alert sent to ... recipient(s)`; the SMTP server accepting a message **does not guarantee mailbox delivery**. + +```bash +sleep 5 +docker logs --since 5m docker-container-watchdog 2>&1 | grep -E "$NAME|SMTP:|watchdog_alert" +``` + +If the alert is missing, investigate the watchdog logs, SMTP relay, spam filtering, channel settings, and delivery receipt before considering another test. Avoid repeated CRITICAL notifications. + +**4. Mandatory cleanup (also after a failed test):** Remove **only** the named disposable container after collecting evidence. This guard refuses removal unless both the name prefix and test label match. + +```bash +case "$NAME" in + ca-smtp-test-*) + if [ "$(docker inspect --format '{{index .Config.Labels "watchdog.validation"}}' "$NAME")" = "smtp" ]; then + docker rm "$NAME" + else + echo "Refusing cleanup: test label missing" >&2 + fi + ;; + *) + echo "Refusing cleanup: unexpected container name" >&2 + ;; +esac +``` + +> **Cleanup caution:** Never use `docker rm -f`, `docker container prune`, or `docker system prune` for this test. If the shell/variable was lost, use `docker ps -a --filter label=watchdog.validation=smtp`, verify the **exact** container identity, and remove that one test container only. Nothing should be removed from the actual CyberController stack. + + ## 4. Configuration reference The installer creates the active configuration files. Edit them only when needed; most installations require no manual changes.