Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@ failures on application containers.

| Version | Date | Author | Changes |
|---------|------------|--------|---------|
| 1.6.1 | Pending | — | **In preparation, not published:** HA-monitoring transient transition diagnostics (zero promoted HA Operator tolerated briefly; ambiguous/safety-critical conditions still reported). See [draft v1.6.1 release notes](docs/operations/RELEASE_NOTES_V1.6.1.md). Additional changes and validation to be recorded before release. |
| 1.6.1 | Pending | — | **In preparation, not published:** HA-monitoring transient transition diagnostics (without hiding safety-critical faults); remove unverified default MariaDB label exception while retaining strict built-in dump-helper handling and optional label policies. See [draft v1.6.1 release notes](docs/operations/RELEASE_NOTES_V1.6.1.md). Additional changes and validation to be recorded before release. |
| 1.6.0 | 2026-10-15 | — | Changes since 1.5.4: PRs [#3](https://github.com/Radware/CyberController-Container-Alert/pull/3), [#4](https://github.com/Radware/CyberController-Container-Alert/pull/4), [#5](https://github.com/Radware/CyberController-Container-Alert/pull/5), [#6](https://github.com/Radware/CyberController-Container-Alert/pull/6), [#7](https://github.com/Radware/CyberController-Container-Alert/pull/7), [#8](https://github.com/Radware/CyberController-Container-Alert/pull/8), [#10](https://github.com/Radware/CyberController-Container-Alert/pull/10); corrected restart-loop/recovery lifecycle, planned-stop/exit-137 handling, event ID/name reuse, independent optional HA failover detection, Syslog and notification consistency, one complete versioned release package and wizard, revised informational alert messages, and isolated-only release smoke tests. See [complete v1.6.0 release notes](docs/operations/RELEASE_NOTES_V1.6.0.md). |
| 1.5.4 | 2026-09-23 | Rahul Kumar | Versioned Docker image (`watchdog:<VERSION>`) driven by the root `VERSION` file; installer loads the packaged `watchdog.tar`, rejects an archive that does not carry the release tag, aborts on Compose failures, and verifies the running image; restart-loop detection now uses Docker `RestartCount` instead of counting poll observations; `container_health_checks` overrides implemented; fixed `.env` migration on root-owned files; `watchdog-config.yaml` made readable by the non-root container user |
| 1.5.3 | 2026-09-16 | Rahul Kumar | Resource Limits Enforced |
Expand Down
8 changes: 3 additions & 5 deletions docs/dev/DEVELOPMENT_AND_RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,13 +291,11 @@ Production deployment requires a published, prebuilt release package.

### Narrow CyberController event exceptions

The observed CyberController MariaDB SQL dump syntax-check helper is handled by an **application-specific built-in predicate**: the helper image, non-primary container name, Docker `AutoRemove=True` status, expected SQL dump file mount and exit status 137 must **all** match. Normal primary MariaDB and unrelated containers remain monitored. This exception does not require `ignored_container_events` in YAML.

The intended MariaDB HA syntax-check helper carries the explicit label `com.radware.cybercontroller.role=mariadb-ha-syntax-check`. The configured `ignored_container_events` policy ignores only the designated temporary helper `die` event; normal service container failures remain monitored.
`ignored_container_events` remains a **generic, optional label-based event policy**, but no default MariaDB label is generated. In the October 2026 HA lab run, the expected dump-helper exit reached the built-in predicate instead of matching the previously advertised `com.radware.cybercontroller.role=mariadb-ha-syntax-check` rule. Do not claim that CyberController sets that label without concrete evidence. If a future supported helper does carry a verified label, an operator may deliberately configure a narrow `ignored_container_events` rule.

A narrow compatibility fallback exists for dynamic MariaDB dump helpers: it applies only when the helper image, non-primary container name, Docker `AutoRemove` status, expected SQL dump file mount and exit status 137 **all** match. This is not a general rule to suppress MariaDB failures. Do not alter these narrow exceptions simply to suppress alerts; validate changes with regression tests.


These exceptions are not blanket MariaDB or OOM alert suppression. Changes to monitoring exceptions require regression validation.
Installer upgrades preserve existing customer YAML, including any earlier label rule; they never silently delete or rewrite configured exceptions. These protections are not blanket MariaDB/OOM alert suppression. Validate all predicate and configuration changes with non-destructive regression tests and only hosted isolated Docker acceptance.

### Historical design and validation

Expand Down
6 changes: 5 additions & 1 deletion docs/operations/INSTALLATION_AND_OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,10 +231,14 @@ HA traps use the same OID and var-binds with `cwFailureType=ha-failover` and `cw
| `auto_health_check.timeout_seconds` | `2` | Probe timeout |
| `container_health_checks` | `{}` | Per-container `http` / `exec` probe overrides |
| `excluded_containers` | `[]` | Containers excluded from alerts |
| `ignored_container_events` | Radware-labelled temporary check | Narrow exceptions for known transient operations |
| `ignored_container_events` | `[]` (optional; not generated by installer) | Opt-in suppression of Docker events based on **verified** container labels; not required for known MariaDB dump helpers |
| `ha_monitor.enabled` | `false` | Optional active-node change alerts |
| `ha_monitor.exporter_url` | `http://127.0.0.1:9664/metrics` | Local HA exporter address |

**Temporary MariaDB SQL dump helpers:** The watchdog recognizes the known CyberController syntax-check helper automatically if the container uses the expected `kvision_infra_mariadb` image, is not the primary MariaDB service, has `AutoRemove=True`, mounts `/mnt/cli/tmp/mariadb_dump.sql.gz`, and exits 137. This narrowly scoped built-in exception does **not** depend on `ignored_container_events`. The installer does not generate an assumed MariaDB role label. Operators can still add explicit `ignored_container_events` entries for other verified, intentionally excluded labelled containers; normal application failures should not be excluded.

**Upgrades:** Existing `watchdog-config.yaml` files remain unchanged, including an older `com.radware.cybercontroller.role: mariadb-ha-syntax-check` rule if present. That old rule is not required for the observed dump-helper path; review it before removing it from a customer configuration rather than changing it automatically.

A custom probe can override discovery for a specific container. Example:

```yaml
Expand Down
3 changes: 2 additions & 1 deletion docs/operations/RELEASE_NOTES_V1.6.1.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,13 @@
| Area | Planned change | Impact |
|---|---|---|
| HA monitoring diagnostics | During failover, tolerate up to two consecutive checks with no promoted HA Operator. On the third consecutive such reading, emit a throttled error; conflicting promoted nodes, loss of quorum and exporter failures remain immediate errors. | Reduces expected transient Syslog ERROR noise without hiding sustained or unsafe HA status. No change to HA failover/failback detection, owner-only notifications, or Docker container monitoring. |
| MariaDB helper configuration clarity | Remove the unverified MariaDB syntax-check Docker label rule from Python defaults, installer-generated configuration and the example template. Retain optional explicit label-based exclusions and the existing narrow built-in dump-helper exception. | Avoids documenting a label not observed on the temporary MariaDB helper; no change to the verified built-in exit-137 suppression. Existing customer YAML remains preserved on upgrades, including older rules. |

**Additional changes:** To be updated as development PRs are integrated.

## Compatibility and operations

No configuration key, new service, port, endpoint or privilege is required by the initial HA diagnostic change. Default HA polling remains 60 seconds; the existing 2-valid-poll failover confirmation and 5-minute repeated-error rate limit are retained.
No new service, port, endpoint or privilege is required. Default HA polling remains 60 seconds; the existing 2-valid-poll failover confirmation and 5-minute repeated-error rate limit are retained. Newly generated configs omit the optional `ignored_container_events` key; the reference template defaults it to `[]`. The event-label mechanism is still available for explicit, verified labels. The strict built-in MariaDB dump-helper exception stays enabled and unchanged. Installer upgrades **preserve existing customer YAML**, so they do not automatically remove old label entries.

## Validation (pending)

Expand Down
8 changes: 0 additions & 8 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -677,14 +677,6 @@ configure_all() {
printf "restart_window_minutes: 10\n"
printf "unhealthy_cycles_threshold: 3\n"
printf "\nexcluded_containers: []\n"
printf "\nignored_container_events:\n"
printf " - label:\n"
printf " com.radware.cybercontroller.role: mariadb-ha-syntax-check\n"
printf " events:\n"
printf " - die\n"
printf "# Temporary MariaDB HA syntax-check containers should carry the label shown above.\n"
printf "# Fallback: suppress kvision_infra_mariadb AutoRemove dump helpers exiting 137\n"
printf "# only when /mnt/cli/tmp/mariadb_dump.sql.gz is mounted; primary MariaDB stays monitored.\n"
printf "\nrunbook_base_url: \"\"\n"
printf "\nlog_level: INFO\n"
printf "log_file: /var/log/watchdog/watchdog.log\n"
Expand Down
9 changes: 8 additions & 1 deletion tests/test_installer_image.py
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,13 @@ def test_existing_configuration_is_not_regenerated(self):

def test_keep_existing_configuration_does_not_add_ha(self):
# Upgrading with the default "keep" answer must preserve active YAML.
existing = "\n".join(["syslog:", " enabled: true", ""])
existing = "\n".join([
"syslog:", " enabled: true",
"ignored_container_events:",
" - label:",
" com.radware.cybercontroller.role: mariadb-ha-syntax-check",
" events: [die]", "",
])
(self.root / "watchdog-config.yaml").write_text(existing)
(self.root / ".env").write_text(
"\n".join(["DOCKER_GID=999", "WATCHDOG_VERSION=1.5.4", ""])
Expand Down Expand Up @@ -170,6 +176,7 @@ def test_generated_config_includes_optional_ha_disabled(self):
self.assertEqual(config["ha_monitor"], {
"enabled": False, "exporter_url": "http://127.0.0.1:9664/metrics",
})
self.assertNotIn("ignored_container_events", config)
self.assertEqual(self.calls(), [])

def test_generated_config_can_enable_ha_with_custom_exporter(self):
Expand Down
87 changes: 87 additions & 0 deletions tests/test_mariadb_event_exceptions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
"""Non-destructive tests for built-in MariaDB helper and optional label policies."""
from pathlib import Path
from types import SimpleNamespace
import tempfile
import unittest

import yaml
import watchdog


ROOT = Path(__file__).resolve().parents[1]
IMAGE = "reg.radware.com:18442/kvision/kvision_infra_mariadb:1.30.0"
MOUNT = "/mnt/cli/tmp/mariadb_dump.sql.gz"


def helper(name="zen_hypatia", *, image=IMAGE, auto_remove=True, mount=MOUNT, labels=None):
labels = {} if labels is None else labels
return SimpleNamespace(
name=name,
labels=labels,
image=SimpleNamespace(tags=[]),
attrs={
"Config": {"Image": image, "Labels": labels},
"HostConfig": {"AutoRemove": auto_remove},
"Mounts": [{"Source": mount, "Destination": mount}],
},
)


class MariaDBEventExceptionsTests(unittest.TestCase):
def test_unlabelled_realistic_helper_uses_builtin_predicate(self):
container = helper()
attrs = {"name": "zen_hypatia", "image": IMAGE, "exitCode": "137"}
self.assertEqual(watchdog.DEFAULT_CONFIG["ignored_container_events"], [])
self.assertFalse(watchdog.is_ignored_container_event(
container, attrs, "die", watchdog.DEFAULT_CONFIG["ignored_container_events"]))
self.assertTrue(watchdog.is_kvision_mariadb_dump_helper_crash(container, attrs, 137))

def test_builtin_predicate_does_not_hide_other_failures(self):
attrs = {"name": "zen_hypatia", "image": IMAGE, "exitCode": "137"}
cases = (
("primary MariaDB", helper(name="config_kvision-infra-mariadb_1"),
{**attrs, "name": "config_kvision-infra-mariadb_1"}, 137),
("exit 1", helper(), attrs, 1),
("exit 0", helper(), attrs, 0),
("not AutoRemove", helper(auto_remove=False), attrs, 137),
("wrong mount", helper(mount="/tmp/other.sql.gz"), attrs, 137),
("wrong image", helper(image="other_image:1.0"),
{**attrs, "image": "other_image:1.0"}, 137),
("normal unlabeled service", helper(name="ordinary-service",
image="other_image:1.0"), {"name": "ordinary-service"}, 137),
)
for description, container, event, exit_code in cases:
with self.subTest(case=description):
self.assertFalse(watchdog.is_kvision_mariadb_dump_helper_crash(
container, event, exit_code))

def test_explicit_verified_label_rule_still_works_when_configured(self):
rule = [{"label": {"com.example.test.role": "temporary"}, "events": ["die"]}]
container = helper(labels={"com.example.test.role": "temporary"})
event = {"name": "zen_hypatia"}
self.assertTrue(watchdog.is_ignored_container_event(container, event, "die", rule))
self.assertFalse(watchdog.is_ignored_container_event(container, event, "oom", rule))
self.assertFalse(watchdog.is_ignored_container_event(helper(), event, "die", rule))
self.assertFalse(watchdog.is_ignored_container_event(container, event, "die", []))

def test_runtime_defaults_and_reference_template_have_no_implicit_label_rule(self):
template = yaml.safe_load((ROOT / "watchdog-config.yaml.example").read_text())
self.assertEqual(template["ignored_container_events"], [])
with tempfile.TemporaryDirectory() as temporary:
path = Path(temporary) / "watchdog-config.yaml"
path.write_text("log_level: WARNING\n")
self.assertEqual(watchdog.load_config(str(path))["ignored_container_events"], [])

path.write_text(
"ignored_container_events:\n"
" - label:\n"
" com.example.test.role: temporary\n"
" events: [die]\n"
)
self.assertEqual(watchdog.load_config(str(path))["ignored_container_events"],
[{"label": {"com.example.test.role": "temporary"},
"events": ["die"]}])


if __name__ == "__main__":
unittest.main()
12 changes: 6 additions & 6 deletions watchdog-config.yaml.example
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,12 @@ lifecycle_detection:
excluded_containers: []
# - my-temp-container

# Narrow exception for labelled temporary syntax-check containers.
ignored_container_events:
- label:
com.radware.cybercontroller.role: mariadb-ha-syntax-check
events:
- die
# Optional label-based Docker event exclusions (disabled by default).
# Use only for verified labels on intentionally excluded temporary containers.
# The known CyberController MariaDB SQL dump helper is handled automatically
# by the watchdog's strict image, AutoRemove, mount, and exit-137 checks.
# No label rule is needed for that helper.
ignored_container_events: []


# ── Logging ───────────────────────────────────────────────────────────────────
Expand Down
12 changes: 1 addition & 11 deletions watchdog.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,6 @@ def setup_logging(level: str, log_file: str | None, syslog_cfg: dict | None = No


# ── Config ────────────────────────────────────────────────────────────────────
_KVISION_MARIADB_SYNTAX_CHECK_LABEL = "com.radware.cybercontroller.role"
_KVISION_MARIADB_SYNTAX_CHECK_ROLE = "mariadb-ha-syntax-check"

DEFAULT_CONFIG = {
"alert_channels": ["slack"],
"check_interval_seconds": 60,
Expand All @@ -111,14 +108,7 @@ def setup_logging(level: str, log_file: str | None, syslog_cfg: dict | None = No
"unhealthy_cycles_threshold": 3,
"alert_on_recovery": True,
"excluded_containers": [],
"ignored_container_events": [
{
"label": {
_KVISION_MARIADB_SYNTAX_CHECK_LABEL: _KVISION_MARIADB_SYNTAX_CHECK_ROLE,
},
"events": ["die"],
},
],
"ignored_container_events": [], # opt-in label policy; MariaDB dump helper uses a separate strict predicate
"log_file": "/var/log/watchdog/watchdog.log",
"log_level": "INFO",
# Alert state survives restarts so recovery alerts still fire after a reboot.
Expand Down
Loading