Skip to content
Draft
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,7 @@ failures on application containers.

| Version | Date | Author | Changes |
|---------|------------|--------|---------|
| 1.6.1 | Pending | — | **In preparation, not published:** HA-monitoring transient transition diagnostics (without hiding safety-critical faults); remove unverified default MariaDB label exception while retaining strict built-in dump-helper handling and optional label policies; complete installer-generated YAML and enforce reference-template parity in tests. See [draft v1.6.1 release notes](docs/operations/RELEASE_NOTES_V1.6.1.md). Additional changes and validation to be recorded before release. |
| 1.6.0 | 2026-10-15 | — | Changes since 1.5.4: PRs [#3](https://github.com/Radware/CyberController-Container-Alert/pull/3), [#4](https://github.com/Radware/CyberController-Container-Alert/pull/4), [#5](https://github.com/Radware/CyberController-Container-Alert/pull/5), [#6](https://github.com/Radware/CyberController-Container-Alert/pull/6), [#7](https://github.com/Radware/CyberController-Container-Alert/pull/7), [#8](https://github.com/Radware/CyberController-Container-Alert/pull/8), [#10](https://github.com/Radware/CyberController-Container-Alert/pull/10); corrected restart-loop/recovery lifecycle, planned-stop/exit-137 handling, event ID/name reuse, independent optional HA failover detection, Syslog and notification consistency, one complete versioned release package and wizard, revised informational alert messages, and isolated-only release smoke tests. See [complete v1.6.0 release notes](docs/operations/RELEASE_NOTES_V1.6.0.md). |
| 1.5.4 | 2026-09-23 | Rahul Kumar | Versioned Docker image (`watchdog:<VERSION>`) driven by the root `VERSION` file; installer loads the packaged `watchdog.tar`, rejects an archive that does not carry the release tag, aborts on Compose failures, and verifies the running image; restart-loop detection now uses Docker `RestartCount` instead of counting poll observations; `container_health_checks` overrides implemented; fixed `.env` migration on root-owned files; `watchdog-config.yaml` made readable by the non-root container user |
| 1.5.3 | 2026-09-16 | Rahul Kumar | Resource Limits Enforced |
Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.6.0
1.6.1
10 changes: 5 additions & 5 deletions docs/dev/DEVELOPMENT_AND_RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ git diff --check

The Python test environment needs the repo's test dependencies (including Docker SDK, Requests, PyYAML). Installer/package tests use fake Docker. **Never** run real failure-injection tests or `tests/release-smoke.sh` on operational/shared Docker hosts.

**YAML parity guard:** `tests/test_installer_image.py` runs the installer with a fake Docker executable, parses its generated YAML and the reference `watchdog-config.yaml.example`, and compares **all active nested configuration key paths in both directions**, plus non-wizard default values. Any future setting added to or removed from either file must be reflected in the other. Wizard-specific destinations and SNMPv3-only conditional settings are intentionally not compared for literal values. Preserve existing configuration on upgrade; never force-regenerate customer YAML as part of a parity fix.

**Step 4 — Commit, push, request review.**

```bash
Expand Down Expand Up @@ -291,13 +293,11 @@ Production deployment requires a published, prebuilt release package.

### Narrow CyberController event exceptions

The observed CyberController MariaDB SQL dump syntax-check helper is handled by an **application-specific built-in predicate**: the helper image, non-primary container name, Docker `AutoRemove=True` status, expected SQL dump file mount and exit status 137 must **all** match. Normal primary MariaDB and unrelated containers remain monitored. This exception does not require `ignored_container_events` in YAML.

The intended MariaDB HA syntax-check helper carries the explicit label `com.radware.cybercontroller.role=mariadb-ha-syntax-check`. The configured `ignored_container_events` policy ignores only the designated temporary helper `die` event; normal service container failures remain monitored.

A narrow compatibility fallback exists for dynamic MariaDB dump helpers: it applies only when the helper image, non-primary container name, Docker `AutoRemove` status, expected SQL dump file mount and exit status 137 **all** match. This is not a general rule to suppress MariaDB failures. Do not alter these narrow exceptions simply to suppress alerts; validate changes with regression tests.

`ignored_container_events` remains a **generic, optional label-based event policy**, but no default MariaDB label is generated. In the October 2026 HA lab run, the expected dump-helper exit reached the built-in predicate instead of matching the previously advertised `com.radware.cybercontroller.role=mariadb-ha-syntax-check` rule. Do not claim that CyberController sets that label without concrete evidence. If a future supported helper does carry a verified label, an operator may deliberately configure a narrow `ignored_container_events` rule.

These exceptions are not blanket MariaDB or OOM alert suppression. Changes to monitoring exceptions require regression validation.
Installer upgrades preserve existing customer YAML, including any earlier label rule; they never silently delete or rewrite configured exceptions. These protections are not blanket MariaDB/OOM alert suppression. Validate all predicate and configuration changes with non-destructive regression tests and only hosted isolated Docker acceptance.

### Historical design and validation

Expand Down
4 changes: 3 additions & 1 deletion docs/dev/HA_MONITORING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
| Run HA polling in a **separate thread and HTTP session** from Docker polling | Slow or blocked Docker API calls must not delay HA observations; HA exporter failures must not interrupt container monitoring |
| Require **quorum, a successful Pacemaker scrape, one local node and exactly one promoted owner** | Missing or conflicting telemetry is not proof of a role change; fail closed instead of inventing a failover |
| Require **two consecutive valid observations** of a new active node | Avoid reporting transient/promoted-resource ambiguity while the cluster is transitioning |
| Allow **two consecutive no-promoted-owner observations** before logging an ERROR on the third | A failover may temporarily have no active/promoted operator. Avoid transient Syslog noise, but do not hide persistent loss of an owner |
| Report **multiple promoted owners, exporter failures and loss of quorum immediately** | These are not an ordinary empty-promotion gap and must not be silenced by the grace period |
| **Only the newly active node** emits the alert | Watchdogs run on both nodes; duplicate notifications from old and new active nodes would be misleading |
| Persist the confirmed active node using the existing state file | Survive watchdog restarts without replaying an old role change or creating a second state store |
| Reuse the existing notification channels and SNMP MIB | Treat HA as another monitored event without adding a new alert transport or breaking container notifications |
Expand All @@ -21,7 +23,7 @@
2. It reads `ha_cluster_pacemaker_resources{resource="HAOperatorRep-Res",role="promoted",status="active"} == 1` to identify the active owner and `ha_cluster_corosync_member_votes{local="true"}` to identify the local node. The Pacemaker scrape-success and quorum metrics must both be 1.
3. The **first valid reading** establishes a silent baseline. An active-node change must appear in **two valid consecutive checks** before it is confirmed.
4. On a confirmed transition A → B, both nodes can update their saved baseline, but **only the watchdog local to B** issues one `WARNING` `ha-failover` event with previous/new node details.
5. Invalid/unavailable metrics leave the last confirmed baseline unchanged and reset a pending candidate. Exporter errors are logged at most once every five minutes; they never trigger a speculative failover alert.
5. Invalid/unavailable metrics leave the last confirmed baseline unchanged and reset a pending candidate. A valid Pacemaker scrape with quorum but **zero promoted HA Operator** matches is treated as a possible promotion gap: the first two consecutive observations are DEBUG-only; the third consecutive observation generates an ERROR. At the default 60-second interval, the third observation is about two minutes after the first. A valid observation resets the counter. **Multiple promoted owners**, failed Pacemaker scrape/lost quorum, and exporter transport failures generate an immediate ERROR. Repeated identical diagnostic errors are throttled to at most once every five minutes; a different failure reason is logged immediately. These observations never trigger a speculative failover alert.
6. HA state is kept separately from container incidents in `watchdog/state.json`. Restart with monitoring enabled restores that state. Running the watchdog with HA monitoring disabled clears its baseline, so a later re-enable starts silently.
7. During the host reboot grace interval, HA role checks are deferred. HA events are not included in container recovery or reboot-summary logic.

Expand Down
84 changes: 82 additions & 2 deletions docs/operations/INSTALLATION_AND_OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ This guide covers **installation, configuration, day-to-day operations, upgrades
- [Get and verify the release package](#21-get-and-verify-the-release-package)
- [Run the installer](#22-run-the-installer)
3. [Verify the installation](#3-verify-the-installation)
- [Optional isolated SMTP end-to-end validation](#31-optional-isolated-smtp-end-to-end-validation)
4. [Configuration reference](#4-configuration-reference)
- [Configuration files and editing](#41-configuration-files-and-editing)
- [Watchdog monitoring settings](#42-watchdog-monitoring-settings)
Expand Down Expand Up @@ -112,7 +113,7 @@ The watchdog should be `running` and become `healthy`; the image should be `watc
| `./watchdog/state.json` | Incident, HA and reboot state; do not edit manually |
| `grep 'watchdog_alert' watchdog/watchdog.log` | Find existing alert records |

Check that the selected Syslog, SMTP, SNMP or Slack receiver can be reached and inspect any alerts already received. No artificial container failure is needed.
Check that the selected Syslog, SMTP, SNMP or Slack receiver can be reached and inspect any alerts already received. No artificial failure is needed for the basic checks. Optional real SMTP delivery validation is described in Section 3.1 below.

**Optional HA check:** If `ha_monitor.enabled: true`, check the local exporter on **each HA node**:

Expand All @@ -123,6 +124,75 @@ curl -fsS http://127.0.0.1:9664/metrics
Use the configured exporter URL if different. The first successful reading establishes a baseline; it does not send a failover alert. Test a real switchover only during planned HA maintenance.


### 3.1 Optional isolated SMTP end-to-end validation

This optional post-install test creates **only a disposable container** to exercise a real Docker `die` event, watchdog CRASH detection, and SMTP dispatch. It never modifies or stops CyberController application containers.

> **PRODUCTION CAUTION:** Run only with approval, after checking free host capacity and notifying the email/on-call recipients. A real **CRITICAL TEST** alert may be sent to **all enabled channels**, not just SMTP. Do not stop, kill, or restart real service containers to test notifications. Wait until the watchdog is healthy and any post-reboot grace period has ended. Do **not** run the test with `--rm`: the watchdog needs time to inspect the stopped container.

**1. Verify configuration (read-only)** from the watchdog installation directory. Confirm `smtp` appears in `alert_channels`, `smtp.enabled: true`, and the recipient is correct. Never print the secret values from `.env`.

```bash
grep -A6 '^alert_channels:' watchdog-config.yaml
grep -A12 '^smtp:' watchdog-config.yaml
docker inspect docker-container-watchdog --format '{{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{end}}'
docker logs --tail 30 docker-container-watchdog
```

**2. Cause one isolated nonzero exit.** Run all commands in the **same shell**, retaining `NAME` for cleanup. Uses only the installed local watchdog image (no pull), without a network, ports, mounts, or restart policy; resource limits apply only to the disposable container.

```bash
NAME="ca-smtp-test-$(date +%s)"
IMG="$(docker inspect --format '{{.Config.Image}}' docker-container-watchdog)"
docker run -d \
--pull=never \
--name "$NAME" \
--label watchdog.validation=smtp \
--network none \
--read-only \
--no-healthcheck \
--restart no \
--memory 64m \
--cpus 0.10 \
--pids-limit 32 \
--entrypoint python3 \
"$IMG" -c 'import time,sys; time.sleep(10); sys.exit(42)'

docker wait "$NAME" # Expect 42
docker inspect "$NAME" --format 'ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}'
```

Expected: `ExitCode=42 OOMKilled=false`. An unexpected nonzero exit generates a CRITICAL `crashed` alert after the default three-second stop-correlation window.

**3. Verify end-to-end SMTP:** Wait for the correlation window, then check the watchdog log and **recipient mailbox**. Expect a subject containing `[CRITICAL] Container CRASHED` and the exact `ca-smtp-test-...` name, plus exit code 42. At INFO log level successful submission produces `SMTP: alert sent to ... recipient(s)`; the SMTP server accepting a message **does not guarantee mailbox delivery**.

```bash
sleep 5
docker logs --since 5m docker-container-watchdog 2>&1 | grep -E "$NAME|SMTP:|watchdog_alert"
```

If the alert is missing, investigate the watchdog logs, SMTP relay, spam filtering, channel settings, and delivery receipt before considering another test. Avoid repeated CRITICAL notifications.

**4. Mandatory cleanup (also after a failed test):** Remove **only** the named disposable container after collecting evidence. This guard refuses removal unless both the name prefix and test label match.

```bash
case "$NAME" in
ca-smtp-test-*)
if [ "$(docker inspect --format '{{index .Config.Labels "watchdog.validation"}}' "$NAME")" = "smtp" ]; then
docker rm "$NAME"
else
echo "Refusing cleanup: test label missing" >&2
fi
;;
*)
echo "Refusing cleanup: unexpected container name" >&2
;;
esac
```

> **Cleanup caution:** Never use `docker rm -f`, `docker container prune`, or `docker system prune` for this test. If the shell/variable was lost, use `docker ps -a --filter label=watchdog.validation=smtp`, verify the **exact** container identity, and remove that one test container only. Nothing should be removed from the actual CyberController stack.


## 4. Configuration reference

The installer creates the active configuration files. Edit them only when needed; most installations require no manual changes.
Expand Down Expand Up @@ -231,10 +301,18 @@ HA traps use the same OID and var-binds with `cwFailureType=ha-failover` and `cw
| `auto_health_check.timeout_seconds` | `2` | Probe timeout |
| `container_health_checks` | `{}` | Per-container `http` / `exec` probe overrides |
| `excluded_containers` | `[]` | Containers excluded from alerts |
| `ignored_container_events` | Radware-labelled temporary check | Narrow exceptions for known transient operations |
| `ignored_container_events` | `[]` | Installer generates an empty list; opt-in Docker-event exclusions require **verified** container labels. The known MariaDB dump-helper exception does not need a label rule |
| `ha_monitor.enabled` | `false` | Optional active-node change alerts |
| `ha_monitor.exporter_url` | `http://127.0.0.1:9664/metrics` | Local HA exporter address |

**Generated configuration completeness (v1.6.1):** New installer-generated `watchdog-config.yaml` files now explicitly include every active setting from the reference template, including `lifecycle_detection.enabled: true`, `lifecycle_detection.correlation_window_seconds: 3`, `restart_recovery_stability_seconds: 120`, `alert_on_recovery: true`, and an empty `ignored_container_events` list. Alert destinations and channel-specific options still reflect wizard selections; optional SNMPv3 fields are generated only when SNMPv3 is selected.

The v1.6.0 installer did not write the first four fields, but the Python application already used those defaults, so their absence did not disable monitoring behavior. **On upgrades, the installer preserves the existing YAML** rather than injecting missing keys. To expose the options on an existing installation, add them manually after backing up the configuration; there is no need to regenerate or overwrite the file merely to retain the original defaults.

**Temporary MariaDB SQL dump helpers:** The watchdog recognizes the known CyberController syntax-check helper automatically if the container uses the expected `kvision_infra_mariadb` image, is not the primary MariaDB service, has `AutoRemove=True`, mounts `/mnt/cli/tmp/mariadb_dump.sql.gz`, and exits 137. This narrowly scoped built-in exception does **not** depend on `ignored_container_events`. The installer does not generate an assumed MariaDB role label. Operators can still add explicit `ignored_container_events` entries for other verified, intentionally excluded labelled containers; normal application failures should not be excluded.

**Upgrades:** Existing `watchdog-config.yaml` files remain unchanged, including an older `com.radware.cybercontroller.role: mariadb-ha-syntax-check` rule if present. That old rule is not required for the observed dump-helper path; review it before removing it from a customer configuration rather than changing it automatically.

A custom probe can override discovery for a specific container. Example:

```yaml
Expand All @@ -258,6 +336,8 @@ ha_monitor:

Enable this only for HA deployments. Verify its URL read-only, then restart **only** the watchdog ([Section 4.6](#46-apply-configuration-changes)). The first valid reading sets a baseline without sending an alert.

Starting in v1.6.1, up to two consecutive HA exporter observations with **no promoted HA Operator** are treated as a possible short failover transition and do not generate Syslog ERROR noise. The third consecutive observation is an ERROR (about two minutes after the first with default 60-second polling). Multiple promoted owners, exporter failures and missing quorum remain immediate errors; the watchdog does not automatically remediate Pacemaker or change application containers.

### 4.5 Environment variables

The installer manages most `.env` values. SMTP passwords, Slack webhooks and SNMPv3 passphrases belong here, not in YAML. The SNMPv1/v2c community is configured in YAML, so protect that file too.
Expand Down
Loading
Loading