From a3cc39a42b768cf19b38529ffeb35fd0f230ee5e Mon Sep 17 00:00:00 2001 From: rldyourmnd Date: Wed, 2 Sep 2026 21:54:49 +0500 Subject: [PATCH] fix(alerts): the update ticket waits for the host's own automation host_standard_updates_available fired the moment any package appeared. unattended-upgrades runs daily on every fleet host with the Ubuntu archive and -security as allowed origins, so a security update announced in the afternoon is gone by the next morning without anyone reading a ticket. And Ubuntu phases updates per machine, so one set of nine packages reached the five hosts on different days: on 2026-09-02 the rule ticketed four times for that single set, once per host as its phase arrived. The expression now asks whether the count stayed above zero through a full automation cycle -- twenty-six hours, the daily apt timer plus its randomised delay. A package the automation will never take, one from noble-updates rather than -security or a phased deferral, still surfaces a day later, which is when it stops being the machine's problem. Claude-Session: https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp --- config/observability-rules.yaml | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/config/observability-rules.yaml b/config/observability-rules.yaml index 09e10a5..0cb04a7 100644 --- a/config/observability-rules.yaml +++ b/config/observability-rules.yaml @@ -345,11 +345,26 @@ rules: summary: A fleet host still requires a reboot after package maintenance. action: Drain jobs naturally and use the rolling reboot procedure while preserving cluster quorum. recovery: No fleet host reports a pending reboot for ten minutes. + # Fires on updates the host's own automation did not take, not on updates + # that merely appeared. unattended-upgrades runs daily on every fleet host + # and its allowed origins are the Ubuntu archive and -security, so a + # security update announced in the afternoon is gone by the next morning + # without anyone reading a ticket. Ubuntu also phases updates per machine, + # so the same package reaches the five hosts on five different days: on + # 2026-09-02 this rule ticketed four times for one set of nine packages, + # each host as its phase arrived. + # + # min_over_time asks the question that is worth a ticket: did the count stay + # above zero through a full automation cycle? Twenty-six hours covers the + # daily apt timer plus its randomised delay. A package the automation will + # never take -- one from noble-updates rather than -security, or a phased + # one it defers -- still surfaces, one day later, which is when it becomes + # a human's problem rather than the machine's. - id: host_standard_updates_available severity: ticket query_language: promql stream_name: gha_fleet_host_standard_updates_available - expression: max by (host_name) (gha_fleet_host_standard_updates_available) + expression: min_over_time((max by (host_name) (gha_fleet_host_standard_updates_available))[26h:10m]) operator: ">" threshold: 0 evaluation_seconds: 300 @@ -362,9 +377,9 @@ rules: enabled: true owner: fleet-operations runbook: https://github.com/NDDev-OpenNetwork/github-actions/blob/main/docs/runbooks/fleet-alerts.md - summary: Standard Ubuntu package updates remain available on a fleet host. - action: Apply the normal rolling package-maintenance procedure and prove service and runner recovery after each member. - recovery: Every fleet host reports zero standard updates for one hour. + summary: Package updates have sat on a fleet host through a full unattended-upgrades cycle. + action: Read whether unattended-upgrades is failing or the package is one it will not take (noble-updates, or a phased deferral), then apply the rolling procedure and prove service and runner recovery. + recovery: Every fleet host has reached zero standard updates at least once in the last twenty-six hours. - id: host_swap_high severity: ticket query_language: promql