Repository navigation
alert_transitions API returns incomplete transitions (raises missing) #1155
Description
Summary
POST /api/v2/spaces/{spaceId}/rooms/{roomId}/alert_transitions returns only part of the alert transitions in the requested window. Transitions into WARNING/CRITICAL are missing far more often than transitions back to CLEAR, so any count of "alerts raised" built on this endpoint undercounts, and raise/clear pairs cannot be matched.
Evidence
Same space and room, same time windows, payload { "after": <s>, "before": <s> } (no node or chart filters):
| Window | Source | Result |
|---|---|---|
| last 3h | alert_transitions (frontend endpoint) |
1,098 transitions in the room; for all azure_monitor.* alerts together: CLEAR→CRITICAL 5, CRITICAL→CLEAR 15, CLEAR→WARNING 9, WARNING→CLEAR 12 |
| last 3h | MCP get_alert_transitions, one single azure_monitor.* alert |
36 transitions for that alert alone: CLEAR→CRITICAL 18, CRITICAL→CLEAR 18 |
| last 24h | alert_transitions (frontend endpoint) |
6,942 transitions in the room; for azure_monitor.* alerts: CLEAR→CRITICAL 7 vs CRITICAL→CLEAR 27, CLEAR→WARNING 10 vs WARNING→CLEAR 43 |
One alert alone has more raises in 3 hours than the endpoint returns for all alerts of that collector, and in every window the endpoint returns many more clears than raises.
Other observations
- The response has no
old.duration, so time spent in the previous state cannot be read directly. - Adding an unsupported
contextsfield to the payload changes the result to a 100-row, unfiltered response instead of being ignored or rejected.
Expected
All transitions in the window are returned (or the response says it is truncated and supports paging), and request fields that are not supported are rejected or ignored without changing the result.
Impact
Frontend cards that count raised alerts over a time range (alerts raised over time, raised/resolved counts, mean time to resolve, noisiest resources) undercount, especially for flapping alerts.
Activity
car12o commentedon Nov 5, 2026
Fixed and deployed. The endpoint now returns every transition in the window, newest first. It pages them and reports a total, and each row now includes
old.duration.Why raises were missing
A history response merged two sources:
node_instance_alarms, presented as a transition.Most instances are CLEAR at any given moment, so the second source was mostly
→CLEARrows, includingUNKNOWN→CLEARinitialisations. A flapping alert showed up once, as its last clear. That is how 3h returned 1,098 rows with only 5CLEAR→CRITICAL. On staging the endpoint covered 119 of 322 transitions over 24h and 147 of 1,763 over 7d, with 4CLEAR→CRITICALagainst 42 real ones.The
contextsobservation was a separate bug. Alarm-processor acceptedcontextsand forwarded it, but lean search never mapped it, so you got the room's newest 100 events unfiltered.chartswas lost the same way: feed-svc had renamed the key, and alarm-processor still sent the old name.What callers get now
page_sizeis 1–1000 and defaults to 1000. Passnext_cursorback ascursoruntil it comes back empty.totalis the match count across all pages. Paging no longer skips events that share a timestamp.contextsandchartsfilter. Names must match exactly:["azure_monitor.*"]matches nothing, so list the full context names.afterandbeforeare Unix timestamps in milliseconds.event_id, which is unique per row (transition_idcan repeat), plusunitsandvalue_stringonoldandnew.old.durationis the number of seconds spent in the status left. The top-levelnon_clear_durationis the number of seconds since the alert left CLEAR. Both are omitted when unknown. Events written before 1.29.3 have noold.duration.transitionsstill read stored state, up to 1000 IDs on one page.Also found and fixed
actions, so any space member could read events their role does not allow.CLEAR→CLEAR). It was dropped from history and from notifications. Those clears now appear and notify.get_alert_transitionstool reported truncated results as complete. It now reports "N of total".Changes
chartsaschart_idsold.duration,non_clear_durationcontextsfilter and action permissionsNetdata.alert.previous.duration