fix(alerts): the rules no longer evaluate in one herd

[rldyourmnd]

The alert channel has been repeating the same page every ten minutes for hours, and alert_evaluation_failed has been firing without a broken rule behind it. Both have one cause.

OpenObserve snaps an aligned alert's next run to the previous interval boundary for its frequency (TriggerCondition::get_aligned_next_trigger_time), so with align_time: true every one-minute rule evaluates at :00 and every ten-minute rule at :00, :10, :20 — thirty-four alerts issuing their searches and persisting their group states in the same millisecond.

Measured on the live services host:

• the simultaneous searches queue (metrics->search: wait in queue, all at the same timestamp) until some exceed the PromQL load-data timeout — that is what raised alert_evaluation_failed, sixteen times in two minutes, with each failing expression completing in 40–57 ms when run alone
• the state writes queue behind SQLite's single serialised writer until they are refused: could not persist group states ...: database is locked, 2386 times in one day, about 190 an hour, peaking on the ten-minute boundaries
• a scheduler that cannot persist that it already notified notifies again on the next tick, which is exactly the repetition in the channel's history

The host is not short of anything: load over 24 h is p50 0.69, p90 1.84, p99 3.49 on four cores, and only one sample of 288 exceeded four. The metadata store is 112 MB with 204 k file_list rows from 326 metric streams, which is why the single writer matters at all.

align_time: false gives every alert its own schedule. Nothing in these rules depends on clock alignment: each expression takes its own range relative to evaluation time. A test refuses an aligned alert.

https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp