Skip to content

test: wait for every trigger to reach the latest due time after scheduler fault recovery - #1380

Closed
kvlonge wants to merge 2 commits into
mainfrom
fix/scheduler-fault-catch-up
Closed

kvlonge wants to merge 2 commits into
mainfrom
fix/scheduler-fault-catch-up

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Why

The two-scheduler fault test failed once on a slow Windows runner with 89 triggers "missing" their latest tick and its duplicate probe accepted. Neither was a scheduler fault. The scheduler ticks only the latest missed occurrence, so a trigger first evaluated after the next minute begins ticks that minute and skips the earlier one. The test's final drain ran a fixed four cycles and then required a tick at exactly the latest minute before settling. Whenever that drain crossed a minute boundary, the test failed.

Changes

  • After settling, recovery cycles until every trigger has ticked at or after the latest due time, bounded by a five-minute deadline, instead of a fixed four cycles.
  • A trigger passes the liveness check when its newest tick is at or after that time.
  • A tick scheduled after the clock reading it was recorded with counts as off schedule, so a scheduler that ticks early still fails.
  • The duplicate probe now inserts a time the trigger actually ticked, so it still proves the uniqueness guard.
  • The duplicate-tick, off-schedule, run/tick, request-event and stale-write fencing checks are unchanged.

Verification

  • A scratch change forced the final drain across a minute boundary after 200 of 400 triggers. The unchanged test then failed with missing_latest_ticks: 200, accepted_duplicate_ticks: 1, the same signature as the CI failure. The changed test reported no anomalies, and the loop stopped as soon as no trigger was behind.
  • A scratch scheduler mutation that ticks the next minute early fails with off_schedule_ticks: 400.
  • The full fault test passes on SQLite and PostgreSQL (5 seeds each).
  • make check-ci and fensu check pass.

DuckShop: no edge case added, because this changes a scheduler test's recovery expectation, not a project shape, data value or lifecycle sequence.

…uler fault recovery

The scheduler ticks only the latest missed occurrence, so a trigger first
evaluated after the next minute begins ticks that minute and skips the
earlier one. When the final recovery drain crossed a minute boundary on a
slow runner, the fault test counted those triggers as missing their latest
tick, and its duplicate probe at that minute was accepted for the same
reason.

Recovery now cycles until every trigger has ticked at or after the latest
due time at settling, bounded by a five-minute deadline. Each trigger's
newest tick must reach that time, and the duplicate probe re-inserts a time
the trigger actually ticked. The duplicate, off-schedule, run, event and
fencing checks are unchanged.
@kvlonge
kvlonge marked this pull request as draft November 8, 2026 19:51
Requiring each trigger's newest tick to be at or after the latest due time
has no upper bound, so a scheduler that ticked the next occurrence early
would pass. Each tick records the clock it was planned with, and no tick
may be for a later time.
@kvlonge
kvlonge marked this pull request as ready for review November 8, 2026 20:14

kvlonge commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Folded into the single combined sqlbuild PR (D625/D626), which the SQLBuild lane is assembling now and opens once #1377 lands. Branch kept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant