Skip to content

test: drain every due trigger after scheduler fault recovery - #1327

Merged
kvlonge merged 2 commits into
mainfrom
test/fault-recovery-drain
Nov 6, 2026
Merged

kvlonge merged 2 commits into
mainfrom
test/fault-recovery-drain

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Why

The two-scheduler fault test can fail on slow runners with missing_latest_ticks greater than zero. After recovery it checks the latest due time with one scheduler cycle, but a cycle evaluates at most 100 due triggers and the test declares 400. When recovery crosses a minute boundary, every trigger becomes due again, and one cycle ticks only 100 of them. Scheduler behaviour is correct: later cycles tick the rest.

Changes

  • After taking the settled time, the test runs enough cycles to drain all 400 triggers (400 / 100 = 4), not just one. Expected ticks are still bounded by the settled time, so the assertion is no weaker.

This is test-only and adds no DuckShop case: it fixes the test harness, not a project shape, data value or lifecycle sequence.

Verification

  • Forced the failure with a temporary sleep across a minute boundary before the settled time (since removed). The old helper reported missing_latest_ticks: 300 (400 triggers minus one 100-trigger page). The new helper reported no anomalies.
  • cargo test -p sqlbuild-orchestration --lib -- scheduler::tests::contracts::ticks on SQLite and PostgreSQL, default 5 fault seeds: 10/10 pass.
  • make check-ci passes.

The fault test checked recovery with one scheduler cycle, which evaluates at
most 100 due triggers. With 400 triggers, a slow runner that crossed a minute
boundary during recovery left the rest without their latest tick. Run enough
cycles to drain every trigger due at the settled time.
@kvlonge
kvlonge enabled auto-merge (squash) November 6, 2026 13:17
@kvlonge
kvlonge merged commit 0455861 into main Nov 6, 2026
48 checks passed
@kvlonge
kvlonge deleted the test/fault-recovery-drain branch November 6, 2026 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant