Optimize polling query plans

[cardmagic]

Summary

• bump solid-objects from 0.14.5 to 0.14.6
• install canonical ordered polling indexes in schema migration 8
• use FOR UPDATE SKIP LOCKED for PostgreSQL and MySQL effect, reminder, and broadcast claims
• compare separate pending/available and stale-recovery heads without locking, lock only the oldest row by primary key, and retry past candidates locked by another claimant
• run SQL claim transactions at transaction-scoped read committed isolation so InnoDB recovery scans cannot defeat SKIP LOCKED with next-key locks

This is the Node equivalent of cardmagic/solid-objects-ruby#60, adapted to the Node implementation rather than copied mechanically.

Production evidence

LEADx Web App on MySQL reported these Solid Objects polling loads around 2026-09-03 06:00 UTC:

• effect: 1187.12 ms for pending work ordered by (available_at, id)
• broadcast: 1229.03 ms for a combined pending/stale OR query ordered by (available_at, id)
• reminder: 1251.63 ms for scheduled work with an available-or-stale claim predicate

The Ruby tables already had plausible leading polling indexes, and the Ruby fix removed the broadcast OR plan. The Node schema had no corresponding effect, reminder, or broadcast polling indexes. Node broadcasts already used a separate recovery statement, but that statement was a broad UPDATE; under MySQL's default repeatable-read isolation it took a next-key lock that blocked another claimant before SKIP LOCKED could help. The reminder cleanup had the same concurrency failure.

Generated SQL and plans

PostgreSQL and MySQL generate one ordered locking probe for effects. Reminders and broadcasts use separate ordered, limited, non-locking probes for their available/pending and stale categories, compare those heads, then reselect only the oldest candidate by primary key with FOR UPDATE SKIP LOCKED. If that row is locked elsewhere, the claimant excludes it and repeats the indexed peeks. SQLite keeps its serialized transaction path and omits the lock clause.

Candidate probes contain only the outbox table so MySQL cannot drive them through the instances join, materialize the queue, and lock extra rows. Actor identity is loaded by primary key after a successful effect or reminder claim.

Plans were measured on dedicated databases with 50,000 production-shaped rows. These are isolated EXPLAIN ANALYZE timings, not the original shared-production latency. The after timings below are for the ordered category probes; the final candidate lock is a primary-key lookup.

Probe	SQLite before -> after	PostgreSQL 18 before -> after	MySQL 8.4 before -> after
effect pending	scan + temp sort, 0.907 ms -> ordered index, 0.008 ms	seq scan + sort, 1.245 ms -> index scan, 0.024 ms	table scan + sort, 12.2 ms -> covering index, 0.102 ms
reminder available	scan + temp sort, 0.889 ms -> ordered index, 0.009 ms	seq scan + sort, 1.410 ms -> index scan, 0.022 ms	table scan + sort, 18.3 ms -> index range scan, 0.163 ms
reminder stale	same ordered index shape	seq scan + sort, 31.471 ms including JIT (4.116 ms plan nodes) -> index scan, 0.094 ms	table scan + sort, 9.70 ms -> index range scan, 0.121 ms
broadcast pending	correlated scans + temp sort, 944.210 ms -> both indexes, 0.018 ms	seq/hash anti join + sort, 4.053 ms -> nested anti join with both indexes, 0.046 ms	table/hash anti join + sort, 37 ms -> both indexes, 1.2 ms
broadcast stale	same indexed broadcast shape	seq/hash anti join + sort, 161.179 ms including JIT (about 4 ms in plan nodes) -> both indexes, 0.074 ms	table/hash anti join + sort, 37 ms -> both indexes, 0.117 ms

The broadcast revision guard needs a separate (instance_id, state_revision, status) index; the outer delivery-order probe uses (status, available_at_ms, id).

The isolated plans show defective Node query/schema shapes before this change. They do not establish that every second of the LEADx samples came from those plans: shared database contention can add lock wait time, and should be investigated separately in production telemetry.

Correctness and concurrency

The claimant tests pause the first transaction after it locks its selected row and require a second claimant to finish inside a one-second database deadline. Mixed available/stale reminder and pending/stale broadcast tests prove the first transaction does not hide the unselected category head. The second claimant retries past the locked oldest row and receives the other row on MySQL and PostgreSQL. MySQL also retains same-category effect coverage.

Pending and stale candidates are compared by (available_at_ms, id) for broadcasts and (run_at_ms, id) for reminders. Exact locking queries recheck eligibility before conditional updates. Attempt counts, per-instance broadcast revision ordering, retry/recovery behavior, ownership checks, and at-least-once delivery remain intact.

Migration and compatibility

Schema migration 8 adds:

• effects_poll(status, available_at_ms, id)
• reminders_due(status, run_at_ms, id)
• broadcasts_poll(status, available_at_ms, id)
• broadcasts_instance_revision(instance_id, state_revision, status)

The migration is additive, records its version only after all indexes exist, and is safe to retry: SQLite/PostgreSQL use IF NOT EXISTS, while MySQL checks information_schema.statistics. A version-7 upgrade test runs installation twice and verifies exact index columns. Index construction still takes each adapter's normal DDL locks, so large existing installations should run install() as part of a controlled deploy.

Database.transaction gains an optional DatabaseTransactionOptions argument. Built-in PostgreSQL and MySQL adapters honor read_committed; SQLite keeps its serialized transaction. Custom SQL adapters are documented to honor the option for claim correctness.

TDD record

Observed red before each behavior change:

• pnpm exec vitest run test/polling-queries.test.ts — expected locking probes; all three original queries ended at LIMIT 1
• pnpm exec vitest run test/dead-letters.test.ts -t "upgrades an existing version-one database" — expected migrations 1-8, received 1-7
• MySQL broadcast concurrency regression — DatabaseDeadlineExceeded while the second claimant waited behind the first transaction's recovery range lock
• MySQL effect/reminder concurrency regressions — both failed with DatabaseDeadlineExceeded; after applying read committed, reminder still returned no second row until broad cleanup was replaced
• strengthened reminder query-shape regression — expected five probes, received four
• no-join candidate regression — MySQL had driven the reminder query through instances, sorted and locked both due rows before LIMIT 1
• mixed recovery-probe regressions — the second claimant received undefined for reminders and broadcasts on MySQL and broadcasts on PostgreSQL because the first transaction locked both category heads

The same focused tests passed green after their implementations. New rejection capture narrows non-Error values before exposing a concrete Error in the tests.

Validation

• pnpm run format:check
• pnpm run check
• pnpm run test
• pnpm run test:coverage — 51 files, 360 passed, 17 skipped
• pnpm run build
• pnpm run pack:check
• pnpm run test:package
• pnpm run test:recovery
• pnpm run test:at-least-once
• pnpm run test:browser — 9 passed
• pnpm audit --audit-level=high — no known vulnerabilities
• SOLID_OBJECTS_DATABASE_URL=postgresql://... pnpm run test:postgresql on PostgreSQL 18 — 13 passed
• SOLID_OBJECTS_DATABASE_URL=mysql://... pnpm run test:mysql on MySQL 8.4 — 9 passed
• SOLID_OBJECTS_REDIS_URL=redis://... pnpm run test:redis on Redis 7 — 3 passed

[greptile-apps]

Greptile Summary

The PR optimizes effect, reminder, and broadcast polling by adding ordered indexes, selecting category heads without locks, and locking only the chosen row under read-committed transactions.

• Adds schema migration 8 with polling and broadcast revision indexes.
• Adds transaction isolation options to the database contract and built-in PostgreSQL/MySQL adapters.
• Refactors reminder and broadcast recovery into separate probes with primary-key locking and retry exclusions.
• Adds cross-database query-shape, migration, and concurrency coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/repository.ts	Refactors effect, reminder, and broadcast claims around ordered probes, exact-row locks, eligibility rechecks, and retry exclusions.
src/schema.ts	Adds retry-safe migration 8 installation for the four polling-related indexes.
src/database/mysql.ts	Adds optional transaction-scoped read-committed isolation before beginning a MySQL transaction.
src/database/postgresql.ts	Adds optional read-committed isolation to PostgreSQL transaction startup.
test/support/pausing-claim-database.ts	Adds reusable claim-pausing and rejection-capture test support.

Sequence Diagram

sequenceDiagram
    participant W as Claiming worker
    participant DB as Database
    W->>DB: Probe available/pending head
    DB-->>W: Category candidate
    W->>DB: Probe stale head
    DB-->>W: Recovery candidate
    W->>W: Select globally earliest candidate
    W->>DB: Lock candidate by primary key (SKIP LOCKED)
    alt Candidate locked elsewhere or changed
        DB-->>W: No row
        W->>W: Exclude candidate ID
        W->>DB: Repeat category probes
    else Candidate locked
        W->>DB: Conditionally update claim ownership
        DB-->>W: Claimed work
    end

Loading

_{Reviews (2): Last reviewed commit: "fix: avoid locking unused candidates" | Re-trigger Greptile}

[cardmagic]

CI follow-up (50b5f21): MySQL was not exposing a timing-only test flake. EXPLAIN ANALYZE showed the reminder locking query choosing instances as the outer side of the join, materializing and sorting both due reminders, and locking both before LIMIT 1; the second claimant therefore skipped every row.

A new red query-shape assertion rejected joins in candidate probes. Effect and reminder probes now lock only their indexed outbox table, then load actor identity by primary key after a successful claim. The MySQL reminder race passed eight consecutive focused runs locally, the complete MySQL 8.4 suite, and both GitHub MySQL 8.0/8.4 jobs. The full replacement CI matrix is green.