diff --git a/gitbooks/job-queue.md b/gitbooks/job-queue.md index 42ad16e..aacd546 100644 --- a/gitbooks/job-queue.md +++ b/gitbooks/job-queue.md @@ -146,14 +146,17 @@ Eligibility is `status='ready' AND available_at_ms <= now AND kind NOT IN (retir ```text ORDER BY CASE kind - WHEN 'seal' THEN 1 - WHEN 'flush_stale' THEN 2 - WHEN 'append_buffer' THEN 3 - ELSE 4 -- extract_chunk, reembed_backfill, seal_document + WHEN 'seal' THEN 1 + WHEN 'reembed_backfill' THEN 2 + WHEN 'flush_stale' THEN 3 + WHEN 'append_buffer' THEN 4 + ELSE 5 -- extract_chunk, seal_document END ASC, available_at_ms ASC ``` +`reembed_backfill` sits right behind `seal` because it is the only path that writes chunk vectors (`extract_chunk` no longer embeds inline) and it holds the same single LLM permit as every `extract_chunk`. Ranked with the extraction backlog, the gate-busy defer would round-robin it behind that whole backlog and vectors would trail extraction by the backlog's length. Each backfill step embeds one bounded batch, defers `REEMBED_BACKFILL_REVISIT_MS` (750 ms), and settles `Done` once the space is covered, so extraction is never starved in return. + `DEFAULT_LOCK_DURATION_MS = 5 * 60 * 1000` (5 min) — comfortably larger than any expected single-job runtime, so a crashed worker's row is recovered after the window without leaving real failures stuck for hours. Retry backoff is exponential: `backoff_ms(attempts)` = `min(60s * 2^(attempts-1), 1h)` (`RETRY_BASE_MS = 60s`, `RETRY_CAP_MS = 1h`), so the first retry waits 60s, then 120s, 240s, … capped at one hour. ## Worker loop diff --git a/src/memory/queue/store.rs b/src/memory/queue/store.rs index 567c211..4ef11ac 100644 --- a/src/memory/queue/store.rs +++ b/src/memory/queue/store.rs @@ -107,6 +107,10 @@ pub(crate) fn enqueue_conn( /// Sets `status=running`, bumps `attempts`, stamps `started_at_ms` and /// `locked_until_ms`. Returns `None` when the queue is empty / not yet due. /// +/// Due rows are ranked by kind — `seal`, then `reembed_backfill`, then +/// `flush_stale`, then `append_buffer`, then everything else — and by +/// `available_at_ms` within a rank (the query below explains why). +/// /// Retired kinds (`topic_route`, `digest_daily`) are excluded from the claim so /// a leftover old-queue row never reaches `row_to_job` (which would fail to /// parse it). [`purge_retired_jobs`] removes such rows. @@ -120,6 +124,18 @@ pub fn claim_next(config: &MemoryConfig, lock_duration_ms: i64) -> Result