From d9851a011b1041abffc639558d503b4f95409fdb Mon Sep 17 00:00:00 2001 From: Shanu Date: Mon, 7 Sep 2026 12:39:13 +0530 Subject: [PATCH] fix(queue): rank reembed_backfill right after seal in claim_next The deduped reembed_backfill row is the only writer of chunk vectors since extract_chunk stopped embedding inline, and it shares the single-permit LLM gate with every extract_chunk. In the ELSE bucket, the 50 ms gate-busy defer round-robined it behind the whole extraction backlog, so vectors trailed extraction by the length of the backlog (bursts 7 to 12 minutes apart in the field). Rank it second so each freed permit goes to the backfill while it has work; it still embeds one bounded batch, defers REEMBED_BACKFILL_REVISIT_MS and settles Done once covered, so extraction is not starved in return. Adds two store tests pinning the claim ladder and updates the job-queue gitbook to match the query. Closes #168 --- gitbooks/job-queue.md | 11 ++- src/memory/queue/store.rs | 25 ++++++- src/memory/queue/store_tests.rs | 126 +++++++++++++++++++++++++++++++- 3 files changed, 153 insertions(+), 9 deletions(-) diff --git a/gitbooks/job-queue.md b/gitbooks/job-queue.md index 42ad16ef..aacd5464 100644 --- a/gitbooks/job-queue.md +++ b/gitbooks/job-queue.md @@ -146,14 +146,17 @@ Eligibility is `status='ready' AND available_at_ms <= now AND kind NOT IN (retir ```text ORDER BY CASE kind - WHEN 'seal' THEN 1 - WHEN 'flush_stale' THEN 2 - WHEN 'append_buffer' THEN 3 - ELSE 4 -- extract_chunk, reembed_backfill, seal_document + WHEN 'seal' THEN 1 + WHEN 'reembed_backfill' THEN 2 + WHEN 'flush_stale' THEN 3 + WHEN 'append_buffer' THEN 4 + ELSE 5 -- extract_chunk, seal_document END ASC, available_at_ms ASC ``` +`reembed_backfill` sits right behind `seal` because it is the only path that writes chunk vectors (`extract_chunk` no longer embeds inline) and it holds the same single LLM permit as every `extract_chunk`. Ranked with the extraction backlog, the gate-busy defer would round-robin it behind that whole backlog and vectors would trail extraction by the backlog's length. Each backfill step embeds one bounded batch, defers `REEMBED_BACKFILL_REVISIT_MS` (750 ms), and settles `Done` once the space is covered, so extraction is never starved in return. + `DEFAULT_LOCK_DURATION_MS = 5 * 60 * 1000` (5 min) — comfortably larger than any expected single-job runtime, so a crashed worker's row is recovered after the window without leaving real failures stuck for hours. Retry backoff is exponential: `backoff_ms(attempts)` = `min(60s * 2^(attempts-1), 1h)` (`RETRY_BASE_MS = 60s`, `RETRY_CAP_MS = 1h`), so the first retry waits 60s, then 120s, 240s, … capped at one hour. ## Worker loop diff --git a/src/memory/queue/store.rs b/src/memory/queue/store.rs index 567c2117..4ef11ac7 100644 --- a/src/memory/queue/store.rs +++ b/src/memory/queue/store.rs @@ -107,6 +107,10 @@ pub(crate) fn enqueue_conn( /// Sets `status=running`, bumps `attempts`, stamps `started_at_ms` and /// `locked_until_ms`. Returns `None` when the queue is empty / not yet due. /// +/// Due rows are ranked by kind — `seal`, then `reembed_backfill`, then +/// `flush_stale`, then `append_buffer`, then everything else — and by +/// `available_at_ms` within a rank (the query below explains why). +/// /// Retired kinds (`topic_route`, `digest_daily`) are excluded from the claim so /// a leftover old-queue row never reaches `row_to_job` (which would fail to /// parse it). [`purge_retired_jobs`] removes such rows. @@ -120,6 +124,18 @@ pub fn claim_next(config: &MemoryConfig, lock_duration_ms: i64) -> Result