fix(queue): rank reembed_backfill right after seal in claim_next

[YellowSnnowmann]

Summary

• claim_next now ranks reembed_backfill second, right after seal: seal > reembed_backfill > flush_stale > append_buffer > everything else, then available_at_ms. The deduped backfill row is the only writer of chunk vectors since extract_chunk stopped embedding inline, and it shares the single-permit LLM gate with every extract_chunk. In the ELSE bucket, the 50 ms gate-busy defer round-robined it behind the whole extraction backlog, so vectors arrived in bursts 7 to 12 minutes apart in the field (Sources row warns "Stored without vectors" for a transient embedding backlog after a large sync openhuman#6025).
• Stays LLM-gated (is_llm_bound unchanged). Each step still embeds one bounded batch, defers REEMBED_BACKFILL_REVISIT_MS (750 ms) and settles Done once the space is covered, so extraction is not starved in return. LLM_GATE_RETRY_MS is untouched (the issue lists it as optional and separate).
• Tests: claim_next_prefers_reembed_backfill_over_older_extract_chunk (the issue's case) and claim_next_ranks_seal_then_backfill_then_flush_then_append_then_age (pins the whole ladder with explicit available_at_ms, so rank, not age, decides).
• Docs: gitbooks/job-queue.md "Claiming and ordering" updated to match the query.

Notes for reviewers

• The rank only decides which due row a worker claims. With the 50 ms gate-busy defer, the backfill row is due for about one claim cycle out of every ~55 ms, so at the instant a permit frees it wins with roughly 15 to 20 % probability per release instead of about 1/N (N = ready extract_chunk rows). Expected lag from chunk write to vector drops from one full rotation of the backlog to a handful of extract completions, i.e. tens of seconds. A fully deterministic hand-off would need a kind-specific gate-busy defer for the backfill; a 0 ms defer at rank 2 would hot-spin the row above flush_stale / append_buffer and starve them while the gate is busy, so that is deliberately left out.
• Downstream: tinymemory bumps vendor/tinycortex and cuts a patch release, then openhuman re-pins the module.

Test plan

• cargo fmt --all -- --check
• cargo clippy --all-targets -- -D warnings
• cargo test (default features)
• cargo test --all-targets --no-default-features --features tokio
• cargo build --all-targets --all-features and cargo test --all-features
• cargo clippy --all-targets --all-features -- -D warnings
• RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps

Closes #168

Summary by CodeRabbit

• Improvements

  • Updated background job scheduling so vector backfill tasks are processed ahead of extraction backlogs.
  • Preserved the priority order for sealing tasks, followed by vector backfill, stale-data cleanup, and buffer appends.
  • Backfill processing continues in bounded batches and completes once all applicable content is covered.

• Documentation

  • Clarified job-priority behavior and the rationale for prioritizing vector backfill tasks.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: b6de3460-8e03-4388-92b4-ba273bcc04e8

📥 Commits

Reviewing files that changed from the base of the PR and between 8dab3d2 and d9851a0.

📒 Files selected for processing (3)

• gitbooks/job-queue.md
• src/memory/queue/store.rs
• src/memory/queue/store_tests.rs

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

---

📝 Walkthrough

Walkthrough

The queue now claims reembed_backfill jobs after seal and before flush_stale, append_buffer, and fallback jobs. Documentation explains the ranking. Tests verify priority ordering and age ordering within ranks.

Changes

Queue priority

Layer / File(s)	Summary
Update claim ordering src/memory/queue/store.rs, gitbooks/job-queue.md	claim_next now ranks seal, reembed_backfill, flush_stale, append_buffer, then other jobs. Comments and documentation describe the ranking.
Validate claim ordering src/memory/queue/store_tests.rs	Tests verify that backfill precedes an older extraction job and that the full ranking is applied before age ordering within the fallback rank.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to d9851

This change prioritizes re-embedding backfills ahead of extraction backlog work while retaining the existing queue behavior. The documented ordering and tests align with the intended behavior, with no current merge-blocking risk identified.

Suggested reviewers: senamakel

Poem

A rabbit queues a batch with care
Backfill hops ahead through stale air
Seal leads on, then vectors flow
Older jobs wait in ordered rows
The queue now thumps a steadier beat 🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes satisfy issue #168 by prioritizing reembed_backfill before older extract_chunk jobs, preserving the existing LLM gate and revisit behavior, adding ordering tests, and updating documentatio…
Out of Scope Changes check	✅ Passed	The changes are limited to the requested queue ordering, related tests, and documentation. No unrelated code changes are present.
Docstring Coverage	✅ Passed	Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 2 files. (1 skipped: 1 u…
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: moving reembed_backfill immediately after seal in claim_next job ranking.

---

_{Comment @coderabbitai help to get the list of available commands.}

[YellowSnnowmann]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[tinysweeper]

How this change flows

1 changed behaviour across 12 relationships. 6 surrounding behaviours are shown (60 graph nodes walked). 41 further behaviours left out to keep the diagram readable.

flowchart LR
  n0["claim_next<br/>changed"]:::changed
  n1["enqueue"]:::impacted
  n2["test_config"]:::impacted
  n3["extract_job"]:::impacted
  n4["get_job"]:::impacted
  n5["mark_failed"]:::impacted
  n6["mixed_outcomes_stress"]:::impacted
  n6 -->|calls| n0
  n6 -->|tests| n0
  n6 -->|calls| n1
  n6 -->|tests| n1
  n6 -->|calls| n2
  n6 -->|tests| n2
  n6 -->|calls| n3
  n6 -->|tests| n3
  n6 -->|calls| n4
  n6 -->|tests| n4
  n6 -->|calls| n5
  n6 -->|tests| n5
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff

Loading

_{Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge.}

[tinysweeper]

tinysweeper found nothing blocking. Approving.

             $0.0075 · 81,734 in / 1,145 out · 0 cached (0%) · deepseek/deepseek-v4-flash, openrouter/openai/text-embedding-3-small · 412 embedded
critique:    $0.0033 · 36,281 in / 288 out   · 0 cached (0%) · deepseek/deepseek-v4-flash
security:    $0.0023 · 24,607 in / 466 out   · 0 cached (0%) · deepseek/deepseek-v4-flash
tests:       $0.0012 · 13,900 in / 95 out    · 0 cached (0%) · deepseek/deepseek-v4-flash
description: $0.0005 · 5,926 in  / 105 out   · 0 cached (0%) · deepseek/deepseek-v4-flash