Repository navigation
Fix listdir limit pushdown and first-ship LSM flush bugs - #14
Open
loverustfs wants to merge 3 commits into
Open
loverustfs wants to merge 3 commits into
loverustfs wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Size-triggered flushing can checkpoint incomplete Raft commands and permanently lose data on restart, while threshold-crossing outbox deletions can deadlock Raft apply.
4 open issues
What changed in this PR
This PR updates directory listing and LSM storage behavior by pushing limits into snapshot scans, restoring compaction density after restart, and activating configurable memtable flushing and backpressure.
Changes:
- Route directory listings through pinned scans that stop after the requested live entries.
- Stream compaction inputs and rebuild tombstone density when opening storage.
- Wire memtable limits into storage and Raft, with regression tests and updated English documentation.
| File | Description |
|---|---|
docs/guides/en/operations-and-tuning.md |
Documents active memtable budgets and backpressure. |
docs/guides/en/deployment.md |
Updates example memtable settings. |
docs/en/operations.md |
Clarifies runtime effects of storage settings. |
docs/en/configuration.md |
Documents flush thresholds and immutable-table caps. |
docs/en/architecture.md |
Describes size-triggered flushing and writer waits. |
docs/designs/correctness_hardening.md |
Updates storage limitations. |
crates/metadb-tx/src/two_pc.rs |
Uses early-stopping snapshot scans for paginated listings. |
crates/metadb-tx/src/pushdown.rs |
Adds limited directory scans and a regression test. |
crates/metadb-storage/tests/first_ship_storage_regressions.rs |
Tests scan limits, restart density, compaction, and backpressure. |
crates/metadb-storage/src/version_set.rs |
Adds a streaming SSTable merge cursor. |
crates/metadb-storage/src/sstable.rs |
Adds forward cursors and scan probes. |
crates/metadb-storage/src/mvcc.rs |
Tracks active bytes and notifies blocked writers. |
crates/metadb-storage/src/lib.rs |
Exports scan control and flush identity types. |
crates/metadb-storage/src/engine.rs |
Implements limited scans, density restoration, and flush maintenance. |
crates/metadb-storage/src/compaction.rs |
Supports restoring bucket density counters. |
crates/metadb-storage/src/block.rs |
Adds incremental entry decoding helpers. |
crates/metadb-storage/src/backend.rs |
Streams compaction inputs into output chunks. |
crates/metadb-server/src/lib.rs |
Pushes SQL directory filters and limits into storage scans. |
crates/metadb-server/src/config.rs |
Passes memtable settings into LSM configuration. |
crates/metadb-raft/src/state_machine.rs |
Attaches automatic-flush hooks and refreshes checkpoint metadata. |
Review effort: Balanced
| raft_index: LogSequenceNumber, | ||
| ) { | ||
| self.mvcc.delete(key, hlc, raft_index); | ||
| self.maintain_memtables().expect("memtable size flush"); |
Comment on lines
+657
to
+661
| let (lsn, hlc, _, _) = self.mvcc.latest_watermarks(); | ||
| let checkpoint = self.checkpoint_for_flush(lsn, hlc); | ||
| let hook = self.flush_hook.read().unwrap().clone(); | ||
| let hook_ref = hook.as_ref().map(|installed| installed.as_ref()); | ||
| self.seal_and_flush_locked(checkpoint, hook_ref)?; |
Comment on lines
+465
to
+468
| let (page, _) = | ||
| self.backend | ||
| .scan_in_version_page(&pinned.version, disk_from, end_key, 1)?; | ||
| page.into_iter().next() |
Comment on lines
+391
to
+395
| fn reheap_pending(&mut self) -> Result<()> { | ||
| for cursor_id in 0..self.sources.len() { | ||
| if !self.sources[cursor_id].needs_reheap { | ||
| continue; | ||
| } |
Stop directory scans once the requested live rows are found, rebuild locality density from SSTables on open, stream compaction instead of loading each SST into one map, and honor memtable_size_bytes plus the immutable-table cap. Co-authored-by: RustFS <hello@rustfs.com>
Defer size-triggered seals until the command and its side effects finish, delete acked outbox keys after releasing the outbox lock, and keep a forward SST cursor that opens later non-overlapping files only when the current file ends. Co-authored-by: RustFS <hello@rustfs.com>
Main replaced bincode with wincode. Size-triggered membership bytes now use the same codec::serialize_serde path as explicit flushes, so locked release builds compile. Co-authored-by: RustFS <hello@rustfs.com>
cursor
Bot
force-pushed
the
cursor/storage-listdir-lsm-fixes-ded3
branch
from
November 17, 2026 15:24
2809945 to
24b8c43
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
Fixes four first-ship storage and listdir bugs on
main(f325f6a). This is the independent RustFS implementation; it does not claim paper-scale or trillion-record production numbers.listdirlimit is applied inside the snapshot scan. The scan keeps one forward merge cursor and opens the next non-overlapping SSTable only after the current file is exhausted.listdir_paginated,PushdownExecutor::listdir, andMetaDbEngine::listdiruse that pinned scan instead of an unpinned fullscan_rangefollowed by truncation.LocalityCompactorbucket counts are rebuilt from the latest SST key of each bucket inopen_lsmandopen_lsm_existing, so a restart no longer reports empty density.compact_rangemerges participating SSTables with a streaming cursor and writes output in the existing size chunks. It no longer callsread_all_entries()into oneBTreeMap. Reheap work follows only the cursors that advanced.storage.memtable_size_bytesis passed throughto_lsm_config()and seals the active memtable when the threshold is crossed. On the Raft apply path that seal waits until the whole command, including its side effects, has been written.storage.max_imm_memtables(default 4) blocks the next seal until an immutable table flushes. Acked outbox keys are deleted after the outbox lock is released, so that seal cannot deadlock on the spill hook.Tests
cargo test -p metadb-storage --testscargo test -p metadb-tx --testscargo test -p metadb-raft --testscargo test -p metadb-server --test integration_tests --test e2e_cluster_evolution_testscargo clippy -p metadb-storage -p metadb-tx -p metadb-raft -p metadb-server --all-targets -- -D warningsEarlier in this branch, the cluster, proxy, and remaining server suites also passed. New regressions in
crates/metadb-storage/tests/first_ship_storage_regressions.rsandPushdownExecutor::listdir_limited:open_lsm_existingcompact_rangedoes not increment theread_all_entriesprobe and still keeps live keysend_command(true), and a failed command does not advance the checkpointRemaining gaps
list_quotasandscan_secondary_indexstill collect a range and then truncate.memtable_size_bytesis unused. Citation-only files from the open attribution PR were left untouched.