Skip to content

Fix listdir limit pushdown and first-ship LSM flush bugs - #14

Open
loverustfs wants to merge 3 commits into
mainfrom
cursor/storage-listdir-lsm-fixes-ded3
Open

loverustfs wants to merge 3 commits into
mainfrom
cursor/storage-listdir-lsm-fixes-ded3

Conversation

@loverustfs

@loverustfs loverustfs commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes four first-ship storage and listdir bugs on main (f325f6a). This is the independent RustFS implementation; it does not claim paper-scale or trillion-record production numbers.

  • listdir limit is applied inside the snapshot scan. The scan keeps one forward merge cursor and opens the next non-overlapping SSTable only after the current file is exhausted. listdir_paginated, PushdownExecutor::listdir, and MetaDbEngine::listdir use that pinned scan instead of an unpinned full scan_range followed by truncation.
  • LocalityCompactor bucket counts are rebuilt from the latest SST key of each bucket in open_lsm and open_lsm_existing, so a restart no longer reports empty density.
  • compact_range merges participating SSTables with a streaming cursor and writes output in the existing size chunks. It no longer calls read_all_entries() into one BTreeMap. Reheap work follows only the cursors that advanced.
  • storage.memtable_size_bytes is passed through to_lsm_config() and seals the active memtable when the threshold is crossed. On the Raft apply path that seal waits until the whole command, including its side effects, has been written. storage.max_imm_memtables (default 4) blocks the next seal until an immutable table flushes. Acked outbox keys are deleted after the outbox lock is released, so that seal cannot deadlock on the spill hook.

Tests

  • cargo test -p metadb-storage --tests
  • cargo test -p metadb-tx --tests
  • cargo test -p metadb-raft --tests
  • cargo test -p metadb-server --test integration_tests --test e2e_cluster_evolution_tests
  • cargo clippy -p metadb-storage -p metadb-tx -p metadb-raft -p metadb-server --all-targets -- -D warnings

Earlier in this branch, the cluster, proxy, and remaining server suites also passed. New regressions in crates/metadb-storage/tests/first_ship_storage_regressions.rs and PushdownExecutor::listdir_limited:

  • limit 1 over 1 live row plus 2048 later tombstones examines one row, counts zero tombstones, and does not touch every data block
  • the same limit against two non-overlapping L1 files opens only the first file
  • 20 tombstones / 80 live keys are still 20/80 after open_lsm_existing
  • compact_range does not increment the read_all_entries probe and still keeps live keys
  • a 1-byte memtable threshold flushes, and a fifth immutable table waits while a snapshot pins the first four
  • two keys in one deferred command stay uncheckpointed until end_command(true), and a failed command does not advance the checkpoint

Remaining gaps

  • Density is rebuilt by streaming SST keys on open. It is not a compact manifest summary, so open cost follows the live SST set.
  • Overlapping L0 files are still opened together, because any of them can hold the next key. list_quotas and scan_secondary_index still collect a range and then truncate.
  • Compaction still holds one output chunk, bounded by the existing 64 MiB SSTable target. It no longer holds every input key in one map.
  • A writer that holds the snapshot pinning the oldest immutable memtable waits until that snapshot is dropped or the MVCC retention window expires.
  • A failed Raft command does not flush, but its memtable writes remain until a later successful flush. Commands are still expected to reject invalid input before writing.
  • English docs now describe the memtable settings as active. The de/ja/zh/fr copies still say memtable_size_bytes is unused. Citation-only files from the open attribution PR were left untouched.
  • Jumbo membership, multi-cell operation, and backup/restore are not part of this change.
Open in Web Open in Cursor 

Copilot AI balanced review requested due to automatic review settings November 17, 2026 15:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Size-triggered flushing can checkpoint incomplete Raft commands and permanently lose data on restart, while threshold-crossing outbox deletions can deadlock Raft apply.

4 open issues
What changed in this PR

This PR updates directory listing and LSM storage behavior by pushing limits into snapshot scans, restoring compaction density after restart, and activating configurable memtable flushing and backpressure.

Changes:

  • Route directory listings through pinned scans that stop after the requested live entries.
  • Stream compaction inputs and rebuild tombstone density when opening storage.
  • Wire memtable limits into storage and Raft, with regression tests and updated English documentation.
File Description
docs/​guides/​en/​operations-and-tuning.md Documents active memtable budgets and backpressure.
docs/​guides/​en/​deployment.md Updates example memtable settings.
docs/​en/​operations.md Clarifies runtime effects of storage settings.
docs/​en/​configuration.md Documents flush thresholds and immutable-table caps.
docs/​en/​architecture.md Describes size-triggered flushing and writer waits.
docs/​designs/​correctness_hardening.md Updates storage limitations.
crates/​metadb-tx/​src/​two_pc.rs Uses early-stopping snapshot scans for paginated listings.
crates/​metadb-tx/​src/​pushdown.rs Adds limited directory scans and a regression test.
crates/​metadb-storage/​tests/​first_ship_storage_regressions.rs Tests scan limits, restart density, compaction, and backpressure.
crates/​metadb-storage/​src/​version_set.rs Adds a streaming SSTable merge cursor.
crates/​metadb-storage/​src/​sstable.rs Adds forward cursors and scan probes.
crates/​metadb-storage/​src/​mvcc.rs Tracks active bytes and notifies blocked writers.
crates/​metadb-storage/​src/​lib.rs Exports scan control and flush identity types.
crates/​metadb-storage/​src/​engine.rs Implements limited scans, density restoration, and flush maintenance.
crates/​metadb-storage/​src/​compaction.rs Supports restoring bucket density counters.
crates/​metadb-storage/​src/​block.rs Adds incremental entry decoding helpers.
crates/​metadb-storage/​src/​backend.rs Streams compaction inputs into output chunks.
crates/​metadb-server/​src/​lib.rs Pushes SQL directory filters and limits into storage scans.
crates/​metadb-server/​src/​config.rs Passes memtable settings into LSM configuration.
crates/​metadb-raft/​src/​state_machine.rs Attaches automatic-flush hooks and refreshes checkpoint metadata.

Review effort: Balanced

Comment thread crates/metadb-storage/src/engine.rs Outdated
raft_index: LogSequenceNumber,
) {
self.mvcc.delete(key, hlc, raft_index);
self.maintain_memtables().expect("memtable size flush");
Comment on lines +657 to +661
let (lsn, hlc, _, _) = self.mvcc.latest_watermarks();
let checkpoint = self.checkpoint_for_flush(lsn, hlc);
let hook = self.flush_hook.read().unwrap().clone();
let hook_ref = hook.as_ref().map(|installed| installed.as_ref());
self.seal_and_flush_locked(checkpoint, hook_ref)?;
Comment thread crates/metadb-storage/src/engine.rs Outdated
Comment on lines +465 to +468
let (page, _) =
self.backend
.scan_in_version_page(&pinned.version, disk_from, end_key, 1)?;
page.into_iter().next()
Comment on lines +391 to +395
fn reheap_pending(&mut self) -> Result<()> {
for cursor_id in 0..self.sources.len() {
if !self.sources[cursor_id].needs_reheap {
continue;
}
cursoragent and others added 3 commits November 17, 2026 15:21
Stop directory scans once the requested live rows are found, rebuild locality density from SSTables on open, stream compaction instead of loading each SST into one map, and honor memtable_size_bytes plus the immutable-table cap.

Co-authored-by: RustFS <hello@rustfs.com>
Defer size-triggered seals until the command and its side effects finish, delete acked outbox keys after releasing the outbox lock, and keep a forward SST cursor that opens later non-overlapping files only when the current file ends.

Co-authored-by: RustFS <hello@rustfs.com>
Main replaced bincode with wincode. Size-triggered membership bytes now use the same codec::serialize_serde path as explicit flushes, so locked release builds compile.

Co-authored-by: RustFS <hello@rustfs.com>
@cursor
cursor Bot force-pushed the cursor/storage-listdir-lsm-fixes-ded3 branch from 2809945 to 24b8c43 Compare November 17, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants