Skip to content

Fix recovery, split transaction consistency, and metadata API correctness - #10

Merged
cursor[bot] merged 5 commits into
mainfrom
fix/audit-correctness-and-operator-tools
Nov 17, 2026
Merged

cursor[bot] merged 5 commits into
mainfrom
fix/audit-correctness-and-operator-tools

Conversation

@loverustfs

@loverustfs loverustfs commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This fixes acknowledged writes disappearing during recovery, committed renames being lost during split/recovery, incomplete snapshots replacing live data, and metadata/API operations reporting success with invalid effects.

  • Validate the complete committed Raft prefix and durable election metadata; verify the full snapshot database image before installation, and compare SST metadata against MANIFEST expectations.
  • Serialize apply/checkpoint/read-cut boundaries, preserve directory accounting, compare 2PC read dependencies, and defer split fencing until transaction state drains. Keep pre-fence plans pending across restart so transaction recovery can run; finish committed fences despite new retained-range transactions and reject malformed transaction state.
  • Enforce exclusive allocator ownership, durable transaction IDs, request-kind/count fingerprints, checked ID exhaustion, persistable name lengths, and pre-mutation size/counter validation. Quota updates preserve signed intermediate balances across independent origin streams and apply batches atomically.
  • Reject stale or incomplete published shard routes and wait through split publication. Validate read ownership and epochs to prevent false misses and partial listings; start transaction TTLs after routing is ready.
  • Correct gRPC participant routing, malformed SQL bindings, atomic quota UPSERT, numeric conversions, and cursor/prefix listing bounds; authenticate all Raft RPCs with the configured shared secret.
  • Make CLI output complete JSON, errors nonzero, and benchmark namespaces/IDs/counts/outcomes trustworthy. Preserve the embedded mixed workload entry point with complete JSONL reports and reject invalid explicit settings. Bound metrics connections and deadlines, report the actual bound listener address, and join connections during shutdown.
  • Refresh singleton peer addresses through guarded Raft membership updates, persist directory ownership, and reject unsupported replicated endpoint changes explicitly.

Validation

  • cargo fmt --all -- --check: passed.
  • cargo clippy --workspace --all-targets -- -D warnings: passed.
  • RUST_BACKTRACE=full cargo test --workspace --no-fail-fast on macOS: 138 passed, 0 failed, 1 ignored.
  • Final-head GitHub Linux CI: 138 passed, 0 failed, 1 ignored, with formatting and strict Clippy also passing. Both PR CI and push CI passed.
  • The ignored local stress profile was run explicitly for 30 seconds with 16 clients, 32 shared directories, and seed 7: 14,975 successful operations, 0 operation errors, 0 correctness errors.

The regression suite includes real TCP/gRPC, authenticated three-voter snapshot/promotion/failover, deterministic split/2PC recovery and delayed route publication, corruption and malformed snapshot matrices, allocator migration/downgrade safety, input atomicity, persistent topology tests, and real CLI/daemon lifecycle tests. Maintenance checks the purged committed prefix, durability watermarks, and restart data. Metrics tests discover the bound ephemeral listener and verify concurrent scrapes, idle expiration, and shutdown.

Compatibility and scope

  • Allocator state gains a version-2 envelope. Legacy watermarks migrate without reuse; cached legacy UUIDs with unverifiable request parameters are rejected. Old decoders reject the new envelope rather than silently reinterpret counters. UUID deduplication is bounded to 4,096 grants.
  • Cell directories persist a local owner marker. Legacy singleton ownership can be established automatically; ambiguous legacy replicated ownership requires offline migration. Replicated endpoint changes must be performed online instead of silently rebinding on restart.
  • CLI SQL output and benchmark output intentionally change to JSON/JSONL; downstream scripts need to account for the documented format. Protobuf SQL pagination fields are additive; new bincode command variants are appended.
  • Internal asynchronous quota balances can temporarily be negative under out-of-order origin delivery and converge once both streams arrive.
  • Splits currently drain all shard-local transaction state conservatively. Requests wait for a consistent published route with a 30-second routing retry budget. Snapshot serialization favors correctness over minimizing write pauses.
  • Disaggregated daemon roles, replicated multi-Cell control-plane state, physical jumbo-node migration, and an active memtable_size_bytes flush threshold remain outside this change.

See docs/designs/correctness_hardening.md for upgrade behavior and remaining implementation limits.

Copilot AI balanced review requested due to automatic review settings November 16, 2026 11:49

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A retryable transaction-drain deferral during split recovery aborts daemon startup, preventing transaction maintenance from running and leaving the persisted split stuck across restarts.

1 open issue
What changed in this PR

This PR hardens MetaDB’s storage, transaction, and service layers against recovery data loss and invalid metadata operations. It also documents upgrade requirements and current deployment limits.

Changes:

  • Strengthens Raft log, snapshot, and SST validation; serializes apply, checkpoint, and read boundaries.
  • Improves transaction-aware splitting, directory and quota accounting, durable ID allocation, and metadata API validation.
  • Adds authenticated Raft transport, bounded metrics serving, JSON response support, and regression coverage.
File Description
docs/​guides/​en/​operations-and-tuning.md Updates CLI behavior, tuning guidance, and implementation limits.
docs/​guides/​en/​deployment.md Clarifies supported deployments and Raft authentication.
docs/​designs/​correctness_hardening.md Documents guarantees, compatibility changes, and remaining limits.
crates/​metadb-tx/​tests/​two_pc_crash_tests.rs Adds transaction ID allocation to the test directory.
crates/​metadb-tx/​tests/​two_pc_compare_tests.rs Tests stale-read rejection and coordinator identity reuse.
crates/​metadb-tx/​tests/​split_attr_apply_tests.rs Tests accounting preservation during splits and metadata updates.
crates/​metadb-tx/​tests/​round2_snapshot_reconcile_tests.rs Tests directory totals across pinned reads and flushes.
crates/​metadb-tx/​tests/​round2_atomic_validation_tests.rs Tests atomic validation, capacity reservations, and split fencing.
crates/​metadb-tx/​tests/​aux_idempotency_tests.rs Adds transaction ID allocation to the test directory.
crates/​metadb-tx/​tests/​attr_reconcile_race_tests.rs Adds transaction ID allocation to the test directory.
crates/​metadb-tx/​src/​validation.rs Centralizes size, directory ID, and counter validation.
crates/​metadb-tx/​src/​pushdown.rs Validates mutations before changing keys or accounting.
crates/​metadb-tx/​src/​lib.rs Registers the validation module.
crates/​metadb-tx/​src/​aux_sink.rs Checks quota arithmetic and retains failed tasks for retry.
crates/​metadb-tx/​src/​aux_outbox.rs Combines command deltas and checks directory counters.
crates/​metadb-tx/​src/​attr_aggregator.rs Validates directory balances against pending deltas.
crates/​metadb-storage/​tests/​round2_integrity_tests.rs Tests SST identity and existing-checkpoint validation.
crates/​metadb-storage/​tests/​lsm_recovery_tests.rs Requires complete CRC-invalid manifest records to fail recovery.
crates/​metadb-storage/​tests/​adversarial_recovery_tests.rs Covers corruption, cache isolation, and checkpoint races.
crates/​metadb-storage/​src/​sstable.rs Validates file structure and isolates reader cache entries.
crates/​metadb-storage/​src/​manifest.rs Strengthens recovery validation and existing-image opening.
crates/​metadb-storage/​src/​engine.rs Adds existing-checkpoint opening without initialization.
crates/​metadb-storage/​src/​cache.rs Scopes cached blocks by reader identity.
crates/​metadb-storage/​src/​backend.rs Checks manifest/SST equality and protects checkpoint linking.
crates/​metadb-server/​tests/​round2_restart_routes_tests.rs Tests restart address refresh and directory ownership.
crates/​metadb-server/​tests/​round2_quota_order_tests.rs Tests quota convergence under reordered origin delivery.
crates/​metadb-server/​tests/​round2_namespace_tests.rs Tests directory identity and voter-count rejection.
crates/​metadb-server/​tests/​round2_name_flush_tests.rs Tests name limits and write-intent key persistence.
crates/​metadb-server/​tests/​round2_mixed_shard_tests.rs Tests splitting a shard containing multiple tables.
crates/​metadb-server/​tests/​round2_api_tests.rs Covers SQL, pagination, and invalid-size regressions.
crates/​metadb-server/​tests/​rename_concurrency_regressions.rs Tests reachability under concurrent directory renames.
crates/​metadb-server/​tests/​input_boundary_regressions.rs Tests identifier, name, clock, and SQL boundaries.
crates/​metadb-server/​tests/​grpc_regressions.rs Tests participant routing, authentication, and unsupported operations.
crates/​metadb-server/​src/​lib.rs Configures Raft authentication and maintenance snapshot behavior.
crates/​metadb-server/​src/​grpc_services.rs Validates bindings, routes prepares, and preserves pagination metadata.
crates/​metadb-server/​src/​bin/​metadbd.rs Adds outbound credentials and bounded metrics lifecycle management.
crates/​metadb-server/​Cargo.toml Adds HTTP, JSON, and test dependencies.
crates/​metadb-raft/​tests/​multi_raft_failover_tests.rs Exercises authenticated replication, snapshot transfer, and failover.
crates/​metadb-raft/​tests/​checkpoint_atomicity_tests.rs Tests that reads and checkpoints wait for complete commands.
crates/​metadb-raft/​src/​state_machine.rs Serializes state operations and validates snapshot images.
crates/​metadb-raft/​src/​read_index.rs Acquires read snapshots through the state-machine lock.
crates/​metadb-raft/​src/​network.rs Adds shared-secret headers to outgoing Raft RPCs.
crates/​metadb-raft/​src/​log_store.rs Validates committed history and strengthens durable file operations.
crates/​metadb-raft/​src/​group_host.rs Integrates serialized reads and shared-secret configuration.
crates/​metadb-proxy/​src/​sql_templates.rs Preserves quota UPSERT branches and checks mode conversions.
crates/​metadb-proxy/​src/​router.rs Routes quota UPSERT commands to atomic execution.
crates/​metadb-proto/​proto/​metadb/​v1/​transaction.proto Adds expected-value dependencies to prepare requests.
crates/​metadb-proto/​proto/​metadb/​v1/​metadata.proto Adds SQL pagination response fields.
crates/​metadb-proto/​Cargo.toml Adds serialization support.
crates/​metadb-proto/​build.rs Derives serialization for generated protocol types.
crates/​metadb-core/​src/​schema.rs Enforces persistable UTF-8 name lengths.
crates/​metadb-core/​src/​lib.rs Exports the name-length limit.
crates/​metadb-cluster/​tests/​split_crash_recovery_tests.rs Verifies exclusive allocator ownership before reopening.
crates/​metadb-cluster/​tests/​round2_split_transaction_tests.rs Tests committed transaction recovery before deferred splitting.
crates/​metadb-cluster/​tests/​round2_allocator_tests.rs Covers locking, request identity, migration, and exhaustion.
crates/​metadb-cluster/​tests/​id_restart_regressions.rs Tests ID uniqueness and allocation boundaries across restarts.
crates/​metadb-cluster/​src/​split_worker.rs Handles mixed-table boundaries and defers splits for transactions.
crates/​metadb-cluster/​src/​id_allocator.rs Adds exclusive ownership, versioned grants, and checked allocation.
crates/​metadb-cluster/​src/​global_manager.rs Allocates transaction IDs from a durable sequence.
crates/​metadb-cluster/​src/​cell_manager.rs Validates ownership and refreshes singleton peer addresses.
crates/​metadb-cluster/​Cargo.toml Adds OpenRaft membership API access.
Cargo.lock Records dependency changes.

Review effort: Balanced

loverustfs and others added 5 commits November 17, 2026 06:48
Validate complete Raft and snapshot histories, serialize checkpoint boundaries, and preserve committed transactions during shard splits.

Enforce durable allocator ownership and request identity, validate mutations before apply, conserve cross-origin quota deltas, and correct SQL, CLI, and metrics behavior. Add deterministic regression and real-network tests with upgrade notes.
Preserve pending pre-fence plans while transaction recovery drains their parent shards. Complete matching committed fences despite new retained-range transactions, reject malformed transaction state, and cover both restart paths with regression tests.
Reject stale or incomplete shard descriptors and wait for published routes with bounded retries. Validate read ownership and epochs to prevent false misses and incomplete directory results, start transaction leases after routing, and cover delayed publication with concurrent reads and writes.
Verify the purged committed prefix, durability watermarks, and restart data instead of racing background log retention. Report the bound metrics address, discover an ephemeral listener with startup diagnostics, and validate concurrent scrapes across multiple idle connections.
Keep sole-voter address moves, exact quota sums that still clamp on read,
fast NotLeader replies, Raft bearer auth, and the JSON operator CLI without
dropping the exclusive allocator, split-range waits, or fail-closed checks.

Co-authored-by: RustFS <hello@rustfs.com>
@cursor
cursor Bot force-pushed the fix/audit-correctness-and-operator-tools branch from 61085f3 to 315a264 Compare November 17, 2026 07:12
@cursor
cursor Bot merged commit f325f6a into main Nov 17, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants