Repository navigation
Fix recovery, split transaction consistency, and metadata API correctness - #10
Merged
Merged
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
A retryable transaction-drain deferral during split recovery aborts daemon startup, preventing transaction maintenance from running and leaving the persisted split stuck across restarts.
1 open issue
What changed in this PR
This PR hardens MetaDB’s storage, transaction, and service layers against recovery data loss and invalid metadata operations. It also documents upgrade requirements and current deployment limits.
Changes:
- Strengthens Raft log, snapshot, and SST validation; serializes apply, checkpoint, and read boundaries.
- Improves transaction-aware splitting, directory and quota accounting, durable ID allocation, and metadata API validation.
- Adds authenticated Raft transport, bounded metrics serving, JSON response support, and regression coverage.
| File | Description |
|---|---|
| docs/guides/en/operations-and-tuning.md | Updates CLI behavior, tuning guidance, and implementation limits. |
| docs/guides/en/deployment.md | Clarifies supported deployments and Raft authentication. |
| docs/designs/correctness_hardening.md | Documents guarantees, compatibility changes, and remaining limits. |
| crates/metadb-tx/tests/two_pc_crash_tests.rs | Adds transaction ID allocation to the test directory. |
| crates/metadb-tx/tests/two_pc_compare_tests.rs | Tests stale-read rejection and coordinator identity reuse. |
| crates/metadb-tx/tests/split_attr_apply_tests.rs | Tests accounting preservation during splits and metadata updates. |
| crates/metadb-tx/tests/round2_snapshot_reconcile_tests.rs | Tests directory totals across pinned reads and flushes. |
| crates/metadb-tx/tests/round2_atomic_validation_tests.rs | Tests atomic validation, capacity reservations, and split fencing. |
| crates/metadb-tx/tests/aux_idempotency_tests.rs | Adds transaction ID allocation to the test directory. |
| crates/metadb-tx/tests/attr_reconcile_race_tests.rs | Adds transaction ID allocation to the test directory. |
| crates/metadb-tx/src/validation.rs | Centralizes size, directory ID, and counter validation. |
| crates/metadb-tx/src/pushdown.rs | Validates mutations before changing keys or accounting. |
| crates/metadb-tx/src/lib.rs | Registers the validation module. |
| crates/metadb-tx/src/aux_sink.rs | Checks quota arithmetic and retains failed tasks for retry. |
| crates/metadb-tx/src/aux_outbox.rs | Combines command deltas and checks directory counters. |
| crates/metadb-tx/src/attr_aggregator.rs | Validates directory balances against pending deltas. |
| crates/metadb-storage/tests/round2_integrity_tests.rs | Tests SST identity and existing-checkpoint validation. |
| crates/metadb-storage/tests/lsm_recovery_tests.rs | Requires complete CRC-invalid manifest records to fail recovery. |
| crates/metadb-storage/tests/adversarial_recovery_tests.rs | Covers corruption, cache isolation, and checkpoint races. |
| crates/metadb-storage/src/sstable.rs | Validates file structure and isolates reader cache entries. |
| crates/metadb-storage/src/manifest.rs | Strengthens recovery validation and existing-image opening. |
| crates/metadb-storage/src/engine.rs | Adds existing-checkpoint opening without initialization. |
| crates/metadb-storage/src/cache.rs | Scopes cached blocks by reader identity. |
| crates/metadb-storage/src/backend.rs | Checks manifest/SST equality and protects checkpoint linking. |
| crates/metadb-server/tests/round2_restart_routes_tests.rs | Tests restart address refresh and directory ownership. |
| crates/metadb-server/tests/round2_quota_order_tests.rs | Tests quota convergence under reordered origin delivery. |
| crates/metadb-server/tests/round2_namespace_tests.rs | Tests directory identity and voter-count rejection. |
| crates/metadb-server/tests/round2_name_flush_tests.rs | Tests name limits and write-intent key persistence. |
| crates/metadb-server/tests/round2_mixed_shard_tests.rs | Tests splitting a shard containing multiple tables. |
| crates/metadb-server/tests/round2_api_tests.rs | Covers SQL, pagination, and invalid-size regressions. |
| crates/metadb-server/tests/rename_concurrency_regressions.rs | Tests reachability under concurrent directory renames. |
| crates/metadb-server/tests/input_boundary_regressions.rs | Tests identifier, name, clock, and SQL boundaries. |
| crates/metadb-server/tests/grpc_regressions.rs | Tests participant routing, authentication, and unsupported operations. |
| crates/metadb-server/src/lib.rs | Configures Raft authentication and maintenance snapshot behavior. |
| crates/metadb-server/src/grpc_services.rs | Validates bindings, routes prepares, and preserves pagination metadata. |
| crates/metadb-server/src/bin/metadbd.rs | Adds outbound credentials and bounded metrics lifecycle management. |
| crates/metadb-server/Cargo.toml | Adds HTTP, JSON, and test dependencies. |
| crates/metadb-raft/tests/multi_raft_failover_tests.rs | Exercises authenticated replication, snapshot transfer, and failover. |
| crates/metadb-raft/tests/checkpoint_atomicity_tests.rs | Tests that reads and checkpoints wait for complete commands. |
| crates/metadb-raft/src/state_machine.rs | Serializes state operations and validates snapshot images. |
| crates/metadb-raft/src/read_index.rs | Acquires read snapshots through the state-machine lock. |
| crates/metadb-raft/src/network.rs | Adds shared-secret headers to outgoing Raft RPCs. |
| crates/metadb-raft/src/log_store.rs | Validates committed history and strengthens durable file operations. |
| crates/metadb-raft/src/group_host.rs | Integrates serialized reads and shared-secret configuration. |
| crates/metadb-proxy/src/sql_templates.rs | Preserves quota UPSERT branches and checks mode conversions. |
| crates/metadb-proxy/src/router.rs | Routes quota UPSERT commands to atomic execution. |
| crates/metadb-proto/proto/metadb/v1/transaction.proto | Adds expected-value dependencies to prepare requests. |
| crates/metadb-proto/proto/metadb/v1/metadata.proto | Adds SQL pagination response fields. |
| crates/metadb-proto/Cargo.toml | Adds serialization support. |
| crates/metadb-proto/build.rs | Derives serialization for generated protocol types. |
| crates/metadb-core/src/schema.rs | Enforces persistable UTF-8 name lengths. |
| crates/metadb-core/src/lib.rs | Exports the name-length limit. |
| crates/metadb-cluster/tests/split_crash_recovery_tests.rs | Verifies exclusive allocator ownership before reopening. |
| crates/metadb-cluster/tests/round2_split_transaction_tests.rs | Tests committed transaction recovery before deferred splitting. |
| crates/metadb-cluster/tests/round2_allocator_tests.rs | Covers locking, request identity, migration, and exhaustion. |
| crates/metadb-cluster/tests/id_restart_regressions.rs | Tests ID uniqueness and allocation boundaries across restarts. |
| crates/metadb-cluster/src/split_worker.rs | Handles mixed-table boundaries and defers splits for transactions. |
| crates/metadb-cluster/src/id_allocator.rs | Adds exclusive ownership, versioned grants, and checked allocation. |
| crates/metadb-cluster/src/global_manager.rs | Allocates transaction IDs from a durable sequence. |
| crates/metadb-cluster/src/cell_manager.rs | Validates ownership and refreshes singleton peer addresses. |
| crates/metadb-cluster/Cargo.toml | Adds OpenRaft membership API access. |
| Cargo.lock | Records dependency changes. |
Review effort: Balanced
Validate complete Raft and snapshot histories, serialize checkpoint boundaries, and preserve committed transactions during shard splits. Enforce durable allocator ownership and request identity, validate mutations before apply, conserve cross-origin quota deltas, and correct SQL, CLI, and metrics behavior. Add deterministic regression and real-network tests with upgrade notes.
Preserve pending pre-fence plans while transaction recovery drains their parent shards. Complete matching committed fences despite new retained-range transactions, reject malformed transaction state, and cover both restart paths with regression tests.
Reject stale or incomplete shard descriptors and wait for published routes with bounded retries. Validate read ownership and epochs to prevent false misses and incomplete directory results, start transaction leases after routing, and cover delayed publication with concurrent reads and writes.
Verify the purged committed prefix, durability watermarks, and restart data instead of racing background log retention. Report the bound metrics address, discover an ephemeral listener with startup diagnostics, and validate concurrent scrapes across multiple idle connections.
Keep sole-voter address moves, exact quota sums that still clamp on read, fast NotLeader replies, Raft bearer auth, and the JSON operator CLI without dropping the exclusive allocator, split-range waits, or fail-closed checks. Co-authored-by: RustFS <hello@rustfs.com>
cursor
Bot
force-pushed
the
fix/audit-correctness-and-operator-tools
branch
from
November 17, 2026 07:12
61085f3 to
315a264
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
This fixes acknowledged writes disappearing during recovery, committed renames being lost during split/recovery, incomplete snapshots replacing live data, and metadata/API operations reporting success with invalid effects.
Validation
cargo fmt --all -- --check: passed.cargo clippy --workspace --all-targets -- -D warnings: passed.RUST_BACKTRACE=full cargo test --workspace --no-fail-faston macOS: 138 passed, 0 failed, 1 ignored.The regression suite includes real TCP/gRPC, authenticated three-voter snapshot/promotion/failover, deterministic split/2PC recovery and delayed route publication, corruption and malformed snapshot matrices, allocator migration/downgrade safety, input atomicity, persistent topology tests, and real CLI/daemon lifecycle tests. Maintenance checks the purged committed prefix, durability watermarks, and restart data. Metrics tests discover the bound ephemeral listener and verify concurrent scrapes, idle expiration, and shutdown.
Compatibility and scope
memtable_size_bytesflush threshold remain outside this change.See
docs/designs/correctness_hardening.mdfor upgrade behavior and remaining implementation limits.