Repository navigation
Fail closed when a required shared secret is missing - #13
Merged
Merged
Conversation
Refuse wildcard and other non-loopback binds when the shared secret is empty, reject misspelled secret and TLS config keys instead of serving plaintext, and require the same bearer on metrics as on gRPC. Co-authored-by: RustFS <hello@rustfs.com>
There was a problem hiding this comment.
🟡 Changes recommended
Nested TLS settings bypass validation and can allow plaintext startup on non-loopback binds, while successful metrics requests with the configured bearer token lack integration coverage.
2 open issues
What changed in this PR
This PR hardens the server’s gRPC and metrics listeners against missing shared secrets while preserving unauthenticated loopback development. Native TLS remains unsupported.
Changes:
- Rejects non-loopback listeners without a usable secret and detects unsupported security configuration keys.
- Moves metrics serving into the cluster lifecycle and adds shared-bearer authentication.
- Adds rejection-focused regression tests and updates security documentation across five languages.
| File | Description |
|---|---|
| docs/zh/operations.md | Updates metrics authentication and security limits. |
| docs/zh/configuration.md | Documents configuration rejection and loopback rules. |
| docs/zh/build-and-run.md | Updates startup security guidance. |
| docs/zh/architecture.md | Describes listener validation and metrics ownership. |
| docs/ja/operations.md | Updates metrics authentication and security limits. |
| docs/ja/configuration.md | Documents configuration rejection and loopback rules. |
| docs/ja/build-and-run.md | Updates startup security guidance. |
| docs/ja/architecture.md | Describes listener validation and metrics ownership. |
| docs/guides/zh/deployment.md | Updates deployment authentication requirements. |
| docs/guides/ja/deployment.md | Updates deployment authentication requirements. |
| docs/guides/fr/deployment.md | Updates deployment authentication requirements. |
| docs/guides/en/deployment.md | Clarifies network exposure and metrics authentication. |
| docs/guides/de/deployment.md | Updates deployment authentication requirements. |
| docs/fr/operations.md | Updates metrics authentication and security limits. |
| docs/fr/configuration.md | Documents configuration rejection and loopback rules. |
| docs/fr/build-and-run.md | Updates startup security guidance. |
| docs/fr/architecture.md | Describes listener validation and metrics ownership. |
| docs/en/operations.md | Updates metrics authentication and security limits. |
| docs/en/configuration.md | Documents configuration rejection and loopback rules. |
| docs/en/build-and-run.md | Updates startup security guidance. |
| docs/en/architecture.md | Describes listener validation and metrics ownership. |
| docs/de/operations.md | Updates metrics authentication and security limits. |
| docs/de/configuration.md | Documents configuration rejection and loopback rules. |
| docs/de/build-and-run.md | Updates startup security guidance. |
| docs/de/architecture.md | Describes listener validation and metrics ownership. |
| crates/metadb-server/tests/auth_fail_closed.rs | Adds startup and credential rejection tests. |
| crates/metadb-server/src/metrics_http.rs | Extracts metrics serving and adds bearer authentication. |
| crates/metadb-server/src/lib.rs | Validates binds and manages the metrics listener lifecycle. |
| crates/metadb-server/src/config.rs | Adds bind validation and security-key screening. |
| crates/metadb-server/src/bin/metadbd.rs | Delegates metrics serving and shutdown to the cluster. |
Review effort: Balanced
Comment on lines
+281
to
+301
| for (key, child) in root { | ||
| if !KNOWN_SECTIONS.contains(&key.as_str()) && (is_tls_key(key) || is_secret_key(key)) { | ||
| return Err(security_key_error(key)); | ||
| } | ||
| let Some(table) = child.as_table() else { | ||
| continue; | ||
| }; | ||
| if key == "security" { | ||
| for sub in table.keys() { | ||
| if sub != "shared_secret" { | ||
| return Err(security_key_error(sub)); | ||
| } | ||
| } | ||
| continue; | ||
| } | ||
| for sub in table.keys() { | ||
| if is_tls_key(sub) || is_secret_key(sub) { | ||
| return Err(security_key_error(sub)); | ||
| } | ||
| } | ||
| } |
Comment on lines
+108
to
+109
| if !bearer_header_matches(presented, shared_secret) { | ||
| return (StatusCode::UNAUTHORIZED, "unauthorized\n".to_owned(), false); |
Walk tables and arrays so a TLS key under a nested table or array of tables cannot be ignored. Allow shared_secret only as a direct key of the root security table. Assert that the configured bearer still receives Prometheus text. Co-authored-by: RustFS <hello@rustfs.com>
The e2e evolution test failed in CI with used_bytes still at 20 files after one maintenance tick. Poll the quota and secondary index for a few ticks so a missed aux propose does not fail the run. Co-authored-by: RustFS <hello@rustfs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
Closes first-ship fail-open gaps on the listeners this process actually serves. This repository is an independent RustFS implementation. It does not claim Baidu paper / arXiv:2610.32247v1 production scale.
MetadataService,TransactionService,ControlPlaneService(GetRoutes, whichmetadb-ctl cluster statuscalls), andRaftTransportServicealready shared one bearer interceptor when[security].shared_secretwas non-empty. That Raft check is unchanged. The gaps were around startup and the metrics listener:shared_secretcombined with a wildcard or other non-loopback bind (grpc_listen_addrormetrics_listen_addr, including a saved advertise address) now refuses to listen instead of serving unauthenticated APIs.GET /metricsrequires the sameAuthorization: Bearer <shared_secret>header as the control and data plane when a secret is configured. A missing, empty, or wrong bearer returns HTTP 401 and a body that is not Prometheus text. The same rejection applies to other paths on that listener, including/status. The configured bearer still receives HTTP 200, the Prometheus content type, and metric text.[node.transport],[[listeners]]).[security].shared_secretis accepted only as a direct key of the root[security]table. The process does not start and does not serve plaintext while those keys are present.Loopback binds (
127.0.0.0/8,::1, or the namelocalhost) with an empty secret still start, and log a warning. That is the existing local development path.Tests
cargo test -p metadb-server --lib config::tests auth_tests— passedcargo test -p metadb-server --test auth_fail_closed— passedwildcard_without_a_secret_does_not_listenmisspelled_secret_and_tls_options_do_not_startconfigured_secret_rejects_missing_and_wrong_credentials(cluster status /GetRoutes, metadataLookup, RaftVote, and metrics, including/status; the configured bearer scrape returns HTTP 200 withtext/plain; version=0.0.4andmetadb_shards_total)config::testsfor[node.transport] tls_cert_fileand[[listeners]] tls = truecargo test -p metadb-server --test grpc_regressions configured_secret_protects_raft_transport— passed (existing Raft bearer test, not regressed)cargo test -p metadb-server --test round2_cli_tests idle_metrics_connection_must_not_block_other_scrapes— passed (loopback metrics with no secret)cargo test -p metadb-server --test e2e_cluster_evolution_tests— passed locally after the CI flake fix belowcargo clippy -p metadb-server --all-targets -- -D warnings— passedCI on
6a8e2a8failed onlytest_e2e_persistent_grpc_cluster_restart_split_2pc_and_locality_compaction: after one maintenance tick, overwrite quota was still10240(20 files) instead of9728(19 files). The test now polls quota and the secondary index for up to eight ticks. That does not change listener authentication.Remaining gaps (not fixed)
localhostis not resolved, so a hosts-file alias is not detected.GetRoutesRPC. A heartbeat-driven control loop is still not implemented.