Skip to content

Fail closed when a required shared secret is missing - #13

Merged
houseme merged 3 commits into
mainfrom
cursor/fail-closed-listener-auth-f050
Nov 17, 2026
Merged

houseme merged 3 commits into
mainfrom
cursor/fail-closed-listener-auth-f050

Conversation

@loverustfs

@loverustfs loverustfs commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes first-ship fail-open gaps on the listeners this process actually serves. This repository is an independent RustFS implementation. It does not claim Baidu paper / arXiv:2610.32247v1 production scale.

MetadataService, TransactionService, ControlPlaneService (GetRoutes, which metadb-ctl cluster status calls), and RaftTransportService already shared one bearer interceptor when [security].shared_secret was non-empty. That Raft check is unchanged. The gaps were around startup and the metrics listener:

  • An empty, whitespace-only, missing, or misspelled shared_secret combined with a wildcard or other non-loopback bind (grpc_listen_addr or metrics_listen_addr, including a saved advertise address) now refuses to listen instead of serving unauthenticated APIs.
  • GET /metrics requires the same Authorization: Bearer <shared_secret> header as the control and data plane when a secret is configured. A missing, empty, or wrong bearer returns HTTP 401 and a body that is not Prometheus text. The same rejection applies to other paths on that listener, including /status. The configured bearer still receives HTTP 200, the Prometheus content type, and metric text.
  • This process does not terminate TLS. A TLS, SSL, certificate, token, or password key fails configuration parsing, including keys nested in tables or arrays ([node.transport], [[listeners]]). [security].shared_secret is accepted only as a direct key of the root [security] table. The process does not start and does not serve plaintext while those keys are present.

Loopback binds (127.0.0.0/8, ::1, or the name localhost) with an empty secret still start, and log a warning. That is the existing local development path.

Tests

  • cargo test -p metadb-server --lib config::tests auth_tests — passed
  • cargo test -p metadb-server --test auth_fail_closed — passed
    • wildcard_without_a_secret_does_not_listen
    • misspelled_secret_and_tls_options_do_not_start
    • configured_secret_rejects_missing_and_wrong_credentials (cluster status / GetRoutes, metadata Lookup, Raft Vote, and metrics, including /status; the configured bearer scrape returns HTTP 200 with text/plain; version=0.0.4 and metadb_shards_total)
    • unit coverage in config::tests for [node.transport] tls_cert_file and [[listeners]] tls = true
  • cargo test -p metadb-server --test grpc_regressions configured_secret_protects_raft_transport — passed (existing Raft bearer test, not regressed)
  • cargo test -p metadb-server --test round2_cli_tests idle_metrics_connection_must_not_block_other_scrapes — passed (loopback metrics with no secret)
  • cargo test -p metadb-server --test e2e_cluster_evolution_tests — passed locally after the CI flake fix below
  • cargo clippy -p metadb-server --all-targets -- -D warnings — passed

CI on 6a8e2a8 failed only test_e2e_persistent_grpc_cluster_restart_split_2pc_and_locality_compaction: after one maintenance tick, overwrite quota was still 10240 (20 files) instead of 9728 (19 files). The test now polls quota and the secondary index for up to eight ticks. That does not change listener authentication.

Remaining gaps (not fixed)

  • Native TLS is not implemented. There is no in-process certificate stack, and listeners stay plaintext. A configured secret is still visible on the wire unless an external proxy terminates TLS. The process does not check that such a proxy exists.
  • Loopback with an empty secret remains unauthenticated. The name localhost is not resolved, so a hosts-file alias is not detected.
  • Unknown TOML keys that do not look like a secret or TLS setting are still ignored. A misspelled storage key does not fail startup.
  • One shared bearer covers metadata, transactions, the control plane, Raft, and metrics. There is no separate metrics credential, mutual TLS, per-RPC authorization, secret rotation, or audit log of rejected callers.
  • No new status API was added. Cluster status is the existing GetRoutes RPC. A heartbeat-driven control loop is still not implemented.
  • This change does not depend on the open attribution-only docs pull request.
Open in Web Open in Cursor 

Refuse wildcard and other non-loopback binds when the shared secret is
empty, reject misspelled secret and TLS config keys instead of serving
plaintext, and require the same bearer on metrics as on gRPC.

Co-authored-by: RustFS <hello@rustfs.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 14:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Nested TLS settings bypass validation and can allow plaintext startup on non-loopback binds, while successful metrics requests with the configured bearer token lack integration coverage.

2 open issues
What changed in this PR

This PR hardens the server’s gRPC and metrics listeners against missing shared secrets while preserving unauthenticated loopback development. Native TLS remains unsupported.

Changes:

  • Rejects non-loopback listeners without a usable secret and detects unsupported security configuration keys.
  • Moves metrics serving into the cluster lifecycle and adds shared-bearer authentication.
  • Adds rejection-focused regression tests and updates security documentation across five languages.
File Description
docs/​zh/​operations.md Updates metrics authentication and security limits.
docs/​zh/​configuration.md Documents configuration rejection and loopback rules.
docs/​zh/​build-and-run.md Updates startup security guidance.
docs/​zh/​architecture.md Describes listener validation and metrics ownership.
docs/​ja/​operations.md Updates metrics authentication and security limits.
docs/​ja/​configuration.md Documents configuration rejection and loopback rules.
docs/​ja/​build-and-run.md Updates startup security guidance.
docs/​ja/​architecture.md Describes listener validation and metrics ownership.
docs/​guides/​zh/​deployment.md Updates deployment authentication requirements.
docs/​guides/​ja/​deployment.md Updates deployment authentication requirements.
docs/​guides/​fr/​deployment.md Updates deployment authentication requirements.
docs/​guides/​en/​deployment.md Clarifies network exposure and metrics authentication.
docs/​guides/​de/​deployment.md Updates deployment authentication requirements.
docs/​fr/​operations.md Updates metrics authentication and security limits.
docs/​fr/​configuration.md Documents configuration rejection and loopback rules.
docs/​fr/​build-and-run.md Updates startup security guidance.
docs/​fr/​architecture.md Describes listener validation and metrics ownership.
docs/​en/​operations.md Updates metrics authentication and security limits.
docs/​en/​configuration.md Documents configuration rejection and loopback rules.
docs/​en/​build-and-run.md Updates startup security guidance.
docs/​en/​architecture.md Describes listener validation and metrics ownership.
docs/​de/​operations.md Updates metrics authentication and security limits.
docs/​de/​configuration.md Documents configuration rejection and loopback rules.
docs/​de/​build-and-run.md Updates startup security guidance.
docs/​de/​architecture.md Describes listener validation and metrics ownership.
crates/​metadb-server/​tests/​auth_fail_closed.rs Adds startup and credential rejection tests.
crates/​metadb-server/​src/​metrics_http.rs Extracts metrics serving and adds bearer authentication.
crates/​metadb-server/​src/​lib.rs Validates binds and manages the metrics listener lifecycle.
crates/​metadb-server/​src/​config.rs Adds bind validation and security-key screening.
crates/​metadb-server/​src/​bin/​metadbd.rs Delegates metrics serving and shutdown to the cluster.

Review effort: Balanced

Comment on lines +281 to +301
for (key, child) in root {
if !KNOWN_SECTIONS.contains(&key.as_str()) && (is_tls_key(key) || is_secret_key(key)) {
return Err(security_key_error(key));
}
let Some(table) = child.as_table() else {
continue;
};
if key == "security" {
for sub in table.keys() {
if sub != "shared_secret" {
return Err(security_key_error(sub));
}
}
continue;
}
for sub in table.keys() {
if is_tls_key(sub) || is_secret_key(sub) {
return Err(security_key_error(sub));
}
}
}
Comment on lines +108 to +109
if !bearer_header_matches(presented, shared_secret) {
return (StatusCode::UNAUTHORIZED, "unauthorized\n".to_owned(), false);
cursoragent and others added 2 commits November 17, 2026 14:55
Walk tables and arrays so a TLS key under a nested table or array of
tables cannot be ignored. Allow shared_secret only as a direct key of
the root security table. Assert that the configured bearer still
receives Prometheus text.

Co-authored-by: RustFS <hello@rustfs.com>
The e2e evolution test failed in CI with used_bytes still at 20 files
after one maintenance tick. Poll the quota and secondary index for a
few ticks so a missed aux propose does not fail the run.

Co-authored-by: RustFS <hello@rustfs.com>
@houseme
houseme merged commit 3c1e80f into main Nov 17, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants