Repository navigation
pg_walserver: standalone replication-protocol server (WAL/basebackup serve + push/pull archiving) - #1193
Open
dimitri wants to merge 130 commits into
Open
pg_walserver: standalone replication-protocol server (WAL/basebackup serve + push/pull archiving)#1193dimitri wants to merge 130 commits into
dimitri wants to merge 130 commits into
Conversation
dimitri
force-pushed
the
pg_walsender-server
branch
2 times, most recently
from
September 26, 2026 19:03
41d702d to
70c09b1
Compare
dimitri
force-pushed
the
pg_walsender-server
branch
from
September 27, 2026 02:15
70c09b1 to
ea06c7b
Compare
dimitri
added a commit
that referenced
this pull request
Sep 27, 2026
…, rewrite restore_command.c to call fetch_client in-process The archiving-integration-base merge just landed brought in pg_walsender's review-round rewrite (flex/bison grammar, HBA tokenizer, CommandLine-based CLI, common/fetch_client.c), which along the way dropped the "monitor" HBA keyword and the refresher/monitor_hosts machinery behind it: PR #1193 has no monitor extension of its own to query, so that feature was deliberately lifted out of it to be reintroduced here, where the SQL (get_group_hosts/ get_group_hosts_hash) actually exists. This commit does that: - hba.c/hba.h: restore the "monitor" ADDRESS keyword and hba_lookup()'s routePath/monitorUriPath/refreshSockPath parameters, on top of the new tokenizer (ported from the pre-merge tree, not re-derived). - monitor_hosts.c/.h, refresher.c/.h: restored from their last reviewed state (the refresher already uses the project's PGSQL wrapper and a retry policy, per an earlier review round, before being deleted along with the rest of the monitor feature). - auth.c/.h: WsAuthConfig grows monitorUriPath/refreshSockPath back; hba_lookup() is called with the route's path so "monitor" can match. - accept_loop.c/.h: the parent forks/supervises/restarts the refresher child and creates its datagram socket, mirroring the postmaster/ refresher split documented in monitor_hosts.h. - cli_root.c: `serve` derives archiver-monitor.uri/archiver-refresh.sock from --pgdata, same as the other archiver-* paths. - defaults.h, ws_util.c/.h: PG_AUTOCTL_ARCHIVER_NODES_FILE and ws_auth_deadline_remaining_ms() restored (both were only removed because the feature using them was gone). - Makefile: build monitor_hosts.c/refresher.c again. Also, per review: `pg_autoctl restore command` (restore_command.c) used to execv() into `pg_walsender fetch-file`, which no longer exists (pg_walsender is a server binary, plus scram-secret, nothing else -- the FETCH_FILE client moved to src/bin/common/fetch_client.c in the pg_walsender-server PR). restore_command_run() now calls ws_fetch_file_client() directly, in-process; docs (archiving-details.rst, pg_autoctl_restore_command.rst) and the leftover stale src/bin/pg_walsender/fetch_client.c/.h copy (dead, unreferenced by the Makefile since the move) are updated/removed to match. Four pre-existing archiver test specs drove the raw `pg_walsender fetch-file` CLI directly (archiver_walsender_auth, archiver_lifecycle, archiver_walsender_protocol) or only had a stale comment about it (archiver_warm_standby); updated to go through `pg_autoctl restore command` instead, matching what a real restore_command line actually runs. Verified: PG17 docker build clean (zero warnings), monitor installcheck 20/6 tests green, citus_indent + banned.h.sh clean, and every archiver-* spec touched by this change re-run green: archiver_walsender_auth (all 8 steps, including the "monitor" keyword and node2's registration/drop follow-along), archiver_lifecycle, archiver_walsender_protocol, archiver_warm_standby, archiver_fsm_quorum, archiver_fsm_maintenance, archiver_archive_command, archiver_wal_segsize, and pg_walsender_standalone (#1193's own suite, unaffected).
dimitri
force-pushed
the
pg_walsender-server
branch
from
September 27, 2026 19:54
96cf12e to
599c702
Compare
Additions to the existing src/bin/common/ shared library, adapted from PostgreSQL's own source rather than reimplemented, prep for a real replication-protocol server (a later PR) to authenticate clients by HBA address/network and by SCRAM, the same way postgres's own postmaster does, instead of accepting connections unauthenticated. Nothing here is used by pg_autoctl today. - ifaddr.c/.h (new): local interface address enumeration (getifaddrs()- based), the same code Postgres's own libpq/backend HBA matching uses to decide whether a connecting client's address falls inside a 'samehost' or 'samenet' pg_hba.conf entry. - scram.c/.h + scram_compat.h (new): a server-side SCRAM-SHA-256 verifier (parse/build a stored SCRAM secret, run the exchange), built on top of Postgres's own common/scram-common.c, common/hmac.c, common/cryptohash.c and common/saslprep.c -- scram_compat.h is the small adapter layer translating between those internal APIs and what a frontend-side caller (not a backend) can actually link against. - ipaddr.c/.h (extended): five new peer-address-matching entry points (ipaddrHostMatchesAddress, ipaddrInCIDR, ipaddrIsSameHostOrNet, ipaddrFindHostnamesFromAddress) ported from postgres's own src/backend/libpq/hba.c, for matching a numeric client address against an HBA file's host/CIDR/samehost/samenet entries. - file_utils.c/.h (extended): write_file_atomic(), a rename()-into-place atomic write for a file another process might read concurrently (a config/state file another running service polls). Makefile.common gains -lssl -lcrypto (scram.c's own crypto primitives) and -lresolv (ifaddr.c/ipaddr.c's own resolver use). Verified: PG17 build (pg_autoctl continues to link and behave identically; everything here is compiled but unreferenced) + monitor installcheck (19/19, unaffected); citus_indent --check and ci/banned.h.sh pass with no exceptions needed (all of it already follows this project's own style).
A new binary (src/bin/pg_walsender/, 48 files) speaking the real Postgres replication wire protocol: IDENTIFY_SYSTEM, SHOW, BASE_BACKUP, TIMELINE_ HISTORY, CREATE/READ/DROP_REPLICATION_SLOT, START_REPLICATION, and this project's own FETCH_FILE (a restore_command-friendly one-shot file fetch, alongside the standard commands). Hardened like a real network-facing server rather than a manual-testing toy: - pg_hba.conf-style HBA file (host/hostssl/hostnossl, CIDR/hostname/ samehost/samenet address forms, trust/scram-sha-256/reject), built on the common/ipaddr.c address-matching this PR adds. - SCRAM-SHA-256 authentication (common/scram.c) and TLS (OpenSSL), matching Postgres's own defaults. - Absolute per-connection auth deadline (--auth-timeout), oversize pre- auth message rejection, a per-route replication-slot cap -- the same postmaster-like hardening a network-facing server needs that a one-off script doesn't. - A routes file (archiver-routes.ini) mapping "<formation>/<group>" to a local storage root, and a small vendored ustar writer (vendor/tar.c, reformatted to this project's own style, unlike pg_receivewal's deliberately-untouched vendoring) for BASE_BACKUP's tar stream. - A single-writer "nodes list" refresher process + read-only children, the same nodehost cache design a real postmaster uses for pg_hba.conf's own cached DNS resolution, so a lookup never blocks a connection. Nothing in this PR calls pg_walsender from pg_autoctl yet: it is a fully functional, independently testable server on its own, with no archiver/ monitor integration -- that wiring is a later PR's job. Includes the two src/bin/common/ additions (ifaddr.c, scram.c/scram_compat.h) and extensions (ipaddr.c's five new address-matching entry points, file_ utils.c's write_file_atomic()) this server needs, since nothing else uses them yet either. Build wiring: src/bin/Makefile builds pg_walsender alongside pg_autoctl/ pgaftest; Dockerfile copies the binary with a bracket-glob COPY (kept optional so tests/upgrade can still build the Dockerfile against an old release's source tree that predates this file); top-level Makefile's docker-check/docker-indent now run correctly from a git worktree and also run ci/banned.h.sh (ci-lint). New test: tests/tap/specs/pg_walsender_standalone.pgaf -- a hand-crafted routes file, HBA rule, and archiver-systemid, real WAL captured off node1 by the stock OS pg_receivewal (not this project's own vendored copy, which is a separate PR), pg_walsender started directly by hand (no pg_autoctl archiver integration exists yet in this PR's stack). Covers IDENTIFY_SYSTEM, SHOW wal_segment_size, FETCH_FILE (byte-for-byte match against the captured segment), and START_REPLICATION (a second, real pg_receivewal streaming from pg_walsender itself, byte-for-byte match against what was captured). Verified: PG17 build + monitor installcheck (19/19, unaffected); citus_indent --check and ci/banned.h.sh pass; pg_walsender_standalone.pgaf passes end to end (4/4 steps) on PG17.
- Strip the incorrect "Copyright (c) Microsoft Corporation" line from every genuinely new file in this PR (pg_walsender's own code, plus the scram.c/.h/scram_compat.h additions to src/bin/common/); none of it is Microsoft's. Genuinely vendored files (ifaddr.c/.h, byte-for-byte from PostgreSQL) keep their real upstream PostgreSQL copyright, untouched. - Remove ws_write_file_atomic (ws_util.c/.h) entirely: pg_walsender already links src/bin/common/, so duplicating write_file_atomic() had no Makefile-split justification (unlike the WAL-segment-arithmetic case, which genuinely can't link across the pg_autoctl/pg_walsender binary boundary). Ported its extra safety into the shared function instead of dropping it: write_file_atomic() now uses a pid-qualified tmp name (safe when two processes write the same path concurrently, which cmd_replication_slot.c's per-connection children can do), fsync()s the file before rename(), and fsync()s the containing directory afterward -- strictly more durable than its previous "tmp + rename" behavior, with no signature change, so every existing pg_autoctl caller benefits automatically. All 3 pg_walsender call sites (refresher.c x2, cmd_replication_slot.c) now call it directly. Verified: PG17 build + monitor installcheck (19/19, unaffected); citus_indent --check and ci/banned.h.sh pass; pg_walsender_standalone.pgaf passes end to end (4/4 steps).
…r.l/repl_gram.y), and fscanf-based read_backup_label Task 1 (highest-value review item): repl_command.c's hand-rolled recursive-descent parser for the replication command language is replaced by a real flex/bison pair adapted from PostgreSQL's own src/backend/replication/repl_scanner.l and repl_gram.y: - repl_scanner.l keeps the same token set, the same xq/xd exclusive states for single-quoted strings and double-quoted identifiers (with '' / "" escaping), and the same UCONST/RECPTR patterns as the real scanner. Backend-only bits are swapped for this project's own conventions: ereport() becomes log_error() plus a parse-failed flag, StringInfo becomes PQExpBuffer, and there is no bison-bridge/reentrant scanner -- pg_walsender forks one child per connection (accept_loop.c), so plain global yylex()/yyparse() state is safe, the same way this project's own pgaftest test_spec_scan.l/test_spec_parse.y already work. One addition beyond the real scanner: an exclusive <fname> state for this project's own FETCH_FILE extension, whose argument is a WAL segment filename (digits and letters, case-sensitive) that fits neither the UCONST nor the IDENT pattern. - repl_gram.y keeps the same grammar shape (command alternatives, generic_option_list for BASE_BACKUP/CREATE_REPLICATION_SLOT options, create_slot_legacy_opt for the pre-9.4 EXPORT_SNAPSHOT/RESERVE_WAL/etc keywords) but the semantic actions fill in a plain WsCommand struct directly (repl_command.h), with a fixed-size WsCommandOption array standing in for a backend DefElem List -- no Node*/makeNode()/palloc(), none of which exist in this frontend project. ALTER_REPLICATION_SLOT, UPLOAD_MANIFEST and logical START_REPLICATION are dropped: pg_walsender never implements them. FETCH_FILE is added as this project's own extension, with no Postgres equivalent. This also lets two pieces of ad hoc text-scanning be deleted, now that their input arrives already parsed: - cmd_base_backup.c's scan_options() (BASE_BACKUP's own option-list tokenizer) is gone; cmd_base_backup() now takes the parsed WsCommandOption array directly. - cmd_replication_slot.c's parse_slot_name() is gone; slot names arrive pre-parsed via the grammar's IDENT token, which already implements the same quoting/case-folding PostgreSQL's own replication clients expect (case-preserved when double-quoted, folded to lower case otherwise -- the old hand-rolled version never folded case on a bare name, which was a latent bug against real Postgres semantics). - cmd_start_replication.c's own SLOT/PHYSICAL/TIMELINE/LSN token-eating is gone in favor of the grammar's parsed slotName/startLsn/timeline fields. Build infra follows the exact pattern already used for src/bin/pgaftest/test_spec_parse.y/test_spec_scan.l: repl_gram.c/ repl_gram.h/repl_scanner.c are generated once with bison/flex (both already present in the Docker build stage, used for pgaftest) and committed, `make generate` regenerates them, and `.gitattributes` marks the generated files -citus-style so citus_indent leaves them alone. The Dockerfile's existing "touch bison/flex generated files" step is extended to cover pg_walsender's new files too, so the Docker build never needs to invoke bison/flex itself (mirrors pgaftest exactly). Task 3: cmd_base_backup.c's read_backup_label is reimplemented following PostgreSQL's own read_backup_label() in src/backend/access/transam/xlogrecovery.c: the same fscanf()-based, fixed-format-string parsing of "START WAL LOCATION" and "START TIMELINE", instead of the previous ad hoc line-splitting version. Unlike the backend's ereport(FATAL, ...), a parse failure here is logged and the function returns false, matching the existing caller's own fallback behavior. Fields the backend also extracts but no caller here needs (BACKUP METHOD, BACKUP FROM, START TIME, LABEL, INCREMENTAL FROM LSN) are not ported, since nothing here uses them; the function's signature (LSN string + timeline int) is unchanged. Verification: - `docker build --target run` (PG17): clean build, "All 19 tests passed." (monitor installcheck). - citus_indent --check && ci/banned.h.sh: clean (fscanf/fopen/fclose marked IGNORE-BANNED, same convention as elsewhere in this file). - tests/tap/specs/pg_walsender_standalone.pgaf: all steps pass, including a new test_004_grammar_edge_cases step added as permanent coverage for two protocol edge cases the port is specifically meant to get right: a double-quoted slot name with an embedded escaped quote (rejected at the *semantic* layer, not a syntax error, proving the <xd> lexer state extracted the identifier correctly) and a garbage command producing a clean ErrorResponse with the connection still usable afterwards. A BASE_BACKUP-with-quoted-option-value case was also verified manually against a standalone --insecure instance (reaches the semantic "no base backup configured" error, not a syntax error) but was not added to the permanent spec, to avoid the multi-layer shell- quoting fragility that would introduce into the .pgaf file for no additional grammar coverage beyond what test_004 already exercises. Not included in this commit: Task 2 (hba.c tokenizer rewrite), Task 4 (main.c CommandLine-tree restructuring), Task 5 (refresher.c retry policy / PGSQL wrapper), and Task 6 (missing-header-comment audit) are deferred -- out of scope for the time available in this pass.
…ext_token()
hba_parse used to split each line on plain whitespace with strtok_r and
strip everything after the first '#' unconditionally, so a quoted field
containing a space or a literal '#', a trailing '# comment' after real
tokens, and a line-continuation backslash all either broke tokenizing
or were not supported at all.
Ported the tokenizing mechanism (not the feature set) of PostgreSQL's
own src/backend/libpq/hba.c next_token()/tokenize_auth_file():
- next_hba_token() replaces strtok_r's per-field split: a field can be
wrapped in double quotes (so it may contain spaces or a literal
'#'), a doubled "" inside a quoted field is a literal '"', and an
unquoted '#' starts a comment that runs to the end of the line.
- hba_read_logical_line()/HbaLineReader join physical lines ending in
a trailing backslash into one logical line before any field
splitting happens, the same order of operations PostgreSQL uses
(continuation joining happens first, so a comment or quote that
spans a continued line behaves the same way it does in Postgres).
This project's own HBA feature set (documented in hba.h) is unchanged:
still exactly 5 whitespace/quote-delimited fields per logical line, no
@-include expansion, no comma-separated lists, no regular expressions.
A malformed logical line still rejects the whole file, matching
PostgreSQL's fail-closed behavior for a bad pg_hba.conf.
Rule fields are now individually strdup'd (freed by hba_rule_free_fields)
rather than pointing into the file's own buffer, since the tokenizer no
longer null-terminates fields in place inside a reusable per-line buffer.
Verified: full docker build (17 tests + monitor installcheck, no
warnings), citus_indent + banned.h.sh clean, the standalone pgaftest
spec pg_walsender_standalone.pgaf (5/5), and manual checks of a
comment-only line, a trailing '# comment', a quoted field with an
embedded space, and a backslash line continuation all parsing as one
correct rule set (confirmed against a deliberately unquoted variant of
the same line, which still triggers the 'Malformed HBA line' rejection
as before).
… framework main.c used to hand-roll its own argv[1] dispatch (string-comparing against "fetch-file"/"scram-secret") and then a manual getopt_long loop for whichever mode matched. Replaced that with a CommandLine tree (src/bin/lib/subcommands.c/commandline.h), the same framework pg_autoctl/cli_root.c and pgaftest/cli_root.c already use, split into a new cli_root.c following pgaftest's own file layout (main.c stays a thin entry point; cli_root.c holds the command table and every sub-command's getopt/run pair) since pg_walsender is a similarly-sized standalone binary, not pg_autoctl's own much larger tree. Three sub-commands: serve (the accept loop), fetch-file, scram-secret. The framework itself has no notion of an optional/default sub-command, so pg_walsender_default_argv() (cli_root.c) is a small, explicit shim: when argv[1] isn't a known sub-command name (or --help/-h, which commandline_run() already intercepts on its own), it splices "serve" into argv before main() calls commandline_run(), so bare "pg_walsender --port ... [--pgdata ... | --insecure] ..." keeps working exactly as before -- only fetch-file and scram-secret require typing their name. Every flag is unchanged: --port/--pgdata/--insecure/--auth-timeout/ --ssl-cert-file/--ssl-key-file for serve, fetch-file's own --host/--port/--route/--filename/--output/--user, and scram-secret's --user (password still via PGPASSWORD, never the command line). Only the dispatch mechanism moved; "pg_walsender --help" output is cosmetically different (the framework's own generated format) as expected. Makefile: added cli_root.c to LOCAL_SRC and lib-commandline.o (built from src/bin/lib/subcommands.c/commandline.c by Makefile.common's own generic rule, already used by pgaftest) to OBJS. Verified: full docker build (17 tests + monitor installcheck, no warnings), citus_indent + banned.h.sh clean, the standalone pgaftest spec pg_walsender_standalone.pgaf (5/5, including the flex/bison grammar edge cases), and manual checks of "pg_walsender --help", "pg_walsender fetch-file --help", "pg_walsender scram-secret --help", a real scram-secret run, and an invalid flag on the default (serve) path producing a clear error plus usage on stderr and exit code 1.
…licy
connect_to_monitor() used a single bare PQconnectdbParams() attempt, and
refresh_route() ran two PQexecParams() calls and parsed PQgetvalue()
results inline, instead of this project's own PGSQL struct + the
pgsql_execute_with_params()-with-callback pattern used throughout
monitor.c (src/bin/common/pgsql.c/pgsql.h).
Connection lifetime: a single long-lived PGSQL connection (monitorPgsql)
is now kept open across every tick of ws_refresher_main()'s own loop and
every route it validates on a given tick, rather than reconnected once
per query. The refresher already runs its own periodic loop (a
per-route coalescing timer plus a WS_REFRESH_HEARTBEAT_MS sweep) and a
negative cache on failure (mark_failure()); a short-lived
connect/query/disconnect cycle on every single check would only add
extra TCP and SCRAM round-trips for no benefit, since the outer loop
already provides the long-horizon retry/backoff a periodic background
process needs. monitor_ensure_connection() is the only place that
(re)connects: it reuses the existing connection unless the monitor URI
file changed or PQstatus() shows the connection died since the previous
tick, in which case it reconnects via pgsql_init()/pgsql_finish().
Retry policy: the connection uses pgsql_set_interactive_retry_policy()
(a bounded, backed-off handful of attempts within pgconnect_timeout
seconds -- 2s by default) instead of a single bare connect(), so one
transient hiccup no longer immediately marks the route as failed;
verified manually below.
Split into the same connect/run-one-query/parse-its-result shape
monitor.c's own monitor_get_*() functions use:
- monitor_ensure_connection(): connect-or-reuse.
- monitor_get_group_hosts_hash() / monitor_get_group_hosts(): each runs
one query via pgsql_execute_with_params(), with its own small parse
callback (parse_hosts_hash_result / parse_hosts_list_result) modeled
on monitor.c's SingleValueResultContext/parseSingleValueResult, sized
for this query's own one- and two-column results.
- refresh_route() itself is now just the control flow over those,
with no raw libpq calls left.
Makefile/defaults.h: pgsql.h needs pg_autoctl's own defaults.h (for
POSTGRES_PING_RETRY_TIMEOUT and friends, used inside pgsql.c, and
MAXIMUM_BACKUP_RATE_LEN, used directly in a pgsql.h struct field), so
-I../pg_autoctl was added to this Makefile's CFLAGS. pg_walsender's own
defaults.h (its own trimmed, standalone-binary substitute for
pg_autoctl's) sits earlier in the same -I search path and so still
shadows pg_autoctl's file by name for every other pg_walsender source;
MAXIMUM_BACKUP_RATE_LEN is therefore also duplicated into it, the same
way PG_AUTOCTL_REPLICA_USERNAME already was, so pgsql.h's own struct
definitions still compile for translation units that only see this
project's local defaults.h.
Verified: full docker build (17 tests + monitor installcheck, no
warnings), citus_indent + banned.h.sh clean, the standalone pgaftest
spec pg_walsender_standalone.pgaf (5/5) -- which does NOT exercise
refresher.c/monitor connectivity at all, since it never creates an
archiver-monitor.uri file (ws_accept_loop only forks the refresher once
that file exists). Confirmed separately, by hand, against a real
pgaftest cluster (pgaftest cluster setup on this same spec) with
pg_walsender started manually against a pgdata pointed at that
cluster's real monitor:
- a reachable but schema-mismatched monitor: the connection succeeds
(proving the retry-policy connect path works end to end), the query
fails server-side ("function pgautofailover.get_group_hosts_hash
does not exist" -- a pre-existing gap in the monitor extension,
unrelated to this change: the old bare-libpq code ran the exact same
SQL text and would fail identically; flagged separately as a
follow-up, out of scope here), and mark_failure() correctly writes
the .err sentinel;
- a deliberately unreachable monitor (a non-routable address): the log
shows "retrying until the server is ready" and "after 2 attempts in
2007 ms", i.e. real bounded retries within the interactive policy's
window, not an instant single-attempt give-up, before mark_failure()
runs.
Audited every function in src/bin/pg_walsender/*.c and src/bin/common/ scram.c against the previous review pass's own starting list, re-verifying each item against the current tree (some had already been resolved by the flex/bison grammar port and by Tasks 2/4/5 of this same pass, and were skipped here since those rewrites already carry fresh comments). Added a genuine explanatory comment -- parameters, return semantics, protocol/behavior context matching the density already used elsewhere in each file -- wherever one was missing; left every existing, adequate comment untouched. Files touched: accept_loop.c, auth.c, cmd_fetch_file.c, cmd_identify_system.c, cmd_show.c, cmd_timeline_history.c, framing.c (most of its ws_send_*/ws_read_* wire-message functions had none), hba.c (parse_method only -- the rest was rewritten fresh by Task 2), monitor_hosts.c, refresher.c (key_is_plausible only -- the rest was rewritten fresh by Task 5), routes.c, startup.c, tar_stream.c, tls.c, wal_dir_scan.c, ws_util.c, and common/scram.c (scram_parse_verifier, scram_mock_init). main.c and cli_root.c are untouched here: Task 4 already gave them fresh, complete comments. Verified: full docker build (17 tests + monitor installcheck, no warnings -- one added comment initially tripped -Wcomment on a stray "*/" substring in prose, fixed), citus_indent + banned.h.sh clean, and the standalone pgaftest spec pg_walsender_standalone.pgaf (5/5).
…ILE client to common/
Task 1: remove the "monitor" HBA address keyword and its supporting
machinery entirely. It depended on two monitor-side SQL functions
(pgautofailover.get_group_hosts_hash, pgautofailover.get_group_hosts) that
do not exist anywhere in this PR's own dependency chain (no monitor
extension is part of this PR at all); the feature only becomes real once a
later archiving PR adds that schema, so it is reintroduced there instead.
Removed:
- refresher.c/refresher.h (the nodes-list refresher child process and its
PGSQL-wrapper-based monitor queries)
- monitor_hosts.c/monitor_hosts.h (the local cache reader/matcher)
- hba.c's "monitor" address-keyword special case: an HBA line using it
is now tokenized as an ordinary address token (resolved via DNS like
any other hostname), no new rejected-keyword path was added
- accept_loop.c's forking/reaping/stopping of the refresher child, its
datagram socket, and the archiver-monitor.uri existence check that
triggered it
- cli_root.c's monitorUriPath/refreshSockPath construction
- WsAuthConfig's monitorUriPath/refreshSockPath fields and the matching
hba_lookup()/rule_address_matches() parameters, threaded back down to
what is actually used
- the now-dead ws_auth_deadline_remaining_ms() (its only caller was
monitor_hosts.c's request_refresh())
- refresher.c/monitor_hosts.c from the Makefile's source list
The default HBA file pg_walsender writes now documents an explicit rule
per host instead of a "monitor" line that would silently do nothing
useful. tests/tap/specs/pg_walsender_standalone.pgaf did not reference any
of this and needed no changes; it still passes 5/5.
Task 2: move the FETCH_FILE *client* out of this binary. It never had any
dependency on pg_walsender's own wire-framing code (framing.h) or any other
pg_walsender-internal header -- it is a plain libpq client (PQconnectdbParams
+ PQexec("FETCH_FILE ...") + PQgetCopyData()) -- so it moves as-is to
src/bin/common/fetch_client.c/fetch_client.h, which both pg_autoctl and
pg_walsender already link. This lets a future caller (pg_autoctl restore
command, in the later archiving PR) call ws_fetch_file_client() directly,
in-process, with no execv()/subprocess indirection.
pg_walsender's own "fetch-file" CLI sub-command is removed from cli_root.c
(and its --help text): pg_walsender is now a server binary (serve, the
default command) plus its scram-secret utility, nothing else.
cmd_fetch_file.c, the *server-side* FETCH_FILE handler, is untouched.
tests/tap/specs/pg_walsender_standalone.pgaf's test_002_fetch_file drives
the server-side FETCH_FILE command via a plain psql -c "FETCH_FILE ..."
and does not use the removed CLI client; it needed no changes.
Verified: docker build (PG17) succeeds with no compiler warnings, monitor
installcheck's 19/19 tests pass, citus_indent --check and ci/banned.h.sh
pass clean, and pg_walsender_standalone.pgaf passes 5/5.
Documents the design decisions in this codebase for the next engineer: what pg_walsender is and isn't (standalone, no monitor/archiver integration yet), the wire protocol commands and the FETCH_FILE extension (and why its client now lives in common/, not here), the replication-slot design in depth (bookkeeping marker file, not a real slot; naming/safety constraints; the still-unwired WAL-retention enforcement), the repl_scanner.l/repl_gram.y grammar port (what's dropped, what's added, the <fname> lexer state), hba.c's tokenizer design and the removed "monitor" keyword, authentication (SCRAM, TLS, the auth-timeout deadline), the routes.ini design, the vendored ustar writer's own formatting choice, and how tests/tap/specs/pg_walsender_standalone.pgaf exercises all of this by hand ahead of the archiver integration.
Adds docs/ref/pg_walsender.rst (Sphinx reference page: synopsis, description, options for server mode and scram-secret -- no fetch-file, which was removed from pg_walsender's own CLI), registers it in docs/conf.py's man_pages list, and adds it to docs/ref/manual.rst's toctree so the html build does not warn about an orphan document. Verified: built the docs' own Sphinx toolchain (docs/requirements.txt) in a throwaway Docker image and ran 'sphinx-build -b man docs docs/_build/man'; it builds pg_walsender.1 along with every other man page, with no warnings specific to this new page (the only remaining warnings are the pre-existing tikz SVG image-not-readable ones, present because this throwaway image does not run the docs/tikz build step). Inspected the generated troff with 'man --local-file' and confirmed it renders cleanly end to end (NAME/SYNOPSIS/DESCRIPTION/OPTIONS/ ENVIRONMENT/EXAMPLES/AUTHOR/COPYRIGHT). The project's own Dockerfile.docs (used by 'make build-docs') could not be used as-is: it COPYs Makefile.azure/Makefile.citus, which no longer exist in this tree, and its base image (debian:bullseye-slim) has aged out of the Debian package mirrors (404s on texlive-bin and others) -- both pre-existing issues, unrelated to this change. Verification instead used a throwaway Dockerfile against docs/requirements.txt for the parts relevant to a man-page build (no LaTeX/tikz needed for -b man).
…ack, not 0.0.0.0/0 Every connection this test makes to pg_walsender is node2 talking to its own server over 127.0.0.1 -- there never was a reason for the hand-written archiver-hba.conf to admit "0.0.0.0/0" (the entire internet, in principle). 127.0.0.1/32 is the actually correct, properly scoped rule for what this test really does, written by hand exactly as an operator running pg_walsender on its own (no archiver, no monitor, see this test's own header comment) would maintain it themselves. Verified: pg_walsender_standalone.pgaf still 5/5 green.
…wildcard route Reframe the route key (routes.ini's section names, matched against a connection's dbname, and archiver-hba.conf's ROUTE field) as what it already is in the code: an opaque, operator-chosen string pg_walsender never parses, splits, or gives filesystem meaning to. Nothing in this codebase ever treats "<formation>/<group>" as anything but a plain string compared with strcmp() -- that convention is pg_auto_failover's own, useful because it reads well and is guaranteed unique, but not a requirement pg_walsender imposes on anyone else driving it. Updated: hba.c/hba.h, routes.c/routes.h, walsender.h, startup.c, cli_root.c's help text, and the README's HBA/routes sections, all to describe this generic mechanism first and pg_auto_failover's convention as one example of it, plus an explicit callout that a key's "/" carries no filesystem meaning (only a route's own explicit "path" property ever does). Add routes.ini's own "*" wildcard route (WS_ROUTES_WILDCARD_KEY, routes.c's routes_find()): a dbname matching no route of its own falls through to it, exactly like PgBouncer's own [databases] "*" entry (pgbouncer.org/config.html) -- same precedence (an exact match always wins), same "*" syntax, chosen so anyone who has run a PgBouncer already knows what to expect. Deliberately NOT PgBouncer's per-request name substitution: PgBouncer's wildcard is safe substituting the requested dbname into a *connection string* handed to a real PostgreSQL server, which validates it; ours would substitute an unauthenticated client's string into a *filesystem path*, and a route key is explicitly allowed to contain "/" (pg_auto_failover's own convention does) -- a naive %r-style substitution would turn that convention into path traversal. Every dbname that falls through to "*" shares that one configured path verbatim. This is what makes running pg_walsender useful outside pg_auto_failover with zero multiplexing ceremony: a single-cluster deployment needs one `[*]` section and never has to learn or type a special dbname value. archiver-hba.conf's own ROUTE matching stays completely independent of this (always against the literal dbname, never against which WsRoute the wildcard resolved to). docs/ref/pg_walsender.rst: a full worked standalone example (no pg_auto_failover anywhere) -- create a replication role on a real primary, seed an initial base backup and start continuous pg_receivewal capture into the exact on-disk layout pg_walsender expects (verified against cmd_base_backup.c's basebackups/.latest + backup_label handling and wal_dir_scan.c's flat WAL layout), configure and start pg_walsender, then build a fresh PGDATA and PITR-restore it with a restore_command driving FETCH_FILE through a plain psql one-liner (the exact invocation shape already proven by this test suite's own test_002_fetch_file). Verified the new .rst builds clean with sphinx-build -b man (no warnings). tests/tap/specs/pg_walsender_standalone.pgaf: new test_005_wildcard_route, proving both the wildcard fallback and exact-match precedence against a second route with a distinguishable systemid. Also fixed a stale README line still describing the HBA rule as "0.0.0.0/0" after an earlier commit scoped it to 127.0.0.1/32. Verified: PG17 docker build clean (zero warnings), citus_indent/ banned.h.sh clean, pg_walsender_standalone.pgaf 6/6 green.
… a fabricated one The previous Synopsis block was hand-assembled, not captured, and wrong on two counts: it invented a "+" marker in front of "serve" that this project's own CommandLine framework never prints for either of pg_walsender's two sub-commands (both are leaves -- "+" means "has sub-commands of its own", as pg_autoctl.rst's own tree shows for create/drop/etc.; it was never a "this one is the default" marker), and it merged the root command's usage line with serve's own detailed flag list into a single block that matches neither `pg_walsender --help`'s actual output nor `pg_walsender serve --help`'s. Replaced with two verbatim captures, built and run against this branch's own binary: `pg_walsender --help` (root help -- note neither "serve" nor "scram-secret" gets "+", correctly) and `pg_walsender serve --help` (the actual flag usage line), with a short explanation of why they differ: --help/-h is the one case pg_walsender_default_argv()'s "no sub-command means serve" splicing deliberately leaves alone (see its own comment, cli_root.c), so `pg_walsender --help` shows the root tree, never serve's flags. Verified: PG17 docker build clean, sphinx-build -b man clean (no warnings).
…dalone archiver Design document only, nothing implemented -- see the file's own header. Covers, in order: - PostgreSQL's own continuous-archiving/PITR contract (archive_command's exit-code and overwrite-safety rules, .backup history files, restore_ command's own expectations, timeline handling) and the gaps it exposes in the current design (no .backup in the FETCH_FILE allow-list, no write path into a route at all, WAL capture entirely external and unsupervised). - Renaming pg_walsender to pg_walserver once it grows past being purely a *sender*: checked PostgreSQL's own glossary (avoiding a collision with its own "archiver process" term and this project's planned archiver feature) and external tools' vocabulary (pgBackRest's "repository" convention, none of which actually speak the native replication protocol the way this binary already does) before settling on it. - routes.ini growing an `upstream` connection-string property per route, and new client-side sub-commands (fetch-systemid, basebackup, setup) built on it -- no new wire protocol needed for any of these. - The archive_command push side (ARCHIVE_FILE, CopyIn) and, importantly, that this is not a green field: the pgaf-integrated design already has a working, monitor-backed, quorum-aware "already captured?" check (archiver_confirm.c's use of pgautofailover.wal_archived()) that should stay the primary mechanism there, gaining only a push *fallback* (a new common/ ws_archive_file_client(), called where archiver_confirm.c currently just retries forever) rather than being replaced. A new ARCHIVE_STATUS wire command (cheap CRC32C-based check, no transfer) is the standalone-only equivalent, for when there's no monitor to ask. - An embedded, per-route supervised WAL capturer (the already-vendored pg_receivewal, relocated to common/ so both binaries link it) so a standalone pg_walserver needs no external, unsupervised pg_receivewal process at all. - Fit as a pg_auto_failover building block, including the explicit caveat that archive_command-push (fires only on completed segments, asynchronous relative to commit) is a defense-in-depth addition for the archiver's failover/quorum-member role, not a replacement for the continuous streaming pull path there. - Open questions and a 3-phase implementation order.
Half its job is no longer "sending": this PR already ships FETCH_FILE (a read), and the design work in DESIGN-standalone-archiving.md adds an actual write path (ARCHIVE_FILE push) and an embedded WAL capturer (pull). "Sender" alone was already an inaccurate description before either of those land, so the rename happens now, in this same PR, rather than being deferred to a separate one purely to avoid touching an already-reviewed diff again -- continuing to iterate on an open PR through review is the normal shape of review. pg_walserver: "sender" replaced by the more general "server" -- picked over pg_archiver/pg_walarchive-shaped alternatives (collide with PostgreSQL's own "archiver process" glossary term and this project's own planned pg_auto_failover archiver feature) and over a pgbouncer-inspired pg_walrepo/pg_walstore-style coinage (a new name over a borrowed one, where pg_walserver needs neither). See DESIGN-standalone-archiving.md's own "Naming" section for the full research trail. Mechanical scope: directory move (src/bin/pg_walsender -> pg_walserver), Makefile (PG_WALSENDER -> PG_WALSERVER, binary name), every pg_walsender/ PG_WALSENDER identifier and string across the moved directory plus every outside reference (.gitattributes, Dockerfile, docs/conf.py, docs/ref/ manual.rst, the docs page itself renamed pg_walsender.rst -> pg_walserver. rst, src/bin/Makefile, src/bin/pg_autoctl/Makefile, common/fetch_client.*, common/ipaddr.h, common/scram.h), walsender.h renamed to walserver.h (and its own include guard), and the test spec renamed pg_walsender_standalone.pgaf -> pg_walserver_standalone.pgaf (including its own step name mentioning the binary). No bare "walsender" text existed anywhere referring to something other than this binary (verified by grep before renaming), so this was unambiguous everywhere. Verified: PG17 docker build clean (zero warnings), a real `pg_walserver --help` run confirms the CLI itself now says "pg_walserver" throughout and "pg_walsender" is gone from the built binary, citus_indent/ banned.h.sh clean, pg_walserver_standalone.pgaf still 6/6 green (including the renamed test_003_start_replication_from_pg_walserver step), and sphinx-build -b man clean on the renamed docs page.
….ini, real-standby test coverage
Implements phase 1 of DESIGN-standalone-archiving.md (the routes.ini
"upstream" property + the three client-side sub-commands), reusing
existing pg_autoctl code rather than reimplementing it:
- routes.h/routes.c: WsRoute gains an "upstream" property (a libpq
connection string), parsed alongside "path".
- cli_upstream.c/.h: shared --route/--path/--upstream/--host/--port/
--user resolution for the three new sub-commands, the same explicit-
flag-wins-over-config layering restore_command_resolve() already uses.
Parses a plain keyword=value "upstream" string directly with
PQconninfoParse() (not this project's own parse_pguri_ssl_settings(),
whose checkForCompleteURI path is built for postgres:// URIs, not this
shape).
- cli_fetch_systemid.c/.h: `pg_walserver fetch-systemid` -- reuses
pgctl_identify_system() (src/bin/common/pgctl.c) unchanged, a real
replication-mode IDENTIFY_SYSTEM. Refuses to overwrite an already-
recorded *different* system identifier unless --force.
- cli_basebackup.c/.h: `pg_walserver basebackup` -- reuses
pg_basebackup_fetch(), which did not yet exist on this branch (it was
only ever added on a separate, more advanced branch, splitting
pg_autoctl's historical pg_basebackup() into a reusable fetch-only
piece plus the standby-init-specific rmtree-and-move wrapper, and later
gaining an HBA-readiness retry preflight). Ported both commits' changes
forward to this branch's own src/bin/common/pgctl.c/pgctl.h/pgsql.h
(a straight cherry-pick conflicted: the original commit also touched a
service_archiver_basebackup.c that doesn't exist here) -- same logic,
reused from where it already existed, not reinvented. Validates the
result (backup_label/PG_VERSION present) before atomically swapping
basebackups/.latest.
- cli_setup.c/.h: `pg_walserver setup` -- the wizard: write/validate the
pg_walserver.ini section, then fetch-systemid (whose own
pgctl_identify_system() connection doubles as the role-permission
check: PostgreSQL refuses a replication-mode connection for a role
lacking REPLICATION at the backend level, independent of HBA -- an
earlier version ran a separate plain-SQL pg_roles.rolreplication query
first, removed because that connection targets an ordinary database,
which the replication role's own HBA rule usually does not admit at
all), and, with --with-basebackup, basebackup -- synchronously, not
returning until the first backup has actually succeeded.
- cli_root.c: wired all three into the CommandLine tree and the "no
sub-command means serve" argv-splicing exemption list.
- routes.ini renamed to pg_walserver.ini throughout (code, docs, tests):
what used to be colloquially "routes.ini" in prose is now the file's
actual name, matching the binary's own.
- docs/ref/pg_walserver.rst's worked example now leads with `pg_walserver
setup --with-basebackup` instead of the previous three hand-run steps
(psql for the systemid, pg_basebackup for the backup, a hand-written
ini section); added a new "real standby" example (primary_conninfo +
standby.signal, no restore_command) alongside the existing PITR one.
- tests/tap/specs/pg_walserver_standalone.pgaf: the setup{} block now
uses `pg_walserver setup --with-basebackup` instead of hand-writing
pg_walserver.ini and archiver-systemid; new test_006_real_standby_
with_core_tools proves a real pg_basebackup BASE_BACKUP from
pg_walserver (exercising cmd_base_backup.c's actual wire serving, not
just checking the file exists on disk) followed by a real,
continuously-streaming standby -- primary_conninfo, standby.signal,
stock PostgreSQL commands only -- that catches up via a genuine
walreceiver (not pg_receivewal) and promotes.
That last step surfaced a real PostgreSQL protocol fact, not a bug:
libpqrcv_connect() (libpqwalreceiver.c) always sends the literal
dbname "replication" for a physical replication connection, silently
discarding whatever dbname a real primary_conninfo carries. A route
reachable by a real physical standby by name therefore needs a second
section literally keyed "replication" pointing at the same path (or,
for a single-route deployment, the "*" wildcard) -- documented in both
the README and docs/ref/pg_walserver.rst, and reflected in the test.
Verified: PG17 docker build clean (zero warnings), citus_indent/
banned.h.sh clean, sphinx-build -b man clean, and
pg_walserver_standalone.pgaf 7/7 green including the new real-standby
step.
…sical standbys
test_006_real_standby_with_core_tools (the previous commit) surfaced a
real PostgreSQL wire-protocol fact: libpqrcv_connect() always sends the
literal dbname "replication" for a physical replication connection,
silently discarding whatever dbname a real primary_conninfo carries
("the database name is ignored by the server in replication mode, but
specify 'replication' for .pgpass lookup", libpqwalreceiver.c's own
comment). dbname-based routing therefore cannot select a route by name
for a real physical standby -- only the "*" wildcard or a route literally
keyed "replication" is reachable by one.
Researched two protocol-native alternatives, both already shipped by real
PostgreSQL and unaffected by the dbname override (they happen at or
before the TLS handshake, below where that override lives):
- TLS SNI: libpq already sends it by default (sslsni, fe-secure-
openssl.c); PostgreSQL's own backend already has a matching, documented
server-side feature for exactly this (ssl_sni/hosts_file, sni_
clienthello_cb() in be-secure-openssl.c, using SSL_client_hello_get0_
ext() rather than the simpler SSL_get_servername() on OpenSSL's own
advice against the latter's callback-ordering fragility).
- TLS client certificate CN: sslcert/sslkey are not touched by the
dbname override, flowing through from the original conninfo unchanged
even for a real standby -- the same mechanism PostgreSQL's own
clientcert=verify-full HBA option already uses to map a certificate to
a role, applied to route selection instead.
pg_walserver's own tls.c already links raw OpenSSL directly (no backend
or libpq-fe code), so both are architecturally straightforward reads, no
new dependency. Not designed in detail or scheduled into a phase yet.
A real physical standby's own walreceiver always sends the literal dbname=replication on the wire (libpqrcv_connect() overrides whatever primary_conninfo's dbname= says), so dbname-based routing alone cannot disambiguate more than one route for it. This adds a second routing signal: the TLS Server Name Indication (SNI) extension libpq already sends (sslsni, on by default) from primary_conninfo's own host= value, exactly the mechanism HTTPS virtual hosting uses. - routes.ini gains a "hostname" route property (routes.h/routes.c), set via `pg_walserver setup --hostname`. - tls.c reads the SNI hostname back post-handshake via the simple SSL_get_servername() (safe here because, unlike PostgreSQL's own ssl_sni backend feature, pg_walserver never switches certificates based on it). - auth.c resolves a route in three steps: exact dbname match, then (TLS only) SNI hostname, then the "*" wildcard. HBA's own ROUTE matching stays independent, always against the literal dbname. - `pg_walserver setup` auto-creates a self-signed certificate for --pgdata (reusing pg_create_self_signed_cert(), unchanged) the moment a second named route needs one, and warns if that route has no --hostname. - `pg_walserver serve` now hard-refuses to start (log_fatal/exit(1)) with more than one named route and no TLS configured; a single named route keeps working with no TLS at all. New tap spec tests/tap/specs/pg_walserver_sni_routing.pgaf: two real, independent routes disambiguated purely by SNI hostname while both present the same useless dbname=replication a real standby forces, plus the auto-cert-creation and no-TLS-refusal behaviors. 4/4 green, alongside pg_walserver_standalone.pgaf's own 7/7. Docs (README.md, docs/ref/pg_walserver.rst, DESIGN-standalone-archiving.md) updated to describe the mechanism and a worked two-route example.
…sh side Task A: `pg_walserver create-cert --pgdata <path> --hostname <name> [--force]` (cli_create_cert.c/.h), a thin CLI wrapper around pg_create_self_signed_cert() (src/bin/common/pgctl.c). Refuses to overwrite an existing server.crt/server.key unless --force, matching this codebase's "never silently replace what's already there" principle (cli_fetch_systemid.c's own systemid overwrite check). cli_setup.c's ensure_tls_for_multiple_routes() is refactored to call the same shared ws_create_cert_run() helper this sub-command uses, instead of duplicating the pg_create_self_signed_cert() call-and-log sequence. Task B: the push side of DESIGN-standalone-archiving.md's "The push side: CHECK_FILE + ARCHIVE_FILE" section. - repl_scanner.l/repl_gram.y: CHECK_FILE and ARCHIVE_FILE grammar productions, following the exact FETCH_FILE pattern (same <fname> lexer state for the filename argument). CHECK_FILE's own "crc32c:<hex>" argument is scanned as one token (a hex CRC can start with a digit, which the ordinary identifier rule can't match). - cmd_check_file.c/.h: CHECK_FILE's wire shape is the lean, SHOW-like RowDescription/DataRow/CommandComplete row the design doc's own open question leaned towards, answering "missing"/"matches"/"differs". - cmd_archive_file.c/.h: ARCHIVE_FILE's CopyIn receive path. The server never trusts a client's own CHECK_FILE checksum: once the whole CopyIn is received, it re-derives the overwrite-safety decision from the real bytes on disk vs. the real bytes just received (identical -> success/ idempotent retry, matching PostgreSQL's own archive_command contract; different -> clean rejection; nothing there yet -> write it via a same-directory temp file plus atomic rename). Hard size cap at the route's own wal_segment_size plus a fixed slack. - cmd_fetch_file.c/.h: ws_fetch_filename_is_servable() extended in place to also accept a base backup's own "<24hex>.<8hex>.backup" history file (Gap #1 in the design doc), shared by both the read and write paths. - ws_util.c/.h: ws_file_crc32c(), a shared size+CRC32C helper for a whole file, built on the same INIT_CRC32C/COMP_CRC32C/FIN_CRC32C facility (port/pg_crc32c.h) real Postgres and pg_autoctl's own nodespec.c already use -- no new CRC32C implementation. - framing.c/.h: ws_send_copy_in_response() ('G' CopyInResponse), the one new wire primitive ARCHIVE_FILE needs. - cli_archive.c/.h + cli_root.c: `pg_walserver archive <path> <filename> --route <key> --host <host> [--port <port>] [--user <name>] [--sslmode <mode>]`, meant to run as (part of) a Postgres archive_command. Implements the design's own 4-step sequence: compute local size+CRC32C, CHECK_FILE, a short bounded intra-invocation recheck (two rechecks, one second apart -- the design doc's own other open question, resolved and documented in cli_archive.c; always run today since pg_walserver.ini has no "capture" property yet, revisit once it does), then ARCHIVE_FILE only if still needed. Exit 0 on success (including "already matches"), nonzero with a clean stderr message otherwise, exactly PostgreSQL's own archive_command contract. Deliberately does not reuse cli_upstream.c's cli_resolve_upstream() -- archive connects to pg_walserver itself, not to a Postgres primary, a different shape of connection entirely; see cli_archive.h's own header comment for the full reasoning. Verification: a clean `docker build --target run` (zero new warnings/errors), `make docker-check`/banned.h.sh clean (one file needed `make docker-indent`, applied), and a new tap spec, tests/tap/specs/pg_walserver_archive_command.pgaf, covering CHECK_FILE "missing"/"matches"/"differs", a real `pg_walserver archive` push and its idempotent-retry/overwrite-safety-rejection behavior, and create-cert's own fresh/refuse-without-force/--force behavior -- 4/4 green via pgaftest against a freshly built pgaf:run-pg17 image. Docs updated: src/bin/pg_walserver/README.md (new sections for the wire commands, `create-cert`, and `pg_walserver archive`'s own 4-step logic), docs/ref/pg_walserver.rst (create-cert and archive_command-based archiving added to the worked example), and DESIGN-standalone-archiving.md (the push side section and its two open questions marked resolved/implemented, Phasing's phase 2 marked done -- following the same "Update: ... implemented" precedent already used there for TLS SNI routing). Out of scope, untouched: the embedded pull WAL capturer (phase 3, "capture = pull") and any vendor relocation of pg_receivewal.
…/cmd_check_file.c
A route with `capture = pull` in pg_walserver.ini (set via `pg_walserver setup --capture pull`) gets its own long-lived, supervised child capturing real WAL straight into the route's own directory, via the vendored pg_receivewal (src/bin/common/vendor/pg_receivewal/, relocated in the previous commit) -- no external pg_receivewal process, no separate supervisor unit to wire up by hand. Supervision reuses this codebase's own existing pattern for long-lived, restart-supervised services, rather than a bespoke fork/waitpid loop: - src/bin/common/process_supervisor.c/h: a generic Service/RestartPolicy child supervisor extracted from pg_autoctl's own supervisor.c core (Erlang-style MaxR/MaxT restart backoff, PID-1-safe orphan reaping via one wildcard waitpid(-1, ...) call site). Not a literal move -- the real supervisor.c is entangled with pg_autoctl's keeper/monitor/nodespec state, so this is a fresh, symbol-distinct (process_supervisor_*/ Process*) extraction of its generic core, living alongside it. - src/bin/pg_walserver/cli_internal.c/h: a hidden `pg_walserver internal service pg-receivewal --route <key> --upstream <conninfo> --path <dir>` sub-command, mirroring pg_autoctl's own `internal service postgres|listener|node-active` pattern exactly -- each capturer child is started via fork()+execv() of the pg_walserver binary itself, re-entering as this hidden sub-command, which then calls the vendored pg_receivewal_main() in-process. This is what makes a restarted capturer safe under a container's PID 1: a live-upgraded binary on disk is picked up on the next restart without ever touching the supervising process, the same property pg_autoctl's own supervisor already relies on. - capture.c's ws_capture_start_all() builds one ProcessService per active `capture = pull` route and hands them to process_supervisor_start(); ws_capture_tick() is the only waitpid(-1, ...) call site in the whole process -- accept_loop.c's own per-connection child bookkeeping is threaded through as its otherChildExited callback instead of running a second, independent wildcard reaper (two independent wildcard waiters were found, during testing, to silently steal each other's zombies). This also gives pg_walserver itself PID-1 orphan-reaping for free. docs/ref/pg_walserver.rst and README.md updated: the standalone worked example's step 1 now shows `pg_walserver setup --capture pull`, with the manual `nohup pg_receivewal ... &` step kept only as the push-only/ external-feed alternative.
added 29 commits
October 14, 2026 16:47
…s, request standby replies Three fixes from an external review (verified against the actual code before acting; several of the review's other findings were checked and refuted -- see PR discussion): - framing.c's ws_read_startup_payload()/ws_read_message() rebuilt the 4-byte big-endian length field via left shifts on a signed int32_t, which is undefined behavior in C when the result doesn't fit in int32_t (e.g. the high length byte has its MSB set). Now reconstructed as uint32_t and only cast to int32_t once assembled. Same treatment for startup.c's protocol-version code field (int32_t -> uint32_t), which was then right-shifted -- implementation-defined rather than UB, but the same fix removes the ambiguity either way. - tls.c only called SSL_CTX_set_cipher_list(), which configures TLS 1.2 and below; TLS 1.3 has its own suite list and its own setter (SSL_CTX_set_ciphersuites()), so a TLS 1.3 handshake was silently falling back to OpenSSL's compiled-in defaults instead of an explicit, reviewed policy. Added the missing call with OpenSSL's own documented TLS 1.3 defaults named explicitly. - cmd_start_replication.c's send_keepalive() always sent replyRequested=false, so a client that only sends a StandbyStatusUpdate when asked (rather than on its own periodic timer, as pg_receivewal/ pg_basebackup already do) would never advance this slot's restart_lsn. Since this project never blocks waiting for the reply either way, there's no cost to always requesting one.
…andLine The prior refactor (ca69b69, 8e96093) only forward-declared each file's getopt/run function pair before relocating its CommandLine definition near the top of the file; every other helper function's full body (static or not) was left wherever it already happened to sit, often still before the CommandLine block. This is pure code motion, no logic/behavior change: every remaining function body (helpers included) is moved below its file's CommandLine declaration, with a forward declaration added for each one ahead of the CommandLine block, so every file's CommandLine is now the first substantive thing after its header comment/includes/trivial static state, matching pg_autoctl's own cli_create_node.c convention with zero exceptions. Files fixed: cli_archive.c, cli_archive_cleanup.c, cli_basebackup.c, cli_create_cert.c, cli_fetch_systemid.c, cli_ls.c, cli_ps.c, cli_restore_wal.c, cli_serve.c, cli_setup.c, cli_status.c. cli_cluster.c, cli_internal.c, cli_list.c, cli_reload.c, cli_scram_secret.c, and cli_stop.c were re-checked and already matched the convention; no changes needed there.
segment filename math into src/bin/common/, matching pg_autoctl's own
layering
Three-part refactor, no user-visible behavior change:
1. archive-wal/restore-wal client -> src/bin/common/push_client.c/.h
(new) and fetch_client.c/.h (existing, signature updated)
- new: src/bin/common/push_client.c, src/bin/common/push_client.h
(ws_push_file_client(), plus private statics push_client_file_
crc32c()/check_file_status()/show_receivewal()/push_file())
- fetch_client.h/.c: ws_fetch_file_client() gains an explicit
applicationName parameter (no more hardcoded "fetch_client"
fallback_application_name baked into common/)
- renamed: cli_archive.c's ws_archive_run() -> ws_archive_push_file()
(now a thin wrapper calling ws_push_file_client() with
"pg_walserver_archive_wal"); cli_restore_wal.c's ws_restore_run()
-> ws_restore_fetch_file() (wraps ws_fetch_file_client() with
"pg_walserver_restore_wal")
- cli_archive.c/cli_restore_wal.c dropped their own PQconnectdbParams()/
CHECK_FILE/ARCHIVE_FILE/FETCH_FILE wire-protocol code entirely; both
now get pgsql_init()/pgsql_open_connection()'s retry/backoff for free
2. WsRetentionAge/ws_parse_retention_age() -> src/bin/common/
string_utils.h/.c
- moved out of cli_archive_cleanup.h/.c (both cli_archive_cleanup.c
and cli_basebackup.c already included string_utils.h)
- cli_archive_cleanup.c's other 5 "/* ---...--- */" dividers flattened
to plain comments in the same pass (the usage-comment divider on
archive_cleanup_command itself, shared house style with ~15 other
pg_walserver files, is left untouched)
3. WAL segment filename arithmetic/predicates -> new src/bin/common/
wal_segment.c/.h, mirroring PostgreSQL's xlog_internal.h
- new: src/bin/common/wal_segment.c, src/bin/common/wal_segment.h:
wal_segments_per_xlogid() (XLogSegmentsPerXLogId),
wal_segment_name_is_valid() (IsXLogFileName),
wal_segment_name_is_partial() (IsPartialXLogFileName),
wal_backup_history_name_is_valid() (IsBackupHistoryFileName),
wal_segment_name_format() (XLogFileName),
wal_segment_name_parse() (XLogFromFileName),
wal_segment_name_extract_prefix() (project-specific: plain/.partial/
.backup in one call), wal_lsn_to_segno() (XLByteToSeg composed with
parseLSN()), wal_lsn_to_segment_name() (composition of the two)
- deleted (replaced by the above): wal_dir_scan.c's private
is_wal_segment_filename() and its exported wal_segment_filename()/
ws_wal_segment_prefix_to_position()/ws_wal_lsn_to_segno(); cmd_base_
backup.c's private is_wal_segment_filename(); cli_archive_cleanup.c's
private is_hex_run()/is_wal_segment_name()/wal_prefix_from_name()/
lsn_to_segno()/lsn_to_segment()/segment_name_to_tli_segno()
- wal_dir_scan.h now #includes wal_segment.h; every call site in
cmd_check_file.c, cmd_replication_slot.c, cmd_start_replication.c,
cli_archive_cleanup.c, cmd_base_backup.c updated to the new names
- wal_lsn_to_segno()/wal_lsn_to_segment_name() are reimplemented on
src/bin/common/parsing.c's parseLSN() instead of a bare sscanf,
matching upstream pg_parse_lsn's stricter validation
- wal_segment_name_format() keeps this project's own pre-existing
argument order for this function (timeline, segno, segSize, dest,
destSize -- dest/destSize last), not the dest-first order the
original plan sketch suggested, since that's what the function
being replaced (wal_dir_scan.c's wal_segment_filename()) already
used and every call site already expects
Rebuilt (--no-cache, PGVERSION=17) and exercised live: archive-wal push +
restore-wal fetch round trip is byte-identical; "basebackup --keep-age
30d" and "archive-cleanup --keep-age 72h" both parse and run their
cleanup pass correctly; "pg_walserver ls"/"list wal --segments" against a
route with a real segment, a .partial segment, and a .backup label file
all classify and size correctly through the new wal_segment_name_is_
valid()/_is_partial()/wal_backup_history_name_is_valid() primitives.
make docker-indent / docker-check both pass clean.
…/ into src/bin/pg_walserver/
src/bin/common/ holds genuinely generic utilities (pgsql.c's connect/
execute-SQL/retry facility has zero knowledge of any specific command).
fetch_client.c/push_client.c are different: both hard-code knowledge of
pg_walserver's own custom wire-protocol extensions (FETCH_FILE,
CHECK_FILE, ARCHIVE_FILE) -- that's pg_walserver business logic, not a
generic reusable utility, even though it's built on top of pgsql.c's
generic connection facility underneath.
fetch_client.c's previous common/ placement predated this change and was
justified by a speculative future need ("a future, separate 'archiving'
PR is expected to grow pg_autoctl's own restore command to call this same
function directly"). Checked: nothing in pg_autoctl references either
function today. Per pg_autoctl's own convention (domain-specific protocol
clients like monitor.c live in the binary that owns the protocol, not in
common/), both files move to src/bin/pg_walserver/ as plain domain files.
If/when that future pg_autoctl PR materializes, promoting a file back to
common/ is a trivial, low-risk move.
Also: push_client.c's private push_client_file_crc32c() existed only to
duplicate ws_util.c's ws_file_crc32c() byte for byte, specifically because
common/ had to stay independent of pg_walserver-specific headers. That
constraint is gone now that the file lives in pg_walserver/, so the
duplicate is removed; push_client.c calls ws_file_crc32c() directly.
Updated every stale "src/bin/common/fetch_client.c"/"push_client.c" path
reference in comments and README.md accordingly, including rewriting the
README's "FETCH_FILE's client" section (the removed common/-placement
rationale and the speculative pg_autoctl-sharing paragraph).
src/bin/pg_walserver/Makefile's LOCAL_SRC gains both files explicitly
(sources are listed there, not globbed); src/bin/common/Makefile.common
needs no change (globs *.c automatically, so removing them from there is
transparent).
ws_file_crc32c() (ws_util.c) was pure file I/O + CRC32C math -- open a regular file, read it sequentially in chunks, compute size and CRC32C in one pass -- with no pg_walserver-specific type anywhere in it. That makes it a genuine common/ utility, unlike the FETCH_FILE/CHECK_FILE/ARCHIVE_FILE protocol clients (fetch_client.c/push_client.c) that stayed in pg_walserver/ for the opposite reason: those know about pg_walserver's own wire-protocol commands, this doesn't know about anything beyond "a file". Also found a second, independent implementation of the same thing: pg_autoctl's nodespec.c had its own nodespec_file_crc(), computing a CRC32C the same way but by reading the whole file into memory first (read_file()) rather than streaming it. Consolidated: nodespec_file_crc() is now a one-line wrapper around the new common file_crc32c(), gaining pg_walserver's own bounded-memory streaming behavior for free. New: src/bin/common/file_crc32c.c/.h, file_crc32c(path, sizeOut, crcOut). Removed: ws_util.c's ws_file_crc32c() and its declaration in ws_util.h. Updated callers: cmd_check_file.c, cmd_archive_file.c, push_client.c (pg_walserver), nodespec.c (pg_autoctl). Updated README.md's reference to the relocated function.
…ervalToString()
Both files had a byte-for-byte identical static format_uptime()
("<h>h<mm>m<ss>s"), a bespoke format invented during this same
not-yet-merged branch rather than a pre-existing contract. Replace both
copies with a thin per-file format_elapsed() that computes elapsed
seconds and calls common/string_utils.c's existing IntervalToString(),
with an explicit "-" sentinel for startedAt <= 0 (not running/unknown),
which IntervalToString itself has no notion of.
This changes the visible shape of the ps/status uptime column (e.g.
IntervalToString drops the smaller unit at larger magnitudes), which is
expected and accepted -- the old shape was never a real contract.
…irectory Fixup: the previous commit staged a stale, pre-fix copy of tar_stream.c/.h and cmd_base_backup.c (an earlier repo-wide 'git add -A' ahead of docker-indent had already staged the version before this fix, and a later path-scoped 'git add' for an unrelated commit didn't touch these paths, so the commit picked up the stale staged blob instead of the working tree's fixed content). tar_stream_directory() now takes a caller-supplied excludeNames/ excludeCount list instead of a hardcoded 'backup_manifest' literal (common/ stays free of pg_walserver-specific knowledge), and cmd_base_backup.c's call site passes 'backup_manifest' explicitly. Confirmed live: a real pg_basebackup run against pg_walserver now correctly excludes backup_manifest from the tar itself (still served via its own separate CopyOut manifest stream), where it was previously being included in the tar by mistake because this wiring was missing.
WsRetentionAge/ws_parse_retention_age and sanitize_for_log kept their pg_walserver-era names when they moved into src/bin/common/string_utils.c in earlier commits, unlike every other symbol relocated into common/ this session (wal_segment_name_format, file_crc32c, etc.), which all dropped their project prefix. Renamed to match this file's own pre-existing native convention (stringToInt64, stringToDouble, IntervalToString, IntString -- bare camelCase/PascalCase, no project prefix): WsRetentionAge -> RetentionAge ws_parse_retention_age -> stringToRetentionAge sanitize_for_log -> sanitizeForLog Updated every call site: cli_archive_cleanup.c/.h, cli_basebackup.c, auth.c, startup.h, cmd_archive_file.c, cmd_check_file.c, accept_loop.c, cmd_fetch_file.c. cli_archive_cleanup.c's own static ws_retention_age_ cutoff() is untouched -- it's a genuinely pg_walserver-specific helper, not one of the renamed symbols.
A previous session in this branch mistakenly introduced a documentation
convention that doesn't match this project's established style: a short
pointer comment in the .c file ("/* fn -- see some_header.h. */") with
the full prose documentation left in the .h file. The real convention,
confirmed by src/bin/common/pgsql.c/pgsql.h and
src/bin/pg_autoctl/monitor.c/monitor.h, is: full function documentation
lives in the .c file directly above the function body; .h files carry
bare prototypes (no comment) for functions, and keep documentation only
for types/macros/structs.
This commit relocates the full doc comments back to the .c files and
leaves bare declarations in the .h files, across:
- src/bin/pg_walserver/receivewal.c/.h
- src/bin/pg_walserver/cli_basebackup.c/.h
- src/bin/pg_walserver/backup_bootstrap.c/.h
- src/bin/pg_walserver/accept_loop.c/.h
- src/bin/pg_walserver/cli_common.c/.h
- src/bin/pg_walserver/wal_dir_scan.c/.h
- src/bin/common/openssl_utils.c/.h
- src/bin/common/tar_stream.c/.h
- src/bin/common/string_utils.c/.h
- src/bin/common/process_supervisor.c/.h
- src/bin/common/wal_segment.c/.h
Beyond the short-pointer occurrences, a spot-check of these headers also
found function doc comments with no short-pointer counterpart at all
(cli_basebackup_run, ws_accept_loop's mutable-config paragraph,
wal_dir_find_latest, wal_dir_has_any_segment, wal_position_cache_read,
ws_receivewal_progress_write/_read, sanitizeForLog, and the
process_restart_counters_*() trio sharing one combined comment) -- these
were relocated the same way. ws_accept_loop's .c comment and its .h
mutable-config paragraph were merged rather than one replacing the
other, since both described genuine, non-overlapping behavior. The
process_restart_counters_*() shared comment was split into one doc per
function, each now living above its own definition.
No logic/behavior change -- pure comment relocation.
Catches the stragglers from the previous pass that used '-- see X.h's own comment' phrasing instead of '-- see X.h.': the full prose doc still lived in the .h while the .c only had a bare pointer back to it. Pure relocation, no behavior change. - src/bin/common/file_utils.c/.h: read_file_capped, open_regular_file, read_file_flags - src/bin/common/system_utils.c/.h: monotonic_ms - src/bin/pg_walserver/cli_ls.c/.h: cli_ls_run - src/bin/pg_walserver/cli_ps.c/.h: cli_ps_run - src/bin/pg_walserver/cli_setup.c/.h: cli_setup_run - src/bin/pg_walserver/cli_status.c/.h: cli_status_run - src/bin/pg_walserver/cli_upstream.c/.h: cli_resolve_upstream - src/bin/pg_walserver/hba.c/.h: hba_ruleset_requires_client_cert - src/bin/pg_walserver/cli_cluster.c/.h: ws_cluster_register_run - src/bin/pg_walserver/routes.c/.h: routes_slot_name, config_load_global, config_set_global_property, config_file_path (WsGlobalConfig's own type doc stays in routes.h, untouched) - src/bin/pg_walserver/tls.c/.h: ws_tls_server_load_ca, ws_tls_get_peer_cert_cn
Same anti-pattern as 18f445f/7075a8f, missed by both of those sweeps: this one had no pointer comment at all in the .c (not even a short "-- see X.h" stub), just a bare function body, with the full doc sitting only in file_crc32c.h -- so neither sweep's text-pattern search (which looked for a pointer phrase IN the .c file) ever matched it.
Found via a clang-19 AST-based sweep (-Xclang -ast-dump, matching every function declared in a file's own .h against its definition in the .c, per this project's file.c/file.h pairing convention) rather than the text-pattern grepping the two prior rounds (18f445f, 7075a8f) used -- those could only catch cases with some pointer phrase left in the .c; these 11 had none at all (a fully bare definition, full doc sitting only in the .h), the exact same silent-miss shape caught in file_crc32c.c (23d4e30) but never checked for tree-wide until now. Fixed: scram_build_verifier/scram_server_first (scram.c/.h), ws_authenticate (auth.c/.h), ws_replication_slot_exists/_read_restart_lsn/ _update_restart_lsn/_oldest_restart_lsn (cmd_replication_slot.c/.h), ws_fetch_file_client (fetch_client.c/.h), hba_parse_file (hba.c/.h), ws_ps_state_path/_write/_read (ps_state.c/.h). Swept all 68 hand-written .c files in src/bin/common/ and src/bin/pg_walserver/ (excluding bison/flex-generated and vendored sources); re-ran after these fixes and confirmed zero remaining cases.
…code routes.c/routes.h become clusters.c/clusters.h, and every identifier built on "route" follows: WsRoute -> WsCluster, WS_ROUTES_WILDCARD_KEY -> WS_CLUSTERS_WILDCARD_KEY, routes_load/routes_find/routes_find_exact/ routes_find_by_hostname/routes_slot_name/routes_persist_path/ routes_set_property/routes_drop_section -> the clusters_* equivalents, and every local variable, parameter, and log/CLI-help string that said "route" for this concept now says "cluster". hba.c/hba.h's HBA column and matching field (generated/parsed pg_walserver_hba.conf format) move from ROUTE/routeKey to CLUSTER/clusterKey; the test .pgaf specs under tests/tap/specs/ that assert on HBA content or log wording are updated to match. The separate, unrelated IP-routing/CIDR-auto-discovery code (ipaddr.c/.h's fetchLocalIPAddressForRouting(), hba.c's ws_setup_autodetect_cidr()/ ws_setup_local_ip_via_interfaces() and their own "kernel routing table" comments) is untouched -- that's a genuine, different "route" meaning (literal OS IP routing for the HBA CIDR setup wizard), not this concept. "route"/"routing" referring to this concept is also gone from source comments and user-visible strings (CLI help text, log messages): the dbname/SNI dispatch mechanism is now described as "addressing" a cluster, not "routing" to a route, to avoid the same ambiguity in prose. The CLI's own "cluster" vocabulary (cluster register/drop/list/ ...) was already in place and is unchanged. pgaftest's test_spec_parse.c/.h are regenerated from test_spec_parse.y via bison to match the two comment-only changes in the grammar source.
Sweeps docs/ref/*.rst and src/bin/pg_walserver/README.md to match the
code rename in the previous commit: every "route"/"Route"/"ROUTE"
referring to the pg_walserver.ini cluster concept becomes "cluster"/
"Cluster"/"CLUSTER", including section headings ("The routes file"
-> "The clusters file", "Route keys are opaque strings, not paths" ->
"Cluster keys...", "The wildcard route (*)" -> "The wildcard cluster
(*)") and the pg_walserver.rst/pg_walserver_serve.rst "Routing" /
"Routing more than one cluster by name: TLS SNI" sections, renamed to
"Addressing a cluster" / "Addressing more than one cluster by name:
TLS SNI" with every cross-reference updated to match. pg_walserver_
cluster.rst's "the clusters (routes) one pg_walserver..." drops the
now-redundant "(routes)" parenthetical.
Every literal captured log/HBA example shown in these docs is updated
to the wording the actually-renamed code now prints (verified against
the source, not guessed); none of this was re-captured live in this
pass, so it's a careful hand-edit rather than a fresh capture.
Left untouched: prose about the unrelated IP-routing/CIDR-auto-discovery
feature, and the literal test spec filename tests/tap/specs/pg_walserver_
sni_routing.pgaf referenced from a README.md heading.
Give pg_autoctl a genuinely new node kind that supervises `pg_walserver serve` as a plain child process: no monitor registration, no keeper FSM participation, just "run this binary" -- the same shape service_postgres_ctl.c already uses to run Postgres, applied to pg_walserver instead. - src/bin/common/pgsetup.h: new PgInstanceKind value NODE_KIND_WALSERVER. - src/bin/pg_autoctl/config.h, config.c: new PG_AUTOCTL_ROLE_WALSERVER / WALSERVER_ROLE = "walserver"; ProbeConfigurationFileRole() recognises a walserver node's own config file from its [pg_autoctl] role=walserver line, same convention as role=monitor/role=keeper. - src/bin/pg_autoctl/service_walserver.h/.c (new): WalServerConfig (the minimal, non-Postgres equivalent of KeeperConfig: pgdata, port, name), walserver_config_write()/read(), service_walserver_ctl_start() (fork + execvp "pg_walserver serve --pgdata ... [--port ...]", modelled on service_postgres_ctl_start()), and start_walserver() (registers that one service with the supervisor, no keeper-init/activation dance). - src/bin/pg_autoctl/cli_create_node.c: new "pg_autoctl create walserver --pgdata --port --name --run" command, writing the minimal config file above and, with --run, starting the service directly. - src/bin/pg_autoctl/cli_service.c: "pg_autoctl run"'s role dispatch gets a PG_AUTOCTL_ROLE_WALSERVER case, calling the new cli_walserver_run(). - src/bin/pg_autoctl/cli_root.c, cli_common.h, supervisor.h: register the new create command and its SERVICE_NAME_WALSERVER. - src/bin/pg_autoctl/nodespec.c: nodespec_read()/write() know the "walserver" node.ini kind string; nodespec_create_argv() builds the much simpler "pg_autoctl create walserver --pgdata ... [--port ...] [--run]" argv for it, skipping all the monitor/formation/SSL/auth machinery that doesn't apply. create/launch-deferred support comes for free from the existing, kind-agnostic gate in cli_node.c. - src/bin/pg_autoctl/cli_show.c: "show file --contents" gets an honest "not supported for a walserver node" case (the switch over pgAutoCtlNodeRole is exhaustive, so this was a build error, not just a style choice -- every other ProbeConfigurationFileRole() call site already had a default: case that cleanly errors out, so those were left untouched). Verified by an actual run inside the build-stage container: `pg_autoctl create walserver --pgdata ... --port ... --run` and the two-step `create walserver` + `run --pgdata ...` both really start and supervise a listening pg_walserver serve child process.
…/launch deferred
`pg_walserver <name>` DSL sugar (test_spec_parse.y's pg_walserver_line)
used to exec "pg_walserver serve" directly as the container's own PID 1,
with zero pg_autoctl involvement. Default that through the new
`pg_autoctl create walserver` / `pg_autoctl node run` path instead,
giving it pg_autoctl's own create/launch-deferred support and process
supervision for free -- while preserving the one thing the old default
did that pg_autoctl itself has no reason to know about: seeding a real,
usable default pg_walserver_hba.conf into the node's pgdata before
"serve" ever starts (otherwise every connection is rejected until
something else writes one).
- test_spec_parse.y: pg_walserver_line now sets current_node->kind =
NODE_KIND_WALSERVER (compose_gen_write_node_ini() then emits "kind =
walserver" into the node's own node.ini), and its default
commandOverride changes from "exec pg_walserver serve ..." to
"mkdir -p <pgdata> && cp <bind-mounted default hba> <pgdata>/
pg_walserver_hba.conf [|| true] && exec pg_autoctl node run <ini>" --
same bind-mount-then-copy pattern already used for SSL certs, same
reason (a bind-mounted file's auto-created parent directory would
otherwise be root:root and un-writable by the container's own user for
everything else pg_walserver needs to create under pgdata). The
explicit "command \"...\"" override escape hatch is untouched --
a spec that still wants the old "bypass pg_autoctl entirely" behavior
keeps working exactly as before.
- test_spec_parse.y: aux_opt_list (shared by postgres_line and
pg_walserver_line) now also accepts the same deferred/launch/create
tokens node_opt_list already has ("deferred", "launch deferred",
"create deferred", "create and launch deferred", "launch
immediate"). Verified safe by rebuilding the generated parser with
`bison -Wall`: zero new conflicts, zero new warnings, before and
after this change (only pre-existing %empty style warnings, unrelated).
test_spec_parse.c/test_spec_scan.c regenerated via `make generate`.
- compose_gen.c: compose_gen_write_node_ini()'s kind switch gets a
"walserver" case; write_pg_walserver_default_hba()'s header comment
updated to describe the new target path and command.
Verified via `pgaftest cluster prepare <spec> <dir>` (writes the real
generated docker-compose.yml/node.ini/hba files without starting
containers): a bare "pg_walserver ws1 deferred" node gets kind =
walserver, [launch] create/run = deferred, a generated default HBA file,
and a command that mkdir/cp's it into place before "pg_autoctl node run".
…o 6543 aux_opt_list (the small, closed modifier set pg_walserver_line/ postgres_line accept after the node name) had no way to set a custom port for a `pg_walserver <name>` node: node->pgPort was only reachable from ordinary node_opt_list's own "T_PORT T_INTEGER" rule, not from aux_opt_list. compose_gen_write_node_ini()'s "node->pgPort > 0 ? node->pgPort : 5432" then silently defaulted every unconfigured walserver node to Postgres's own 5432, which makes no sense for pg_walserver (this project's own convention elsewhere is 6543). - test_spec_parse.y: add "port N" to aux_opt, reusing the exact same rule body ordinary node_opt's own T_PORT T_INTEGER rule already has. Verified with `bison -Wall -d`: identical warning set before/after (the pre-existing %empty-rule warnings only), zero new shift/reduce or reduce/reduce conflicts introduced -- T_PORT never starts a bare top-level cluster_item, so it is unambiguous here the same way T_DEFERRED/T_LAUNCH/T_CREATE/T_IMMEDIATE already are. - compose_gen.c: compose_gen_write_node_ini()'s default now falls back to 6543 for NODE_KIND_WALSERVER nodes specifically (only when the spec didn't set an explicit "port N"); every other node kind keeps the 5432 default unchanged. - test_spec_parse.c/.h regenerated via `make generate` (bison/flex), same convention as commit 920fade.
Convert the primary `pg_walserver <name>` node in each of the 4 specs
that still used the old "command \"tail -f /dev/null\", exec
`pg_walserver cluster register`/`serve` directly by hand" pattern over
to the new mechanism: `port <n>` (previous commit) + `launch
deferred`, with setup{}/test steps doing their cluster-register/
certificate/HBA provisioning against the standard NODE_PGDATA
(/var/lib/postgres/pgaf) and then releasing the node with
`exec <node> pg_autoctl node start` (the same precedent every other
launch-deferred node in this suite already uses, e.g.
launch_deferred_set_metadata.pgaf) instead of a bare
`nohup pg_walserver serve ... &`. Every spec's own assertions are
otherwise unchanged -- only how/where `pg_walserver serve` is started
and configured changes, not what each test actually verifies.
pg_walserver_list_ps_status.pgaf:
- node "walserver": command override -> port 6543 + launch deferred.
- setup{}: drop the hand-written pg_walserver_hba.conf (now the DSL's
own auto-generated default); cluster register moved from /tmp/lps
to /var/lib/postgres/pgaf.
- test_002: "pg_autoctl node start" releases serve (was
nohup+redirect-to-logfile); the "both clusters got their bootstrap
backup" wait now polls `pg_walserver list backups` instead of
grepping a serve log file, because serve's stdout/stderr is now
inherited from pg_autoctl (the container's own stdout), not a file
an exec step can read.
pg_walserver_pitr_and_secondary.pgaf:
- node "server": command override -> port 6543 + launch deferred.
- setup{}: drop the hand-written pg_walserver_hba.conf; cluster
register and all later path references moved from /tmp/ws to
/var/lib/postgres/pgaf; "pg_autoctl node start" releases serve
(was nohup+redirect-to-logfile) at the same point in setup{} the
old nohup was.
pg_walserver_clientcert.pgaf:
- node "walserver": command override -> port 6546 + launch deferred.
- setup{}: CA/cert generation, create-cert, cluster ini, scram
secret and the spec's own custom clientcert=verify-full HBA all
moved from /tmp/cc to /var/lib/postgres/pgaf (the custom HBA still
overwrites the DSL's auto-generated default before "serve" ever
starts, same timing as before).
- test_000: "pg_autoctl node start" releases the primary instance
(was nohup+redirect-to-logfile), AFTER the negative-test assertion
above it. The negative-test instance itself (a separate, isolated
/tmp/ccnoca scratch dir, port 6599, proving "clientcert=verify-full
with no usable TLS CA fails closed at startup") is untouched: still
a one-off, directly-exec'd `pg_walserver serve`, unrelated to and
unaffected by the primary node's own conversion.
pg_walserver_sni_routing.pgaf:
- node "walserver": command override -> port 6544 + launch deferred;
"alias" kept as-is.
- setup{}: drop the hand-written pg_walserver_hba.conf (the compose-
subnet-wide default already admits every peer this spec needs);
cluster register and all later path references moved from
/tmp/sni to /var/lib/postgres/pgaf.
- test_001: "pg_autoctl node start" releases serve (was
nohup+redirect-to-logfile).
- test_004 (negative test: "more than one named cluster without TLS
refuses to start"): rewritten. The old version stopped the live
instance with `pkill -x pg_walserver`, moved its certs away, tried
a one-off `pg_walserver serve`, then restored the certs -- but the
primary node's `pg_walserver serve` is now pg_autoctl-supervised
(RP_PERMANENT, service_walserver.c's start_walserver()), so
pkill-ing it would just make pg_autoctl respawn it immediately,
defeating the test. Instead this now takes a throwaway copy of the
same pgdata (same two already-registered named clusters,
server.crt/key and any pid/state file removed from the copy only)
and runs the one-off `pg_walserver serve` against that copy on its
own port -- the same "bypass pg_autoctl, exec directly" shape, now
without ever touching the still-running primary instance. (cli_
serve.c's own startup validation rejects the missing-TLS case
before ever writing/checking a pidfile, so no further cleanup of
the copy is needed.) This is a deliberate adaptation, not a literal
"leave the negative test untouched" -- see this task's final
report for the full reasoning.
…ferred modifiers pg_walserver <name> now defaults to going through pg_autoctl create walserver / pg_autoctl node run (commit 920fade), not a direct PID-1 exec of pg_walserver serve -- the doc still described the old behavior. Also documented two pre-existing gaps in the node-modifiers table that were never written up at all, caught while touching this section: the bare "deferred"/"create deferred"/"create and launch deferred"/"launch immediate" forms (only "launch deferred" had a row), and "port <N>" itself (existed in the grammar for ordinary nodes well before this session, just never documented, and now also meaningful for a pg_walserver <name> node with its own correct default of 6543 instead of 5432).
…el/WAL-continuity helpers into common/, extract backup_list.c Four related reorganizations to cli_archive_cleanup.c, zero behavior change: 1. The boxed, dash-divider usage comment above archive_cleanup_command is flattened to a plain comment, matching this file's own convention everywhere else (and out of scope: every other cli_*.c file's dash-divider header deliberately stays as-is). 2. ws_retention_age_cutoff() moves next to stringToRetentionAge() in src/bin/common/string_utils.c/.h, renamed retentionAgeCutoff() to match that file's own camelCase convention (stringToInt64, IntervalToString, stringToRetentionAge). 3. parse_backup_label_time() -- fully generic, no pg_walserver types in its signature -- moves to src/bin/common/wal_segment.c/.h, renamed wal_backup_label_parse_time() to match that file's own wal_<noun>_<verb> convention. 4. WsBackupInfo/ws_backup_list_load()/backup_cmp() move out of cli_archive_cleanup.c entirely into a new, pg_walserver-specific src/bin/pg_walserver/backup_list.c/.h pair (added to the Makefile's LOCAL_SRC). cli_list.c and cli_ls.c, which both used WsBackupInfo/ ws_backup_list_load() via cli_archive_cleanup.h purely as a side effect, now include backup_list.h directly instead -- the same "one cli_*.c depending on another cli_*.c's header for a shared utility" smell already fixed once this session for RetentionAge/ cli_basebackup.c. Also moves the WAL-continuity pre-flight check's generic core to src/bin/common/wal_segment.c/.h: - read_last_history_line() -> wal_history_read_last_line(): fully generic, no pg_walserver types. - check_wal_range() -> wal_check_range(), and its WsContinuityProblem parameter type -> WalContinuityProblem (dropping the "Ws" prefix, same as every other common/ type, e.g. IntString, RetentionAge): fully generic already, just uses the renamed struct. - ws_check_wal_continuity() is NOT generic as written: it takes WsCluster/WsBackupInfo (pg_walserver types) directly, purely to call wal_dir_find_latest() for the newest kept backup's own end boundary. Split it: a new generic wal_check_continuity() in wal_segment.c takes plain scalars (segSize, a WalContinuityEntry[] of already-resolved per-backup start tli/segno pairs, and the already-resolved "latest on-disk" tli/segno as plain parameters) and does the logging-free continuity-checking loop, returning one WalContinuityProblem per entry. ws_check_wal_continuity() in cli_archive_cleanup.c stays as a thin pg_walserver-side wrapper: it resolves WsCluster/WsBackupInfo into those plain entries (via wal_dir_find_latest(), wal_dir_scan.c), calls wal_check_continuity(), then does all the cluster/backup-specific log_error() reporting, reproducing the exact same two message templates the original function logged (the "no WAL at all to compare against" case is distinguished from a genuine wal_check_range() gap purely by position -- only the newest entry, when there was nothing on disk to anchor it, can hit that case -- so the wrapper picks the matching template without needing the common function to know about either one by name). src/bin/common/wal_segment.c/.h and string_utils.c/.h end up with zero WsCluster/WsBackupInfo/WsRoute references (verified via grep). Every new/moved function has a full header comment in its own .c file, bare declaration in its own .h, matching this session's established convention.
…amed phase helpers ws_archive_cleanup_run() did five distinct things in one function (validate inputs + load the backup list + resolve ".latest"; decide the kept/removed set per backup via --keep-count/--keep-age plus the replication-slot floor; the WAL-continuity pre-flight check and whether to proceed; actually remove non-kept backups; actually remove expired WAL/.partial/.backup files) and, under its old name, read almost identically to this file's own CLI dispatcher cli_archive_cleanup_command_run() right below it -- the same "_run" naming collision already fixed once this session for ws_archive_run -> ws_archive_push_file. Split into five static phase helpers, each named for what it does: - resolve_latest_backup(): looks up and validates "basebackups/.latest" - decide_kept_backups(): --keep-count/--keep-age math plus the replication-slot floor, logging which cutoff rule(s) applied - run_continuity_preflight(): runs ws_check_wal_continuity() and decides whether the deletion phases may proceed - remove_expired_backups(): the backup-directory removal loop - remove_expired_wal_files(): the WAL/.partial/.backup removal loop The renamed top-level function, ws_archive_cleanup_execute(), does nothing but call these five in order and return the final result -- same computation, same log messages, same return value as before, purely reorganized. Its one external call site (cli_basebackup.c's cli_basebackup_command_run()) and its own cli_archive_cleanup_command_run() call site are updated to match, as is cli_archive_cleanup.h's declaration and cli_basebackup.h's comment mentioning it by name.
Same stylistic fix already applied to cli_archive_cleanup.c this session: replace the ASCII-art '/* ----...---- */' divider above each command's CommandLine/usage synopsis with a plain '/* ... */' comment carrying the same content, no box. Applies to cli_archive.c, cli_create_cert.c, cli_fetch_systemid.c, cli_ps.c, cli_reload.c, cli_restore_wal.c, cli_scram_secret.c, cli_serve.c, and cli_stop.c (each had exactly one). cli_root.c had two: the one above cli_help_run/help_command, and the one labeling the 'Root command table' section before root_subcommands[] -- both flattened the same way, for consistency across the file set. cli_cluster.c's own stray divider, above ws_cluster_register_run, was mislabeled as covering 'register | drop | list | set-upstream' even though the file defines six commands (its label did not even mention enable/prune) -- flattened into an accurate, register-specific plain comment instead; each command's own run function already has its own doc comment below it from this session's earlier doc-comment-convention sweep, so nothing needed duplicating. No behavior change.
…to ws_ cli_basebackup_run, cli_fetch_systemid_run, cli_setup_run, cli_status_run, cli_ls_run and cli_ps_run each collided, by name, with their own file's static <command>_command_run dispatcher, and (the first two) are also called directly from cli_cluster.c as real cross-file APIs, not just their own file's dispatcher -- exactly the case this project's 'cli_-prefix = thin CLI wrapper only, ws_-prefix = the real, reusable implementation' convention exists for (already applied this session to ws_archive_push_file, ws_restore_fetch_file, ws_archive_cleanup_execute). Renamed: cli_basebackup_run -> ws_basebackup_execute cli_fetch_systemid_run -> ws_fetch_systemid_execute cli_setup_run -> ws_setup_execute cli_status_run -> ws_status_report cli_ls_run -> ws_ls_report cli_ps_run -> ws_ps_report Updated every definition, its .h declaration, and every call site, including cli_cluster.c's own calls to ws_basebackup_execute()/ ws_fetch_systemid_execute() and the matching mention in README.md. cli_cluster.c's own six ws_cluster_*_run functions and cli_create_cert.c's ws_create_cert_run are untouched: already differentiated by the ws_/cli_ prefix split, not real collisions. No behavior change.
…se helpers
Both functions had grown to do several, clearly separable things in one
body. Pure extraction, same logic, same order, same error handling --
each helper below is called exactly once, in the original order.
cli_serve.c: cli_serve_run (~283 lines) split into:
cli_serve_check_pgdata_or_insecure -- --pgdata/--insecure guard
cli_serve_load_global_defaults -- config paths + persisted
port/auth-timeout/ssl-*-file
defaults
cli_serve_setup_tls -- cert/key + optional CA loading
cli_serve_load_clusters_and_hba -- HBA default-creation + parsing
pg_walserver.ini and the HBA
file
cli_serve_check_sni_requirements -- clientcert/named-cluster-vs-TLS
sanity checks
cli_serve_start_receivewal_children -- start "receivewal=pull" children
cli_serve_start_backup_bootstraps -- kick off missing-backup bootstraps
cli_serve_write_pidfile_or_die -- write the pidfile
cli_serve_run itself now only orchestrates these in order, then seeds the
SCRAM mock secret, calls ws_accept_loop(), and cleans up the pidfile.
cli_setup.c: ws_setup_execute (~179 lines, after the earlier rename)
split into:
cli_setup_validate_pgdata -- pgdata validation/creation
cli_setup_persist_global_config -- persist port/ssl-*/auth-timeout
global config properties
cli_setup_create_cert_if_missing -- auto-create a self-signed cert
cli_setup_create_hba_if_missing -- auto-create the default HBA file
(with CIDR autodetection)
ws_setup_execute itself now only orchestrates these in order.
No behavior change.
…tils.c cli_ps.c and cli_status.c each had a byte-for-byte identical static format_elapsed() (only a one-clause doc-comment difference) wrapping IntervalToString() (common/string_utils.c) with a "-" sentinel for an unset/zero timestamp. Moved to src/bin/common/system_utils.c/.h as format_elapsed_time(), a bare name matching that file's own existing convention (monotonic_ms, pretty_print_bytes, get_system_info -- no ws_/ cli_ prefix). cli_ps.c and cli_status.c now both call it and no longer keep their own static copy. No behavior change.
…ader CLI sweep - de-duplicate format_utc/directory_size (byte-identical between cli_ls.c and cli_list.c) into common/system_utils.c and common/file_utils.c, removing both now-redundant static copies - fix cli_list.c's own 3 "_run" collisions with its "_command_run" dispatchers: cli_list_clusters_run/cli_list_backups_run/cli_list_wal_run -> ws_list_clusters_report/ws_list_backups_report/ws_list_wal_report - flatten cli_list.c's remaining 6 dash-divider header comments to plain prose, matching the rest of this round's cli_*.c files - flatten the 4 dash-dividers my own earlier task-scoping for the broader sweep agent missed: cli_status.c, cli_basebackup.c, cli_setup.c, cli_ls.c - fix 2 stale ws_basebackup_execute() comment mentions in cli_archive_cleanup.c left over from the cli_basebackup_run rename Verified: make docker-indent/docker-check clean, a --no-cache Docker build with zero new warnings, and a live smoke test of pg_walserver ls/list clusters/list backups/list wal against a synthetic pgdata fixture confirming identical output after the relocation.
walserver_config_write() used a raw fopen(path, "w")/fclose() pair (IGNORE-BANNED) instead of this project's own facilities, unlike every other *_config_write_file() (keeper_config.c, monitor_config.c). Fixed to match: fopen_with_umask() + a temp-file + rename, so a concurrent reader always sees either the old complete file or the new complete one, never a partial write. Swept the rest of pg_walserver/pg_autoctl/common for the same pattern and fixed every other instance found: - cmd_base_backup.c (read_backup_label, send_manifest), cmd_start_ replication.c (segment file open), tar_stream.c (emit_file_contents): raw fopen(path, "r"/"rb") -> fopen_read_only(), the existing designated wrapper (file_utils.c) for exactly this case. - nodespec.c's nodespec_write_to_path(): raw fopen(path, "w")/fclose() -> fopen_with_umask()/fclose()-with-error-check, matching every other config writer in pg_autoctl. - pghba.c's ident-map append: raw fopen(path, "a")/fformat()/fclose() (no error check on fclose at all) -> the existing append_to_file() facility (file_utils.c), which already does this exact thing. Left alone (not instances of this pattern): cmd_replication_slot.c's open()+flock() advisory lock (no existing facility provides flock semantics; a genuinely distinct, lower-level need) and every other open()/read()/write() use project-wide, which this project already prefers over stdio for plain fd-based I/O -- only fopen()/fclose() pairs that duplicate an existing file_utils.h wrapper were in scope. Pre-existing fopen() use in pgaftest/parson (vendored) is untouched: predates this PR, out of scope. Verified: make docker-indent/docker-check clean, a --no-cache Docker build with zero new warnings, and a live smoke test of "pg_autoctl create walserver" confirming the new config write leaves no stray .tmp file and produces identical ini content.
walserver_config_write() and nodespec_write_to_path() built their output by issuing one fformat()/appendPQExpBuffer() call per line directly against an open stream, then (after the previous commit's fix) closed it and renamed a temp file into place by hand. Switch both to the pattern pg_walserver's own clusters.c/hba.c already use throughout: build the whole file in memory with a PQExpBuffer, then hand it to write_file_atomic() (common/file_utils.c) as a single write -- which already does the temp-file + fsync + rename + directory fsync dance, strictly stronger than the hand-rolled temp+rename this file had (no fsync at all) before this change. nodespec.c: extracted nodespec_build(spec, PQExpBuffer) as the one shared serializer behind both nodespec_write(spec, FILE *out) (still takes a FILE* for its stdout-display caller, "pg_autoctl node show", but now does one fwrite() of the finished buffer instead of ~15 separate fformat() calls against the stream) and nodespec_write_to_path(), so the two can never drift against each other. Verified: make docker-indent/docker-check clean, a --no-cache Docker build with zero new warnings, and live smoke tests of "pg_autoctl create walserver" (walserver_config_write) and "pg_autoctl node start" (nodespec_write_to_path, clearing the [launch] deferred flag) -- both confirmed to write correct content with no stray .tmp file left behind.
pghba_ensure_host_rule_exists()/pghba_ensure_host_rules_exist() already built the new pg_hba.conf contents in memory with a PQExpBuffer, but wrote them out with the plain write_file() (no fsync, no temp-file rename -- a crash mid-write can leave a truncated pg_hba.conf behind). Switch both to write_file_atomic(), matching the pattern this same pass just applied to walserver_config_write()/nodespec_write_to_path(), and that pg_walserver's own clusters.c/hba.c already use throughout. Verified: make docker-indent/docker-check clean, a --no-cache Docker build with zero new warnings, and a live "pg_autoctl create monitor" run (exercises pghba_ensure_host_rules_exist() for real, granting the autoctl_node replication HBA rule) confirming the expected "# Auto-generated by pg_auto_failover" rule lands in pg_hba.conf with no stray .tmp file left behind.
dimitri
force-pushed
the
pg_walsender-server
branch
from
October 14, 2026 21:52
81f6f17 to
fb117fd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pg_walserver(renamed frompg_walsenderpartway through review — see "Naming" below): a standalone binary speaking the real PostgreSQL replication wire protocol, hardened like a real network-facing server rather than a manual-testing toy, that can servepg_basebackup,pg_receivewaland a real standby's own walreceiver directly out of a directory tree of WAL segments and base backups it owns — no livepostmasterbehind it, nopg_auto_failovermonitor or archiver integration in this PR at all. It is a standalone, independently useful building block (works outsidepg_auto_failoverentirely — seedocs/ref/pg_walserver.rst's full worked example with plain PostgreSQL core tooling), and the intended foundation the follow-up archiving-integration PR builds on, not a replacement for it.What it serves
On the wire:
IDENTIFY_SYSTEM,SHOW,BASE_BACKUP,TIMELINE_HISTORY,CREATE/READ/DROP_REPLICATION_SLOT,START_REPLICATION— exactly as a real primary — plus three project-specific extensions:FETCH_FILE(arestore_command-friendly one-shot file fetch), andCHECK_FILE/ARCHIVE_FILE(the push-side counterpart, idempotent and overwrite-safe, used by thearchive-walclient below).Sub-commands
serve— the accept loop (default command;pg_walserver [options]with no sub-command name works too).scram-secret— print anarchiver-passwdline for a role.setup— write/validate apg_walserver.iniroute (--route/--path/--upstream/--hostname), fetch the upstream system identifier, optionally take the first base backup synchronously (--with-basebackup), and enable the embedded WAL capturer by default (--no-captureto opt out).fetch-systemid/basebackup— the two stepssetupcalls internally, also usable standalone.create-cert— create (or--force-rotate) a self-signed TLS certificate for--pgdata;setupcalls this automatically the moment a second named route needs TLS for SNI-based routing.archive-wal %p %f/restore-wal %f %p— drop-inarchive_command/restore_commandclients, usingCHECK_FILE/ARCHIVE_FILEandFETCH_FILErespectively (named-walrather than a barearchive/restore, matching pgBackRest'sarchive-push/archive-getand WAL-G'swal-push/wal-fetchconvention, sincebasebackupalready exists in this same CLI).internal service pg-receivewal— hidden,pg_autoctl-style re-exec entry point the embedded capturer uses (see below).Routing:
dbname, TLS SNI, and a PgBouncer-style wildcardA route key is an opaque string (pg_auto_failover's own
<formation>/<group>convention is one choice among many, never given filesystem meaning), matched first by exactdbname, then — for a real physical standby, whose walreceiver always sends the literaldbname=replicationregardless ofprimary_conninfo(libpqrcv_connect()'s own unconditional override) — by TLS SNI hostname, then by a"*"wildcard route. More than one named route requires TLS;pg_walserverrefuses to start otherwise. Seedocs/ref/pg_walserver.rst's "Routing more than one cluster by name: TLS SNI" section for the full mechanism and its DNS/CNAME prerequisite.The embedded pull capturer
A route (on by default via
setup,capture = pullinpg_walserver.ini) gets its own long-lived, supervised child running the vendoredpg_receivewal(src/bin/common/vendor/pg_receivewal/, shared withpg_autoctl) againstupstream, captured directly into the route's own directory — no externalpg_receivewalprocess required. Supervision reuses this codebase's own pattern for long-lived services rather than a bespoke loop:fork()+execv()ofpg_walserveritself, re-entering as the hiddeninternal service pg-receivewalsub-command, matchingpg_autoctl's owninternal service postgres|listener|node-activeshape (and its live-upgrade-under-PID-1 rationale).archive-walremains a fully independent, monitor-agnostic defense-in-depth backstop on top of this, not a replacement for it.Shared infrastructure
src/bin/common/process_supervisor.c/.h: the generic Erlang-style MaxR/MaxT restart-backoff ring buffer and PID-1-safe orphan-reap classification, now genuinely shared —pg_autoctl/supervisor.c's own internals call into it too (previously a symbol-renamed parallel implementation; consolidated in this PR, kept as an isolated first commit so it can be split into its own preliminary PR the same waysrc/bin/common/vendor/pg_receivewal/'s relocation was in pg_autoctl: relocate vendored pg_receivewal to src/bin/common #1194).src/bin/common/fetch_client.c/.h: theFETCH_FILEclient, used in-process byrestore-wal(and, later, bypg_autoctl restore_command.conce the archiving-integration PR adds it).archiver-hba.conf,host/hostssl/hostnossl, CIDR/hostname/samehost/samenet, trust/scram-sha-256/reject), SCRAM-SHA-256 auth, TLS, absolute per-connection auth deadlines, and a per-route replication-slot cap.pgaftest(this project's own docker-compose-driven test DSL) gainedpostgres <name>/pg_walserver <name>cluster-level node sugar (a bare, unmanaged Postgres orpg_walserverbox, deliberately distinct frompg_auto_failover's own futurearchivernode kind — see the sibling archiving-integration branch), plus a per-nodecommand "..."override and hostnamealiassupport (for TLS SNI testing) andinit: truesupport.Naming
Renamed
pg_walsender→pg_walservermid-PR once the scope grew to include receiving (the embedded capturer,ARCHIVE_FILE) alongside sending — "sender" undersold half the job. Seesrc/bin/pg_walserver/README.md's own "Naming" section for the full survey against PostgreSQL's own glossary and external tools (pgBackRest, Barman, WAL-G).Tests
Five
tests/tap/specs/pg_walserver_*.pgafspecs, 22 steps total:standalone(7 — setup wizard,IDENTIFY_SYSTEM/SHOW/FETCH_FILE, realpg_receivewalSTART_REPLICATION, grammar edge cases, wildcard routing, a realpg_basebackup+ real standby promoted via core tooling alone),sni_routing(4 — two real routes disambiguated purely by TLS SNI, auto-cert-creation, the no-TLS-with-2-routes hard failure),archive_command(5 —CHECK_FILEmissing/matches/differs, overwrite-safety,create-cert,restore-wal),capture(3 — the embedded capturer runs, restarts on kill, stops cleanly with the server),pid1(3 — the same capturer as a container's real PID 1, orphan-reaped, orderly-shutdown-not-SIGKILL).citus_indent/ci/banned.h.shclean; PG17 build clean, zero warnings.What's still out of scope for this PR
No
pg_auto_failoverarchiver node kind, no monitor schema, nopg_autoctl restore command/archive commandwith quorum/full-archiver-node participation — all of that is the follow-up archiving-integration PR's job, building on top of what's here.