Repository navigation
Conversation
…or API) Adds the SQL-only foundation for the Archiver process identity / ARCHIVING node membership / base-backup policy / PITR schema described in the archiving-disaster-recovery design doc, milestone 1 (schema + monitor API only -- no service_archiver process involved yet, everything is exercised via direct SQL calls against a plain cluster). - pgautofailover.replication_state gains a new 'archiving' terminal state. - pgautofailover.node gains haspgdata bool, distinguishing ordinary Postgres instances from lightweight ARCHIVING membership rows (a pg_receivewal client, no PGDATA). The old unconditional UNIQUE (nodehost, nodeport) constraint is replaced with a partial unique index scoped to haspgdata rows, since one archiver's (hostname, 0) pair is deliberately shared across every group it serves. - New types, tables and ~26 plpgsql/SQL functions covering: archiver registration and storage targets (local + rclone), formation/group archiver policy (quorum, base-backup policy, replication-quorum eligibility), WAL capture confirmation (wal_archived()/ report_wal_received()), base-backup lifecycle and pruning, warm-standby archiver_node rows with a maxresidentreplay cap, and PITR node lifecycle + command queue. - pgautofailover--2.2--2.3.sql mirrors the same DDL incrementally, since 2.3 hasn't shipped yet; verified end-to-end against a real 1.0 -> ... -> 2.2 -> 2.3 upgrade (including the pre-existing node_nodehost_nodeport_key1 constraint name quirk from two earlier migrations each recreating the table). - New archiving_schema regress test exercising the full schema end-to-end via direct SQL, added at the end of regress_schedule (after cluster_init_failover_rule_attribution, before the dummy_update/ drop_extension/upgrade trio that must stay last) since its expected output pins literal id values tied to its exact position in the shared contrib_regression database, same as every other test in this schedule. Full local regress (20/20) + isolation (6/6) schedules pass, plus a verified real extension upgrade from 2.2 to 2.3.
Adds monitor-side (SQL FSM + C) support for the ARCHIVING replication state, so an ARCHIVING node row (haspgdata = false, created by M1's archiver_add_formation()) is driven through the same node_active() protocol as an ordinary node instead of being stuck at wait_standby forever. No keeper-side/service_archiver work yet -- this is groundwork, verified via node_active() calls made directly against the monitor. - ReplicationState gains REPLICATION_STATE_ARCHIVING (C) / 'archiving' already existed on the SQL enum from M1. - AutoFailoverNode gains hasPgData, populated via TupleToAutoFailoverNode. Looked up by name (SPI_fnumber), not the file's usual hardcoded Anum_ constant: this function is also called against a "RETURNING node.*" tuple descriptor whose physical column order diverges from the explicit SELECT list's logical order once pg_versionnum/pg_version/ pg_versionstring/citus_version are in the mix, so a hardcoded ordinal would silently read the wrong (and wrongly-typed) column for that caller. - MonitorFSM[]: pos 307/309/315/317/319 (report_lsn/wait_standby, primary converged -> secondary/catchingup) gain an explicit hasPgData = TRUE restriction, paired with 5 new hasPgData = FALSE mirror rows (pos 394-398) assigning ARCHIVING instead -- appended after the existing MS-failover cluster since the ordinary rows are numbered with no room between them for 5 more, and the hasPgData split makes their relative order irrelevant to first-match-wins. Pos 367's MS-failover fan-out row (and BuildCandidateList's own C-side secondaryStates list) now also admits ARCHIVING, pulling it into report_lsn during elections exactly like SECONDARY/CATCHINGUP. - system_identifier_is_null_at_init_only loosened to also allow a NULL sysidentifier while reportedstate is 'archiving' or 'report_lsn': an ARCHIVING row never gets a real one. The 2.2--2.3 migration mirror casts the column to text instead of the literals to the enum, since this script's own earlier ADD VALUE 'archiving' and this constraint run in the same ALTER EXTENSION UPDATE transaction and Postgres refuses to create new instances of a not-yet-committed enum value. - keeper_fsm_edges.sql's own "expect zero rows" comment updated: 8 rows are now expected there, a real and currently correct gap -- the monitor side landed first, with no service_archiver/KeeperFSM[] support yet to report ARCHIVING or drive pg_receivewal (next milestone). Verified against a hand-run node_active() scenario (register primary + secondary, converge to primary/secondary, attach an archiver, confirm wait_standby -> archiving instead of catchingup, steady-state archiving stays archiving, replication_quorum = true fans out apply_settings to the primary exactly like an ordinary quorum standby, and rule_pos attribution points at the new rows) in addition to the full regress (20/20) + isolation (6/6) suites and a real 2.2 -> 2.3 extension upgrade.
Adds the keeper-side counterpart to the monitor-side ARCHIVING FSM support (previous commit): KeeperFSM[] rows for WAIT_STANDBY->ARCHIVING/ARCHIVING->REPORT_LSN/REPORT_LSN->ARCHIVING, each dispatching to a new, archiver-specific transition function (fsm_init_archiver/fsm_archiver_report_lsn/fsm_archiver_follow_new_primary, fsm_transition.c) -- mirroring the existing NODE_KIND_CITUS_* pattern of adding separate functions per node kind rather than branching inside the shared ones (fsm_init_standby, keeper_update_pg_state, keeper_ensure_current_state, keeper_node_active_loop stay untouched). New service_archiver.c launches and tracks the one pg_receivewal child an ARCHIVING node keeps running against its group's current primary -- milestone 2's own "colocated fast path" scope (see the Build order in ~/dev/temp/archiving-disaster-recovery.md): pg_receivewal is a real, unmodified Postgres client talking straight to the real primary's own walsender, so no new wire protocol is needed here at all. Not yet wired into supervisor.c's Service/RestartPolicy machinery or a replication slot -- both noted as follow-ups in that file's own header comment. Also: ARCHIVING_STATE added to NodeState (state.h/state.c) and to nodestate_utils.c's nodestateConnectionType() switch (grouped with the other "Postgres known to be stopped" states, since an ARCHIVING row never has a postmaster of its own -- this switch has no default case by design, so a missing case here would have failed the build). Verified: full regress (20/20) + isolation (6/6) suites still pass, and the monitor/keeper reachability cross-check in keeper_fsm_edges.sql -- which the previous commit deliberately left showing 8 unresolved rows, documented as this milestone's own known gap -- now shows zero rows again, both directions, confirming the two sides agree. Live-checked via `pg_autoctl inspect fsm list --json`, which is also how keeper_fsm_edges.json was regenerated (pretty-printed to match the existing file's own review-friendly formatting, not the CLI's compact default). citus_indent and ci/banned.h.sh both pass (the latter caught a raw strerror()/fprintf(stderr) call in service_archiver.c's own exec- failure path, fixed to the project's own log_fatal(..., "%m") convention already used at the other execv() call sites in this codebase).
… (M2 continued)
Adds NODE_KIND_ARCHIVER as a real PgInstanceKind (pgsetup.h/pgsetup.c,
name<->enum both directions), two monitor RPC wrapper functions
(monitor_register_archiver/monitor_archiver_add_formation, monitor.c,
calling M1's own register_archiver()/archiver_add_formation() plpgsql
functions -- not the ordinary C register_node() RPC, since an Archiver is
a process identity, not a (formation, group) membership by itself), and
`pg_autoctl create archiver` (cli_create_node.c): a deliberately minimal,
hand-rolled getopts (not the shared cli_create_node_getopts every ordinary
node kind uses, since that parser's defaults assume a real PostgresSetup
an archiver never has) that registers with the monitor, writes a
KeeperConfig + initial state file (WAIT_STANDBY_STATE, mirroring
archiver_add_formation()'s own starting point), and with --run hands off
to service_archiver_loop() (previous commit).
Verified live against the real monitor RPC layer (not just static
review): registration, formation attachment, and config/state file
writing all confirmed end-to-end against a real running monitor extension
instance, including two real bugs the empirical run caught that manual
review missed --
- config_find_pg_ctl() unconditionally clears pgSetup.pg_ctl before
searching, silently discarding a caller-supplied --pgctl value; fixed
by only calling it when pg_ctl is still empty (also added the missing
--pgctl flag itself -- this dev machine has two pg_ctl on PATH and
needs it to disambiguate, a real scenario, not a test-only one).
- keeper_config_write_file() requires pg_autoctl.role set (validated
against KEEPER_ROLE, not defaulted on write) -- config.role was never
populated, since this path deliberately skips keeper_config_init()'s
ordinary defaults (Postgres-specific probing that doesn't apply here).
`--run`'s actual pg_receivewal launch is still unverified against a real
streaming primary -- needs a real replication-configured Postgres pair,
which is exactly what the next step (pgaftest wiring) provides.
citus_indent and ci/banned.h.sh both pass.
autoctl_node was only ever granted EXECUTE on the function, never SELECT on pgautofailover.basebackup itself, matching every other autoctl_node- callable helper that reads a table it has no direct grant on (e.g. archiver_add_formation) -- get_latest_basebackup was the odd one out. Found via a real end-to-end test of `pg_autoctl archiver serve` against a live monitor: calling it as autoctl_node failed with "permission denied for table basebackup".
The archiver's serving half: a standalone binary (no pg_autoctl/*.c
dependency, only src/bin/common/ and src/bin/lib/log/) that speaks enough
of the real Postgres replication protocol to serve IDENTIFY_SYSTEM, SHOW,
BASE_BACKUP, and a non-standard FETCH_FILE side-channel, backed by an
archiver's local WAL cache and base backups instead of a live postmaster.
No frontend-linkable server-side protocol library exists anywhere in
Postgres (confirmed against pqcomm.c/backend_startup.c/repl_gram.y/
walsender.c, all backend-only) -- this is a genuine reimplementation
guided by that source, not a linking exercise. Two pieces are vendored
near-verbatim since they're already frontend-safe: vendor/tar.c + pgtar.h
(PostgreSQL's own ustar header/checksum logic, src/port/tar.c).
Also ships fetch_client.c / `pg_walsender fetch-file`, the client side of
the FETCH_FILE side-channel, for use as pg_autoctl's own restore_command.
Verified against real, unmodified PostgreSQL client tools:
- psql (replication=1): IDENTIFY_SYSTEM, SHOW wal_segment_size
- pg_basebackup --format=plain -X none --no-manifest: fetched a real
base backup byte-identical to the source, then booted a live Postgres
instance from the result
- pg_walsender fetch-file: fetched a full 16MB WAL segment byte-
identical, plus clean error handling (missing file, path traversal,
unknown route)
See ~/dev/temp/archiving-disaster-recovery.md for the design this
implements milestone 2 of.
`pg_autoctl archiver serve` (cli_archiver.c) is the supervisor verb that
execs pg_walsender as a persistent child (service_archiver_serve.c),
mirroring exactly how service_archiver.c already execs real pg_receivewal
for the outbound WAL-capture direction -- same pattern, new direction.
Keeps pg_walsender's routes file ("<formation>/<group>" -> { walcache,
basebackup }) current, refreshed periodically and on SIGHUP.
The routes file is built from *local* config (formation/groupId/pgSetup.
pgdata), not a monitor round-trip: archiver_add_formation()'s own SQL
inserts the new archiver_node row's pgdata as an empty string, since the
monitor has no way to know an archiver's local WAL cache path -- that's
inherently archiver-host-local information. The one genuinely monitor-
tracked piece is the latest base backup's storage location
(monitor_get_latest_basebackup_location, new in monitor.c).
KeeperConfig gains archiverId/archiverIdStr (keeper_config.h/.c) so a
later, separate `archiver serve` invocation can identify itself to the
monitor -- ini_file.c's INI_INT_T is a plain int, too narrow for a
bigserial id, so this follows citusRoleStr/citusRole's existing string-
plus-parsed-value pattern in the same struct.
Verified against a real, freshly-created cluster (create monitor ->
create postgres -> create archiver -> archiver serve): archiverId
persists and round-trips correctly, the routes file is generated
correctly from live monitor state, pg_walsender starts and serves real
clients through it, and SIGTERM shuts the whole thing down cleanly.
…(M2c)
Completes milestone 2's command surface:
- TIMELINE_HISTORY <tli>: serves a "<tli>.history" file straight out of
the WAL cache directory (RowDescription/DataRow, no COPY involved --
traced from walsender.c's own SendTimeLineHistory()).
- CREATE_REPLICATION_SLOT / READ_REPLICATION_SLOT (physical only): a
slot is a bookkeeping marker file under the WAL cache directory, not
a real Postgres slot -- there's no live server to hold one. Not yet
wired into WAL-retention enforcement (prune_archiver_wal()'s job).
- START_REPLICATION [SLOT <name>] <lsn> TIMELINE <tli>: streams raw WAL
bytes straight from the WAL cache directory. Deliberately does NOT
vendor xlogreader.c: real walsender's own WalSndSegmentOpen just opens
a path computed from TLI+segno and streams bytes -- no WAL *record*
decoding is needed to serve a byte range, only the offset/segment
bookkeeping this file does directly. Handles the actively-growing
(".partial") segment case by polling, matching pg_receivewal's own
producer on the other end of this same protocol.
- wal_dir_scan.c: shared helper -- finds the newest fully-captured WAL
segment and derives its boundary LSN from the filename (XLogFileName
format, fixed 16MB segments). Used by START_REPLICATION's default
position, CREATE_REPLICATION_SLOT's consistent_point, and improves
IDENTIFY_SYSTEM's xlogpos (previously a "0/0" placeholder).
One correctness fix alongside: IDENTIFY_SYSTEM's dbname column must be
NULL for a plain replication=1/true connection (pg_receivewal's style) --
only replication=database (pg_basebackup's style) gets a real dbname back.
Always returning a value broke real pg_receivewal outright ("replication
connection using slot ... is unexpectedly database specific"), caught by
this milestone's own end-to-end testing, not by any narrower unit check.
Verified against real, unmodified PostgreSQL client tools:
- psql: TIMELINE_HISTORY round-trips real file content
- psql: CREATE_REPLICATION_SLOT / READ_REPLICATION_SLOT round-trip
consistent_point/restart_lsn correctly, including the "slot doesn't
exist" all-NULL-row case
- pg_receivewal -S <slot> --endpos=...: streamed a full 16MB WAL segment
byte-identical to the source via START_REPLICATION
(and `pg_autoctl stop`) now supervises an archiver's two
halves together -- WAL capture (service_archiver.c's service_archiver_loop,
outbound pg_receivewal against the primary) and serving (service_archiver_
serve.c's service_archiver_serve_loop, inbound pg_walsender) -- as two real
supervisor.c Service[] entries under one restart-on-crash process tree, the
same way start_keeper() already supervises postgres + node-active together
for an ordinary node (service_archiver_run.c). Dispatched from cli_service.
c's cli_keeper_run(), which already reaches an archiver's config file via
the existing role=keeper path; branches on nodeKind before the Postgres-
instance-specific local_postgres_init()/start_keeper() calls, which don't
apply to an archiver.
Two real bugs surfaced by actually running the full archiver process tree
end-to-end for the first time this session (service_archiver_loop's own
monitor-reporting loop was never previously exercised against a live
monitor for more than a few ticks):
- keeper->postgres.currentLSN was never initialized for an archiver (it
has no real Postgres instance to query it from, so keeper_update_pg_
state() -- the only place that ever set it -- is never called on this
path). node_active()'s own pg_lsn parameter rejected the resulting
empty string outright. Fixed by seeding it to "0/0" once, matching
keeper_update_pg_state()'s own placeholder before a real reading
exists; an archiver's actual capture progress is tracked separately
via archiver_wal, not through this per-node report.
- An ordinary node's own get_other_nodes()/current_state listings now
legitimately include ARCHIVING rows with nodeport = 0 (a deliberate
sentinel, see archiver_add_formation()'s own SQL comment: no
postmaster to be reachable on) -- but monitor.c's node-parsing helpers
treated a parsed port of exactly 0 as an unconditional error, so any
ordinary primary/secondary in a formation with an archiver attached
would fail its own node-active loop entirely. Relaxed the two multi-
node-listing parsers to only reject a genuine parse failure, not the
value 0 itself; left the single-node lookup (which can never
legitimately return an archiver, candidate_priority = 0 excludes it)
unchanged.
Verified against a real, freshly-created cluster (monitor + primary +
archiver): `pg_autoctl run --pgdata archiver1` starts both services
cleanly, the FSM transitions wait_standby -> archiving and real
pg_receivewal starts against the primary, pg_walsender serves real
clients through it, and `pg_autoctl stop` cascades a graceful shutdown
through both services and their own child processes (pg_walsender,
pg_receivewal) with no orphans left behind. Also re-verified the primary
node's own node-active loop, previously broken by the port=0 regression,
now runs cleanly with an archiver attached to its formation.
service_archiver.c gains service_archiver_report_captured_wal(), called once per node_active tick from service_archiver_loop(): it scans the archiver's local WAL cache directory for segments pg_receivewal has completed (i.e. no longer ".partial") since the last one reported, and calls the new monitor_report_wal_received() (monitor.c/.h) -- a thin wrapper around the already-existing pgautofailover.report_wal_received() SQL function -- for each one. This is what actually populates archiver_wal and makes wal_archived() return true; until now nothing in the codebase ever called that SQL function. Also fixes a liveness gap this uncovered: pg_receivewal was only ever (re)started from the FSM transition functions that move a node *into* ARCHIVING_STATE (fsm_init_archiver, fsm_archiver_follow_new_primary). An archiver process restarted while already ARCHIVING (or one whose pg_receivewal child died on its own) had nothing to bring it back up, despite this file's own header comment already describing that as the design. service_archiver_loop() now checks service_archiver_pgreceivewal_is_running() every tick and restarts it when needed, exactly matching that comment. Verified end-to-end against a real monitor + primary + archiver: forced WAL switches on the primary, confirmed archiver_wal gets populated with the correct end-of-segment LSNs and wal_archived() correctly reflects archiver_quorum, confirmed the liveness restart itself by killing and restarting the archiver process while already ARCHIVING. Full SQL regression schedule (src/monitor, 20/20) still passes. Dockerfile: copy pg_walsender into the "run" stage image alongside pg_autoctl -- needed for any archiver node in a pgaftest Docker environment, and a prerequisite for M4's own pgaftest specs.
Adds the "archiver" node kind to pgaftest's own DSL, needed to write any
.pgaf spec that includes an ARCHIVING node:
- test_spec_scan.l/.y: new "archiver" keyword (T_ARCHIVER), usable the
same way "coordinator"/"worker" already are: `archiver1 archiver`
inside a formation{} block.
- compose_gen.c: writes kind = archiver into the node's .ini, and links
service_archiver.c into pgaftest's own SHARED_SRCS (Makefile) so the
binary can drive an archiver node the same way it already drives
postgres/coordinator/worker ones.
- nodespec.c (pg_autoctl, not pgaftest): teaches `pg_autoctl node run
<node.ini>` -- the one command every pgaftest container actually
execs -- to recognize kind = archiver and build the right `pg_autoctl
create archiver` argv. An archiver's own getopts is deliberately
minimal (no --pgport/--ssl-*/--auth/...), so it gets its own argv
branch rather than falling through into the generic postgres-flags
path every other kind shares.
Also fixes a real bug in cli_indent.c's print_node() found while
writing the first archiver spec and round-tripping it through `pgaftest
indent`: the node-kind-to-keyword switch only had cases for coordinator
and worker, so indenting a spec containing an archiver node silently
dropped the "archiver" keyword on write-back, turning it into a plain
postgres node. Added the missing NODE_KIND_ARCHIVER case.
test_spec_parse.c/.h and test_spec_scan.c are bison/flex output,
regenerated from the .y/.l changes above.
First pgaftest spec exercising an ARCHIVING node, covering the two things Milestone 4 adds: - test_001/test_002: forcing WAL segment switches on the primary gets each completed segment reported to the monitor (service_archiver_report_captured_wal(), service_archiver.c) and reflected by pgautofailover.wal_archived() -- the archive_command confirmation check nothing populated before this milestone. - test_002 also exercises the liveness fix in service_archiver_loop(): killing and restarting the archiver process while it's already ARCHIVING must bring pg_receivewal back up on its own, not just on the FSM transition that first enters that state. - test_003: fails node1 over to node2 and confirms the archiver passes through REPORT_LSN_STATE and back to ARCHIVING_STATE (following the new primary), and that segments recorded before the failover are still there afterwards. Segment filenames are asserted directly (a fresh cluster deterministically starts WAL at 000000010000000000000001, and each pg_switch_wal() on an idle test database advances exactly one segment) since autoctl_node has no direct SELECT on archiver_wal -- wal_archived() is the only accessor it can call. Registered in tests/tap/schedule and tests/tap/schedules/node.sch, alongside the other node-lifecycle/FSM specs.
First half of Milestone 5 per the design doc's own Build order ("live
first, then replay/volatile"). New file service_archiver_basebackup.c
adds service_archiver_maybe_generate_basebackup(), called once per tick
from service_archiver_loop() alongside the M4 WAL-report/liveness calls.
Trigger scope for this pass: bootstrap only -- a group with zero
existing base backups (monitor_get_latest_basebackup_location() reports
not-found) gets one immediately. Scheduled/timeline-change/retention
triggers need basebackup_policy wired through the CLI first, a later
milestone.
Target selection follows the design doc's `live` precedence, minus its
warm-standby tier (nothing to select from yet, also later): the first
healthy secondary in the group (monitor_get_nodes(), skipping port == 0
ARCHIVING rows), falling back to the primary when none exists.
Generation itself is a one-shot forked child (basebackupPid, tracked the
same way service_archiver.c tracks pgReceivewalPid) rather than a
persistent service, so a potentially long-running pg_basebackup can't
stall the main loop's own node_active()/WAL-report tick. The child execs
the real, unmodified pg_basebackup client with --wal-method=none -- this
backup is deliberately not self-consistent on its own, since the
archiver's already-running WAL capture is what supplies the WAL needed
to reach consistency on replay -- then reads the resulting backup_label
for the authoritative start LSN/timeline and reports both start and
completion to the monitor via two new wrappers,
monitor_report_basebackup_started()/_completed() (monitor.c/.h), calling
the SQL functions M1's schema already shipped but nothing had called
yet. endlsn is best-effort: a live read of the source's current WAL (or
last-replayed, if the source is a standby) position right after the
backup finishes: not Postgres's own internal stop-backup LSN (not
observable from a plain CLI wrapper around pg_basebackup), but a
reasonable upper bound, and never fatal to the backup itself if that one
query fails.
Verified end-to-end against a real monitor + primary + archiver: the
bootstrap backup fires automatically, archiver_wal / basebackup rows
land correctly (source = 'live', status = 'complete', a real endlsn
distinct from startlsn), and the resulting directory passes
pg_verifybackup. Full SQL regression schedule (src/monitor, 20/20)
still passes.
Second half of Milestone 5 ("live first, then replay/volatile" per the
design doc's Build order). service_archiver_maybe_generate_basebackup()
now takes a bootstrap `live` backup as before, then -- on the very next
tick -- exercises the `replay`/`volatile` pipeline exactly once: extract
the last retained backup into a throwaway staging directory, point its
recovery at this archiver's own already-captured WAL (restore_command +
recovery.signal, entirely local, no network round trip), let it replay
forward and promote once it runs out of locally-captured segments, then
pg_basebackup it over loopback and discard the staging instance --
'volatile' means nothing survives between cycles.
Real frequency-driven scheduling (basebackup_policy's own frequency/
onpromotion/retention, resolved through get_archiver_policy()/
get_basebackup_policy()) is a deliberate follow-up, not built here: the
milestone-defining new capability is the replay mechanism itself, not a
general scheduler (matching the design doc's own build order, which
lists warm standby's scheduling machinery as a later milestone).
monitor_report_basebackup_started() (added in the `live`-only commit)
now takes real source/replaymode parameters instead of a hardcoded
'live', and monitor_get_latest_basebackup_location() is renamed to
monitor_get_latest_basebackup_info() and returns the latest backup's
source alongside its storage location -- what the trigger above uses to
tell "only the bootstrap has run" from "the replay exercise is already
done".
Getting a working staging instance up took two real, load-bearing fixes
along the way:
- pg_ctl start, invoked here through both a hand-rolled fork()/execl()
and this project's own run_program() helper, reproducibly misparsed
its own arguments in this exact process tree (deep in a supervised
archiver's own fork chain) even though byte-identical argv worked
fine in every standalone reproduction attempted. Root cause not
fully isolated; worked around by execing the real "postgres" binary
directly instead of going through pg_ctl at all -- the same
fork()/execv() pattern already used for pg_receivewal and
pg_basebackup in this codebase, with readiness confirmed by polling
a real SQL connection rather than relying on pg_ctl's own "-w".
- recovery_target_lsn set to "the end of the latest complete segment"
is not actually a reachable record boundary on a mostly-idle source
(a renamed, "complete" segment file is always its full fixed size
regardless of how much of it is real WAL) -- recovery correctly
refused to pause there ("recovery ended before configured recovery
target was reached"). Replaying to "everything locally available"
and letting Postgres promote on its own sidesteps needing a precise
target at all, which a `volatile`, discard-after-use snapshot never
actually needed in the first place.
Verified end-to-end against a real monitor + primary + archiver, from a
cold start through both the live bootstrap and the replay follow-up:
basebackup rows land correctly (source/replaymode/status all correct,
real distinct startlsn/endlsn across the sequence), and both resulting
directories pass pg_verifybackup. Full SQL regression schedule
(src/monitor) passed 20/20 twice earlier against this same unchanged
schema in this session; a later re-run hit an apparent local
pg_regress/DROP DATABASE environment hang (ProcSignalBarrier) unrelated
to any change in this commit -- no .sql files are touched here.
pgaftest coverage for Milestone 5's own base backup generation (both live and replay/volatile): brings up a monitor + primary + archiver, then waits for both the bootstrap live backup and the one-time replay/volatile follow-up to land, checking the group's final pgautofailover.get_latest_basebackup() row (source = 'replay', replaymode = 'volatile', status = 'complete'). No explicit trigger step is needed here, unlike archiver_wal_capture.pgaf's pg_switch_wal() calls -- both backups fire on their own within a couple of service_archiver_loop() ticks of the archiver starting. That also makes the intermediate 'live'-only state unsafe to assert on directly (this pass's own trigger logic produces at most one live and one replay backup before going quiet for the group, a couple of ticks apart, with nothing in this spec's control over exactly when to look) -- only the final state, once both have landed, is deterministic. Registered in tests/tap/schedule and tests/tap/schedules/node.sch, alongside archiver_wal_capture.pgaf.
New "Archiving & Disaster Recovery Architecture" section in intro.rst, between "Single Standby Architecture" and "Multiple Standby Architecture" -- an archiver is orthogonal to standby count, so it reads best as the thing you add on top of the simplest case before the doc branches into standby-count variations. New docs/tikz/arch-archiver.tex, rendered to .svg the same way every other architecture diagram in this directory is (latexmk -lualatex + pdftocairo, verified locally): primary + secondary + archiver, with the archiver's own WAL cache / base backups called out, a distinct WAL streaming (pg_receivewal) edge separate from real streaming replication, and the monitor's health-check/WAL-report edges to all three. common.tex gains one new color pair (abox/atxt, MS amber) and one new edge style (wal, dashed) for the archiver box and its WAL-streaming edge -- deliberately not reusing the primary/standby colors, since an archiver is a different kind of entity, not a replica. Terminology: uses "archiver" for the physical entity and "archiving node" for its per-group FSM membership, per-project decision -- avoids colliding with pgautofailover.archiver_node, the broader schema table that also covers warm-standby/pitr instances which don't participate in elections at all.
keeper->postgres.currentLSN was set to "0/0" once at service_archiver_ loop() startup and never updated again -- an archiving node's own reportedlsn in pgautofailover.node stayed at that placeholder forever, no matter how much WAL it had actually captured. This mattered more than it looked: pgautofailover.get_most_advanced_ standby() -- the query fast-forward uses to pick a WAL source during a failover election -- has no kind-based exclusion at all, and an archiving node already passes through REPORT_LSN_STATE during an election exactly like any other node (ARCHIVING_STATE -> REPORT_LSN_STATE, fsm.c). A "0/0" reportedlsn was the only thing keeping an archiver from ever being ranked as a candidate WAL source. service_archiver_update_current_lsn() now scans the local WAL cache for the newest complete segment each tick and updates currentLSN to its end LSN before keeper_node_active() reports it -- verified against a real cluster: after two pg_switch_wal() calls, the archiver's own pgautofailover.node.reportedlsn row tracks the primary's position almost exactly (0/A000000 vs. the primary's own 0/A000060).
…m an archiver
Confirms (and builds out) the reframing from the previous commit: an
archiving node already passes through REPORT_LSN_STATE during elections
and get_most_advanced_standby() has no kind-based exclusion, so once its
currentLSN is real, fast-forward's existing streaming-replication code
path can already select and target one -- no new restore_command
plumbing needed. Four real gaps stood between that and actually working,
found and fixed by testing a genuine, unmodified Postgres standby against
a real archiver end to end (not just pg_receivewal, which never exercises
any of these):
- pg_walsender routing is dbname-based (formation/group as dbname), but a
real standby's own walreceiver never forwards the operator's dbname for
a physical replication connection -- it always sends the literal
"replication", confirmed against a real standby. accept_loop.c now
falls back to the single configured route when it sees that sentinel,
matching this milestone's own one-membership-per-archiver scope; a
multi-route archiver (later milestone) needs a different mechanism
(e.g. application_name, which real walreceiver does forward).
- IDENTIFY_SYSTEM's systemid always fell back to the "unknown" placeholder
"0" because nothing ever populated route->systemId: service_archiver_
serve.c's own routes-file writer never wrote a systemid key, even
though routes.c already knew how to parse one. A real standby rejects
a mismatched system identifier outright ("database system identifier
differs between the primary and standby"). Fixed with a new monitor
RPC, monitor_get_group_system_identifier() (pgautofailover.
get_group_system_identifier(), new SQL function in both
pgautofailover.sql and the 2.2--2.3 migration -- an archiving node has
no sysidentifier of its own, but every other node in its group shares
the same one), wired into the routes-file refresh.
- cmd_start_replication.c read raw fread() bytes from a ".partial"
segment without knowing where pg_receivewal's actually-written data
ends -- pg_receivewal pre-allocates the full segment size up front
(matching real Postgres's own WAL file pre-allocation), so reading past
the real tail returns zeros indistinguishable from real content at the
byte level. Sending that tail as WAL data is exactly what a real
standby's recovery logic flags as "invalid record length ... got 0",
and on seeing it, terminates its own walreceiver outright rather than
treating it as "nothing new yet, retry" -- with no automatic
reconnection afterward. Fixed by trimming any trailing zero run before
ever sending a ".partial" chunk (self-correcting: an in-progress
boundary just gets re-read next tick instead of shipped early).
- get_most_advanced_standby() returns an ARCHIVING row's real nodeport,
which is the port == 0 sentinel (no postmaster of its own), not the
archiver's actual pg_walsender serve port -- the monitor has no column
for that (archiver-host-local information, same reasoning service_
archiver_serve.c's own routes file exists for). keeper_get_most_
advanced_standby() now resolves a port == 0 candidate to
PG_AUTOCTL_ARCHIVER_SERVE_PORT, matching this milestone's single-
well-known-port scope.
Verified end-to-end: a real pg_basebackup-seeded standby, given nothing
but an ordinary primary_conninfo pointing at the archiver's serve port,
completed backup recovery, reached consistent recovery state, streamed
live via START_REPLICATION, stayed connected indefinitely (pg_stat_wal_
receiver: status = streaming), and correctly applied newly-written data
(a table created and populated on the real primary afterward) -- with
zero restore_command, zero new replication-source machinery, and zero
changes to fsm_fast_forward's own selection logic beyond the port fix
above.
Bootstraps a brand new node from a registered archiver's base backup
plus captured WAL instead of the group's live primary -- the disaster-
recovery case: rebuild after every live standby (or even the primary)
is gone, with only the archiver left standing. Verified end to end
against a real cluster: `create postgres --from-archiver` completed
pg_basebackup from the archiver, replayed WAL, and settled into a
genuinely healthy "secondary" (pg_stat_wal_receiver: status =
streaming), matching reportedlsn against the real primary once it
re-parented there.
New plumbing:
- KeeperConfig.fromArchiver (keeper_config.h) plus the `--from-archiver`
CLI flag on `create postgres` (cli_create_node.c, cli_common.c) --
runtime-only, same as createAndRun, since reach_initial_state() runs
in the same `pg_autoctl create` invocation that parses it.
- pgautofailover.get_archiver_node() (pgautofailover.sql, the 2.2--2.3
migration) plus its monitor_get_archiver_node()/keeper_get_archiver_
node() C wrappers (monitor.c, keeper.c): finds the ARCHIVING row for
(formation, group) directly. Deliberately not get_most_advanced_
standby() -- that function filters on reportedstate = 'report_lsn', a
transient state an archiving node only visits during a FAST_FORWARD
election, never during its normal steady-state 'archiving' operation,
so it can never find an idle archiver outside of an election.
- fsm_init_standby() (fsm_transition.c) branches on config->fromArchiver
to resolve the archiver via the above instead of keeper_get_primary(),
and passes an empty replication slot name -- pg_walsender has no
slot-based retention in this milestone (cmd_start_replication.c's own
header comment), so standby_init_database's pre-flight replication-
slot check must be skipped rather than asked to verify a slot that
will never exist, matching that function's own existing "initialising
from another standby, no primary yet" precedent.
Four further real, narrow gaps stood between that and actually working,
each found by running the real `pg_basebackup`/`pg_autoctl` code paths
end to end rather than by inspection:
- pg_walsender's BASE_BACKUP had no manifest support (documented scope
cut, cmd_base_backup.c), but PG13+ pg_basebackup requests one by
default -- ReplicationSource.noManifest (pgsql.h) plus pg_basebackup()
passing --no-manifest when set (pgctl.c) works around it for an
archiver-sourced clone specifically, without touching a real primary's
own backup path.
- pgctl_identify_system() (pgctl.c) built its replication connection
string with no dbname at all, relying on real pg_basebackup's and
real walreceiver's own respective "default unset dbname to the literal
'replication'" behaviors -- neither of which this is: it's pg_auto_
failover's own raw libpq connection, which has no such default and
instead falls back to plain libpq's own "dbname = username" rule
(fe-connect.c), a route pg_walsender's routes file was never going to
have an entry for. Passing "replication" explicitly matches what every
other replication client already sends on the wire, and is a no-op
against a real primary (which ignores dbname for replication=true
connections regardless).
- A "replay" base backup (basebackup_replay_mode, milestone 5) promotes
a throwaway extracted copy to make it self-consistent, which genuinely
puts it on a *later* timeline than whatever the archiver's own walcache
has actually captured (which only ever advances on the real primary's
timeline) -- serving that pairing breaks a real pg_basebackup's own
timeline consistency check once it reaches its background WAL-streaming
step ("starting timeline N is not present in the server", comparing
the backup's own timeline against IDENTIFY_SYSTEM's). Fixed at the
source: pgautofailover.get_latest_basebackup() grew an optional
preferred_source filter (both SQL files), and service_archiver_serve.c's
routes refresh now asks for 'live' specifically -- a live-sourced
backup always shares the walcache's timeline by construction. A second,
independent, defense-in-depth check (walcache_current_timeline(),
comparing the walcache's own newest captured segment's embedded
timeline against whatever's about to be advertised) keeps the routes
file from ever serving a mismatched pairing even if that invariant is
ever violated by a future backup mode. monitor_get_latest_basebackup_
info() also grew a timeline out-param, threaded into the routes file's
own (previously unpopulated) "timeline" key -- already parsed by
routes.c, never written by anyone until now.
- cmd_start_replication.c ended a stream with bare CopyDone and nothing
else. A real, long-lived streaming client (real walreceiver, via
primary_conninfo) never triggers the gap because it never decides to
stop on its own -- which is exactly why this went unnoticed through
all of the earlier fast-forward-from-archiver verification. But
pg_basebackup's --wal-method=stream background WAL receiver does
decide to stop, once it reaches its own target LSN, and real receive-
log.c's ReceiveXlogStream only accepts that as a *successful* stop
when it can read a matching CommandComplete afterward (matching real
walsender.c's own WalSndDone, which sends exactly that on controlled
shutdown) -- without it, the client falls through to "unexpected
termination of replication stream" and exits non-zero even though
nothing was actually wrong on the wire. Fixed by sending a CommandComplete
tagged "COPY" right after CopyDone.
…iver Adds archiver_bootstrap_and_fast_forward.pgaf, the disaster-recovery scenario this whole investigation was driven by: a primary, an archiver, and a secondary that's created via `pg_autoctl create postgres --from-archiver` (not from the live primary) after the archiver's first live base backup is ready, then a FAST_FORWARD election where the archiver is the only node with the WAL the winning candidate is missing. node2 is declared `create and launch deferred`: the normal ini-driven node bring-up (`pg_autoctl node start`) has no hook for a custom flag like --from-archiver (NodeSpec/nodespec.c carries no such field -- fromArchiver lives only in KeeperConfig, populated exclusively by cli_create_node.c's own direct CLI parsing), so test_001 `exec`s into node2's own container and runs `pg_autoctl create postgres --from-archiver` by hand, then backgrounds `pg_autoctl run` the same way debug_citus_worker_switchover.pgaf backgrounds a long-lived process (`bash -c "nohup ... &"` -- a foreground `pg_autoctl run` would hang `docker compose exec -T` forever otherwise). test_002-004 engineer a real WAL gap rather than relying on race timing: stop node2 so it can't stream from node1 anymore, generate more WAL on the primary and give the archiver (still capturing independently via pg_receivewal) time to land it, kill the primary, then bring node2 back -- at that point the archiver is strictly ahead of node2 and is the only viable FAST_FORWARD WAL source. The final row-count check on node2 post-promotion confirms real WAL bytes were fetched and applied, not just that the FSM passed through the right state label. Verified: `pgaftest show spec`/`show compose` parse this spec cleanly (exit 0) and `pgaftest indent` round-trips it losslessly, confirming the DSL usage (deferred node declaration, exec/nohup backgrounding, multi-state `passing through` clause) is syntactically valid against the real grammar. Could not run it against a live docker compose cluster in this session: `make -f Makefile.docker build-pg17` fails fetching ghcr.io/hapostgres/pg_auto_failover/pgaf-base (401 Unauthorized, no registry credentials available here), and no local base image is cached to build from instead. Every C-level behavior this spec exercises (--from-archiver's own bootstrap, and fast-forward sourcing WAL from an archiver) was independently verified working end-to-end by hand against a real cluster in the two preceding commits on this branch.
wal_archived() is a plain LANGUAGE sql function (not SECURITY DEFINER), so it runs under the caller's own privileges. autoctl_node never got a direct SELECT grant on pgautofailover.archiver_wal: the blanket `GRANT SELECT ON ALL TABLES IN SCHEMA pgautofailover TO autoctl_node` only covers tables that already existed when that statement ran, and archiver_wal (like every other table in the M1 archiving schema) was created after it. Confirmed live: calling wal_archived() as autoctl_node (the role node_active() actually uses) failed with "permission denied for table archiver_wal". get_latest_basebackup() had this exact same bug, already fixed the same way (SECURITY DEFINER) in a prior commit -- apply the same fix here.
BuildForPrimaryNodeNodeActiveContext() counted every other node in the group toward replicationQuorumCount/secondaryNodesCount/ secondaryQuorumNodesCount, including ARCHIVING rows -- which are never real Postgres secondaries and can never report SECONDARY. In a formation with only a primary and an archiver, that miscount let the archiver's own bootstrap WAIT_STANDBY reading trip anyOtherNodeWaitingStandby (pos 401) and bump the primary off SINGLE, while secondaryQuorumNodesCount could then never legitimately reach zero -- so the primary got stuck between SINGLE and PRIMARY forever. Skip ARCHIVING (hasPgData=false) rows in that loop, matching the hasPgData-based exclusion this file's own REPORTING_NODE section already applies for a different purpose. Also adds the archiver-mirror FSM rows (pos 394/396/399) their own SINGLE|WAIT_PRIMARY|JOIN_PRIMARY match set, since a primary attached only to an archiver legitimately stays SINGLE the whole time instead of ever reaching WAIT_PRIMARY.
…(M5)
Several pieces of the Archiving & Disaster Recovery milestone, landing
together since they build on each other:
WAL-capture reliability
- service_archiver_start_pgreceivewal() now creates a replication slot
for pg_receivewal (pgautofailover_standby_<nodeId>), the same one
keeper_create_and_drop_replication_slots() already creates eagerly
on any primary for every other node regardless of kind. Without a
slot, a pg_receivewal that loses the startup HBA-propagation race
restarts from the server's then-current position, silently and
permanently skipping whatever WAL existed in between.
- pg_walsender's START_REPLICATION now fails loudly ("58P01") instead
of waiting forever when asked for a segment that predates this
archiver's own captured history and will never arrive.
- The archiver's real captured-WAL position is now tracked out of
band (a position file, service_archiver_position_path() and
friends) so it can cross the fork() boundary between the capture
and serve processes -- consumed by cmd_base_backup.c's own
end-of-backup position (previously could re-send a stale start
position and hang a real pg_basebackup's background WAL streamer
forever) and by cmd_identify_system.c indirectly via the routes
file's new "position" key.
- service_archiver_loop() now sets pgIsRunning = true for the
archiver's own keeper state, which the monitor's NodeIsHealthy()
unconditionally requires before ever selecting a node as a
FAST_FORWARD WAL source.
Telemetry
- service_archiver_report_storage() reports disk usage/free space to
the monitor periodically; monitor_get_archivers() surfaces it (and
each archiver's FSM state) to `pg_autoctl watch`'s new archivers
section.
Base-backup production/retention policy
- New SQL: get_basebackup_policy_for_group(), list_basebackups();
get_basebackup_policy() gains SECURITY DEFINER (needed now that
`pg_autoctl show basebackup-policy` calls it directly).
- service_archiver_basebackup.c's scheduling is now policy-driven
instead of the previous hardcoded "bootstrap live, then exactly one
replay, then quiet" scope: frequency/source/replaymode/onpromotion
read from whichever policy resolves for the group, plus
maxcount/maxage retention pruning after each successful backup. The
very first backup for a group is always sourced live regardless of
policy (nothing to replay from yet).
- The replay/volatile staging instance now starts with ssl = off:
the copied postgresql.conf/postgresql.auto.conf still carries the
source node's own ssl_cert_file/ssl_key_file paths, meaningless
here since the archiver has no Postgres SSL certs of its own --
left enabled, the staging instance failed outright at startup.
- New CLI: `pg_autoctl create/show/set basebackup-policy`, and
`pg_autoctl create archiver --basebackup-policy <name>` to attach
one at creation time via set_archiver_policy().
Verified via a full --no-cache Docker rebuild plus the archiver_wal_
capture, archiver_basebackup_generation, archiver_basebackup_policy,
and archiver_bootstrap_and_fast_forward pgaftest specs, all passing.
archiver_wal_capture.pgaf: fixed a wrong segment-1 assumption (the archiver's replication slot only protects WAL from its own creation time onward -- by the time it's created, node1+node2's own bootstrap has typically already consumed segments up to the empirically observed floor, segment 3) and switched two `wait until ... state is primary` assertions to the real terminal state after a permanent primary loss (`wait_primary`: WAIT_PRIMARY -> PRIMARY requires another node to reach reported SECONDARY, which an archiver never will). archiver_basebackup_generation.pgaf: the schema's own 'default' policy (frequency 24h) no longer produces a second, replay-sourced backup within any sane test window now that scheduling is policy-driven instead of hardcoded "bootstrap live, then exactly one replay". Attach a short-frequency, source=replay policy during setup so the spec's own remaining job -- proving the replay/volatile generation pipeline itself still works -- is still genuinely exercised. archiver_bootstrap_and_fast_forward.pgaf: same wait_primary fix as above, applied where this spec also stops the original primary for good partway through. New: archiver_basebackup_policy.pgaf, covering the base-backup policy feature end to end -- a fast-cycling, maxcount=3 policy created via the real CLI, attached via set_archiver_policy(), reaching and holding a stable retained count after several times its frequency has elapsed. Registered both archiver_basebackup_policy and (previously missing) archiver_bootstrap_and_fast_forward in tests/tap/schedules/node.sch. All four specs verified passing against a from-scratch --no-cache Docker rebuild.
Intro - Rewrote the opening paragraph: pg_auto_failover is a complete system (pg_autoctl runs as its own pid 1 supervising postmaster), not just an extension -- dynamic topology, automated or operator- driven, two modes of operation (command-driven CLI and node.ini + `pg_autoctl node run`). - New "High Availability, Disaster Recovery, and Backups: One System" section with a new two-panel diagram (arch-ha-dr-unified.tex/.svg) contrasting the typical separate-HA-tool/separate-backup-tool split against pg_auto_failover's single control plane for both. Failover State Machine - New "Archiving" subsection in the State reference, covering the ARCHIVING state's real transitions (verified against live `pg_autoctl inspect fsm list --json` output), its exclusion from candidacy/quorum, and its role as a Fast_forward-eligible WAL source. - Added the 3 real archiving edges to the "Node init / join" and "Failover / promotion" mermaid diagrams, with a new archiverState color class and cross-reference notes. Updated the "20 states and 77 transitions" summary line to 21/80. Fault Tolerance - New "Archiving Nodes and Disaster Recovery" section: WAL capture independent of any standby, base backups on a policy, rebuilding a node (or a whole formation) from an archiver's cache, and how archiving nodes participate in (and are excluded from) failover. Operations - New docs/archiving.rst page: registering an archiver, creating and attaching base-backup policies, watching an archiver, and rebuilding a node with `pg_autoctl create postgres --from-archiver` -- including the disaster-recovery case of rebuilding a whole formation from a single surviving archiver. Reference - New CLI reference pages for `pg_autoctl create/show/set basebackup-policy`, registered in their respective toctrees. Verified with a clean `sphinx-build -W --keep-going` (no warnings, no broken references).
…anels Replaces the single stacked arch-ha-dr-unified diagram with two separate figures, each a "production architecture" style pair of dashed service-boundary boxes with a header + inner service pills: - arch-ha-dr-typical: High Availability (Patroni, repmgr) next to Disaster Recovery + Backups (pgBackRest, pgBarman) -- two entirely separate boundaries, naming the actual products a typical setup reaches for. - arch-ha-dr-pgautofailover: High Availability + Disaster Recovery collapse into a single pg_auto_failover box; Backups (pgBackRest, pgBarman) remains its own separate boundary. Colors are a muted, readable palette local to these two diagrams (dark-tinted text, pale tints for fills) rather than raw saturated brand colors used directly as text -- the previous version's bright green header/body text (mbox, #9BF00B) was a real readability problem. Node heights are compact (1.35cm pills) instead of the previous 2.3cm/6.4cm boxes, since most of these boxes hold a single line of text. intro.rst's "High Availability, Disaster Recovery, and Backups: One System" section is retitled "High Availability and Disaster Recovery: One System" and its body adjusted to match: Backups, in the narrower sense of retention/cataloguing/cloud tiers, is now described as its own remaining concern rather than folded into "one system," matching what the new diagrams actually show.
…maid The five keeper-FSM mermaid diagrams had drifted from real KeeperFSM[] output -- verified by re-running `pg_autoctl inspect fsm mermaid <phase>` for all five phases and diffing byte-for-byte against what was checked into the docs. Real gaps found and fixed: - Failover / promotion was missing the entire "wherever you were, you're being demoted now" fan-out (init/single/catchingup/secondary/ prepare_promotion/stop_replication/maintenance/prepare_maintenance/ wait_maintenance/report_lsn/fast_forward, each with both a -> demoted and -> demote_timeout edge), plus several report_lsn fan-in edges (fast_forward/prepare_promotion/stop_replication/ demote_timeout/join_secondary -> report_lsn) -- 28 missing edges in this diagram alone. - Node removal / drop was missing wait_maintenance -> single and fast_forward -> single. - Maintenance was missing wait_maintenance -> report_lsn. - The archiving state's edges (added in an earlier, hand-written pass) are now the tool's own generated labels/coloring (electionState amber, not a separate hand-added archiverState class) instead of hand-embellished text not backed by any real KeeperFSM[] comment. Node init / join and Steady-state / config changes already matched exactly. Updated the summary line and the "replaces the old Graphviz diagram" note from the stale 80/68 transition counts to the real total: 21 states, 102 transitions (111 raw KeeperFSM[] edges minus the 9 excluded join_primary ones). Fixed the Failover / promotion intro paragraph's "still less than half the size of the full graph" claim -- at 57 of 102 edges it's now over half, which the added fan-out edges explain (most of that diagram's size is exactly that "interrupted from anywhere" fan-out). Added an explicit `archiving_state` label on the State reference's Archiving entry so other pages can :ref: it directly instead of relying on an implicit, same-document-only section-title link.
New docs/archiving-internals.rst, in the Architecture toctree: the technical reference for how archiving is actually built, meant to be the main place to extend for later milestones (warm standby, PITR, cloud push). Covers, grounded directly in the current source (function names, exact invocations, exact file paths): - The two forked processes per archiver (capture, serve) and the two files (archiver-position, archiver-routes.ini) that are their only channel to each other -- new arch-archiver-internals diagram. - WAL capture: how an archiver's replication slot reuses the exact same mechanism a real standby's slot uses, with zero primary-side special-casing; the exact pg_receivewal invocation; how the real captured position is computed (including .partial-segment trailing- zero trimming) and shared across the fork boundary; what happens to pg_receivewal across a failover. - Base backup generation: the basebackup_policy table and its 3-tier resolution chain; exactly when a backup is due (bootstrap, onpromotion, frequency); the live pg_basebackup invocation; the full replay/volatile pipeline (staging instance, recovery config, promote, basebackup over loopback, discard); retention pruning. - pg_walsender: why it's a from-scratch reimplementation (no frontend-linkable server-side replication library exists), its process model, the routes-file-based auth/routing mechanism, and a full table of every wire command it implements. - How pg_autoctl create postgres --from-archiver and FAST_FORWARD reuse the port==0 archiver-serve-port resolution trick to talk to pg_walsender with no archiver-specific code past that one lookup. - Build/process wiring, and an explicit "extension points" section listing what M6/M7/M8 build on top of, and what's schema-complete but not yet enforced (concurrency, allowed_hosts). Verified clean with sphinx-build -W --keep-going (no warnings, no broken references).
Mermaid diagrams already get pan/scroll-to-zoom via mermaid_d3_zoom (conf.py), but that's specific to Mermaid's own inline-SVG rendering and never applied to the tikz-rendered figures the rest of the docs embed via `.. figure::` -- those render as plain <img src="....svg">, which d3-zoom can't attach to. New docs/_static/js/zoom.js + css/zoom.css: a small, dependency-free overlay wired to every `figure img` at page load. Click (or Enter/Space when focused) opens the image full-screen on a dark backdrop; scroll to zoom, drag to pan, double-click to reset, Esc/backdrop-click/close-button to dismiss. Wired site-wide via conf.py's existing add_css_file/ add_js_file setup() hook, the same mechanism already used for the project's custom CSS. Verified interactively: click opens the overlay, wheel/drag/dblclick/Esc all behave as expected, and a clean sphinx-build -W --keep-going.
New top-level section right after the page's own intro, before "The pg_auto_failover Monitor": frames High Availability as two distinct guarantees -- Service Availability (the Postgres service stays reachable, what the rest of this page/failover-state-machine.rst/ fault-tolerance.rst describe) and Disaster Recovery (the data survives even total loss of every node that ever held it, what archiving-internals.rst and the archiver covers) -- cross-referencing into both rather than duplicating either. Adds a page-level `fault_tolerance` label to fault-tolerance.rst (it had no explicit label of its own) so this new section can :ref: it directly.
…d its TLS defaults SCRAM on libpgcommon. common/scram.c no longer implements the primitives: scram_SaltedPassword/ClientKey/ServerKey/H/build_secret, pg_hmac_*, pg_b64_*, pg_saslprep and pg_strong_random come from libpgcommon/libpgport (already linked). scram_compat.h hides the four signature variants across PG 14-19. The exchange follows auth-scram.c and gains what we lacked: SASLprep (non-ASCII passwords), SCRAM-SHA-256-PLUS channel binding (tls-server-end-point, hash as be_tls_get_certificate_hash), and mock authentication so an unknown role is indistinguishable from a wrong password. Vendored ifaddr.c: pg_range_sockaddr(), pg_sockaddr_cidr_mask() and pg_foreach_ifaddr() from PostgreSQL, replacing the hand-written CIDR match and adding the samehost/samenet HBA keywords. Reverse DNS is resolved once per connection, like check_hostname(). fetch-file on libpq. FETCH_FILE '<name>' is now an ordinary command on an ordinary connection served as a regular COPY OUT (the fetch/ dbname side-channel and the raw line reader are gone); the client is a plain libpq connection, so TLS, PGPASSWORD/.pgpass, PGSSLMODE... work as for pg_basebackup. About 350 lines of client startup/TLS/SCRAM code are deleted. TLS: the context settings of be_tls_init (no tickets, no session cache, moving write buffers, server cipher order, HIGH:!aNULL, X25519:prime256v1, auto DH). Tests: the auth spec now covers channel_binding=require, a non-ASCII password, a role without a verifier and samenet. Passes on PG 14-19 with the archiver and archiver-multi schedules on PG 17.
…t fsm view on upgrade
- The ${NAME} expansion added with `let` failed existing specs that use shell
variables in exec strings (basic_operation.pgaf's ${PGDATA}): an unknown name
is now passed through untouched instead of being an error.
- pgautofailover--2.2--2.3.sql: the fsm view lacked the SELECT grant to
autoctl_node that a fresh 2.3 install gets from its blanket grant (found by
diffing the catalogs of an upgraded and a fresh database).
…riter node list) Modelled on PostgreSQL's backend_startup/pqcomm/postmaster: - absolute authentication deadline (--auth-timeout, default 30s, SIGALRM -> _exit) covering startup, TLS, DNS and SCRAM; TLS reads no longer loop forever - message length limits before allocation (1 KiB pre-auth, 1 MiB after) - child accounting by reaping in the accept loop, no SIGCHLD handler/counter - the "monitor" node list has ONE writer: a refresher process (only process using libpq), children are read-only and ask it over a datagram socket; requests coalesced, failures negative-cached, list warmed at startup and kept fresh by a heartbeat, fail closed after 1h without the monitor - HBA malformed line = reject everything; file reads capped - authenticate before revealing anything (unknown route reveals only after auth), generic rejections, sanitized logging, empty user rejected - --insecure required without --pgdata - after an error inside COPY the connection is closed - FETCH_FILE streams and serves only WAL segments and .history files; slot names/caps/atomic writes, DROP_REPLICATION_SLOT; keyword word boundaries - TLS: free on failure, no passphrase prompt, fstat on the opened key - per-route WAL segment size (archiver-walsegsize) instead of 16 MiB - ipaddr: clamp res_query answer length
… guards - archiver.serveport (register_archiver in_serveport, get_archiver_node OUT serveport) instead of a hard-coded 6543 - the upgrade script finds the old UNIQUE (nodehost, nodeport) by columns, not by a guessed generated name - health-check worker skips archiver rows (no postmaster at host:0); their health derives from their reports - get_basebackup_policy_for_archiver_group: group, formation, archiver default, seeded default; archiver_policy rows with a NULL policy fall through - get_archivers returns one row per (archiver, membership) - candidate priority > 0 refused on archiver rows; synchronous_standby_names accepts an archiver quorum member in the two-node path; perform_failover needs two DATA nodes (the group-size shortcut stays keyed on the FSM's count) - set_archiver_policy no longer resets the quorum when passed NULL - archiver_node.formationid cascades on delete - fsm view granted on upgrade; new function get_group_hosts(_hash)
…quorum name, serve port - BUG: the closed-segment hook named XLByteToSeg(end LSN), i.e. segment N+1 when N closes, so wal_archived() confirmed a segment the archiver did not have yet; now XLByteToPrevSeg - pg_receivewal child: SIGINT/SIGTERM set time_to_stop, bounded stop with SIGKILL escalation, reconnect sleep is interruptible (the failover-continuity hang), stale children/staging Postgres are verified before being signalled - application_name pgautofailover_standby_<nodeid> and synchronous flush, so an archiver with replication-quorum is a real synchronous standby of the primary - WAL segment size learned from the primary (archiver-walsegsize) instead of 16 MiB everywhere - local WAL cache is pruned to the oldest retained backup; scanner floor - `archiver backup now` - a detached and re-attached archiver starts its membership from a fresh state (the old node id no longer exists) - create archiver --serve-port, persisted, registered with the monitor and used by clients instead of the 6543 constant
…check, adding-an-archiver guide New specs: archiver_walsender_protocol, archiver_quorum, archiver_lifecycle, archiver_serve_port (archiver.sch / new archiver-dr.sch, wired into the CI matrix); ci/check-extension-upgrade.sh + CI job comparing an upgraded 2.2->2.3 catalog with a fresh 2.3 one; docs: rolling-upgrade procedure for adding an archiver to an existing cluster, operating notes, pg_walsender reference, CHANGELOG.
…archive_confirmed() - BuildForPrimaryNodeNodeActiveContext counts an ARCHIVING row that is a replication-quorum member and not stale (NodeIsUnhealthy) as a quorum standby. When the secondary dies, the archiver alone satisfies the quorum: the primary no longer drops to wait_primary (synchronous replication off) or blocks writes. With neither the secondary nor the archiver left it falls back to wait_primary as before. - pgautofailover.archive_confirmed(formation, group, walfile): true when the group has no archiver, else wal_archived(); hardened like the other functions; regress-tested.
…pecRole) `pg_autoctl node run` reads a node.ini `kind`; it now parses into a NodeSpecRole (POSTGRES, MONITOR, ARCHIVER, and the reserved WARM_STANDBY and PITR, rejected with "planned but not implemented yet") plus, for POSTGRES only, the PgInstanceKind used by keeper FSM guards. The two concepts no longer share one enum, ahead of the warm-standby and PITR milestones. FSM/keeper behaviour and every existing spelling and error message are unchanged; docs list the values.
- pg_autoctl archiver confirm --pgdata D <walfile>: exit 0 iff the archiver already holds the segment (non-segment files and clusters without archiver succeed; a monitor outage uses a tiny local cache of the last answer). - pg_autoctl writes archive_mode = on and archive_command = 'pg_autoctl archiver confirm ...' for new nodes (create postgres --archive-confirm on|off, default on); a keeper warning says when a running Postgres still has archive_mode = off and needs a restart. - PG_AUTOCTL_INITDB_OPTIONS adds initdb options (--wal-segsize=32 can only be chosen at initdb time).
archiver_fsm_quorum (primary stays primary with only the archiver as its quorum, then wait_primary with none, then back), archiver_wal_segsize (a 32 MB cluster end to end: capture, reported LSNs, pg_walsender SHOW/base backup/pg_receivewal, standby, failover), archiver_archive_command and _none (confirmation with the archiver up/down, the no-archiver and monitor-outage escapes); docs for archiver confirm.
…ring maintenance with a quorum archiver; document the FSM change - node.walsegsize, set_node_wal_segment_size(), get_group_wal_segment_size(), reported by keepers next to the system identifier; the archiver reads it from the monitor. - start_maintenance on the last secondary no longer sends the primary to wait_primary when a healthy quorum-member archiver remains. - New spec archiver_fsm_maintenance; docs and CHANGELOG for the archiver as quorum standby.
…t out of this PR pg_autoctl restore command wraps pg_walsender fetch-file (cli_archiver.c, restore_command.c/.h): restore_command = 'pg_autoctl restore command %f %p' resolves host/port/route/user from this node's own pg_auto_failover registration, or from a small cache file written once by --set-up for an ad hoc replica with no pg_auto_failover node of its own. The remote wal-receiver placement (a separate NodeSpecRole, monitor-side attach/detach, its own capture loop, and pg_walsender FETCH_FILE proxying to it) turned out to add more moving parts than it was worth: running another archiver node covers the same need with a design already proven in this branch, so that work is dropped rather than carried forward. This commit resets to the state right after the archive-command rename and re-applies only the WAL-segment-size fix (drop the vendored pg_receivewal writer, refuse to guess 16MB when the size is unknown) and pg_autoctl restore command on top of it. Verified on PG14: build + monitor installcheck (20/20), archiver_warm_standby, archiver_walsender_protocol, archiver_wal_segsize, archiver_fsm_quorum, archiver_fsm_maintenance, and archiver_archive_command all pass.
CI's style_checker caught what the local edits missed. No functional change.
…unctions LANGUAGE plpgsql vs sql) check-extension-upgrade.sh's own comparison query concatenated pg_proc. provolatile (the catalog's internal "char" type) with || against text, which Postgres can resolve to more than one operator -- cast it to text explicitly. That was masking a real, pre-existing mismatch between the two SQL files: 11 functions (archiver_remove_formation(bigint,text), create_basebackup_policy, create_rclone_config, pitr_next_command, pitr_queue_command, prune_archiver_wal, report_basebackup_remote_deleted, report_basebackup_synced, report_pitr_status, report_timeline_history, set_archiver_policy) are LANGUAGE sql in a fresh pgautofailover.sql install but LANGUAGE plpgsql (with an equivalent BEGIN/END-wrapped body) in the 2.2->2.3 upgrade script, so an upgraded and a fresh 2.3 catalog disagreed. Replaced each of the 11 CREATE FUNCTION blocks in pgautofailover--2.2--2.3.sql with the exact text of its pgautofailover.sql counterpart. Verified: ci/check-extension-upgrade.sh now reports catalogs identical (410 lines compared); PG17 build + monitor installcheck (20/20) pass.
…e failures were flakes No code change. See prior commit's diagnosis: the extension-upgrade catalog check was a real bug (fixed); archiver_lifecycle reproduced as a clean pass locally on the same commit; the other three (ssl test_007, archiver_ walsender_auth test_001, pytest single test_007_004) showed no connection to anything this branch touched. Re-running CI to see if they clear.
Found while reviewing the branch's diff against main for code quality; harmless doc-only leftover.
Vendored byte-for-byte from PostgreSQL's src/bin/pg_basebackup/pg_receivewal.c
at REL_14_STABLE, not a later branch: REL_17_STABLE's own walmethods.h pulls
in common/compression.h, which doesn't exist as of PG14's own header tree.
This project rebuilds pg_autoctl fresh against each of PG14 through PG19's
own headers, so the vendored copy has to compile against all of them --
PG14's simpler, pre-compression-refactor source does, and the replication
protocol itself is unaffected by that later refactor, so it still talks to
a PG17/18/19 primary exactly the same way a real, unmodified pg_receivewal
built from any of those versions would.
Two changes from upstream, both marked "PGAF:" at their exact location,
everything else byte-for-byte unchanged (see pg_receivewal.c's own header
comment for the full rationale):
1. main() renamed to pg_receivewal_main() and no longer static/the process
entry point -- meant to be called directly, once per (re)connection
cycle, by a caller that forks a real child process and calls it there
(a real fork(), not just a function call: pg_receivewal_main() behaves
like the CLI tool it's vendored from, including calling exit(), so it
needs its own process-wide state and address space).
2. stream_stop's callback now also exposes two hook points --
pgaf_wal_segment_closed_hook() when a segment finishes, and
pgaf_wal_progress_hook() otherwise -- for a caller to be notified of
WAL capture progress without a second wire protocol or polling the
output directory.
Nothing calls pg_receivewal_main() yet in this PR: it compiles into
pg_autoctl (linked, not yet referenced) as pure preparation for the
archiver feature (a later PR) to use it as the vendor/pg_receivewal/
pg_receivewal.c header comment already describes.
Not tested at runtime in this PR (there is no caller yet); its correctness
rests on (a) being byte-for-byte identical to upstream outside the two
marked hooks, and (b) the follow-up PR that adds a real caller and
end-to-end WAL-capture test coverage.
Verified: PG17 build (pg_autoctl links successfully with pg_receivewal_main()
present but unused) + monitor installcheck (19/19, unaffected); citus_indent
--check and ci/banned.h.sh pass (needs a .gitattributes -citus-style
exemption, same as pg_walsender's own vendored tar.c, since this is kept
byte-for-byte as upstream formats it rather than reindented to this
project's own style).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Additions to the existing src/bin/common/ shared library, adapted from PostgreSQL's own source rather than reimplemented, prep for a real replication-protocol server (a later PR) to authenticate clients by HBA address/network and by SCRAM, the same way postgres's own postmaster does, instead of accepting connections unauthenticated. Nothing here is used by pg_autoctl today. - ifaddr.c/.h (new): local interface address enumeration (getifaddrs()- based), the same code Postgres's own libpq/backend HBA matching uses to decide whether a connecting client's address falls inside a 'samehost' or 'samenet' pg_hba.conf entry. - scram.c/.h + scram_compat.h (new): a server-side SCRAM-SHA-256 verifier (parse/build a stored SCRAM secret, run the exchange), built on top of Postgres's own common/scram-common.c, common/hmac.c, common/cryptohash.c and common/saslprep.c -- scram_compat.h is the small adapter layer translating between those internal APIs and what a frontend-side caller (not a backend) can actually link against. - ipaddr.c/.h (extended): five new peer-address-matching entry points (ipaddrHostMatchesAddress, ipaddrInCIDR, ipaddrIsSameHostOrNet, ipaddrFindHostnamesFromAddress) ported from postgres's own src/backend/libpq/hba.c, for matching a numeric client address against an HBA file's host/CIDR/samehost/samenet entries. - file_utils.c/.h (extended): write_file_atomic(), a rename()-into-place atomic write for a file another process might read concurrently (a config/state file another running service polls). Makefile.common gains -lssl -lcrypto (scram.c's own crypto primitives) and -lresolv (ifaddr.c/ipaddr.c's own resolver use). Verified: PG17 build (pg_autoctl continues to link and behave identically; everything here is compiled but unreferenced) + monitor installcheck (19/19, unaffected); citus_indent --check and ci/banned.h.sh pass with no exceptions needed (all of it already follows this project's own style). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A new binary (src/bin/pg_walsender/, 48 files) speaking the real Postgres replication wire protocol: IDENTIFY_SYSTEM, SHOW, BASE_BACKUP, TIMELINE_ HISTORY, CREATE/READ/DROP_REPLICATION_SLOT, START_REPLICATION, and this project's own FETCH_FILE (a restore_command-friendly one-shot file fetch, alongside the standard commands). Hardened like a real network-facing server rather than a manual-testing toy: - pg_hba.conf-style HBA file (host/hostssl/hostnossl, CIDR/hostname/ samehost/samenet address forms, trust/scram-sha-256/reject), built on the common/ipaddr.c address-matching this PR adds. - SCRAM-SHA-256 authentication (common/scram.c) and TLS (OpenSSL), matching Postgres's own defaults. - Absolute per-connection auth deadline (--auth-timeout), oversize pre- auth message rejection, a per-route replication-slot cap -- the same postmaster-like hardening a network-facing server needs that a one-off script doesn't. - A routes file (archiver-routes.ini) mapping "<formation>/<group>" to a local storage root, and a small vendored ustar writer (vendor/tar.c, reformatted to this project's own style, unlike pg_receivewal's deliberately-untouched vendoring) for BASE_BACKUP's tar stream. - A single-writer "nodes list" refresher process + read-only children, the same nodehost cache design a real postmaster uses for pg_hba.conf's own cached DNS resolution, so a lookup never blocks a connection. Nothing in this PR calls pg_walsender from pg_autoctl yet: it is a fully functional, independently testable server on its own, with no archiver/ monitor integration -- that wiring is a later PR's job. Includes the two src/bin/common/ additions (ifaddr.c, scram.c/scram_compat.h) and extensions (ipaddr.c's five new address-matching entry points, file_ utils.c's write_file_atomic()) this server needs, since nothing else uses them yet either. Build wiring: src/bin/Makefile builds pg_walsender alongside pg_autoctl/ pgaftest; Dockerfile copies the binary with a bracket-glob COPY (kept optional so tests/upgrade can still build the Dockerfile against an old release's source tree that predates this file); top-level Makefile's docker-check/docker-indent now run correctly from a git worktree and also run ci/banned.h.sh (ci-lint). New test: tests/tap/specs/pg_walsender_standalone.pgaf -- a hand-crafted routes file, HBA rule, and archiver-systemid, real WAL captured off node1 by the stock OS pg_receivewal (not this project's own vendored copy, which is a separate PR), pg_walsender started directly by hand (no pg_autoctl archiver integration exists yet in this PR's stack). Covers IDENTIFY_SYSTEM, SHOW wal_segment_size, FETCH_FILE (byte-for-byte match against the captured segment), and START_REPLICATION (a second, real pg_receivewal streaming from pg_walsender itself, byte-for-byte match against what was captured). Verified: PG17 build + monitor installcheck (19/19, unaffected); citus_indent --check and ci/banned.h.sh pass; pg_walsender_standalone.pgaf passes end to end (4/4 steps) on PG17. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eivewal vendoring comment The vendored source (pg_receivewal.c's own header comment) is REL_14_STABLE; .gitattributes' own comment said REL_17_STABLE. Found while splitting the vendoring out into its own PR.
dimitri
force-pushed
the
archiving/m1-5-schema-and-wal-capture
branch
from
September 26, 2026 19:06
476264a to
ba307f6
Compare
…and-wal-capture-fixes # Conflicts: # src/bin/pg_walsender/repl_command.c # src/bin/pg_walsender/wal_dir_scan.c # src/bin/pg_walsender/wal_dir_scan.h
dimitri
force-pushed
the
archiving-integration-base
branch
from
September 27, 2026 10:20
ad8c3c5 to
7f80e41
Compare
…and-wal-capture-fixes # Conflicts: # .gitattributes # docs/ref/pg_walsender.rst # src/bin/common/file_utils.c # src/bin/common/scram.c # src/bin/common/scram.h # src/bin/common/scram_compat.h # src/bin/pg_walsender/Makefile # src/bin/pg_walsender/accept_loop.c # src/bin/pg_walsender/accept_loop.h # src/bin/pg_walsender/auth.c # src/bin/pg_walsender/auth.h # src/bin/pg_walsender/cmd_base_backup.c # src/bin/pg_walsender/cmd_base_backup.h # src/bin/pg_walsender/cmd_fetch_file.c # src/bin/pg_walsender/cmd_fetch_file.h # src/bin/pg_walsender/cmd_identify_system.c # src/bin/pg_walsender/cmd_identify_system.h # src/bin/pg_walsender/cmd_replication_slot.c # src/bin/pg_walsender/cmd_replication_slot.h # src/bin/pg_walsender/cmd_show.c # src/bin/pg_walsender/cmd_show.h # src/bin/pg_walsender/cmd_start_replication.c # src/bin/pg_walsender/cmd_start_replication.h # src/bin/pg_walsender/cmd_timeline_history.c # src/bin/pg_walsender/cmd_timeline_history.h # src/bin/pg_walsender/defaults.h # src/bin/pg_walsender/framing.c # src/bin/pg_walsender/framing.h # src/bin/pg_walsender/hba.c # src/bin/pg_walsender/hba.h # src/bin/pg_walsender/main.c # src/bin/pg_walsender/repl_command.c # src/bin/pg_walsender/repl_command.h # src/bin/pg_walsender/routes.c # src/bin/pg_walsender/routes.h # src/bin/pg_walsender/startup.c # src/bin/pg_walsender/startup.h # src/bin/pg_walsender/tar_stream.c # src/bin/pg_walsender/tar_stream.h # src/bin/pg_walsender/tls.c # src/bin/pg_walsender/tls.h # src/bin/pg_walsender/wal_dir_scan.c # src/bin/pg_walsender/wal_dir_scan.h # src/bin/pg_walsender/walsender.h # src/bin/pg_walsender/ws_util.c # src/bin/pg_walsender/ws_util.h # tests/tap/specs/pg_walsender_standalone.pgaf
…, rewrite restore_command.c to call fetch_client in-process The archiving-integration-base merge just landed brought in pg_walsender's review-round rewrite (flex/bison grammar, HBA tokenizer, CommandLine-based CLI, common/fetch_client.c), which along the way dropped the "monitor" HBA keyword and the refresher/monitor_hosts machinery behind it: PR #1193 has no monitor extension of its own to query, so that feature was deliberately lifted out of it to be reintroduced here, where the SQL (get_group_hosts/ get_group_hosts_hash) actually exists. This commit does that: - hba.c/hba.h: restore the "monitor" ADDRESS keyword and hba_lookup()'s routePath/monitorUriPath/refreshSockPath parameters, on top of the new tokenizer (ported from the pre-merge tree, not re-derived). - monitor_hosts.c/.h, refresher.c/.h: restored from their last reviewed state (the refresher already uses the project's PGSQL wrapper and a retry policy, per an earlier review round, before being deleted along with the rest of the monitor feature). - auth.c/.h: WsAuthConfig grows monitorUriPath/refreshSockPath back; hba_lookup() is called with the route's path so "monitor" can match. - accept_loop.c/.h: the parent forks/supervises/restarts the refresher child and creates its datagram socket, mirroring the postmaster/ refresher split documented in monitor_hosts.h. - cli_root.c: `serve` derives archiver-monitor.uri/archiver-refresh.sock from --pgdata, same as the other archiver-* paths. - defaults.h, ws_util.c/.h: PG_AUTOCTL_ARCHIVER_NODES_FILE and ws_auth_deadline_remaining_ms() restored (both were only removed because the feature using them was gone). - Makefile: build monitor_hosts.c/refresher.c again. Also, per review: `pg_autoctl restore command` (restore_command.c) used to execv() into `pg_walsender fetch-file`, which no longer exists (pg_walsender is a server binary, plus scram-secret, nothing else -- the FETCH_FILE client moved to src/bin/common/fetch_client.c in the pg_walsender-server PR). restore_command_run() now calls ws_fetch_file_client() directly, in-process; docs (archiving-details.rst, pg_autoctl_restore_command.rst) and the leftover stale src/bin/pg_walsender/fetch_client.c/.h copy (dead, unreferenced by the Makefile since the move) are updated/removed to match. Four pre-existing archiver test specs drove the raw `pg_walsender fetch-file` CLI directly (archiver_walsender_auth, archiver_lifecycle, archiver_walsender_protocol) or only had a stale comment about it (archiver_warm_standby); updated to go through `pg_autoctl restore command` instead, matching what a real restore_command line actually runs. Verified: PG17 docker build clean (zero warnings), monitor installcheck 20/6 tests green, citus_indent + banned.h.sh clean, and every archiver-* spec touched by this change re-run green: archiver_walsender_auth (all 8 steps, including the "monitor" keyword and node2's registration/drop follow-along), archiver_lifecycle, archiver_walsender_protocol, archiver_warm_standby, archiver_fsm_quorum, archiver_fsm_maintenance, archiver_archive_command, archiver_wal_segsize, and pg_walsender_standalone (#1193's own suite, unaffected).
…ack, not 0.0.0.0/0 Every connection this test makes to pg_walsender is node2 talking to its own server over 127.0.0.1 -- there never was a reason for the hand-written archiver-hba.conf to admit "0.0.0.0/0" (the entire internet, in principle). 127.0.0.1/32 is the actually correct, properly scoped rule for what this test really does, written by hand exactly as an operator running pg_walsender on its own (no archiver, no monitor, see this test's own header comment) would maintain it themselves. Verified: pg_walsender_standalone.pgaf still 5/5 green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on two extracted PRs, split out for independent review: #1191 vendors
pg_receivewalon its own, and #1193 addspg_walsenderas a standalone, independently-tested server with no archiver/monitor integration. This PR's base branch (archiving-integration-base) is the two of them merged; its own diff is now scoped to what's genuinely specific to the archiving feature (schema, keeper-side archiver logic, the FSM change, and the WAL-segment-size/archive_command wiring on top of the basepg_walsender).Archiving & Disaster Recovery, Milestones 1–5 of the design (schema through base-backup generation + policy), plus the docs coverage for all of it.
An archiver is a new node kind that captures a group's WAL continuously (
pg_receivewal, via the same replication-slot mechanism a standby already uses) and produces periodic base backups (live or replayed locally from its own WAL cache), independent of whether any standby is healthy or even present. A new standalone binary,pg_walsender, serves that captured data back out over a real (subset of the) PostgreSQL replication protocol, so a realpg_basebackup, a real streaming standby, or this project's ownrestore_commandcan all talk to an archiver with no archiver-aware code beyond finding it.This is not ready to merge — opening now for early review of direction and approach while the remaining work continues. See "Known gaps" below.
What's in this PR
pgautofailoverSQL schema for archivers —archiver,basebackup_policy,archiver_policy,archiver_wal,basebackup,wal_archived(), retention/pruning functions.service_archiver,archiver serve,pg_walsender(M2): the keeper-sideARCHIVINGFSM state;service_archiver.c's WAL-capture loop; a brand-new standalone binary (src/bin/pg_walsender/) implementing enough of the replication wire protocol from scratch (no frontend-linkable server-side implementation exists anywhere in PostgreSQL to link against) —IDENTIFY_SYSTEM,SHOW,BASE_BACKUP,TIMELINE_HISTORY,CREATE/READ_REPLICATION_SLOT,START_REPLICATION, and aFETCH_FILEside channel forrestore_command.pg_autoctl node runsupport forkind = archiver(M3).archiver_wal/wal_archived()tracking, re-pointingpg_receivewalat a new primary after a failover.live(realpg_basebackupagainst a healthy node) andreplay/volatile(extract the last backup, replay locally captured WAL forward against a throwaway staging instance, snapshot over loopback, discard) sources; policy-driven scheduling (frequency,onpromotion) and retention (maxcount,maxage); new CLI (pg_autoctl create/show/set basebackup-policy,--basebackup-policyoncreate archiver).pg_autoctl create postgres --from-archiver(bootstrap a new node straight from an archiver's cache) andFAST_FORWARDreusing an archiver as a WAL source during a multi-standby election, both via the same well-known-port resolution trick, no archiver-specific code in the ordinary standby-init/fast-forward paths themselves.SINGLEandPRIMARYforever (group_state_machine.c), and the replay staging instance failed to start under SSL (missing certs it has no reason to have).archiving-internals.rst) written as the technical reference/extension point for the milestones after this one; a new Operations page;ARCHIVINGstate coverage in the FSM docs (plus a full regeneration of the five FSM mermaid diagrams frompg_autoctl inspect fsm mermaid— real drift was found and fixed there, unrelated to archiving); fault-tolerance coverage; a rewritten intro with new architecture diagrams; and a small site-wide docs feature (click-to-zoom on figures, generalizing the zoom Mermaid diagrams already had).Known gaps — why this isn't ready yet
pg_walsender(all-new wire-protocol code) and the FSM fix ingroup_state_machine.c.conf.pychange (adds the click-to-zoom JS/CSS): Sphinx's incremental build doesn't reliably re-emit the<script>/<link>tags on every already-built page just becauseconf.pychanged — only pages whose own.rstsource changed get regenerated. If a localdocs/_buildpredates this PR,make -C docs htmlalone won't retrofit the zoom feature onto older pages; runmake -C docs clean html(or deletedocs/_build) once to pick it up everywhere.archiving-internals.rst's "Extension points" section is written to be where that work plugs in.Testing
New/updated
pgaftestspecs:archiver_wal_capture,archiver_bootstrap_and_fast_forward,archiver_basebackup_generation,archiver_basebackup_policy— all passing against a from-scratch--no-cacheDocker rebuild.citus_indent --checkclean.sphinx-build -W --keep-goingclean (no warnings, no broken references).