Repository navigation
Postgres 17+ CDC continuity: synchronized_standby_slots, demotion slot cleanup (fixes #1188) - #1197
Merged
Merged
Conversation
dimitri
force-pushed
the
cdc-continuity-slots
branch
from
October 13, 2026 18:17
2f40d4f to
59b3678
Compare
added 8 commits
October 14, 2026 12:35
… slot cleanup (#1188) Lets a failover-enabled logical replication slot survive a pg_auto_failover promotion on Postgres 17+, by automating the pieces Postgres itself needs: - Monitor: new pgautofailover.synchronized_standby_slots(formation, group) SQL function, added to the extension's current unreleased schema revision (still '2.3', the last tagged release being v2.2 -- folded into the existing pgautofailover--2.2--2.3.sql migration alongside the other schema changes already accumulated there since v2.2, rather than bumping to a new 2.4; this project only bumps the extension version at actual release time). Computed like synchronous_standby_names but listing every current failover candidate's physical slot name, so a failover-enabled logical slot only advances once every promotable node has replayed that far. - pg_autoctl: fetches and applies that value on the primary via the same fsm_apply_settings() path that already maintains synchronous_standby_names; enables sync_replication_slots by default on 17+ nodes (alongside the existing hot_standby_feedback default), gated by a new pg_setup_synchronized_standby_slots_supported() version check. - Fix issue #1188: a node rejoining as standby after an interrupted demotion could keep its own stale (failover AND NOT synced) logical slots, permanently blocking the slot-sync worker. Extend the existing demotion slot cleanup to drop those too. New `pg_autoctl show synchronized-standby-slots` command (mirrors `show standby-names`), and `pg_autoctl watch`'s header now shows a live "CDC Slots" count next to "Sync Standbys". wal_level stays user-managed, as does logical slot creation itself -- CDC tooling (Debezium, etc) still owns slot lifecycle; pg_auto_failover only keeps the sync infrastructure correct across role changes. Documented in a new docs/cdc.rst page (including a comparison against Patroni, CloudNativePG, EDB PGD, and repmgr's own support for this Postgres feature, and why a single merged CDC stream across Citus workers needs a separate, standalone proxy component rather than an extension of pg_auto_failover itself) and a faq.rst entry. Covered by a new monitor regression test (synchronized_standby_slots.sql) and a pgaftest spec (cdc_continuity.pgaf) exercising a clean failover and the #1188 demote_timeout scenario -- run for real against `pg_auto_failover:pg17` Docker containers via `pgaftest run`, all 5 steps passing. That run surfaced and fixed two issues in the test itself (not the feature): setup{} restarting the primary's Postgres after it already had a synchronous standby attached (the risky pattern basic_operation.pgaf warns about), and a scenario that re-promoted the slot's original creator a second time, which separately hits a genuine Postgres-level limitation (a node round-tripping back to primary can get its own slot stuck in an unresolvable sync loop and lose it) -- now documented as a known limitation rather than conflated with the #1188 regression check. Also verified the real ALTER EXTENSION upgrade path end to end: installed at 2.2, updated to 2.3, confirmed synchronized_standby_slots() is callable immediately after, against a live Postgres 17 instance.
…ut runbook pg_auto_failover now creates nodes with wal_level=logical by default (required for CDC continuity's failover-enabled replication slots), patched into the existing default GUC settings array at node creation time. A --wal-level replica flag (also settable later via the keeper config file) opts back out for deployments that don't need logical decoding. docs/cdc.rst gains the manual runbook for flipping wal_level on an already-running cluster via maintenance mode, and docs/orchestrated-operations.rst explores a generalized monitor/client primitive for driving that kind of whole-cluster, multi-step change as a single declared, resumable, cancellable operation. cdc_continuity.pgaf is simplified to rely on the new default instead of manually toggling wal_level in its setup block.
… doc The spec hardcoded image "pg_auto_failover:pg17", a tag that only ever existed on a local Docker build used for manual validation. CI builds and loads pgaf:run-pg<N> and passes it via PGAF_IMAGE, which an explicit image "..." in the spec overrides -- so CI tried to pull a nonexistent "pg_auto_failover:pg17" from a registry and failed. Dropping the explicit image line falls back to PGAF_IMAGE, same as every other spec in the suite. Also removes the orchestrated-operations design exploration from the user-facing docs (toctree entry and the cdc.rst cross-reference) and deletes the page -- not ready to present as part of this feature's documentation.
…ce restarts disable maintenance's FSM transition for a standby always ends by calling ensure_postgres_service_is_running() (via primary_rewind_to_standby(), reused by fsm_restart_standby()), so Postgres comes back up on its own as part of rejoining -- no separate start/restart step needed, the operator only has to stop it after editing the config. Also notes that pg_autoctl's own Postgres controller explicitly steps aside during enable maintenance (local_postgres_unlink_status_file(), "so that this process won't interfere with whatever the user is doing during maintenance (such as stop Postgres)"), so a plain pg_ctl stop here is already safe and doesn't race pg_autoctl's supervision.
That section was internal research material, not something that belongs in user-facing documentation.
…g secondary GroupListCandidates() only filters by candidatePriority > 0, regardless of the node's current reportedState -- a candidate the monitor has pushed back to catchingup (lagging, missed a reporting window, ...) was still listed, which would make a failover-enabled logical slot's progress hostage to that node recovering. Filter the candidate list down to AutoFailoverNode.reportedState == secondary. Deliberately checks reportedState, not goalState: an extra standby beyond what number_sync_standbys needs can sit at goalState catchingup indefinitely for quorum bookkeeping alone while genuinely being a fully caught-up secondary, and that node belongs in the list -- this is why the existing regression test's "both standbys are candidates" assertion is unchanged (sss_s2's goalstate is catchingup there, but it had already reported secondary). Adds a new regression scenario exercising the actual lag case: a standby that self-reports catchingup drops out of synchronized_standby_slots, and reappears once it reports secondary again, verified against a live PG17 instance.
The FAQ still said "you still need to set wal_level = logical yourself" -- stale since wal_level now defaults to logical at node creation (this same PR). Point to the new default/opt-out and rollout sections instead. Also stop lying in the C source: DEFAULT_GUC_SETTINGS_FOR_PG_AUTO_FAILOVER hardcoded wal_level to 'replica' with a comment saying it gets patched to 'logical' by default -- so the array never actually produced 'replica' in the common case, only in the --wal-level replica opt-out. Flip the hardcoded literal to 'logical' (the real default) and only patch the array when a non-Citus node explicitly opted out to 'replica', instead of unconditionally copying and patching on every node creation just to re-assert the same default value.
Replaces the four manual steps per node (enable maintenance, edit config, stop Postgres, disable maintenance) with the new pg_autoctl restart postgres --maintenance [--allow-failover] command from a separate, in-progress PR: edit the config, then one command per node. --allow-failover is still required for the primary, same gate as enable maintenance itself.
dimitri
force-pushed
the
cdc-continuity-slots
branch
from
October 14, 2026 12:36
a938c6a to
c65be23
Compare
added 5 commits
October 14, 2026 12:36
…docs pg_autoctl restart postgres (#1198) is now on main after rebasing this branch on top of it. Turn the plain-text command name into a proper :ref: to its manual page, and point to the Operations guide's own restart_postgres section for more worked examples of the same command (including the --exec case, which doesn't apply to a plain wal_level edit but is worth knowing about).
…by_names siblings Same methodology as the restart postgres PR (#1198) review: this PR introduced four near-duplicates of pre-existing functions, each differing only in which monitor/SQL function to call and a setting name used in error messages or JSON output. Extracted the shared logic into one new helper per layer, keeping each pair's own thin wrapper: - node_active_protocol.c: synchronized_standby_slots() duplicated the "build a comma-separated pgautofailover_standby_<id> list" loop from synchronous_standby_names(). Extracted AppendStandbySlotNames(). - monitor.c: monitor_synchronized_standby_slots() duplicated monitor_synchronous_standby_names()'s entire body (same SQL execution, same error handling) bar the SQL function name and setting name. Extracted monitor_get_group_setting_from_sql(). - cli_show.c: cli_show_synchronized_standby_slots() duplicated cli_show_standby_names()'s entire body (same fetch, same JSON/plain printing) bar the monitor function and setting name. Extracted cli_show_monitor_computed_setting(). - fsm_transition.c: fsm_apply_settings() had two copies of the same "fetch a monitor-computed setting into a buffer, or fall back to a default when there's no monitor" block, one for synchronous_standby_names and one for synchronized_standby_slots. Extracted fsm_fetch_monitor_setting(), taking the monitor function as a function pointer (both have the exact same signature). No behavior change. Verified: the monitor's own synchronized_standby_slots regression test still produces byte-identical output against a live PG17 instance, and a real 2-node cluster still correctly shows synchronous_standby_names, synchronized_standby_slots, and wal_level after the refactor.
The "No N of M" limitation bullet was stale: it described a permanently lagging/unreachable candidate as permanently stalling decoding, before the fix that filters synchronized_standby_slots down to candidates currently reporting themselves as secondary. A lagging candidate now drops out of the list on its own (pushed to catchingup by the monitor) and reappears once it catches back up, so the remaining limitation is narrower: there's still no knob to require only some of several simultaneously-healthy candidates, not that a single bad node blocks everything indefinitely.
Step 1.b was just "b. ::" with a bare command block, and the explanation of what the command does trailed after it instead of introducing it -- a reader hit the command with zero context first. Restructured to match step 2's existing pattern: explain what's about to happen, then show the command.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lets a
failover => truelogical replication slot survive a pg_auto_failover promotion on Postgres 17+, by automating the pieces Postgres itself needs:pgautofailover.synchronized_standby_slots(formation, group)SQL function, computed likesynchronous_standby_namesbut listing every current failover candidate's physical slot name, so a failover-enabled logical slot only advances once every promotable node has replayed that far. Added to the extension's current unreleased schema revision (still'2.3', last tagged releasev2.2) by folding into the existingpgautofailover--2.2--2.3.sqlmigration alongside the other schema changes already accumulated there since v2.2 — no version bump, matching this project's convention of only bumping the extension version at actual release time.fsm_apply_settings()path that already maintainssynchronous_standby_names; enablessync_replication_slotsby default on 17+ nodes (alongside the existinghot_standby_feedbackdefault), gated by a newpg_setup_synchronized_standby_slots_supported()version check.failoverAND NOTsynced) logical slots, permanently blocking the slot-sync worker. Extends the existing demotion slot cleanup to drop those too.pg_autoctl show synchronized-standby-slotscommand (mirrorsshow standby-names), andpg_autoctl watch's header now shows a live "CDC Slots" count next to "Sync Standbys".wal_levelnow defaults tologicalon every node pg_auto_failover creates, so CDC continuity works out of the box; a new--wal-level replicaflag (also settable later via the keeper config) opts back out for deployments that don't need logical decoding. Logical slot creation itself stays user-managed — CDC tooling (Debezium, etc) still owns slot lifecycle; pg_auto_failover only keepswal_leveland the sync infrastructure correct across role changes.docs/cdc.rstdocuments the manual runbook for turningwal_level = logicalon for an already-running cluster (via maintenance mode), and newdocs/orchestrated-operations.rstexplores a generalized monitor/client primitive for driving that kind of whole-cluster, multi-step change as a single declared, resumable, cancellable operation — a design exploration, not implemented in this PR.See the new
docs/cdc.rstpage for the full picture, including a comparison against Patroni, CloudNativePG, EDB PGD, and repmgr's own support for this Postgres feature, a documented known limitation (a node round-tripping back to primary a second time can hit a Postgres-level sync-loop limitation unrelated to this change), and why a single merged CDC stream across Citus workers needs a separate, standalone proxy component rather than an extension of pg_auto_failover itself.Testing
src/monitor/sql/synchronized_standby_slots.sql, part of the regularmake installcheckschedule.tests/tap/specs/cdc_continuity.pgaf, covering a clean failover and the Demotion can leave stale failover slots behind, permanently blocking the slot sync worker on the new standby #1188 demote_timeout scenario, now relying on the newwal_level=logicaldefault instead of toggling it manually in setup. Run for real againstpg_auto_failover:pg17Docker containers viapgaftest run— all 5 steps pass end to end.ALTER EXTENSIONupgrade path end to end: installed at 2.2, updated to 2.3, confirmedsynchronized_standby_slots()is callable immediately after, against a live Postgres 17 instance.Test plan
make installcheckpasses, including the newsynchronized_standby_slotstestpgaftest run tests/tap/specs/cdc_continuity.pgafpasses (5/5 steps) against real Docker containers, including with the new defaultwal_level=logicalpg_autoctl show synchronized-standby-slotsandpg_autoctl watchmanually verifiedCREATE EXTENSION ... VERSION '2.2'→ALTER EXTENSION ... UPDATE TO '2.3'verified to install the new function correctly