Skip to content

Fix health check worker never acknowledging ProcSignalBarrier (DROP DATABASE hang) - #1196

Merged
dimitri merged 1 commit into
mainfrom
fix/health-check-worker-procsignalbarrier-hang
Oct 13, 2026
Merged

dimitri merged 1 commit into
mainfrom
fix/health-check-worker-procsignalbarrier-hang

Conversation

@dimitri

@dimitri dimitri commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Summary

LatchWait() -- the sole sleep point in both the health-check launcher's and every per-database health-check worker's main loop -- calls WaitLatch() + ResetLatch() but never CHECK_FOR_INTERRUPTS().

A background worker only acknowledges a ProcSignalBarrier (the mechanism DROP DATABASE uses to wait until every backend in the cluster has released smgr state) from inside CHECK_FOR_INTERRUPTS() -- WaitLatch() wakes the process up on the pending signal, but doesn't process it itself.

Practical effect: as long as any pgautofailover health-check worker is running anywhere in the cluster (which is always, by default, once the extension is loaded via shared_preload_libraries), every DROP DATABASE statement anywhere in that cluster blocks forever, repeatedly logging:

LOG:  still waiting for backend with PID ... to accept ProcSignalBarrier
STATEMENT:  DROP DATABASE IF EXISTS "..."

This includes pg_regress's own schedule setup, so make installcheck against a live Postgres 17 instance hangs indefinitely before running a single test.

Root cause

health_check_worker.c's LatchWait() is the common per-iteration sleep point for both worker loops. Standard Postgres background worker loops call CHECK_FOR_INTERRUPTS() right after waking from WaitLatch()/ResetLatch() precisely so that pending interrupts -- including a ProcSignalBarrier -- get processed promptly. This worker never did.

Fix

Add CHECK_FOR_INTERRUPTS() right after ResetLatch() in LatchWait(), matching the idiom used by every other Postgres background worker.

Reproduction / verification

Reproduces on an unmodified checkout against Postgres 17.11: CREATE DATABASE and ordinary queries work fine, but any DROP DATABASE against a cluster with shared_preload_libraries = 'pgautofailover' loaded hangs forever once the extension's health-check worker has started.

With the fix: rebuilt and reinstalled the extension against a live PG17.11 instance; the full monitor regression schedule (20 tests) and isolation test suite (6 tests) now complete and pass via make installcheck, where they previously hung.

Test plan

  • make installcheck completes (was hanging indefinitely before this fix)
  • All existing monitor regression tests still pass
  • All existing isolation tests still pass

…lcheck hang)

LatchWait() -- the sole sleep point in both the health-check launcher's and
each per-database health-check worker's main loop -- called WaitLatch() and
ResetLatch() but never CHECK_FOR_INTERRUPTS(). A background worker only
acknowledges a ProcSignalBarrier (the mechanism DROP DATABASE uses to wait
for every backend to release smgr state) from inside CHECK_FOR_INTERRUPTS(),
so as long as any health-check worker was running, every DROP DATABASE
anywhere in the cluster -- including pg_regress's own schedule setup --
blocked forever, logging "still waiting for backend ... to accept
ProcSignalBarrier" every 5s.

Found while working on an unrelated change, but reproduces on an unmodified
checkout against Postgres 17.11, so it's a pre-existing issue: as long as
any pgautofailover health-check worker is running anywhere in the cluster
(which is always, by default), every DROP DATABASE blocks forever.

Confirmed fixed: the full monitor regression schedule (20 tests) and
isolation suite (6 tests) now complete and pass against a live PG17
instance.
@dimitri
dimitri merged commit 5c7f7e7 into main Oct 13, 2026
147 of 148 checks passed
@dimitri
dimitri deleted the fix/health-check-worker-procsignalbarrier-hang branch October 13, 2026 16:44
@dimitri dimitri self-assigned this Oct 13, 2026
@dimitri dimitri added the bug Something isn't working label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant