[rebaseandpanic]

Scenario

The client uses Sentinel mode (ServiceName) and is connected to primary A.

1. Sentinel fails over to node C, often at an address the client has never seen (e.g. a recreated pod on Kubernetes).
2. A is demoted to a replica, but the client's connections to A stay up.
3. The +switch-master message never reaches the client.

Today the client sees the demotion (INFO replication reports role:slave) but only re-elects among the nodes it knows; all of them are replicas, and sentinel is not asked again. Writes fail until restart:

No connection (requires writable - not eligible for replica) is active/available to service this operation

#3130 fixed the case where the new primary is already known; this is the unknown-address case it left untouched. We hit this class of failure in production (Valkey + Sentinel on Kubernetes).

Fix

On the sentinel connection's heartbeat (every second), using local state only: if the primary sentinel last reported is connected but reports itself as a replica, run one recovery attempt in the background:

• ask sentinel who the primary is; no answer: try again next heartbeat;
• sentinel still names the demoted node: re-read its role with one INFO (no full reconfigure);
• otherwise switch, as +switch-master would have.

Skipped while a reconfigure is running, after dispose, or when disconnected (the existing reconnect timer handles that). No public API change, no periodic sentinel polling. This goes slightly beyond the sentinel client guidelines ("Handling reconnections"), which rely on sentinel disconnecting the demoted node's clients.

Tests

In-process sentinel (InProcessSentinelServer) plus SetReplicaOf on the toy server:

• demotion to an unknown address: fails without the fix, passes with it;
• sentinel lagging: sentinel is still polled, but no reconfigure on every heartbeat.

Also checked against real Valkey 8.1 + Sentinel in docker: the client switched ~1 s after noticing the demotion.

Refs #1891, #2487, #1670

Checklist

• I fully and freely contribute this code in accordance with the project license (and am legally able to do so)
• I take responsibility for this contribution's quality and correctness, including any portions produced with AI assistance (see CONTRIBUTING.md).

[mgravell]

looking now 👍

[mgravell]

Thanks - the demotion-in-place gap is real and the heartbeat approach here is sound. I've reviewed and pushed a follow-up commit (b4d330a) to this branch addressing a few things found in review; summary below so the changes have context.

Race: the switch could reshape the connection and then have its reconfigure refused

RecoverDemotedPrimary peeked at activeConfigCause and then called SwitchPrimary, but SwitchPrimary rewrote EndPoints (Clear/TryAdd) and retired servers before ReconfigureAsync ran its own CAS on that field. A reconfigure starting in that window (the child's role-change SetConfig, a late +switch-master, the initial-connect switch) would refuse the switch's reconfigure after the connection had been reshaped - leaving retired/dropped servers and no primary for at least a heartbeat, on the exact path meant to restore writability. Two concurrent SwitchPrimary calls could also interleave their endpoint rewrites.

Fix: SwitchPrimary now claims the reconfigure gate before touching anything, via new TryClaimReconfigure/ReleaseReconfigureClaim helpers and a claimed handover parameter on the internal ReconfigureAsync (which releases the claim however it exits, including the disposed-throw path, so it can't leak). A lost race now mutates nothing and returns false; the heartbeat simply retries. This also serializes concurrent switches, and fixes the same latent race for the pre-existing callers (+switch-master handler, reconnect timer, restore handler).

The switch re-queried sentinel after the guards validated a different answer

RecoverDemotedPrimary validated one sentinel answer, then SwitchPrimary asked again and acted on whatever came back - under a flapping sentinel that could re-trigger the per-heartbeat full-reconfigure storm the INFO-recheck branch was built to avoid. SwitchPrimary now takes an optional reportedPrimary, and the recovery path hands over the answer its guards evaluated - which also drops 1-2 redundant blocking sentinel round-trips from the claimed section.

Smaller items

• LastException: a successful demotion recovery now clears it (as OnManagedConnectionRestored does, and for the same reason), once the reported primary is connected as a primary - so a stale soft-connect discovery failure stops being folded into every timeout message after the client is healthy again.
• Memory visibility: currentSentinelPrimaryEndPoint is now volatile - it's written by SwitchPrimary on worker/handler threads and read by the heartbeat demotion check, and previously relied on surrounding Interlocked traffic incidentally publishing it.
• Duplication: the disposed-child cleanup shared by the +switch-master handler and the heartbeat sweep is extracted into ForgetDisposedChild.
• Test determinism: DemotedPrimaryDoesNotReconfigureWhileSentinelStillNamesIt no longer uses wall-clock quiet detection + a fixed 4s window + an exact event count (flaky on loaded CI, and a mandatory ~5.5s per green run). It now waits for a stretch of three sentinel polls with zero configuration changes, re-baselining if a straggler reconfigure lands - the buggy behavior (a reconfigure per poll) can never produce a clean stretch, so it still fails deterministically.
• Docs: docs/Configuration.md now describes how a sentinel-managed connection tracks failovers (+switch-master, reconnect polling, and the new demotion-in-place re-resolution), per repo convention for user-visible behavior changes.

Validated locally: full Release build clean on all TFMs (warnings-as-errors), the in-process demotion tests green across repeated runs, all 58 sentinel integration tests green against the docker topology, and the long-running SentinelFailoverTests end-to-end failover green against real sentinels.