Repository navigation
fix(sentinel): reopen the sentinel connection after the sentinel restarts - #3526
Merged
nkaradzhov merged 1 commit intoOct 21, 2026
Merged
Conversation
…arts The sentinel client is created with reconnectStrategy false, so when the sentinel it watches restarts the client closes and a reset observes the same node again. observe() reported that node from the closed client, so analyze() saw no sentinel change and the client with its +switch-master subscription was never rebuilt. The next failover then produced no MASTER_CHANGE and pub/sub listeners stayed on the old master until a command forced a reset. Report no current sentinel while the sentinel client is closed, so the reset opens a new one. The check is isOpen rather than isReady, so a sentinel client that is still connecting is not opened a second time.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When the only sentinel a
RedisSentinelwatches restarts (an upgrade, a container restart), the client never rebuilds its sentinel connection or the+switch-mastersubscription on it, so it misses the next failover: noMASTER_CHANGE,getMasterNode()is undefined, andsubscribe()listeners stay on the dead master until some command on the sentinel object forces a reset.The sentinel client has
reconnectStrategy: false, so the restart closes it and the reset that follows reaches the same sentinel. observe() then recordscurrentSentinelfromgetSentinelNode(), which returns the closed client's node (L1587, L1881-L1887), and analyze() decides "sentinel node hasn't changed" and opens nothing. With several reachable sentinels the client moves to another one instead, so this hits a single configured sentinel or roots restored from the seeds.Redis 8, one sentinel (quorum 1),
docker restartof the sentinel thendocker stopof the master, built dist of each side:observe() now reports no current sentinel while the sentinel client is closed, so analyze() reopens it like a changed node. I used
isOpenrather thanisReady(whichgetMasterNode()checks) so a sentinel client that is still connecting isn't opened twice, and kept the check out ofgetSentinelNode()because the publicRedisSentinel.getSentinelNode()delegates to it.SENTINEL_CHANGEnow also fires when the client reconnects to the same sentinel.Added a case to sentinel/index.spec.ts next to 'Should recover after full outage'; it fails on master with "no MASTER_CHANGE after the failover". sentinel/index.spec.ts (102 passing) and the multi-db and other sentinel specs that import it pass on Linux, tsc and eslint are clean.
Checklist
npm testpass with this change (including linting)?Note
Medium Risk
Changes Sentinel topology observation/reconnect logic on failover paths; behavior is targeted but affects when sentinel clients and pub/sub subscriptions are recreated.
Overview
Fixes a case where
RedisSentinelcould miss the next failover when the only configured Sentinel restarts (upgrade, container restart).observe()now setscurrentSentinelonly when the persistent sentinel client is stillisOpen. A closed client is no longer reported as the current sentinel, soanalyze()treats the topology as needing a new sentinel connection andtransform()recreates the client and+switch-masterpub/sub subscription instead of assuming the node is unchanged.Adds an integration regression test: single reachable sentinel, restart sentinel, then stop master and assert
MASTER_CHANGEfires.Reviewed by Cursor Bugbot for commit 678bbdf. Bugbot is set up for automated code reviews on this repo. Configure here.