Skip to content

fix(sentinel): reopen the sentinel connection after the sentinel restarts - #3526

Merged
nkaradzhov merged 1 commit into
redis:masterfrom
maxymlyskov:fix/reopen-sentinel-after-restart
Oct 21, 2026
Merged

nkaradzhov merged 1 commit into
redis:masterfrom
maxymlyskov:fix/reopen-sentinel-after-restart

Conversation

@maxymlyskov

@maxymlyskov maxymlyskov commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

When the only sentinel a RedisSentinel watches restarts (an upgrade, a container restart), the client never rebuilds its sentinel connection or the +switch-master subscription on it, so it misses the next failover: no MASTER_CHANGE, getMasterNode() is undefined, and subscribe() listeners stay on the dead master until some command on the sentinel object forces a reset.

The sentinel client has reconnectStrategy: false, so the restart closes it and the reset that follows reaches the same sentinel. observe() then records currentSentinel from getSentinelNode(), which returns the closed client's node (L1587, L1881-L1887), and analyze() decides "sentinel node hasn't changed" and opens nothing. With several reachable sentinels the client moves to another one instead, so this hits a single configured sentinel or roots restored from the seeds.

Redis 8, one sentinel (quorum 1), docker restart of the sentinel then docker stop of the master, built dist of each side:

# master
after sentinel restart: pattern-subscribed connections on the sentinel: 0
sentinel reports master port 56401 | client getMasterNode undefined
topology-change events after failover: [] | subscriber received: []
# this branch
after sentinel restart: pattern-subscribed connections on the sentinel: 1
sentinel reports master port 56401 | client getMasterNode {"host":"127.0.0.1","port":56401}
topology-change events after failover: ["MASTER_CHANGE:56401"] | subscriber received: ["after-failover"]

observe() now reports no current sentinel while the sentinel client is closed, so analyze() reopens it like a changed node. I used isOpen rather than isReady (which getMasterNode() checks) so a sentinel client that is still connecting isn't opened twice, and kept the check out of getSentinelNode() because the public RedisSentinel.getSentinelNode() delegates to it. SENTINEL_CHANGE now also fires when the client reconnects to the same sentinel.

Added a case to sentinel/index.spec.ts next to 'Should recover after full outage'; it fails on master with "no MASTER_CHANGE after the failover". sentinel/index.spec.ts (102 passing) and the multi-db and other sentinel specs that import it pass on Linux, tsc and eslint are clean.


Checklist

  • Does npm test pass with this change (including linting)?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?

Note

Medium Risk
Changes Sentinel topology observation/reconnect logic on failover paths; behavior is targeted but affects when sentinel clients and pub/sub subscriptions are recreated.

Overview
Fixes a case where RedisSentinel could miss the next failover when the only configured Sentinel restarts (upgrade, container restart).

observe() now sets currentSentinel only when the persistent sentinel client is still isOpen. A closed client is no longer reported as the current sentinel, so analyze() treats the topology as needing a new sentinel connection and transform() recreates the client and +switch-master pub/sub subscription instead of assuming the node is unchanged.

Adds an integration regression test: single reachable sentinel, restart sentinel, then stop master and assert MASTER_CHANGE fires.

Reviewed by Cursor Bugbot for commit 678bbdf. Bugbot is set up for automated code reviews on this repo. Configure here.

…arts

The sentinel client is created with reconnectStrategy false, so when the
sentinel it watches restarts the client closes and a reset observes the
same node again. observe() reported that node from the closed client, so
analyze() saw no sentinel change and the client with its +switch-master
subscription was never rebuilt. The next failover then produced no
MASTER_CHANGE and pub/sub listeners stayed on the old master until a
command forced a reset.

Report no current sentinel while the sentinel client is closed, so the
reset opens a new one. The check is isOpen rather than isReady, so a
sentinel client that is still connecting is not opened a second time.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-20T21:24:12.083533Z
We couldn’t complete this review. Comment @codex review to try again.
678bbdf PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@nkaradzhov nkaradzhov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM.

@nkaradzhov
nkaradzhov merged commit a9484c8 into redis:master Oct 21, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants