Leader heartbeat succeeds but ReadIndex/MsgApp requests timeout

[Goend]

I have a cluster with three nodes:

10.10.1.4

10.10.1.5

10.10.1.6 (leader)

etcd version: v3.4.18

In this scenario, the leader (10.10.1.6) can send heartbeats to 10.10.1.4 without timeout. However, when 10.10.1.4 tries to perform a linearizable read request, it times out.this situation keeps occurring on the etcd cluster nodes, and each timeout only lasts for about ten seconds. I believe this is due to a ReadIndex timeout.

At that time, the logs clearly showed an I/O timeout between the leader and follower:

On 10.10.1.6 (leader):

rafthttp: lost the tcp streaming connection with peer.... (stream MsgApp v2 reader)
rafthttp: lost the tcp streaming connection with peer.... (stream MsgApp reader)
failed to read .. on stream MsgApp v2 (read tcp 10.10.1.6:34674 -> 10.10.1.4:2380: io timeout)

On 10.10.1.4:

rafthttp: lost the tcp streaming connection with peer.... (stream MsgApp v2 writer)

But during this period, leader heartbeats did not time out. leader node dont have this message,

failed to send out heartbeat on time

ETCD_HEARTBEAT_INTERVAL=250

Question

• Do leader heartbeats and ReadIndex/MsgApp messages use the same underlying TCP connection?

• If they share the same connection, why could heartbeats still succeed while MsgApp and ReadIndex appeared to timeout?

• If they use different connections, what differences between them could cause this situation? and possible solutions

By checking the disk and WAL log synchronization latency, it seems unlikely to be related to disk performance, and etcd has no logs indicating slow disk issues.

When the issue occurred, I found that only part of the TCP connections were closed.

I suspect that the issue might be related to the TCP keepalive parameters, which could cause the leader’s heartbeat frequency to be very low, so the underlying TCP connections are able to stay alive.
···
--grpc-keepalive-min-time '5s' (default)
--grpc-keepalive-interval '2h' (default)
--grpc-keepalive-timeout '20s' (default)

net.ipv4.tcp_keepalive_time = 30
net.ipv4.tcp_keepalive_probes = 8
net.ipv4.tcp_keepalive_intvl = 3
···
Will these gRPC parameter settings affect the gRPC connections which are used by passing raft message between etcd servers?

Thanks for any suggestions.

[ahrtr]

etcd version: v3.4.18

Please try with a new etcd version: 3.5.x or 3.6.x.

[Goend]

Thank you for the explanation, but I am unable to test the new version in the target environment.

[Goend]

Actually, the environment contains up to 8 NUMA nodes and 128 CPUs. After binding etcd to specific NUMA nodes, it stabilized and no longer experiences disconnections. If a scenario occurs where overall node resource usage is low, disk I/O utilization is minimal, network performance is good, and CPU usage is not high, but etcd performance remains unstable, you can try binding CPU cores. Additionally, referring to golang/go#28808, reducing the value of GOMAXPROCS can help stabilize etcd performance.