lb: route strongly-consistent requests to the tablet leader

For a table in a strongly-consistent keyspace, one replica of each tablet is the
Raft leader that coordinates its writes and its linearizable reads. Promote that
replica to the front of the query plan, so the request reaches it directly
instead of being forwarded there by another coordinator.

The server publishes the leader nowhere: not in schema, not in system.tablets, and
a Raft re-election can move it at any time. The one place it appears is the
tablets-routing-v2 payload, which lists the leader first. The driver caches the
replicas in that order, which is what makes replicas[0] of a cached tablet the
leader.

should_route_to_leader() gates this on the keyspace being strongly consistent
(ConsistencyMode::Global) and the consistency level not being ONE/LOCAL_ONE --
those are satisfied by any single replica, so preferring the leader would only
concentrate load on it without buying any consistency.

leader_to_prefer() calls that predicate, and then answers a second question: is
the leader actually known? Tablet::known_leader() decides that.

That a cached tablet's replica list is leader-ordered at all does not need
checking per request. The server serves statements against a strongly-consistent
table through a path that only ever attaches a tablets-routing-v2 payload, never
a v1 one; strongly-consistent tables are enabled only once the whole cluster
supports tablets-routing-v2; and a keyspace changing its consistency mode purges
its cached tablets. So a tablet cached for a strongly-consistent keyspace came
from a v2 payload, which lists the leader first.

What is checked is that a tablet is cached for the token at all -- before the
first payload for a table arrives, its replica set is filled in from the token
ring, which knows nothing about tablets, let alone which replica leads one --
and that every replica resolved to a known node. Unresolved replicas are dropped
from the tablet's replica list, so if the leader is the one that could not be
resolved, the first remaining replica is a follower. That does not make the
tablet version unreadable: an unresolved replica does not invalidate the cached
mapping, and re-requesting it would only cost a round trip and return the same
still-unresolvable replicas. It is transient -- perform_maintenance() re-resolves
those replicas, or drops the tablet, at the next topology refresh.

Either way the request keeps plain token-aware routing and the server does the
forwarding.

known_leader() is reached through ReplicaLocator::tablet_leader_for_token(), so
the whole decision costs one tablet lookup, instead of resolving the same tablet
once for its replicas and once more for anything else asked of it.

Finally it decides whether that leader is a host this policy is willing to
contact. The leader outranks distance: it is tried ahead of nearer
replicas, a leader in a remote datacenter ahead of one in the preferred rack,
because a nearer replica would only add a forwarding hop and, the table being
globally consistent, keeping the request in one datacenter buys no consistency.
It does not override the policy's own filter, though: the leader must satisfy the
pick predicate (which includes liveness), and it must sit in a datacenter the
policy would use at all -- the preferred one, any of them when none is
preferred, or a remote one when datacenter failover is permitted. With a
preferred datacenter and failover disabled, a leader elsewhere is skipped, the
request takes plain token-aware routing, and the server does the forwarding.

This mirrors the Python driver's TokenAwarePolicy contract (scylladb/python-driver#913),
so the two drivers agree on how leader awareness interacts with locality
preferences.

Only the leader is promoted. The ordering of the remaining replicas is left
alone, so retries after the leader still spread; deterministic replica ordering
stays reserved for LWTs, which need it to reduce Paxos contention.

Adds unit tests, including the datacenter/rack preference matrix, the two cases
where the leader is unknown -- a table with no cached tablet at all, and a
tablet whose leader did not resolve -- and the case where the leader is known
but not alive, which exercises leader_to_prefer's predicate rather than
known_leader: the plan falls back to the other replicas, trying the leader only
as an absolute last resort like any other unreachable node. Also adds a
RawTablet test constructor, and widens update_tablets to pub(crate).

Author: dawmd

scylla/src/cluster/state.rs

@@ -656,7 +656,7 @@ impl ClusterState {
         &self.known_nodes
     }
 
-    pub(super) fn update_tablets(&mut self, raw_tablets: Vec<(TableSpec<'static>, RawTablet)>) {
+    pub(crate) fn update_tablets(&mut self, raw_tablets: Vec<(TableSpec<'static>, RawTablet)>) {
         let replica_translator = |uuid: Uuid| self.known_nodes.get(&uuid).cloned();
 
         for (table, raw_tablet) in raw_tablets.into_iter() {