Implement TABLETS_ROUTING_V2 - #913
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe driver adds tablets routing v2 negotiation, tablet-version blocks on prepared executions, connection-level routing, and custom-payload caching. Schema parsing tracks strongly consistent keyspaces, while token-aware plans can prioritize tablet leaders. Unit and integration tests cover protocol behavior, metadata refresh, shard routing, payload decoding, and leader selection. Sequence Diagram(s)sequenceDiagram
participant Client
participant ResponseFuture
participant HostConnection
participant ScyllaServer
participant ClusterMetadata
Client->>ResponseFuture: execute prepared query
ResponseFuture->>HostConnection: borrow connection with routing context
HostConnection->>ScyllaServer: send EXECUTE with tablet version block
ScyllaServer-->>ResponseFuture: return tablet routing payload when stale
ResponseFuture->>ClusterMetadata: cache tablet metadata
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
There was a problem hiding this comment.
Pull request overview
Implements Scylla TABLETS_ROUTING_V2 negotiation and request/response handling in the driver, including encoding the per-request tablet_version_block, parsing the new tablets-routing-v2 custom payload (with tablet_version), and adding leader-aware routing behavior for strongly-consistent keyspaces.
Changes:
- Add TABLETS_ROUTING_V2 protocol feature negotiation and ensure V2 subsumes V1 in STARTUP options.
- Attach a per-request
tablet_version_blockto EXECUTE messages on V2-negotiated connections; parse and cache V2 tablet routing payloads includingtablet_version. - Introduce strongly-consistent keyspace detection (Scylla-only) and leader-first routing in
TokenAwarePolicy; add unit and integration coverage.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_tablets.py | Adds unit tests for tablet_version_block encoding and for storing tablet_version on Tablet. |
| tests/unit/test_protocol_features.py | Adds negotiation tests ensuring V2 is preferred over V1 in STARTUP options. |
| tests/unit/test_policies.py | Adds unit tests for leader-aware routing behavior (leader-first, fallback, and non-SC cases). |
| tests/integration/standard/test_tablets_routing_v2.py | Adds opt-in end-to-end tests validating negotiation, payload decoding, and wire behavior against a V2-capable Scylla cluster. |
| cassandra/tablets.py | Adds tablet_version_block helpers and extends Tablet to store tablet_version. |
| cassandra/query.py | Adds memoization for routing-key token computation via Statement.routing_token(). |
| cassandra/protocol.py | Extends ExecuteMessage to optionally append tablet_version_block byte when present. |
| cassandra/protocol_features.py | Adds TABLETS_ROUTING_V2 constant and negotiation logic; makes V2 mutually exclusive with V1 in STARTUP. |
| cassandra/pool.py | Adds per-pool V2 capability detection and reuses memoized routing token to avoid repeated hashing. |
| cassandra/policies.py | Adds leader-first routing for strongly-consistent keyspaces when tablet replicas are available. |
| cassandra/metadata.py | Adds KeyspaceMetadata.strongly_consistent and populates it from Scylla system_schema.scylla_keyspaces. |
| cassandra/cluster.py | Computes/attaches tablet_version_block per connection and parses V2 routing payloads based on the serving connection’s negotiated features. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| token = self._routing_token | ||
| if token is None: | ||
| token = token_class.from_key(routing_key) | ||
| self._routing_token = token | ||
| return token |
There was a problem hiding this comment.
This statement is technically correct:
- We and Cassandra used to support other partitioners than Murmur3. Looks like Cassandra still supports them, though it recommends Murmur3: https://cwiki.apache.org/confluence/spaces/CASSANDRA2/pages/120732668/Partitioners.
- We do have our own partitioner - the CDC partitioner - which applies only to CDC tables and is different than Murmur3.
Theoretically, it would be possible to have either two Cassandra clusters that use different partitioners, or have two Scylla clusters where you have a CDC log table created in the usual way, and the other where the CDC log table is "faked" via CREATE TABLE and thus uses Murmur3. I see that you added the _routing_token_class field but that does not protect us from a case where somebody sends two requests in parallel - routing_token is not protected with a mutex so there is a potential race condition.
I don't know if this is significant enough for us to care. On the other hand, the statement now contains data which is dependent on the context (i.e. the cluster it is sent to) which is something that I think should be avoided in principle. I think it would be better if the token were computed in one place on the statement execution path and then passed manually via e.g. function arguments and not hidden in the Statement class which probably should be immutable.
@sylwiaszunejko What do you think?
There was a problem hiding this comment.
I think it would be better if the token were computed in one place on the statement execution path and then passed manually via e.g. function arguments and not hidden in the Statement class which probably should be immutable.
That would be ideal if possible, we should do something to avoid potential race condition
There was a problem hiding this comment.
I looked into it, and it looks like we cannot easily avoid computing the token multiple times.
If we discard the current approach, the only other option I see is computing it early-on and passing on along the control flow. Unfortunately, along the way, we go through make_query_plan, which is part of the public API and so we cannot modify it -- the user is allowed to implement their own LoadBalancingPolicy and override it. The existing arguments don't really allow for hiding the token there, but it's necessary for the function.
However, it might be possible to reduce the number of computations to two. I'll see what can be done about it.
The alternative is to give up on it and accept computing the token three times.
There was a problem hiding this comment.
Implemented the two-computation approach in v8.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (2)
tests/unit/test_policies.py (1)
1043-1082: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueOptional: test name/docstring don't match what's actually exercised.
This test asserts the "no tablet_version (V1)" path, but the production code (
make_query_plan) never readstablet.tablet_version; leader-first is gated solely onks_meta.strongly_consistent, which is set toFalsehere. So the test passes because of thestrongly_consistent=Falseflag, not becausetablet_version=None. Consider renaming/reframing it (or settingstrongly_consistent=Truewithtablet_version=None) so the test actually guards the behavior its name implies; otherwise it overlaps withtest_no_leader_routing_for_eventually_consistent_keyspace.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/test_policies.py` around lines 1043 - 1082, The test name and docstring in test_no_leader_routing_without_tablet_version do not match the behavior actually being exercised, because make_query_plan only gates leader-first routing on keyspace metadata strong consistency, not Tablet.tablet_version. Update the test so it truly covers the intended case by either renaming/reframing it to reflect strongly_consistent=False, or by setting cluster.metadata.keyspaces['ks'].strongly_consistent to True while keeping tablet_version=None to verify the V1 tablet path; keep the assertions aligned with the TokenAwarePolicy routing behavior.cassandra/policies.py (1)
510-514: 🚀 Performance & Scalability | 🔵 TrivialOptional: Cache the child query plan to prevent double invocation and potential inconsistency.
child.make_query_plan(keyspace, query)is invoked at line 512 to filter replicas and is called again at line 551 to build the final plan. If the child policy is stateful (e.g.,RoundRobinPolicyorDCAwareRoundRobinPolicy), the second invocation may advance internal pointers and return a different host sequence, causing the filtering logic to diverge from the execution order and introducing unnecessary overhead.♻️ Suggested approach
Materialize the child plan once and reuse it:
- if tablet is not None: - replicas_mapped = set(map(lambda r: r[0], tablet.replicas)) - child_plan = child.make_query_plan(keyspace, query) - - replicas = [host for host in child_plan if host.host_id in replicas_mapped] + child_plan = list(child.make_query_plan(keyspace, query)) + if tablet is not None: + replicas_mapped = set(map(lambda r: r[0], tablet.replicas)) + replicas = [host for host in child_plan if host.host_id in replicas_mapped]Update the subsequent loop (line 551) to iterate over the cached
child_planinstead of re-invokingchild.make_query_plan(...).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cassandra/policies.py` around lines 510 - 514, Cache the result of child.make_query_plan(keyspace, query) in the policy logic so it is only evaluated once, then reuse that same child_plan for both the replica filtering in the tablet block and the final plan construction loop. Update the query planning flow in the policy method that handles tablet replicas to iterate over the cached child_plan instead of calling child.make_query_plan(...) a second time, which avoids stateful policy divergence and redundant work.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cassandra/cluster.py`:
- Around line 5055-5075: The ExecuteMessage tablet_version_block is only being
set in the pooled connection path, so control-connection fallback can miss the
mandatory V2 trailing byte. Update _query_control_connection() in
cassandra/cluster.py to use the same tablets_routing_v2 check and
_compute_tablet_version_block(self.query) logic as the ExecuteMessage handling
in the main send path, placing it after self._connection = connection and before
connection.send_msg(...) so V2 control connections always include the block and
non-V2 ones do not.
- Around line 5212-5231: The tablet-routing cache is using only
self.query.keyspace when adding tablets, which can store entries under a None
key and miss cache hits for session-level keyspaces. Update the tablet handling
in ResponseFuture to use the effective keyspace consistently, matching
_compute_tablet_version_block() by falling back to self.keyspace when
self.query.keyspace is unset, and then pass that resolved keyspace into
metadata._tablets.add_tablet().
In `@cassandra/metadata.py`:
- Around line 2658-2660: The fallback in metadata schema refresh is too broad
because the except block in the system_schema.scylla_keyspaces read path catches
every Exception, which masks unexpected failures. Narrow the handler around the
code in metadata.py that logs “Could not read system_schema.scylla_keyspaces” so
it only catches the specific expected driver/server schema-unavailable error(s),
and let other refresh errors propagate. Keep the existing debug log and exc_info
behavior for the expected case, but do not use a blanket Exception catch in this
branch.
In `@cassandra/pool.py`:
- Around line 451-453: The tablets_routing_v2 property in Pool should not
iterate the live _connections view directly, because concurrent
replacement/shutdown can mutate it and closed connections can incorrectly keep
V2 enabled. Snapshot the current connections first, then compute the flag only
from still-open/live connections before checking each connection’s
features.tablets_routing_v2.
In `@cassandra/query.py`:
- Line 278: The cached routing token in Statement is being reused without
considering token_class, which can cause the wrong ring/token to be used across
sessions or clusters. Update the token cache logic in Statement-related routing
code so _routing_token is keyed by token_class as well, and make the
lookup/reuse path in the affected routing methods distinguish tokens by the
partitioner class used. Ensure the changes cover the Statement fields and the
routing/token selection flow that currently reads and writes _routing_token.
---
Nitpick comments:
In `@cassandra/policies.py`:
- Around line 510-514: Cache the result of child.make_query_plan(keyspace,
query) in the policy logic so it is only evaluated once, then reuse that same
child_plan for both the replica filtering in the tablet block and the final plan
construction loop. Update the query planning flow in the policy method that
handles tablet replicas to iterate over the cached child_plan instead of calling
child.make_query_plan(...) a second time, which avoids stateful policy
divergence and redundant work.
In `@tests/unit/test_policies.py`:
- Around line 1043-1082: The test name and docstring in
test_no_leader_routing_without_tablet_version do not match the behavior actually
being exercised, because make_query_plan only gates leader-first routing on
keyspace metadata strong consistency, not Tablet.tablet_version. Update the test
so it truly covers the intended case by either renaming/reframing it to reflect
strongly_consistent=False, or by setting
cluster.metadata.keyspaces['ks'].strongly_consistent to True while keeping
tablet_version=None to verify the V1 tablet path; keep the assertions aligned
with the TokenAwarePolicy routing behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 587d3541-27be-4132-8dda-c90e66cff85c
📒 Files selected for processing (12)
cassandra/cluster.pycassandra/metadata.pycassandra/policies.pycassandra/pool.pycassandra/protocol.pycassandra/protocol_features.pycassandra/query.pycassandra/tablets.pytests/integration/standard/test_tablets_routing_v2.pytests/unit/test_policies.pytests/unit/test_protocol_features.pytests/unit/test_tablets.py
|
v2:
Tested against scylladb/scylladb#30291 (scylladb/scylladb@58ab555). |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cassandra/metadata.py`:
- Around line 805-810: Update the documentation for the strongly_consistent
metadata field so it matches the behavior in _is_strongly_consistent() and the
new tests. In the keyspace metadata docstring near strongly_consistent, change
the description to state that only ScyllaDB keyspaces with consistency set to
global are marked True, while local, eventual, and non-ScyllaDB cases remain
False. Keep the public contract aligned with the implementation by tightening
the wording around strongly_consistent.
In `@tests/integration/standard/test_tablets_routing_v2.py`:
- Around line 279-294: The global monkeypatch on
ProtocolFeatures.add_startup_options in the test setup can leak if Cluster
creation or shutdown fails. Move the restore logic for add_startup_options into
a nested finally that always runs after the patch is applied, and keep the
cleanup tied to the cluster/session lifecycle in the same test helper around
Cluster.connect and cluster.shutdown. Use the existing
ProtocolFeatures.add_startup_options, Cluster, and session setup block to ensure
the original value is restored no matter which step raises.
- Line 41: The module-level skip logic in the test setup is too broad because
the `except Exception as exc` block can mask unrelated startup problems. Narrow
the handler around the cluster start path to only skip on the specific expected
unsupported-feature/startup failure, using the same setup flow where the cluster
is initialized, and re-raise any other exception so real regressions still fail
fast.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 712ffcf1-f346-4fd7-931d-303a5bc26920
📒 Files selected for processing (12)
cassandra/cluster.pycassandra/metadata.pycassandra/policies.pycassandra/pool.pycassandra/query.pycassandra/tablets.pydocs/scylla-specific.rsttests/integration/standard/test_tablets_routing_v2.pytests/unit/test_metadata.pytests/unit/test_policies.pytests/unit/test_response_future.pytests/unit/test_tablets.py
🚧 Files skipped from review as they are similar to previous changes (6)
- tests/unit/test_tablets.py
- cassandra/policies.py
- cassandra/query.py
- tests/unit/test_policies.py
- cassandra/tablets.py
- cassandra/cluster.py
| # advertisement of TABLETS_ROUTING_V2_EXPERIMENTAL. | ||
| 'experimental_features': ['lwt', 'udf', 'strongly-consistent-tables'], | ||
| }) | ||
| except Exception as exc: |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Find existing integration-test patterns and harness exception types to catch specifically.
rg -n -C3 "def use_cluster|pytest.skip\\(|Could not start a Scylla cluster|except .*Exception" tests cassandraRepository: scylladb/python-driver
Length of output: 50378
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== tests/integration/standard/test_tablets_routing_v2.py =="
nl -ba tests/integration/standard/test_tablets_routing_v2.py | sed -n '1,120p'
echo
echo "== tests/integration/__init__.py (use_cluster) =="
nl -ba tests/integration/__init__.py | sed -n '421,560p'
echo
echo "== integration test patterns for skip/start failures =="
rg -n -C2 "pytest\.skip\(|Could not start|except .*Exception|DependencyException|NoHostAvailable|OperationTimedOut|FeatureNotSupported|Unsupported" tests/integration tests | sed -n '1,220p'Repository: scylladb/python-driver
Length of output: 260
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== tests/integration/standard/test_tablets_routing_v2.py =="
sed -n '1,140p' tests/integration/standard/test_tablets_routing_v2.py | cat -n
echo
echo "== tests/integration/__init__.py (use_cluster) =="
sed -n '421,560p' tests/integration/__init__.py | cat -n
echo
echo "== relevant integration skip / exception patterns =="
rg -n -C2 "pytest\.skip\(|Could not start|DependencyException|Unsupported|OperationTimedOut|NoHostAvailable|except .*Exception" tests/integration tests | sed -n '1,220p'Repository: scylladb/python-driver
Length of output: 32197
Limit the module-level skip to the expected startup failure. Catching Exception here can hide setup bugs, port conflicts, and config regressions behind a skip. Narrow this to the cluster-start/unsupported-feature failure you expect, or re-raise everything else.
🧰 Tools
🪛 Ruff (0.15.20)
[warning] 41-41: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/integration/standard/test_tablets_routing_v2.py` at line 41, The
module-level skip logic in the test setup is too broad because the `except
Exception as exc` block can mask unrelated startup problems. Narrow the handler
around the cluster start path to only skip on the specific expected
unsupported-feature/startup failure, using the same setup flow where the cluster
is initialized, and re-raise any other exception so real regressions still fail
fast.
Source: Linters/SAST tools
There was a problem hiding this comment.
I think this is a valid suggestion. The startup should only be allowed fail due to the strongly-consistent-tables feature being unsupported. Can we detect this?
There was a problem hiding this comment.
I only did some quick research, but I don't see any way to detect this without inspecting the logs.
|
v2.1:
Also tested against scylladb/scylladb#30291 (scylladb/scylladb@58ab555). |
| # ScyllaDB-only: per-keyspace consistency option. The column is null for | ||
| # eventually-consistent keyspaces (and the whole table is absent on Cassandra | ||
| # and on Scylla versions without strongly-consistent tablets). | ||
| _SELECT_SCYLLA_KEYSPACES = "SELECT keyspace_name, consistency FROM system_schema.scylla_keyspaces" |
There was a problem hiding this comment.
It's a real pity that we need to perform yet another fetch from system_schema in order to obtain this information - especially when strong consistency is still experimental, people will be paying for support for a feature which they not only do not use, but can't use.
Maybe we should have considered sending some information about whether a table uses strong consistency or not in the prepared statement. I think this could also apply to information like the partitioner and whether a table uses tablets or not. This is, of course, out of scope.
There was a problem hiding this comment.
That's a good point. However, if we want to avoid it, we need to modify the server-code before we can merge this PR since we need some way to learn that a table is strongly consistent. I can start working on it in parallel of course.
There was a problem hiding this comment.
I think that modifications like those I propose would require another round of design review. I don't think we should be implementing this at the moment. The other suggestion about skipping the query to scylla_keyspaces should make the current PR palatable enough, we will only pay the cost on experimental clusters with strong consistency enabled.
There was a problem hiding this comment.
Alright, we can remove this code for the time being. Unfortunately, the consequence will be ditching all leader awareness from the driver, but it shouldn't be difficult to add it back later on. If that's acceptable, I can proceed with the changes.
There was a problem hiding this comment.
Unfortunately, the consequence will be ditching all leader awareness from the driver
That would defeat the whole point of this PR.
What I meant by my "we should not be implementing this at the moment" is that we should not be working on further extending the protocol in the way as I suggested in my first message. I did not mean to drop the code that this conversation is attached to (cassandra/metadata.py, lines 2590-2593), we need it to distinguish whether it's a strongly consistent table or not and whether to do leader awareness routing or not.
In #913 (comment) I suggested that we can skip issuing the query if we know that the cluster does not support strong consistency. If we do this, we will not incur the cost for regular users who are not testing strong consistency at the moment. While not great, I don't think an additional metadata query is tragic; we still have some time before release of strong consistency to address it (if there will be a need to address it at all, given the python-over-rust effort).
There was a problem hiding this comment.
Gated behind the protocol extension in v3. Leaving the discussion as open since this is still something we might want to improve. It should be easier to remember this way until we create an issue.
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cassandra/connection.py`:
- Around line 1224-1226: Preserve backward compatibility in
send_msg()/encode_message handling: the new protocol_features argument passed
from self._protocol_handler.encode_message can break custom protocol handler
subclasses that still implement the older signature. Update the call path in
CassandraConnection.send_msg (and any related encode_message wrapper) to either
accept and ignore extra keyword arguments via **kwargs or add a compatibility
shim that only passes protocol_features when the handler supports it, keeping
existing subclasses working.
In `@tests/integration/standard/test_tablets_routing_v2.py`:
- Around line 155-156: The assertion in the tablet routing test is checking the
bound supports_tablet_routing method object instead of its return value. Update
the loop over self.session._pools.values() to invoke supports_tablet_routing()
on each pool before asserting, so the test validates the negotiated feature
rather than the method reference itself.
- Around line 69-77: The setup in the cluster bootstrap path can leak resources
if Cluster.connect() or _create_schema() fails before teardown_class runs. Wrap
the logic in the class setup flow around cls.cluster, cls.session, and
cls._create_schema so any exception triggers immediate cluster shutdown/cleanup
before re-raising, and make sure the cleanup is tied to the existing setup
method that initializes the Cluster and session.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 359bd0fd-f061-419c-83e3-12b654fcc40e
📒 Files selected for processing (16)
cassandra/cluster.pycassandra/connection.pycassandra/metadata.pycassandra/policies.pycassandra/pool.pycassandra/protocol.pycassandra/protocol_features.pycassandra/query.pycassandra/tablets.pydocs/scylla-specific.rsttests/integration/standard/test_tablets_routing_v2.pytests/unit/test_metadata.pytests/unit/test_policies.pytests/unit/test_protocol_features.pytests/unit/test_response_future.pytests/unit/test_tablets.py
✅ Files skipped from review due to trivial changes (1)
- docs/scylla-specific.rst
🚧 Files skipped from review as they are similar to previous changes (10)
- tests/unit/test_protocol_features.py
- cassandra/tablets.py
- cassandra/protocol_features.py
- tests/unit/test_response_future.py
- tests/unit/test_metadata.py
- cassandra/query.py
- tests/unit/test_policies.py
- cassandra/policies.py
- cassandra/pool.py
- tests/unit/test_tablets.py
|
v3:
|
|
v4:
What remains to be fixed is the violation of a public API pointed out by coderabbit here: #913 (comment). |
|
The new CI failure seems unrelated to these changes, but I didn't find it in the Issues on GitHub or Jira. Summary of it (since the logs are going to be deleted soon. I have them saved, but let's persist at least this here too): @Lorak-mmk can you take a look and confirm it's unrelated? Then I can create an issue for it. Btw. I asked Claude to summarise the full logs. This is what it said: Bottom lineThe single failure is unrelated to your PR. It's a Client Routes / Private Link (NLB) test, and your branch doesn't touch that feature or test at all ( What failedThe test is a 4-stage topology-churn scenario (start 3 nodes → bootstrap 3 → decommission the original 3 → verify session survives). It fails in Stage 4, at test_client_routes.py: right after url = "http://%s:10000/v2/client-routes" % contact_point # Scylla REST API, port 10000
...
response = urllib.request.urlopen(req) # <-- returns 500Port 10000 is Scylla's own REST API, so the 500 originates from the Scylla node (release 2026.1.8), not from the test's Noise to ignore
|
|
Doesn't sound like something you could cause. @sylwiaszunejko is a better person to look at this, she implemented PrivateLink tests IIRC. |
| # If we have a leader hint, yield it first -- but respect the child | ||
| # policy's own filter: never front-run a host the child policy would | ||
| # exclude (e.g. one a custom policy reports as IGNORED). | ||
| if (leader_host is not None and leader_host.is_up |
There was a problem hiding this comment.
This also front-runs REMOTE leaders. With DCAwareRoundRobinPolicy(..., used_hosts_per_remote_dc>0), local CLs like LOCAL_QUORUM/LOCAL_SERIAL can go remote before local replicas. Please preserve the child distance buckets, or only prefer local leaders for local CLs.
There was a problem hiding this comment.
This is something we probably need to discuss a bit more. With global consistency, there's only one leader in the cluster and every request (if it's not a read with CL \in {ONE, LOCAL_ONE}) must go through it. Sending it to a local replica will result in a bounce to the leader, which seems wasteful. However, that's in conflict with the policy itself.
I see three solutions:
- Ignore the policy and for strongly-consistent requests, route them all to the leader (at least for consistency global).
- Honour the policy and accept that bounces will happen. Create a new policy that respects leader awareness.
- Modify the policy: exempt strongly-consistent requests from the existing constraints and route them to the leader.
Since the true goal of this PR is introducing leader awareness in the driver, i.e. really routing requests to the leader, option 2, we cannot ignore this problem. Otherwise, this patch would simply build a dead infrastructure only future PRs could leverage.
@dkropachev @piodul what do you think? My recommendation is going with either approach 1, or approach 3. Approach 2 might result in an even bigger diff, and since this PR is already quite large, that would make it harder to review. Leader awareness is an experimental feature, so this code won't affect active users of the driver. This way, we achieve our goal and can improve it in a follow-up.
There was a problem hiding this comment.
Policy should not be ignored, ever. User can for example have a policy that explicitly routes to specific node.
Leader-awareness should be implemented in the default policy, not bypass it. So option 1 is out.
Imo it makes no sense to prioritize DC-awareness with global strongly consistent tables. DC/rack awareness is about:
- Performance (local network is cheaper), latency: both are negated if coordinator needs to forward to global leader anyway.
- Correctness. If you send
LOCAL_QUORUMrequests only to local DC, you have some consistency guarantees you don't have if you also send to other DCs. This again doesn't matter for global strongly consistent tables.
So imo LBP should return global leader for global strongly consistent tables, even if DC awareness is present and leader is in remote DC. Other options are strictly worse.
There was a problem hiding this comment.
I agree, we just need to adjust policy code to route properly for strong consistent queries, not ignore it.
Maybe my reading of the core code is wrong, but I don't see LOCAL_* in cl breaking reaching the leader.
Now given that and fact that some clients still want to route queries to a "closest" nodes we need to give then this opportunity.
Let's have an option on policies or policy that controls if policy preferre leader despite it being in another dc or not and have default to send to the leader anyways (as long as it is not IGNORED)
There was a problem hiding this comment.
I agree, we just need to adjust policy code to route properly for strong consistent queries, not ignore it.
Maybe my reading of the core code is wrong, but I don't see LOCAL_* in cl breaking reaching the leader. Now given that and fact that some clients still want to route queries to a "closest" nodes we need to give then this opportunity. Let's have an option on policies or policy that controls if policy preferre leader despite it being in another dc or not and have default to send to the leader anyways (as long as it is not
IGNORED)
Could you elaborate on this new option? Should it be a configuration option of the driver, a parameter of the LBP (in the code), or something else?
My current understanding is like this:
- We should only modify the code of the default LBP to return the leader for strongly consistent queries, regardless of what the policy is normally supposed to do. The "default LBP" is ambiguous, but I guess the one @Lorak-mmk meant is TokenAwarePolicy (since we need the token to know where to route a request. Also, cf. Listing 1 below). We should not modify any other LBP.
- We should document that strongly consistent requests (to keyspaces with consistency global) will be routed directly to the leader, regardless of its data center placement, as long as it's not marked as
IGNORED. - The user can override this behaviour by implementing a custom LBP.
Please confirm this or point out what I should change in this summary and how.
Listing 1:
python-driver/cassandra/cluster.py
Lines 272 to 275 in d99dc46
| routing_key = query.routing_key | ||
| if routing_key is None: | ||
| # Non-token-aware query: the server won't version-check it, so skip | ||
| # generating random bits and just send 0. | ||
| return 0 | ||
|
|
There was a problem hiding this comment.
Why? Didn't you already check that when computing routing_token?
There was a problem hiding this comment.
I'm not sure if I understand your comment here, but I changed the code in v11 -- now we pass routing_key as an argument of the function. Note that it can still be None.
| # Fall back to a random block (a version miss on the server) when the | ||
| # token could not be computed (no token map) or when this table has no | ||
| # cached tablets (vnode tables, or tablet tables on cold start). | ||
| if routing_token is None or not self.cluster.metadata._tablets.table_has_tablets(keyspace, table): | ||
| return random_tablet_version_block() | ||
|
|
||
| tablet = self.cluster.metadata._tablets.get_tablet_for_key(keyspace, table, routing_token) |
There was a problem hiding this comment.
You perform double lookup. Why not use get_tablet_for_key, and handle None case?
I think that this tablet metadata is mutable in Pythin Driver, so tablet could disappear between those lines.
There was a problem hiding this comment.
I tried to handle this in v11. Let me know if you see something's still wrong or I misunderstood the comment.
| # If we have a leader hint, yield it first -- but respect the child | ||
| # policy's own filter: never front-run a host the child policy would | ||
| # exclude (e.g. one a custom policy reports as IGNORED). |
There was a problem hiding this comment.
I wonder if this is the right approach. Python having nested LBPs makes it so difficult..
For example, I think it could be ignored if DC fallback is disabled (correct me if I'm wrong).
But user could want to disable DC fallback for eventual consistency (very reasonable for correctness and latency), while still wanting to use strongly consistent tables.
I'm not sure what the good solution here is tbh.
|
Turning the PR into a draft to avoid running CI checks (I'm going to first just rebase the PR for a cleaner diff later on). |
|
v10:
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (4)
cassandra/policies.py:538
Noneis not the effective consistency level here.Session._create_response_futureresolves it to the selected execution profile atcassandra/cluster.py:3031, but the policy receives the original statement, so the defaultLOCAL_ONEpath is treated as leader-requiring and pins ordinary reads to the leader. Thread the resolved consistency level into query planning (without mutating a potentially shared statement) so theONE/LOCAL_ONEcarve-out also applies to profile defaults.
effective_cl = query.consistency_level
prefer_leader = effective_cl not in (ConsistencyLevel.ONE, ConsistencyLevel.LOCAL_ONE)
tests/integration/standard/test_tablets_routing_v2.py:45
- Catching every startup exception turns unrelated regressions—invalid generated configuration, a broken server binary, networking failures, or a
use_clusterbug—into a successful module-wide skip, so CI can silently run none of the new end-to-end coverage. Only skip when the experimental feature is specifically known to be unsupported, and let other startup failures propagate.
except Exception as exc:
pytest.skip("Could not start a Scylla cluster with the "
f"'strongly-consistent-tables' experimental feature: {exc}",
allow_module_level=True)
cassandra/cluster.py:3067
- This hashes every routed request before knowing whether either new consumer needs the token. With the default
TokenAwarePolicy, the policy hashes the key again; on Cassandra/vnode tables the pool has no sharding info and never uses this precomputed value, so the hot path goes from one partition-key hash to two. Precompute only when a cached tablet makes the token useful; cold/vnode requests can retain the pool's existing fallback hash.
if routing_key is not None:
token_map = self.cluster.metadata.token_map
if token_map is not None:
routing_token = token_map.token_class.from_key(routing_key)
docs/scylla-specific.rst:181
- This describes any
consistencyoption as strongly consistent, but the implementation explicitly treats only_ConsistencyMode.GLOBALas strong (cassandra/metadata.py:2655-2658) and mapslocalto false. Document the currently supportedconsistency = 'global'condition so users do not expect leader routing for other modes.
**Leader-aware routing for strongly-consistent tables.** Tables in a
strongly-consistent keyspace -- one created with a ``consistency`` option and
backed by Raft -- have a tablet leader that coordinates operations. For those
|
I didn't expect CI to run, but it did. From what I can tell, the failure is unrelated to the changes. It looks to be more related to the run itself: in |
Add per-connection negotiation of the TABLETS_ROUTING_V2 extension, the successor to TABLETS_ROUTING_V1. When the server advertises it in the SUPPORTED response, the driver echoes it back during STARTUP to opt in; a driver that negotiates v2 does not negotiate v1. While the feature is experimental the wire name carries the `_EXPERIMENTAL` suffix (TABLETS_ROUTING_V2_EXPERIMENTAL), and the server only advertises it when started with the `strongly-consistent-tables` experimental feature enabled. Also add the trailing tablet_version_block byte to the EXECUTE message body. The server reads exactly one such byte per EXECUTE on a connection that negotiated the extension, so the encoder writes one whenever the connection did -- coalescing an unset value to 0 -- and none otherwise. Later commits fill in the value from the cached tablet version. Deciding this from the connection's negotiated features rather than from the message is what lets one ExecuteMessage be sent, unmodified, on connections that negotiated differently.
Store the server-provided 64-bit tablet_version on each cached Tablet and add helpers to encode it into the one-byte tablet_version_block exchanged on the wire. The version stays None until learned: on a cold start, and on a TABLETS_ROUTING_V1 connection, which never reports one. * Tablet.from_row normalizes the version to an unsigned 64-bit value. The server sends an unsigned hash, but the driver deserializes the payload field as a signed long, so the raw value can come back negative; masking to [0, 2**64) keeps the nibble extraction in choose_tablet_version_block consistent with the server's unsigned layout. * choose_tablet_version_block() packs a randomly chosen block index in the high nibble and that block's value in the low nibble, matching the server's locator::compare_tablet_version_block layout. Blocks are indexed from the least significant bits, so block i covers bits [i*4, i*4 + 4) of the version. A random index avoids any shared mutable counter on the hot path while still probing every nibble often enough to detect a server-side version change quickly. * random_tablet_version_block() returns a random byte for cold start, when no version is cached yet.
With TABLETS_ROUTING_V2 the server returns, on a tablet_version mismatch, the tablet's replica set plus the new tablet_version, so the driver can keep its routing cache fresh without the per-response overhead v1 incurs. * Every EXECUTE on a V2 connection carries a tablet_version_block computed from the cached version -- or a random byte on cold start (a token-aware request with no cached version yet), or 0 for a non-token-aware request, which the server never version-checks. * The routing key and its ring token are resolved once per request, in _create_response_future, and handed to both consumers that need them while sending: the tablet_version_block here and shard selection in HostConnection. This keeps cluster-dependent state off the statement, which a caller may share between concurrent requests. The cached tablet is likewise looked up once -- the cache is mutable, so a second lookup could disagree with the first. * On the response, the routing payload is parsed according to what the serving connection negotiated; the v2 tuple additionally carries the tablet_version, which is stored back on the tablet. The tablet is cached under the effective keyspace -- the statement's, else the session's -- so a prepared statement executed in a session keyspace lands under the same key the send path looks it up by. * HostConnection.tablets_routing_v1 becomes supports_tablet_routing: shard selection is identical under both versions, since the request goes to this host either way and the pool picks the shard this host owns for the tablet. Refs: SCYLLADB-288 Refs: SCYLLADB-291
Cover the end-to-end behaviour against a live ScyllaDB started with the `strongly-consistent-tables` experimental feature: v2 negotiation, payload-driven cache population, and the tablet_version_block matching rules (no payload on a matching block, exactly one matching value per index, and v2 taking precedence over v1 on a wrong-shard request). The last of those needs a connection that negotiated both extensions, which the driver never does on its own, so the test patches ProtocolFeatures.add_startup_options. The patch delegates to the real implementation and only adds v1 on top. Enumerating the options itself would silently stop requesting any extension added later while ProtocolFeatures still reported it as negotiated -- that is parsed from SUPPORTED, not from what STARTUP asked for -- and an extension that changes the frame layout, such as SCYLLA_USE_METADATA_ID, would then desynchronize every request on the connection.
Extend the "Tablet Awareness" section of the Scylla-specific guide to cover the V2 protocol extension: the per-connection negotiation and the tablet_version_block byte that lets the server skip re-sending routing information the driver already has.
Add KeyspaceMetadata._consistency_mode, derived from the per-keyspace `consistency` option in system_schema.scylla_keyspaces. It is a _ConsistencyMode enum -- EVENTUAL, LOCAL or GLOBAL -- so the mode the server reported is kept verbatim instead of being flattened into a boolean at parse time. The lookup is cached per schema refresh and degrades to EVENTUAL on non-Scylla clusters, on a control connection that did not negotiate TABLETS_ROUTING_V2, and on Scylla versions that lack the table or column. Scylla only implements `global` so far, so a keyspace's tablets have a Raft leader exactly when its mode is GLOBAL; `local` is reserved for a mode that does not exist yet and behaves like `eventual` everywhere. Callers that care compare against _ConsistencyMode.GLOBAL directly, so implementing `local` later only widens those comparisons and leaves the parser and the metadata untouched. A change of mode also invalidates the keyspace's cached tablets, the same way a replication-strategy change does: a tablet cached while the keyspace was eventually consistent carries no leader ordering and must not survive into a strongly-consistent keyspace, where it would be misread as a leader hint. Both names are underscore-prefixed to keep them private: they are not yet stable and we do not want to commit to a public API for them.
For a strongly-consistent tablet the TABLETS_ROUTING_V2 server orders the replica set with the Raft leader first (replicas[0]) and keeps it fresh via the tablet_version already tracked in the previous commits. TokenAwarePolicy uses this to send reads and writes for such tables straight to the leader, saving the extra coordinator->leader hop. The leader is yielded first only when the keyspace's consistency mode is GLOBAL -- the only mode Scylla implements, and so the only one whose tablets have a leader -- and when the tablet carries a tablet_version: eventually-consistent tablet tables are assigned a tablet_version too, and a versionless (v1-sourced or stale) tablet must not be mistaken for a leader hint. Requests at consistency level ONE or LOCAL_ONE are left alone. Any single replica satisfies them, so preferring the leader would only concentrate load on it without buying any consistency. The level is read from the statement, so a request that inherits it from an execution profile looks unset here and is routed to the leader anyway; that costs a little leader contention and nothing in correctness, and is tracked separately in scylladb#953. The hint stays bounded by the wrapped policy -- the leader is front-run only if the child policy would consider it at all (never a host it reports as IGNORED, nor a cross-datacenter leader under a DCAwareRoundRobinPolicy with no remote hosts); otherwise the usual token-aware (optionally shuffled) ordering applies and the server forwards to the leader as it would without v2. Refs: SCYLLADB-288 Fixes: SCYLLADB-291
Extend the TABLETS_ROUTING_V2 integration suite with a strongly-consistent (consistency='global', Raft-backed) keyspace and cover, against a live ScyllaDB: that the driver reads each keyspace's _consistency_mode from system_schema.scylla_keyspaces (statically, and as keyspaces are created and dropped), and that TokenAwarePolicy sends a leader-requiring request for such a table to the Raft leader (replicas[0]).
Extend the Scylla-specific guide's TABLETS_ROUTING_V2 section, which the previous docs commit introduced for tablet-version tracking, to cover leader-aware routing: strongly-consistent (Raft-backed) tablet tables have a leader that the driver targets directly to save the coordinator->leader hop, the behaviour is best-effort and bounded by the load-balancing policy, and eventually-consistent tables keep their usual token-aware ordering. Include a table of how ScyllaDB serves each operation on such a table, so the routing distinction has a visible reason: ONE and LOCAL_ONE reads are non-linearizable and take no Raft read barrier, so they keep normal token-aware ordering, while QUORUM and LOCAL_QUORUM reads and writes go through the leader and are routed to it.
|
v11:
Btw. I realised some tests might not be passing with partially implemented tablets routing v2, but it turns out that's not the case, so unless I missed something, we should be good. I didn't touch the load balancing code as the thread is not resolved yet. I hope that's the last thing to change, though. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (4)
cassandra/policies.py:548
- When a statement leaves
consistency_levelunset, this treatsNoneas leader-requiring. However,Session._create_response_futureresolves that same request to the selected execution profile’s consistency level (cluster.py:3017-3032), which is commonlyLOCAL_ONE. Those reads will therefore be pinned to the leader despite the documented ONE/LOCAL_ONE carve-out, creating the hotspot this branch is intended to avoid. The effective consistency level needs to be passed into query-plan construction rather than inferred from the raw statement.
effective_cl = query.consistency_level
prefer_leader = effective_cl not in (ConsistencyLevel.ONE, ConsistencyLevel.LOCAL_ONE)
tests/integration/standard/test_tablets_routing_v2.py:245
Nonedoes not omit the byte:ExecuteMessage.send_bodycoalesces it to0x00whenever V2 is negotiated. This docstring currently claims a wire behavior the helper cannot produce.
Send an EXECUTE directly on a specific shard connection with a chosen
tablet_version_block (or None to omit the byte entirely, i.e. behave like
the pre-V2 protocol), and return the decoded response message.
cassandra/policies.py:542
- Correct the typo in “consistency.”
# TODO: Figure out how to obtain the actual effective consisteny
cassandra/metadata.py:834
- Remove the duplicated word.
The consistency mode of the keyspace, derived from the the ``consistency``
column ScyllaDB stores in ``system_schema.scylla_keyspaces``.
In dfccfff, we introduced tablet awareness.
Thanks to it, the driver was finally able to learn about the replicas
of a given tablet and successfully route requests to the right nodes
and shards. Unfortunately, the feature isn't sufficient to handle
strongly consistent tables.
Just like an eventually consistent tablet, a strongly consistent one
has a set of replicas that own it. The difference is that the latter
also has a distinguished replica called the leader that coordinates all
of the writes and (almost) all reads to the tablet. That creates a need
for the driver to route its requests to the leader instead of an
arbitrary replica. However, the current tablet awareness doesn't allow
the driver to learn about the identity of the leader.
To solve that problem, we devise a new protocol extension that will
effectively function as a new version of tablet awareness. It affects
both eventually and strongly consistent tablets. This PR implements it
in the driver.
We introduce the notion of a tablet version -- a 64-bit hash that
corresponds to the replicas of a tablet and (in the case of strongly
consistent tablets) its leader.
With every EXECUTE request, the driver provides information about the
tablet version it knows. If the server detects a mismatch, the response
to the request will contain full routing information: the boundary
tokens of the tablet, the list of its replicas, and the tablet version.
The driver then updates its cache and routes subsequent requests to the
right replicas.
Tests have been provided to verify that the implementation is correct.
Fixes: SCYLLADB-291
Refs: SCYLLADB-288
Pre-review checklist
./docs/source/.Fixes:annotations to PR description.