DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension

[nikagra]

Summary

Fixes: https://scylladb.atlassian.net/browse/DRIVER-153

Implements the SCYLLA_USE_METADATA_ID Scylla CQL protocol extension (DRIVER-153), which backports the prepared-statement metadata-ID mechanism from CQL v5 to earlier protocol versions.

When the extension is negotiated:

• The server includes a result metadata hash in the PREPARE response
• The driver sends that hash back with every EXECUTE request, allowing the server to skip sending full result metadata on every response (skip_meta=True)
• If the result schema has changed, the server sets the METADATA_CHANGED flag and includes the new metadata ID + new column metadata in the response — the driver picks this up and updates its cached metadata automatically

Rebased on #934 (merged groundwork that threads each connection's negotiated ProtocolFeatures into message serialization) and reworked to resolve the review discussion below.

Design change from the previous version of this PR

Previously, ExecuteMessage carried a use_metadata_id flag and ResponseFuture._query() mutated skip_meta/result_metadata_id on the shared message after borrowing a connection. Per @Lorak-mmk's review (#770 (comment) and the surrounding thread), that design is gone:

• ExecuteMessage is immutable once constructed — skip_meta and result_metadata_id are set from the prepared statement's cached metadata at construction time (in _create_response_future), same as before this extension existed.
• send_body decides what actually reaches the wire from (protocol_version, protocol_features) — the connection's negotiated features, supplied by protocol: pass negotiated ProtocolFeatures to message serialization #934's plumbing. The metadata-id field is written whenever the serving connection speaks CQL v5+ or negotiated the extension — always, not conditionally — with an empty b'' sentinel when the statement has no id yet (mixed cluster / rolling upgrade). _SKIP_METADATA_FLAG is only set when that same condition holds, so a statement executed against a connection without the extension never asks the server to skip metadata it has no way to recover.

This fixes two problems the old design had:

• Speculative-execution race: the old _query() mutated the one shared ExecuteMessage per connection borrowed; with speculative retries or a mixed-feature cluster, two connections could interleave mutation and encoding of the same message. Immutability removes the race entirely.
• Control-connection fallback gap (@dkropachev, DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment)): _query_control_connection never went through the mutation block and would omit the id field on an extension-enabled control connection. Every send path now serializes correctly with zero extra code, because the decision is made in send_body from the connection actually being used.

Other review threads resolved

• Torn metadata/id pair (@Lorak-mmk DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment), @dkropachev DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment)): PreparedStatement now stores (result_metadata, result_metadata_id) as one tuple replaced in a single attribute assignment (update_result_metadata()), with result_metadata/result_metadata_id as compatibility properties over it. A reader can no longer observe the metadata of one schema version paired with the id of another — the previous per-field write ordering (and its GIL-dependent correctness comment) is gone. The compatibility setters are documented as non-atomic relative to each other, pointing callers at update_result_metadata().
• Behavioral fix beyond what was asked: when the server sends a new metadata id without column metadata (_set_result), the old code adopted the id anyway. That manufactures the exact unrecoverable state the pair-atomicity fix is meant to prevent: the server would then match the id and stop sending metadata, while the driver decodes against stale cached columns forever. Now that response is logged as a warning and nothing is cached — the next EXECUTE resends the old id, the server detects the mismatch, and the driver recovers with full metadata.
• Rolling-upgrade docs (@Lorak-mmk DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment)): reworded to describe the actual self-healing behavior — a statement prepared before the extension was negotiated sends the empty sentinel on its first execute over an extension-enabled connection and acquires an id from the resulting METADATA_CHANGED response; no client restart needed. Also reworded the section's opening paragraph so skip_meta reads as conditional on the extension, not pre-existing default behavior.
• Wrong spec link (@Lorak-mmk DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment)): replaced the opensource-docs CQL-language-extensions link with the scylladb repo's docs/dev/protocol-extensions.md, which actually documents this extension.
• bool(result_metadata) question (@Lorak-mmk DRIVER-153: negotiate and implement SCYLLA_USE_METADATA_ID extension #770 (comment)): result_metadata is None for statements with NO_METADATA in their PREPARE response (LWT/conditional statements) and [] for statements returning zero columns (plain INSERT/UPDATE/DELETE). Both are falsy, and both correctly keep skip_meta off — there's nothing to decode against in either case. Documented in a code comment at the construction site.

Changes

cassandra/protocol_features.py

• Add USE_METADATA_ID = "SCYLLA_USE_METADATA_ID" constant and use_metadata_id field to ProtocolFeatures
• Parse the extension from the SUPPORTED frame; include it in STARTUP when present

cassandra/protocol.py

• Bug fix: _SKIP_METADATA_FLAG is now actually written to the wire — it was stored on _QueryMessage but never sent (effectively dead code upstream)
• recv_results_prepared: read result_metadata_id for the Scylla extension (pre-v5) in addition to standard CQL v5+
• ExecuteMessage.send_body: decides both the metadata-id field and the skip-metadata flag from (protocol_version, protocol_features) at serialization time; writes the b'' sentinel when the statement has no id

cassandra/query.py

• PreparedStatement.result_metadata/result_metadata_id become properties over a single (result_metadata, result_metadata_id) tuple; add update_result_metadata() for atomic pair replacement

cassandra/cluster.py

• _create_response_future: snapshot the metadata pair once, construct ExecuteMessage immutably with skip_meta/result_metadata_id set from that snapshot
• _set_result: on METADATA_CHANGED, replace the cached pair atomically; ignore (with a warning) a response that carries a new id without column metadata
• _execute_after_prepare: same atomic update on reprepare, keeping the previous id when the response carries none

docs/scylla-specific.rst

• Document the extension and its behaviour; corrected rolling-upgrade description and spec reference link

CHANGELOG.rst

• Added a Features entry for the extension

Live-server verification of the b'' sentinel

The one open question from review was whether Scylla actually treats the empty b'' metadata-id sentinel — sent by a statement that was prepared before the extension was negotiated, e.g. mid rolling-upgrade — as a mismatch, versus rejecting it as a malformed frame. Resolved two ways:

Cross-driver precedent. The identical scenario is already covered by a merged, live-server-tested case in the Java driver: scylladb/java-driver#758 adds PreparedStatementIT.should_handle_empty_metadata_id_when_executing_statement_when_supported, which nulls a prepared statement's cached id, executes, and asserts the id comes back non-null against a real Scylla node. The gocql implementation (scylladb/gocql#590) independently reaches the identical wire convention — its Test_framer_writeExecuteFrame unit test asserts a nil id serializes to a zero-length short-bytes field, the same convention as this PR's b''. All three drivers trace back to the same server-side fix (scylladb/scylladb#23292).

Live run against this driver. Added tests/integration/standard/test_scylla_metadata_id.py (3 tests) and ran it against a real Scylla node via CCM:

source ~/Envs/scylla/bin/activate   # venv with ccm + a Scylla build already cached locally, no download needed
cd python-driver
SCYLLA_VERSION=release:2026.1.9 PROTOCOL_VERSION=4 pytest tests/integration/standard/test_scylla_metadata_id.py -v

tests/integration/standard/test_scylla_metadata_id.py::ScyllaMetadataIdTests::test_empty_sentinel_id_triggers_metadata_changed PASSED [ 33%]
tests/integration/standard/test_scylla_metadata_id.py::ScyllaMetadataIdTests::test_extension_is_negotiated PASSED [ 66%]
tests/integration/standard/test_scylla_metadata_id.py::ScyllaMetadataIdTests::test_metadata_changed_recovers_after_schema_change PASSED [100%]
======================== 3 passed, 3 warnings in 25.09s ========================

test_empty_sentinel_id_triggers_metadata_changed is the direct check: prepare a statement, force its result_metadata_id back to None via update_result_metadata() (simulating a statement prepared before the extension was known), execute it, and confirm both that no error is raised and that a fresh result_metadata_id comes back afterward — proving Scylla treats the sentinel as METADATA_CHANGED, not a protocol error.

Test plan

• Unit tests across test_protocol_features.py, test_protocol.py, test_query.py, test_response_future.py covering: feature negotiation, STARTUP options, wire-format assertions for the metadata-id field and skip-metadata flag (present/suppressed, sentinel for None, v4/v5), PREPARE response decoding with/without the extension, atomic metadata-pair replacement (constructor, update_result_metadata, compatibility setters), _create_response_future construction gating (id present/absent, LWT/NO_METADATA, zero-column), a regression proving _query sends the message unmutated (the old speculative-execution race), _set_result METADATA_CHANGED and the anomalous-response warning path (nothing cached), and _execute_after_prepare reprepare handling
• Full unit test suite passes (693 passed, 111 skipped)
• Cython build (build_ext --inplace) compiles the reworked signatures; tests pass against the compiled modules
• Integration tests against a Scylla node with the extension: verified live via CCM (Scylla 2026.1.9) that schema changes after PREPARE are detected and metadata is updated without re-preparation, and that the empty-id sentinel is accepted as a mismatch, not rejected — see "Live-server verification" above

Pre-review checklist

• I have split my patch into logically separate commits.
• All commit messages clearly explain what they change and why.
• I added relevant tests for new features and bug fixes.
• All commits compile, pass static checks and pass test.
• PR description sums up the changes and reasons why they should be introduced.
• I have provided docstrings for the public items that I want to introduce.
• I have adjusted the documentation in ./docs/.
• I added appropriate Fixes: annotations to PR description.

[Copilot]

Pull request overview

Implements negotiation and support for Scylla’s SCYLLA_USE_METADATA_ID protocol extension to enable metadata-id based skip_meta behavior (backporting CQL v5 prepared-statement metadata-id semantics to earlier protocol versions).

Changes:

• Adds SCYLLA_USE_METADATA_ID parsing from SUPPORTED and includes it in STARTUP when negotiated.
• Extends protocol encode/decode to read/write result_metadata_id for PREPARE/EXECUTE on pre-v5 when the extension is used, and fixes on-wire encoding of _SKIP_METADATA_FLAG.
• Updates execution/result handling to conditionally use skip_meta and to refresh cached prepared metadata when the server reports metadata changes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
cassandra/protocol_features.py	Adds the SCYLLA_USE_METADATA_ID feature flag and includes it in negotiated STARTUP options.
cassandra/protocol.py	Writes _SKIP_METADATA_FLAG in query params; adds pre-v5 extension handling for result_metadata_id in PREPARE/EXECUTE.
cassandra/cluster.py	Adjusts when skip_meta is enabled and updates cached prepared metadata/id on METADATA_CHANGED responses.
tests/unit/test_protocol_features.py	Adds unit tests for feature parsing and STARTUP option inclusion.
tests/unit/test_protocol.py	Adds unit tests for skip-meta flag encoding and metadata-id handling in pre-v5 PREPARE/EXECUTE paths.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mykaul]

I'm not sure where, but we should document this - with reference mainly to the scylladb docs about this feature.

[nikagra]

@mykaul Documentation I'm aware of is MetadataId extension in CQLv4 Requirement Document

[dkropachev]

One blocking correctness issue below: skip_meta is being enabled for prepared statements that can still have empty/absent cached result metadata.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds SCYLLA_USE_METADATA_ID negotiation, protocol metadata-id serialization and parsing, atomic prepared-statement metadata caching, and cache refresh handling for changed schemas. Execute messages conditionally send skip_meta and result_metadata_id; responses update cached metadata when column metadata is returned. Documentation and unit tests cover protocol versions, feature negotiation, reprepare behavior, cache consistency, and warning paths.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant ResponseFuture
  participant Server
  participant PreparedStatement
  Client->>ResponseFuture: execute prepared statement
  ResponseFuture->>PreparedStatement: snapshot metadata and id
  ResponseFuture->>Server: send ExecuteMessage
  Server-->>ResponseFuture: return result metadata id and optional columns
  ResponseFuture->>PreparedStatement: update cached metadata pair
  ResponseFuture-->>Client: return rows

Loading

Possibly related PRs

• scylladb/python-driver#913: Changes protocol message encoding based on negotiated protocol features in cassandra/protocol.py.

Suggested labels: area/Driver_-_python-driver

Suggested reviewers: dkropachev, sylwiaszunejko

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 59.26% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly names the main change: negotiating and implementing the SCYLLA_USE_METADATA_ID extension.
Description check	✅ Passed	The description matches the template well and includes the required checklist items, summary, Fixes annotation, tests, and docs updates.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 0

[nikagra]

🤖: All review issues (1–9) have been addressed and the branch has been squashed to the required 2-commit shape. Requesting re-review from @Lorak-mmk and @dkropachev.

What changed since last review:

Production commit (8accdb5a) — no functional changes, two cleanups squashed in:

• Issue 8 (docs scope claim): corrected docs/scylla-specific.rst to say the extension applies to EXECUTE requests, not PREPARE
• Issue 9 (GIL comment): added a note in _set_result explaining why the metadata update write ordering is safe under the GIL

Test commit (bfc97602) — all 6 fix/addition commits squashed in, commit message fully replaced to enumerate every test:

• Issue 1: test_query_no_skip_meta_without_extension fixture corrected (result_metadata=[] was falsy, defeating the assertion)
• Issue 2: test_execute_after_prepare_updates_result_metadata_id and test_execute_after_prepare_no_metadata_id_in_response added to cover the _execute_after_prepare reprepare path
• Issue 3: test_recv_results_metadata_no_metadata_flag_skips_metadata_id tightened — now asserts not hasattr(result, 'result_metadata_id') rather than is None, and checks column_metadata not result_metadata
• Issue 4: test_recv_results_prepared_v5_reads_metadata_id added — covers the v5 native uses_prepared_metadata() decode path
• Issue 5: test_execute_message_v5_skip_meta_sets_flag added — confirms _SKIP_METADATA_FLAG is correctly written into the 4-byte v5 flags word (this flag was dead code in upstream before this PR)
• Issue 6: test_repeat_orig_query_after_succesful_reprepare fixed — result_metadata_id value changed from str to bytes; assertion that value is stored on prepared_statement added
• Issue 7: test_set_result_warns_when_metadata_id_but_column_metadata_is_none added — covers the column_metadata=None (absent) variant of the METADATA_CHANGED warning path

CI on the 8-commit pre-squash branch: 18/19 passed; test libev (3.14t) was cancelled after a 6-hour runner stall (infrastructure timeout — test asyncio (3.14t) on the same commit passed cleanly).

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

[nikagra]

🤖: Both of @dkropachev's latest comments are addressed, and the branch keeps the 2-commit shape: 8f3c3f50 (impl) + 8bc97cf8 (tests).

Review comments

• docs (or speak CQL v5) claim — removed; added a bullet stating the native-v5 behaviour explicitly so the claim can't creep back.
• setUpClass resource leak — the probe is now wrapped so an unsupported build or a probe failure tears down the cluster/keyspace before raising, since unittest skips tearDownClass when setUpClass raises.

Four fixes from a self-review pass, folded into the same two commits

1. update_result_metadata() now re-arms _warned_missing_column_metadata, so recovery via the reprepare path re-arms it too — previously only the _set_result METADATA_CHANGED branch did, contradicting the flag's own comment. Duplicate reset in cluster.py removed.
2. The ROWS-response cache update is now gated on prepared_statement.result_metadata is not None, so a statement prepared with NO_METADATA (conditional/LWT, PYTHON-847) can never acquire cached metadata and become skip_meta-eligible. Its result shape varies per response (applied vs not-applied), so caching one shape could have led to decoding the other against the wrong columns. This was previously unreachable only if the server sets METADATA_CHANGED solely in reply to a skip request — now it is enforced driver-side instead of assumed. New unit test covers it.
3. test_empty_sentinel_id_triggers_metadata_changed now spies on ResponseFuture._reprepare and asserts zero calls, so it provably exercises the METADATA_CHANGED-on-ROWS path rather than the UNPREPARED/reprepare fallback.
4. The feature probe no longer borrows a connection (borrow_connection() pops a stream id that only process_msg gives back, so it leaked one per call); it reads features off the pool's existing connection.

Testing: unit suite 696 passed / 111 skipped. Both new production changes were verified non-vacuous by reverting each in turn and confirming the corresponding test fails. Not yet re-run against a live cluster — test_scylla_metadata_id.py and test_prepared_statements.py -k conditional are the two worth watching in CI, the latter confirming fix 2 left LWT behaviour untouched.

[nikagra]

@dawmd Agreed on not shipping two mechanisms for the same problem — and I think that is now settled in a way that works for both of us: #934 ("protocol: pass negotiated ProtocolFeatures to message serialization") merged on 2026-07-16 and is exactly the shared plumbing.

What it gives us:

• _ProtocolHandler.encode_message and decode_message both receive the ProtocolFeatures negotiated on the connection carrying the message (encode_message as a required protocol_features parameter), and forward it to send_body.
• Messages therefore stay connection-independent — they carry request data only — and send_body decides the wire format from (protocol_version, protocol_features). A field belonging to a negotiated extension is emitted on exactly the connections that negotiated it.
• That is the shape Lorak-mmk suggested in https://github.com/scylladb/python-driver/pull/770/changes#r3442046951, and this PR is rebased on it: result_metadata_id and the skip-metadata flag are now decided inside ExecuteMessage.send_body from the serving connection's features, with no per-request mutation of the message.

On the public API concern: the encode_message signature extension was taken once, in #934, rather than separately in each feature PR. It is documented as a contract change in docs/api/cassandra/protocol.rst (overrides must keep the protocol_features parameter name) with a CHANGELOG entry, so custom _ProtocolHandler subclasses have one documented break instead of two.

So tablet_version_block in #913 should be able to ride the same path — encode it in send_body off protocol_features, no extra parameter needed. If anything in #913 still needs something #934 does not cover, happy to go through it on Slack and fold the difference in here.

[nikagra]

@mykaul Following up on this — the feature is now documented in-tree, with the ScyllaDB reference you asked for.

docs/scylla-specific.rst has a "Prepared Statement Metadata Caching (SCYLLA_USE_METADATA_ID)" section covering:

• what the extension does and why change detection is needed (stale cached metadata after a schema change),
• that it is negotiated automatically when the node supports it, and the exact conditions under which the driver asks the server to omit result metadata from EXECUTE responses,
• how a METADATA_CHANGED response refreshes both the cached column metadata and the hash, with no application change,
• what happens to statements prepared before the extension was negotiated (rolling upgrade): they acquire a hash automatically on their first execute, no re-prepare or restart,
• which statements the optimization applies to in practice (anything whose PREPARE returns result columns, i.e. SELECTs).

It closes with the link to the ScyllaDB protocol extensions doc: https://github.com/scylladb/scylladb/blob/master/docs/dev/protocol-extensions.md — plus the internal requirement document I linked above for the design rationale.

[dkropachev]

@Lorak-mmk , can you please check it, or just tell us it is good to go