Repository navigation
SLO: transactional topic<->table workloads (sync/async), exactly-once under TLI #854
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # Topic-tx SLO scenario — transactional topic ↔ table pipeline | ||
|
|
||
| What the `sync-topic-tx` / `async-topic-tx` workloads do: an **exactly-once** | ||
| processing pipeline where both ends run inside a YDB transaction. Producers write | ||
| to the topic with a `tx_writer` (a transactional topic write); consumers read from | ||
| the topic with `receive_batch_with_tx` and, in the *same* transaction, UPSERT the | ||
| message into a sink table. The commit atomically advances the topic offset and | ||
| persists the sink rows — so every produced message lands in the sink table exactly | ||
| once, even while a chaos monkey keeps killing YDB nodes. | ||
|
|
||
| ## Topology | ||
|
|
||
| ``` | ||
| producer tx (transactional topic write) per-ref topic: /Root/testdb/slo_topic_tx_<ref> | ||
| │ tx_writer(tx).write(msg); commit ┌──────────────────────────────────────────────┐ | ||
| w0 ──┼──produce p0──▶ partition 0 ────────────────▶│ p0: m1 m2 m3 … (server-ordered per producer) │ | ||
| w1 ──┴──produce p1──▶ partition 1 ────────────────▶│ p1: m1 m2 m3 … │ | ||
| └───────────────────────┬──────────────────────┘ | ||
| │ one consumer group | ||
| ┌──────────────────────────────────────────▼──────────────────────┐ | ||
| │ consumer tx: receive_batch_with_tx(tx) → decode → validate │ | ||
| │ → UPSERT sink(writer_id, seqno); commit │ | ||
| └───────────────────────────────────────────┬───────────────────────┘ | ||
| ▲ ▼ sink(writer_id, seqno) — exactly once | ||
| chaos-monkey: kills a YDB node every ~20-60s | ||
| ``` | ||
|
|
||
| Each message is self-describing: `writer_id : seqno : write_ts_ns : <padding>` | ||
| (same codec as the plain topic scenario, `jobs/topic_payload.py`). | ||
|
|
||
| ## Flow (happy path + chaos) | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| autonumber | ||
| participant W as Producer tx (tx_writer) | ||
| participant S as YDB topic · partition i | ||
| participant R as Consumer tx (receive_batch_with_tx) | ||
| participant T as sink table | ||
| participant M as Metrics (OTLP) | ||
|
|
||
| Note over W: retry_tx: tx_writer.write(msgs), commit | ||
| loop until --time | ||
| W->>S: tx_writer.write(payload) (buffered, flushed on commit) | ||
| alt commit ok | ||
| S-->>W: topic write committed | ||
| Note right of W: seqno advances only on commit (no gap) | ||
| else Unavailable (chaos) | ||
| Note right of W: retry_tx re-runs callee; nothing persisted (no gap) | ||
| end | ||
| end | ||
|
|
||
| loop drain as fast as messages arrive | ||
| S-->>R: receive_batch_with_tx(tx) | ||
| R->>T: UPSERT sink(writer_id, seqno) — keyed, idempotent | ||
| alt commit ok | ||
| R->>M: for each msg: delivered++, e2e = now - write_ts, validate seqno | ||
| Note right of R: offset advance + sink rows committed atomically | ||
| else Unavailable (chaos) | ||
| R->>M: retry_tx re-reads from last committed offset | ||
| Note right of R: rolled-back tx advanced no offset ⇒ re-processed, upsert idempotent | ||
| end | ||
| end | ||
| ``` | ||
|
|
||
| ## Invariants we assert | ||
|
|
||
| | Signal | Meaning | Expectation | | ||
| |--------|---------|-------------| | ||
| | `topic_lost_errors` | forward gap in a producer's seqno (real loss) | **0** — hard-fails the run, under chaos | | ||
| | exactly-once into sink | sink keyed by `(writer_id, seqno)` — idempotent UPSERT | re-processing after a rollback creates no duplicate row | | ||
| | `topic_e2e_latency_p50/p99` | write-commit → read-commit latency | reported | | ||
| | `topic_duplicates` | redelivery after a rolled-back tx | informational (at-least-once at the app level) | | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Severity: Minor This invariant row documents Suggested fix: reword the row to describe the producer re-commit duplicate case, or note that consumer-side rollback redelivery is invisible to |
||
| | `read/write_availability` | liveness under chaos | ~100% | | ||
|
|
||
| ## Why it stays correct under chaos | ||
|
|
||
| - **The topic read and the table write share one transaction.** A consumer commit | ||
| advances the topic offset *and* writes the sink row atomically; an abort (chaos) | ||
| rolls back both. There is no window where one happened without the other. | ||
| - **seqno advances only after a successful commit** — an aborted/retried producer | ||
| tx leaves no gap. Validation reads the seqno embedded in the payload, so a | ||
| retried commit is at worst an app-level duplicate, never a false loss. | ||
| - **sink is keyed by `(writer_id, seqno)`** — a consumer tx that rolls back | ||
| (chaos) re-reads the same messages from the last committed offset and | ||
| re-UPSERTs the same keys: idempotent, so the table stays exactly-once. | ||
| - **validation runs only after the commit returns** — only durable offsets/rows | ||
| feed the per-producer next-expected-seqno check (shared across readers, since a | ||
| partition can move between them on rebalance). | ||
| - **per-ref topic + per-ref sink table** — the `current` and `baseline` containers | ||
| run on the same cluster in parallel; ref-scoping keeps their validation from | ||
| mixing. | ||
|
|
||
| ## Not covered here | ||
|
|
||
| - Reading the sink table back for an independent full-table exactly-once audit | ||
| (validation is the in-memory per-producer next-expected-seqno check, fed only by | ||
| committed batches; the atomic offset+row commit makes that a faithful proxy). | ||
| - Snapshot/read-only tx modes — the pipeline runs serializable read-write. | ||
| - Deliberate non-retriable failures (SchemeError/BadRequest) — those are code/config | ||
| bugs, out of scope for a liveness SLO. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| import asyncio | ||
| import logging | ||
| import time | ||
|
|
||
| from core.metrics import OP_TYPE_READ, OP_TYPE_WRITE, REF | ||
|
|
||
| import ydb | ||
| import ydb.aio | ||
|
|
||
| from .async_topic_jobs import AsyncTopicJobManager | ||
| from .topic_payload import decode_payload, encode_payload | ||
| from .topic_tx_jobs import SINK_UPSERT_TEMPLATE | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| class AsyncTopicTxJobManager(AsyncTopicJobManager): | ||
| """Async mirror of :class:`TopicTxJobManager` (transactional topic <-> table | ||
| pipeline). See that class for the scenario description.""" | ||
|
|
||
| def __init__(self, driver, args, metrics, sink_table): | ||
| super().__init__(driver, args, metrics) | ||
| self.sink_table = sink_table | ||
| self.messages_per_tx = max(1, int(getattr(self.args, "messages_per_tx", 10))) | ||
| self._sink_upsert_query = SINK_UPSERT_TEMPLATE.format(sink_table) | ||
| self.retry_settings = ydb.RetrySettings(idempotent=True) | ||
|
|
||
| # --- shared query helper -------------------------------------------------- | ||
|
|
||
| async def _upsert_sink(self, tx, decoded): | ||
| writer_id, seqno, write_ts_ns = decoded | ||
| async with await tx.execute( | ||
| self._sink_upsert_query, | ||
| { | ||
| "$writer_id": (writer_id, ydb.PrimitiveType.Uint64), | ||
| "$seqno": (seqno, ydb.PrimitiveType.Uint64), | ||
| "$write_ts_ns": (write_ts_ns, ydb.PrimitiveType.Uint64), | ||
| }, | ||
| commit_tx=False, | ||
| ) as result_sets: | ||
| async for _ in result_sets: | ||
| pass | ||
|
|
||
| # --- producer (transactional topic write) -------------------------------- | ||
|
|
||
| async def _run_topic_writes(self, writer_id, limiter): | ||
| start_time = time.time() | ||
| partition_id = writer_id % self.partitions | ||
| producer_id = f"{REF}-p{writer_id}" | ||
| logger.info("Start async topic tx writer %s (partition %s, producer %s)", writer_id, partition_id, producer_id) | ||
|
|
||
| # `seqno` lives across recreations and advances only after a successful | ||
| # commit, so the sequence the reader sees is gap-free. | ||
| seqno = 1 | ||
| while time.time() - start_time < self.args.time: | ||
| try: | ||
| async with ydb.aio.QuerySessionPool(self.aio_driver) as pool: | ||
| while time.time() - start_time < self.args.time: | ||
| base_seqno = seqno | ||
| payloads = [] | ||
| for k in range(self.messages_per_tx): | ||
| if time.time() - start_time >= self.args.time: | ||
| break | ||
| async with limiter: | ||
| payloads.append( | ||
| encode_payload( | ||
| writer_id, base_seqno + k, time.monotonic_ns(), self.args.message_size | ||
| ) | ||
| ) | ||
| if not payloads: | ||
| break | ||
|
|
||
| async def callee(tx): | ||
| tx_writer = self.aio_driver.topic_client.tx_writer( | ||
| tx, | ||
| self.args.path, | ||
| producer_id=producer_id, | ||
| partition_id=partition_id, | ||
| codec=ydb.TopicCodec.RAW, | ||
| ) | ||
| for payload in payloads: | ||
| await tx_writer.write(ydb.TopicWriterMessage(data=payload)) | ||
|
|
||
| ts = self.metrics.start((OP_TYPE_WRITE,)) | ||
| try: | ||
| await pool.retry_tx_async(callee, retry_settings=self.retry_settings) | ||
| self.metrics.stop((OP_TYPE_WRITE,), ts) | ||
| seqno = base_seqno + len(payloads) # advance only on commit | ||
| except Exception as e: | ||
| self.metrics.stop((OP_TYPE_WRITE,), ts, error=e) | ||
| logger.error("Tx write error (recreating writer): %s", e) | ||
| break | ||
| except Exception as e: | ||
| logger.error("Topic tx writer %s recreate: %s", writer_id, e) | ||
| await asyncio.sleep(0.2) | ||
|
|
||
| logger.info("Stop async topic tx writer %s", writer_id) | ||
|
|
||
| # --- consumer (topic -> table) ------------------------------------------- | ||
|
|
||
| async def _run_topic_reads(self, reader_id): | ||
| start_time = time.time() | ||
| read_timeout = self.args.read_timeout / 1000 | ||
| logger.info("Start async topic tx reader %s", reader_id) | ||
|
|
||
| while time.time() - start_time < self.args.time: | ||
| try: | ||
| async with ydb.aio.QuerySessionPool(self.aio_driver) as pool: | ||
| async with self.aio_driver.topic_client.reader(self.args.path, self.args.consumer) as reader: | ||
| while time.time() - start_time < self.args.time: | ||
| committed = [] | ||
|
|
||
| async def callee(tx): | ||
| nonlocal committed | ||
| committed = [] | ||
| batch = await asyncio.wait_for( | ||
| reader.receive_batch_with_tx(tx, max_messages=self.messages_per_tx), read_timeout | ||
| ) | ||
| if batch is None: | ||
| return | ||
| for msg in batch.messages: | ||
| decoded = decode_payload(msg.data) | ||
| if decoded is not None: | ||
| await self._upsert_sink(tx, decoded) | ||
| committed.append(decoded) | ||
|
|
||
| ts = self.metrics.start((OP_TYPE_READ,)) | ||
| try: | ||
| await pool.retry_tx_async(callee, retry_settings=self.retry_settings) | ||
| except asyncio.TimeoutError: | ||
| # No batch within read_timeout: an idle reader or | ||
| # the SDK transparently reconnecting under chaos, | ||
| # not a read failure. Don't count the op (real | ||
| # errors surface as exceptions the tx retries). | ||
| self.metrics.cancel((OP_TYPE_READ,), ts) | ||
| continue | ||
| except Exception as e: | ||
| self.metrics.stop((OP_TYPE_READ,), ts, error=e) | ||
| logger.error("Tx read error (recreating reader): %s", e) | ||
| break | ||
|
|
||
| if not committed: | ||
| self.metrics.cancel((OP_TYPE_READ,), ts) | ||
| continue | ||
|
|
||
| # A batch committed: the offsets/rows are durable, so | ||
| # the seqnos are exactly-once delivered. | ||
| self.metrics.stop((OP_TYPE_READ,), ts) | ||
| for decoded in committed: | ||
| self._record(decoded) | ||
| except Exception as e: | ||
| logger.error("Topic tx reader %s recreate: %s", reader_id, e) | ||
| await asyncio.sleep(0.2) | ||
|
|
||
| logger.info("Stop async topic tx reader %s", reader_id) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Severity: Minor
Confidence: Medium
The table defines
topic_e2e_latency_p50/p99as "write-commit → read-commit latency", but the code captureswrite_ts_nsviatime.monotonic_ns()insideencode_payload, which runs before the producer transaction starts (and before the limiter pacing loop completes for the rest of the batch). If a chaos abort forcesretry_tx_*to re-run the callee, backoff (RetrySettings defaults grow to seconds per attempt over 10 retries) is absorbed into the consumer-side e2e measurement, so the metric drifts from its documented meaning and can inflate the gatedtopic_e2e_latency_p50after aborts.Suggested fix: either reword the doc to "encode → read-commit", or capture the write timestamp on the final attempt (e.g. right before
tx.commit()/ after the first successful write ack) so e2e measures what the doc claims.