Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@
!setup.py
!examples/opentelemetry/requirements.txt
!examples/opentelemetry/otel_example.py
!examples/opentelemetry/load_tank.py
!examples/opentelemetry/load_tank.py
!examples/opentelemetry/verify_metrics.py
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
* Add `ydb.topic.writer.sending.messages`, `ydb.topic.writer.sending.bytes`, `ydb.topic.writer.written.messages`, and `ydb.topic.writer.session.errors` counters, `ydb.topic.writer.buffer.used.bytes`, `ydb.topic.writer.buffer.limit.bytes`, and `ydb.topic.writer.sending.oldest_age` observable gauges, and `ydb.topic.writer.buffer.wait.duration`, `ydb.topic.writer.message.processing.duration`, `ydb.topic.writer.compression.duration`, and `ydb.topic.writer.message.ack.duration` histograms for explicitly named topic writers; metrics-enabled clients now advertise `ydb-sdk-metrics/0.25.0` in `x-ydb-sdk-build-info`
* Add `ydb.topic.reader.received.messages`, reader-level `ydb.topic.reader.received.bytes`, `ydb.topic.reader.delivered.messages`, `ydb.topic.reader.commit.queued`, `ydb.topic.reader.commit.acknowledged`, and `ydb.topic.reader.session.errors` counters, `ydb.topic.reader.compression.duration` and `ydb.topic.reader.message.processing.duration` histograms, plus `ydb.topic.reader.commit_offset.lag.max`, `ydb.topic.reader.credit_balance_bytes`, `ydb.topic.reader.local_buffer.messages`, `ydb.topic.reader.local_buffer.message_age.max`, and `ydb.topic.reader.partition_session.count` observable metrics for explicitly named topic readers; metrics-enabled clients now advertise `ydb-sdk-metrics/0.14.0` in `x-ydb-sdk-build-info`
* Add optional topic reader names for identifying logical readers in server diagnostics and SDK logs
## 3.35.0 ##
* Include streaming queries in scheme directory listings and path descriptions
* Add timeout to the token exchange request so a stalled token endpoint cannot hang credential refresh
Expand Down
285 changes: 273 additions & 12 deletions docs/observability.rst

Large diffs are not rendered by default.

195 changes: 188 additions & 7 deletions docs/opentelemetry.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,14 @@ driver initialization, and retries — into OpenTelemetry spans, and propagates
context to the YDB server through gRPC metadata using the
`W3C Trace Context <https://www.w3.org/TR/trace-context/>`_ standard. Enabling metrics
records client-side OpenTelemetry instruments for the same operations, retries, and
query session pools.
query session pools, as well as receive, delivery, and buffer metrics for explicitly
named topic Readers and Writers.

Like every backend, it has a **zero-cost no-op path when disabled**: operations, query
sessions, and pools share no-op instrumentation without metric allocations, timers,
locks, or state tracking. The SDK does not import ``opentelemetry`` until you call
``enable_tracing()`` or ``enable_metrics()``.
sessions, pools, and topic Readers share no-op instrumentation without metric
allocations, timers, locks, or state tracking; topic Writers use the same no-op path.
The SDK does not import
``opentelemetry`` until you call ``enable_tracing()`` or ``enable_metrics()``.


Installation
Expand Down Expand Up @@ -139,6 +141,183 @@ The full list of instruments and their attributes is catalogued on the
:doc:`observability` page.


Topic Writer Metrics
--------------------

Enable metrics before creating the Writer and pass a stable, non-empty
``writer_name``:

.. code-block:: python

with driver.topic_client.writer(
"/local/orders",
writer_name="orders-writer",
) as writer:
writer.write(["one", "two"])

``ydb.topic.writer.sending.messages`` counts messages accepted from the application
after validation and any buffer wait. It carries ``endpoint``, ``database``, ``topic``,
and ``writer.name``. ``ydb.topic.writer.sending.bytes`` counts their uncompressed body
bytes, excluding metadata and envelope overhead, and carries the same attributes.
``ydb.topic.writer.written.messages`` counts server acknowledgements for written,
already-written/skipped, and transactional messages. It carries the same attributes
without a ``status`` attribute; a transactional acknowledgement does not imply commit.
``ydb.topic.writer.session.errors`` counts session and connection failures once each.
It adds ``retry_decision=retry|stop``, ``status_code``, and ``error.type``; transactional
Writers report ``stop`` because they do not reconnect. Normal shutdown and validation
or encoding failures are not counted. Retries and reconnects do not count a message or
acknowledgement again. ``ydb.topic.writer.buffer.used.bytes`` is an observable gauge
of the current buffer-limiter reservation. It follows the Writer's backpressure
accounting, including metadata and envelope overhead, is released on acknowledgement,
and is not reset by reconnects. ``ydb.topic.writer.buffer.limit.bytes`` reports the
configured ``max_buffer_size_bytes`` from the same limiter. It is absent when that
limit is ``None`` instead of reporting ``0`` for an unbounded Writer. Closing the
Writer removes both gauge series. ``ydb.topic.writer.buffer.wait.duration`` is a
histogram in seconds. It records one monotonic-clock measurement only when a Write
batch that actually waited for byte or message capacity is accepted; immediate
admission and waits ending in timeout, cancellation, or shutdown produce no sample.
Repeated wakeups remain one measurement.
``ydb.topic.writer.message.processing.duration`` records one histogram measurement per
accepted message from buffer admission until transport-ready data is available. It
includes queueing, codec selection, and compression, and uses ``codec`` plus
``outcome=ok|error`` attributes; failures also include ``error.type``. Buffer wait,
stream readiness, retries, and acknowledgement are excluded.
``ydb.topic.writer.compression.duration`` records each non-RAW encoder invocation and
excludes executor queueing. Adaptive codec-selection trials and repeated compression
produce separate measurements with the same ``codec``, ``outcome``, and optional
``error.type`` attributes. Resending already encoded data produces no new measurement.
``ydb.topic.writer.message.ack.duration`` records one monotonic-clock measurement per
server acknowledgement, from the first transport send; retries and reconnects do not
reset the start time, and messages ending without an acknowledgement produce no sample.
``ydb.topic.writer.sending.oldest_age`` is an observable gauge for the oldest accepted,
unfinished message. It includes processing and reconnect waits, reports ``0`` when the
Writer has no unfinished messages, and aggregates identical Writer attributes by
maximum age. If ``writer_name`` is omitted or empty, the Writer works normally but does
not publish these metrics.


.. _opentelemetry-topic-reader-metrics:

Topic Reader Metrics
--------------------

Enable metrics before creating the Reader and pass an explicit, non-empty
``reader_name``. The name identifies one logical Reader and remains the same when its
underlying stream reconnects:

.. code-block:: python

import asyncio
import uuid

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import (
ConsoleMetricExporter,
PeriodicExportingMetricReader,
)

import ydb
from ydb.opentelemetry import enable_metrics

metric_reader = PeriodicExportingMetricReader(
ConsoleMetricExporter(),
export_interval_millis=1000,
)
meter_provider = MeterProvider(metric_readers=[metric_reader])
enable_metrics(meter_provider)


async def read_one_message():
topic_path = f"/local/otel-reader-metrics-example-{uuid.uuid4().hex}"
consumer = "orders-consumer"

async with ydb.aio.Driver(
endpoint="grpc://localhost:2136",
database="/local",
) as driver:
await driver.wait(timeout=5)

await driver.topic_client.create_topic(topic_path, consumers=[consumer])
try:
async with driver.topic_client.writer(
topic_path,
codec=ydb.TopicCodec.GZIP,
writer_name="payments-writer",
) as writer:
await writer.write("OpenTelemetry topic Reader metrics")

async with driver.topic_client.reader(
topic_path,
consumer=consumer,
reader_name="payments-worker",
) as reader:
await asyncio.wait_for(reader.wait_message(), timeout=10)
await asyncio.sleep(2)
message = await asyncio.wait_for(reader.receive_message(), timeout=10)
await reader.commit_with_ack(message)
await asyncio.sleep(2)
finally:
await driver.topic_client.drop_topic(topic_path)


try:
asyncio.run(read_one_message())
finally:
meter_provider.shutdown()

This registers six cumulative counters, two histograms, and five observable instruments:

* ``ydb.topic.reader.received.messages`` counts received messages before decoding. It
includes ``endpoint``, ``database``, ``reader.name``, optional ``consumer``, and
``topic`` attributes.
* ``ydb.topic.reader.received.bytes`` adds ``ReadResponse.bytes_size`` once for every
response received by the logical Reader. It has the same Reader attributes but no
``topic``, because one response may contain batches from several topics.
* ``ydb.topic.reader.delivered.messages`` counts messages returned to the application
by ``receive_message``, ``receive_batch``, or ``receive_batch_with_tx``. It carries
the Reader attributes and ``topic``.
* ``ydb.topic.reader.compression.duration`` records time spent inside the decoder for
each non-RAW message, excluding decoder-executor queueing. It uses seconds and carries
``topic``, ``codec``, ``outcome``, and ``error.type`` on failures.
* ``ydb.topic.reader.message.processing.duration`` records time from receipt until each
non-RAW message's batch is ready for delivery. It includes decoder queueing and uses
the same attributes as the compression histogram.
* ``ydb.topic.reader.commit.queued`` counts messages in commit ranges accepted by the
SDK, using ``range.end - range.start``. It carries the Reader attributes and ``topic``.
* ``ydb.topic.reader.commit.acknowledged`` counts monotonic committed-offset advances
reported by regular acknowledgements. Successful transaction commits count their
submitted ranges; duplicate or stale regular acknowledgements and failed transactions
add nothing. It carries the Reader attributes and ``topic``.
* ``ydb.topic.reader.commit_offset.lag.max`` reports the maximum gap between the highest
requested commit end offset and the acknowledged offset across active partitions of
each ``topic``. It tracks regular stream commits; transactional ranges remain visible
in the commit counters but do not contribute to this gauge. Stale acknowledgements
cannot increase it.
* ``ydb.topic.reader.session.errors`` counts failed sessions or connection attempts.
The ``retry_decision`` attribute is ``retry`` when the Reader will reconnect and
``stop`` when it terminates. ``status_code`` identifies the YDB status or exception
class, while ``error.type`` classifies transport, YDB, and other failures. Normal
Reader shutdown is not counted. This Reader-level metric has no ``topic`` attribute.
* ``ydb.topic.reader.credit_balance_bytes`` reports the protocol byte credit granted to
the server and not yet consumed by read responses. It is ``0`` between stream
sessions, uses unit ``By``, and has the Reader attributes but no ``topic``.
* ``ydb.topic.reader.local_buffer.messages`` reports the current number of messages
retained by the logical Reader, including messages waiting for decoding and decoded
messages waiting for the application. It has the Reader attributes but no ``topic``.
* ``ydb.topic.reader.local_buffer.message_age.max`` reports, in seconds, the age of the
oldest retained message. It uses a monotonic clock, becomes ``0`` when the buffer is
empty, and has the Reader attributes but no ``topic``.
* ``ydb.topic.reader.partition_session.count`` reports active partition sessions per
``topic``. Configured topics report ``0`` before the first session starts. The metric
has the Reader attributes and ``topic``, but deliberately has no ``partition``
attribute.

If ``reader_name`` is omitted or empty, the Reader still works and receives its normal
process-local diagnostic name, but these thirteen metrics are not published. The runnable
``examples/opentelemetry/otel_example.py`` creates a topic, writes one message, reads it
with an explicitly named Reader, and exports its Reader metrics through OTLP.


Instrumented Spans and Attributes
---------------------------------

Expand Down Expand Up @@ -226,9 +405,11 @@ Running the Examples
--------------------

The runnable script is ``examples/opentelemetry/otel_example.py`` (bank table + concurrent
Serializable transactions and ``app_startup`` / ``example_tli`` application spans). **Start
Docker (YDB or the full stack) first**, then install and run on the host — see
``examples/opentelemetry/README.md`` for the full order of commands and environment variables.
Serializable transactions, ``app_startup`` / ``example_tli`` application spans, and an
explicitly named topic Reader that emits receive, delivery, and buffer metrics). **Start Docker
(YDB or the full stack) first**, then install and run on the host — see
``examples/opentelemetry/README.md`` for the full order of commands and environment
variables.

**Full stack in one command** (YDB + OTLP + Tempo + Grafana; the ``otel-example`` service is built from ``examples/opentelemetry/Dockerfile`` and runs the script once):

Expand Down
11 changes: 11 additions & 0 deletions docs/topic.rst
Original file line number Diff line number Diff line change
Expand Up @@ -183,12 +183,15 @@ Writer Parameters
codec=ydb.TopicCodec.GZIP, # Compress messages. Default: RAW (no compression).
auto_seqno=True, # Auto-increment sequence numbers (default: True).
auto_created_at=True, # Auto-set message timestamps (default: True).
writer_name="payments-writer", # Stable name that enables Writer metrics.
)

* ``producer_id`` is used for deduplication: if the same ``(producer_id, seqno)`` pair is
received again, YDB silently skips it.
* When ``auto_seqno=True`` the SDK assigns monotonically increasing sequence numbers.
Set ``auto_seqno=False`` and provide ``seqno`` manually when you need deterministic IDs.
* Pass a stable, non-empty ``writer_name`` to publish Writer metrics. Omitting it does
not change Writer behavior, but disables its metrics.


Sending Messages
Expand Down Expand Up @@ -381,8 +384,16 @@ Reader Parameters
consumer="my-consumer",
buffer_size_bytes=50 * 1024 * 1024, # client-side buffer (default: 50 MB)
buffer_release_threshold=0.5, # see below (default: 0.5)
reader_name="payments-worker", # optional diagnostic name
)

``reader_name`` identifies a logical reader in server-side diagnostics and SDK logs. If it is
omitted or empty, the SDK generates a process-local name in the form ``reader-N``. The name
remains unchanged when the reader reconnects. Explicit names do not have to be unique. Topic
Reader metrics are published only when the application supplies a
non-empty ``reader_name``. See :ref:`topic-reader-receive-metrics` for their semantics
and attributes.

``buffer_size_bytes`` controls how many bytes the server is allowed to send before the client
signals that it is ready for more. The server will not exceed this limit.

Expand Down
1 change: 1 addition & 0 deletions examples/opentelemetry/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,4 @@ RUN pip install --no-cache-dir -e '.[opentelemetry]' -r examples/opentelemetry/r
# Demo scripts.
COPY examples/opentelemetry/otel_example.py ./examples/opentelemetry/otel_example.py
COPY examples/opentelemetry/load_tank.py ./examples/opentelemetry/load_tank.py
COPY examples/opentelemetry/verify_metrics.py ./examples/opentelemetry/verify_metrics.py
Loading
Loading