Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions .github/workflows/slo-2dc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
name: Hosted 2DC SLO

on:
pull_request:
paths:
- .github/workflows/slo-2dc.yml
- tests/slo/**
- ydb/driver.py
- ydb/resolver.py
- ydb/pool.py
- ydb/aio/driver.py
- ydb/aio/pool.py
workflow_dispatch:
inputs:
server_image:
description: YDB server image to qualify (resolved to a digest)
default: cr.yandex/crpb3d0up7s6a7iuofo9/ydb-appteam-ci/local-ydb:trunk
required: true

permissions:
contents: read

jobs:
bridge:
runs-on: ubuntu-24.04
timeout-minutes: 30
concurrency:
group: hosted-2dc-${{ github.ref }}
cancel-in-progress: true
env:
WORKLOAD_MEMORY_LIMIT: 1G
WORKLOAD_CPU_LIMIT: '1.00'
WORKLOAD_CURRENT_IMAGE: py-slo-bridge:current
REQUESTED_IMAGE: ${{ inputs.server_image || 'cr.yandex/crpb3d0up7s6a7iuofo9/ydb-appteam-ci/local-ydb@sha256:bce9caa6a1a6aac9b513510b46f61c4d4b58c1f88d937bcb54c20dc1bda6c8d4' }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
repository: ydb-platform/ydb-slo-action
ref: 54c3c03de1ce8c6a2d8b0828b328bd5db82ef2bb
path: .slo-action
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install the Compose version required by the action
run: |
set -euo pipefail
mkdir -p "$HOME/.docker/cli-plugins"
curl -fsSL --retry 3 \
https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \
-o "$HOME/.docker/cli-plugins/docker-compose"
chmod +x "$HOME/.docker/cli-plugins/docker-compose"
docker compose version
docker compose up --help | grep -F -- --quiet-build
- name: Record runner and resolve server image
run: |
set -euo pipefail
mkdir -p bridge-artifacts
free -h
df -h
docker version
docker compose version
docker pull --platform linux/amd64 "$REQUESTED_IMAGE"
docker image inspect "$REQUESTED_IMAGE" > bridge-artifacts/server-image.json
SERVER_IMAGE=$(docker image inspect --format '{{index .RepoDigests 0}}' "$REQUESTED_IMAGE")
printf '%s\n' "$SERVER_IMAGE" > bridge-artifacts/server-image.txt
python tests/slo/bridge/prepare.py .slo-action "$SERVER_IMAGE"
- name: Build current Python SDK workload
run: docker build -f tests/slo/Dockerfile -t py-slo-bridge:current .
- name: Run all five Bridge scenarios
id: slo
uses: ./.slo-action/init
timeout-minutes: 20
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
workload_name: sync-query
workload_duration: '600'
workload_current_ref: current
workload_current_image: py-slo-bridge:current
workload_current_command: >-
--read-rps 50 --write-rps 10 --read-threads 2 --write-threads 2
--read-timeout 30000 --write-timeout 30000
--prefer-primary-pile
bridge_mode: true
disable_compose_profiles: extra-nodes
fail_on_workload_error: true
- name: Verify scenario coverage and workload recovery
if: always()
timeout-minutes: 3
env:
ACTION_OUTCOME: ${{ steps.slo.outcome }}
run: python tests/slo/bridge/verify.py
- name: Upload qualification evidence
if: always()
uses: actions/upload-artifact@v4
with:
name: python-hosted-2dc
path: bridge-artifacts/
retention-days: 14
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
* Add opt-in PRIMARY/PROMOTED Bridge pile preference with synchronized-pile fallback for sync and async drivers

## 3.35.0 ##
* Include streaming queries in scheme directory listings and path descriptions
* Add timeout to the token exchange request so a stalled token endpoint cannot hang credential refresh
Expand Down
26 changes: 26 additions & 0 deletions docs/driver.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ DriverConfig
* - ``discovery_request_timeout``
- ``10``
- Timeout in seconds for each discovery request.
* - ``prefer_primary_pile``
- ``False``
- Prefer PRIMARY/PROMOTED Bridge endpoints with synchronized-pile fallback; no effect without Bridge metadata.
* - ``table_client_settings``
- ``None``
- :class:`~ydb.TableClientSettings` instance to configure the Table service client.
Expand All @@ -62,6 +65,29 @@ DriverConfig
- gRPC-level compression (e.g. ``grpc.Compression.Gzip``).


Preferring the primary Bridge pile
----------------------------------------

Enable this policy explicitly for sync or async drivers:

.. code-block:: python

config = ydb.DriverConfig(
endpoint="grpc://discovery.example:2135",
database="/Root/mydb",
prefer_primary_pile=True,
)
driver = ydb.Driver(config)

PRIMARY and PROMOTED endpoints take priority over SYNCHRONIZED endpoints.
Other or unknown pile states are last-resort candidates. The policy follows role
changes from discovery without recreating the driver, and takes precedence over
local-DC preferences while Bridge metadata is present, even with ``use_all_nodes=True``.
Without Bridge metadata the existing balancing behavior is preserved.
Requests bound to a session retain their node affinity; the option does not move
an existing session or perform a server-side failover. It is disabled by default.


Creating a Driver
-----------------

Expand Down
177 changes: 177 additions & 0 deletions tests/aio/test_discovery_detect_local_dc.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# -*- coding: utf-8 -*-
import asyncio
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from ydb import driver, connection
from ydb import _apis, resolver
from ydb.aio import pool, _utilities


Expand Down Expand Up @@ -141,3 +143,178 @@ def mock_init(self, endpoint, driver_config, endpoint_options=None):
await discovery.execute_discovery()

assert detect_mock.call_count == 0, "detect_local_dc should NOT be called when use_all_nodes=True"


class PrimaryConnection:
failed = set()
created = []

def __init__(self, endpoint, driver_config, endpoint_options=None):
self.endpoint = endpoint
self.node_id = endpoint_options.node_id
self.callbacks = []
self.closed = False
self.created.append(self)

async def connection_ready(self, ready_timeout):
if self.endpoint in self.failed:
raise asyncio.TimeoutError()

def add_cleanup_callback(self, callback):
self.callbacks.append(callback)

async def close(self, grace=30):
self.closed = True
for callback in self.callbacks:
callback(self)


def primary_result(states, nodes=((1, "A"), (2, "B"))):
message = _apis.ydb_discovery.ListEndpointsResult(self_location="local")
for node_id, pile in nodes:
message.endpoints.add(
address=f"node{node_id}",
port=2135,
node_id=node_id,
location="local" if pile == "A" else "remote",
bridge_pile_name=pile,
)
for name, state in states.items():
message.pile_states.add(pile_name=name, state=state)
response = _apis.ydb_discovery.ListEndpointsResponse()
response.operation.ready = True
response.operation.status = 400000
response.operation.result.Pack(message)
return resolver.DiscoveryResult.from_response(None, response, use_all_nodes=True)


def primary_discovery(monkeypatch, result, enabled=True, use_all_nodes=True, failed=()):
PrimaryConnection.failed, PrimaryConnection.created = set(failed), []
monkeypatch.setattr(pool, "Connection", PrimaryConnection)
config = driver.DriverConfig("grpc://test:2135", "/local", use_all_nodes=use_all_nodes, prefer_primary_pile=enabled)
discovery = pool.Discovery(pool.ConnectionsCache(use_all_nodes), config)
discovery._resolver.resolve = AsyncMock(return_value=result)
return discovery


@pytest.mark.asyncio
async def test_primary_role_change_and_affinity(monkeypatch):
discovery = primary_discovery(monkeypatch, primary_result({"A": 1, "B": 3}))
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {1}
assert (await discovery._cache.get(connection.EndpointKey("node2:2135", 2))).node_id == 2
discovery._resolver.resolve.return_value = primary_result({"A": 3, "B": 1})
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {2}
assert len(PrimaryConnection.created) == 2
assert not any(conn.closed for conn in PrimaryConnection.created)
assert (await discovery._cache.get(connection.EndpointKey("node1:2135", 1))).node_id == 1
await (await discovery._cache.get()).close()
assert (await discovery._cache.get()).node_id == 1


@pytest.mark.asyncio
async def test_primary_off_and_ordinary_cluster_locality(monkeypatch):
discovery = primary_discovery(monkeypatch, primary_result({"A": 1, "B": 3}), enabled=False)
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {1, 2}
discovery = primary_discovery(monkeypatch, primary_result({}), use_all_nodes=False)
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {1}
discovery._resolver.resolve.return_value = primary_result({"A": 3, "B": 1})
assert await discovery.execute_discovery()
assert (await discovery._cache.get()).node_id == 2
discovery._resolver.resolve.return_value = primary_result({})
assert await discovery.execute_discovery()
assert (await discovery._cache.get()).node_id == 1


@pytest.mark.asyncio
async def test_primary_promoted_and_unknown_states(monkeypatch):
discovery = primary_discovery(
monkeypatch,
primary_result({"A": 1, "B": 2, "C": 3, "D": 99}, ((1, "A"), (2, "B"), (3, "C"), (4, "D"), (5, "missing"))),
)
assert await discovery.execute_discovery()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {1, 2}
await (await discovery._cache.get(connection.EndpointKey("node1:2135", 1))).close()
await (await discovery._cache.get(connection.EndpointKey("node2:2135", 2))).close()
assert (await discovery._cache.get()).node_id == 3
await (await discovery._cache.get()).close()
assert {(await discovery._cache.get()).node_id for _ in range(10)} == {4, 5}


@pytest.mark.asyncio
async def test_primary_full_cache_admission_and_failed_connection(monkeypatch):
old = tuple((i, "A") for i in range(1, 10))
discovery = primary_discovery(monkeypatch, primary_result({"A": 1}, old))
assert await discovery.execute_discovery() and discovery._cache.size == 9
discovery._resolver.resolve.return_value = primary_result({"A": 3, "B": 1}, old + ((10, "B"),))
PrimaryConnection.failed = {"node10:2135"}
assert await discovery.execute_discovery()
assert discovery._cache.size == 9
assert not any(conn.closed for conn in PrimaryConnection.created if conn.node_id != 10)
PrimaryConnection.failed.clear()
assert await discovery.execute_discovery()
assert (await discovery._cache.get()).node_id == 10
assert discovery._cache.size == 9
created_count = len(PrimaryConnection.created)
assert await discovery.execute_discovery()
assert len(PrimaryConnection.created) == created_count


@pytest.mark.asyncio
async def test_primary_cache_preserves_synchronized_fallback(monkeypatch):
nodes = tuple((i, "A") for i in range(1, 11)) + ((11, "B"),)
discovery = primary_discovery(monkeypatch, primary_result({"A": 1, "B": 3}, nodes))
assert await discovery.execute_discovery() and discovery._cache.size == 9
assert discovery._cache.already_exists("node11:2135")
for conn in discovery._cache.values():
if conn.node_id != 11:
await conn.close()
assert (await discovery._cache.get()).node_id == 11


@pytest.mark.asyncio
async def test_primary_alias_retirement_keeps_node_affinity():
cache = pool.ConnectionsCache()
options = connection.EndpointOptions(node_id=1)
old = PrimaryConnection("old:2135", None, options)
new = PrimaryConnection("new:2135", None, options)
cache.add(old)
cache.add(new)
await old.close()
assert await cache.get(connection.EndpointKey(None, 1)) is new


@pytest.mark.asyncio
async def test_primary_driver_keyword_and_config_override(monkeypatch):
from ydb import topic
from ydb.aio import driver as aio_driver, table, scheme, coordination

seen = []
monkeypatch.setattr(pool.ConnectionPool, "__init__", lambda self, config: seen.append(config))
for module, name in (
(topic, "TopicClientAsyncIO"),
(table, "TableClient"),
(scheme, "SchemeClient"),
(coordination, "CoordinationClient"),
):
monkeypatch.setattr(module, name, MagicMock())
aio_driver.Driver(endpoint="grpc://test:2135", database="/local", prefer_primary_pile=True)
assert seen[-1].prefer_primary_pile
config = aio_driver.DriverConfig("grpc://test:2135", "/local", prefer_primary_pile=True)
aio_driver.Driver(config)
assert seen[-1].prefer_primary_pile
aio_driver.Driver(config, prefer_primary_pile=False)
assert not seen[-1].prefer_primary_pile


@pytest.mark.asyncio
async def test_primary_discovery_cancellation_closes_candidate(monkeypatch):
discovery = primary_discovery(monkeypatch, primary_result({"A": 1, "B": 3}))
monkeypatch.setattr(PrimaryConnection, "connection_ready", AsyncMock(side_effect=asyncio.CancelledError))
with pytest.raises(asyncio.CancelledError):
await discovery.execute_discovery()
assert len(PrimaryConnection.created) == 1
assert PrimaryConnection.created[0].closed
16 changes: 16 additions & 0 deletions tests/slo/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,21 @@
# SLO workload

## GitHub-hosted Bridge SLO

`.github/workflows/slo-2dc.yml` runs the standard five Bridge scenarios from the
pinned SLO Action v2 on `ubuntu-24.04`. The ordinary `sync-query` workload runs for
10 minutes with `--prefer-primary-pile` enabled.

The runner uses two storage and two database nodes, `erasure=none` and SectorMap disks.
The adapter bounds resources and cold-start readiness time, restricts bootstrap DNS
to active compute nodes, and sets a 30-second gap between fault scenarios to fit the
10-minute workload. The upstream scenario scripts and SDK discovery intervals are unchanged.

The job requires all five scenarios to complete, successful recent reads and writes,
and no workload failure, OOM or unexpected container restart. Transient operation
errors are recorded. Server image metadata, workload and chaos logs, final container
states and counters are retained in the `python-hosted-2dc` artifact for 14 days.

SLO is the type of test where app based on ydb-sdk is tested against falling YDB cluster nodes, tablets, network
(that is possible situations for distributed DBs with hundreds of nodes)

Expand Down
1 change: 1 addition & 0 deletions tests/slo/bridge/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
__pycache__/
Loading
Loading