Skip to content

feat cdc_aggregator : compute the stream heartbeat in the CdcAggregator tablet - #59751

Draft
rgnlax wants to merge 57 commits into
ydb-platform:mainfrom
rgnlax:cdc-aggregator-tablet-heartbeats
Draft

rgnlax wants to merge 57 commits into
ydb-platform:mainfrom
rgnlax:cdc-aggregator-tablet-heartbeats

Conversation

@rgnlax

@rgnlax rgnlax commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

KIKIMR-27708

The CdcAggregator tablet now accepts TEvWriteHeartbeats: it persists the last heartbeat of each source DataShard and advances the stream heartbeat to the minimum once every source has reported. The stream heartbeat never moves back, including after split/merge and tablet restarts. An in-memory index of sources ordered by version keeps a write at O(log S) for tables with ~100k shards.

Builds on #59120 (its commits are included until it is merged). The unused Heartbeats table (id 3) is dropped from the schema; table id 3 must not be reused.

Maksim Pedchenko and others added 30 commits November 10, 2026 22:37
Fix every boundary of the CdcAggregator work (KIKIMR-27694) before the
implementation, so the tablet, the node service, the CDC sender, the PersQueue
publication and the SchemeShard lifecycle can be developed in parallel.

Adds the CdcAggregator tablet type, its event space and log component, the
tablet protocol, the local service protocol, the PersQueue publish command,
the stream barrier mode with the topic binding and the EnableCdcAggregator
feature flag. No behaviour changes.
Diagnostics need the committed state of a stream, in particular the sources
that hold the barrier back. Also fixes type references in the protocol.
Model of one CDC stream for CdcAggregator (KIKIMR-27708): the set of sources,
their delivered progress and the ready barrier as the minimum over all of them.
The aggregate keeps the invariants (monotonic progress and barrier, a silent
source blocks the barrier, atomic replacement on split/merge, a retired source
never returns); use cases validate requests and persist changes through a
repository port. No actors, LocalDB or protobuf, so it is tested with plain
command sequences.
Tablet of KIKIMR-27708 that runs the stream use cases in LocalDB
transactions: Configure, Activate, ReportDeliveredBatch, ReplaceSource and
DescribeStream. Every reply and the publishable barrier leave the tablet only
from Complete of their own transaction, so nothing is acknowledged before it
is committed; after a restart the stream is restored with its progress,
barrier and replacements. The factory is registered for nodes and test
runtimes; publication to PersQueue is left to its own stage.
Model of one CDC stream for CdcAggregator (KIKIMR-27708): the set of sources,
their delivered progress and the ready barrier as the minimum over all of them.
The aggregate keeps the invariants (monotonic progress and barrier, a silent
source blocks the barrier, atomic replacement on split/merge, a retired source
never returns); use cases validate requests and persist changes through a
repository port. No actors, LocalDB or protobuf, so it is tested with plain
command sequences.
The tablet is registered in Local, creates its LocalDB tables, loads its
state and becomes active. Event handlers come with the tracking logic.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Rebuild the branch as the tablet skeleton on top of main with the merged
contracts; the tracking logic moves to a separate pull request.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Pure library for the CdcAggregator tablet: tracked sources, the minimum
heartbeat over them in O(log S), split/merge source replacement and use
cases speaking the TEvConfigureTracking, TEvWriteHeartbeats,
TEvPartitioningChanged and TEvDescribeTracking contracts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Name DataShard parameters after the contract fields, make the change flags
read as flags and drop batch and payload limits that the contract does not
define.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The tablet restores the stored tracking config through the same parsing
that validates ConfigureTracking requests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Handle ConfigureTracking, WriteHeartbeats and PartitioningChanged in tablet
transactions that persist the stream changes and reply after commit;
DescribeTracking answers from memory. TxInit restores the stream from
LocalDB.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The tablet persists the tracking config and the source DataShards, replaces a
parent with its children on PartitioningChanged and restores both on restart.
Heartbeats are still answered with NOT_READY.

Transaction types are no longer aligned and tabletId moves to the log context,
as requested in the review of the skeleton.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Keep the protobuf response alive until the sources have been copied. This fixes SIGSEGV in the five tablet tests that describe non-empty tracking state.
…the tablet test

Select the source-list overload explicitly for the empty configuration. This preserves the BAD_REQUEST case after adding the raw-request overload.
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-17 02:27:21 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8c7c2a5 has started.
⚪ 2026-11-17 02:30:14 UTC Artifacts will be uploaded here
⚪ 2026-11-17 02:31:46 UTC ya make is running...
⚫ 2026-11-17 03:07:20 UTC Check cancelled

@github-actions

Copy link
Copy Markdown
Contributor

✅ Documentation build

Revision built successfully
Revision preview link

@robot-vibe-db robot-vibe-db Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Summary

Verdict: ✅ No Critical or Major issues found

Critical and Major issues

No Critical or Major issues found.

Other findings

  • Minor | Medium: Duplicate source entries in one TEvWriteHeartbeats batch are silently deduplicated last-wins and only one result entry is reported for the two submitted heartbeats — ydb/core/tx/cdc_aggregator/tablet/cdc_aggregator_stream_proto.cpp:108
  • Minor | Medium: Replaying an already-applied partitioning change after the completion notification has been processed (history pruned) returns UNKNOWN_SOURCE for a change that was successfully applied — ydb/core/tx/cdc_aggregator/tablet/cdc_aggregator_stream.cpp:78
  • Minor | Medium: No test covers a heartbeat batch containing duplicate source entries, so the last-wins dedup behavior in ParseHeartbeatsBatch is unspecified by tests — ydb/core/tx/cdc_aggregator/tablet/ut/tablet_ut.cpp:99
  • Nit | High: The !diff->Deleted.empty() guard is misleading — for any freshly applied change Deleted always contains the parent, so the guard effectively means "not a duplicate replay" — ydb/core/tx/cdc_aggregator/tablet/tx_partitioning_changed.cpp:58
  • Nit | Medium: The old dead Heartbeats table (id 3) is not actually dropped from existing deployments (db.Materialize<Schema>() never removes tables absent from the schema), and the "table id 3 must not be reused" constraint is only stated in the PR description — ydb/core/tx/cdc_aggregator/tablet/schema.h:31

This review was generated automatically. Critical and Major issues require attention; Minor and Nit findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.

batch.InvalidSourceDataShardIds.insert(sourceDataShardId);
continue;
}
batch.HeartbeatsBySourceDataShardId[sourceDataShardId] = std::move(heartbeat);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

batch.HeartbeatsBySourceDataShardId[sourceDataShardId] = std::move(heartbeat); silently overwrites an earlier entry for the same source, so a TEvWriteHeartbeats request containing the same SourceDataShardId twice produces only one THeartbeatResult (from batch.HeartbeatsBySourceDataShardId) for two submitted heartbeats, and only the last heartbeat survives — the earlier one is neither rejected nor acknowledged.

ParseHeartbeatsBatch carefully validates every other edge case (zero step, empty data, unknown source) but treats duplicate entries as an unspecified last-wins merge. If duplicates are a protocol violation, mark them BAD_REQUEST like other invalid inputs; if they are legal, document the last-wins semantics so callers know the per-source result count may be smaller than the request count.

Suggested fix: detect duplicates (e.g. check count(sourceDataShardId) before assignment or track a seen-set) and add the entry to InvalidSourceDataShardIds with a BAD_REQUEST result.

}

const auto parent = LastHeartbeatsBySourceDataShardId.find(change.ParentDataShardId);
if (parent == LastHeartbeatsBySourceDataShardId.end()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

A replay of an already-applied partitioning change now returns std::nullopt (→ UNKNOWN_SOURCE ack) once the corresponding history rows have been pruned by TTxForgetPartitioningChanges (which runs as soon as the SchemeShard completion notification arrives). Before this PR the ack of a partitioning change was at least persistent across restarts; after completion processing, the same retried TEvPartitioningChanged — e.g. the original ack was lost in transit and the sender retries — gets an error for a change that was applied and is reflected in the source set.

The pruning is clearly intentional (history cleanup), and the retrying sender can recover by re-reading the source set, but the returned error code conflates "change never seen" with "change seen, applied, and forgotten". Consider a distinct status (e.g. SUCCESS/ALREADY_APPLIED for a change whose parent is no longer tracked and whose children are all known) or documenting that UNKNOWN_SOURCE may arrive for acknowledged changes after completion, so the future caller of this API can handle retries safely.

}

void RebootTablet() {
CompletionSubscribers.clear();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

All new heartbeat tests submit at most one entry per source per batch. The last-wins deduplication of duplicate SourceDataShardId entries in ParseHeartbeatsBatch (and the fact that the result then contains fewer THeartbeatResult entries than request entries) is not covered by any test, so that behavior is unspecified and could change unnoticed. Consider adding a test that sends {{1,100},{1,200}} and asserts both the resulting sources' heartbeat and the result-entry count.

return true;
}

if (!diff->Deleted.empty()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Nit
Confidence: High

if (!diff->Deleted.empty()) reads as a special case for inserts-only diffs, but TCdcAggregatorStream::ApplyPartitioningChange always inserts the parent into diff.Deleted for a freshly applied change; the only case with empty Deleted is a duplicate replay (which returns an empty diff to skip persistence). Suggest naming the intent explicitly, e.g. if (!applied duplicate) via a dedicated flag on the diff (or an early return true before the persistence block for replays), so future readers do not assume inserts-only changes exist.

struct VersionStep : Column<1, NScheme::NTypeIds::Uint64> {};
struct VersionTxId : Column<2, NScheme::NTypeIds::Uint64> {};
struct Data : Column<3, NScheme::NTypeIds::String> {};
struct PartitioningChanges : Table<4> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Nit
Confidence: Medium

Table id 3 is being freed by removing Heartbeats from the schema. Note that TTxInitSchema runs db.Materialize<Schema>(), which only creates the tables/columns listed in the schema and never removes tables absent from it — so on existing deployments the old table 3 and its data remain in the tablet. That is harmless (the table was unused), but the constraint "table id 3 must not be reused" is only stated in the PR description and not visible to future editors of this file. Consider adding a short comment near the PartitioningChanges : Table<4> definition, e.g. // Table id 3 was used by the removed Heartbeats table; do not reuse id 3.

Maksim Pedchenko added 2 commits November 17, 2026 06:01
Activate the tablet with an explicit configuration load error and reject reconfiguration instead of overwriting persisted sources and partitioning history. Cover malformed and semantically invalid blobs, repeated reboots, and recovery after restoring the original configuration.
Merge the tracking dependency while retaining per-source and current heartbeat persistence and the transactional heartbeat writer.
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-17 03:09:46 UTC Pre-commit check linux-x86_64-relwithdebinfo for b2e867f has started.
⚪ 2026-11-17 03:10:44 UTC Artifacts will be uploaded here
⚪ 2026-11-17 03:12:13 UTC ya make is running...
🔴 2026-11-17 03:58:53 UTC Build failed, see the logs. Also see fail summary
🟡 2026-11-17 03:59:22 UTC ydbd size 2.9 GiB changed* by +20.4 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 7c1bcec merge: b2e867f diff diff %
ydbd size 3 045 473 272 Bytes 3 066 883 736 Bytes +20.4 MiB +0.703%
ydbd stripped size 643 672 192 Bytes 646 109 216 Bytes +2.3 MiB +0.379%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

Copy link
Copy Markdown
Contributor

✅ Documentation build

Revision built successfully
Revision preview link

@robot-vibe-db robot-vibe-db Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Summary

Verdict: ❌ Critical: 1; Major: 1

Critical and Major issues

  • Critical | High: The PR head does not build: linking ydb/core/tx/cdc_aggregator/tablet/ut fails with undefined Out<NKikimrCdcAggregator::TEvConfigureTrackingResult_EStatus> and Out<NKikimrCdcAggregator::TEvPartitioningChangedAck_EStatus> (confirmed by CI run 44007853556). The new tests compare these proto enums with UNIT_ASSERT_VALUES_EQUAL, which requires Out<> symbols generated by GENERATE_ENUM_SERIALIZATION(cdc_aggregator.pb.h), missing in ydb/core/protos/ya.make — ydb/core/tx/cdc_aggregator/tablet/ut/tablet_ut.cpp:216
  • Major | Medium: tablet_impl.h includes <ydb/core/tx/schemeshard/schemeshard.h> but ydb/core/tx/cdc_aggregator/tablet/ya.make does not add PEERDIR(ydb/core/tx/schemeshard), so compilation relies on include dirs leaking in transitively from other targets; in stricter/other build configurations the include is not resolvable — ydb/core/tx/cdc_aggregator/tablet/ya.make:16

Other findings

  • Minor | Medium: After a split, if the deleted parent had the highest observed heartbeat but the stream heartbeat was still unset (another source lagged), children may report lower versions and the stream heartbeat starts below the version the parent had already reached — the reported watermark can regress relative to data the deleted parent had durably written — ydb/core/tx/cdc_aggregator/tablet/cdc_aggregator_stream.cpp:60
  • Minor | Medium: A TEvWriteHeartbeats batch containing two entries for the same source, one valid and one invalid, produces two THeartbeatResult records for the same SourceDataShardId (one SUCCESS and one BAD_REQUEST); the client cannot resolve which entry was applied — ydb/core/tx/cdc_aggregator/tablet/cdc_aggregator_stream_proto.cpp:132
  • Minor | High: TTxWriteHeartbeats persists every heartbeat in the batch even when it did not advance the source's last heartbeat (RegisterHeartbeats returns unchanged entries too), so repeated or stale heartbeat batches rewrite unchanged Sources rows on every write — ydb/core/tx/cdc_aggregator/tablet/tx_write_heartbeats.cpp:59
  • Minor | Medium: A redelivered TEvPartitioningChanged for an operation that was already applied, completed at SchemeShard and forgotten gets UNKNOWN_SOURCE even though the change was applied; senders that treat UNKNOWN_SOURCE as a permanent failure will not re-ack, and the semantics differ from an in-flight replay (which returns SUCCESS) — ydb/core/tx/cdc_aggregator/tablet/tx_partitioning_changed.cpp:55
  • Minor | Low: On a failed pipe connect (TEvClientConnected with non-OK status) the tablet closes and reschedules the pipe itself via a 1s TEvWakeup, while the pipe client was created with TClientRetryPolicy::WithRetries() and retries internally; the two overlapping retry loops can postpone resubscription of completion notifications beyond either policy's intent — ydb/core/tx/cdc_aggregator/tablet/tablet.cpp:108
  • Minor | Low: LoadLastHeartbeats guards only LastHeartbeatStep with HaveValue and then reads LastHeartbeatTxId/LastHeartbeatData unconditionally; a row where LastHeartbeatStep is set but LastHeartbeatData is missing aborts the tablet at boot instead of surfacing a recoverable load error — ydb/core/tx/cdc_aggregator/tablet/cdc_aggregator_stream_repository.cpp:104

This review was generated automatically. Critical and Major issues require attention; Minor and Nit findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.


void CheckInvalidPersistedTrackingConfig(const TString& config, const TString& error) {
TTestEnv env;
UNIT_ASSERT_VALUES_EQUAL(env.ConfigureTracking({1, 2}).GetStatus(), TConfigureResult::SUCCESS);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Critical
Confidence: High

The PR head fails to link: ydb/core/tx/cdc_aggregator/tablet/ut fails with undefined symbol: void Out<NKikimrCdcAggregator::TEvConfigureTrackingResult_EStatus>(IOutputStream&, ...) and Out<NKikimrCdcAggregator::TEvPartitioningChangedAck_EStatus> (see CI run 44007853556, fail_summary.txt). These symbols are required because the new tests compare proto enum values with UNIT_ASSERT_VALUES_EQUAL(...GetStatus(), TConfigureResult::SUCCESS) here and throughout the suite, which instantiates CompareAndMakeStrings/Out<> for the enum type.

The symbols are normally provided by GENERATE_ENUM_SERIALIZATION, but ydb/core/protos/ya.make does not generate enum serialization for cdc_aggregator.pb.h (only blobstorage_pdisk_config.pb.h, datashard_load.pb.h, shared_cache.pb.h are listed there).

Suggested fix: add GENERATE_ENUM_SERIALIZATION(cdc_aggregator.pb.h) to ydb/core/protos/ya.make (or avoid streaming/comparing these enums directly in the tests).

tx_write_heartbeats.cpp
)

PEERDIR(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Major
Confidence: Medium

tablet_impl.h now includes <ydb/core/tx/schemeshard/schemeshard.h> (needed for NSchemeShard::TEvSchemeShard::TEvNotifyTxCompletion/TEvNotifyTxCompletionResult), but this module does not declare PEERDIR(ydb/core/tx/schemeshard). I traced the PEERDIR closure of all peers listed here (ydb/core/base, ydb/core/tablet, ydb/core/tablet_flat, ydb/core/persqueue/common, etc.) — none of them reaches ydb/core/tx/schemeshard, so the include resolves only via transitive include-directory leakage from unrelated targets. That makes the build fragile and breaks in configurations where those leakage paths differ (Arcadia requires an explicit PEERDIR for cross-module includes).

Suggested fix: add ydb/core/tx/schemeshard to PEERDIR here, and also add it to the ut/ya.make PEERDIR set (its tablet_ut.cpp includes ydb/core/tx/schemeshard/schemeshard.h as well; ydb/core/testlib/default does not export it).

if (SourceDataShardIdsByLastHeartbeatVersion.size() != LastHeartbeatsBySourceDataShardId.size()) {
return std::nullopt;
}
const auto& [minVersion, minSourceDataShardId] = *SourceDataShardIdsByLastHeartbeatVersion.begin();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

The "stream heartbeat never moves back" invariant only holds while CurrentHeartbeat is already set. Scenario: configure sources {1, 2}, source 1 reports version 100 (source 2 has not reported, so CurrentHeartbeat is still unset), then shard 1 splits into 3 and 4. The parent's heartbeat (100) is dropped together with the parent; children start with no heartbeat. When children later report versions lower than 100 (e.g. they restart from a lower resolved timestamp) and source 2 catches up, the stream heartbeat is established at the children's lower version — i.e. below the version the deleted parent had already durably reached.

The existing test ShouldNotMoveCurrentHeartbeatBackWhenChildReportsOlderVersion only covers the case where the stream heartbeat was already advanced before the split. Consider seeding the new children with a floor (e.g. the parent's last heartbeat version at split time) to keep the watermark monotonic across the parent removal, or documenting that a fresh child reset may lower the first-published watermark.


TResult proto;
proto.SetStatus(TResult::SUCCESS);
for (const auto& [sourceDataShardId, _] : batch.HeartbeatsBySourceDataShardId) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

A TEvWriteHeartbeats request with two entries for the same SourceDataShardId — one parseable and one invalid — yields two THeartbeatResult records for the same source: one SUCCESS (from HeartbeatsBySourceDataShardId, because the earlier valid entry wins) and one BAD_REQUEST (from InvalidSourceDataShardIds). Conversely, two valid entries silently keep only the last one. A client matching results back to its own entries by position or by id gets an ambiguous outcome.

Suggested fix: on a duplicate source id either reject the whole batch entry set for that source with BAD_REQUEST, or make the last entry win consistently in both maps.

const auto batch = Self->CdcAggregatorStreamProtoConverter.ParseHeartbeatsBatch(Ev->Get()->Record);
const auto diff = Self->CdcAggregatorStream->WriteHeartbeats(batch.HeartbeatsBySourceDataShardId);
NIceDb::TNiceDb db(txc.DB);
Self->CdcAggregatorStreamRepository.ApplyCdcAggregatorStreamHeartbeatsDiff(db, diff);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: High

RegisterHeartbeats includes every heartbeat from the batch in the returned diff, even when it did not advance the source's last heartbeat (an older or equal version returns the unchanged value), and ApplyCdcAggregatorStreamHeartbeatsDiff then rewrites the Sources row with identical values. DataShards resend heartbeats periodically, so batches that change nothing still produce a full set of row updates (3 columns per source) in every such transaction.

Suggested fix: return only sources whose version actually advanced from RegisterHeartbeats (or filter unchanged entries before persisting) and report SUCCESS with the already-written version for the rest.

const auto diff = Self->CdcAggregatorStream->ApplyPartitioningChange(*change);
if (!diff) {
Result->Record.SetStatus(TResult::UNKNOWN_SOURCE);
Result->Record.SetError(TStringBuilder() << "Unknown partitioning change " << change->OperationId);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

Once an operation completes at SchemeShard, TTxForgetPartitioningChanges erases the (parent, operationId) history. A TEvPartitioningChanged redelivered after that point reaches ApplyPartitioningChange, no longer finds the parent among the sources, and is answered UNKNOWN_SOURCE — even though the change was applied and acknowledged earlier. An in-flight replay (operation not yet completed) instead returns SUCCESS. The same sender therefore sees two different outcomes for semantically identical retries, and a sender that maps UNKNOWN_SOURCE to "this shard is not tracked, stop retrying" (as opposed to treating it as retryable) would silently drop a retry after an ack was lost.

The tests codify the UNKNOWN_SOURCE behavior, so this may be intended, but the distinction between "never applied" and "applied and forgotten" is lost on the wire. Consider a dedicated status (or SUCCESS) for replays of forgotten-but-applied operations if senders need to distinguish them.

result->Record.SetStatus(NKikimrCdcAggregator::TEvPartitioningChangedAck::NOT_READY);
result->Record.SetOrigin(TabletID());
ctx.Send(ev->Sender, result.Release(), 0, ev->Cookie);
void TCdcAggregator::Handle(TEvTabletPipe::TEvClientConnected::TPtr& ev, const TActorContext& ctx) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Low

The pipe client is created with TClientRetryPolicy::WithRetries() (unlimited internal retries with up to 10-minute backoff), but the owner additionally reacts to every failed connect by closing the client and scheduling a 1-second TEvWakeup retry. With WithRetries the client usually keeps retrying internally and does not report each failure to the owner; when it does report (e.g. resolve failure paths), the owner tears down a client that was about to retry on its own. The two overlapping recovery mechanisms make the actual resubscription delay hard to reason about. Consider relying on the pipe client's own retry policy and only reacting to TEvClientDestroyed, or using WithoutRetries plus the owner-driven wakeup loop.

TCdcAggregatorStream::TLastHeartbeats lastHeartbeats;
while (!rowset.EndOfSet()) {
auto& lastHeartbeat = lastHeartbeats[rowset.GetValue<Schema::Sources::DataShardId>()];
if (rowset.HaveValue<Schema::Sources::LastHeartbeatStep>()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Low

The row is accepted whenever LastHeartbeatStep has a value, but LastHeartbeatTxId and LastHeartbeatData are then read unconditionally. ApplyCdcAggregatorStreamHeartbeatsDiff always writes all three columns together, so this cannot happen with the current writer, but a partially written row (future schema change, manual edit, backup/restore skew) would abort the tablet during boot inside LoadLastHeartbeats instead of producing a recoverable load error like the TrackingConfig path does. Consider validating all heartbeat columns with HaveValue before constructing the heartbeat.

Maksim Pedchenko added 2 commits November 17, 2026 07:19
Keep the recovery assertions unchanged while reporting complete protobuf responses instead of requiring missing enum output specializations.
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

⚪ 2026-11-17 04:28:18 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2af1462 has started.
⚪ 2026-11-17 04:28:38 UTC Artifacts will be uploaded here
⚪ 2026-11-17 04:30:01 UTC ya make is running...
🟡 2026-11-17 05:43:35 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
62797 60979 0 6 1803 9

⚪ 2026-11-17 05:43:54 UTC ya make is running... (failed tests rerun, try 2)
🟢 2026-11-17 05:45:08 UTC Tests successful.

Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
98 (only retried tests) 97 0 0 0 1

🟢 2026-11-17 05:45:12 UTC Build successful.
🟡 2026-11-17 05:45:40 UTC ydbd size 2.9 GiB changed* by +20.4 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 7c1bcec merge: 2af1462 diff diff %
ydbd size 3 045 473 272 Bytes 3 066 883 736 Bytes +20.4 MiB +0.703%
ydbd stripped size 643 672 192 Bytes 646 109 216 Bytes +2.3 MiB +0.379%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions

Copy link
Copy Markdown
Contributor

✅ Documentation build

Revision built successfully
Revision preview link

@robot-vibe-db robot-vibe-db Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Summary

Verdict: ✅ No Critical or Major issues found

Critical and Major issues

No Critical or Major issues found.

Other findings

  • Minor | Medium: tablet_impl.h includes <ydb/core/tx/schemeshard/schemeshard.h> but the tablet LIBRARY does not declare a PEERDIR on ydb/core/tx/schemeshard; tests only build via the transitive ydb/core/testlib dependency — ydb/core/tx/cdc_aggregator/tablet/ya.make:16
  • Minor | Medium: the removed Heartbeats (Table<3>) is never dropped from existing LocalDBs (Materialize<Schema>() does not drop removed tables), leaving orphaned data; consider db.Cleanup<Schema>() in TxInitSchema — ydb/core/tx/cdc_aggregator/tablet/tx_init_schema.cpp:23
  • Minor | High: restart tests never assert the persisted heartbeat Data payload, so a regression in the LastHeartbeatData/CurrentHeartbeatData persistence path would go unnoticed — ydb/core/tx/cdc_aggregator/tablet/ut/tablet_ut.cpp:820
  • Nit | High: !diff->Deleted.empty() is always true for freshly applied changes (parent is always deleted) and only filters replays; the persistence gate would be clearer as an explicit "applied for the first time" flag — ydb/core/tx/cdc_aggregator/tablet/tx_partitioning_changed.cpp:59
  • Nit | High: duplicate heartbeats for the same source in one batch silently resolve last-wins instead of being rejected like the other malformed inputs — ydb/core/tx/cdc_aggregator/tablet/cdc_aggregator_stream_proto.cpp:108

This review was generated automatically. Critical and Major issues require attention; Minor and Nit findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.

tx_write_heartbeats.cpp
)

PEERDIR(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

tablet_impl.h includes <ydb/core/tx/schemeshard/schemeshard.h> (for NSchemeShard::TEvSchemeShard::TEvNotifyTxCompletion / TEvNotifyTxCompletionResult), but this LIBRARY does not declare a PEERDIR on ydb/core/tx/schemeshard. I traced the PEERDIR closure of ydb/core/tx/cdc_aggregator/tablet and ydb/core/tx/schemeshard is not reachable from it; the unit tests only build because ut -> ydb/core/testlib/default -> ydb/core/testlib -> ydb/core/tx/schemeshard happens to provide the include path transitively. The library itself relies on permissive include-path resolution, so strict dependency checking (or building/consuming this LIBRARY outside the test world, e.g. from a new caller) can break.

Please add ydb/core/tx/schemeshard to the PEERDIR list (other subscribers to TEvNotifyTxCompletion, like ydb/core/kqp/gateway/actors, declare it explicitly).

YDB_LOG_DEBUG("Execute");

NIceDb::TNiceDb db(txc.DB);
db.Materialize<Schema>();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

The PR removes Heartbeats : Table<3> from the schema, but db.Materialize<Schema>() only creates tables/columns that are missing (EMaterializationMode::All calls AddTable/AddColumn); it never drops tables that are no longer part of the schema. A CdcAggregator tablet whose LocalDB was created while the intermediate schema (with table 3) was deployed keeps the orphaned Heartbeats table and its rows forever. The PR description correctly says table id 3 must not be reused, but nothing cleans up the leftover data.

Consider using db.Cleanup<Schema>() here (it drops tables and columns that are absent from the current schema, see NIceDb::Schema::Cleanup in flat_cxx_database.h), or add a one-time migration that drops table 3.

UNIT_ASSERT_VALUES_EQUAL_C(describeResult.GetCurrentHeartbeat().GetStep(), 200, describeResult.ShortDebugString());
}

Y_UNIT_TEST(ShouldKeepHeartbeatsWhenTabletRestarted) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: High

Test gap: restart round-trip of the persisted heartbeat Data payload is never asserted. ShouldKeepHeartbeatsWhenTabletRestarted (and the other restart tests) only compare Step values, while LoadLastHeartbeats/LoadCdcAggregatorStream additionally read Sources::LastHeartbeatData and State::CurrentHeartbeatData. Note that TEvDescribeTrackingResult does not expose the data at all (only versions), so if persistence of the data columns regressed (e.g. the LastHeartbeatData/CurrentHeartbeatData update was dropped or reordered), no existing test would notice, and the payload would silently be lost (or replaced by an empty string) after a tablet restart.

Suggest asserting the persisted bytes via a LocalQuery on the Sources/State tables (the test already uses this technique in ReadTrackingConfig) in at least one restart test.

return true;
}

if (!diff->Deleted.empty()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Nit
Confidence: High

For every newly applied change diff.Deleted always contains the parent (TCdcAggregatorStream::ApplyPartitioningChange inserts it unconditionally), and the only path that reaches this line with an empty Deleted is the replayed-identical-change early return. So !diff->Deleted.empty() is effectively "this is a fresh apply, not a replay", which is hard to see from the condition and easy to break later (e.g. if a change type that only inserts sources is added, persistence would be silently skipped).

Suggest making the intent explicit, e.g. have ApplyPartitioningChange return a flag (or std::optional vs. empty-diff distinction) that says whether the change was applied for the first time, and gate persistence on that.

batch.InvalidSourceDataShardIds.insert(sourceDataShardId);
continue;
}
batch.HeartbeatsBySourceDataShardId[sourceDataShardId] = std::move(heartbeat);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Nit
Confidence: High

Duplicate entries for the same SourceDataShardId inside one TEvWriteHeartbeats batch are accepted and silently resolve to last-wins (batch.HeartbeatsBySourceDataShardId[sourceDataShardId] = ... overwrites the earlier entry), and the reply reports SUCCESS for the surviving version only. Every other malformed input (zero step, empty data, missing heartbeat) is rejected with BAD_REQUEST, so silently coalescing duplicates is inconsistent; the sender gets no signal that part of its batch was discarded.

If duplicates are expected from senders, a comment documenting last-wins would help; otherwise mark duplicates as BAD_REQUEST like the other invalid inputs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant