any-sync v0.13.x: files-v2, resharding, pubsub relay - #237
Open
cheggaaa wants to merge 27 commits into
Open
Conversation
Co-Authored-By: Claude Fable 5 <[email protected]>
Co-Authored-By: Claude Opus 4.8 <[email protected]>
.diskgen marker in the storage root; DiskGen().Fresh() reports a node started on an empty root (new node or replaced disk) so resharding machinery can treat local absence as 'missing', never 'deleted'. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
…er, AdoptArchive RPC The data plane for resharding: spaces move between tree nodes as archive snapshots inside a shared S3 bucket, never streamed node-to-node. - archivestore: Exists (HEAD), Key, CopyFrom (server-side CopyObject), Shared config flag gating the migration machinery - archive.ForceArchive: snapshot a live space to S3 without deleting the local DB or flipping its status (uses DumpStorage, works on open spaces) - archive.Restore keeps the S3 object (copy, not move): idempotent restores; stale objects are cleaned by the next archive cycle / archive sweeper - archive.QueueRestore + background worker: eager restore of adopted spaces - AdoptArchive RPC (nodesyncproto): receiver copies the offered snapshot into its own prefix, verifies with HEAD, registers Archived+heads in its index and nodehead; node-only, rejects tombstoned spaces, repairs lost objects - indexstorage.MarkArchivedRemote: register a never-opened space as archived Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
The draining side of resharding, gated on the shared archive store: - resharder component: on nodeconf change (and periodically), finds local spaces this node is no longer responsible for and hands each off through the shared bucket: archived spaces via CopyObject+AdoptArchive with zero restores; live spaces via ForceArchive snapshot + eager adoption, with the post-snapshot delta converged by tree sync - deletion safety: local copy removed only after >=2 current owners durably hold the same heads (adopt Ok / AlreadyHaveSame); diverged owners trigger hotsync convergence and the space parks for the next cycle; heads recheck before deletion catches writes landed mid-handoff - AdoptArchive responses are head-aware (AlreadyHaveSame vs Diverged) so a stale owner copy can never justify deleting the newest replica - SpaceStatusMoved terminal status; drained spaces leave ldiff/nodehead - spacedeleter: archived spaces delete their S3 object directly instead of restoring first; live deletions clean up leftover archive objects Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
- archivestore.List: paginated iteration of this node's prefix - daily archive sweep: deletes objects that no longer back an Archived index entry (restore leftovers, finished handoffs, deleted spaces, wiped-disk leaks); a 7-day age guard protects in-flight handoffs and fresh restores - resharder reconciles space dirs without index entries before each drain cycle so legacy orphans become drainable Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
…+ real-S3 tests Verified against Google Cloud Storage S3 interop: HEAD responses can be stale for seconds after a DELETE (observed 200 for a just-deleted object while ListObjectsV2 already showed it gone). Exists backs the durable-ACK check of the drain handoff, where a stale positive could acknowledge an object that no longer exists — reimplemented over ListObjectsV2 with an exact-key match, which proved read-after-write consistent. - TestRealS3 (archivestore): opt-in conformance test for Put/Get/Exists/ CopyFrom/List/Delete against a real bucket (ARCHIVE_TEST_S3_* env vars) - the drain integration test now runs against the real bucket too when the same env vars are set; verified against GCS Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Findings from a high-effort multi-agent review, all confirmed and fixed: - drain handoff: heads are now read from the snapshot itself (ForceArchive returns them) so ACKed hashes exactly describe the uploaded object; the drain parks when the index moves past the snapshot, and the pre-delete recheck compares against the snapshot heads - adopter: RemovePrepare (cancellable) no longer reads as 'deleted' — new ErrSpacePendingDeletion parks the sender instead of dropping its copy; only the final Remove status returns ErrSpaceDeleted - adopter: receivers reject AdoptArchive for spaces they are not responsible for (ErrNotResponsible) so a stale/misbehaving sender can't count a non-owner as a durable ACK - resharder: empty owner list parks instead of deleting with zero handoffs (a force-applied broken config could otherwise wipe drained partitions) - resharder: the archived-but-object-missing repair path flips the status to Ok before snapshotting — previously it triggered a doomed restore whose error path deleted the only remaining local db - archive.Restore: no longer removes the space dir when the archive object is missing (nothing was extracted; the dir may hold a live db) - sweeper: serialized with the archiver via a mutex, decisions re-read the index under the lock; Ok-status objects are only collected when the local db actually exists; unknown objects are kept until the storage root is 30 days old (fresh-disk protection); Moved/Remove are collected as before - spacedeleter: deletion-cancel keeps archived spaces Archived (previously flipped to Ok, presenting the only copy as sweeper garbage); final deletion also removes lingering space dirs of archived spaces - coordinator: unique partial index on epoch + duplicate-key retry makes epoch assignment safe under concurrent confapply runs Verified: full unit suites + drain integration test against real GCS. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
- adopter recognizes senders that were tree nodes in recently retained config epochs (nodeconf history): a node removed from the configuration can finish draining after the removal — receivers no longer reject it as a non-node - scenario tests over real chash rings (resharder/scenarios_test.go): TestScenario_AddNode — epoch bump adds a node; losers hand affected spaces off with 2-owner ACKs and delete their copies; unaffected spaces stay put. TestScenario_RemoveNode — removed node drains everything to the current owners (authenticated via config history) and ends up empty; every space remains available on all remaining owners - docs/resharding-operations.md: operator guide for the two supported operations (prerequisites, steps, metrics to watch, safety properties); linked from README Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
- node_resharder_state (0 disabled / 1 idle / 2 draining), node_resharder_epoch (config epoch acted on — shows propagation), node_resharder_last_cycle_unix, node_resharder_errors - adopter gets its own metrics: node_adopter_adopted / already_have_same / already_have_diverged / rejected (the archive-level 'adopted' gauge was registered but never incremented — removed) - operator guide updated with the full metric list and idle-state 'done' definition Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
…fore merge) Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
… client code) Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
…fig changes A config publish that only changes addresses or non-tree nodes produces an identical chash ring — no space changes hands. The change observer now compares tree member sets and skips the (guaranteed-empty) full index scan for such epochs; the periodic cycle remains as a backstop. Correctness was already given (zero candidates); this removes the wasted work. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
An Error entry needs an operator: the local db may hold data that failed to archive. Adopting a snapshot over it would let the later restore overwrite that data, and ACKing the sender could justify deleting the freshest copy. Error owners now answer 'diverged' — no adoption, no ACK; the two healthy owners carry the handoff. Operator guide gains an Error-state playbook. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Periodic per-node repair (default hourly) for spaces in Error status that the node is responsible for: - repair in place when the db opens fine (transient errors like a failed archive upload) — no data touched - otherwise quarantine the corrupted db under <root>/.quarantine (preserved for the operator) and pull a valid copy from a responsible neighbor via the regular coldsync (works without a shared bucket), validating it before the Error status clears - nodestorage.QuarantineSpace + IndexStorage.ReadSpacesByStatus - metrics: node_repairer_errored/repaired/repaired_in_place/quarantined - operator guide updated: manual playbook shrinks to the cases the repairer deliberately leaves alone Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
…anywhere Tracing the all-replicas-broken scenarios surfaced a bricking bug: an Error entry blocks space creation (checkStatus), so an orphaned entry of an uncommitted space push — no data on any owner, no heads ever recorded — would keep the space id unusable forever and alert forever. - coldsync already returns coded ErrSpaceMissing when the source definitively lacks the space; the repairer now distinguishes that from unreachable/broken owners - when every owner definitively lacks the space, we hold nothing and the entry has no recorded heads, the entry is deleted outright: the id becomes usable again and the alert clears (node_repairer_dropped_entries) - if heads WERE recorded, the entry stays Error: that is a real data-loss signal that must keep alerting Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Two real node stacks with an in-process drpc pair; node B holds a real GenStorage space, node A repairs through the actual protocol: - corrupted db: quarantined under .quarantine, valid copy pulled from B, status Ok, heads registered from the pulled state, storage opens as a valid space - missing db file: pulled from B without quarantine - space missing everywhere: B answers coded ErrSpaceMissing over the wire and the heads-less entry is garbage-collected Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Add nodespace/pubsubrelay, a component that drives the any-sync commonspace/pubsub engine as a relay: - Relay: IsResponsible / IsResponsibleNode via nodeconf, OtherResponsiblePeers dials the other responsible nodes (excluding self) through the pool. - MembershipChecker: authorizes subscribe/publish against the hosted space's ACL (Permissions at current head via nodespace.Service.GetSpace + Acl()). - Crypto/Peers left nil: the node relays ciphertext it cannot read and routes via Relay, not a client peer provider. - Registers the PubSub DRPC service and owns the engine's Init/Run/Close. - Evicts removed members: nodespace gains SetAclObserver, invoked from nodeSpace.AddConsensusRecords (after releasing the acl lock) so the relay re-checks subscribers via engine.RevalidateMembers on every ACL change. Registered in the node bootstrap after nodespace. go.mod/go.sum bumps come from building against the any-sync pubsub branch.
The ACL-change observer fires from the consensusclient stream reader while it holds the consensusclient mutex. The previous synchronous onAclUpdate called GetSpace (space ocache) inline, which deadlocks: a concurrent space close does UnWatch (same mutex), and GetSpace waits on the closing entry — ABBA — or, on an ocache miss, reloads on the same goroutine and re-enters Watch on the held mutex (self-deadlock). Either bricks consensus delivery node-wide. - onAclUpdate now only enqueues the spaceId (non-blocking, deduping); a relay-owned worker drains the queue and revalidates with a bounded timeout. - Use PickSpace (never loads) so a miss is a no-op and an evicted space is not resurrected; members are re-checked on the next subscribe/publish anyway. - Fire the observer even when AddRawRecords errors (records apply one by one, so a partial batch may have changed membership). - Guard SetAclObserver(nil) against a nil-func panic and unregister the observer in relay.Close before tearing down the worker and engine. Adds regression tests: onAclUpdate is non-blocking and never touches the space cache synchronously; a PickSpace miss is a no-op; the worker drains the queue.
Replace the local-path dev replace with a pseudo-version pinning the any-sync feat/pubsub commit (f2bca56) that ships commonspace/pubsub. Bump to the tagged any-sync release once it is cut.
Picks up the WithMetric fix (f05d5319) so the node's private pubsub pool no longer clobbers the app sync streampool's OutgoingMsg telemetry.
Resolutions:
- go.mod/go.sum: repin any-sync to the merged v0.13.x
(v0.13.0-alpha.5.0.20260710230326-2c00c2b7f186).
- Adapt to any-sync's diffsync-V2 removal, which dropped the distinct old
hash from statestorage (State.OldHash gone; OnHashChange/SetHash now carry
a single hash, with "oh" mirrored to "nh"). The node keeps its oldHash
plumbing (resharder/archive/nodehead/index/proto) and feeds newHash for
both at the any-sync boundary points:
- nodestorage/spacestorage.go OnHashChange(newHash)
- nodestorage/storageservice.go IndexSpace
- nodespace/migrator/migrator.go
Tests updated for statestorage.SetHash's single-hash signature and the
ringConf fake gains FileV2NodeIds/FileV2Peers.
# Conflicts: # go.mod # go.sum
# Conflicts: # go.mod # go.sum
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
Coverage provided by https://github.com/seriousben/go-patch-cover-action |
zarkone
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release branch for the any-sync v0.13.x train.
Contains:
mainmerged inv0.13.0-alpha.8Build, vet and full test suite green against the pinned module.
🤖 Generated with Claude Code
https://claude.ai/code/session_01AhQwGDBDiQfv2rJWLS31GH