Skip to content

Fix transfer reliability, chunk safety, and bounded concurrency - #1

Open
daltschu22 wants to merge 2 commits into
masterfrom
fix/transfer-reliability
Open

daltschu22 wants to merge 2 commits into
masterfrom
fix/transfer-reliability

Conversation

@daltschu22

@daltschu22 daltschu22 commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Dsync could return success while transfers were still running or had failed, omit files during chunking, and write/delete testfile.dsync even during cloud dry runs. This change makes the command wait for bounded parallel transfers, report failures, and preserve filename and dry-run behavior across both backends.

  • Replace shell-built commands with argument lists and quoted SSH payloads; stream chunk lists over stdin so worker hosts do not need shared working files. Preserve remote path spelling instead of resolving it against controller symlinks.
  • Honor --rclone-config, remove cloud write probes, avoid creating local directories for remote destinations, and fix recursive rclone basic chunking.
  • Include hidden files; preserve special filenames, rsync symlinks, and empty directories. Use NUL-delimited rsync lists and raw rclone lists.
  • Replace incompatible fpart -Z usage, detect nonzero failures and traversal diagnostics even when fpart exits zero, and avoid unintended recursive re-copying of fpart chunks.
  • Stream basic chunk generation, skip empty jobs, enforce -n during reuse, and lock working directories.
  • Handle SIGINT and SIGTERM during partitioning and transfers. Register new processes before delivering pending cancellation, finish cleanup despite further signals, and kill surviving local process-group members after a five-second grace period even if their leader exited.
  • Stage chunk generation and validate reuse against source identity, mode, and checksums. Replace only verified, unchanged chunks listed in a valid manifest; preserve unrecognized files and refuse symlinked chunk directories, including during dry runs. Reject overlapping local source/destination/work paths.
  • Refresh documentation and add a Python 3.10/3.14 CI matrix.

Validation: python3 -m unittest discover -s tests -v — 39 tests passed, with no skips, on Python 3.14.4, rsync 3.4.1, fpart 1.7.0, and rclone 1.60.1-DEV. Tests include real local transfers with both chunking modes and both backends, a configured rclone alias, special filenames, dry runs, unreadable directories, failure propagation, ownership checks, reuse, concurrency, locking, SIGTERM during partitioning and transfers, termination-resistant descendants, cancellation during process registration/cleanup, remote path preservation, and worker execution through a local SSH shim. git diff --check and Python compilation also passed. Actual SSH hosts and external cloud services were not exercised; large-directory throughput has not been benchmarked.

Compatibility: regenerate legacy chunks once before using --reuse; reuse requires matching backend/chunking flags and does not discover source-tree changes. Modified/unrecognized working files are preserved and require review or a new working directory. For older rclone compatibility, newline/carriage-return filenames fail explicitly before cloud transfer. Host files currently support DNS/IPv4/SSH aliases, not IPv6 literals. SSH filesystem destinations must be absolute; overlap validation covers local destinations. Remote worker cleanup depends on SSH and the remote tool's disconnect behavior. Rclone retains its default symlink/empty-directory behavior. Fpart's nonverbose stderr is checked against its normal partition statistics; unexpected diagnostics stop the run rather than risk a partial copy being reported as successful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant