Repository navigation
Fix transfer reliability, chunk safety, and bounded concurrency - #1
Open
daltschu22 wants to merge 2 commits into
Open
daltschu22 wants to merge 2 commits into
daltschu22 wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dsync could return success while transfers were still running or had failed, omit files during chunking, and write/delete
testfile.dsynceven during cloud dry runs. This change makes the command wait for bounded parallel transfers, report failures, and preserve filename and dry-run behavior across both backends.--rclone-config, remove cloud write probes, avoid creating local directories for remote destinations, and fix recursive rclone basic chunking.-Zusage, detect nonzero failures and traversal diagnostics even when fpart exits zero, and avoid unintended recursive re-copying of fpart chunks.-nduring reuse, and lock working directories.Validation:
python3 -m unittest discover -s tests -v— 39 tests passed, with no skips, on Python 3.14.4, rsync 3.4.1, fpart 1.7.0, and rclone 1.60.1-DEV. Tests include real local transfers with both chunking modes and both backends, a configured rclone alias, special filenames, dry runs, unreadable directories, failure propagation, ownership checks, reuse, concurrency, locking, SIGTERM during partitioning and transfers, termination-resistant descendants, cancellation during process registration/cleanup, remote path preservation, and worker execution through a local SSH shim.git diff --checkand Python compilation also passed. Actual SSH hosts and external cloud services were not exercised; large-directory throughput has not been benchmarked.Compatibility: regenerate legacy chunks once before using
--reuse; reuse requires matching backend/chunking flags and does not discover source-tree changes. Modified/unrecognized working files are preserved and require review or a new working directory. For older rclone compatibility, newline/carriage-return filenames fail explicitly before cloud transfer. Host files currently support DNS/IPv4/SSH aliases, not IPv6 literals. SSH filesystem destinations must be absolute; overlap validation covers local destinations. Remote worker cleanup depends on SSH and the remote tool's disconnect behavior. Rclone retains its default symlink/empty-directory behavior. Fpart's nonverbose stderr is checked against its normal partition statistics; unexpected diagnostics stop the run rather than risk a partial copy being reported as successful.