Skip to content

perf(upload): spread a batch over upload_workers CLI processes - #10

Merged
jshvn merged 1 commit into
josh/checkpoint-dietfrom
josh/upload-workers
Sep 15, 2026
Merged

jshvn merged 1 commit into
josh/checkpoint-dietfrom
josh/upload-workers

Conversation

@jshvn

@jshvn jshvn commented Sep 15, 2026

Copy link
Copy Markdown
Member

Stacked on #9 (shares the mutation retry path). Retarget to main once #9 merges.

Why

Upload is 66% of every run: one proton-drive filesystem upload per batch, measured at 0.208 s per file plus 12.8 MB/s, on a link that does two orders of magnitude more. The CLI has no concurrency flag, so the only lever is several processes.

Change

  • proton.upload_workers (default 1, so nothing changes until it is raised) and the run flag UPLOAD_WORKERS=n after the double dash or in the workflow's vars input, for one run at a time. The experiment is a dispatch with UPLOAD_WORKERS=2, then 4, reading upload_seconds from batches.details_json.
  • upload spreads the batch's top-level staging entries over at most that many groups, heaviest first into the lightest group (bytes plus 2.7 MB per file, the measured per-file cost). Disjoint top-level entries share no folder to create, so -d merge never races. Each call writes its own report, upload-<batch>-<k>.json, whose first line names the entries it was handed.
  • upload_trees on the provider runs the groups at once, each process from its own copy of the session exactly as the reconcile walk's workers do (WorkerSessions). Worker threads only run the CLI; every attempt is recorded and logged afterwards on the calling thread, which is the one that owns the state. A group the CLI signed out because its token refresh lost the race is re-run once on the copy that won, after that copy is adopted and every copy re-seeded; the session is written back once after adoption. Any group that still fails raises, and the batch fails as today.
  • confirm holds each call's summary against the files and folders under the entries that call was handed: transferred plus skipped plus failed must equal that call's files plus folders, every failure must name a file among them, and the batch confirms only when every call does and every verified file was covered by some call. A failure now marks twins within its own call rather than across the batch, which is strictly narrower. With one worker this is the accounting that ran before, over the whole batch.

Test plan

  • task test: 164 passed. New: two workers split a batch and a failure in one call marks only that call's file; a report that balances but does not cover every file records nothing; two groups run at once and a signed-out group is rescued on the adopted session, with the session written back once; a group that fails for good raises
  • task lint, task check
  • After merge: dispatch sync with vars: UPLOAD_WORKERS=2, compare upload_seconds per batch against the fitted 0.208 * files + bytes / 12.8e6; then 4. If it scales, set upload_workers in config/mirror.toml. Watch for "adopted the refreshed Proton session" warnings and for any duplicate folders in the next reconcile

@jshvn
jshvn merged commit 6fa63aa into josh/checkpoint-diet Sep 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant