Add additional cloud sync providers (rclone)

[wiverson]

Support vaults that already live in a consumer cloud — Dropbox, OneDrive and
similar — so someone can point a Railway-hosted NoteMesh at notes where they
already are, without an Obsidian Sync subscription and without git.

Summary

rclone is the right vehicle; the hard part is not rclone. One binary covers
~70 providers and the existing SyncBackend contract is already the right
shape, so a third implementation largely fills in a contract rather than
inventing one. The cost is concentrated entirely in provider auth models,
which vary enormously.

Recommendation: ship rclone as a bring-your-own-sync backend with tiered
setup, and do not support iCloud.

Providers split into three tiers by auth model

Tier	Providers	Setup cost
Key-based	S3 / R2 / B2, SFTP, WebDAV	Paste two keys. No browser, nothing expires. Fully self-service.
OAuth	Dropbox, OneDrive	Browser flow rclone assumes happens on a machine with a browser.
iCloud	—	Not viable, see below.

This is the whole design driver: "support rclone" and "support Dropbox/OneDrive"
are separable decisions.

iCloud: recommend against

rclone marks the iCloud Drive backend Tier 4 – Experimental, "Use with
care; expect gaps/changes." Beyond stability:

• It rejects app-specific passwords. The docs: "App-specific passwords are not
  accepted. Only use your regular Apple ID password and 2FA." That means
  storing a user's full Apple ID password on the server.
• The trust token it issues expires after 30 days, after which the user must
  re-authenticate interactively via rclone reconnect or rclone config.

Monthly hand-reauthentication on a headless box, in exchange for holding the
credential to someone's entire Apple account. No amount of UI design fixes
either half. Worth stating explicitly in the docs so it does not keep getting
re-proposed.

Headless OAuth is the real UX problem

rclone's own documented options for a machine with no browser are all poor fits
for a non-technical audience:

1. Install rclone on a second machine, run rclone authorize, paste the token.
2. SSH port-forward localhost:53682.
3. Configure locally and transfer the config file.

A fourth option exists: NoteMesh is already an OAuth server, so it could be an
OAuth client and redirect back to its own Railway domain. But the redirect URI
must be pre-registered, and a shared ChangeNode app cannot register every
self-hosted domain — so each user creates their own app: 8 steps for Dropbox,
5 for an Azure app registration. Viable later; not the MVP.

Both providers also ship a shared default client ID that is rate-limited.
Dropbox: "When you use rclone with Dropbox in its default configuration you are
using rclone's App ID. This is shared between all the rclone users."
OneDrive: "You may choose to create and use your own Client ID, in case the
default one does not work well for you. For example, you might see throttling."

Proposed setup: three ways in

Ship 1 and 2 together; 3 is documentation only.

1. A form, for key-based remotes. Provider dropdown → endpoint, key, secret,
   bucket/path; NoteMesh generates the rclone.conf stanza server-side. Nothing
   leaves the browser, nothing expires. This is the tier a non-technical user
   can finish unaided, and should be the default.
2. A textarea, for everything else — including Dropbox and OneDrive. The user
   runs rclone config once on their own machine, where the browser already is,
   and pastes the resulting rclone.conf. This is rclone's documented option 3,
   it covers all ~70 providers for free, and it is a copy-paste rather than a
   CLI session.
3. railway ssh as an escape hatch. Confirmed to exec inside the deployed
   container, so rclone config can be run against /data/rclone.conf
   directly. Docs only, no code.

Operational findings

bisync is stateful, not fire-and-forget. Its own docs open with: "bisync is
considered an advanced command, so use with care. Make sure you have read and
understood the entire manual (especially the Limitations section) before using,
or data loss can result." Concretely:

• It keeps listing files between runs; losing them requires --resync.
• Certain errors cause a lockout until someone intervenes. --resilient,
  --recover and --max-lock enable automatic recovery and should be on.
• --max-delete defaults to 50%; renaming a folder reads as mass deletion.
• The v1.66 snapshot redesign substantially reduced the risk from files changing
  mid-run.

This needs a new SyncState — a resync-required lockout is neither
backoff (retrying will not help) nor conflict — plus a Status callout and a
recovery button, or users are simply stuck.

Tokens cannot live only in our encrypted store. rclone's docs: "the
configuration file must be writable, because rclone needs to update the tokens
inside it." So rclone.conf on the volume is authoritative while rclone runs.
Plan: persist it encrypted in settings via the existing encryptSecret /
decryptSecret, materialise to /data/rclone.conf at 0600 on boot, watch the
file, and re-encrypt when rclone rewrites a refreshed token. This matters
because the Security tab claims credentials are encrypted at rest.

What this costs in the codebase

The abstraction holds; most of the work is additive.

Reuse directly:

• SyncBackend — src/server/sync/types.ts:66. Eight methods; health is folded
  into status().state rather than a separate method.
• LogRing / appendLogLine / logLinesOf — types.ts:102-131. The log
  surface the Status tab already consumes, including heartbeat collapsing.
• src/server/sync/git-exec.ts:31 — the model for rclone-exec.ts: secrets off
  argv, output redaction, no-prompt env, timeouts.
• GitBackend — src/server/sync/git.ts:82. Its start() → interval +
  debounced cycle(), busy serialisation, and backoff / needs-reauth
  latching from classified subprocess output transfer almost line for line.

New: src/server/sync/rclone.ts (~300–400 lines paralleling git.ts),
rclone-exec.ts, rclone-config.ts. Widen SyncKind (types.ts:5) — tsc
then names every site needing an update. Wizard work is a new stage in
setup.ts, a step component and card in setup.tsx, and RPC entries in both
allowlists plus src/lib/api.ts. Dockerfile gains one pinned binary beside
git/git-lfs.

Two genuine leaks in the abstraction:

• ConflictRecord.strategy is the literal union "file" | "branch" | "inline"
  (types.ts:58), hardcoded in the Status callout and Settings select. bisync's
  model (--conflict-resolve, ..conflict1 suffixes) has no branch
  analogue.
• Re-auth is Obsidian-shaped end to end. admin.reauth() calls obLogin
  directly and the needs-reauth UI renders Obsidian email/password/MFA fields.
  Git can already enter needs-reauth with no way out — a pre-existing gap a
  third backend makes acute. A generic "re-authenticate this backend" affordance
  is the one piece of real design work.

No shared conformance suite exists — backends are tested separately and
GitBackend's own state machine is untested. Adding a third implementation is
the natural moment to write one; it retro-covers git.

Suggested sequencing

1. rclone in the image, rclone-exec.ts, config storage and materialisation.
2. The backend class against a local/SFTP remote — prove the loop, states
   and logs with no auth complexity in the way.
3. SyncBackend conformance suite; make git and Obsidian pass it too.
4. The key-based form.
5. The paste-a-config textarea, which brings Dropbox and OneDrive in.
6. Docs: per-provider walkthroughs, the railway ssh hatch, why not iCloud.
7. Only then consider in-app OAuth with user-supplied client ID/secret.

Open questions

• bisync throughput on a large vault on Railway's smaller instances — worth a
  timing spike at step 2 before building UI on top.
• .obsidian/ churn through a general-purpose file sync; probably exclude it by
  default with an rclone filter.
• Is needs-resync safe to recover from a button, or should it show
  --resync --dry-run output as a confirmation first?

Note on the original link

dropbox/dbxcli is still actively maintained
(not archived; pushed recently), but it is a scriptable CLI for file operations
on a single provider, not a bidirectional sync engine. rclone supersedes it for
this purpose and covers Dropbox along with everything else.

Sources

• https://rclone.org/ · https://rclone.org/bisync/ · https://rclone.org/remote_setup/
• https://rclone.org/iclouddrive/ · https://rclone.org/dropbox/ · https://rclone.org/onedrive/
• https://rclone.org/rc/ · https://rclone.org/docs/

[wiverson]

https://rclone.org/