Context
resume_stream_pass and hash_plaintext_streaming hash with a single-core hasher.update call on 64 KiB chunks, while cpu_stage uses update_rayon above RAYON_HASH_THRESHOLD (100 MiB) to parallelize hashing across cores.
Resuming an 88 GB file currently re-hashes on a single core during startup reconcile, which is slow and was part of the surface area of the 2026-08-14 OOM incident (fixed in #279).
The catch
Naively adding update_rayon per 64 KiB chunk would be placebo parallelism - rayon only pays off at slice sizes >= 128 KiB, well above the 64 KiB chunk size these paths currently use. So the real fix is not "swap update for update_rayon" but a chunk-size-aware rayon hashing strategy (e.g. batching/accumulating into larger slices before handing off to rayon) applied consistently across:
cpu_stage (already uses update_rayon, but should share the same chunking strategy)resume_stream_passhash_plaintext_streaming- scanner
hash_file
Ask
Introduce a shared chunk-size-aware rayon hashing helper and apply it consistently across all four call sites above, rather than hashing 64 KiB slices one at a time on a single core.
Reference
PR #279 (fix(core): stream the resumable-upload resume instead of buffering the whole file) - deferred hardening item from its multi-agent review.
Context
resume_stream_passandhash_plaintext_streaminghash with a single-corehasher.updatecall on 64 KiB chunks, whilecpu_stageusesupdate_rayonaboveRAYON_HASH_THRESHOLD(100 MiB) to parallelize hashing across cores.Resuming an 88 GB file currently re-hashes on a single core during startup reconcile, which is slow and was part of the surface area of the 2026-08-14 OOM incident (fixed in #279).
The catch
Naively adding
update_rayonper 64 KiB chunk would be placebo parallelism - rayon only pays off at slice sizes >= 128 KiB, well above the 64 KiB chunk size these paths currently use. So the real fix is not "swap update for update_rayon" but a chunk-size-aware rayon hashing strategy (e.g. batching/accumulating into larger slices before handing off to rayon) applied consistently across:cpu_stage(already usesupdate_rayon, but should share the same chunking strategy)resume_stream_passhash_plaintext_streaminghash_fileAsk
Introduce a shared chunk-size-aware rayon hashing helper and apply it consistently across all four call sites above, rather than hashing 64 KiB slices one at a time on a single core.
Reference
PR #279 (fix(core): stream the resumable-upload resume instead of buffering the whole file) - deferred hardening item from its multi-agent review.