Archive sharding: streaming hash, abandon, and size-ordered budget

[raphaeltm]

Summary

The SAM root ProjectData Durable Object sits at 10.256 GB against the 10 GB SQLite ceiling and is being reset for CPU and memory. The archive-sharding sweep that should drain it could not: its terminal-version hash loaded every row of a session with toArray() and hashed one string, so a 100,000-message session and a tool-heavy 9,906-message session both reset the object with Durable Object's isolate exceeded its memory limit. Three production sessions are now fenced (migrating/frozen) with failed/poisoned journals that copy-back cannot fix because it requires source_deleted.

This PR makes three changes:

1. Streamed terminal-version hash. computeTerminalVersion → tableAggregateSha256 pages PROJECT_DATA_ARCHIVE_HASH_PAGE_ROWS (default 500) rows per statement through createCanonicalRowsHasher (node:crypto incremental SHA-256), which is byte-identical to the one-shot sha256Hex(canonicalizeArchiveRows(...)), so every recorded proof stays valid. finalizeSourceDelete, rebuildTargetFts, and the new abandon path page grouped rows the same way, seeking on the indexed (created_at, id) order.
2. Superadmin abandon for pre-copy migrations. POST /api/admin/project-data/storage/:projectId/archive-sharding/migrations/:migrationId/abandon reserves the journal lease (CAS, epoch bump), removes the root source intent under the source transcript lock (refuses once the payload is deleted), only then drops the partial shard copy, freezes the journal as operator_abandoned without opening the project breaker, and returns the D1 location to root.
3. Size-ordered, budgeted sweep selection. Candidates are selected largest-first (session_summaries.message_count DESC) under a per-tick PROJECT_DATA_ARCHIVE_SWEEP_MESSAGE_BUDGET (default 20,000; a single oversize session is still selected alone). PROJECT_DATA_ARCHIVE_SWEEP_SESSIONS becomes the hard ceiling (default 1 → 10), and the sweep journals (fences) each candidate only immediately before processing it, so a tick that runs out of wall time never leaves unreached sessions fenced until the next cadence-gated sweep.

Rule 47 candidate-selection statement. selectCandidates / selectScopedCandidates changed only their ORDER BY (updated_at ASC → message_count DESC, updated_at ASC, id ASC) and gained the in-memory budget filter; the WHERE predicate is unchanged. Expected candidate volume per tick: at most sweepSessions (10) rows after the budget, from a global scan of terminal sessions past the 7-day grace (staging today: ~700 such sessions across projects; production SAM project alone holds 6.7M of 8.9M messages in terminal sessions past grace). Worst-case per-candidate cost is one migration (~4 ms/message observed on the production canary), bounded by the existing wall-time break, which now runs before each journal row is created. The scan itself is unindexed on ended_at/message_count (pre-existing; the global sweep runs once per PROJECT_DATA_ARCHIVE_GLOBAL_SWEEP_INTERVAL_MS, 24 h in production); an index is a follow-up, not a change in this PR.

Config: PROJECT_DATA_ARCHIVE_SWEEP_MESSAGE_BUDGET, PROJECT_DATA_ARCHIVE_HASH_PAGE_ROWS (new, plumbed through env.ts, DO types.ts, top-level wrangler.toml, sync-wrangler-config.ts, both wrangler_sync_env blocks, .env.example, env-reference skill, configuration docs). No D1 or DO migration: frozen and error_code already exist.

Validation

• pnpm lint
• pnpm typecheck
• pnpm test (api: 665 files / 8,984 tests; web: 302 files)
• Additional validation run: pnpm check:fast; apps/api Workers-pool suite tests/workers/project-data-archive-sharding.test.ts (real DO SQLite, 7/7 incl. a 1,001-message multi-page migration and abandon-then-reselect); mutation checks recorded below
• Candidate selection change: expected volume and worst-case per-candidate cost stated in the summary

Mutation checks (each guard deleted, exactly the intended test went red, then restored):

• one-shot toArray() hash restored → never materialises more than one page per statement fails expected 1205 to be less than or equal to 500
• abandon page size forced to 999,999,999 → keeps the grouped-row and FTS teardown paged fails
• state !== 'failed' lease conjunct deleted → does not let a stale lease on an already-failed migration block abandon fails
• target-first abandon ordering restored → never drops the shard copy when the source is finalized between the inspect read and the abandon fails
• lease reservation bypassed → the race test and fences the journal before touching any object fail
• eager candidate journaling restored → leaves sessions it never reached unfenced when wall time runs out mid-tick fails
• rowid paging: EXPLAIN QUERY PLAN test asserts the index range scan, with the rejected rowid shape as a per-page-sort control

Staging Verification (REQUIRED for all code changes — merge-blocking)

• Staging deployment green — Deploy Staging run 33987974789 for 870f73f3 succeeded (19:59Z); https://api.sammy.party/health healthy; staging D1 ledger current (no migration in this PR), 0 active / 0 fenced archive migrations before the feature pass
• Live app verified via Playwright — PENDING: SAM_PLAYWRIGHT_PRIMARY_USER is not set in this session; script .tmp/staging-archive-verify.mjs is prepared and runs as soon as the token is available
• Existing workflows confirmed working — PENDING (same blocker)
• New feature/fix verified on staging — PENDING (same blocker): plan is a non-dry manual canary on a 1,650-message session (>1 hash page) that must publish, then a second migration abandoned via the new route with the session reading again from root
• Infrastructure verification completed — N/A: no infra changes (Worker + DO code, no cloud-init/vm-agent/DNS/TLS)
• Mobile and desktop verification notes added for UI changes — N/A: no UI changes

Staging Verification Evidence

Deploy: run 33987974789 green for 870f73f3. Health: {"status":"healthy"} at 20:01Z. D1 (via Cloudflare API): d1_migrations head 0143_project_data_storage_relief_preflight.sql; project_data_archive_migrations 2 rows, none active; project_data_session_locations 0 migrating. Feature pass not yet run — blocked on the staging smoke token (rule 13: not substituting page-load checks). This section will be updated with canary/abandon evidence before merge; the PR must not merge until then.

UI Compliance Checklist (Required for UI changes)

N/A: no UI changes (apps/web, packages/ui, packages/terminal untouched).

UI Screenshot Evidence

N/A: no UI changes.

End-to-End Verification (Required for multi-component changes)

• Data flow traced from user input to final outcome with code path citations
• Capability test exercises the complete happy path across system boundaries (tests/workers/project-data-archive-sharding.test.ts: coordinator → root DO → shard DO → D1 through real RPC)
• All spec/doc assumptions about existing behavior verified against code
• Gaps documented below

Data Flow Trace

Abandon:

1. Operator POST .../migrations/:migrationId/abandon → apps/api/src/routes/admin/project-data-storage.ts (superadmin via adminRoutes.use in routes/admin.ts; jsonValidator(ProjectDataArchiveRecoveryControlSchema))
2. → abandonProjectDataArchiveMigration (apps/api/src/scheduled/project-data-archive-sharding.ts): journal read, project match, state/lease guards, reserveAbandonLease (D1 CAS)
3. → root DO archiveSourceInspectIntent (early refusal) then archiveSourceAbandonIntent (durable-objects/project-data/index.ts, under withArchiveTranscriptLock) → abandonArchiveSourceIntent (archive-sharding.ts) deletes the intent row, refuses source_deleted/rehome_exported
4. → shard DO archiveTargetAbandonSession → abandonArchiveTargetSession pages grouped rows for FTS delete markers, deletes messages/tool archives/chunks/target session/anchor
5. → D1 batch: journal → frozen/operator_abandoned (guarded by reserved lease_epoch), location → root
6. → next user read of the session resolves root via resolveExactReadOwner and succeeds

Sweep selection: cron runProjectDataArchiveSharding → selectMigrationWork (reclaimable + budgeted unjournaled pending) → processArchiveMigrationBatch (wall-time check → createCandidateJournal → migrateCandidate, one at a time) → prepareArchiveSourceIntent → computeTerminalVersion (paged tableAggregateSha256) → chunk export/commit → sealArchiveTarget (paged) → manifest → finalizeSourceDelete (paged) → publish.

Untested Gaps

• The DO memory ceiling itself is not enforced by any harness (better-sqlite3 or the Workers pool), so the bound is proven by the page-shape assertion (max rows materialised per statement ≤ page size) plus digest parity, not by observing a reset. Staging runs real multi-page sessions (852 and 1,650 messages) through the real DO.
• The abandon/finalize race is reproduced with a fake root DO whose abandon RPC refuses after the inspect read; a true concurrent finalize is not scheduled in-test.

Post-Mortem (Required for bug fix PRs)

What broke

Archive sharding could not drain the largest sessions from the SAM root ProjectData object: every session above one page of rows reset the object with a memory-limit error, and the failed attempts left three sessions unreadable (migrating/frozen) with no recovery path.

Root cause

tableAggregateSha256 (introduced with the archive-sharding bridge) selected every row of a session per table with no LIMIT, called toArray(), and canonicalised one string; memory scaled with session bytes. The 2026-09-04 bind-parameter fix paged the chunk export but not the hash.

Class of bug

Harness-ceiling divergence (rule 69): a platform limit (the isolate memory ceiling) that no test engine enforces, exercised only by fixtures far below it. Plus an unbounded per-session scan in a control loop (rule 47). The review round surfaced a second class in the fix itself: a cross-object check-then-act whose destructive step ran on the wrong side of the only serialization point (rules 45/58), fixed before merge.

Why it wasn't caught

All DO fixtures were ≤ 12 rows; the canary's nine "successes" were two-message sessions. The suite never measured how many rows a statement materialised.

Process fix included in this PR

.claude/rules/69-emergency-config-paths-need-their-own-coverage.md gains "Memory Is A Ceiling Too, And It Scales With The Row You Did Not Page": unbounded SELECT ... WHERE <scope> = ? over user-growing tables must page, and the discriminating test asserts the page shape and digest parity because no harness can observe the reset.

Post-mortem file

tasks/archive/2026-09-05-archive-sharding-streaming-hash-abandon-and-size-budget.md

Specialist Review Evidence (Required for agent-authored PRs)

• All local reviewers completed and findings addressed before merge
• If any reviewer did NOT complete: needs-human-review label added and merge deferred to human — not applicable, all nine completed

Reviewer	Status	Outcome
task-completion-validator	ADDRESSED	PASS on all six checks; LOWs fixed: abandon_reason_required → 400 + route test (808fde8), resolver clamp tests (a8e4391)
cloudflare-specialist	ADDRESSED	CRITICAL abandon/finalize race: lease reservation + source-first ordering + release on failure + epoch-guarded freeze, two mutation-verified tests (870f73f); MEDIUM DO-invariant→500 pre-existing copy-back pattern, now a rare race path; LOW lone-surrogate encoding unchanged (both encoders emit U+FFFD)
security-auditor	ADDRESSED	HIGH same race, fixed as above; MEDIUM dead published target guard documented in code; MEDIUM real-middleware auth test deferred (repo-wide mock pattern; mount order verified by reading routes/admin.ts); LOW blank reason fixed (808fde8); LOW project_id on shard deletes not applicable (tables carry no project_id, shard DO is per-project, identity checked by validateTargetOwner)
test-engineer	ADDRESSED	HIGH abandon route untested → route test (808fde8); MEDIUM wide-shard abandon paging + state !== 'failed' conjunct tests, both mutation-verified (412e471); LOW tie-break determinism + blank-reason coordinator tests (412e471)
constitution-validator	ADDRESSED	PASS on Principle XI; MEDIUM clamp/fallback coverage for resolveArchiveHashPageRows added (a8e4391)
env-validator	PASS	Both new vars typed, top-level only, in sync-wrangler-config, both workflow env blocks, and all three docs; quality:wrangler-bindings and the deploy-workflow env-parity test pass; LOW root .env.example drift pre-existing
doc-sync-validator	PASS	Every doc claim matches code and cited tests; no stale "one session per pass"/"no thaw" wording remains
architecture-reviewer	ADDRESSED	PASS; doc comments added for DO-owned hashPageRows, single sanctioned sourceIntactVerified caller, second consumer of ACTIVE_RECLAIMABLE_STATES, and the rowid-vs-cursor paging reason (412e471); file split of the two >2,500-line modules deferred as follow-up
performance-reviewer	ADDRESSED	HIGH fence-before-process → lazy per-candidate journaling, mutation-verified test; HIGH rowid per-page sort → indexed (created_at, id) seek with EXPLAIN QUERY PLAN test (870f73f); MEDIUM session_summaries index deferred, candidate volume stated above; LOW redundant ensureProjectId pre-flight pre-existing

CodeRabbit Review Evidence (Required for agent-authored PRs)

• coderabbit-review label applied after local review, staging if applicable, and CI gates passed
• All CodeRabbit findings implemented or explicitly reviewed and closed/resolved
• Incremental CodeRabbit review completed after final pushed fixes, or no fixes were needed
• Latest CodeRabbit review has no unresolved feedback

CodeRabbit Notes

Not yet requested: the coderabbit-review label is applied only after CI and the staging feature pass are green (Phase 7 gate). CodeRabbit's auto-review is label-gated in this repo (see its skip notice on this PR).

Exceptions (If any)

• Scope: none
• Rationale: —
• Expiration: —

Agent Preflight (Required)

• Preflight completed before code changes

Classification

• external-api-change
• cross-component-change
• business-logic-change
• public-surface-change
• docs-sync-change
• security-sensitive-change
• ui-change
• infra-change

External References

Cloudflare Durable Object SQLite limits (10 GB per object, ~128 MB isolate memory) and nodejs_compat node:crypto availability, checked via the Cloudflare docs MCP; production D1 journal/location rows and the two failing canary results (Durable Object's isolate exceeded its memory limit and was reset) queried via the Cloudflare API before coding.

Codebase Impact Analysis

apps/api/src/project-data-archive/hashing.ts (incremental hasher), apps/api/src/project-data-archive/contract.ts (defaults), apps/api/src/durable-objects/project-data/archive-sharding.ts (paged hash, paged grouped loops, abandon primitives), apps/api/src/durable-objects/project-data/index.ts (RPC wrappers, env-resolved page size), apps/api/src/scheduled/project-data-archive-sharding.ts (size-ordered budgeted selection, lazy journaling, abandon coordinator with lease reservation), apps/api/src/routes/admin/project-data-storage.ts (abandon route), env plumbing (env.ts, DO types.ts, wrangler.toml, scripts/deploy/sync-wrangler-config.ts, .github/workflows/deploy-reusable.yml).

Documentation & Specs

apps/www/src/content/docs/docs/reference/configuration.md, apps/api/.env.example, .claude/skills/env-reference/SKILL.md, .claude/skills/api-reference/SKILL.md, CLAUDE.md (Recent Changes), .claude/rules/69-emergency-config-paths-need-their-own-coverage.md.

Constitution & Risk Check

Principle XI: every new limit has a DEFAULT_* constant, an env override, and a clamp (MAX_*). Data-safety risk: abandon is destructive on the shard side; it is gated on a lease reservation, the lock-protected source abandon running first, refusal once the source is deleted, and an epoch-guarded D1 freeze; the breaker is untouched. Rule 31: no schema changes. Tradeoff: the sourceIntactVerified flag remains a caller assertion (single sanctioned caller documented) because the shard cannot see the root object.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BZCe9xphZ9LWBK64sz5PWa

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• coderabbit-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: adffc03e-eb3f-4264-b906-2b25c25d304d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.1% Duplication on New Code

See analysis details on SonarQube Cloud