[Madraka]

RFC-0045 rollout step 3b: publishes purge the pages that show references (§2 "Purges", §9 step 3, watch-list item 2). It follows 3a (#211). The types stay refused on every production write path; the tests enable them with SiteGraphStoreOptions.fieldTypes.

First commit: a 3a verification leftover

packages/cli/test/reference-links.test.ts builds the real starter with cloudhal build, under prefix-except-default and under prefix-all. The pages go through the real EntryFields.astro and the real locale middleware. In both routings:

• a Turkish referrer links /tr/places/salon exactly once;
• a fallback card links the default locale's member;
• the default locale's own page is right too.

Putting href={c.url} back into EntryFields.astro makes both builds fail with /tr/tr/places/salon.

The closure (core-ts/src/references/purge.ts)

• Seeds: the groups of every changed, added or removed entry, before and after, and of every entry the asset rule reached. planAssetPurge now returns them.
• Two hops: the referrers of the seeds, then the referrers of their groups. Each one purges its node tag and every member of its group's.
• "Used by": every member of each group a changed or reached entry names, before or after.
• Resolved values: resolveFields runs with the snapshot's same-site original and without it, so a translation that only inherits counts.
• Reads: only the immutable from/to bodies, never the working copy or entry_refs.
  • One hash-keyed cache: the planner's own.
  • One bundle, to's, else from's. It is streamed through the new readSnapshotBundle options: keep cuts each body to what the closure needs as its shard is parsed, and maxBytes/maxNodes are checked before each parse (BundleBudgetError).
  • Then only the missing hashes, through the counted reader. Shard reads count against maxReads.
• Fallbacks: past any budget (bytes 32 MB, nodes 50,000, maxReads, maxTags) the publish purges everything. So does any field-operation marker on either site node.
• Short-circuit: sites without reference definitions skip all of it; the test asserts no bundle is read.
• No new tags. SnapshotReader.readSnapshotBundle is the one reader addition. The store's method takes the options and stays at its baselined line count.
• Size: planCachePurge goes from 207 lines to 184 (diffNodes, countedReader, ancestry).

Releases (core-ts/src/references/release-purge.ts)

• After a committed publishRelease, the executor plans from → snapshotId and sends that, rather than the plan stored at pin time. from is the pointer the batch moved.
• The stored plan still fills the outcome the batch writes, and a differing result is corrected through recordPurge, as deferred purges already were.
• RELEASE_EXECUTOR_COST goes from 16 to 18. The comment says why: two getSnapshot reads, while bundle shards and node bodies are blob reads, bounded by maxReads.

Tests: core-ts/test/references-purge.test.ts, every case of the Test plan's Renderer bullet

Each case asserts the existing cloudhal-node-<id> tags only.

• A target retitled, unpublished, newly published, newly translated, or with a changed cover.
• Depth 2: A → B → C with C unpublished purges A; a fourth hop doesn't.
• A "used by" page when a referrer appears or goes, a translation that only inherits included, and when a referrer is edited away.
• A rollback whose referrer was since edited to draft.
• An approved request publishing an older snapshot, after the referrer was edited, then the target unpublished: the referrer's page is purged.
• A referrer write between createSnapshot and publish. A referrer edited to stop naming the target, or deleted, between snapshot and purge, is still purged from to's bodies.
• A translation in to that only inherits, deleted since.
• A snapshot without a bundle (a release run's scheduled snapshot, or an approval's manual one): two bundle attempts, node reads only for the diff, maxReads 20, no purge-everything.
• A site without reference definitions: no bundle read.
• The byte, node, tag and read budgets fall back to purge-everything.
• Any field-operation marker on from or to purges everything: on the marker's own publish, on both snapshots, and on from only.
• The release case is in the shared release suite (SQLite and D1): a referrer published after the pin is purged by the re-plan.

Mutation checks:

• disabling the closure fails 9 of the 12 purge tests;
• sending the stored plan instead of the re-plan fails the release case on both backends.

No new SQL: the closure reads blobs, plus the existing getSnapshot.

Deviations (also in the RFC's "Amended in 3b")

• Seed groups' own members are purged. A translation inheriting a changed original's reference is neither a referrer nor changed.
• The marker rule runs after the short-circuit, so sites without references keep today's purges. A marker's own publish changes the site node, which already purges the site.
• maxNodes counts the entry bodies held across both snapshots, plus the bundle's node count.

Checks

• Lint, size ratchet and typecheck: pass (20/20).
• Full pnpm check: it failed on one Studio UI test (chrome-settings, 11 s) under load, and turbo stopped after it.
• Rest of the tests: every other package passes with --continue (17/17); Studio passes alone (83 files, 1,339 tests).
• This PR's files, run one at a time: references-purge (12), cache-purge (12), bundles (4), the release suite (30), Cloudflare purge, purge-site and scheduler, and reference-links (both builds).

Review fixes (1cb2520a)

• A. No seeds, no reads. A publish that changes no entry and no shown asset returns before any read, so a term-only publish reads no bundle. Tested.

• B. The closure's own reads. Missing hashes go through readSnapshotBundle({ only }):

  • six at a time;
  • counted against maxReads;
  • added to the same byte total as the bundle (spent) before each parse;
  • kept as Slim, never in the planner's cache of full bodies.

  A reader that can't stream bodies purges everything. Tested: no bundles and a small maxBodyBytes.

• C. A failed re-plan never drops the purge.

  • The re-plan runs over the tick's counting driver (ReleaseExecutorOptions.counted, which the tick passes; releases.ts stays at its baseline).
  • If it throws, the run sends { mode: "everything" } through the purger.
  • RELEASE_EXECUTOR_COST is 26: the batch up to 8, refreshSearch 7, the bundle and re-plan reads 3, and bookkeeping 4. The site-moved retry stays outside it, as before; it fits Workers Free (40).
  • Tested on SQLite and D1: snapshot reads fail after the batch, and the run purges everything, recorded as everything.

• D. Back-to-back releases. Each run writes its scheduled snapshot's bundle first, best effort, for snapshots of at most 1,000 nodes. A snapshot left without one is passed to the planner as unbundled, so no shard reads are spent on it. Tested on SQLite and D1 (two consecutive releases, each snapshot bundled, both plans tags), plus a planner test for unbundled.

• E. The pin-time plan skips the closure (references: false). It is documented as the expected outcome only.

• F. A field still converting from references counts as one (migration.fromType: "reference", on the field or via sub). Tested by writing the reference → text state directly.

• G. The closure itself hits maxReads. The test measures the planner's own reads first, with the closure off, and asserts purge-everything at that count plus 5.

• Mutation checks: undoing each of A, B, C, D, F and G fails exactly its own test.

• The RFC's "Amended in 3b" is rewritten, including the corrected marker and conversion rule.

Local checks for this push:

• Lint, the size ratchet and typecheck (20/20) pass.
• Full pnpm check failed on load and environment, not on this change.
  • mcp-server's render_page test timed out under load.
  • The --continue run then hit core-ts and Studio timeouts.
  • Runs after that hit EADDRNOTAVAIL: the machine ran out of ephemeral ports during Miniflare's D1 tests (TIME_WAIT peaked at about 16,200 of 16,384). Another project's workerd processes were running at the same time.
  • origin/main's store.test.ts fails the same way on this machine (120 failures, the same peak).
• Run separately, all green:
  • core-ts without store.test.ts and list-values-query.test.ts: 64 files, 740 tests;
  • those two files' SQLite variants;
  • the release suite on SQLite and D1 (34);
  • the live-reads suite on all three backends;
  • references-purge (17), cache-purge and bundles;
  • Cloudflare purge, purge-site and scheduler;
  • Studio alone (1,339) and mcp-server alone (183).
• Not run locally: core-ts's Miniflare-variant suites beyond the release and reads cases. CI is the clean run for those.

Verification round 2 (b4b19b0b)

1. Release runs and subrequests.
   • a. Sites without references behave as before 3b. The run sends the stored pin-time plan, or nothing when it purges nothing, and makes no extra read. The decision uses the site row publishRelease returns, which is to's site node. Since from's site node isn't read, the "two site-node reads" aren't needed. A site whose last reference field was removed since from gets the stored plan; that edge is in the amendment.
     • Test: a tick with a purger makes exactly as many blob gets as the same tick without one, and sends the stored plan unchanged.
   • b. The bundle write comes after the purge, only for reference sites and only when from has a bundle.
     • It is built from from's bodies plus at most 100 changed nodes, for snapshots of at most 1,000 nodes.
     • When the re-plan found from bundle-less, the write is skipped, so from's shards are tried only once.
     • The re-plan never reads to's shards.
   • c. A per-tick blob allowance, shared by every run in the tick: 400 on Free, 4,000 on Paid (releaseBlobAllowance, passed by runScheduledTick). A re-plan reads at most 250 blobs and a bundle write at most 108. Past the allowance the run purges everything. The numbers sit next to RELEASE_EXECUTOR_COST.
   • d. Tests count blob gets through a wrapped BlobStore, on a reference site:
     • from bundled: to's shards are never read, from's are read once by the re-plan and once by the bundle write, and the referrer tag is present;
     • from bundle-less: from's shards are tried exactly once, to's never; the plan has the referrer tag or purges everything; and no bundle is written.
   • Mutation checks: removing the non-reference path fails test (a), and removing the bundle-less skip fails test (d).
2. RELEASE_EXECUTOR_COST stays at 26, and that is correct. It now counts recordPurge. The bundle write's own getSnapshot is gone (it reuses the re-plan's two reads), so the sum is 1 + 3 + 8 + 7 + 2 + 4 + 1 = 26, written out in the comment.
3. A discarded partial bundle gives its bytes back. Test: a mismatching to shard that fits the budget on its own but not together with from's.
4. The G test pins the shard spend. It uses a bundled from and a manual to: tags at own + shards (with no extra node reads), purge-everything at own + shards − 1. Deleting spend(bundleShardCount) fails it.
5. Changeset wording is now exact for publishes and release runs on sites without references.
6. The no-seeds return now comes before the site-node reads.

Local checks for this push:

• Lint, the size ratchet and typecheck (20/20) pass.
• Full pnpm check: 15 packages passed. Cloudflare's playground-workerd "20 sessions at once" failed under load and passes alone (4/4). Turbo stopped before core-ts, api and cli finished.
• Run alone:
  • api: 641 tests;
  • cli: 76 files, 568 tests;
  • core-ts: its full run hit the port exhaustion again (TIME_WAIT peak 16,358), as main does on this machine. Split up, it passes: 741 tests without the two Miniflare-heavy files; both files' SQLite variants; the release suite on SQLite and D1 (38); the live reads on all three backends.
• This PR's files, one at a time: references-purge (18), cache-purge, bundles, and Cloudflare purge, purge-site and scheduler.

Verification round 3 (740e2e9c): removing machinery

• A. The gate is the published site node. publishRelease returns its ref (publishedSite), with its 212-line baseline kept.

  • The round's assumption that the working row can equal the published node doesn't hold: a snapshot's site node is captured before the publish flips the pointer, so publishedSnapshotId and rev always differ.
  • So the gate reads the published node once per release run with a purger: one charged blob read. A site without references then sends the stored plan exactly as before. The test asserts the stored plan is sent, and exactly one blob get more than a purger-less run.
  • Tests: a draft that drops the last reference field still re-plans and purges the referrer; a draft that adds a first one sends the stored plan.
  • Mutation check: gating on the working row fails both, on SQLite and D1.

• B. The run-time bundle write is deleted (writeBundle, its cost, its caps). Back-to-back releases with no visitor re-plan node by node within the allowance, or purge everything. The test asserts two plans (tags or everything, with the referrer when tags) and the allowance never below 0.

• C. One allowance per invocation, charged by actual calls.

  • A charging BlobStore wraps the planner's blobs; nothing is reserved up front. Each re-plan gets maxReads = min(2,000, left), and RELEASE_REPLAN_MAX_READS is gone.
  • The cron handler shares one allowance between its two ticks (cronTick, keyed by the invocation's controller; createAdminWorker stays at its baseline). The alarm and the backstop run one tick each, with the default.
  • The arithmetic (Free 350, Paid 3,000) is written once in references/release-purge.ts; the RFC, the changeset and RELEASE_EXECUTOR_COST's comment refer to it.
  • On Paid, 50 releases of 100 items pass the default cap with refreshSearch alone, re-plan or not; that is pre-existing, and noted.

• D. Tests, on SQLite and D1:

  • back-to-back releases;
  • both gate directions;
  • 13 reference-site releases in one Paid tick, all tags;
  • a 260-term reference site on Paid, tags;
  • two ticks sharing the allowance, then purge-everything at 0 with nothing read.

  Plus a Cloudflare unit test for cronTick.

• RELEASE_EXECUTOR_COST stays at 26 D1 statements: 1 + 3 + 8 + 7 + 2 + 4 + 1, recordPurge included. labelOf's read is a blob call, not a statement. If you want 27 as a margin, it's a one-line change.

pnpm check: green in one full run. Typecheck 20/20, 19/19 test tasks (core-ts 1,393, cli 568, Studio 1,339), exit 0.

Verification round 4 (01a2ce71, narrow)

1. The gate read is counted but never refused. If it fails, the working row decides: no references, the stored plan; otherwise everything. Tests on a site without references: with left = 0, and with a gate read that throws, it still sends the stored plan. Re-adding the refusal fails the test.
2. A hopeless re-plan stops before reading. When the two snapshots hold at least as many distinct terms as there are reads left, the run purges everything; the gate read is the only call it made. Removing the pre-check fails the test.
3. The tick counts the release graph's other blob calls. It wraps the release graph's store with chargedBlobs (count only), so refreshSearch, labelOf and a first publish's head and put subtract from left. A store that is already charged is not wrapped twice.
4. "Publish now" sizes its allowance from the deployment's tick budget: ReleaseServiceOptions.tickBudget and SchedulesDeps.tickBudget, set by the admin Worker through publishNowPurge. ReleaseServiceOptions and the request types moved to release-service-options.ts, re-exported unchanged, so releases.ts stays at its baseline.
5. Docs:
   • labelOf keeps its JSDoc.
   • "except on a first publish".
   • Paid fits about 27 releases by D1 (33 statements each).
   • "this invocation".
   • Accepted over-purges are listed: the asset rule on Free above about 350 nodes, and the uncounted AI Search mirror.

pnpm check: green in one full run (typecheck 20/20, test tasks 19/19; core-ts 1,395, cli 568).

🤖 Generated with Claude Code