feat(core-ts): purge referrers from the snapshots' bodies (RFC-0045 step 3b)

.changeset/rfc45-step3b-purges.md

@@ -0,0 +1,34 @@
+---
+"@cloudhal/core-ts": patch
+"@cloudhal/cloudflare": patch
+"@cloudhal/mcp-server": patch
+---
+
+RFC-0045 rollout step 3b: publishes purge the pages that show references. Nothing changes for sites without reference definitions. Their publish purge plans are the same, with no bundle read. Their release runs send the plan stored at pin time, as before, with no re-plan; the only addition is one read of the published site node, to tell them apart.
+
+- **The referrer closure** (`references/purge.ts`, called by `planCachePurge`).
+  - The seeds are the groups of every changed, added or removed entry, and of every entry whose shown asset changed.
+  - It purges the referrers of those groups, two hops deep, with every language version of each referrer.
+  - It also purges every member of each group a changed entry names, before or after ("used by").
+  - Values are resolved, so a translation that only inherits counts.
+  - It reads only the `from` and `to` snapshots' bodies, never the working copy or `entry_refs`.
+- **Reads and budgets.** A publish that changes no entry and no shown asset reads nothing. Otherwise the closure keeps one hash-keyed cache:
+  - one bundle (`to`'s, else `from`'s), streamed and cut down as each shard is parsed;
+  - then the missing hashes, read by the closure itself, six at a time, and cut down the same way.
+
+  A passed budget purges everything:
+  - bytes (`maxBodyBytes`, 32 MB);
+  - nodes (`maxNodes`, 50,000);
+  - reads (`maxReads`, shard reads included);
+  - tags (`maxTags`).
+
+  So does any field-operation marker on either snapshot's site node, for sites that hold references or convert from them.
+- **`readSnapshotBundle`** (`SnapshotReader.readSnapshotBundle`, `SiteGraphStore.readSnapshotBundle`, `readSnapshotBundle` in bundles) takes optional `maxBytes`, `maxNodes`, `keep`, `only` (bodies node by node) and `spent` (a shared byte total). A passed budget throws `BundleBudgetError` before the shard is parsed. No new cache tags.
+- **Release runs on sites with references re-plan their purge at run time** (`references/release-purge.ts`), from the pointer they moved to the snapshot they made, instead of sending the plan stored at pin time.
+  - The test is the published site node, which `publishRelease` now returns (`publishedSite`). That costs one blob read per run with a purger, except on a first publish; the read is counted, never refused. If it fails, the working row decides.
+  - The re-plan runs over the tick's counting driver.
+  - The release graph's blob calls (`refreshSearch`, `labelOf`, the re-plan) are charged to one allowance per invocation, shared by the cron's ticks (`ScheduledTickOptions.releaseBlobs`): 350 on Workers Free, 3,000 on Paid. "Publish now" sizes it from the deployment's tick budget (`ReleaseService` and `SchedulesDeps.tickBudget`).
+  - Past the allowance, or if the re-plan fails, the run purges everything. A re-plan with more distinct terms than reads left purges everything before reading anything.
+  - The pin-time plan skips the closure (`CachePurgeOptions.references: false`): for these sites it is the expected outcome only.
+  - `RELEASE_EXECUTOR_COST` is 26 (it counts `refreshSearch` and `recordPurge` now).
+- **`@cloudhal/cloudflare`:** the cron handler passes one `releaseBlobs` allowance to both ticks of its invocation (`cronTick`); the alarm and the request backstop run one tick each, with the default allowance.

docs/rfcs/0045-collections-stage-4-references-and-repeaters.md

@@ -1210,3 +1210,39 @@ Rollout step 2 shipped in two PRs: 2a, store and integrity, and 2b, the consumer
 
 - **`ReferencedEntry.path`** (§4): the member's graph path (`entryPath`, no locale prefix). Themes link with it, so the locale middleware localizes it once, like every other entry link. `url` stays the canonical URL, already localized, and absolute only on a locale's own domain.
 - **`referencedBy` follows each page** (§4): a translation resolves with its same-site original while the view shows that original, and without it otherwise, as its page reads it through the gated `entryById`. Ties on date order by id, so both sources agree on order and on what `limit` keeps.
+
+### Amended in 3b (2026-11-12, after the PR #212 review)
+
+- **Seeds purge their own groups' members** (§2 Purges, point 1). A translation inheriting a changed original's reference shows a new card, but it is neither a referrer of the seed nor changed itself.
+- **No seeds, no reads.** A publish that changes no entry and no shown asset (term-only, asset-only, site-only) reads nothing for the closure, so the budgets cannot turn it into purge-everything.
+- **Which sites count.** A site counts as having references when its `from` or `to` site node defines a reference field, a repeater with a reference sub-field, or a field (or `sub`) still converting *from* references (`migration.fromType: "reference"`). During `change_field_type` reference → text, the converted values render their targets' titles through the conversion maps until phase B rewrites them. On such sites any field-operation marker purges everything; other sites keep today's purge.
+- **The closure's own reads.** Bodies the planner already holds are reused. The missing hashes are read by the closure itself (`readSnapshotBundle` with `only`):
+  - six at a time;
+  - counted against `maxReads`;
+  - added to the same byte total as the bundle before each parse;
+  - cut to what the closure needs, and never cached whole.
+- **The node budget** (`maxNodes`) counts the entry bodies the closure holds across both snapshots, plus the node count of the one bundle it reads.
+- **Release runs.**
+  - **The gate is the published site node,** not the working row. A release never carries the site node, so its snapshot publishes the live one's; `publishRelease` returns that ref. Only a node that defines references, or still converts from them, re-plans. Every other site sends the plan stored at pin time, as before 3b (nothing when it purges nothing).
+    - That gate costs one blob read per release run that has a purger, except on a first publish: a snapshot's site node is captured before the publish flips the pointer, so the working row equals it only on a site's first publish.
+    - The read is counted but never refused, so a site without references never depends on what other runs left. If it fails, the working row decides: no references, the stored plan; otherwise purge everything.
+  - **The re-plan** goes from the pointer the batch moved (`from`) to the new snapshot. That is `baseSnapshotId` whenever something was live; a first publish purges everything.
+    - It runs over the tick's counting driver, never refused, and spends no shard reads on `to`.
+    - If it throws, the run purges everything, never the stored plan.
+  - **No bundle write at run time.** The site Worker writes the new snapshot's bundle on its first read. Back-to-back releases of one site with no visitor between them therefore re-plan node by node, within the allowance, or purge everything: an over-purge, never a stale page.
+  - **Subrequests.** R2 and D1 calls are capped per invocation: 1,000 on Workers Free, by default 10,000 on Paid.
+    - Release runs charge their *actual* blob calls to one allowance per invocation, shared by both ticks of the cron's invocation. That includes `refreshSearch`, `labelOf` and a first publish's own blobs, counted but never refused. Each re-plan gets `maxReads = min(2,000, left)`.
+    - A re-plan whose two snapshots hold more distinct terms than reads left purges everything before reading anything.
+    - "Publish now" sizes its allowance from the deployment's tick budget (`CLOUDHAL_TICK_BUDGET`).
+    - Free: 1,000 − 2 × (40 + 7) D1 − 2 × 40 built-in blobs − 2 × (211 `refreshSearch` + 2 + 4 purge sends) = 392, so 350.
+    - Paid: about 27 releases fit a tick's 900 statements (33 each with claim and finish). The allowance is sized for ticks of up to 10 releases of 100 items, 3,000. Twenty-seven such releases pass the default cap with `refreshSearch` alone, re-plan or not; that is RFC-0008's budget, unchanged here.
+    - **Accepted over-purges, not fixed:**
+      - On Free, a reference site with more terms than the allowance leaves purges everything on every release, because the planner reads every term. No cache priming is built for it.
+      - On Free, a reference-site release that changes an existing asset reads every entry, page and term one by one in the asset rule, so above about 350 nodes it purges everything.
+      - The AI Search mirror's R2 calls (RFC-0029) are not counted.
+  - **`RELEASE_EXECUTOR_COST` is 26 statements:**
+    - the stored plan (1), the reads (3), the batch (8) and `refreshSearch` (7);
+    - then the re-plan's two `getSnapshot` reads, the bookkeeping (4) and `recordPurge` (1).
+
+    The rare site-moved retry stays outside it, as before.
+- **The pin-time plan skips the closure** (`references: false`). It would read every body one by one, and for sites with references it is the batch's *expected* outcome only: the run-time re-plan, or purge-everything, covers referrers.

packages/cli/test/reference-links.test.ts

@@ -0,0 +1,152 @@
+/**
+ * RFC-0045 §4 through the real pipeline: the starter's `EntryFields.astro` draws a reference's
+ * card by its graph path, and the locale middleware (`localizeLinks`) gives it the page's locale
+ * exactly once. Built with `cloudhal build` under `prefix-except-default` and `prefix-all`: a
+ * Turkish referrer links its venue's Turkish member at `/tr/places/…` (never `/tr/tr/…`), and a
+ * fallback card (`default-with-notice`) links the default locale's member. The types stay refused
+ * in production; the test enables them on its own store, as the shared suites do.
+ */
+import { spawn } from "node:child_process";
+import { mkdtemp, readFile, rm, symlink } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import type { ContentEntry, Site } from "@cloudhal/core-contracts";
+import { GATES_ENFORCED, nodeHash, SiteGraphStore } from "@cloudhal/core-ts";
+import { openLocalStore } from "@cloudhal/core-ts/local";
+import { afterAll, describe, expect, it } from "vitest";
+import { init, readProjectConfig, snapshot } from "../src/index.ts";
+
+const here = path.dirname(fileURLToPath(import.meta.url));
+const STARTER_NODE_MODULES = path.resolve(here, "../../../starters/blog/node_modules");
+const BIN = path.resolve(here, "../bin/cloudhal.ts");
+const author = { kind: "human", id: "user_01HZX8K3Q9CD", name: "Halil" };
+
+async function runBuild(projectDir: string): Promise<{ code: number | null; output: string }> {
+  const { VITEST: _v, NODE_ENV: _n, ...env } = process.env;
+  const child = spawn(process.execPath, [BIN, "build"], {
+    cwd: projectDir,
+    env,
+    stdio: ["ignore", "pipe", "pipe"],
+  });
+  let output = "";
+  child.stdout?.on("data", (c: Buffer) => (output += c.toString()));
+  child.stderr?.on("data", (c: Buffer) => (output += c.toString()));
+  const code = await new Promise<number | null>((resolve) => child.once("exit", resolve));
+  return { code, output };
+}
+
+/** Places with a Turkish member (`hall`/`salon`) and without (`pier`), and events naming them. */
+async function addReferences(
+  projectDir: string,
+  routing: "prefix-except-default" | "prefix-all",
+): Promise<void> {
+  const cfg = await readProjectConfig(projectDir);
+  const local = await openLocalStore(path.join(projectDir, cfg.dataDir));
+  const store = new SiteGraphStore({
+    sql: local.sql,
+    blobs: local.blobs,
+    gates: GATES_ENFORCED,
+    fieldTypes: { supported: () => true },
+  });
+  try {
+    const site = (await store.getSite(cfg.siteId)) as Site;
+    // the seed's required category applies to every collection: reuse the seeded post's terms
+    const [hello] = await store.listContentEntries(cfg.siteId, "posts");
+    await store.updateSite({
+      ...site,
+      locales: ["en", "tr"],
+      localization: { ...site.localization, routing, fallback: "default-with-notice" },
+      collections: {
+        ...site.collections,
+        places: {},
+        events: {
+          fields: [
+            {
+              key: "venue",
+              type: "reference",
+              labels: { en: { label: "Venue" }, tr: { label: "Mekan" } },
+              localized: false,
+              collections: ["places"],
+            },
+          ],
+        },
+      },
+    });
+    const make = (collection: string, slug: string, over: Partial<ContentEntry>) =>
+      store.createContentEntry({
+        siteId: cfg.siteId,
+        collection,
+        slug,
+        locale: "en",
+        status: "published",
+        title: `The ${slug}`,
+        body: { format: "markdown", value: `A page about the ${slug}, with enough words to read.` },
+        fields: {},
+        tags: [],
+        ...(hello?.terms ? { terms: hello.terms } : {}),
+        author,
+        ...over,
+      } as never) as Promise<ContentEntry>;
+    const translate = async (of: ContentEntry, slug: string) =>
+      make(of.collection, slug, {
+        locale: "tr",
+        title: `Türkçe ${slug}`,
+        translation: {
+          group: of.id,
+          method: "human",
+          source: { id: of.id, locale: of.locale, rev: of.rev, hash: await nodeHash(of) },
+        },
+      });
+    const hall = await make("places", "hall", {});
+    await translate(hall, "salon");
+    const pier = await make("places", "pier", {});
+    await translate(await make("events", "launch", { fields: { venue: hall.id } }), "lansman");
+    await translate(await make("events", "gig", { fields: { venue: pier.id } }), "konser");
+  } finally {
+    await local.close();
+  }
+  await snapshot({ projectDir, label: "references", publish: true });
+}
+
+/** The hrefs inside a page's typed-fields list. */
+const fieldLinks = (html: string) =>
+  [
+    ...(html.match(/<dl class="ch-fields"[\s\S]*?<\/dl>/)?.[0] ?? "").matchAll(/href="([^"]*)"/g),
+  ].map((m) => m[1]);
+
+describe.each(["prefix-except-default", "prefix-all"] as const)(
+  "reference links in built pages (%s)",
+  (routing) => {
+    let tmp: string;
+    afterAll(async () => {
+      if (tmp) await rm(tmp, { recursive: true, force: true });
+    });
+
+    it("links each card once in the page's locale, the fallback in the default locale", async () => {
+      tmp = await mkdtemp(path.join(os.tmpdir(), "cloudhal-ref-links-"));
+      const projectDir = path.join(tmp, "site");
+      await init({ dir: projectDir, name: "Reference Links" });
+      await symlink(STARTER_NODE_MODULES, path.join(projectDir, "node_modules"), "dir");
+      await addReferences(projectDir, routing);
+      const { code, output } = await runBuild(projectDir);
+      expect(code, output.slice(-4000)).toBe(0);
+      const read = (...p: string[]) => readFile(path.join(projectDir, "dist", ...p), "utf8");
+      const en = routing === "prefix-all" ? ["en"] : [];
+      // a Turkish referrer: the venue's Turkish member, prefixed once
+      expect(fieldLinks(await read("tr", "events", "lansman", "index.html"))).toEqual([
+        "/tr/places/salon",
+      ]);
+      // no Turkish member: the default locale's, through `default-with-notice`
+      expect(fieldLinks(await read("tr", "events", "konser", "index.html"))).toEqual([
+        `${en.map((l) => `/${l}`).join("")}/places/pier`,
+      ]);
+      // the default locale's own page
+      expect(fieldLinks(await read(...en, "events", "launch", "index.html"))).toEqual([
+        `${en.map((l) => `/${l}`).join("")}/places/hall`,
+      ]);
+      const all = await read("tr", "events", "lansman", "index.html");
+      expect(all).not.toMatch(/href="\/(en|tr)\/(en|tr)\//);
+    });
+  },
+);

packages/cloudflare/src/admin-worker.ts

@@ -71,8 +71,9 @@ import { adminMail, type MailEnv } from "./mail.ts";
 import { purgeSiteCache } from "./purge.ts";
 import {
   type BackstopState,
+  cronTick,
   followUpDelay,
-  releasePurger,
+  publishNowPurge,
   requestBackstop,
   runAdminTick,
   type SchedulerEnv,
@@ -537,7 +538,7 @@ export function createAdminWorker(options: AdminWorkerOptions = {}) {
         // RFC-0008 §12–§13: the schedule routes and tools; "Publish now" purges like a tick.
         schedules: {
           store: new SqlScheduleStore(sql),
-          purge: releasePurger(env, sql, { log }),
+          ...publishNowPurge(env, sql, log),
           automationsEnabled: env.CLOUDHAL_AUTOMATIONS !== "off",
         },
         scheduler: {
@@ -571,8 +572,7 @@ export function createAdminWorker(options: AdminWorkerOptions = {}) {
       await ensureMigrated(sql);
       const tick = (now: Date) =>
         runAdminTick(env, {
-          trigger: "cron",
-          now,
+          ...cronTick(controller, env, now),
           log,
           analytics: (s, at) => adminAnalytics(env, s, { now: () => at, log }),
           commentsOn: commentsOn(env),

packages/cloudflare/src/scheduler.ts

@@ -30,6 +30,7 @@ import { SqlCommentStore } from "@cloudhal/comments/sql";
 import {
   type AiSearchEnv,
   type CachePurgePlan,
+  releaseBlobAllowance,
   SiteGraphStore,
   searchProviderFromEnv,
   siteCacheTag,
@@ -380,10 +381,35 @@ export interface AdminTickOptions {
   readonly commentsOn: boolean;
   /** Keeps a deferred purge's flush alive after the tick (the handler's `waitUntil`). */
   readonly waitUntil?: ((p: Promise<unknown>) => void) | undefined;
+  /**
+   * RFC-0045: the R2 calls release purges may still make in this invocation, shared by every tick
+   * it runs (the cron's follow-up); absent: one allowance per tick.
+   */
+  readonly releaseBlobs?: { left: number } | undefined;
 }
 
 const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
 
+/** "Publish now"'s purge, and the deployment's tick budget that sizes its blob calls (RFC-0045). */
+export function publishNowPurge(env: SchedulerEnv, sql: SqlDriver, log: (line: string) => void) {
+  return { purge: releasePurger(env, sql, { log }), tickBudget: tickBudget(env) };
+}
+
+const invocationBlobs = new WeakMap<object, { left: number }>();
+
+/**
+ * A cron tick's trigger and clock, with RFC-0045's R2 allowance for release purges: one per
+ * invocation (`controller`), so the cron's follow-up tick shares it (the cap is per invocation).
+ */
+export function cronTick(controller: object, env: SchedulerEnv, now: Date) {
+  let releaseBlobs = invocationBlobs.get(controller);
+  if (!releaseBlobs) {
+    releaseBlobs = { left: releaseBlobAllowance(tickBudget(env)) };
+    invocationBlobs.set(controller, releaseBlobs);
+  }
+  return { trigger: "cron" as const, now, releaseBlobs };
+}
+
 /** One tick of the admin Worker (cron, alarm, request backstop or the operator's run-due). */
 export function runAdminTick(env: SchedulerEnv, o: AdminTickOptions): Promise<TickReport> {
   const sql = new D1Driver(env.DB);
@@ -425,6 +451,7 @@ export function runAdminTick(env: SchedulerEnv, o: AdminTickOptions): Promise<Ti
         log: o.log,
       }),
     purge: ({ counted }) => releasePurger(env, counted, { log: o.log, wake: flush }),
+    ...(o.releaseBlobs ? { releaseBlobs: o.releaseBlobs } : {}),
     automationsEnabled: env.CLOUDHAL_AUTOMATIONS !== "off",
     // RFC-0008 §11.1: the built-in jobs read analytics' 404 rows and the site's own feed.
     builtinInputs: ({ counted }) => adminBuiltinInputs(env, o.analytics(counted, o.now)),