feat: RFC-0045 step 4, field operations on references and repeaters

.changeset/rfc45-step4-field-operations.md

@@ -0,0 +1,18 @@
+---
+"@cloudhal/core-ts": patch
+"@cloudhal/api": patch
+"@cloudhal/cli": patch
+---
+
+RFC-0045 rollout step 4: the four field operations work on reference and repeater fields. The types themselves stay refused on every production write path until step 7 (`fields_unsupported`, now from the dry run's check of the definitions the operation ends with); the shared suites enable them with `SiteGraphStoreOptions.fieldTypes`. Nothing changes for operations on the nine existing types.
+
+- **`sub`** on all four operations (`FieldOperation.sub`, `FieldMigration.sub`): with it, `rename_field`, `change_field_type` and `remove_field` rename, convert or remove one sub-field of a repeater in every item. An item left empty is dropped, and a list left empty is no value. `set_field_localized` refuses `sub` (sub-fields share their repeater's `localized`). Readers, writers and the rows read items through the marker; a top-level reader ignores a sub-field rename's `fromKey` (`renamedFrom`).
+- **New `change_field_type` inputs:** `collections`, `maxItems`, `subFields` and `itemLabel`, each refused where the target type does not take it. A reference changes to a reference only to switch `multiple`. A conversion that carries ids over may only widen the target collections. A repeater converts to text or to a reference only with exactly one matching sub-field.
+- **New conversion rows** (`fields/conversions-references.ts`): reference single ↔ multiple, text ↔ reference (through the prefetched `ConvertContext.entries` maps, never the entry's own group), multiline text → repeater, repeater → text, and multiple reference ↔ repeater. The maps are read once per operation pass, and by `EntryEngine.write` while such a conversion runs.
+- **Text ↔ reference values** are decided by one rule (`fields/text-reference.ts`), per value, from the `entry_refs` rows that phase C's pending check reads too: each operation pass loads the path's rows once, and a person's write reads its node's. A value with its row is kept; a legacy id-shaped text is written as a reference when its group is usable, else reported; a value that does not convert keeps its rows as they are. Readers use a display rule. Rebuilds (pull, push, import, restore) read a converting path through its earlier type. The renderer's conversion maps read archived entries too, so an archived target lends its title.
+- **Phase B's target guard** covers only the ids a rewrite newly names; ids a reshape or a rename carries over stay as stored, rows kept. **`set_field_localized`** updates the rows' `shared` in the batch of its site write (phase A, and the revert's).
+- **Revert** drops restored references whose group has no member left in the site, and reports them in `FieldRevertResult.dropped`.
+- **Values that stop converting while an operation runs** (a target archived or deleted, a second entry with the same title) are reported in `unconvertible` and keep the operation running. `continueFieldOperation` and `revertFieldOperation` take `onInvalid: "clear"` to drop them, as do the admin API's continue and revert routes (in the body) and `cloudhal fields continue|revert --clear`.
+- **Releases (#213):** when the read of the published site node fails, the run retries it once, then purges everything; it no longer falls back to the working site row.
+- **`multiple`** on a type change to a type that has none is ignored, as before.
+- **CLI:** `cloudhal fields retype` takes `--collections a,b`, `--max-items n` and `--sub-fields <file.json>`, and every action takes `--sub <sub-field>`. The admin API's operation route parses the same inputs. MCP's operation tools take them with rollout step 5.

packages/api/src/field-operations.ts

@@ -5,17 +5,18 @@
  * handlers through `/api/admin` with the deployment token (`cloudhal fields … --remote`).
  *
  * - `POST …/collections/:key/fields/operations` ← `{ op, …, dryRun? }` → the result (`done: false`
- *   when the budget ran out); `?continue=<operationId>` resumes one instead
+ *   when the budget ran out); `?continue=<operationId>` resumes one instead, with
+ *   `{ onInvalid: "clear" }` dropping values that stopped converting (RFC-0045 §7)
  * - `GET  …/collections/:key/fields/operations/:operationId` → its status
- * - `POST …/collections/:key/fields/operations/:operationId/revert` → the revert result (call again
- *   while `done: false`)
+ * - `POST …/collections/:key/fields/operations/:operationId/revert` ← `{ onInvalid? }` → the revert
+ *   result (call again while `done: false`)
  *
  * Refusals are the store's codes: `operation_in_progress` (409), `field_values_unconvertible` (422,
  * with the dry run's report in `details.report`), `field_operation_incomplete` (409), and the
  * definition codes (`unknown_field`, `field_key_duplicate`, `field_key_reserved`, …).
  */
 import { holds } from "@cloudhal/agent-runtime";
-import type { Actor, FieldOption } from "@cloudhal/core-contracts";
+import type { Actor, FieldOption, SubFieldDefinition } from "@cloudhal/core-contracts";
 import {
   continueFieldOperation,
   FIELD_OPERATIONS,
@@ -25,7 +26,7 @@ import {
   runFieldOperation,
   type SiteGraphStore,
 } from "@cloudhal/core-ts";
-import { type EnabledFieldType, FIELD_TYPES, type FieldType } from "@cloudhal/core-ts/fields";
+import { ALL_FIELD_TYPES, type FieldType } from "@cloudhal/core-ts/fields";
 import type { Context } from "hono";
 import { Hono } from "hono";
 import type { Env } from "./context.ts";
@@ -41,20 +42,52 @@ const str = (b: Body, name: string): string => {
   return v;
 };
 
-/** The request body as an operation on `collection`; 400 for a malformed one. */
+/** RFC-0045 §7: a repeater's sub-field, when the body names one. */
+function subOf(b: Body): { sub?: string } {
+  if (b.sub === undefined) return {};
+  if (typeof b.sub !== "string" || !b.sub) throw badRequest("sub must be a sub-field key");
+  return { sub: b.sub };
+}
+
+/** RFC-0045 §7: a type change's inputs for references and repeaters, each only when given. */
+function stage4Of(b: Body) {
+  const strings = (v: unknown) => Array.isArray(v) && v.every((x) => typeof x === "string");
+  if (b.collections !== undefined && !strings(b.collections))
+    throw badRequest("collections must be a list of collection keys");
+  if (b.maxItems !== undefined && !Number.isInteger(b.maxItems))
+    throw badRequest("maxItems must be a whole number");
+  const record = (v: unknown) => typeof v === "object" && v !== null && !Array.isArray(v);
+  if (b.subFields !== undefined && (!Array.isArray(b.subFields) || !b.subFields.every(record)))
+    throw badRequest("subFields must be a list of sub-field definitions");
+  if (b.itemLabel !== undefined && typeof b.itemLabel !== "string")
+    throw badRequest("itemLabel must be a sub-field key");
+  return {
+    ...(b.collections !== undefined ? { collections: b.collections as string[] } : {}),
+    ...(b.maxItems !== undefined ? { maxItems: b.maxItems as number } : {}),
+    ...(b.subFields !== undefined ? { subFields: b.subFields as SubFieldDefinition[] } : {}),
+    ...(b.itemLabel !== undefined ? { itemLabel: b.itemLabel as string } : {}),
+  };
+}
+
+/**
+ * The request body as an operation on `collection`; 400 for a malformed one. Reference and
+ * repeater targets parse; the store refuses them (`fields_unsupported`) until RFC-0045 step 7.
+ */
 export function operationOf(collection: string, b: Body): FieldOperation {
   const op = b.op;
   if (typeof op !== "string" || !FIELD_OPERATIONS.includes(op as FieldOperation["op"]))
     throw badRequest(`op must be one of ${FIELD_OPERATIONS.join(", ")}`);
-  if (op === "rename_field") return { op, collection, from: str(b, "from"), to: str(b, "to") };
-  if (op === "remove_field") return { op, collection, key: str(b, "key") };
+  const sub = subOf(b);
+  if (op === "rename_field")
+    return { op, collection, from: str(b, "from"), to: str(b, "to"), ...sub };
+  if (op === "remove_field") return { op, collection, key: str(b, "key"), ...sub };
   if (op === "set_field_localized") {
     if (typeof b.localized !== "boolean") throw badRequest("localized must be a boolean");
-    return { op, collection, key: str(b, "key"), localized: b.localized };
+    return { op, collection, key: str(b, "key"), localized: b.localized, ...sub };
   }
   const type = str(b, "type");
-  if (!FIELD_TYPES.includes(type as EnabledFieldType))
-    throw badRequest(`type must be one of ${FIELD_TYPES.join(", ")}`);
+  if (!ALL_FIELD_TYPES.includes(type as FieldType))
+    throw badRequest(`type must be one of ${ALL_FIELD_TYPES.join(", ")}`);
   if (b.onInvalid !== undefined && b.onInvalid !== "clear")
     throw badRequest('onInvalid is "clear" or absent');
   if (b.multiple !== undefined && typeof b.multiple !== "boolean")
@@ -66,12 +99,24 @@ export function operationOf(collection: string, b: Body): FieldOperation {
     collection,
     key: str(b, "key"),
     type: type as FieldType,
+    ...sub,
+    ...stage4Of(b),
     ...(b.options ? { options: b.options as FieldOption[] } : {}),
     ...(b.multiple !== undefined ? { multiple: b.multiple as boolean } : {}),
     ...(b.onInvalid === "clear" ? { onInvalid: "clear" as const } : {}),
   };
 }
 
+/**
+ * RFC-0045 §7: `{ onInvalid: "clear" }` on a continue or a revert drops the values that stopped
+ * converting while the operation ran, so it can always finish.
+ */
+function clearOf(b: Body): { onInvalid?: "clear" } {
+  if (b.onInvalid === undefined) return {};
+  if (b.onInvalid !== "clear") throw badRequest('onInvalid is "clear" or absent');
+  return { onInvalid: "clear" };
+}
+
 /** The operation must belong to the collection in the path. */
 async function ownStatus(store: SiteGraphStore, siteId: string, key: string, id: string) {
   const status = await fieldOperationStatus(store, siteId, id);
@@ -87,11 +132,14 @@ export function fieldOperationHandlers(actor: (c: Context<Env>) => Actor) {
       const siteId = param(c, "siteId");
       const key = param(c, "key");
       const resume = c.req.query("continue");
+      const b = ((await c.req.json().catch(() => null)) ?? {}) as Body;
       if (resume) {
+        const clear = clearOf(b);
         await ownStatus(store, siteId, key, resume);
-        return c.json(await continueFieldOperation(store, siteId, resume, { actor: actor(c) }));
+        return c.json(
+          await continueFieldOperation(store, siteId, resume, { actor: actor(c), ...clear }),
+        );
       }
-      const b = ((await c.req.json().catch(() => null)) ?? {}) as Body;
       if (b.dryRun !== undefined && typeof b.dryRun !== "boolean")
         throw badRequest("dryRun must be a boolean");
       const op = operationOf(key, b);
@@ -114,7 +162,10 @@ export function fieldOperationHandlers(actor: (c: Context<Env>) => Actor) {
       const siteId = param(c, "siteId");
       const id = param(c, "operationId");
       await ownStatus(store, siteId, param(c, "key"), id);
-      return c.json(await revertFieldOperation(store, siteId, id, { actor: actor(c) }));
+      const b = ((await c.req.json().catch(() => null)) ?? {}) as Body;
+      return c.json(
+        await revertFieldOperation(store, siteId, id, { actor: actor(c), ...clearOf(b) }),
+      );
     });
 }
 

packages/api/src/testing-fields.ts

@@ -183,14 +183,54 @@ export function describeFieldsApi(label: string, factory: ApiStoreFactory): void
           expect(refused.body.error).toBe("fields_unsupported");
         }
       expect((await site()).collections?.events).toBeUndefined();
-      // and no field operation turns a field into one (the API takes the nine types only)
+      // and no field operation turns a field into one: the route parses RFC-0045's inputs, and
+      // the store's dry run refuses the end state (rollout step 4)
       expect((await put(EVENTS)).status).toBe(200);
       const ops = `/api/sites/${siteId}/collections/events/fields/operations`;
-      const retype = await call(ops, {
+      for (const dryRun of [true, false]) {
+        const retype = await call(ops, {
+          method: "POST",
+          body: JSON.stringify({
+            op: "change_field_type",
+            key: "venue",
+            type: "reference",
+            collections: ["events"],
+            dryRun,
+          }),
+        });
+        expect(retype.status).toBe(409);
+        expect(retype.body.error).toBe("fields_unsupported");
+      }
+      const malformed = await call(ops, {
         method: "POST",
-        body: JSON.stringify({ op: "change_field_type", key: "venue", type: "reference" }),
+        body: JSON.stringify({
+          op: "change_field_type",
+          key: "venue",
+          type: "reference",
+          maxItems: "x",
+        }),
       });
-      expect(retype.status).toBe(400);
+      expect(malformed.status).toBe(400);
+      for (const subFields of [[1], "x", [["a"]]]) {
+        const bad = await call(ops, {
+          method: "POST",
+          body: JSON.stringify({
+            op: "change_field_type",
+            key: "venue",
+            type: "repeater",
+            subFields,
+          }),
+        });
+        expect(bad.status).toBe(400);
+      }
+      const unknownClear = await call(`${ops}?continue=snap_nothing`, {
+        method: "POST",
+        body: JSON.stringify({ onInvalid: "drop" }),
+      });
+      expect(unknownClear.status).toBe(400);
+      expect((await site()).collections?.events?.fields?.find((d) => d.key === "venue")?.type).toBe(
+        "text",
+      );
     });
 
     it("field operations: dry run, run, status, continue and revert (RFC-0041 §6)", async () => {

packages/astro-renderer/src/live-source.ts

@@ -281,7 +281,9 @@ export class LiveSource implements GraphSource {
               [...needs].map((c) => this.#store.listContentEntries(this.#siteId, c)),
             )
           ).flat();
-      return visible(read, true);
+      // Every entry, archived ones too: an archived target still lends its title (reference →
+      // text), and `conversionEntries` keeps archived ones out of what text matches.
+      return read;
     });
     this.#conversions.set(collection, { at, maps });
     maps.catch(() => this.#conversions.delete(collection));

packages/astro-renderer/src/reference-conversions.ts

@@ -3,11 +3,14 @@
  * (`ConvertContext.entries`), so `resolveFields` converts an entry phase B has not rewritten yet
  * exactly as the writers and the operation will.
  *
- * - The snapshot source builds them from the snapshot's own entries (every one not archived,
- *   drafts included: the graph it renders), lazily, once per snapshot, isolate and collection,
- *   and only while the collection carries such a marker.
- * - The live source builds them from the working copy (the graph phase B reads) before resolving
- *   an entry of a collection that carries one.
+ * - The snapshot source builds them from the snapshot's own entries (every one, drafts and
+ *   archived ones included: the graph it renders), lazily, once per snapshot, isolate and
+ *   collection, and only while the collection carries such a marker.
+ * - The live source builds them from the working copy (the graph phase B reads), every entry too,
+ *   before resolving an entry of a collection that carries one.
+ *
+ * Both pass every entry, as the store does: `conversionEntries` keeps archived ones out of what
+ * text matches (`byText`, `usable`), and lets every member lend a title (`titles`).
  *
  * Without such a marker neither reads anything. Worker-safe.
  */

packages/astro-renderer/src/source.ts

@@ -174,7 +174,10 @@ interface Loaded {
   published: ContentEntry[];
   /** Published entries with a public URL (unlisted ones too); `entries` holds the listed. */
   reachable: ContentEntry[];
-  /** RFC-0045 §7: every entry not archived, drafts included; kept only while a conversion runs. */
+  /**
+   * RFC-0045 §7: every entry, drafts and archived ones included (`conversionEntries` filters what
+   * matches text itself); kept only while a conversion runs.
+   */
   current: ContentEntry[];
   /** Every published entry's derived gate (RFC-0042 §3), computed once per snapshot. */
   derived: ReadonlyMap<string, DerivedGate>;
@@ -385,7 +388,7 @@ export class SnapshotSource implements GraphSource {
         entries: shownEntries,
         published: shownEntries,
         reachable: shownEntries,
-        current: siteConverts(site as Site) ? visible(entries, true) : [],
+        current: siteConverts(site as Site) ? entries : [],
         derived,
         resolved: new Map(),
         taxonomies,

packages/astro-renderer/test/references.test.ts

@@ -5,6 +5,7 @@
  * active); gate instants; depth, cycles and the per-request budget.
  */
 import type { ContentEntry, Site } from "@cloudhal/core-contracts";
+import { canonicalJson, nodeHash } from "@cloudhal/core-ts";
 import { afterEach, beforeAll, describe, expect, it, vi } from "vitest";
 import { entryFields, type FieldsSource, type RenderedFields } from "../src/fields.ts";
 import { linkLocalizer, localizeLinks } from "../src/locales.ts";
@@ -372,3 +373,48 @@ describe("reference links go through the locale middleware once (RFC-0045 §4)",
         expect(h, h).not.toMatch(/^\/(en|tr)\/(en|tr)\//);
     });
 });
+
+describe("reference → text titles on both sources (RFC-0045 §7, PR #214 round 2)", () => {
+  it("an archived target and an archived default-locale member lend their titles, as in the store", async () => {
+    const w = await refsWorld();
+    const { e, store } = w;
+    // the hall's English member is archived; its Turkish member stays
+    await store.updateContentEntry({
+      ...((await store.getContentEntry(e.hall?.id as string)) as ContentEntry),
+      status: "archived",
+    });
+    // `change_field_type` reference → text, phase A: venue is text with the marker
+    const site = (await store.getSite(w.site.id)) as Site;
+    const events = site.collections?.events;
+    const fields = (events?.fields ?? []).map((d) =>
+      d.key === "venue"
+        ? {
+            key: "venue",
+            type: "text",
+            labels: d.labels,
+            localized: false,
+            migration: { id: "fop_01HZX8K3Q9CD", fromType: "reference" },
+          }
+        : d,
+    );
+    const next = {
+      ...site,
+      rev: site.rev + 1,
+      collections: { ...site.collections, events: { ...events, fields } },
+    } as unknown as Site;
+    await store.sql.run("UPDATE nodes SET rev = ?, hash = ?, body = ? WHERE id = ?", [
+      next.rev,
+      await nodeHash(next),
+      canonicalJson(next),
+      site.id,
+    ]);
+    const snap = await snapshotOf(w, next);
+    const live = new LiveSource(store, site.id, { includeDrafts: false, now: () => T });
+    for (const source of [snap, live]) {
+      const night = await source.entryFields(e.night as ContentEntry);
+      expect(night.venue?.text).toBe("Title vault");
+      const launch = await source.entryFields(e.launch as ContentEntry);
+      expect(launch.venue?.text).toBe("Title hall");
+    }
+  });
+});

packages/cli/src/cli.ts

@@ -75,6 +75,10 @@ export const OPTIONS = {
   options: { type: "string" },
   multiple: { type: "boolean" },
   clear: { type: "boolean" },
+  collections: { type: "string" },
+  "max-items": { type: "string" },
+  "sub-fields": { type: "string" },
+  sub: { type: "string" },
 } as const;
 
 /**

packages/cli/src/cli/help.ts

@@ -97,7 +97,7 @@ const CONTENT = `  cloudhal export [--out <file>] [--site <id>] [--no-media] [--
   cloudhal migrate tags-to-terms [--dry-run] [--revert <id>]
                                                           move entry tags into tag terms (RFC-0002);
                                                           resumable; --revert <id> undoes a run
-  cloudhal fields rename <c> <from> <to> | retype <c> <key> <type> [--options a,b] [--multiple] [--clear] | localize <c> <key> on|off | remove <c> <key> | continue|revert <c> <id>  [--dry-run] [--remote]
+  cloudhal fields rename <c> <from> <to> | retype <c> <key> <type> [--options a,b] [--multiple] [--collections a,b] [--max-items n] [--sub-fields file.json] [--clear] | localize <c> <key> on|off | remove <c> <key> | continue|revert <c> <id> [--clear]  [--sub <sub-field>] [--dry-run] [--remote]
                                                           change a collection's typed fields with their values
                                                           (RFC-0041): snapshotted, resumable; --dry-run reports
                                                           and greps the theme; revert <id> undoes a run