From 74d709895346e9e5eed03274078a427f2f3ba24d Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Fri, 13 Nov 2026 20:53:46 +0300 Subject: [PATCH 1/2] feat(core-ts,cloudflare,api,cli,storage): RFC-0045 step 7b, enable reference and repeater fields - The gate accepts both types wherever the renderer serving the site reads them: the admin Worker asks the site Worker for `fields: 2` (siteSupport, beside the gates check); openLocalStore and the playground pass FIELDS_SUPPORTED; the admin push route passes the store's capability; cloudhal pull, import (dry runs too) and --revert pass the local store's. Without a capability the types stay refused, bundles included. - One-time per-site entry_refs rebuild (rebuildEntryRefs): rows from the import rebuild's own code, each node anchored on its read hash, the collections anchor on every statement, resumable by cursor in the new entry_refs_state table (migration 0034), orphans removed per site. Wired as the admin Worker's refs.rebuild system job and in openLocalStore. - Tests: the job on SQLite, D1 and DO (R17, cursor, orphans, definitions change, rows equal writer and import rebuild); bundle gate with and without the capability for import, push, pull and restore; a capable site's held site body cannot be re-pointed by another site's snapshot; API push and workerd e2e against old and new site Workers. - docs/guides/fields.md documents both types; RFC "Amended in 7b"; ADR-0046 and comments updated; changeset (patch). Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- .changeset/rfc45-step7b-enable.md | 15 + .../components/fields/useReferenceTypes.ts | 5 +- docs/adr/0046-references-and-repeaters.md | 2 +- docs/guides/fields.md | 94 ++++- ...ctions-stage-4-references-and-repeaters.md | 18 + packages/api/src/field-operations.ts | 2 +- packages/api/src/routes/admin/sync.ts | 2 + packages/api/test/admin.test.ts | 27 +- packages/api/test/studio-endpoints.test.ts | 2 +- packages/astro-renderer/README.md | 2 +- .../astro-renderer/test/references-fixture.ts | 2 +- packages/cli/src/commands/fields.ts | 2 +- packages/cli/src/commands/import-apply.ts | 6 +- packages/cli/src/commands/import-revert.ts | 8 +- packages/cli/src/commands/import.ts | 16 +- packages/cli/src/commands/sync.ts | 2 + packages/cli/test/cloudflare-gates.test.ts | 48 ++- packages/cli/test/fields-cli.test.ts | 11 +- packages/cloudflare/src/admin-worker.ts | 16 +- packages/cloudflare/src/entry-refs-job.ts | 46 +++ packages/cloudflare/src/features.ts | 43 +- packages/cloudflare/src/playground-site.ts | 5 +- packages/cloudflare/src/scheduler.ts | 2 + packages/cloudflare/test/comments.test.ts | 15 +- .../cloudflare/test/entry-refs-job.test.ts | 136 +++++++ packages/cloudflare/test/features.test.ts | 37 +- packages/core-ts/src/errors.ts | 2 +- .../core-ts/src/fields-operations/plan.ts | 4 +- packages/core-ts/src/fields-operations/run.ts | 4 +- packages/core-ts/src/fields/index.ts | 2 +- packages/core-ts/src/fields/types.ts | 9 +- packages/core-ts/src/index.ts | 6 + packages/core-ts/src/local.ts | 8 +- .../core-ts/src/references/definitions.ts | 6 +- .../core-ts/src/references/rebuild-job.ts | 284 +++++++++++++ packages/core-ts/src/references/rebuild.ts | 59 ++- packages/core-ts/src/site-delete.ts | 1 + packages/core-ts/src/store/types.ts | 5 +- packages/core-ts/src/sync-bundle.ts | 2 +- .../core-ts/src/testing-entry-refs-job.ts | 377 ++++++++++++++++++ packages/core-ts/src/testing-gates-fields.ts | 185 +++++++++ packages/core-ts/src/testing-gates.ts | 4 + .../core-ts/src/testing-reference-gate.ts | 7 +- packages/core-ts/src/testing.ts | 2 + packages/mcp-server/src/collection-fields.ts | 4 +- .../migrations/0034_entry_refs_state.ts | 25 ++ packages/storage/migrations/index.ts | 2 + packages/storage/test/migrate.test.ts | 6 + scripts/mcp-tools-doc.ts | 6 +- scripts/size-baseline.json | 4 +- 50 files changed, 1497 insertions(+), 81 deletions(-) create mode 100644 .changeset/rfc45-step7b-enable.md create mode 100644 packages/cloudflare/src/entry-refs-job.ts create mode 100644 packages/cloudflare/test/entry-refs-job.test.ts create mode 100644 packages/core-ts/src/references/rebuild-job.ts create mode 100644 packages/core-ts/src/testing-entry-refs-job.ts create mode 100644 packages/core-ts/src/testing-gates-fields.ts create mode 100644 packages/storage/migrations/0034_entry_refs_state.ts diff --git a/.changeset/rfc45-step7b-enable.md b/.changeset/rfc45-step7b-enable.md new file mode 100644 index 00000000..0864e483 --- /dev/null +++ b/.changeset/rfc45-step7b-enable.md @@ -0,0 +1,15 @@ +--- +"@cloudhal/core-ts": patch +"@cloudhal/storage": patch +"@cloudhal/cloudflare": patch +"@cloudhal/api": patch +"@cloudhal/cli": patch +"@cloudhal/mcp-server": patch +--- + +RFC-0045 rollout step 7b: **reference and repeater fields are now enabled.** Sites can define `reference` fields (entries of one to five collections of the same site, one or several, with optional conditions) and `repeater` fields (ordered lists of items with up to ten flat sub-fields) wherever the renderer serving them reads the two types. See [docs/guides/fields.md](https://github.com/cloudhal-dev/cloudhal/blob/main/docs/guides/fields.md). + +- **Where.** On Cloudflare, once the site Worker answers `GET /__cloudhal/features` with `fields: 2` (0.9.0 and later): the admin Worker asks it, as it asks for time gates, and remembers a yes. Deploy the site Worker first; until it reports `fields: 2`, the types stay refused with `fields_unsupported` on every path. On Node (`cloudhal dev`, `studio`, self-host, Docker) and in the playground, always, since the renderer runs in the same process (`openLocalStore` passes `FIELDS_SUPPORTED`). +- **Every write path follows the capability:** Studio, MCP, the API, field operations and their dry runs, and every bundle: `cloudhal push` (the admin route passes the store's capability), `pull`, `cloudhal import` (dry runs included), `--replace` and `--revert`. A bundle that defines either type is still refused whole where the capability does not hold, and a site's new snapshot cannot point at another site's held site body to bring the types in. +- **MCP and Studio.** `update_collection` and `change_field_type` list both types, and `/api/instance` answers `features.referenceFields: true`, on servers that have the capability. The store still decides per site. +- **A one-time `entry_refs` check per site** (`rebuildEntryRefs`, new core-ts export): the admin Worker's new `refs.rebuild` system job (every five minutes, offset from `lists.backfill`, until every site is done; then two reads a run), and `openLocalStore` at start-up. Rows are written per entry, anchored on the hash each entry was read with, so a concurrent edit keeps its own rows; each chunk carries the site's collections anchor; progress is in the new `entry_refs_state` table (storage migration `0034_entry_refs_state`), so a run stopped anywhere resumes. It is a safety net and should find nothing to change. diff --git a/apps/studio/src/components/fields/useReferenceTypes.ts b/apps/studio/src/components/fields/useReferenceTypes.ts index fd16cf20..1ed10a10 100644 --- a/apps/studio/src/components/fields/useReferenceTypes.ts +++ b/apps/studio/src/components/fields/useReferenceTypes.ts @@ -3,8 +3,9 @@ import { api } from "../../lib/api.ts"; /** * RFC-0045: whether the server takes reference and repeater fields (`/api/instance`, - * `features.referenceFields`). Until rollout step 7 production answers no, and the builder offers - * neither type, so it looks as before; an older server omits it, which reads as no. + * `features.referenceFields`): true where the store has the capability (rollout step 7: the admin + * Worker, Node and the playground). Without it the builder offers neither type, so it looks as + * before; an older server omits it, which reads as no. The store still decides per site. */ export function useReferenceTypes(): boolean { const instance = useQuery({ queryKey: ["instance"], queryFn: api.instance, retry: false }); diff --git a/docs/adr/0046-references-and-repeaters.md b/docs/adr/0046-references-and-repeaters.md index 14661b21..8489f6e3 100644 --- a/docs/adr/0046-references-and-repeaters.md +++ b/docs/adr/0046-references-and-repeaters.md @@ -8,7 +8,7 @@ Collections have typed fields (ADR-0043), but a venue, a speaker or a brand is still plain text retyped on every entry, and a recipe's ingredients are lines of one text field. Two field types fix that, and each needs integrity rules that hold under D1, which has only atomic batches (AGENTS.md rule 4). ## Decision -1. **Two types.** `reference` (one to five target collections of the same site, one target or up to 50) and `repeater` (up to 50 items of up to ten flat scalar sub-fields; no nesting). Both are refused on every write path until rollout step 7, when the site Worker reports `fields: 2`: definitions with `fields_unsupported`, field operations that name them with `field_operation_unsupported` (until step 4). +1. **Two types.** `reference` (one to five target collections of the same site, one target or up to 50) and `repeater` (up to 50 items of up to ten flat scalar sub-fields; no nesting). Both were refused on every write path until rollout step 7. Since then they are accepted where the renderer serving the site reads them: on Cloudflare once the site Worker reports `fields: 2`, on Node always (`FIELDS_SUPPORTED`). Elsewhere definitions are refused with `fields_unsupported`, bundles included; an absent capability refuses. 2. **A reference stores a translation group id**, `COALESCE(translation.group, id)`, the one identity shared by every language version in every site. It resolves only through members of the page's own site, in the page's locale, then `localization.fallback`. Membership is that expression everywhere (guards, renderer, purges). 3. **`entry_refs`** holds one row per (holder, field path, stored group id), whatever the target's state, written in the holder's own batch, anchored on its new hash, with a plain `INSERT` (never `OR IGNORE`) that reads `shared` from the site row at batch time. Every guard and lookup filters `site_id`. 4. **Strong where the live site depends.** Only what a published entry shows locks: a delete, a move out of every allowed collection, or a group change of a group's last usable member is refused while a published entry shows a reference to it (`entry_referenced`, `reference_target_moved`). Archiving and unpublishing are always allowed; an unresolved reference renders nothing. Publishing a *site* warns (`referenceIssues`) and never fails; publishing a *referrer* whose own value names a group with no usable member is refused (`unknown_entry`). diff --git a/docs/guides/fields.md b/docs/guides/fields.md index 0a724bbe..d53c70dd 100644 --- a/docs/guides/fields.md +++ b/docs/guides/fields.md @@ -2,7 +2,7 @@ A collection can declare **typed fields**: an event has a venue, a start time and a ticket link; a job has a deadline and a salary range. Agents fill them through a JSON Schema instead of guessing, people get a form in Studio, and themes can rely on every value being there in the right shape. -The design and its reasons are in [RFC-0041](../rfcs/0041-typed-custom-fields.md) and [ADR-0043](../adr/0043-typed-custom-fields.md). +The design and its reasons are in [RFC-0041](../rfcs/0041-typed-custom-fields.md) and [ADR-0043](../adr/0043-typed-custom-fields.md); references and repeaters in [RFC-0045](../rfcs/0045-collections-stage-4-references-and-repeaters.md) and [ADR-0046](../adr/0046-references-and-repeaters.md). ## What a field is @@ -19,6 +19,10 @@ Each field has a **key** (lower camelCase, like `startsAt`), **labels** per lang | `url` | `https:`, `http:`, `mailto:`, `tel:`, a site-relative path or a `#fragment` | a link | | `image` | an image asset of the same site (`asset_…`) | an `` with the asset's alt text in the reader's language | | `select` | one of the field's options, or a list of them with `multiple` | the options' labels in the reader's language | +| `reference` | another entry of the same site, by its translation group id (`entry_…`), or a list of them with `multiple` | a link to the target in the reader's language, or nothing while no published version shows | +| `repeater` | an ordered list of items, each a flat object of sub-fields | a nested list, one item each | + +References and repeaters have [their own section](#references-and-repeaters) below. A field can be **required to publish** (`requiredToPublish`, every type but `boolean`): drafts save without it, but an entry cannot go to review or be published while the value is missing. Text and rich-text fields marked **searchable** add their words to site search from the next publish on (off by default, so internal notes never reach search snippets; see the [search guide](search.md)). @@ -44,6 +48,74 @@ Adding fields, reordering, editing labels and help, adding options and toggling Each runs a **dry run** first (`--dry-run`): how many entries change, and which values move, convert or would be cleared. A type change converts values with a fixed table (text to rich text, number, select or link; any of those back to text; date to datetime at midnight in the site's time zone and back); values that do not convert block it unless you opt in to clearing them (`--clear`, `onInvalid: "clear"`). Every operation takes a snapshot first and can be reverted (`cloudhal fields revert `). A long operation resumes where it stopped (`cloudhal fields continue`). Operations change the draft graph only; publish as usual. +## References and repeaters + +### Reference fields + +A **reference** points to entries of the same site: an event's venue, a talk's speakers, a product's brand. + +- **Targets.** `collections` names one to five collections a target may belong to (declared collections of the site, or ones its entries already use). A target is never the entry itself or another language version of it. +- **One or several.** Without `multiple` the value is one target. With `multiple: true` it is an ordered list without repeats, of at most `maxItems` targets (20 when absent, 50 at most). +- **Conditions.** With exactly one target collection, `where` (up to four conditions on that collection's list fields, [RFC-0042](../rfcs/0042-collections-stage-3.md), no `now`) narrows what may be *newly* chosen, as Studio's picker does: "only places with `capacity` ≥ 500". A target that stops meeting it later stays. +- **What is stored.** The target's **translation group id**: the id of its group's original, the same in every language version and in every language edition of the project. A page in Turkish shows the Turkish version of the venue, else the default language's under `localization.fallback: default-with-notice`, else nothing. When the target has no published version the page can show (a draft, archived, embargoed or hidden entry, or one deleted elsewhere), the field shows nothing: no id, no title, no empty link. +- **Publishing warns, never fails.** Studio's site publish dialog lists published entries whose references will not resolve, and `cloudhal import --dry-run` reports them for an archive. + +### Repeater fields + +A **repeater** is an ordered list of items with the same sub-fields: an agenda's sessions, a recipe's ingredients, a job's benefits. + +- **Sub-fields.** `fields` holds one to ten sub-fields, each with a key, labels and a type: `text`, `richText`, `number`, `boolean`, `date`, `url`, `image`, a single `select` or a single `reference`. No `datetime`, no multiple select or reference, and no repeater inside a repeater, so every schema agents see has a fixed depth. +- **Items.** At most `maxItems` items (20 when absent, 50 at most). An item holds only sub-field keys, and an empty item is refused. `requiredToPublish` on a sub-field means every item of an entry in review or published needs it; on the repeater, at least one item. +- **Item labels.** `itemLabel` names the text, select or reference sub-field that names an item in Studio's legends ("Item 2: Ada Lovelace") and in approval prompts. Absent: the first such sub-field. +- **Languages.** `localized` is the repeater's and applies to the whole list: each language version has its own list, or only the original holds one. Sub-fields have no `localized` of their own. +- **Images and rich text** inside items are checked like top-level ones (an image sub-field takes image assets only) and count as uses of the asset. + +### Limits + +- **100 leaves per collection:** fields plus repeater sub-fields. Top-level fields stay at 30 per collection. +- **200 per site**, sub-fields included. +- **100 references per entry**, across fields and repeater items (`too_many_references`). + +### The delete lock and "Used by" + +A reference holds its target in place while a **published** entry shows it, its own value or one a published translation inherits: + +- **Deleting** the target's last version in the site, **moving** it to a collection the field does not allow, or **relinking** it into another translation group is refused with `409 entry_referenced`, which names up to 20 published referrers and the total. Studio's delete dialog lists them before you confirm; drafts that would lose the value are listed too, without blocking. Deleting one language version while another remains is allowed: that language's pages then fall back or show nothing, and the dialog says which. +- **Archiving or unpublishing** the target is always allowed. It renders nothing from then on, but it does **not** release the lock: the referrers still hold it, so the delete stays refused until they change or are unpublished. Unpublishing a referrer releases its hold at once. +- Drafts, entries in review and archived entries lock nothing on their own. A contributor, who cannot publish, therefore never pins anyone's entry. +- **"Used by"** on an entry's page in Studio lists the entries that reference it, published or not, with the field. Agents use `list_references` (`direction: "to"` or `"from"`). + +### Field operations on references and repeaters + +The four operations above work on both types, and take `sub` to work on one sub-field of a repeater (`cloudhal fields rename events speakers who --sub person` renames the sub-field `person` of `speakers` to `who`; in Studio, pick the sub-field in the operation's dialog): + +- **Rename and remove** a sub-field. Removing a repeater's only sub-field is refused: remove the repeater instead. `set_field_localized` takes no `sub`, since sub-fields share their repeater's setting. +- **Change type** has these conversions besides RFC-0041's table: + - text ↔ reference: a text value becomes the target whose title it matches in the field's collections (exactly one, not the entry itself); a reference becomes its target's title; + - reference single ↔ multiple, and reference (multiple) ↔ repeater with one reference sub-field; + - a repeater with exactly one text, rich-text or reference sub-field → text (one line per item) or reference (multiple); + - a sub-field's type (`--sub`), with `--collections` for a new reference sub-field. + + A conversion that carries ids over may only widen the target collections. The CLI takes `--collections a,b`, `--max-items n` and `--sub-fields file.json` (a JSON list of sub-field definitions). +- **While a conversion into references runs**, deletes, moves and relinks of entries in its target collections wait (`409 field_operation_running`), so no value it converts can lose its target. +- **"Clear and finish."** A value can stop converting while an operation runs (its target was archived, or a second entry took the same title). The operation then stops with those entries listed as unconvertible and keeps its marker. Studio's dialog offers **Clear and finish** behind a confirmation; the CLI's `continue` and `revert` take `--clear`, the API and MCP `onInvalid: "clear"`. It drops only what no longer converts. + +### Agents (MCP) + +- **Schemas.** `list_collections` returns `fieldsSchema` (references as `entry_…` strings, repeaters as arrays of flat objects; no `$ref`, depth at most 4) and `fieldsSchemaStrict` for providers' strict modes, with `strictCost` (`properties`, `depth`, `enumValues`, `unions`) so an agent can tell when to send a subset. Each property of the strict schema stands alone, so splitting it is safe. +- **Writing.** `create_entry` and `update_entry` take `fields` (a merge patch: `null` removes a key) or **`fieldsStrict`** (strict output: `null` means "leave as it is", and inside an item "unset"), with **`clearFields`** for removals. A given reference or repeater value replaces the whole value. Items left empty by `null`s are dropped and reported (`droppedItems`). The prechecks run the store's own checks, and approval prompts name targets readably (`venue → "Main hall" (entry_…, Places)`). +- **Translations.** Item paths are `key/index/sub` (`speakers/0/talk`). `get_translation_source` returns `sourceRev`, which `save_translation` needs with item paths. Once a translation holds its own list for a repeater, item paths into it are refused (`translation_shape_mismatch`): `get_translation_source` then returns the source's whole list under the repeater's key in `fields` and names it in **`wholeLists`**. Translate its text and rich-text sub-values and send the whole list back, in the source's order. A whole list sent alone is the translation's own and is stored as sent. References and other sub-values are not translated. +- **`update_collection`** and **`change_field_type`** list both types where the server can render them; the store still decides per site (below). + +### Where they are available + +Both types are accepted on every write path (Studio, MCP, the API, `cloudhal fields`, imports, `push`, `pull`, `cloudhal import --replace` and `--revert`) wherever the renderer serving the site reads them: + +- **Cloudflare:** once the site Worker answers `GET /__cloudhal/features` with `fields: 2` (0.9.0 and later). Deploy the site Worker first; until it reports it, the admin Worker refuses both types with `fields_unsupported`, and a push or archive that defines them is refused whole. +- **Node** (`cloudhal dev`, `studio`, self-host, Docker): always, since the renderer runs in the same process. + +Upgrading runs a one-time check of every site's reference index (`refs.rebuild`, every five minutes on the admin Worker until each site is done; at start-up on Node). It is a safety net and should find nothing to change. + ## Languages Each field decides once whether it is **localized**: @@ -80,6 +152,17 @@ It draws a plain definition list of the fields that have a value, in definition It has no styles of its own. Style `.ch-fields`, `.ch-field[data-type="…"]` or one field with `[data-field=""]`. Multi-line text fields carry `data-multiline` (give them `white-space: pre-line`), and values read from the original carry `data-from-original`. +A **reference** is a link to each target the page shows, by its graph path (localized like every entry link, with `lang` on a card from the fallback language), several in a `ul.ch-references`. A **repeater** is an `ol.ch-items` with one nested `dl` per item that has something to show: + +```html + +
Speakers
    +
  1. +
+``` + +Themes from v0.5.0 on draw both with no change of their own. + ### Your own markup: `entryFields` `source.entryFields(entry, options)` (or `entryFields(source, entry, options)` from `@cloudhal/astro-renderer/fields`, for any source) returns every defined field by key, in definition order, ready to draw: @@ -94,8 +177,12 @@ It has no styles of its own. Style `.ch-fields`, `.ch-field[data-type="…"]` or | `image` | `image` only: `` attributes (`src`, `srcset`, `sizes`, `width`, `height`, `alt`, …) and `assetId` | | `datetime` | `date` and `datetime`: the machine-readable value for `