diff --git a/.agents/skills/deco-v7-to-v8-migration/SKILL.md b/.agents/skills/deco-v7-to-v8-migration/SKILL.md index d1204bf2a..d085a4873 100644 --- a/.agents/skills/deco-v7-to-v8-migration/SKILL.md +++ b/.agents/skills/deco-v7-to-v8-migration/SKILL.md @@ -5,7 +5,7 @@ description: Moves a v7 Deco site (@decocms/blocks 7.x with @decocms/tanstack or # Deco v7 → v8 Migration -Moves a v7 site onto the next major in two parts: a script that does the mechanical, content-safe part in one pass, and a list of manual steps it prints. Proven on `deco-sites/storefront-tanstack` (Shopify, TanStack Start) and `deco-sites/blog-tanstack` (TanStack Start). +Moves a v7 site onto the next major in two parts: a script that does the mechanical, content-safe part in one pass, and a list of manual steps it prints. Proven on `deco-sites/storefront-tanstack` (Shopify, TanStack Start), `deco-sites/blog-tanstack` (TanStack Start), a Next.js App Router storefront on VTEX, a TanStack Start storefront on VTEX and a non-ejected FastStore storefront on VTEX (hundreds of saved blocks, private). The spec is the docs page **Migrating from v7** (`/next/renames-and-migrations`). When this skill and the docs disagree, the docs win. @@ -14,6 +14,10 @@ The spec is the docs page **Migrating from v7** (`/next/renames-and-migrations`) - `package.json` depends on `@decocms/blocks` `^7`, and on `@decocms/tanstack` or `@decocms/nextjs`, `@decocms/blocks-admin`, `@decocms/blocks-cli` or v7 `@decocms/apps-*`. - Saved content lives in `.deco/blocks/*.json`, or the site serves it at `/.decofile`. +**FastStore storefronts that aren't ejected** (FastStore owns the Next.js app; v7 monkey-patched its content requests) keep that seam with v8 behind it: follow `reference/faststore.md` for steps 2, 3 and 6. + +**Native or mobile apps** whose v7 package only generated Studio's files (bundled JSON, native rendering) also apply. The script still runs, but most manual steps don't: follow `reference/native-apps.md`. + Not for sites still on `@decocms/start@6.x` (upgrade to 7.x first with `decocms-v6-to-v7-upgrade`) or on Fresh/Deno (migrate to TanStack first with `deco-to-tanstack-migration`; both live on the 7.x branch). ## Prerequisites @@ -50,7 +54,8 @@ rm -rf .deco-migrate What it does, in order (`scripts/migrate.ts`): -1. **content**: saved blocks into `.deco/blocks`, v7 generated files removed, the seven legacy names outside the alias table rewritten (`reference/legacy-names.md`), A/B tests keyed on a random matcher get its saved-block name as `experiment`; +1. **content**: saved blocks into `.deco/blocks`, v7 generated files removed, v7 async-rendering wrappers (`website/sections/Rendering/Lazy.tsx`, `SingleDeferred.tsx`, `Deferred.tsx`) unwrapped to the sections they held (v8 has no async rendering), the seven legacy names outside the alias table rewritten (`reference/legacy-names.md`), A/B tests keyed on a random matcher get its saved-block name as `experiment`; + **settings**: site settings folded into the `CMS` block (`.deco/blocks/CMS.json`, type `cms-settings`): the v7 Site block's (`Site`/`site`) `previewHosts` → `preview.hosts` (the field moves, ports kept, each entry trimmed and lowercased as v7 compared it; an entry that isn't a host is reported; on TanStack Start the `.deco.site` and `.deco-cx.workers.dev` hosts v7 always added for `DECO_SITE_NAME` are added too, or reported when the name isn't found), a literal `collectorAddress` on `OneDollarStats` → `analytics.collector`, and an earlier prerelease's `Telemetry.json`/`Analytics.json` → their sections, field for field and variants included (the old files are deleted). It keeps what `CMS.json` already has, writes nothing when there's nothing to fold, and a second run changes nothing. The report lists the environment-only settings to move by hand (`DECO_ALLOWED_PREVIEW_HOSTS`, `DECO_OTEL_*` sampling, `DECO_ANALYTICS_ENABLED`/`ONEDOLLAR_ENABLED`/`ONEDOLLAR_COLLECTOR`); 2. **secrets**: v7 secrets re-encrypted with `.deco/secrets.pub`; 3. **block map**: `.deco/index.ts` with each block under a short name plus an alias under its v7 name, after vendoring the app loaders and actions the content calls into `src/vendor`; 4. **imports**: the codemod over `src/` (`reference/import-map.md`); @@ -64,12 +69,12 @@ The script prints **Done** and **Left to do**, grouped by step. Every "Left to d ## Manual steps (what the report leaves) -1. **Dependencies.** Depend on `@decocms/blocks@^8.1` and the v8 `@decocms/apps-` client. Remove `@decocms/tanstack`/`@decocms/nextjs`, `@decocms/blocks-admin`, `@decocms/blocks-cli`, `@decocms/apps-commerce`, `@decocms/apps-website` and `@decocms/apps-blog` once nothing imports them, and drop the v7 codegen from `build`. +1. **Dependencies.** Depend on `@decocms/blocks@^8.1` and the v8 `@decocms/apps-` client. Remove `@decocms/tanstack`/`@decocms/nextjs`, `@decocms/blocks-admin`, `@decocms/blocks-cli`, `@decocms/apps-commerce`, `@decocms/apps-website` and `@decocms/apps-blog` once nothing imports them, and drop the v7 codegen from `build`. On Next.js, add `@decocms/blocks` to `transpilePackages` in `next.config`: the `8.1.0-next.*` prereleases publish TypeScript source (`exports` → `src/*.ts`), v7's `withDeco` wrapper used to add it, and without it a clean install fails `next build` with `Module parse failed: Unexpected token`. 2. **Render pages with `createCMS`.** Follow the framework guide (`/next/tanstack-start-descriptors`, `/next/nextjs`): `createCMS` over the content, `matchRoute`, one promise per block, a view registry. Delete the v7 setup files, admin routes and `/deco/*` handlers. 3. **Move framework code into the site** (`reference/gotchas.md`): edge cache, image, SEO/head, device detection, cookies, cart/user/wishlist flows, commerce loaders and converters. -4. **Replace `/deco/invoke`** with server functions (TanStack `createServerFn`) or Next server actions/route handlers. +4. **Replace `/deco/invoke`** with server functions (TanStack `createServerFn`) or Next server actions/route handlers. One exported server function per loader or action the browser called, and each call site imports the one it calls: `invoke.site.loaders.spin(props)` becomes `$siteLoadersSpin({ data: props })`. Don't rebuild the `invoke.x.y` tree or a string-keyed dispatcher over the commerce-loader map (see DO NOT); the server function calls its handler directly. Details in `reference/gotchas.md` (`/deco/invoke` → server functions). 5. **Fix the content `deco check` rejects**: fields no type declares, `.tsx`-named preview blocks, v7 app blocks (apps are code now), `requestToParam` blocks in string fields. -6. **Telemetry and analytics**: the `telemetry` option of `createCMS` (`/next/telemetry`); the built-in analytics block with `AnalyticsScript`/`track` from `@decocms/blocks/analytics` (the report carries GTM/GA4 IDs over). +6. **Telemetry, analytics and previews**: the `telemetry` option of `createCMS` (`/next/telemetry`), with switches and rates in the `telemetry` section of `CMS.json`; `AnalyticsScript` with `const { analytics } = await cms.settings()` and `track` from `@decocms/blocks/analytics`; drafts through `await cms.draftPointer(request)` / `cms.draftCookie(request)`, which serve the release on hosts outside `preview.hosts` (list the dev host, `localhost`, too: v8 ignores `DECO_ALLOWED_PREVIEW_HOSTS`) (with no `previewHosts`, v7 allowed previews only on a TanStack site's `.deco.site` and `.deco-cx.workers.dev`, or nowhere; here every host may preview unless you list some, `/next/releases-and-drafts#allow-previews-per-host`). The report carries GTM/GA4 IDs over). ## Verify @@ -78,11 +83,19 @@ bunx deco schema && bunx deco content && bunx deco check # 0 errors bun run typecheck && bun run build ``` -Then compare the migrated site with the v7 one page by page (a parity harness: SSR HTML, JSON-LD, analytics calls, cache headers, third-party requests). Encode every difference the product owner approves as an explicit rule, never a blanket ignore (`reference/parity.md`). +The committed `.deco/schema.gen.json` must carry the top-level `"blocksMajor": 8` that `deco schema` writes: Studio treats a site as v8 **only** when its schema has `blocksMajor === 8` (the file name doesn't count; anything else edits as v7), and `deco check` fails without it. A schema generated by a CLI that predates the field must be regenerated with one that writes it, then committed. + +A v7 site often carries type errors of its own. Count them on the v7 commit first, and report the v8 count against it per file: name every remaining error in a file the migration touched instead of claiming there are none. Fix them with type-only changes, and rerun `deco schema` + `deco check` after each: a section's `Props` type *is* its editor form, so repairing a broken type import there (one that resolved to `any`) changes the schema and can make `deco check` reject saved content. Leave such an error, or fix the content with it, never silently. Don't fix an error by giving code that never ran (an undefined variable in a component) real values: that changes what renders. + +Run it once more **without any local link** (a `decocms/blocks` checkout linked into `node_modules` ships compiled `dist/`, the npm package may not): a clean `install --frozen-lockfile && build` from the committed lockfile, the way CI and the deploy run it. Then `git diff .deco/schema.gen.json`: the build regenerates it, so commit what the committed dependency's CLI writes, or every clean build leaves the tree dirty. If it differs from the linked checkout's, check both against v7's forms; when the published one is the worse, list a follow-up to bump once the fix is released. + +**Probe drafts by hand** on the dev server and on a production build: `?__draft=` on an allowed host renders the draft and sets the cookie, the cookie alone keeps the next page in the draft, a host outside `preview.hosts` and a malformed pointer get the release, `?__draft=off` clears the cookie. A parity harness can't catch a dead draft path when v7's was dead too: both sides render the release. + +Then compare the migrated site with the v7 one page by page (a parity harness: SSR HTML, JSON-LD, analytics calls, cache headers, third-party requests). Compare the editor forms as well (v7 `meta.gen.json` vs v8 `schema.gen.json`): many differences are stale v7 files or v7 heuristics v8 drops on purpose, a few are CLI bugs to fix. Encode every difference the product owner approves as an explicit rule, never a blanket ignore. Keep explained-but-unapproved ones as `pending`, and have a strict compare fail on them (`reference/parity.md`). ## DO NOT -- Add compat shims that re-create v7 APIs (`useCart` from a package, `invoke`, `RequestContext`, a fake `@decocms/tanstack`). What v7's packages did is site code now; copy it in and own it. +- Add compat shims that re-create v7 APIs (`useCart` from a package, `invoke`, `RequestContext`, a fake `@decocms/tanstack`). What v7's packages did is site code now; copy it in and own it. A site-local `invoke` object with v7's paths, or a `runInvoke("site/loaders/…")` dispatcher, is the same shim. - Strip `"use client"` from components that need hooks, events or browser APIs to make them render. - Rename saved type names. Content stores them; keep the v7 name as an alias (`/next/renames-and-migrations#rename-a-type-with-an-alias`). - Hand-edit the script's output in the same commit as the run. @@ -92,5 +105,7 @@ Then compare the migrated site with the v7 one page by page (a parity harness: S - `reference/import-map.md`: every v7 import and its v8 replacement. - `reference/legacy-names.md`: the alias table and the seven rewritten names. - `reference/gotchas.md`: what the site migrations taught. -- `reference/parity.md`: approved v7 → v8 differences. +- `reference/parity.md`: how to approve differences, the pending state, editor-form causes, approved differences so far. +- `reference/native-apps.md`: apps that bundle content and render natively. +- `reference/faststore.md`: FastStore storefronts that aren't ejected (vendored runtime, content-client patch, drafts on Pages Router SSG). - `scripts/`: the migration (`main.ts` entry). Its tests run with the repo's `bun run test`. diff --git a/.agents/skills/deco-v7-to-v8-migration/reference/faststore.md b/.agents/skills/deco-v7-to-v8-migration/reference/faststore.md new file mode 100644 index 000000000..ac6542831 --- /dev/null +++ b/.agents/skills/deco-v7-to-v8-migration/reference/faststore.md @@ -0,0 +1,40 @@ +# FastStore storefronts that aren't ejected + +Learned on a VTEX FastStore storefront (`@faststore/cli` 4.x, Next.js 16 Pages Router, private) whose v7 integration monkey-patched FastStore's content requests. FastStore owns the Next.js app (it generates it into `.faststore/` at build) and only compiles some of the store's folders, so there is no place for a framework guide's `createCMS` page. v8 keeps the same seam; only what sits behind it changes. The patches are site-owned glue: say so in the code and the README. + +## The seam + +- **Vendored runtime.** FastStore won't compile `@decocms/blocks` or `.deco/index.ts` from the store, so bundle them (esbuild, CommonJS) into a file under a folder FastStore copies (e.g. `src/server/vendor/deco-cms.cjs`), exporting `createCMS`, `parseDraftPointer` and the block map. Build it in `dev`, `test` and `build`, commit it, and build the committed copy from the **published** package: a bundle built from a linked checkout ships unpublished code. Say in the PR which version it was built from. +- **Content module.** Bundle what `deco content` writes (the blocks, revision and alias table) as JSON next to the runtime, and import it statically so the server build traces it. Don't read `.deco/` from disk at runtime: FastStore's deploy doesn't ship it. +- **Content client patch.** Patch the VTEX Content Platform client FastStore's content service calls (`@vtex/client-cp`) so it answers from `cms.forRelease()` / `cms.forDraft(pointer)`, in FastStore's `{ $componentKey, data, name }` section shape. Section blocks return a descriptor (`{ component, props }`) and FastStore renders the component. +- **Schema-derived defaults by block map, not by name.** A rule that reads a section's schema (a `default` that forces server rendering, say) must map `deco schema`'s `base64()@Props` keys to sections through the block map's registrations. Parsing a v7 alias (`site/sections/X.tsx`) works only until the aliases go, and then every section silently changes behavior. + +## Drafts on Pages Router SSG pages + +FastStore's CMS pages are SSG and its CLI forbids `middleware`/`proxy` and `/api` routes, so the only point where a request can turn a static page into a per-request `getStaticProps` is Next's `tryGetPreviewData` (a non-`false` return is draft mode: no SSG cache, `Cache-Control: private, no-cache, no-store`, the value as `context.previewData`). + +- **Patch the compiled runtimes, not `next/dist/server`.** Next 16 serves pages from `next/dist/compiled/next-server/pages.runtime.{dev,prod}.js`, and each inlines its own `tryGetPreviewData`. Replacing `next/dist/server/api-utils/node/try-get-preview-data` in `require.cache` reaches neither, so drafts never run, in dev or production. Add one statement after the function's per-request memo in both runtimes (webpack builds; FastStore builds with `--webpack`, so the turbo runtimes are unused) that calls a global hook and returns its answer unless it's `undefined`. A `patch-package` file of a minified runtime is hundreds of KB; an idempotent install/build script that fails when the anchor is missing is reviewable. Re-check it on every Next upgrade. +- **The hook.** Answer synchronously (Next calls it twice per request, from the route's `prepare` and from the renderer, so memoize on the request). Only a `?__draft=` or draft cookie that `parseDraftPointer` accepts is a signal: junk on a production host must not switch a page to per-request rendering. `cms.draftPointer`/`cms.draftCookie` are async, so return a token as preview data, let the content patch await the CMS's answer for it, and remember hosts the CMS refused so they go back to the SSG cache. +- **`x-nextjs-cache: HIT` is no evidence.** Next stamps it on draft-mode responses too. The draft ran when the response is `Cache-Control: private, no-cache, no-store, …` and shows the draft's content. +- **Password-protected preview domains.** FastStore protects `*.vtex.app` with a password gate that runs before the page. A store that patched it to let the Studio preview through must read the draft cookie (`__deco_draft`, the same name in v7 and v8), and accept only a value shaped like a pointer, never any `?__draft=`. +- **Prove drafts end to end; parity can't.** If the v7 draft path was dead (a `require.cache` patch is), the v7 baseline recorded the published page for the draft cases and a dead v8 path passes the compare. Probe the dev server and a production build (`next start`) by hand: `?__draft=` on an allowed host shows the draft and sets the cookie; the cookie alone keeps the next page in the draft; a host outside `preview.hosts` and `?__draft=` get the published page; `?__draft=off` expires the cookie. Then let the draft cases differ from the baseline and list that for approval. + +## Preview hosts and draft sources + +- `preview.hosts` in `CMS.json` must list the dev host (`localhost`) as well as the store's hosts, within the code cap in `createCMS({ preview })`. v8 ignores v7's `DECO_ALLOWED_PREVIEW_HOSTS`: remove it from env files and harness env too. +- Drafts need no site loader: `cms.forDraft` fetches the pointer itself, only from v7's preview API domains (`*.decocms.com` and the loopback hosts by default; `DECO_PREVIEW_API_DOMAINS` replaces the list, as in v7). Delete any draft-fetching loader the site kept from v7. +- If the site keeps v7's policy that a draft that fails to load falls back to the release, write that down: the docs say a failed draft is an error. + +## Content imported from FastStore's CMS + +- Pages imported from FastStore's Headless CMS carry FastStore's settings (`__settings`) next to the SEO object the editor's form edits (`seo`). Merge the form's `seo` over the imported one, or an SEO edit never renders. +- Don't route on undeclared `__`-prefixed metadata (`__cms.contentType`). The docs say the site editor keeps only declared fields, and whether `_`-keys survive a save isn't documented. Encode what routing needs in a declared field (the template's synthetic `path`, say) or declare it. +- The store's build runs `deco check`, so an importer that writes content check rejects breaks the next deploy. Make the importer apply the same cleanup rules as the migration's content commit, and run `deco check` at its end. + +## Saved-block names that differ only by letter case + +A case-insensitive disk (macOS) checks out one of the two files, so v7 silently bundled one of them on macOS while Linux had both. Before migrating, find which one content references (grep every saved block for both names), compare the two, and keep the referenced one; record the decision in its own commit. Make the build fail on such a pair, reading the git index too (the disk can't show it). v8's write path refuses a new name that differs from an existing one only by case, but `deco content` and `deco check` don't flag a pair that is already there. + +## Global sections with per-URL matchers + +v7 resolved a store-wide "global sections" list per request, so a matcher in it could show a section on some URLs only. In v8 a matcher reads the request only through the site's own request-scoped storage (`/next/blocks#reading-the-request`), and FastStore fetches global sections apart from the page, for SSG pages at build time. An adapter that resolves the list once per revision drops per-URL matching. If content uses it, keep it (resolve the list per page path, passing the path to the matcher through the site's storage); if it doesn't, list the loss for the product owner anyway, since editors could rely on it. diff --git a/.agents/skills/deco-v7-to-v8-migration/reference/gotchas.md b/.agents/skills/deco-v7-to-v8-migration/reference/gotchas.md index 5dfb86f52..eea60af53 100644 --- a/.agents/skills/deco-v7-to-v8-migration/reference/gotchas.md +++ b/.agents/skills/deco-v7-to-v8-migration/reference/gotchas.md @@ -1,19 +1,23 @@ # Gotchas from the site migrations -Learned on `deco-sites/storefront-tanstack` (Shopify) and `deco-sites/blog-tanstack`. Commit hashes refer to those repos. +Learned on `deco-sites/storefront-tanstack` (Shopify), `deco-sites/blog-tanstack`, a Next.js storefront on VTEX, a TanStack Start storefront on VTEX and a non-ejected FastStore storefront on VTEX (the last three private, so no hashes; FastStore specifics are in `faststore.md`). Commit hashes refer to the two public repos. ## Before you run the script - **Install `@decocms/blocks@^8.1` first** (`^8` resolves to the accidental v7 build published as `8.0.0`). A v7 site already has `@decocms/blocks@7`, which has no `/cli` or `/protocol/keys`; the script fails to import. Keep v7 `@decocms/apps-*` installed until after the run: vendoring copies from them. - **Commit the script's output unedited** (`046de4d`, `de03793`), then do the manual work in follow-up commits. +- **The script vendors every app loader the content calls**, including ones the v7 site never ran (a v7 Next.js site resolved many VTEX loader blocks to `null`). Check which ones actually rendered before keeping a vendored copy; delete the rest along with the saved blocks nothing references. ## Rendering -- **Lazy wrappers render normally.** v7's `website/sections/Rendering/Lazy.tsx` (and `Deferred`/`SingleDeferred`) is registered as a block that renders its section. Blocks resolve on the server, so content that used to arrive after hydration (JSON-LD in particular) is now in the first SSR HTML. That is an approved parity difference, not a bug (`5a5a4d4`). +- **v8 has no async rendering; the script removes it.** v7's `website/sections/Rendering/Lazy.tsx` and `SingleDeferred.tsx` (`{ section }`) and `Deferred.tsx` (`{ sections: [...] }`) are unwrapped in the saved content: each wrapper becomes the section(s) it held, with their props, and its own options (`loading`, `display`, `behavior`) go. Don't register a block under those names. Content that used to arrive after hydration (JSON-LD in particular) is now in the first SSR HTML. That is an approved parity difference, not a bug (`5a5a4d4`). A wrapper the script reports (several sections where one block goes) is unwrapped by hand. - **Blocks that render nothing** (theme fonts, SEO sections' `jsonLD`): wrap their data arguments in `lazy` in content so nothing is fetched for them (`9a097a1`). - **Awaited blocks are an exception.** The TanStack guide keeps block promises unawaited. A site whose blocks only read in-memory content may await them so every section lands in the first HTML chunk and the hero preload stays in ``; stop awaiting a block once it fetches upstream (blog `dee30a5`). +- **Awaited because of response headers.** A page whose response headers depend on how its blocks resolved (a degraded-page header when a loader failed, an A/B cookie with the groups the matchers picked) has to await every block before answering, so time to first byte is the slowest block's, as in v7. That departs from the guide: say so in `open-page.server.ts`'s header and in the PR, and list streaming as a follow-up. +- **Side effects of deferred sections now start on load.** v7 rendered a deferred section once its placeholder came within 300px of the viewport, one frame after mount, so a popup, banner, timer or view event inside it started then; now it starts on page load. Gating it on the section's position in v8's layout does *not* reproduce v7: v7's pages were short while their other deferred sections were still placeholders, so a footer near the fold of a short v7 layout (a product page) mounted on load, and the gate then hides a popup v7 showed. Compare a page whose v7 layout was short as well as a long one before keeping such a gate; otherwise keep the difference as pending and explain it. - **Section keys and wrappers.** Keep v7's `
` wrapper and section keys so CSS and client state survive navigations (`5bcc12b`, `0512e60`). -- **Tailwind.** Code moved out of the v7 packages changes Tailwind's scan; list the classes v7 generated from deleted files inline so the stylesheet stays byte-identical. +- **Tailwind reads every text file it scans, comments and Markdown included.** A word that is a utility name (`isolate`, `hidden`, `fixed`) in a new comment or an `AGENTS.md` line generates that class and changes the stylesheet's hash, which shows up as a different `Link` preload header on every page. Diff the built CSS against v7's after editing docs or comments. +- **Tailwind.** Code moved out of the v7 packages changes Tailwind's scan; list the classes v7 generated from deleted files inline so the stylesheet stays byte-identical. If you exclude the vendored folder from the scan (`@source not "../vendor"`, because v7 never scanned `node_modules`), a class written only there is never generated: say so in the site's agent/contributor notes. ## Framework code moves into the site @@ -26,20 +30,57 @@ v8 has no framework binding and its apps are thin clients. What `@decocms/tansta - **Cookies and request state**: pass the request and response headers explicitly, or use a site-owned `AsyncLocalStorage`, instead of v7's `RequestContext`. - **Cart, user, wishlist, sign-in flows and commerce loaders**: vendored next to the loaders the script copied, sending through the v8 client. - **Section loaders** run inside their block functions. +- **Module-level config from v7 apps is cross-request state.** A v7 app's `configureX({...})` singleton (the website app's SEO defaults, for one) called from the page loader on every request is shared by every request in the isolate. A preview host and production share the worker, so a draft's values can land in a concurrent release render that the edge then caches. Put per-page values in the page's own state (the `AsyncLocalStorage` the page loader runs blocks in) and pass them to whatever read the singleton. +- **Identification headers in the copied worker entry** (`x-powered-by`, the outbound `User-Agent`) are the site's own values now, not the framework's. If they change (v7 sent its package version), list it as the site's difference. +- **Keep v7's platform client, or adopt the v8 one?** Keeping the vendored v7 client (and its cached loaders and fetch cache) keeps every upstream request identical, so recorded upstream fixtures replay. If you keep it, drop the v8 `@decocms/apps-` dependency nothing imports, and list the move to its client plus the `/next/caching` recipe (`cachedLoader` is gone in v8) as a follow-up. Prune what the copied worker entry still carries for v7's admin (preview shells, `/live/previews`, `/deco/render`). +- **Error reporting.** v7's `onResolveError` hook sent every failed section or loader to the site's exception reporting. v8 has no hook: `client.resolve` returns `[value, error]`, and a section loader that throws is caught in its block function. Report both explicitly there (same attributes as before, so dashboards keep working); `console.error` alone only reaches the SDK's sampled log export. ## `/deco/invoke` → server functions v8 has no invoke endpoint. Every call the browser made through `/deco/invoke` becomes a TanStack Start server function (`createServerFn`, e.g. `src/server/site.functions.ts`) or a Next server action/route handler, taking the request and response headers explicitly (`89895c3`). Validate inputs with zod; reject protocol-relative hrefs. Loaders that 404'd through invoke and fell back to placeholders now return real state: check those flows. +- **One server function per key, imported by the call site.** Name it after the key (`site/loaders/gateway/wishlist/add-item` → `$siteLoadersGatewayWishlistAddItem`) and rewrite each call: `invoke.a.b(props)` → `$aB({ data: props })` (a script over the call sites does most of it; chains split across lines and bracket keys like `["can-spin"]` are the usual misses). The server function imports its handler module and calls it with the props and the request, then forwards the cookies the handler set. Never a `runInvoke(key)` lookup over the commerce-loader map: that is v7's endpoint again. +- **v7's invoke passed no app context.** It called `handler(body, request)`. A site loader that reads its third argument (account, secrets) worked when a page ran it (the block map passed them) and threw through invoke. Give its server function the context the block map passes. +- **Keys v7 had no handler for** failed with "handler not found". Keep them failing explicitly at the call site (a rejected promise with a comment saying v7 failed too), not as stubs in a rebuilt `invoke` tree, and list them as follow-ups. +- **Check the client bundle.** TanStack Start's compiler replaces each handler with an RPC stub and drops its imports; grep `dist/client` JavaScript (not source maps) for secret env names and `node:async_hooks` after the build. +- v7's generated `invoke.gen.ts` (cart and session actions) is site code now: export each action as its own server function and delete the ones nothing calls. + ## Caching and privacy - **Keep per-visitor data out of the edge cache** (`844ce9f`): cache a GET server function only when its payload is page data; cart, user, wishlist, addresses answer `Cache-Control: private, no-store`, and the cache wrapper never stores such a response. Detect signed-in shoppers by the cookie the sign-in flow actually sets. - **One QueryClient per router**, never per isolate: a shared one leaks the previous request's cart and user. - v7's admin paths (`/deco/*`, `/live/*`, `/.decofile`) should answer 404, not a cacheable page. -- Read drafts with `draftPointer()`. +- Read drafts with `await cms.draftPointer(request)`; it ignores drafts on hosts outside `preview.hosts` (the `CMS` block, capped by `createCMS({ preview })`), so list your staging and dev hosts there. +- **Gate the copied worker entry's draft bypass on the pointer too.** v7's edge cache skipped a request with `?__draft=` or the draft cookie only on an allowed preview host. A copy that checks the raw param or cookie lets anyone skip the edge cache on production (`?__draft=x`), so every such request reaches the worker and the commerce API. Compute `const previewing = (await cms.draftPointer(request)) !== null` once per request and use it wherever the copy checked for a draft. +- **Don't let hydration wait on a cross-origin call.** Prefetching the shopper's profile in the router's `hydrate` so widgets hydrate with it keeps v7's pixels, but `hydrate` is awaited: a slow or hung gateway keeps the page from ever becoming interactive. Cap the wait (a few hundred milliseconds) or don't await it. A parity harness replays upstream calls, so it never shows this cost. + +## Next.js App Router + +- **`transpilePackages: ['@decocms/blocks']`** while the published package ships `.ts` source (see SKILL.md, step 1). A linked local checkout ships `dist/` and hides this; so does every parity run made against it. +- **Drafts on Pages Router SSG pages** (no proxy available, as on FastStore): Next 16's compiled pages runtimes inline `tryGetPreviewData`, so a `require.cache` patch never runs. See `faststore.md`. +- **Drafts on static pages.** `force-static`/ISR pages get stubbed `cookies()`, so they can't read the draft pointer. Divert drafted requests (`?__draft=` or the draft cookie) in `proxy.ts` onto a dynamic route group that binds the pointer, and 404 direct hits on that internal route. Send `Cache-Control: no-store, private` and `X-Robots-Tag: noindex` on both signals. +- **List pages once per revision.** `list('page')` expands every page's saved-block references; on a site with hundreds of pages that is tens of milliseconds of CPU. Keep the routable list per revision (a revision never changes, a draft has its own) and hand the same array to `matchRoute` so it reuses its lookup. Health and readiness probes go through the same cache, never a fresh `list`. +- **Head meta order.** If v7 emitted `theme-color`/`color-scheme` after the root layout's metas on some routes, that was a race with Next's viewport resolution. To keep the order deterministic, render those tags from the segment layout (React hoists in tree order) rather than approving a reorder. +- **Jest** is CJS-only: transform `@decocms/blocks` (ESM) with ts-jest and keep it out of `transformIgnorePatterns`. When mocking a site module in a test, spread `jest.requireActual` so sibling exports other code imports stay real. + +## Hosted releases and bundled content + +**v7 fast deploy becomes the hosted CMS.** A v7 TanStack site that published content without a deploy (`DECO_FAST_DEPLOY` plus a `DECO_KV` namespace) needs `site` and `token` on `createCMS` (`DECO_SITE` and `DECO_SITE_TOKEN` secrets) for the same feature; without them every content change needs a deploy. Put the two `wrangler secret put` commands in the deploy checklist. v7's fast deploy also kept the content out of the bundle; v8 bundles it, and hosted mode loads a second copy. On a site with ~300 saved blocks (11 MB of JSON), Node measured about 13 MB per parsed copy, plus the bundled module's source (about the size of the JSON) and a transient string up to twice the JSON while a snapshot parses: under the Workers 128 MB limit, but measure memory and cold start with `wrangler tail` on a preview before turning it on. + +With `site` and `token` set on `createCMS`, a published release goes live without a deploy, but only for what reads through the CMS client. Before turning it on, grep for direct imports of `.deco/blocks/*.json` (redirects, proxy tables, config blocks) and for build-time scripts that read `.deco/blocks`: those stay on the last deploy. v7 behaved the same, so it isn't a parity break; route them through the client or document the limitation and keep hosted releases off until you do. + +## v7 leftovers to delete + +- Env vars and comments for the v7 admin protocol (`DANGEROUSLY_ALLOW_PUBLIC_ACCESS`, admin public keys) and for settings that lived in the v7 `site` app block. +- Middleware/proxy exclusions for `/deco`, `/live` and `/.decofile`: those routes are gone, and keeping them special only hides that they now 404. +- Add the new env vars (`DECO_SITE`, `DECO_SITE_TOKEN`) to the site's env example, marked secret, and blank them in parity runs. +- **Editor settings that disappear with the `site` block.** Deleting the v7 `site` app block (apps are code now) also deletes any setting editors changed there, such as draft preview hosts. Moving it to code or env is fine, but it is an editor-visible change: list it for the product owner. ## Content +- **`deco check` in the build gates the deploy.** Once `build` runs it, a Studio save or a content importer that writes something check rejects fails the next deploy. Make importers emit check-clean content (and run `deco check` at their end), and say in the README that the gate exists. +- **Don't let rendering depend on undeclared `__`-prefixed fields.** `deco check` accepts `_`/`$`/`@` keys, but the docs say the site editor keeps only declared fields, and whether those keys survive a save isn't documented. Routing data (a page type) belongs in a declared field. +- **Case-only name pairs.** Two saved blocks whose names differ only by letter case can't coexist on macOS disks; v7 silently bundled one. Check which one content references, keep it, and fail the build on such a pair (reading the git index too). v8's write path refuses new ones; `deco content`/`deco check` don't flag an existing pair. - `deco check` rejects fields no type declares, `.tsx`-named preview blocks and dangling references: delete or type them (`9a097a1`, `ffa866a`). - v7 app blocks (`deco-shopify`, `deco-blog`, `site`) go: apps are code now, configured from env (e.g. `SHOPIFY_STORE_NAME`). Don't re-encrypt secrets runtime code never reads. - `website/functions/requestToParam.ts` in a string field becomes the page's route `param` (`/:slug`) the block reads itself. @@ -54,9 +95,12 @@ v8 ships no dev hook for content changes; two pieces of template code make a sav ## Telemetry and analytics -- Telemetry: set `OTEL_EXPORTER_OTLP_ENDPOINT` to the OTLP collector and the auth header as the `OTEL_EXPORTER_OTLP_HEADERS` secret in production; send nothing in dev or parity runs (`89eae7d`). -- Analytics: the built-in `analytics` block plus `AnalyticsScript`/`track` replace OneDollarStats; it sends collector beacons directly instead of loading the SDK (`4a50b13`, blog `9e97455`). +- Telemetry: set `OTEL_EXPORTER_OTLP_ENDPOINT` to the OTLP collector and the auth header as the `OTEL_EXPORTER_OTLP_HEADERS` secret in production; send nothing in dev or parity runs (`89eae7d`). v7's `DECO_OTEL_{METRICS,LOGS,TRACES}_ENDPOINT`, `DECO_OTEL_HEADERS` and `DECO_OTEL_AUTH_TOKEN` keep working as aliases (the standard name wins when both are set), so a site's existing variables don't need renaming. +- `service.version` comes from `DECO_COMMIT_SHA` (or another host's commit variable). Workers have none at runtime: set it from the build's commit (a Vite `define`) before `createCMS`. v7 read the deployment id from the `CF_VERSION_METADATA` binding. +- **v7 Workers bindings with no v8 writer.** v7's `instrumentWorker` wrote per-route request counts, edge cache hit/miss and request duration to an Analytics Engine dataset (`DECO_METRICS`) and read `CF_VERSION_METADATA`; v8 has no metric API for them. Don't keep the bindings as dead config, and don't drop them silently: list each by name (and the edge layer's `deco.cache.requests` counter) as a product decision in the PR, since the dashboards built on them go dark. +- Analytics: `AnalyticsScript` with `(await cms.settings()).analytics` plus `track` replace OneDollarStats (the collector lives in the `analytics` section of `CMS.json`); it sends collector beacons directly instead of loading the SDK (`4a50b13`, blog `9e97455`). ## Dependencies - `bun.lock`: don't commit a lock produced by `bun link`; regenerate once the v8 packages resolve from npm. +- A linked checkout writes `"version": "0.0.0"` into `.deco/schema.gen.json`; the published CLI writes its own version. Commit what a clean `install --frozen-lockfile && build` writes. diff --git a/.agents/skills/deco-v7-to-v8-migration/reference/import-map.md b/.agents/skills/deco-v7-to-v8-migration/reference/import-map.md index dbfae4a99..e1517aeda 100644 --- a/.agents/skills/deco-v7-to-v8-migration/reference/import-map.md +++ b/.agents/skills/deco-v7-to-v8-migration/reference/import-map.md @@ -11,7 +11,7 @@ What `scripts/imports.ts` does with each v7 `@decocms/*` import in the site's `s ## Left alone (already the v8 API) -`@decocms/blocks` (the root's v8 names: `createCMS`, `matchRoute`, `remoteLoader`, `draftPointer`, `Blocks`, `Lazy`, `Seo`, …), `@decocms/blocks/fetch`, `/analytics`, `/secrets`, `/cli`. +`@decocms/blocks` (the root's v8 names: `createCMS`, `matchRoute`, `remoteLoader`, `parseDraftPointer`, `Blocks`, `Lazy`, `Seo`, …; drafts are read with `cms.draftPointer`), `@decocms/blocks/fetch`, `/analytics`, `/secrets`, `/cli`. ## Reported, with where the replacement lives (first match wins) @@ -19,14 +19,14 @@ What `scripts/imports.ts` does with each v7 `@decocms/*` import in the site's `s |---|---| | `@decocms/start`, `@decocms/start/*` | a 6.x import: upgrade the site to 7.x first | | `@decocms/blocks/sdk/cachedLoader` | gone: an upstream cache is your own `fetch`, passed to `createInstrumentedFetch` (`/next/caching#upstream-data`) | -| `…/invoke`, `createInvoke`, `@decocms/blocks/sdk/invoke` | `/deco/invoke` is gone: call upstream clients from server functions or route handlers (`/next/renames-and-migrations#loaders-actions-and-invoke`) | +| `…/invoke`, `createInvoke`, `@decocms/blocks/sdk/invoke` | `/deco/invoke` is gone: one server function per loader or action, imported by each call site, never a rebuilt invoke tree (`/next/renames-and-migrations#loaders-actions-and-invoke`) | | `@decocms/blocks-admin/*` | v8 sites serve no admin endpoints; the site editor uses the content protocol (`/next/studio-compatibility`) | | `@decocms/blocks-cli/*` | codegen is the `deco` CLI in `@decocms/blocks`: `deco schema`, `deco content`, `deco check` (`/next/cli`) | | `@decocms/blocks/setup`, `@decocms/blocks/cms/*` | `createCMS` from `@decocms/blocks`, with the block map in `.deco/index.ts` (`/next/content#create-the-cms`) | | `@decocms/blocks/sdk/instrumentedFetch` (other names) | `createInstrumentedFetch({ provider, fetch?, retry?, circuitBreaker? })` from `@decocms/blocks/fetch` | | `@decocms/blocks/sdk/otel`, `/sdk/observability`, `/sdk/logger`, `/middleware/observability` | the `telemetry` option of `createCMS` (`/next/telemetry`) | | `@decocms/blocks/types/widgets` | type the field as a `string` with a `@format` tag (`/next/schema#widgets`) | -| `…/Analytics`, `…/OneDollarStats`, `@decocms/blocks/sdk/analytics` | `AnalyticsScript` and `track` from `@decocms/blocks/analytics`, with the built-in `analytics` block (`/next/analytics`) | +| `…/Analytics`, `…/OneDollarStats`, `@decocms/blocks/sdk/analytics` | `AnalyticsScript` and `track` from `@decocms/blocks/analytics`, with the `analytics` section of `cms.settings()` (`/next/analytics`) | | `@decocms/apps-salesforce`, `@decocms/apps/salesforce` | `@decocms/apps-sfmc-personalization` | | `@decocms/apps-commerce`, `@decocms/apps/commerce` | converters, hooks and shared commerce types live in your platform template (copy them into the site) | | `@decocms/apps-website`, `@decocms/apps/website` | SEO, sitemaps, redirects live in your platform template | diff --git a/.agents/skills/deco-v7-to-v8-migration/reference/native-apps.md b/.agents/skills/deco-v7-to-v8-migration/reference/native-apps.md new file mode 100644 index 000000000..3a85e45fc --- /dev/null +++ b/.agents/skills/deco-v7-to-v8-migration/reference/native-apps.md @@ -0,0 +1,43 @@ +# Apps that render content natively + +Some v7 projects aren't websites. They're native or mobile apps (a v7 binding such as `@decocms/eitri`) that bundle `.deco/blocks/*.json` at build time and render sections through their own component table. For these apps, the v7 package only generated Studio's files (`.deco/meta.gen.json`, `.deco/blocks.gen.json`). No runtime file imports a `@decocms/*` package. + +v8 has no binding for them, and needs none. The app keeps reading its JSON. Only the editor tooling moves to `@decocms/blocks` v8. Keep app-specific code in the app; don't add it to a package. + +## What changes + +1. **Run the script as usual** and commit its output unedited. It writes `.deco/index.ts` with descriptor-returning section blocks, which `deco schema` and `deco check` need. Manual steps 2 to 4 (`createCMS`, framework code, `/deco/invoke`) don't apply. +2. **Dependencies.** + - Remove the v7 binding package. + - Add `@decocms/blocks@^8.1` and `typescript`, which `deco schema` needs as a peer. + - Check the Node version that `@decocms/blocks`'s `engines` requires against the app's toolchain. +3. **Scripts.** + - These apps usually have no `dev`/`build` npm scripts. Run `deco schema && deco check` in whatever runs before the app's own start (`prestart`, a CI step). + - Keep the old v7 script names as aliases when other tooling, such as a content importer, calls them. +4. **Generated files.** + - Commit `.deco/schema.gen.json`, which Studio reads for the editor forms, and delete `meta.gen.json`, `blocks.gen.json` and `generate.digests.json`. + - Skip `deco content` when the app imports the JSON directly, and gitignore `.deco/blocks.gen.ts`. +5. **The block map isn't auto-discovered.** v7 listed every file under `sections/` as a section. In v8, a section that isn't in `.deco/index.ts` has no editor form. Say so in the app's AGENTS.md or contributor docs: a new section goes in the block map **and** in the app's component table. + - Remove files that only hold shared types (`types.ts`) from the block map. They were never sections. +6. **Studio.** + - `deco serve` serves the editor, from Studio at any origin (local or hosted). + - Expect an empty preview pane: there's no web renderer. Ignore the dev-server URL that `deco serve` prints, which is a default. + - A save rewrites the whole block file as formatted JSON, so compact inline objects become multi-line. Before the first editor session, commit a one-time reformat (`deco serve`'s formatting) so later diffs show only the edited fields. + +## Parity + +There's no SSR HTML to compare. A static harness covers what changes (Tier 0, no device or network needed): + +| Check | What it compares | +|---|---| +| bundle | sha256 of every file the app's bundler packs, per module, minus the Deco tooling files. Any change fails. | +| content | canonical-JSON hash of each `.deco/blocks/*.json`. | +| forms | each section's editor form, from v7 `meta.gen.json` vs v8 `schema.gen.json`, reduced to form keywords (`type`, `title`, `format`, `enum`, `required`, `items`, `properties`, `anyOf`/`oneOf`/`allOf`, `default`, `description`, min/max, `additionalProperties`). | + +Record the baseline on the last v7 commit, before the script runs. Add device screenshots (Tier 1) once a device or emulator build is available. Until then, rendering parity rests on the bundle check showing the packed files didn't change: say that in the PR. + +Most form differences are expected. Sort them with `reference/parity.md` ("Editor-form differences") before you approve or fix anything. + +## Before merging + +`prestart` regenerates `schema.gen.json` from the **installed** `@decocms/blocks`. Pin a range that includes every CLI fix the committed schema depends on. Otherwise the next start brings the old form differences back. The form-fidelity fixes below shipped after `8.1.0-next.1`. diff --git a/.agents/skills/deco-v7-to-v8-migration/reference/parity.md b/.agents/skills/deco-v7-to-v8-migration/reference/parity.md index d7bad2970..b008d8cbd 100644 --- a/.agents/skills/deco-v7-to-v8-migration/reference/parity.md +++ b/.agents/skills/deco-v7-to-v8-migration/reference/parity.md @@ -6,11 +6,54 @@ Compare the migrated site with the v7 one with a parity harness (SSR HTML, hydra Add it to an `approvedDifferences` list (e.g. in `parity/pages.json`), applied at compare time to both snapshots. Each rule names **one snapshot field**, **the cases it covers**, and **exactly what the new value must be**, so any other change to that field still fails. List the rules applied per case in the summary. Never ignore a field wholesale. +### Pending approval + +A difference you've explained but the product owner hasn't approved yet still needs a state. Otherwise it's either a failing diff or an approval nobody signed. List it like an approval, with its reason, plus `"status": "pending"`: + +- the default compare matches the entry, prints it as `PENDING` on every run and reports `PASS 0 diffs (N pending approval)`; +- `compare --strict` fails until each one is approved, which means deleting its `status` field. Run it in CI or before merging. + +Never hide a pending entry: the compare summary and the PR list every one. + +## Running it + +- **Build against what ships.** Do the final compare on a build from the committed lockfile, not a linked local checkout: a published package can differ (source vs compiled output, older CLI). +- **ISR and prerendered routes.** Pages prerendered at build time with `revalidate` were stale when the baseline recorded them. Compare a build older than the longest `revalidate`, or those cases differ only in `x-nextjs-cache`/`x-nextjs-prerender`. +- **Flakes.** Rerun a single failing case (`--only `) before treating it as a regression, and avoid running two harnesses on one machine at once: font rasterization under load produced one-off weight diffs. Record each full run's flake count and which cases flipped; a count that grows between runs is a determinism regression to document or fix, not noise. +- **Pin the Workers `request.cf` object.** A local Workers runtime (Miniflare/wrangler) fills `request.cf` from a cache file fetched for the machine's network location (`node_modules/.mf/cf.json`), once per checkout. Two checkouts fetched at different times can disagree (region, city, colo), and anything keyed on the region (a cache segment, regional prices) differs. Copy one checkout's file into the other before recording, and record and compare on the same ports: URLs built from the origin can carry the port. +- **A bad baseline is re-recorded, never masked.** If the v7 baseline itself captured a rendering flake (an icon laid out but not painted), re-record that case from the v7 site (`record --only `) instead of adding a pixel-ignore rect. The re-record also refreshes that case's upstream fixtures live, so its page can change (prices, stock, page height); run a full compare afterwards. +- **Fix in code what code can fix, then prove it on every page type.** Before asking for approval, try removing a pending difference in site code; then run the full compare, not just the cases the rule named. A popup gated on the footer's position fixed the home page cases and broke the product page ones (`gotchas.md`, Rendering), so that rule stayed pending. +- **Not covered by pixels.** Name what the cases don't exercise (draft rendering through the pointer, coupons, region pricing, signed-in shoppers, device-only variants on per-request pages, hosted releases) in the PR so it gets a manual check. A replaying harness also can't see latency: a page that now waits on an upstream call before hydrating, or a draft check that no longer lets the edge cache serve, passes every compare. Review those code paths by hand. + +## Editor-form differences (v7 `meta.gen.json` → v8 `schema.gen.json`) + +Compare each section's form, not just the content. Sort each difference into one of three causes before you approve or fix it. + +**1. The v7 file was stale.** v7 committed `meta.gen.json`, and sections changed without anyone regenerating it. For each field, check the source's history after the file's last change (`git log .. --
`). If a field was added or removed there, v8 is right. Approve it with the commit as the reason. + +**2. v7 heuristics v8 drops on purpose.** Approve these, don't "fix" them: +- **"Select from saved" on every list.** v7 offered it on every array. v8 offers it only where a saved block fits the field's type. Check that no content uses it, and whether the renderer could even resolve a saved block there. +- **Every file under `sections/` as a section.** v7 listed helper files (shared `types.ts`) as sections. v8 lists only what the block map declares. +- **Dynamic option pickers.** v7's `@format dynamic-options` fields ran a site loader through `/deco/invoke` to suggest values as the editor typed. The v8 site editor never runs site code, so they become text fields (`/next/schema`). Make sure the field still accepts what an editor would type (a bare ID, a plain place name), update copy that promises suggestions, and list each field for approval. +- **Page form order and labels.** A site page type that `extends Route` and adds `seo`/`sections` lists its own fields first and labels `seo` as "Seo". Redeclare `name` and `path` (with their `@title`) before them and give `seo` an explicit `@title SEO` to keep v7's form. +- **The rich-text widget from a substring.** v7 gave the rich-text editor to any type whose name contained `RichText` (`PromoRichText`). v8 matches the alias name as a whole word. When editors rely on the toolbar, keep it with `/** @format rich-text */` on the alias rather than approving its loss. + +**A color picker from a substring: keep it, don't approve its loss.** v7 added the color widget whenever the type's *name* contained `Color`, even on a string-literal union such as a black/white choice. v8 doesn't guess from names: put `/** @format color */` on the alias (`type TextColor = "black" | "white"`) or on the field, and `deco schema` writes the same `format: "color"` v7 did, with the select's values kept (`/next/schema#widgets`). + +**3. CLI fidelity bugs: fix them in the CLI, don't approve them.** If your `@decocms/blocks` predates these fixes, you'll see: +- a literal union's dropdown in TypeScript's internal order instead of the source order; +- `@format datetime` without the date-time picker; +- `Record` offering "Select from saved" and every block type. + +Upgrade to a release that includes them rather than approving the differences. + +**Literal order depends on the v7 generator.** Check the site's own v7 `meta.gen.json` before calling an order a bug. Deno-era generators wrote a union's values in source order. `@decocms/blocks-cli` 7 (TanStack and Next.js sites) took the TypeScript checker's order, which is source order unless the same literals already appeared in another type (`'both' | 'desktop' | 'mobile'` came out `desktop, mobile, both`). On such a site the source-order fix reorders those dropdowns relative to v7. That's a stable, intended order, but it's still an editor-visible change, so list it for approval. + ## Approved so far storefront-tanstack (`5a5a4d4`): -- **JSON-LD in the first SSR HTML** for Lazy-wrapped sections (`ssr.jsonLd`): equal to the hydrated DOM's. Lazy wrappers now render on the server. +- **JSON-LD in the first SSR HTML** for Lazy-wrapped sections (`ssr.jsonLd`): equal to the hydrated DOM's. v8 has no async rendering: the script unwraps Lazy wrappers to their sections, which render on the server. - **Extra product image requests** (`thirdPartyRequests`/`harMisses`) on flows that render more images server-side. - **PLP "show more" page-2 `view_item_list` item indexes.** - **Fixed cache headers** where v7 sent the wrong ones (the v7 QueryClient bug on `home@mobile`). @@ -18,5 +61,14 @@ storefront-tanstack (`5a5a4d4`): blog-tanstack (`9e97455`): -- **Collector beacons recorded as analytics**: the built-in analytics block sends One Dollar Stats beacons straight to the collector instead of loading the SDK; decode them into the same view/event records, and compare `location.pathname` without the query string. +- **Collector beacons recorded as analytics**: `AnalyticsScript` sends One Dollar Stats beacons straight to the collector instead of loading the SDK; decode them into the same view/event records, and compare `location.pathname` without the query string. - **Real 404s** for unknown slugs and `/404` (v7 answered 200). + +A native app with a v7 binding (bundled JSON, native rendering; `reference/native-apps.md`), pending product-owner approval: + +- **Bundle and content unchanged.** No packed file and no saved block differs. +- **Editor forms.** The differences fall under the three causes above: + - stale v7 `meta.gen.json`: a shared action type's new field across 15 sections, plus one removed CTA field; + - list fields without "Select from saved"; + - two black/white text-color fields without the color picker: not approved (the product owner keeps the picker); annotate their alias `@format color`; + - `types.ts` no longer listed as a section. diff --git a/.agents/skills/deco-v7-to-v8-migration/scripts/asyncRendering.ts b/.agents/skills/deco-v7-to-v8-migration/scripts/asyncRendering.ts new file mode 100644 index 000000000..af9f8f574 --- /dev/null +++ b/.agents/skills/deco-v7-to-v8-migration/scripts/asyncRendering.ts @@ -0,0 +1,73 @@ +/** + * Async rendering: v7 could defer a section behind a wrapper block, rendered + * after the page (`website/sections/Rendering/Lazy.tsx`, its re-export + * `SingleDeferred.tsx`, both `{ section }`) or on a trigger + * (`Deferred.tsx`, `{ sections: [...] }`). The next major has no async + * rendering (spec: renames-and-migrations › Migrating from v7), so the + * migration unwraps them in saved content: each wrapper is replaced by the + * section(s) it held, props untouched, and the wrapper's own options + * (`loading`, `display`, `behavior`) go. In a list, a `Deferred` gives way to + * all its sections in place. Running it again changes nothing. + */ +import fs from "node:fs"; +import path from "node:path"; +import { serializeBlock } from "@decocms/blocks/protocol/keys"; +import type { Report } from "./report"; +import { isPlainObject, type JsonObject, readContent } from "./walk"; + +const WRAPPER = /^website\/sections\/Rendering\/(Lazy|SingleDeferred|Deferred)\.tsx?$/; + +/** The sections a wrapper held, or `null` when `value` isn't a wrapper. */ +function wrapped(value: unknown): unknown[] | null { + if (!isPlainObject(value) || typeof value.__resolveType !== "string") return null; + const kind = WRAPPER.exec(value.__resolveType)?.[1]; + if (kind === undefined) return null; + const inner = kind === "Deferred" ? value.sections : [value.section]; + return Array.isArray(inner) ? inner.filter(isPlainObject) : []; +} + +/** + * `value` with every wrapper unwrapped. Outside a list, a wrapper that doesn't + * hold exactly one section stays, and its type goes in `stuck`. + */ +function unwrap(value: unknown, stuck: Set): unknown { + if (Array.isArray(value)) { + return value.flatMap((item) => { + const inner = wrapped(item); + return inner === null ? [unwrap(item, stuck)] : (unwrap(inner, stuck) as unknown[]); + }); + } + if (!isPlainObject(value)) return value; + const inner = wrapped(value); + if (inner?.length === 1) return unwrap(inner[0], stuck); + if (inner !== null) stuck.add(value.__resolveType as string); // kept; its sections still unwrap + const out: JsonObject = {}; + for (const [key, child] of Object.entries(value)) out[key] = unwrap(child, stuck); + return out; +} + +export function unwrapAsyncRendering(root: string, report: Report): void { + const { blocks, files } = readContent(root); + for (const [entry, block] of Object.entries(blocks)) { + const stuck = new Set(); + const next = unwrap(block, stuck); + const before = serializeBlock(block); + const after = serializeBlock(next); + if (after !== before) { + fs.writeFileSync(path.join(root, ".deco", "blocks", files[entry]!), after); + report.done.push({ + step: "content", + subject: files[entry]!, + message: "unwrapped v7 async-rendering wrappers (Lazy/Deferred) to the sections they held", + }); + } + for (const type of stuck) { + report.manual.push({ + step: "content", + subject: `${type} (in ${files[entry]})`, + message: + "a v7 async-rendering wrapper holding no section or several where one block goes; the next major has no async rendering: replace it with its section(s) by hand", + }); + } + } +} diff --git a/.agents/skills/deco-v7-to-v8-migration/scripts/blockMap.ts b/.agents/skills/deco-v7-to-v8-migration/scripts/blockMap.ts index 895a7b932..4ec0214e5 100644 --- a/.agents/skills/deco-v7-to-v8-migration/scripts/blockMap.ts +++ b/.agents/skills/deco-v7-to-v8-migration/scripts/blockMap.ts @@ -47,7 +47,7 @@ const HINTS: [RegExp, string][] = [ ], [ /^website\/sections\/Rendering\/(Lazy|Deferred|SingleDeferred)\.tsx?$/, - "v7's Lazy section wrapper: register a block under this name that returns its `section` (it renders normally), or unwrap it in the content", + "a v7 async-rendering wrapper the migration couldn't unwrap (no section, or several where one block goes); the next major has no async rendering: replace it with its section(s) in the content", ], [ /\/sections\/Seo\//, diff --git a/.agents/skills/deco-v7-to-v8-migration/scripts/imports.ts b/.agents/skills/deco-v7-to-v8-migration/scripts/imports.ts index 82a43ddad..0e5e02792 100644 --- a/.agents/skills/deco-v7-to-v8-migration/scripts/imports.ts +++ b/.agents/skills/deco-v7-to-v8-migration/scripts/imports.ts @@ -30,11 +30,10 @@ export const V8_API: Record | "*"> = { "Client", "CMS", "CMSError", + "CMSSettings", "createCMS", - "DRAFT_COOKIE", "DraftPointer", - "draftCookie", - "draftPointer", + "EffectiveSettings", "formatDraftPointer", "Lazy", "ListOptions", @@ -44,6 +43,7 @@ export const V8_API: Record | "*"> = { "Page", "parseDraftPointer", "Redirect", + "RequestLike", "Result", "Route", "remoteLoader", @@ -70,7 +70,7 @@ const HINTS: [RegExp, string][] = [ ], [ /(^|\/)(invoke|createInvoke)$|\/sdk\/invoke/, - "/deco/invoke is gone: call upstream clients from server functions or route handlers (/next/renames-and-migrations#loaders-actions-and-invoke)", + "/deco/invoke is gone: one server function per loader or action, imported by each call site, never a rebuilt invoke tree (/next/renames-and-migrations#loaders-actions-and-invoke)", ], [ /^@decocms\/blocks-admin(\/|$)/, @@ -98,7 +98,7 @@ const HINTS: [RegExp, string][] = [ ], [ /(OneDollarStats|Analytics)$|^@decocms\/blocks\/sdk\/analytics$/, - "AnalyticsScript and track from @decocms/blocks/analytics, with the built-in analytics block (/next/analytics)", + "AnalyticsScript and track from @decocms/blocks/analytics, with the analytics section of cms.settings() (/next/analytics)", ], [ /^@decocms\/apps-salesforce(\/|$)|^@decocms\/apps\/salesforce(\/|$)/, @@ -122,6 +122,10 @@ const HINTS: [RegExp, string][] = [ ], ]; const DEFAULT_HINT = "no v8 equivalent"; +/** Prerelease root exports that became CMS methods. */ +const DRAFT_HELPERS = new Set(["draftPointer", "draftCookie", "DRAFT_COOKIE"]); +const DRAFT_HINT = + "draftPointer and draftCookie are the async cms.draftPointer and cms.draftCookie now (they check the preview hosts), and DRAFT_COOKIE is gone: pass cms.draftPointer the request, or { url, headers } (/next/api-reference#draft-pointers)"; const SOURCE = /\.(tsx?|mts|cts)$/; @@ -195,7 +199,10 @@ function isDocumented(specifier: string, names: string[]): boolean { return api === "*" || names.every((n) => api.has(n)); } -function hintFor(specifier: string): string { +function hintFor(specifier: string, names: Iterable): string { + if (specifier === "@decocms/blocks" && [...names].some((n) => DRAFT_HELPERS.has(n))) { + return DRAFT_HINT; + } return HINTS.find(([pattern]) => pattern.test(specifier))?.[1] ?? DEFAULT_HINT; } @@ -328,7 +335,7 @@ export function rewriteImports(root: string, report: Report, vendored: Map { ); // v7 content: a page with both sections, the shelf fed by the app loader, a - // v7 secret, a Lazy wrapper (no v8 equivalent) and v7's generated files. + // v7 secret, Lazy wrappers (no async rendering in v8) and v7's generated files. const crypto7 = await v7CryptoKey(); process.env.DECO_CRYPTO_KEY = crypto7.env; const keys = await keyPair(); @@ -165,8 +165,11 @@ beforeAll(async () => { sections: [ { __resolveType: "site/sections/Hero.tsx", title: "Welcome" }, { - __resolveType: "site/sections/Shelf.tsx", - items: { __resolveType: "acme/loaders/ItemList.ts", query: "shirt", count: 3 }, + __resolveType: "website/sections/Rendering/Lazy.tsx", + section: { + __resolveType: "site/sections/Shelf.tsx", + items: { __resolveType: "acme/loaders/ItemList.ts", query: "shirt", count: 3 }, + }, }, ], }); @@ -179,6 +182,8 @@ beforeAll(async () => { __resolveType: "website/sections/Rendering/Lazy.tsx", section: { __resolveType: "site/sections/Hero.tsx", title: "Later" }, }); + // A prerelease next-major settings block: deco check fails on it until it's folded. + write(root, ".deco/blocks/Telemetry.json", { __resolveType: "telemetry", errorSampleRate: 0.01 }); write(root, ".deco/meta.gen.json", { v7: true }); write(root, ".deco/sections.gen.ts", "export {};\n"); @@ -220,11 +225,17 @@ describe("migrate", () => { expect(fs.existsSync(path.join(root, "src/vendor/acme/loaders/Unused.ts"))).toBe(false); }); - it("reports a type with no v8 equivalent instead of registering it", () => { - expect(read(".deco/index.ts")).not.toContain("Rendering/Lazy"); - expect(manual("block map").join("\n")).toContain( - "website/sections/Rendering/Lazy.tsx (in Lazy%20Hero.json): v7's Lazy section wrapper", - ); + it("unwraps v7 Lazy wrappers to the section they held, in pages and saved blocks", () => { + expect(read(".deco/index.ts")).not.toContain("Rendering/"); + expect(JSON.parse(read(".deco/blocks/Lazy%20Hero.json"))).toEqual({ + __resolveType: "site/sections/Hero.tsx", + title: "Later", + }); + expect(JSON.parse(read(".deco/blocks/pages-home.json")).sections[1]).toEqual({ + __resolveType: "site/sections/Shelf.tsx", + items: { __resolveType: "acme/loaders/ItemList.ts", query: "shirt", count: 3 }, + }); + expect(manual("block map").join("\n")).not.toContain("Rendering/"); }); it("rewrites the imports that have a v8 equivalent and reports the rest", () => { @@ -259,19 +270,22 @@ describe("migrate", () => { expect(read(".deco/blocks/API_KEY.json")).not.toContain("s3cret-value"); }); - it("leaves a tree that passes deco schema, deco content and (after the reported fix) deco check", async () => { + it("folds settings into the CMS block and deletes the prerelease block", () => { + expect(fs.existsSync(path.join(root, ".deco/blocks/Telemetry.json"))).toBe(false); + expect(JSON.parse(read(".deco/blocks/CMS.json"))).toEqual({ + __resolveType: "cms-settings", + telemetry: { errorSampleRate: 0.01 }, + }); + expect(manual("settings").join("\n")).toContain("DECO_ALLOWED_PREVIEW_HOSTS"); + }); + + it("leaves a tree that passes deco schema, deco content and deco check", async () => { expect(await schema({ root, reporter: quiet })).toBe(0); expect(await content({ root, reporter: quiet })).toBe(0); - // The one reported item: the Lazy wrapper has no block. Drop that entry, as a person would. - expect(check({ root, reporter: quiet })).toBe(1); - expect(quiet.text.join("\n")).toContain( - 'unknown block type "website/sections/Rendering/Lazy.tsx"', - ); - fs.rmSync(path.join(root, ".deco/blocks/Lazy%20Hero.json")); - quiet.text.length = 0; expect(check({ root, reporter: quiet })).toBe(0); expect(quiet.text).toEqual([]); - }); + // deco schema runs the TypeScript compiler: seconds on a loaded machine. + }, 30_000); it("resolves the migrated content: v7 names through the aliases, the secret with the new key", async () => { await content({ root, reporter: quiet }); @@ -300,8 +314,11 @@ describe("migrate", () => { it("keeps an existing block map and copies on a second run", async () => { const before = read(".deco/index.ts"); + const settings = read(".deco/blocks/CMS.json"); const second = await migrate({ root }); expect(read(".deco/index.ts")).toBe(before); + expect(read(".deco/blocks/CMS.json")).toBe(settings); + expect(second.done.filter((n) => n.step === "settings")).toEqual([]); expect( second.manual.some( (n) => n.subject === ".deco/index.ts" && n.message.startsWith("already exists"), diff --git a/.agents/skills/deco-v7-to-v8-migration/scripts/migrate.ts b/.agents/skills/deco-v7-to-v8-migration/scripts/migrate.ts index 0db8100f5..2059d3964 100644 --- a/.agents/skills/deco-v7-to-v8-migration/scripts/migrate.ts +++ b/.agents/skills/deco-v7-to-v8-migration/scripts/migrate.ts @@ -3,8 +3,10 @@ * renames-and-migrations › Migrating from v7). The steps, in order: * * 1. content: saved blocks in `.deco/blocks`, v7 generated files removed, + * v7 async-rendering wrappers (Lazy, Deferred) unwrapped to their sections, * legacy type names outside the alias table renamed, and A/B tests keyed * on a random matcher get its name as `experiment`; + * then site settings folded into the `CMS` block (`.deco/blocks/CMS.json`); * 2. secrets: v7 secrets re-encrypted with `.deco/secrets.pub`; * 3. block map: `.deco/index.ts` with aliases under the v7 names, after * vendoring the app loaders and actions the content calls; @@ -16,6 +18,7 @@ */ import fs from "node:fs"; import path from "node:path"; +import { unwrapAsyncRendering } from "./asyncRendering"; import { writeBlockMap } from "./blockMap"; import { moveContent } from "./content"; import { copyExperimentIds } from "./experiments"; @@ -23,6 +26,7 @@ import { rewriteImports } from "./imports"; import { renameLegacyTypes } from "./legacyNames"; import { createReport, type Report } from "./report"; import { reencryptSecrets } from "./secrets"; +import { foldSiteSettings } from "./siteSettings"; interface MigrateOptions { /** The app root: the folder with the site's package.json. */ @@ -78,8 +82,10 @@ export async function migrate(options: MigrateOptions): Promise { } const report = createReport(); moveContent(root, report, { decofile: options.decofile }); + unwrapAsyncRendering(root, report); renameLegacyTypes(root, report); copyExperimentIds(root, report); + foldSiteSettings(root, report); await reencryptSecrets(root, report); const { vendored } = writeBlockMap(root, report); rewriteImports(root, report, vendored); diff --git a/.agents/skills/deco-v7-to-v8-migration/scripts/report.ts b/.agents/skills/deco-v7-to-v8-migration/scripts/report.ts index fa58219a1..a871c762c 100644 --- a/.agents/skills/deco-v7-to-v8-migration/scripts/report.ts +++ b/.agents/skills/deco-v7-to-v8-migration/scripts/report.ts @@ -2,7 +2,7 @@ * What a migration did and what it left for a person: every step adds to one * report, which the command prints at the end. */ -type Step = "content" | "secrets" | "block map" | "vendor" | "imports" | "scripts"; +type Step = "content" | "settings" | "secrets" | "block map" | "vendor" | "imports" | "scripts"; export interface Note { step: Step; diff --git a/.agents/skills/deco-v7-to-v8-migration/scripts/siteSettings.ts b/.agents/skills/deco-v7-to-v8-migration/scripts/siteSettings.ts new file mode 100644 index 000000000..3b90ca441 --- /dev/null +++ b/.agents/skills/deco-v7-to-v8-migration/scripts/siteSettings.ts @@ -0,0 +1,397 @@ +/** + * Site settings (spec: renames-and-migrations › Site settings): the next + * major keeps site-wide settings in one saved block, `CMS`, of the built-in + * type `cms-settings`. This step folds what it finds into + * `.deco/blocks/CMS.json`: + * + * - the v7 Site block's (`Site` or `site`) `previewHosts` → `preview.hosts`, + * each entry trimmed and lowercased as v7 compared it (the field moves; the + * rest of the Site block stays). An entry that isn't a host pattern is left + * out and reported. On TanStack Start, v7 also allowed the site's + * `.deco.site` and `.deco-cx.workers.dev` (from `DECO_SITE_NAME`): + * they're added when the name is found, and reported when it isn't; + * - a literal `collectorAddress` on v7's `OneDollarStats` component in + * `src/` → `analytics.collector`; + * - a prerelease `Telemetry` / `Analytics` saved block (type `telemetry` / + * `analytics`) → its section, field for field, variants included; the old + * file is deleted. + * + * What's already in `CMS.json` is kept; nothing is written when there's + * nothing to fold, and a second run changes nothing. Settings v7 read from + * the hosting environment are listed in the report. + */ +import fs from "node:fs"; +import path from "node:path"; +import { formatHostPattern, LEGACY_ALIASES, parseHostPattern } from "@decocms/blocks/cli"; +import { blockFileName, serializeBlock } from "@decocms/blocks/protocol/keys"; +import type { Report } from "./report"; +import { isPlainObject, type JsonObject, readContent } from "./walk"; + +const SETTINGS_BLOCK = "CMS"; +const SETTINGS_TYPE = "cms-settings"; +const SITE_BLOCKS = ["site", "Site"]; +/** Prerelease well-known blocks, by name, and the built-in type each had. */ +const PRERELEASE: { name: string; type: "telemetry" | "analytics" }[] = [ + { name: "Telemetry", type: "telemetry" }, + { name: "Analytics", type: "analytics" }, +]; +const SOURCE = /\.(tsx?|jsx?|mts|cts)$/; + +type Section = "preview" | "telemetry" | "analytics"; + +function isMultivariate(type: unknown): boolean { + return ( + type === "multivariate" || + (typeof type === "string" && + Object.hasOwn(LEGACY_ALIASES, type) && + LEGACY_ALIASES[type] === "multivariate") + ); +} + +/** + * A prerelease `telemetry`/`analytics` block as its section: its fields + * without `__resolveType`. A `multivariate` stays one, with each variant's + * value turned into a section the same way; anything else (a reference to + * another saved block) is kept as is. + */ +function asSection(node: unknown, type: string): unknown { + if (!isPlainObject(node)) return node; + if (node.__resolveType === type) { + const { __resolveType: _, ...fields } = node; + return fields; + } + if (isMultivariate(node.__resolveType) && Array.isArray(node.variants)) { + return { + ...node, + variants: node.variants.map((variant) => { + if (!isPlainObject(variant)) return variant; + const value = variant.value; + if (isPlainObject(value) && value.__resolveType === "lazy") { + return { ...variant, value: { ...value, value: asSection(value.value, type) } }; + } + return { ...variant, value: asSection(value, type) }; + }), + }; + } + return node; +} + +/** Whether a prerelease block is (or picks between) the given built-in. */ +function isPrereleaseBlock(node: unknown, type: string): boolean { + if (!isPlainObject(node)) return false; + if (node.__resolveType === type) return true; + if (!isMultivariate(node.__resolveType) || !Array.isArray(node.variants)) return false; + return node.variants.some((variant) => { + if (!isPlainObject(variant)) return false; + const value = variant.value; + const inner = isPlainObject(value) && value.__resolveType === "lazy" ? value.value : value; + return isPlainObject(inner) && inner.__resolveType === type; + }); +} + +/** + * Whether v7 ran on `@decocms/tanstack`, whose worker entry added the site's + * deco-hosted domains to the preview hosts. v7's Next.js binding never did. + */ +function isTanStackSite(root: string): boolean { + try { + const pkg = JSON.parse(fs.readFileSync(path.join(root, "package.json"), "utf8")); + return ["dependencies", "devDependencies"].some( + (field) => isPlainObject(pkg[field]) && Object.hasOwn(pkg[field], "@decocms/tanstack"), + ); + } catch { + return false; + } +} + +/** + * The `DECO_SITE_NAME` v7's TanStack worker read: from the Workers vars in + * `wrangler.*` first, else from `.env`/`.dev.vars` and the Vite config's + * fallback. `undefined` when there's none, or several that disagree. + */ +function v7SiteName(root: string): { name?: string; candidates: string[] } { + const read = (file: string) => { + const full = path.join(root, file); + return fs.existsSync(full) ? fs.readFileSync(full, "utf8") : ""; + }; + const NAME = /^[a-z0-9][a-z0-9-]*$/; + const collect = (files: string[], pattern: RegExp) => { + const found = new Set(); + for (const file of files) { + for (const match of read(file).matchAll(pattern)) { + const name = match[1]?.trim().toLowerCase(); + if (name && NAME.test(name)) found.add(name); + } + } + return [...found]; + }; + const fromWrangler = collect( + ["wrangler.jsonc", "wrangler.json", "wrangler.toml"], + /["']?DECO_SITE_NAME["']?\s*[:=]\s*["']([^"']+)["']/g, + ); + const candidates = + fromWrangler.length > 0 + ? fromWrangler + : [ + ...new Set([ + ...collect([".env", ".dev.vars"], /^\s*DECO_SITE_NAME\s*=\s*["']?([^"'\s#]+)/gm), + ...collect( + ["vite.config.ts", "vite.config.js", "vite.config.mts"], + /DECO_SITE_NAME\s*\|\|\s*["']([^"']+)["']/g, + ), + ]), + ]; + return { name: candidates.length === 1 ? candidates[0] : undefined, candidates }; +} + +/** The hosts v7's TanStack worker always allowed for a named site. */ +function decoHostedHosts(site: string): string[] { + return [`${site}.deco.site`, `${site}.deco-cx.workers.dev`]; +} + +function listSources(dir: string): string[] { + if (!fs.existsSync(dir)) return []; + return fs + .readdirSync(dir, { recursive: true, withFileTypes: true }) + .filter((d) => d.isFile() && SOURCE.test(d.name) && !d.name.endsWith(".d.ts")) + .map((d) => path.join(d.parentPath, d.name)) + .filter((f) => !f.split(path.sep).includes("node_modules")); +} + +/** Literal `collectorAddress` values on `` elements in `src/`. */ +function oneDollarCollectors(root: string): { values: Set; dynamic: string[] } { + const values = new Set(); + const dynamic: string[] = []; + for (const file of listSources(path.join(root, "src"))) { + const text = fs.readFileSync(file, "utf8"); + if (!text.includes("OneDollarStats")) continue; + for (const element of text.matchAll(/]*)\/?>/g)) { + const attributes = element[1]; + if (!/\bcollectorAddress\s*=/.test(attributes)) continue; + const literal = + /\bcollectorAddress\s*=\s*(?:"([^"]*)"|'([^']*)'|\{\s*"([^"]*)"\s*\}|\{\s*'([^']*)'\s*\}|\{\s*`([^`$]*)`\s*\})/.exec( + attributes, + ); + const value = literal?.slice(1).find((v) => v !== undefined); + if (value) values.add(value); + else dynamic.push(path.relative(root, file).split(path.sep).join("/")); + } + } + return { values, dynamic }; +} + +/** + * Sets `value` at `section` (and `field`, when given) of `target` unless + * something is already there. Returns whether it changed anything. + */ +function fold(target: JsonObject, section: Section, value: unknown, field?: string): boolean { + if (field === undefined) { + if (target[section] === undefined) { + target[section] = value; + return true; + } + const existing = target[section]; + // Both plain sections: fold field by field; a section with variants is kept as it is. + if (!isPlainObject(existing) || "__resolveType" in existing) return false; + if (!isPlainObject(value) || "__resolveType" in value) return false; + let changed = false; + for (const [key, v] of Object.entries(value)) + changed = fold(target, section, v, key) || changed; + return changed; + } + const existing = target[section]; + if (existing === undefined) { + target[section] = { [field]: value }; + return true; + } + if (!isPlainObject(existing) || "__resolveType" in existing) return false; + if (existing[field] !== undefined) return false; + existing[field] = value; + return true; +} + +export function foldSiteSettings(root: string, report: Report): void { + const blocksDir = path.join(root, ".deco", "blocks"); + const { blocks, files } = readContent(root); + + const current = blocks[SETTINGS_BLOCK]; + if (current !== undefined && current.__resolveType !== SETTINGS_TYPE) { + report.manual.push({ + step: "settings", + subject: `.deco/blocks/${files[SETTINGS_BLOCK]}`, + message: `"${SETTINGS_BLOCK}" is the name of the CMS settings block (type "${SETTINGS_TYPE}"); rename this "${String(current.__resolveType)}" block, then run the migration again to fold your settings (/next/renames-and-migrations#site-settings)`, + }); + return; + } + const settings: JsonObject = current + ? (JSON.parse(JSON.stringify(current)) as JsonObject) + : { __resolveType: SETTINGS_TYPE }; + const done: string[] = []; + const remove: string[] = []; + /** Saved blocks to write back, without a field that moved. */ + const rewrite = new Map(); + + // The v7 Site block's preview hosts, plus the deco-hosted ones v7's TanStack worker added. + const tanstack = isTanStackSite(root); + const siteName = tanstack ? v7SiteName(root) : { candidates: [] }; + let previewHosts: string[] | undefined; + for (const name of SITE_BLOCKS) { + const site = blocks[name]; + if (!site || !Array.isArray(site.previewHosts)) continue; + previewHosts = []; + for (const raw of site.previewHosts) { + // v7 trimmed and lowercased each entry and compared it to the request's host[:port]. + const entry = typeof raw === "string" ? raw.trim().toLowerCase() : raw; + const pattern = typeof entry === "string" ? parseHostPattern(entry) : null; + if (pattern === null || pattern.kind === "any" || pattern.kind === "wildcard") { + report.manual.push({ + step: "settings", + subject: `.deco/blocks/${files[name]}`, + message: `previewHosts entry ${JSON.stringify(raw)} isn't a host v7 could match; it was left out of CMS.json's preview.hosts, so add the host you meant by hand (/next/api-reference#host-patterns)`, + }); + continue; + } + previewHosts.push(formatHostPattern(pattern)); + } + if (tanstack && siteName.name) previewHosts.push(...decoHostedHosts(siteName.name)); + previewHosts = [...new Set(previewHosts)]; + if (tanstack && !siteName.name) { + report.manual.push({ + step: "settings", + subject: "preview hosts", + message: `v7 also allowed previews on .deco.site and .deco-cx.workers.dev for the site named by DECO_SITE_NAME${siteName.candidates.length > 1 ? ` (found several: ${siteName.candidates.join(", ")})` : ""}; add the ones you use to CMS.json's preview.hosts (/next/releases-and-drafts#allow-previews-per-host)`, + }); + } + const folded = fold(settings, "preview", previewHosts, "hosts"); + const preview = settings.preview; + const hosts = isPlainObject(preview) ? preview.hosts : undefined; + // The field moves when CMS.json now holds those hosts; one that disagrees is left for a person. + if (folded || JSON.stringify(hosts) === JSON.stringify(previewHosts)) { + const { previewHosts: _, ...rest } = site; + rewrite.set(files[name], rest); + done.push(`preview.hosts from the ${name} block's previewHosts`); + } else { + report.manual.push({ + step: "settings", + subject: `.deco/blocks/${files[name]}`, + message: + "its previewHosts differ from preview.hosts in CMS.json; keep the right list in CMS.json and delete previewHosts", + }); + } + break; + } + + // A literal collectorAddress on OneDollarStats. + const collectors = oneDollarCollectors(root); + if (collectors.values.size === 1) { + const [collector] = collectors.values; + if (fold(settings, "analytics", collector, "collector")) { + done.push("analytics.collector from OneDollarStats' collectorAddress"); + } + } else if (collectors.values.size > 1) { + report.manual.push({ + step: "settings", + subject: "OneDollarStats collectorAddress", + message: `several collectors (${[...collectors.values].join(", ")}): put the one to keep in CMS.json's analytics.collector (/next/analytics)`, + }); + } + for (const file of collectors.dynamic) { + report.manual.push({ + step: "settings", + subject: file, + message: + "OneDollarStats' collectorAddress isn't a literal: put its value in CMS.json's analytics.collector (/next/analytics)", + }); + } + + // Prerelease Telemetry.json / Analytics.json. + for (const { name, type } of PRERELEASE) { + const block = blocks[name]; + if (!isPrereleaseBlock(block, type)) continue; + const section = asSection(block, type); + const changed = fold(settings, type, section); + if (!changed && settings[type] !== undefined && !isSubset(section, settings[type])) { + report.manual.push({ + step: "settings", + subject: `.deco/blocks/${files[name]}`, + message: `CMS.json already has a ${type} section that differs; merge this block into it by hand, then delete it`, + }); + continue; + } + remove.push(files[name]); + done.push(`the ${type} section from ${files[name]} (deleted)`); + } + + // Leftover blocks of the folded built-ins fail deco check. + for (const [name, block] of Object.entries(blocks)) { + if (remove.includes(files[name])) continue; + const type = block.__resolveType; + if (type === "telemetry" || type === "analytics") { + report.manual.push({ + step: "settings", + subject: `.deco/blocks/${files[name]}`, + message: `the "${type}" built-in is gone: fold this block into the ${type} section of CMS.json (/next/renames-and-migrations#site-settings)`, + }); + } + } + + if (done.length > 0) { + fs.mkdirSync(blocksDir, { recursive: true }); + const file = files[SETTINGS_BLOCK] ?? blockFileName(SETTINGS_BLOCK); + const next = serializeBlock(settings); + const full = path.join(blocksDir, file); + if (!fs.existsSync(full) || fs.readFileSync(full, "utf8") !== next) { + fs.writeFileSync(full, next); + } + for (const [old, block] of rewrite) + fs.writeFileSync(path.join(blocksDir, old), serializeBlock(block)); + for (const old of remove) fs.rmSync(path.join(blocksDir, old), { force: true }); + report.done.push({ + step: "settings", + subject: `.deco/blocks/${file}`, + message: `folded ${done.join("; ")}`, + }); + } + + // Settings v7 read from the hosting environment never reach the repository. + report.manual.push( + { + step: "settings", + subject: "DECO_ALLOWED_PREVIEW_HOSTS", + message: + "if your hosting sets it, put its hosts in CMS.json's preview.hosts, or in createCMS({ preview: { hosts } }) to make them the most content may allow; its none is an empty list (/next/releases-and-drafts#allow-previews-per-host)", + }, + { + step: "settings", + subject: "DECO_OTEL_* sampling", + message: + "if your hosting sets sampling variables, move them to the telemetry section's errorSampleRate and traceSampleRate, within the limits in code (/next/telemetry#sampling)", + }, + { + step: "settings", + subject: "DECO_ANALYTICS_ENABLED, ONEDOLLAR_ENABLED, ONEDOLLAR_COLLECTOR", + message: + "if your hosting sets them, the enabled ones become the analytics section's enabled and ONEDOLLAR_COLLECTOR its collector (/next/analytics)", + }, + ); + if (previewHosts === undefined && settings.preview === undefined && current === undefined) { + const v7Hosts = siteName.name + ? `only on ${decoHostedHosts(siteName.name).join(" and ")} (from DECO_SITE_NAME)` + : tanstack + ? "only on .deco.site and .deco-cx.workers.dev for the site named by DECO_SITE_NAME, or nowhere without one" + : "nowhere"; + report.manual.push({ + step: "settings", + subject: "preview hosts", + message: `with no previewHosts, v7 allowed previews ${v7Hosts}; the next major allows every host unless preview.hosts lists some, so add that list to CMS.json if you relied on it (/next/releases-and-drafts#allow-previews-per-host)`, + }); + } +} + +/** Whether every field of `part` is already in `whole` with the same value. */ +function isSubset(part: unknown, whole: unknown): boolean { + if (!isPlainObject(part) || !isPlainObject(whole)) { + return JSON.stringify(part) === JSON.stringify(whole); + } + return Object.entries(part).every(([key, value]) => isSubset(value, whole[key])); +} diff --git a/.agents/skills/deco-v7-to-v8-migration/scripts/steps.test.ts b/.agents/skills/deco-v7-to-v8-migration/scripts/steps.test.ts index 88c541a45..b3ff90b24 100644 --- a/.agents/skills/deco-v7-to-v8-migration/scripts/steps.test.ts +++ b/.agents/skills/deco-v7-to-v8-migration/scripts/steps.test.ts @@ -3,6 +3,7 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; import { afterEach, describe, expect, it } from "vitest"; +import { unwrapAsyncRendering } from "./asyncRendering"; import { writeBlockMap } from "./blockMap"; import { decofileEntries, moveContent } from "./content"; import { copyExperimentIds } from "./experiments"; @@ -10,6 +11,7 @@ import { rewriteImports, V8_API } from "./imports"; import { renameLegacyTypes } from "./legacyNames"; import { createReport } from "./report"; import { reencryptSecrets } from "./secrets"; +import { foldSiteSettings } from "./siteSettings"; import { locateAppModule, vendorModule } from "./vendor"; let root: string; @@ -27,6 +29,78 @@ function site(files: Record): string { const home = { __resolveType: "website/pages/Page.tsx", name: "Home", path: "/", sections: [] }; +describe("async rendering", () => { + const LAZY = "website/sections/Rendering/Lazy.tsx"; + const DEFERRED = "website/sections/Rendering/Deferred.tsx"; + // storefront-tanstack's home: a ProductShelfTabbed behind v7's Lazy wrapper. + const shelf = { + __resolveType: "site/sections/Product/ProductShelfTabbed.tsx", + title: "Hottest Deals", + tabs: [ + { + title: "Accessories", + products: { __resolveType: "shopify/loaders/ProductList.ts", props: { count: 10 } }, + }, + ], + }; + const hero = { __resolveType: "site/sections/Hero.tsx", title: "Hi" }; + const footer = { __resolveType: "Footer" }; + + it("replaces each wrapper with the section(s) it held, props untouched; a second run changes nothing", () => { + site({ + ".deco/blocks/pages-home.json": { + ...home, + sections: [ + { __resolveType: LAZY, section: shelf, loading: "lazy" }, + { __resolveType: DEFERRED, sections: [hero, { __resolveType: LAZY, section: footer }] }, + { __resolveType: "website/sections/Rendering/SingleDeferred.tsx", section: hero }, + { __resolveType: LAZY }, + ], + }, + ".deco/blocks/Lazy%20Footer.json": { __resolveType: LAZY, section: footer }, + ".deco/blocks/Banner.json": { + __resolveType: "site/sections/Banner.tsx", + slot: { __resolveType: LAZY, section: hero }, + }, + ".deco/blocks/Plain.json": hero, + }); + const report = createReport(); + unwrapAsyncRendering(root, report); + const read = (file: string) => + JSON.parse(fs.readFileSync(path.join(root, ".deco/blocks", file), "utf8")); + // In a list, a Deferred gives way to all its sections in place; an empty wrapper goes. + expect(read("pages-home.json").sections).toEqual([shelf, hero, footer, hero]); + expect(read("Lazy%20Footer.json")).toEqual(footer); + expect(read("Banner.json")).toEqual({ __resolveType: "site/sections/Banner.tsx", slot: hero }); + expect(report.done.map((n) => n.subject).sort()).toEqual([ + "Banner.json", + "Lazy%20Footer.json", + "pages-home.json", + ]); + expect(report.manual).toEqual([]); + + const again = createReport(); + unwrapAsyncRendering(root, again); + expect(again).toEqual({ done: [], manual: [] }); + }); + + it("reports a wrapper holding several sections where one block goes, unwrapping inside it", () => { + site({ + ".deco/blocks/Slot.json": { + __resolveType: DEFERRED, + sections: [{ __resolveType: LAZY, section: hero }, footer], + }, + }); + const report = createReport(); + unwrapAsyncRendering(root, report); + expect(JSON.parse(fs.readFileSync(path.join(root, ".deco/blocks/Slot.json"), "utf8"))).toEqual({ + __resolveType: DEFERRED, + sections: [hero, footer], + }); + expect(report.manual.map((n) => n.subject)).toEqual([`${DEFERRED} (in Slot.json)`]); + }); +}); + describe("content", () => { it("reads a decofile flat or under blocks", () => { expect(decofileEntries({ Home: home })).toEqual({ Home: home }); @@ -234,6 +308,16 @@ describe("imports", () => { ]); }); + it("points prerelease draft helpers at the CMS methods", () => { + site({ + "src/a.ts": 'import { createCMS, draftPointer, DRAFT_COOKIE } from "@decocms/blocks";\n', + }); + const report = createReport(); + rewriteImports(root, report, new Map()); + expect(report.manual).toHaveLength(1); + expect(report.manual[0]?.message).toContain("cms.draftPointer"); + }); + it("reports every framework-binding import, kvLoader included: the next major has no binding", () => { site({ "src/a.ts": @@ -265,3 +349,295 @@ describe("the codemod's list of the root's next-major exports", () => { expect([...(V8_API["@decocms/blocks"] as Set)].sort()).toEqual(names.sort()); }); }); + +describe("site settings", () => { + const read = (file: string) => + JSON.parse(fs.readFileSync(path.join(root, ".deco/blocks", file), "utf8")); + const exists = (file: string) => fs.existsSync(path.join(root, ".deco/blocks", file)); + const snapshotDir = () => + Object.fromEntries( + fs + .readdirSync(path.join(root, ".deco/blocks")) + .sort() + .map((f) => [f, fs.readFileSync(path.join(root, ".deco/blocks", f), "utf8")]), + ); + + it("folds a v7 site's preview hosts and OneDollarStats collector into CMS.json; a second run changes nothing", () => { + site({ + "package.json": { name: "site" }, + ".deco/blocks/site.json": { + __resolveType: "site/apps/site.ts", + theme: { primary: "#000" }, + previewHosts: ["staging.example.com", "localhost:5173"], + }, + ".deco/blocks/pages-home.json": home, + "src/routes/__root.tsx": `import { OneDollarStats } from "@decocms/apps-website/components/OneDollarStats"; +export function Root() { + return ; +} +`, + }); + const report = createReport(); + foldSiteSettings(root, report); + expect(read("CMS.json")).toEqual({ + __resolveType: "cms-settings", + preview: { hosts: ["staging.example.com", "localhost:5173"] }, + analytics: { collector: "https://stats.example.com/events" }, + }); + // previewHosts moved; the rest of the Site block stays. + expect(read("site.json")).toEqual({ + __resolveType: "site/apps/site.ts", + theme: { primary: "#000" }, + }); + expect(report.done.map((n) => n.subject)).toEqual([".deco/blocks/CMS.json"]); + const manual = report.manual.map((n) => n.subject); + expect(manual).toContain("DECO_ALLOWED_PREVIEW_HOSTS"); + expect(manual).toContain("DECO_OTEL_* sampling"); + expect(manual).toContain("DECO_ANALYTICS_ENABLED, ONEDOLLAR_ENABLED, ONEDOLLAR_COLLECTOR"); + expect(manual).not.toContain("preview hosts"); + + const before = snapshotDir(); + const again = createReport(); + foldSiteSettings(root, again); + expect(snapshotDir()).toEqual(before); + expect(again.done).toEqual([]); + }); + + it("trims and lowercases v7's entries, and reports one that isn't a host", () => { + site({ + "package.json": { name: "site" }, + ".deco/blocks/site.json": { + __resolveType: "site/apps/site.ts", + previewHosts: [ + " Staging.Example.com ", + "localhost:3100", + "https://bad.example.com/", + "*.x.com", + ], + }, + }); + const report = createReport(); + foldSiteSettings(root, report); + expect(read("CMS.json").preview).toEqual({ hosts: ["staging.example.com", "localhost:3100"] }); + const bad = report.manual.filter((n) => n.subject === ".deco/blocks/site.json"); + expect(bad.map((n) => n.message).join("\n")).toContain("https://bad.example.com/"); + expect(bad.map((n) => n.message).join("\n")).toContain("*.x.com"); + }); + + it("on TanStack Start, adds the deco-hosted hosts v7 allowed for DECO_SITE_NAME", () => { + site({ + "package.json": { name: "site", dependencies: { "@decocms/tanstack": "7.0.0" } }, + "wrangler.jsonc": `{ "vars": { "DECO_SITE_NAME": "acme" } }`, + ".env": "DECO_SITE_NAME=other\n", + ".deco/blocks/site.json": { + __resolveType: "site/apps/site.ts", + previewHosts: ["staging.example.com", "acme.deco.site"], + }, + }); + const report = createReport(); + foldSiteSettings(root, report); + expect(read("CMS.json").preview).toEqual({ + hosts: ["staging.example.com", "acme.deco.site", "acme.deco-cx.workers.dev"], + }); + expect(report.manual.map((n) => n.subject)).not.toContain("preview hosts"); + const before = snapshotDir(); + foldSiteSettings(root, createReport()); + expect(snapshotDir()).toEqual(before); + }); + + it("on TanStack Start without a site name, reports the deco-hosted hosts; a Next.js site never gets them", () => { + site({ + "package.json": { name: "site", dependencies: { "@decocms/tanstack": "7.0.0" } }, + ".deco/blocks/site.json": { + __resolveType: "site/apps/site.ts", + previewHosts: ["a.example.com"], + }, + }); + const report = createReport(); + foldSiteSettings(root, report); + expect(read("CMS.json").preview).toEqual({ hosts: ["a.example.com"] }); + const note = report.manual.find((n) => n.subject === "preview hosts"); + expect(note?.message).toContain(".deco.site"); + + fs.rmSync(root, { recursive: true, force: true }); + site({ + "package.json": { name: "site", dependencies: { "@decocms/nextjs": "7.0.0" } }, + ".env": "DECO_SITE_NAME=acme\n", + ".deco/blocks/site.json": { + __resolveType: "site/apps/site.ts", + previewHosts: ["a.example.com"], + }, + }); + const next = createReport(); + foldSiteSettings(root, next); + expect(read("CMS.json").preview).toEqual({ hosts: ["a.example.com"] }); + expect(next.manual.map((n) => n.subject)).not.toContain("preview hosts"); + }); + + it("with no previewHosts on a named TanStack site, says v7 allowed only its deco-hosted hosts", () => { + site({ + "package.json": { name: "site", dependencies: { "@decocms/tanstack": "7.0.0" } }, + ".env": "DECO_SITE_NAME=acme\n", + ".deco/blocks/pages-home.json": home, + }); + const report = createReport(); + foldSiteSettings(root, report); + const note = report.manual.find((n) => n.subject === "preview hosts"); + expect(note?.message).toContain("acme.deco.site and acme.deco-cx.workers.dev"); + expect(note?.message).not.toContain("kept previews off"); + }); + + it("reads a Site block named Site, and collectorAddress={'…'}", () => { + site({ + "package.json": { name: "site" }, + ".deco/blocks/Site.json": { __resolveType: "site/apps/site.ts", previewHosts: [] }, + "src/app/layout.tsx": ``, + }); + foldSiteSettings(root, createReport()); + expect(read("CMS.json")).toEqual({ + __resolveType: "cms-settings", + preview: { hosts: [] }, + analytics: { collector: "https://c.example/e" }, + }); + }); + + it("folds prerelease Telemetry.json and Analytics.json field for field, variants included, and deletes them", () => { + const variants = { + __resolveType: "multivariate", + variants: [ + { + rule: { __resolveType: "date", start: "2026-11-01" }, + value: { __resolveType: "lazy", value: { __resolveType: "analytics", enabled: true } }, + }, + { + rule: { __resolveType: "always" }, + value: { __resolveType: "lazy", value: { __resolveType: "analytics", enabled: false } }, + }, + ], + }; + site({ + "package.json": { name: "site" }, + ".deco/blocks/Telemetry.json": { + __resolveType: "telemetry", + enabled: true, + errorSampleRate: { __resolveType: "Rates" }, + traceSampleRate: 0, + }, + ".deco/blocks/Analytics.json": variants, + }); + const report = createReport(); + foldSiteSettings(root, report); + expect(exists("Telemetry.json")).toBe(false); + expect(exists("Analytics.json")).toBe(false); + expect(read("CMS.json")).toEqual({ + __resolveType: "cms-settings", + telemetry: { enabled: true, errorSampleRate: { __resolveType: "Rates" }, traceSampleRate: 0 }, + analytics: { + __resolveType: "multivariate", + variants: [ + { + rule: { __resolveType: "date", start: "2026-11-01" }, + value: { __resolveType: "lazy", value: { enabled: true } }, + }, + { + rule: { __resolveType: "always" }, + value: { __resolveType: "lazy", value: { enabled: false } }, + }, + ], + }, + }); + // A prerelease site with no preview hosts gets the behaviour-change note. + expect(report.manual.map((n) => n.subject)).toContain("preview hosts"); + const before = snapshotDir(); + foldSiteSettings(root, createReport()); + expect(snapshotDir()).toEqual(before); + }); + + it("keeps what CMS.json already has, and adds only what's missing", () => { + site({ + "package.json": { name: "site" }, + ".deco/blocks/CMS.json": { + __resolveType: "cms-settings", + preview: { hosts: ["kept.example.com"] }, + telemetry: { metrics: false }, + }, + ".deco/blocks/site.json": { + __resolveType: "site/apps/site.ts", + previewHosts: ["v7.example.com"], + }, + ".deco/blocks/Telemetry.json": { + __resolveType: "telemetry", + metrics: true, + errorSampleRate: 0.01, + }, + }); + const report = createReport(); + foldSiteSettings(root, report); + expect(read("CMS.json")).toEqual({ + __resolveType: "cms-settings", + preview: { hosts: ["kept.example.com"] }, + telemetry: { metrics: false, errorSampleRate: 0.01 }, + }); + expect(exists("Telemetry.json")).toBe(false); + // The Site block's different list is left for a person, untouched. + expect(read("site.json").previewHosts).toEqual(["v7.example.com"]); + expect(report.manual.map((n) => n.subject)).toContain(".deco/blocks/site.json"); + }); + + it("writes nothing when there's nothing to fold; a v7 Analytics section named Analytics is left alone", () => { + site({ + "package.json": { name: "site" }, + ".deco/blocks/pages-home.json": home, + ".deco/blocks/Analytics.json": { + __resolveType: "website/sections/Analytics/Analytics.tsx", + trackingIds: ["GTM-1"], + }, + }); + const before = snapshotDir(); + const report = createReport(); + foldSiteSettings(root, report); + expect(snapshotDir()).toEqual(before); + expect(report.done).toEqual([]); + expect(report.manual.map((n) => n.subject)).toContain("preview hosts"); + }); + + it("never replaces a CMS block of another type; leftover telemetry/analytics blocks are reported", () => { + site({ + "package.json": { name: "site" }, + ".deco/blocks/CMS.json": { __resolveType: "site/sections/Cms.tsx" }, + ".deco/blocks/site.json": { + __resolveType: "site/apps/site.ts", + previewHosts: ["a.example.com"], + }, + }); + const report = createReport(); + foldSiteSettings(root, report); + expect(read("CMS.json")).toEqual({ __resolveType: "site/sections/Cms.tsx" }); + expect(report.done).toEqual([]); + expect(report.manual[0]?.message).toContain("rename"); + + site({ + "package.json": { name: "site" }, + ".deco/blocks/OtherTelemetry.json": { __resolveType: "telemetry", enabled: false }, + }); + const leftovers = createReport(); + foldSiteSettings(root, leftovers); + expect(exists("OtherTelemetry.json")).toBe(true); + expect(leftovers.manual.map((n) => n.subject)).toContain(".deco/blocks/OtherTelemetry.json"); + }); + + it("several collectors, or one that isn't a literal, are left for a person", () => { + site({ + "package.json": { name: "site" }, + "src/a.tsx": ``, + "src/b.tsx": ``, + "src/c.tsx": "", + }); + const report = createReport(); + foldSiteSettings(root, report); + expect(fs.existsSync(path.join(root, ".deco/blocks/CMS.json"))).toBe(false); + const subjects = report.manual.map((n) => n.subject); + expect(subjects).toContain("OneDollarStats collectorAddress"); + expect(subjects).toContain("src/c.tsx"); + }); +}); diff --git a/.cursor/skills/deco-site-deployment/SKILL.md b/.cursor/skills/deco-site-deployment/SKILL.md index 747b64b13..52e2a211b 100644 --- a/.cursor/skills/deco-site-deployment/SKILL.md +++ b/.cursor/skills/deco-site-deployment/SKILL.md @@ -191,7 +191,7 @@ kubectl get secret state -n sites- -o json \ Extract, modify, and write back: ```bash -SITENAME="fila-store" +SITENAME="my-store" NS="sites-${SITENAME}" # 1. Extract current state @@ -216,7 +216,7 @@ if [ -z "$ADMIN_API_KEY" ]; then exit 1 fi -SITENAME="fila-store" +SITENAME="my-store" # Deploy to AWS curl -s --location "https://admin.deco.cx/live/invoke/deco-sites/admin/actions/hosting/deploy.ts" \ @@ -317,7 +317,7 @@ if [ -z "$ADMIN_API_KEY" ]; then exit 1 fi -SITENAME="fila-store" +SITENAME="my-store" NS="sites-${SITENAME}" # Ask user which cluster to target, then set: diff --git a/CLAUDE.md b/CLAUDE.md index 3bcf046c7..dae3d49f0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -74,7 +74,7 @@ Each export is built from the source file below (`dist/.js`, see abov | Import path | File | |---|---| -| `@decocms/blocks` | `src/index.ts` — the v8 runtime API only (`createCMS`, `matchRoute`, `remoteLoader`, `draftPointer`, `Blocks`, `Lazy`, …), re-exported from `src/v8/index.ts` as one explicit `export { … } from "./v8/index.ts"` block (a conformance test parses that form) | +| `@decocms/blocks` | `src/index.ts` — the v8 runtime API only (`createCMS`, `matchRoute`, `remoteLoader`, `parseDraftPointer`, `Blocks`, `Lazy`, …; the draft helpers are `cms.draftPointer`/`cms.draftCookie`, since they check the preview hosts in `cms.settings()`), re-exported from `src/v8/index.ts` as one explicit `export { … } from "./v8/index.ts"` block (a conformance test parses that form) | | `@decocms/blocks/analytics` | `src/v8/analytics.ts` — `AnalyticsScript`, `track` | | `@decocms/blocks/fetch` | `src/v8/fetch.ts` — `createInstrumentedFetch` | | `@decocms/blocks/secrets` | `src/v8/secrets.ts` | diff --git a/examples/nextjs-smoke/.deco/schema.gen.json b/examples/nextjs-smoke/.deco/schema.gen.json index 730859c43..2d98ea9a7 100644 --- a/examples/nextjs-smoke/.deco/schema.gen.json +++ b/examples/nextjs-smoke/.deco/schema.gen.json @@ -1,5 +1,6 @@ { "major": 1, + "blocksMajor": 8, "version": "0.0.0", "namespace": "site", "site": "nextjs-smoke-fixture", @@ -96,12 +97,8 @@ "$ref": "#/definitions/c2Vv", "namespace": "site" }, - "telemetry": { - "$ref": "#/definitions/dGVsZW1ldHJ5", - "namespace": "deco" - }, - "analytics": { - "$ref": "#/definitions/YW5hbHl0aWNz", + "cms-settings": { + "$ref": "#/definitions/Y21zLXNldHRpbmdz", "namespace": "deco" } } @@ -624,9 +621,10 @@ } } }, - "dGVsZW1ldHJ5": { - "title": "telemetry", - "description": "Telemetry switches and sample rates", + "Y21zLXNldHRpbmdz": { + "title": "cms-settings", + "description": "Preview hosts, telemetry and analytics: the saved block named CMS", + "icon": "settings", "type": "object", "required": [ "__resolveType" @@ -635,59 +633,68 @@ "__resolveType": { "type": "string", "enum": [ - "telemetry" + "cms-settings" ], - "default": "telemetry" - }, - "enabled": { - "type": "boolean", - "title": "Enabled", - "default": true - }, - "metrics": { - "type": "boolean", - "title": "Metrics", - "default": true + "default": "cms-settings" }, - "errorSampleRate": { - "type": "number", - "title": "Error sample rate", - "minimum": 0, - "maximum": 1, - "default": 0.05 - }, - "traceSampleRate": { - "type": "number", - "title": "Trace sample rate", - "minimum": 0, - "maximum": 1, - "default": 0 - } - } - }, - "YW5hbHl0aWNz": { - "title": "analytics", - "description": "Page view analytics settings", - "type": "object", - "required": [ - "__resolveType" - ], - "properties": { - "__resolveType": { - "type": "string", - "enum": [ - "analytics" - ], - "default": "analytics" + "preview": { + "type": "object", + "title": "Preview", + "properties": { + "hosts": { + "type": "array", + "title": "Hosts", + "description": "Host patterns drafts may be previewed on, such as staging.example.com or *.example.com; empty turns previews off. Without it, every host (or the hosts code allows).", + "items": { + "type": "string" + } + } + } }, - "collector": { - "type": "string", - "title": "Collector" + "telemetry": { + "type": "object", + "title": "Telemetry", + "properties": { + "enabled": { + "type": "boolean", + "title": "Enabled", + "default": true + }, + "metrics": { + "type": "boolean", + "title": "Metrics", + "default": true + }, + "errorSampleRate": { + "type": "number", + "title": "Error sample rate", + "minimum": 0, + "maximum": 1, + "default": 0.05 + }, + "traceSampleRate": { + "type": "number", + "title": "Trace sample rate", + "minimum": 0, + "maximum": 1, + "default": 0 + } + } }, - "enabled": { - "type": "boolean", - "title": "Enabled", - "default": true + "analytics": { + "type": "object", + "title": "Analytics", + "properties": { + "collector": { + "type": "string", + "title": "Collector" + }, + "enabled": { + "type": "boolean", + "title": "Enabled", + "default": true + } + } } } }, @@ -1301,10 +1308,7 @@ "$ref": "#/definitions/c2Vv" }, { - "$ref": "#/definitions/dGVsZW1ldHJ5" - }, - { - "$ref": "#/definitions/YW5hbHl0aWNz" + "$ref": "#/definitions/Y21zLXNldHRpbmdz" } ] } diff --git a/examples/tanstack-rsc-smoke/.deco/schema.gen.json b/examples/tanstack-rsc-smoke/.deco/schema.gen.json index 9bc3b5764..3deaf0bf6 100644 --- a/examples/tanstack-rsc-smoke/.deco/schema.gen.json +++ b/examples/tanstack-rsc-smoke/.deco/schema.gen.json @@ -1,5 +1,6 @@ { "major": 1, + "blocksMajor": 8, "version": "0.0.0", "namespace": "site", "site": "tanstack-rsc-smoke-fixture", @@ -92,12 +93,8 @@ "$ref": "#/definitions/c2Vv", "namespace": "site" }, - "telemetry": { - "$ref": "#/definitions/dGVsZW1ldHJ5", - "namespace": "deco" - }, - "analytics": { - "$ref": "#/definitions/YW5hbHl0aWNz", + "cms-settings": { + "$ref": "#/definitions/Y21zLXNldHRpbmdz", "namespace": "deco" } } @@ -582,9 +579,10 @@ } } }, - "dGVsZW1ldHJ5": { - "title": "telemetry", - "description": "Telemetry switches and sample rates", + "Y21zLXNldHRpbmdz": { + "title": "cms-settings", + "description": "Preview hosts, telemetry and analytics: the saved block named CMS", + "icon": "settings", "type": "object", "required": [ "__resolveType" @@ -593,59 +591,68 @@ "__resolveType": { "type": "string", "enum": [ - "telemetry" + "cms-settings" ], - "default": "telemetry" - }, - "enabled": { - "type": "boolean", - "title": "Enabled", - "default": true - }, - "metrics": { - "type": "boolean", - "title": "Metrics", - "default": true + "default": "cms-settings" }, - "errorSampleRate": { - "type": "number", - "title": "Error sample rate", - "minimum": 0, - "maximum": 1, - "default": 0.05 - }, - "traceSampleRate": { - "type": "number", - "title": "Trace sample rate", - "minimum": 0, - "maximum": 1, - "default": 0 - } - } - }, - "YW5hbHl0aWNz": { - "title": "analytics", - "description": "Page view analytics settings", - "type": "object", - "required": [ - "__resolveType" - ], - "properties": { - "__resolveType": { - "type": "string", - "enum": [ - "analytics" - ], - "default": "analytics" + "preview": { + "type": "object", + "title": "Preview", + "properties": { + "hosts": { + "type": "array", + "title": "Hosts", + "description": "Host patterns drafts may be previewed on, such as staging.example.com or *.example.com; empty turns previews off. Without it, every host (or the hosts code allows).", + "items": { + "type": "string" + } + } + } }, - "collector": { - "type": "string", - "title": "Collector" + "telemetry": { + "type": "object", + "title": "Telemetry", + "properties": { + "enabled": { + "type": "boolean", + "title": "Enabled", + "default": true + }, + "metrics": { + "type": "boolean", + "title": "Metrics", + "default": true + }, + "errorSampleRate": { + "type": "number", + "title": "Error sample rate", + "minimum": 0, + "maximum": 1, + "default": 0.05 + }, + "traceSampleRate": { + "type": "number", + "title": "Trace sample rate", + "minimum": 0, + "maximum": 1, + "default": 0 + } + } }, - "enabled": { - "type": "boolean", - "title": "Enabled", - "default": true + "analytics": { + "type": "object", + "title": "Analytics", + "properties": { + "collector": { + "type": "string", + "title": "Collector" + }, + "enabled": { + "type": "boolean", + "title": "Enabled", + "default": true + } + } } } }, @@ -1256,10 +1263,7 @@ "$ref": "#/definitions/c2Vv" }, { - "$ref": "#/definitions/dGVsZW1ldHJ5" - }, - { - "$ref": "#/definitions/YW5hbHl0aWNz" + "$ref": "#/definitions/Y21zLXNldHRpbmdz" } ] } diff --git a/examples/tanstack-rsc-smoke/src/cms.ts b/examples/tanstack-rsc-smoke/src/cms.ts index df1f55f30..037de63e9 100644 --- a/examples/tanstack-rsc-smoke/src/cms.ts +++ b/examples/tanstack-rsc-smoke/src/cms.ts @@ -7,4 +7,4 @@ export const cms = createCMS({ blocks, content }); // The client for this request. Every page gets its client here, so this is the one place to change // if requests ever need different content. -export const client = (_request: Request) => cms.forRelease(); +export const client = async (_request: Request) => cms.forRelease(); diff --git a/examples/tanstack-rsc-smoke/src/open-page.server.ts b/examples/tanstack-rsc-smoke/src/open-page.server.ts index 7902702a4..2513fc4fe 100644 --- a/examples/tanstack-rsc-smoke/src/open-page.server.ts +++ b/examples/tanstack-rsc-smoke/src/open-page.server.ts @@ -4,7 +4,7 @@ import { client } from "./cms"; import type { StoredPage } from "./model"; export async function openPage(href: string, request: Request) { - const c = client(request); + const c = await client(request); const [pages, pagesError] = await c.list("page"); if (pagesError) throw pagesError; const [redirects, redirectsError] = await c.list("redirect"); diff --git a/examples/tanstack-smoke/.deco/schema.gen.json b/examples/tanstack-smoke/.deco/schema.gen.json index af70cfe7e..e98733d48 100644 --- a/examples/tanstack-smoke/.deco/schema.gen.json +++ b/examples/tanstack-smoke/.deco/schema.gen.json @@ -1,5 +1,6 @@ { "major": 1, + "blocksMajor": 8, "version": "0.0.0", "namespace": "site", "site": "tanstack-smoke-fixture", @@ -92,12 +93,8 @@ "$ref": "#/definitions/c2Vv", "namespace": "site" }, - "telemetry": { - "$ref": "#/definitions/dGVsZW1ldHJ5", - "namespace": "deco" - }, - "analytics": { - "$ref": "#/definitions/YW5hbHl0aWNz", + "cms-settings": { + "$ref": "#/definitions/Y21zLXNldHRpbmdz", "namespace": "deco" } } @@ -502,9 +499,10 @@ } } }, - "dGVsZW1ldHJ5": { - "title": "telemetry", - "description": "Telemetry switches and sample rates", + "Y21zLXNldHRpbmdz": { + "title": "cms-settings", + "description": "Preview hosts, telemetry and analytics: the saved block named CMS", + "icon": "settings", "type": "object", "required": [ "__resolveType" @@ -513,59 +511,68 @@ "__resolveType": { "type": "string", "enum": [ - "telemetry" + "cms-settings" ], - "default": "telemetry" - }, - "enabled": { - "type": "boolean", - "title": "Enabled", - "default": true - }, - "metrics": { - "type": "boolean", - "title": "Metrics", - "default": true + "default": "cms-settings" }, - "errorSampleRate": { - "type": "number", - "title": "Error sample rate", - "minimum": 0, - "maximum": 1, - "default": 0.05 - }, - "traceSampleRate": { - "type": "number", - "title": "Trace sample rate", - "minimum": 0, - "maximum": 1, - "default": 0 - } - } - }, - "YW5hbHl0aWNz": { - "title": "analytics", - "description": "Page view analytics settings", - "type": "object", - "required": [ - "__resolveType" - ], - "properties": { - "__resolveType": { - "type": "string", - "enum": [ - "analytics" - ], - "default": "analytics" + "preview": { + "type": "object", + "title": "Preview", + "properties": { + "hosts": { + "type": "array", + "title": "Hosts", + "description": "Host patterns drafts may be previewed on, such as staging.example.com or *.example.com; empty turns previews off. Without it, every host (or the hosts code allows).", + "items": { + "type": "string" + } + } + } }, - "collector": { - "type": "string", - "title": "Collector" + "telemetry": { + "type": "object", + "title": "Telemetry", + "properties": { + "enabled": { + "type": "boolean", + "title": "Enabled", + "default": true + }, + "metrics": { + "type": "boolean", + "title": "Metrics", + "default": true + }, + "errorSampleRate": { + "type": "number", + "title": "Error sample rate", + "minimum": 0, + "maximum": 1, + "default": 0.05 + }, + "traceSampleRate": { + "type": "number", + "title": "Trace sample rate", + "minimum": 0, + "maximum": 1, + "default": 0 + } + } }, - "enabled": { - "type": "boolean", - "title": "Enabled", - "default": true + "analytics": { + "type": "object", + "title": "Analytics", + "properties": { + "collector": { + "type": "string", + "title": "Collector" + }, + "enabled": { + "type": "boolean", + "title": "Enabled", + "default": true + } + } } } }, @@ -982,10 +989,7 @@ "$ref": "#/definitions/c2Vv" }, { - "$ref": "#/definitions/dGVsZW1ldHJ5" - }, - { - "$ref": "#/definitions/YW5hbHl0aWNz" + "$ref": "#/definitions/Y21zLXNldHRpbmdz" } ] } diff --git a/examples/tanstack-smoke/src/cms.ts b/examples/tanstack-smoke/src/cms.ts index df1f55f30..037de63e9 100644 --- a/examples/tanstack-smoke/src/cms.ts +++ b/examples/tanstack-smoke/src/cms.ts @@ -7,4 +7,4 @@ export const cms = createCMS({ blocks, content }); // The client for this request. Every page gets its client here, so this is the one place to change // if requests ever need different content. -export const client = (_request: Request) => cms.forRelease(); +export const client = async (_request: Request) => cms.forRelease(); diff --git a/examples/tanstack-smoke/src/open-page.server.ts b/examples/tanstack-smoke/src/open-page.server.ts index 7902702a4..2513fc4fe 100644 --- a/examples/tanstack-smoke/src/open-page.server.ts +++ b/examples/tanstack-smoke/src/open-page.server.ts @@ -4,7 +4,7 @@ import { client } from "./cms"; import type { StoredPage } from "./model"; export async function openPage(href: string, request: Request) { - const c = client(request); + const c = await client(request); const [pages, pagesError] = await c.list("page"); if (pagesError) throw pagesError; const [redirects, redirectsError] = await c.list("redirect"); diff --git a/packages/blocks/conformance/observability/.deco/blocks.gen.ts b/packages/blocks/conformance/observability/.deco/blocks.gen.ts index ca174f715..19bc2d70d 100644 --- a/packages/blocks/conformance/observability/.deco/blocks.gen.ts +++ b/packages/blocks/conformance/observability/.deco/blocks.gen.ts @@ -7,7 +7,7 @@ const content: { revision: "rev-observability", root: "conformance/observability/.deco", blocks: { - Analytics: { __resolveType: "analytics" }, + CMS: { __resolveType: "cms-settings", analytics: {} }, Banner: { __resolveType: "promo-banner", title: "Sale", href: "/sale" }, }, }; diff --git a/packages/blocks/conformance/observability/layout.tsx b/packages/blocks/conformance/observability/layout.tsx index 7794b7d44..743733d56 100644 --- a/packages/blocks/conformance/observability/layout.tsx +++ b/packages/blocks/conformance/observability/layout.tsx @@ -1,10 +1,9 @@ // analytics.mdx › Render the script, verbatim (app/layout.tsx). -import type { Analytics } from "@decocms/blocks"; import { AnalyticsScript } from "@decocms/blocks/analytics"; import { cms } from "./cms"; export default async function RootLayout({ children }: { children: React.ReactNode }) { - const [analytics] = await cms.forRelease().resolve("Analytics"); // the settings, defaults filled in + const { analytics } = await cms.settings(); // the release's settings, defaults filled in return ( diff --git a/packages/blocks/src/index.ts b/packages/blocks/src/index.ts index eba90a742..06171cca0 100644 --- a/packages/blocks/src/index.ts +++ b/packages/blocks/src/index.ts @@ -8,11 +8,10 @@ export { type Client, type CMS, type CMSError, + type CMSSettings, createCMS, - DRAFT_COOKIE, type DraftPointer, - draftCookie, - draftPointer, + type EffectiveSettings, formatDraftPointer, type Lazy, type ListOptions, @@ -22,6 +21,7 @@ export { type Page, parseDraftPointer, type Redirect, + type RequestLike, type Result, type Route, remoteLoader, diff --git a/packages/blocks/src/protocol/__tests__/fixtures.ts b/packages/blocks/src/protocol/__tests__/fixtures.ts index 271061e72..1026b72ea 100644 --- a/packages/blocks/src/protocol/__tests__/fixtures.ts +++ b/packages/blocks/src/protocol/__tests__/fixtures.ts @@ -96,6 +96,3 @@ export const route = (rpc: (request: Request) => Promise, assets: (request: Request) => Promise) => (request: Request) => new URL(request.url).pathname.startsWith("/assets/") ? assets(request) : rpc(request); - -/** Limits low enough for the conformance suite to probe the list and batch-response bounds. */ -export const PROBE_LIMITS = { maxListBytes: 256 * 1024, maxBatchResponseBytes: 384 * 1024 }; diff --git a/packages/blocks/src/protocol/__tests__/fs.conformance.test.ts b/packages/blocks/src/protocol/__tests__/fs.conformance.test.ts index 23200ecbd..886dea638 100644 --- a/packages/blocks/src/protocol/__tests__/fs.conformance.test.ts +++ b/packages/blocks/src/protocol/__tests__/fs.conformance.test.ts @@ -13,7 +13,6 @@ import { createAssetHandler } from "../server/assets"; import { createFsStorage } from "../storage/fs"; import { generateSecretsKeyPair, - PROBE_LIMITS, route, SECRET_BLOCK, SECRET_FIELD, @@ -21,13 +20,11 @@ import { } from "./fixtures"; import { serve } from "./nodeServer"; -const TOKEN = "fs-conformance-token"; let PUBLIC_KEY: string; let root: string; let server: Awaited>; let readOnlyServer: Awaited>; -let schemaLimitedServer: Awaited>; beforeAll(async () => { PUBLIC_KEY = (await generateSecretsKeyPair()).publicKeyPem; @@ -36,25 +33,18 @@ beforeAll(async () => { await writeFile(join(root, ".deco", "schema.gen.json"), JSON.stringify(schemaFixture, null, 2)); await writeFile(join(root, ".deco", "secrets.pub"), PUBLIC_KEY); await writeFile(join(root, ".deco", "index.ts"), "export default {};\n"); - const storage = createFsStorage({ root, limits: PROBE_LIMITS }); - const handler = createContentHandler(storage, { - token: TOKEN, - server: { name: "deco-cli", version: "test" }, - }); - server = await serve(route(handler, createAssetHandler(storage, { token: TOKEN }))); + const storage = createFsStorage({ root }); + const handler = createContentHandler(storage, { server: { name: "deco-cli", version: "test" } }); + server = await serve(route(handler, createAssetHandler(storage))); const readOnlyStorage = createFsStorage({ root, readOnly: true }); readOnlyServer = await serve( route(createContentHandler(readOnlyStorage), createAssetHandler(readOnlyStorage)), ); - schemaLimitedServer = await serve( - createContentHandler(createFsStorage({ root, limits: { maxSchemaBytes: 128 } })), - ); }); afterAll(async () => { await server?.close(); await readOnlyServer?.close(); - await schemaLimitedServer?.close(); if (root) await rm(root, { recursive: true, force: true }); }); @@ -63,7 +53,6 @@ defineConformanceSuite( () => ({ endpoint: `${server.origin}/rpc`, assetsEndpoint: `${server.origin}/assets/`, - token: TOKEN, secretField: { blockType: SECRET_BLOCK, field: SECRET_FIELD }, secretsPublicKey: PUBLIC_KEY, }), @@ -79,10 +68,3 @@ defineConformanceSuite( }), "read-only filesystem storage conformance (over HTTP)", ); - -defineConformanceSuite( - { describe, it }, - () => ({ endpoint: `${schemaLimitedServer.origin}/rpc`, schemaOverLimit: true }), - "filesystem storage with a schema over its limit (over HTTP)", - (testCase) => testCase.id === "limits/schema-bytes", -); diff --git a/packages/blocks/src/protocol/__tests__/memory.conformance.test.ts b/packages/blocks/src/protocol/__tests__/memory.conformance.test.ts index d034de478..a319b574f 100644 --- a/packages/blocks/src/protocol/__tests__/memory.conformance.test.ts +++ b/packages/blocks/src/protocol/__tests__/memory.conformance.test.ts @@ -6,7 +6,6 @@ import { createAssetHandler } from "../server/assets"; import { createMemoryStorage, type MemoryStorage } from "../storage/memory"; import { generateSecretsKeyPair, - PROBE_LIMITS, route, SECRET_BLOCK, SECRET_FIELD, @@ -15,39 +14,19 @@ import { const { publicKeyPem: PUBLIC_KEY } = await generateSecretsKeyPair(); -let storage: MemoryStorage = createMemoryStorage({ +const storage: MemoryStorage = createMemoryStorage({ state: { schema: JSON.stringify(schemaFixture), secretsPublicKey: PUBLIC_KEY }, }); -let handler = createHandler(); - -function createHandler() { - const authorize = (request: Request) => { - const auth = request.headers.get("authorization"); - if (auth === "Bearer tenant-a") return { scope: "tenant-a" }; - if (auth === "Bearer tenant-b") return { scope: "tenant-b" }; - return "unauthorized" as const; - }; - return route( - createContentHandler(storage, { authorize, limits: PROBE_LIMITS }), - createAssetHandler(storage, { authorize }), - ); -} +const handler = route(createContentHandler(storage), createAssetHandler(storage)); defineConformanceSuite( { describe, it }, { endpoint: "http://memory.test/rpc", - token: "tenant-a", fetch: (request) => handler(request), assetsEndpoint: "http://memory.test/assets/", secretField: { blockType: SECRET_BLOCK, field: SECRET_FIELD }, secretsPublicKey: PUBLIC_KEY, - otherTenant: { token: "tenant-b" }, - restart: async () => { - // A new process: fresh handler, storage rebuilt from its durable state. - storage = createMemoryStorage({ state: storage.dump() }); - handler = createHandler(); - }, }, ); @@ -59,22 +38,6 @@ defineConformanceSuite( "schemaless memory storage conformance", ); -const schemaOverLimit = createContentHandler( - createMemoryStorage({ state: { schema: JSON.stringify(schemaFixture) } }), - { limits: { maxSchemaBytes: 128 } }, -); - -defineConformanceSuite( - { describe, it }, - { - endpoint: "http://memory.test/rpc", - fetch: (request) => schemaOverLimit(request), - schemaOverLimit: true, - }, - "memory storage with a schema over its limit", - (testCase) => testCase.id === "limits/schema-bytes", -); - const readOnlyStorage = createMemoryStorage({ description: { readOnly: true } }); const readOnly = route(createContentHandler(readOnlyStorage), createAssetHandler(readOnlyStorage)); diff --git a/packages/blocks/src/protocol/canonical.test.ts b/packages/blocks/src/protocol/canonical.test.ts index 5973c262a..f2b04108c 100644 --- a/packages/blocks/src/protocol/canonical.test.ts +++ b/packages/blocks/src/protocol/canonical.test.ts @@ -1,14 +1,7 @@ // @vitest-environment node import { createHash } from "node:crypto"; import { describe, expect, it } from "vitest"; -import { - APPLY_DIGEST_DOMAIN, - applyRequestDigest, - CanonicalJsonError, - canonicalJson, - computeContentRevision, - sha256Hex, -} from "./canonical"; +import { CanonicalJsonError, canonicalJson, computeContentRevision, sha256Hex } from "./canonical"; import { contentHashFixtures } from "./conformance/contentHashFixtures"; describe("canonicalJson", () => { @@ -67,13 +60,6 @@ describe("hashing", () => { "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad", ); }); - - it("binds a request digest to every parameter, with a domain prefix", async () => { - const a = await applyRequestDigest({ set: { x: { v: 1 } }, requestKey: "k" }); - expect(a).toBe(await applyRequestDigest({ requestKey: "k", set: { x: { v: 1 } } })); - expect(a).not.toBe(await applyRequestDigest({ set: { x: { v: 2 } }, requestKey: "k" })); - expect(a).toBe(await sha256Hex(`${APPLY_DIGEST_DOMAIN}{"requestKey":"k","set":{"x":{"v":1}}}`)); - }); }); describe("computeContentRevision: the golden fixtures", () => { diff --git a/packages/blocks/src/protocol/client.test.ts b/packages/blocks/src/protocol/client.test.ts index ed8e99564..79c6fb69b 100644 --- a/packages/blocks/src/protocol/client.test.ts +++ b/packages/blocks/src/protocol/client.test.ts @@ -9,12 +9,11 @@ const json = (body: unknown, status = 200) => new Response(JSON.stringify(body), { status, headers: { "content-type": "application/json" } }); describe("createContentClient", () => { - it("posts JSON-RPC 2.0 with the bearer token and extra headers", async () => { + it("posts JSON-RPC 2.0 with the extra headers", async () => { const fetch = vi.fn(async (request: Request) => { expect(request.method).toBe("POST"); expect(request.url).toBe("http://h/rpc"); expect(request.headers.get("content-type")).toBe("application/json"); - expect(request.headers.get("authorization")).toBe("Bearer t0k"); expect(request.headers.get("x-project")).toBe("p"); const body = await request.json(); expect(body).toEqual({ @@ -31,7 +30,6 @@ describe("createContentClient", () => { }); const client = createContentClient({ endpoint: "http://h/rpc", - token: "t0k", headers: { "x-project": "p" }, fetch, }); @@ -65,7 +63,7 @@ describe("createContentClient", () => { }); }); - it("throws the whole-request error of a 401 or 413", async () => { + it("throws the whole-request error of a 401 (from the host's own auth) or 413", async () => { for (const [status, code] of [ [401, ErrorCode.Unauthorized], [413, ErrorCode.LimitExceeded], diff --git a/packages/blocks/src/protocol/client.ts b/packages/blocks/src/protocol/client.ts index a910dfe5d..a79f56001 100644 --- a/packages/blocks/src/protocol/client.ts +++ b/packages/blocks/src/protocol/client.ts @@ -13,10 +13,8 @@ import { } from "./types.ts"; export interface ContentClientOptions { - /** The endpoint URL, such as `http://127.0.0.1:4545/rpc`. */ + /** The endpoint URL, such as `http://localhost:4545/rpc`. */ endpoint: string | URL; - /** Sent as `Authorization: Bearer `. */ - token?: string; /** Extra headers on every request. */ headers?: Record; /** A fetch implementation (defaults to the global one). */ @@ -100,7 +98,6 @@ export function createContentClient(options: ContentClientOptions): ContentClien accept: "application/json", ...options.headers, }; - if (options.token !== undefined) headers.authorization = `Bearer ${options.token}`; const response = await doFetch( new Request(options.endpoint, { method: "POST", headers, body: JSON.stringify(body) }), ); diff --git a/packages/blocks/src/protocol/conformance/cases/guards.ts b/packages/blocks/src/protocol/conformance/cases/guards.ts index 6cd50a592..7c4270ff2 100644 --- a/packages/blocks/src/protocol/conformance/cases/guards.ts +++ b/packages/blocks/src/protocol/conformance/cases/guards.ts @@ -1,8 +1,9 @@ /** - * Limits, the secret guard and request-key idempotency. + * `blocks.apply`'s own size guards and the secret guard. */ import { encryptToCiphertext, publicKeyPemFromDer } from "../../ciphertext.ts"; import { ErrorCode, type InvalidBlockData } from "../../errors.ts"; +import { MAX_BLOCK_BYTES, MAX_OPS_PER_APPLY, MAX_REQUEST_BYTES } from "../../server/core.ts"; import { assert, assertEqual, @@ -18,12 +19,6 @@ async function writable(ctx: ConformanceContext) { return d; } -async function withIdempotency(ctx: ConformanceContext) { - const d = await writable(ctx); - if (!d.writes.idempotency) return ctx.skip("request keys aren't advertised"); - return d; -} - /** A throwaway public key, for endpoints that don't report one. */ async function ephemeralPublicKey(): Promise { const pair = (await crypto.subtle.generateKey( @@ -45,17 +40,14 @@ async function realCiphertext(ctx: ConformanceContext, value: string): Promise - `${ctx.prefix}-${label}-${Math.random().toString(36).slice(2)}`; - export const guardCases: ConformanceCase[] = [ { id: "limits/ops-per-apply", - title: "more names than maxOpsPerApply is LimitExceeded", + title: "more names than one blocks.apply takes is LimitExceeded", async run(ctx) { - const d = await writable(ctx); + await writable(ctx); const names = Array.from( - { length: d.limits.maxOpsPerApply + 1 }, + { length: MAX_OPS_PER_APPLY + 1 }, (_, i) => `${ctx.prefix}-op-${i}`, ); await expectError( @@ -67,16 +59,14 @@ export const guardCases: ConformanceCase[] = [ }, { id: "limits/block-bytes", - title: "an entry over maxBlockBytes is refused and nothing is written", + title: "an entry over the block size limit is refused and nothing is written", async run(ctx) { - const d = await writable(ctx); - if (d.limits.maxBlockBytes + 1024 > d.limits.maxRequestBytes) - return ctx.skip("the request limit is lower"); + await writable(ctx); const big = ctx.name("big"); const small = ctx.name("small"); const error = await expectError( ctx.client.blocksApply({ - set: { [big]: { text: "x".repeat(d.limits.maxBlockBytes) }, [small]: { ok: true } }, + set: { [big]: { text: "x".repeat(MAX_BLOCK_BYTES) }, [small]: { ok: true } }, }), ErrorCode.InvalidBlock, "an oversized entry", @@ -92,106 +82,19 @@ export const guardCases: ConformanceCase[] = [ }, { id: "limits/request-bytes", - title: "a body over maxRequestBytes is HTTP 413 with LimitExceeded", + title: "a body over the request size limit is HTTP 413 with LimitExceeded", async run(ctx) { - const d = await ctx.describe(); - if (d.limits.maxRequestBytes > 64 * 1024 * 1024) - return ctx.skip("the request limit is too large to probe"); const body = JSON.stringify({ jsonrpc: "2.0", id: 1, method: "blocks.apply", - params: { set: { [ctx.name("huge")]: { text: "x".repeat(d.limits.maxRequestBytes) } } }, + params: { set: { [ctx.name("huge")]: { text: "x".repeat(MAX_REQUEST_BYTES) } } }, }); const response = await ctx.raw(body); assertEqual(response.status, 413, "HTTP status"); assertEqual(rawErrorCode(response.body), ErrorCode.LimitExceeded, "the error code"); }, }, - { - id: "limits/list-bytes", - title: "a block list over maxListBytes is LimitExceeded, never a partial map", - async run(ctx) { - const d = await writable(ctx); - const { maxListBytes, maxBlockBytes, maxOpsPerApply } = d.limits; - if (maxListBytes > 512 * 1024) return ctx.skip("maxListBytes is too large to probe"); - const size = Math.min(maxBlockBytes, 64 * 1024) - 64; - const count = Math.ceil(maxListBytes / size) + 1; - if (count > maxOpsPerApply * 4) return ctx.skip("would need too many writes"); - for (let i = 0; i < count; i++) { - await ctx.client.blocksApply({ set: { [ctx.name("fill")]: { text: "x".repeat(size) } } }); - } - await expectError(ctx.client.blocksList(), ErrorCode.LimitExceeded, "an oversized list"); - }, - }, - { - id: "limits/batch-response-bytes", - title: - "a batch response over maxBatchResponseBytes answers later reads with LimitExceeded, never a write", - async run(ctx) { - const d = await writable(ctx); - const { maxBatchResponseBytes, maxBlockBytes, maxListBytes } = d.limits; - if (maxBatchResponseBytes > 1024 * 1024) - return ctx.skip("maxBatchResponseBytes is too large to probe"); - // Enough content that one list fits the budget but four don't. - const target = Math.floor(maxBatchResponseBytes / 3); - if (target > maxListBytes / 2) return ctx.skip("maxListBytes is lower than the probe needs"); - const size = Math.min(maxBlockBytes, 64 * 1024) - 64; - for (let filled = 0; filled < target; filled += size) { - await ctx.client.blocksApply({ set: { [ctx.name("fill")]: { text: "x".repeat(size) } } }); - } - const write = ctx.name("batch-write"); - const req = (id: number, method: string, params: unknown) => ({ - jsonrpc: "2.0", - id, - method, - params, - }); - const response = await ctx.rpc([ - req(1, "blocks.list", {}), - req(2, "blocks.list", {}), - req(3, "blocks.list", {}), - req(4, "blocks.list", {}), - req(5, "blocks.apply", { set: { [write]: { kept: true } } }), - req(6, "blocks.list", {}), - ]); - const items = response.body as Array<{ - id: number; - result?: unknown; - error?: { code: number; data?: { limit?: string } }; - }>; - assert(Array.isArray(items) && items.length === 6, "one response per call"); - assert(!items[0].error, "the first read fits"); - const over = items.filter((i) => i.error?.code === ErrorCode.LimitExceeded); - assert(over.length > 0, "a read over the budget is LimitExceeded"); - assert( - over.every((i) => i.error?.data?.limit === "maxBatchResponseBytes"), - "the error names maxBatchResponseBytes", - ); - assert(items[4].result !== undefined, "the write still answers with its result"); - const list = await ctx.client.blocksList(); - assert(!list.notModified && write in list.blocks, "the write landed"); - }, - }, - { - id: "limits/schema-bytes", - title: "a schema over maxSchemaBytes is LimitExceeded for schema.get and for writes", - async run(ctx) { - if (!ctx.options.schemaOverLimit) - return ctx.skip("the harness serves a schema within limits"); - await expectError(ctx.client.schemaGet(), ErrorCode.LimitExceeded, "schema.get"); - const d = await ctx.describe(); - if (d.readOnly) return; - const name = ctx.name("schema-limited"); - await expectError( - ctx.client.blocksApply({ set: { [name]: { v: 1 } } }), - ErrorCode.LimitExceeded, - "a write checked against the schema", - ); - const list = await ctx.client.blocksList(); - assert(!list.notModified && !(name in list.blocks), "nothing was written"); - }, - }, { id: "secrets/guard", title: "a Secret field accepts only a secret block with a well-formed ciphertext", @@ -230,85 +133,4 @@ export const guardCases: ConformanceCase[] = [ assert(!list.notModified && ok in list.blocks, "an encrypted value is saved"); }, }, - { - id: "idempotency/unadvertised", - title: "a request key is refused when the endpoint doesn't advertise receipts", - async run(ctx) { - const d = await writable(ctx); - if (d.writes.idempotency) return ctx.skip("request keys are advertised"); - await expectError( - ctx.client.blocksApply({ requestKey: key(ctx, "k"), delete: [ctx.name()] }), - ErrorCode.Unsupported, - "a request key", - ); - }, - }, - { - id: "idempotency/retry", - title: "retrying with the same key returns the original result, even after the content moved", - async run(ctx) { - await withIdempotency(ctx); - const name = ctx.name("retry"); - const params = { requestKey: key(ctx, "retry"), set: { [name]: { v: 1 } } }; - const first = await ctx.client.blocksApply(params); - await ctx.client.blocksApply({ set: { [ctx.name("advance")]: { v: 1 } } }); - const second = await ctx.client.blocksApply(params); - assertEqual(second, first, "the replayed result"); - const list = await ctx.client.blocksList(); - assert(!list.notModified && list.versions[name] === first.versions[name], "written once"); - }, - }, - { - id: "idempotency/different-params", - title: "reusing a key for a different request is Invalid params", - async run(ctx) { - await withIdempotency(ctx); - const requestKey = key(ctx, "reuse"); - const name = ctx.name("reuse"); - await ctx.client.blocksApply({ requestKey, set: { [name]: { v: 1 } } }); - await expectError( - ctx.client.blocksApply({ requestKey, set: { [name]: { v: 2 } } }), - ErrorCode.InvalidParams, - "a different body", - ); - }, - }, - { - id: "idempotency/simultaneous", - title: "simultaneous duplicates commit once and share one result", - async run(ctx) { - await withIdempotency(ctx); - const params = { requestKey: key(ctx, "dup"), set: { [ctx.name("dup")]: { v: 1 } } }; - const results = await Promise.all([1, 2, 3].map(() => ctx.client.blocksApply(params))); - assertEqual(results[1], results[0], "the second result"); - assertEqual(results[2], results[0], "the third result"); - }, - }, - { - id: "idempotency/restart", - title: "receipts survive a server restart", - async run(ctx) { - await withIdempotency(ctx); - if (!ctx.options.restart) return ctx.skip("no restart hook"); - const params = { requestKey: key(ctx, "restart"), set: { [ctx.name("restart")]: { v: 1 } } }; - const first = await ctx.client.blocksApply(params); - await ctx.options.restart(); - assertEqual(await ctx.client.blocksApply(params), first, "the replayed result"); - }, - }, - { - id: "idempotency/tenant-isolation", - title: "one tenant's receipts never answer another tenant's requests", - async run(ctx) { - await withIdempotency(ctx); - if (!ctx.options.otherTenant) return ctx.skip("no otherTenant credentials"); - const requestKey = key(ctx, "tenant"); - const a = ctx.name("tenant-a"); - const b = ctx.name("tenant-b"); - await ctx.client.blocksApply({ requestKey, set: { [a]: { v: 1 } } }); - const other = ctx.clientFor(ctx.options.otherTenant); - const result = await other.blocksApply({ requestKey, set: { [b]: { v: 1 } } }); - assert(typeof result.versions[b] === "string", "the other tenant's write ran"); - }, - }, ]; diff --git a/packages/blocks/src/protocol/conformance/cases/reads.ts b/packages/blocks/src/protocol/conformance/cases/reads.ts index e5039147f..fb31b6911 100644 --- a/packages/blocks/src/protocol/conformance/cases/reads.ts +++ b/packages/blocks/src/protocol/conformance/cases/reads.ts @@ -1,24 +1,14 @@ /** * `describe`, `schema.get` and `blocks.list`, including conditional reads. */ -import { ErrorCode } from "../../errors.ts"; import { PROTOCOL_NAME, SCHEMA_FORMAT } from "../../types.ts"; -import { assert, assertEqual, expectError } from "../context.ts"; +import { assert, assertEqual } from "../context.ts"; import type { ConformanceCase } from "./types.ts"; -const LIMIT_KEYS = [ - "maxOpsPerApply", - "maxBlockBytes", - "maxRequestBytes", - "maxListBytes", - "maxSchemaBytes", - "maxBatchResponseBytes", -] as const; - export const readCases: ConformanceCase[] = [ { id: "describe/shape", - title: "describe reports the protocol, its version, kind, root, limits and features", + title: "describe reports the protocol, its version, kind, root and features", async run(ctx) { const d = await ctx.client.describe(); assertEqual(d.protocol, PROTOCOL_NAME, "protocol"); @@ -29,14 +19,7 @@ export const readCases: ConformanceCase[] = [ assert(typeof d.readOnly === "boolean", "readOnly"); assert(typeof d.root === "string" && !d.root.startsWith("/"), "root is relative"); assertEqual(d.schemaFormat, SCHEMA_FORMAT, "schemaFormat"); - assert(d.refs === null || typeof d.refs.default === "string", "refs"); - assert( - d.writes.idempotency === null || d.writes.idempotency.retentionMs > 0, - "writes.idempotency", - ); - assert(typeof d.writes.schemaPreconditions === "boolean", "writes.schemaPreconditions"); assert(d.pollIntervalMs > 0, "pollIntervalMs"); - for (const key of LIMIT_KEYS) assert(d.limits[key] > 0, `limits.${key}`); assert( d.preview === null || /^https?:\/\//.test(String(d.preview?.url)), "preview is null or { url } with an http(s) URL", @@ -63,12 +46,9 @@ export const readCases: ConformanceCase[] = [ id: "schema/read", title: "schema.get returns the schema and a version, and 'not modified' for that version", async run(ctx) { - if (ctx.options.hasSchema === false) { - await expectError(ctx.client.schemaGet(), ErrorCode.NotFound, "no schema"); - return; - } + if (ctx.options.hasSchema === false) return ctx.skip("no schema (see schema/absent)"); const first = await ctx.client.schemaGet(); - assert(!first.notModified, "the first read returns the schema"); + assert(!first.notModified && first.schema !== null, "the first read returns the schema"); assert(typeof first.version === "string" && first.version.length > 0, "a version"); assert(typeof first.schema === "object" && first.schema !== null, "a schema object"); const again = await ctx.client.schemaGet({ ifNoneMatch: first.version }); @@ -77,6 +57,26 @@ export const readCases: ConformanceCase[] = [ assert(!stale.notModified, "another version returns the schema"); }, }, + { + id: "schema/absent", + title: "without a schema, schema.get is schema: null (not an error) and blocks still list", + async run(ctx) { + if (ctx.options.hasSchema !== false) return ctx.skip("the endpoint has a schema"); + const result = await ctx.client.schemaGet(); + assertEqual( + result, + { + notModified: false, + version: null, + resolvedRef: result.notModified ? null : result.resolvedRef, + schema: null, + }, + "the no-schema result", + ); + const list = await ctx.client.blocksList(); + assert(!list.notModified, "blocks.list returns the map without a schema"); + }, + }, { id: "list/read", title: "blocks.list returns every entry with versions, and 'not modified' for its revision", @@ -90,8 +90,10 @@ export const readCases: ConformanceCase[] = [ Object.keys(first.blocks).sort(), "one version per entry", ); - const d = await ctx.describe(); - if (d.refs === null) assertEqual(first.resolvedRef, null, "resolvedRef without refs"); + assert( + first.resolvedRef === null || typeof first.resolvedRef === "string", + "resolvedRef is a branch or null", + ); const again = await ctx.client.blocksList({ ifNoneMatch: first.revision }); assertEqual( again, @@ -112,7 +114,7 @@ export const readCases: ConformanceCase[] = [ : [ { method: "schema.get" as const, - params: { ifNoneMatch: (await ctx.client.schemaGet()).version }, + params: { ifNoneMatch: (await ctx.client.schemaGet()).version ?? undefined }, }, ...calls, ], @@ -123,46 +125,4 @@ export const readCases: ConformanceCase[] = [ } }, }, - { - id: "refs/unsupported", - title: "a ref on an endpoint without branches is Unsupported", - async run(ctx) { - const d = await ctx.describe(); - if (d.refs !== null) return ctx.skip("the endpoint has branches"); - await expectError( - ctx.client.blocksList({ ref: "main" }), - ErrorCode.Unsupported, - "blocks.list ref", - ); - await expectError( - ctx.client.schemaGet({ ref: "main" }), - ErrorCode.Unsupported, - "schema.get ref", - ); - if (!d.readOnly) { - await expectError( - ctx.client.blocksApply({ ref: "main", delete: [ctx.name()] }), - ErrorCode.Unsupported, - "blocks.apply ref", - ); - } - }, - }, - { - id: "auth/token", - title: "a missing or wrong bearer token is HTTP 401 with Unauthorized", - async run(ctx) { - if (ctx.options.token === undefined) return ctx.skip("the endpoint has no token"); - const body = JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }); - for (const authorization of ["", "Bearer wrong-token"]) { - const response = await ctx.raw(body, { headers: { authorization } }); - assertEqual(response.status, 401, `HTTP status with "${authorization}"`); - assertEqual( - (response.body as { error?: { code: number } }).error?.code, - ErrorCode.Unauthorized, - "the error code", - ); - } - }, - }, ]; diff --git a/packages/blocks/src/protocol/conformance/cases/writes.ts b/packages/blocks/src/protocol/conformance/cases/writes.ts index a9e554763..a14ccb6c4 100644 --- a/packages/blocks/src/protocol/conformance/cases/writes.ts +++ b/packages/blocks/src/protocol/conformance/cases/writes.ts @@ -273,47 +273,19 @@ export const writeCases: ConformanceCase[] = [ assert(!(from in after.blocks) && to in after.blocks, "the entry moved"); }, }, - { - id: "apply/schema-precondition", - title: "ifSchemaMatch rejects a write when the schema changed", - async run(ctx) { - const d = await writable(ctx); - const name = ctx.name("schema-guarded"); - if (!d.writes.schemaPreconditions) { - await expectError( - ctx.client.blocksApply({ set: { [name]: {} }, ifSchemaMatch: "x" }), - ErrorCode.Unsupported, - "an unadvertised guard", - ); - return; - } - if (ctx.options.hasSchema === false) return ctx.skip("no schema"); - const { version } = await ctx.client.schemaGet(); - const error = await expectError( - ctx.client.blocksApply({ set: { [name]: {} }, ifSchemaMatch: `${version}-old` }), - ErrorCode.Conflict, - "a stale schema version", - ); - assertEqual( - (error.data as { schema?: unknown }).schema, - { expected: `${version}-old`, actual: version }, - "the conflict's schema versions", - ); - assert(!(name in (await list(ctx)).blocks), "nothing was written"); - await ctx.client.blocksApply({ set: { [name]: {} }, ifSchemaMatch: version }); - }, - }, { id: "apply/unknown-params", title: "an unknown parameter is refused, so an unknown guard never becomes an unguarded write", async run(ctx) { await writable(ctx); const name = ctx.name("unknown-param"); - await expectError( - ctx.client.call("blocks.apply", { set: { [name]: {} }, ifUnmodifiedSince: "x" } as never), - ErrorCode.InvalidParams, - "an unknown guard", - ); + for (const param of ["ifUnmodifiedSince", "ifSchemaMatch", "requestKey", "ref"]) { + await expectError( + ctx.client.call("blocks.apply", { set: { [name]: {} }, [param]: "x" } as never), + ErrorCode.InvalidParams, + `the unknown parameter ${param}`, + ); + } assert(!(name in (await list(ctx)).blocks), "nothing was written"); }, }, diff --git a/packages/blocks/src/protocol/conformance/context.ts b/packages/blocks/src/protocol/conformance/context.ts index 27c7968db..8af50f018 100644 --- a/packages/blocks/src/protocol/conformance/context.ts +++ b/packages/blocks/src/protocol/conformance/context.ts @@ -7,12 +7,13 @@ import { type ContentClient, createContentClient, readResponseJson } from "../cl import { ContentProtocolError } from "../errors.ts"; import type { DescribeResult } from "../types.ts"; +/** Names deleted per `blocks.apply` when a case cleans up. */ +const CLEANUP_CHUNK = 100; + export interface ConformanceOptions { - /** The endpoint URL, such as `http://127.0.0.1:4545/rpc`. */ + /** The endpoint URL, such as `http://localhost:4545/rpc`. */ endpoint: string | URL; - /** The bearer token the endpoint expects, if any. */ - token?: string; - /** Extra headers on every request (an auth cookie, a tenant header). */ + /** Extra headers on every request (such as the host's auth cookie). */ headers?: Record; /** A fetch implementation (defaults to the global one); use it to call a handler in process. */ fetch?: (request: Request) => Promise; @@ -23,28 +24,13 @@ export interface ConformanceOptions { secretField?: { blockType: string; field: string }; /** The `.deco/secrets.pub` content the endpoint should report, if the harness wrote one. */ secretsPublicKey?: string; - /** Whether the endpoint has a schema (default true). `false` checks NotFound instead. */ + /** Whether the endpoint has a schema (default true). `false` checks the `schema: null` result instead. */ hasSchema?: boolean; - /** - * The endpoint serves a schema larger than its `limits.maxSchemaBytes`, so - * reading it and writes that depend on it must be LimitExceeded. Run such - * an endpoint with only the `limits/schema-bytes` case (pass a filter to - * `runConformance` or `defineConformanceSuite`): every other case expects - * a schema it can read. - */ - schemaOverLimit?: boolean; /** * The URL prefix uploads are served under (`PUT `), - * such as `http://127.0.0.1:4545/assets/`. Leave out to skip the upload cases. + * such as `http://localhost:4545/assets/`. Leave out to skip the upload cases. */ assetsEndpoint?: string | URL; - /** - * Restarts the server, keeping its durable state, to check that request-key - * receipts survive a restart. Leave out to skip that case. - */ - restart?: () => Promise; - /** Credentials of another tenant on the same storage, to check receipt isolation. */ - otherTenant?: { token?: string; headers?: Record }; /** Prefix of every name the suite creates (default `conformance-`). */ namePrefix?: string; } @@ -83,11 +69,6 @@ export class ConformanceContext { this.client = createContentClient(options); } - /** A client with other credentials (for tenant isolation). */ - clientFor(credentials: { token?: string; headers?: Record }): ContentClient { - return createContentClient({ ...this.options, ...credentials }); - } - async describe(): Promise { this.describeResult ??= await this.client.describe(); return this.describeResult; @@ -112,10 +93,8 @@ export class ConformanceContext { this.created.clear(); const description = await this.describe(); if (description.readOnly) return; - for (let i = 0; i < names.length; i += description.limits.maxOpsPerApply) { - await this.client - .blocksApply({ delete: names.slice(i, i + description.limits.maxOpsPerApply) }) - .catch(() => {}); + for (let i = 0; i < names.length; i += CLEANUP_CHUNK) { + await this.client.blocksApply({ delete: names.slice(i, i + CLEANUP_CHUNK) }).catch(() => {}); } } @@ -133,9 +112,6 @@ export class ConformanceContext { ...this.options.headers, ...init.headers, }; - if (this.options.token !== undefined && !("authorization" in headers)) { - headers.authorization = `Bearer ${this.options.token}`; - } for (const [key, value] of Object.entries(headers)) if (value === "") delete headers[key]; const method = init.method ?? "POST"; const doFetch = this.options.fetch ?? ((request: Request) => fetch(request)); @@ -163,7 +139,6 @@ export class ConformanceContext { ...this.options.headers, "content-type": contentType, }; - if (this.options.token !== undefined) headers.authorization = `Bearer ${this.options.token}`; const doFetch = this.options.fetch ?? ((request: Request) => fetch(request)); const url = new URL(encodeURIComponent(name), new URL(String(base)).href.replace(/\/?$/, "/")); const response = await doFetch( diff --git a/packages/blocks/src/protocol/conformance/index.ts b/packages/blocks/src/protocol/conformance/index.ts index a412dcca1..1d54f5085 100644 --- a/packages/blocks/src/protocol/conformance/index.ts +++ b/packages/blocks/src/protocol/conformance/index.ts @@ -10,14 +10,14 @@ * import { describe, it } from "vitest"; * import { defineConformanceSuite } from "@decocms/blocks/protocol/conformance"; * - * defineConformanceSuite({ describe, it }, { endpoint: "http://127.0.0.1:4545/rpc", token }); + * defineConformanceSuite({ describe, it }, { endpoint: "http://localhost:4545/rpc" }); * ``` * * or collect a report with `runConformance(options)`. The suite writes only * names under its own prefix and deletes them after each case (uploads, which * the protocol can't delete, stay in the asset folder). Cases that - * don't apply to an endpoint (a read-only one, one without request keys) - * skip themselves. Browser-safe: it only needs `fetch`. + * don't apply to an endpoint (a read-only one, one without a schema) skip + * themselves. Browser-safe: it only needs `fetch`. */ import { assetCases } from "./cases/assets.ts"; import { guardCases } from "./cases/guards.ts"; diff --git a/packages/blocks/src/protocol/errors.ts b/packages/blocks/src/protocol/errors.ts index ccc95fbfa..40f0bccfb 100644 --- a/packages/blocks/src/protocol/errors.ts +++ b/packages/blocks/src/protocol/errors.ts @@ -1,7 +1,9 @@ /** * Content-protocol errors. Every failure travels as a JSON-RPC 2.0 `error` - * object (HTTP 200), except a missing or invalid bearer token (401) and a body - * over the size limit (413), which apply to the whole batch. + * object (HTTP 200), except a body over the size limit (413), which applies + * to the whole batch. `Unauthorized` and `Forbidden` are for the server that + * mounts the handler and authenticates its callers (Studio's session); the + * handler itself never raises them. */ export const ErrorCode = { @@ -41,8 +43,7 @@ export interface VersionMismatch { /** `error.data` of a Conflict (-32002). */ export interface ConflictData { - entries?: Record; - schema?: { expected: string; actual: string | null }; + entries: Record; } /** One rule a `blocks.apply` breaks. */ @@ -134,9 +135,3 @@ export const parseError = () => new ContentProtocolError(ErrorCode.ParseError, " export const internalError = () => new ContentProtocolError(ErrorCode.InternalError, "internal error"); - -export const unauthorized = () => - new ContentProtocolError(ErrorCode.Unauthorized, "missing or invalid bearer token"); - -export const forbidden = () => - new ContentProtocolError(ErrorCode.Forbidden, "not allowed for this project"); diff --git a/packages/blocks/src/protocol/index.ts b/packages/blocks/src/protocol/index.ts index 41c755f02..d84c4cd7c 100644 --- a/packages/blocks/src/protocol/index.ts +++ b/packages/blocks/src/protocol/index.ts @@ -14,11 +14,10 @@ * * The SDK's runtime never imports the protocol, so it never reaches an app * bundle. The canonical hash it re-exports is the SDK's own dependency-free - * leaf module (see ./canonical.ts). + * leaf module (see ./canonical.ts). Draft previews and the draft lifecycle + * are outside the protocol (see /next/content-delivery#draft-previews). */ export { - APPLY_DIGEST_DOMAIN, - applyRequestDigest, CanonicalJsonError, CONTENT_HASH_FORMAT, canonicalJson, diff --git a/packages/blocks/src/protocol/params.test.ts b/packages/blocks/src/protocol/params.test.ts index 7c1741765..23709f0c6 100644 --- a/packages/blocks/src/protocol/params.test.ts +++ b/packages/blocks/src/protocol/params.test.ts @@ -38,7 +38,7 @@ describe("validateParams", () => { expect(code(() => validateParams("blocks.apply", { ifMatch: { a: 1 } }))).toBe( ErrorCode.InvalidParams, ); - expect(code(() => validateParams("blocks.apply", { requestKey: "" }))).toBe( + expect(code(() => validateParams("blocks.apply", { ifMatch: { a: "" } }))).toBe( ErrorCode.InvalidParams, ); expect(code(() => validateParams("blocks.list", { ifNoneMatch: 5 }))).toBe( diff --git a/packages/blocks/src/protocol/params.ts b/packages/blocks/src/protocol/params.ts index 69a649607..35574c947 100644 --- a/packages/blocks/src/protocol/params.ts +++ b/packages/blocks/src/protocol/params.ts @@ -13,14 +13,10 @@ import type { BlocksApplyParams, MethodName, ReadParams } from "./types.ts"; const opaque = z.string().min(1).max(1024); const readParams = z.strictObject({ - ref: z.string().min(1).max(255).optional(), ifNoneMatch: opaque.optional(), }); const applyParams = z.strictObject({ - ref: z.string().min(1).max(255).optional(), - requestKey: z.string().min(1).max(256).optional(), - ifSchemaMatch: opaque.optional(), // Entry shapes are checked later and reported as InvalidBlock, all at once. set: z.record(z.string(), z.unknown()).optional(), delete: z.array(z.string()).optional(), diff --git a/packages/blocks/src/protocol/secrets.test.ts b/packages/blocks/src/protocol/secrets.test.ts index e71458b08..a8f191216 100644 --- a/packages/blocks/src/protocol/secrets.test.ts +++ b/packages/blocks/src/protocol/secrets.test.ts @@ -161,6 +161,38 @@ describe("the secret guard", () => { ); }); + it("leaves a legacy secret loader block's encrypted string alone", () => { + const loader = "website/loaders/secret.ts"; + const meta = { + manifest: { + blocks: { + ...schemaFixture.manifest?.blocks, + loaders: { [loader]: { $ref: "#/definitions/djc=" } }, + }, + }, + schema: { + definitions: { + ...schemaFixture.schema?.definitions, + "djc=": { + type: "object", + properties: { + name: { type: "string" }, + encrypted: { type: "string", format: "secret" }, + }, + }, + }, + }, + }; + const entry = { + __resolveType: "settings", + apps: { __resolveType: loader, name: "API_KEY", encrypted: "0a1b2c" }, + }; + expect(rules(entry, meta)).toEqual([]); + expect(rules({ __resolveType: loader, encrypted: secretBlock("bad") }, meta)).toEqual([ + "secret-ciphertext@/encrypted", + ]); + }); + it("names the entry in every violation", () => { const [violation] = checkSecrets("Newsletter", newsletter("x"), schemaFixture); expect(violation).toMatchObject({ diff --git a/packages/blocks/src/protocol/secrets.ts b/packages/blocks/src/protocol/secrets.ts index a6f4a671e..970ee5a24 100644 --- a/packages/blocks/src/protocol/secrets.ts +++ b/packages/blocks/src/protocol/secrets.ts @@ -9,7 +9,8 @@ * entry must carry a well-formed `ciphertext`, whatever its field. * * The schema marks a `Secret` field with `"format": "secret"` — the contract - * `deco schema` emits for the `Secret` type. Browser-safe. + * `deco schema` emits for the `Secret` type. A legacy secret loader block + * (`website/loaders/secret.ts`) is left alone. Browser-safe. */ import { parseCiphertext } from "./ciphertext.ts"; import type { BlockViolation } from "./errors.ts"; @@ -23,6 +24,12 @@ export const SECRET_FORMAT = "secret"; const MULTIVARIATE_TYPES = new Set(["multivariate", "website/flags/multivariate.ts"]); const LAZY_TYPE = "lazy"; +/** + * The legacy (v7) secret loader. Its `encrypted` prop is a string marked + * `"format": "secret"` that already holds the site's own ciphertext, so it + * isn't a `Secret` field; its blocks are migrated, not guarded. + */ +const LEGACY_SECRET_LOADER = /(^|\/)loaders\/secret\.ts$/; const MAX_DEPTH = 512; type Json = Record; @@ -180,7 +187,8 @@ class SecretWalker { 'a "secret" block must carry a well-formed "ciphertext" (v1...)', ); } - const definition = this.meta ? blockIndex(this.meta).get(type) : undefined; + const definition = + this.meta && !LEGACY_SECRET_LOADER.test(type) ? blockIndex(this.meta).get(type) : undefined; this.walkObject(block, definition, pointer, depth); } diff --git a/packages/blocks/src/protocol/server/apply.test.ts b/packages/blocks/src/protocol/server/apply.test.ts index 4df539cbe..a1d1fbda0 100644 --- a/packages/blocks/src/protocol/server/apply.test.ts +++ b/packages/blocks/src/protocol/server/apply.test.ts @@ -85,42 +85,15 @@ describe("two spellings of one name under guards", () => { expect(result.versions).toEqual({ "a b": null, "a%20b": null }); expect(storage.dump().files).toEqual({}); }); - - it("replays the same versions, deleted spellings included, for a retried request key", async () => { - const { client } = setup(encodedOnly); - const params = { requestKey: "k1", set: { "a b": { v: 2 } } }; - const first = await client.blocksApply(params); - expect(first.versions["a%20b"]).toBeNull(); - expect(await client.blocksApply(params)).toEqual(first); - }); }); describe("the schema a write depends on", () => { - it("rejects ifSchemaMatch when the schema changes between the check and the commit", async () => { - let changed = false; - const { client, storage } = setup({ - beforeCommit: (_attempt, s) => { - if (changed) return; - changed = true; - s.setSchema({ ...schemaFixture, changed: true }); - }, - }); - const { version } = await client.schemaGet(); - const error = await rejects( - client.blocksApply({ set: { a: {} }, ifSchemaMatch: version }), - ErrorCode.Conflict, - ); - expect(error.data).toEqual({ schema: { expected: version, actual: expect.any(String) } }); - expect(storage.dump().files).toEqual({}); - }); - it("passes the schema version to the commit, so the storage checks it atomically", async () => { const attempts: Array = []; const { client } = setup({ beforeCommit: (a) => void attempts.push(a.expectedSchemaVersion) }); const { version } = await client.schemaGet(); await client.blocksApply({ set: { a: {} } }); await client.blocksApply({ delete: ["a"] }); - await client.blocksApply({ delete: ["b"], ifSchemaMatch: version }); // A set depends on the schema (the secret guard); a plain delete doesn't. expect(attempts).toEqual([version, undefined]); }); @@ -236,23 +209,6 @@ describe("commit retries", () => { }); }); -describe("response budgets", () => { - const names = Object.fromEntries(Array.from({ length: 20 }, (_, i) => [`entry-${i}`, {}])); - - it("never replaces a single write's response, even over maxBatchResponseBytes", async () => { - const { client, storage } = setup({}, { limits: { maxBatchResponseBytes: 64 } }); - const result = await client.blocksApply({ set: names }); - expect(Object.keys(result.versions)).toHaveLength(20); - expect(storage.commits).toBe(1); - }); - - it("still answers a single read over maxBatchResponseBytes with LimitExceeded", async () => { - const { client } = setup({}, { limits: { maxBatchResponseBytes: 64 } }); - const error = await rejects(client.schemaGet(), ErrorCode.LimitExceeded); - expect(error.data).toEqual({ limit: "maxBatchResponseBytes" }); - }); -}); - describe("names a storage can't hold", () => { it("refuses a leading-dot name before any storage sees it", async () => { const { client, storage } = setup(); diff --git a/packages/blocks/src/protocol/server/assets.test.ts b/packages/blocks/src/protocol/server/assets.test.ts index 58f5b1d22..4fd03fac7 100644 --- a/packages/blocks/src/protocol/server/assets.test.ts +++ b/packages/blocks/src/protocol/server/assets.test.ts @@ -10,9 +10,9 @@ import { ErrorCode } from "../errors"; import { createMemoryStorage, type MemoryStorageOptions } from "../storage/memory"; import { createAssetHandler } from "./assets"; -function setup(options: MemoryStorageOptions = {}, token?: string, allowSvg?: boolean) { +function setup(options: MemoryStorageOptions = {}, allowSvg?: boolean) { const storage = createMemoryStorage(options); - return { storage, handler: createAssetHandler(storage, { token, allowSvg }) }; + return { storage, handler: createAssetHandler(storage, { allowSvg }) }; } const put = ( @@ -106,27 +106,11 @@ describe("PUT /assets/", () => { it("refuses SVG (it can carry scripts) unless allowSvg is set", async () => { const svg = ''; expect((await setup().handler(put("logo.svg", svg, "image/svg+xml"))).status).toBe(415); - const allowed = await setup({}, undefined, true).handler(put("logo.svg", svg, "image/svg+xml")); + const allowed = await setup({}, true).handler(put("logo.svg", svg, "image/svg+xml")); expect(allowed.status).toBe(201); expect(await allowed.json()).toEqual({ path: "/assets/logo.svg" }); }); - it("refuses an empty token at construction", () => { - expect(() => createAssetHandler(createMemoryStorage(), { token: "" })).toThrow(TypeError); - }); - - it("checks the same bearer token as the protocol", async () => { - const { handler } = setup({}, "tok"); - expect((await handler(put("a.png", new Uint8Array([1])))).status).toBe(401); - expect( - ( - await handler( - put("a.png", new Uint8Array([1]), "image/png", { authorization: "Bearer tok" }), - ) - ).status, - ).toBe(201); - }); - it("refuses uploads over the advertised size with HTTP 413", async () => { const { handler } = setup({ description: { assets: { dir: "public/assets", maxBytes: 4 } } }); const response = await handler(put("a.png", new Uint8Array(5))); diff --git a/packages/blocks/src/protocol/server/assets.ts b/packages/blocks/src/protocol/server/assets.ts index 494688a4f..9b79d1661 100644 --- a/packages/blocks/src/protocol/server/assets.ts +++ b/packages/blocks/src/protocol/server/assets.ts @@ -2,7 +2,7 @@ * `createAssetHandler(storage)`: uploads beside the content protocol. * * Uploads aren't one of the four methods. The site editor sends each file to - * `PUT /assets/` with the same token as the protocol; on that path the + * `PUT /assets/`; on that path the * file's own image, video, font or PDF content type is accepted instead of * JSON, and the name's extension must match it (`banner.html` sent as * `image/png` is refused; a name without an extension gets the type's). SVG @@ -21,21 +21,16 @@ import { import { type ContentProtocolError, ErrorCode, - forbidden, invalidRequest, limitExceeded, readOnly, - unauthorized, unsupported, } from "../errors.ts"; import type { ContentStorage } from "../storage.ts"; import { ASSETS_URL_PREFIX } from "../types.ts"; -import { type AuthOptions, assertAuthOptions, authenticate } from "./auth.ts"; import { BodyEncodingError, BodyTooLargeError, jsonResponse, readBody } from "./http.ts"; const STATUS: Record = { - [ErrorCode.Unauthorized]: 401, - [ErrorCode.Forbidden]: 403, [ErrorCode.ReadOnly]: 403, [ErrorCode.Unsupported]: 404, [ErrorCode.LimitExceeded]: 413, @@ -44,13 +39,12 @@ const STATUS: Record = { export type AssetHandler = (request: Request) => Promise; -export type AssetHandlerOptions = AuthOptions & AssetTypeOptions; +export type AssetHandlerOptions = AssetTypeOptions; export function createAssetHandler( storage: ContentStorage, options: AssetHandlerOptions = {}, ): AssetHandler { - assertAuthOptions(options); const fail = (request: Request, error: ContentProtocolError, status?: number) => jsonResponse( request, @@ -69,10 +63,6 @@ export function createAssetHandler( }, ); } - const auth = await authenticate(request, options); - if (!auth.ok) - return fail(request, auth.reason === "unauthorized" ? unauthorized() : forbidden()); - const description = await storage.describe(); if (description.readOnly) return fail(request, readOnly()); if (description.assets === null || !storage.putAsset) { diff --git a/packages/blocks/src/protocol/server/auth.ts b/packages/blocks/src/protocol/server/auth.ts deleted file mode 100644 index 94f6e8706..000000000 --- a/packages/blocks/src/protocol/server/auth.ts +++ /dev/null @@ -1,49 +0,0 @@ -/** - * Request authorization shared by the content handler and the asset handler. - */ -import { bearerToken, timingSafeEqualStrings } from "./http.ts"; - -/** - * What an `authorize` hook decides: - * - `true` or `{ scope }` — allowed. `scope` names the tenant and principal, - * and scopes request-key receipts, so one tenant never replays another's. - * - `false` or `"unauthorized"` — a missing or invalid bearer token (HTTP 401). - * - `"forbidden"` — authenticated, but not allowed for this project. - */ -export type AuthorizeResult = boolean | "unauthorized" | "forbidden" | { scope?: string }; - -export interface AuthOptions { - /** When set, every request must carry `Authorization: Bearer `. */ - token?: string; - /** Decides whether a request may proceed (runs after the token check). */ - authorize?: (request: Request) => AuthorizeResult | Promise; -} - -/** - * Refuses options that would lock every client out by accident: an empty - * token can never be presented, so every request would be 401. - */ -export function assertAuthOptions(options: AuthOptions): void { - if (options.token !== undefined && (typeof options.token !== "string" || options.token === "")) { - throw new TypeError("token must be a non-empty string; omit it to accept requests without one"); - } -} - -type AuthOutcome = - | { ok: true; scope: string } - | { ok: false; reason: "unauthorized" | "forbidden" }; - -export async function authenticate(request: Request, options: AuthOptions): Promise { - if (options.token !== undefined) { - const presented = bearerToken(request); - if (presented === null || !(await timingSafeEqualStrings(presented, options.token))) { - return { ok: false, reason: "unauthorized" }; - } - } - if (!options.authorize) return { ok: true, scope: "" }; - const result = await options.authorize(request); - if (result === true) return { ok: true, scope: "" }; - if (result === false || result === "unauthorized") return { ok: false, reason: "unauthorized" }; - if (result === "forbidden") return { ok: false, reason: "forbidden" }; - return { ok: true, scope: result.scope ?? "" }; -} diff --git a/packages/blocks/src/protocol/server/content.test.ts b/packages/blocks/src/protocol/server/content.test.ts index 766ccdc68..31869b291 100644 --- a/packages/blocks/src/protocol/server/content.test.ts +++ b/packages/blocks/src/protocol/server/content.test.ts @@ -40,7 +40,7 @@ describe("a file that changes while bodies are read", () => { }); const result = await list(storage); expect(result.blocks.b).toEqual({ v: 2 }); - const fresh = await storage.snapshot({}); + const fresh = await storage.snapshot(); expect(result.revision).toBe(fresh.revision); expect(result.versions.b).toBe(fresh.files.find((f) => f.file === "b.json")!.version); }); @@ -91,7 +91,7 @@ describe("a file that changes while bodies are read", () => { }); const result = await list(storage); expect(Object.keys(result.blocks)).toEqual(["a"]); - expect(result.revision).toBe((await storage.snapshot({})).revision); + expect(result.revision).toBe((await storage.snapshot()).revision); }); it("detects the race with a storage whose readFiles returns changed content", async () => { diff --git a/packages/blocks/src/protocol/server/content.ts b/packages/blocks/src/protocol/server/content.ts index 3bed20fe1..0f17ffad3 100644 --- a/packages/blocks/src/protocol/server/content.ts +++ b/packages/blocks/src/protocol/server/content.ts @@ -4,11 +4,12 @@ * Shared by `blocks.list` (which needs every body) and `blocks.apply` (which * needs bodies only to break ties between spellings). */ -import { limitExceeded, unavailable } from "../errors.ts"; +import { unavailable } from "../errors.ts"; import { entryHasPath, fullyDecodeFileName, isBlockFileName, resolveSpellings } from "../keys.ts"; import type { ContentStorage, StorageFile, StorageSnapshot } from "../storage.ts"; -import type { Diagnostic, Limits } from "../types.ts"; +import type { Diagnostic } from "../types.ts"; import { type BodyCache, type ParsedBody, parseBody } from "./bodyCache.ts"; +import { MAX_BLOCK_BYTES } from "./core.ts"; export interface LoadedEntry { file: string; @@ -38,14 +39,13 @@ export interface LoadedContent { interface Options { /** Read every body (`blocks.list`), not only the ones that break ties. */ readAll: boolean; - limits: Limits; cache: BodyCache; } async function loadContent( storage: ContentStorage, snapshot: StorageSnapshot, - { readAll, limits, cache }: Options, + { readAll, cache }: Options, ): Promise { const files = snapshot.files.filter((f) => isBlockFileName(f.file)); const groups = new Map(); @@ -60,35 +60,27 @@ async function loadContent( let moved = false; const parsed = new Map(); const toRead: StorageFile[] = []; - let knownBytes = 0; for (const group of groups.values()) { if (!readAll && group.length < 2) continue; for (const file of group) { - // An existing file over maxBlockBytes is unreadable, like invalid JSON: it's + // An existing file over MAX_BLOCK_BYTES is unreadable, like invalid JSON: it's // left out of the map and reported as a `too-large` diagnostic, so one // oversized hand edit can't take the whole editor down. The map plus its - // diagnostics still accounts for every file in the snapshot; only the - // aggregate (maxListBytes) answers LimitExceeded. - if (file.size !== undefined && file.size > limits.maxBlockBytes) { + // diagnostics still accounts for every file in the snapshot. + if (file.size !== undefined && file.size > MAX_BLOCK_BYTES) { parsed.set(file.file, { ok: false, kind: "too-large", bytes: file.size, - message: `the file is over ${limits.maxBlockBytes} bytes`, + message: `the file is over ${MAX_BLOCK_BYTES} bytes`, }); continue; } - knownBytes += file.size ?? 0; const hit = cache.get(file.file, file.version); if (hit) parsed.set(file.file, hit); else toRead.push(file); } } - if (readAll && knownBytes > limits.maxListBytes) { - throw limitExceeded(`the block list is over ${limits.maxListBytes} bytes`, { - limit: "maxListBytes", - }); - } if (toRead.length > 0) { const bodies = await storage.readFiles( snapshot, @@ -100,7 +92,7 @@ async function loadContent( moved = true; // vanished since the snapshot continue; } - const body = parseBody(read.text, limits.maxBlockBytes); + const body = parseBody(read.text, MAX_BLOCK_BYTES); // Cache under the version of the bytes actually read, never the snapshot's. cache.set(file.file, read.version, body); if (read.version !== file.version) { @@ -127,11 +119,6 @@ async function loadContent( bytes += body.bytes; candidates.push({ ...file, hasPath: entryHasPath(body.value), value: body.value }); } - if (readAll && bytes > limits.maxListBytes) { - throw limitExceeded(`the block list is over ${limits.maxListBytes} bytes`, { - limit: "maxListBytes", - }); - } const entries = new Map(); for (const { name, winner, shadowed } of resolveSpellings(candidates).values()) { @@ -161,12 +148,11 @@ const MAX_SNAPSHOT_READS = 3; */ export async function loadCurrentContent( storage: ContentStorage, - ref: string | undefined, options: Options, shortCircuit?: (snapshot: StorageSnapshot) => boolean, ): Promise<{ snapshot: StorageSnapshot; content: LoadedContent | null }> { for (let read = 1; read <= MAX_SNAPSHOT_READS; read++) { - const snapshot = await storage.snapshot({ ref }); + const snapshot = await storage.snapshot(); if (shortCircuit?.(snapshot)) return { snapshot, content: null }; const content = await loadContent(storage, snapshot, options); if (!content.moved) return { snapshot, content }; diff --git a/packages/blocks/src/protocol/server/core.ts b/packages/blocks/src/protocol/server/core.ts index 0d63e8a02..87e4c7ddc 100644 --- a/packages/blocks/src/protocol/server/core.ts +++ b/packages/blocks/src/protocol/server/core.ts @@ -1,13 +1,7 @@ /** * The shared state of one content handler: its storage, options and caches. */ -import { - ContentProtocolError, - invalidBlock, - notFound, - unavailable, - unsupported, -} from "../errors.ts"; +import { ContentProtocolError, invalidBlock, notFound, unavailable } from "../errors.ts"; import { blockNameFromFile } from "../keys.ts"; import { type ContentStorage, @@ -16,15 +10,21 @@ import { StorageNotFoundError, StorageUnavailableError, } from "../storage.ts"; -import { DEFAULT_LIMITS, type DecoMeta, type Limits } from "../types.ts"; -import type { AuthOptions } from "./auth.ts"; +import type { DecoMeta } from "../types.ts"; import { BodyCache } from "./bodyCache.ts"; -export interface ContentHandlerOptions extends AuthOptions { +const MiB = 1024 * 1024; + +/** `blocks.apply`'s own guards: names (set plus delete) in one call. Internal, not advertised. */ +export const MAX_OPS_PER_APPLY = 500; +/** Bytes of one stored entry. */ +export const MAX_BLOCK_BYTES = 1 * MiB; +/** Bytes of one HTTP request body, uncompressed. */ +export const MAX_REQUEST_BYTES = 8 * MiB; + +export interface ContentHandlerOptions { /** Reported by `describe`, such as `{ name: "deco-cli", version: "8.0.0" }`. */ server?: { name: string; version: string }; - /** Lowers the protocol's default limits. Higher values are ignored. */ - limits?: Partial; /** Overrides the poll interval the storage suggests. */ pollIntervalMs?: number; /** The app the site editor previews, reported by `describe` (`{ url }`). */ @@ -48,8 +48,6 @@ const DEFAULT_RETRY_DELAY = { minMs: 50, maxMs: 200 }; export class Core { readonly cache: BodyCache; - /** Requests in flight per scoped request key, so simultaneous duplicates commit once. */ - readonly inflight = new Map>(); /** The last parsed schema, by version: the secret guard reads it on every write. */ parsedSchema: { version: string; meta: DecoMeta } | null = null; @@ -75,31 +73,11 @@ export class Core { return this.storage.describe(); } - /** The effective limits: the lowest of the defaults, the storage's and the options'. */ - limits(description: StorageDescription): Limits { - const out = { ...DEFAULT_LIMITS }; - for (const source of [description.limits, this.options.limits]) { - if (!source) continue; - for (const key of Object.keys(out) as (keyof Limits)[]) { - const value = source[key]; - if (typeof value === "number" && value > 0 && value < out[key]) out[key] = value; - } - } - return out; - } - pollIntervalMs(description: StorageDescription): number { return ( this.options.pollIntervalMs ?? description.pollIntervalMs ?? DEFAULT_POLL_MS[description.kind] ); } - - /** Refuses a `ref` on a storage without branches. */ - checkRef(description: StorageDescription, ref: string | undefined): void { - if (ref !== undefined && description.refs === null) { - throw unsupported("this endpoint has no branches; omit ref"); - } - } } /** Maps storage errors to protocol errors; anything unknown is rethrown. */ diff --git a/packages/blocks/src/protocol/server/handler.test.ts b/packages/blocks/src/protocol/server/handler.test.ts index 06d95ee74..6aceb2252 100644 --- a/packages/blocks/src/protocol/server/handler.test.ts +++ b/packages/blocks/src/protocol/server/handler.test.ts @@ -9,7 +9,8 @@ import { ErrorCode } from "../errors"; import type { ContentStorage } from "../storage"; import { StorageNotFoundError, StorageUnavailableError } from "../storage"; import { createMemoryStorage } from "../storage/memory"; -import { DEFAULT_LIMITS, MAX_BATCH_CALLS } from "../types"; +import { MAX_BATCH_CALLS } from "../types"; +import { MAX_REQUEST_BYTES } from "./core"; import { createContentHandler } from "./handler"; const URL_ = "http://test.local/rpc"; @@ -81,48 +82,11 @@ describe("HTTP", () => { expect(response.body.error.code).toBe(ErrorCode.InvalidBlock); }); - it("refuses a missing or invalid bearer token with HTTP 401", async () => { - const { handler } = setup({ token: "s3cret" }); - const attempts: Record[] = [ - {}, - { authorization: "Bearer nope" }, - { authorization: "Basic s3cret" }, - ]; - for (const headers of attempts) { - const response = await call(handler, rpc(1, "describe"), headers); - expect(response.status).toBe(401); - expect(response.body).toEqual({ - jsonrpc: "2.0", - id: null, - error: { code: ErrorCode.Unauthorized, message: "missing or invalid bearer token" }, - }); - } - const ok = await call(handler, rpc(1, "describe"), { authorization: "Bearer s3cret" }); - expect(ok.status).toBe(200); - expect(ok.body.result.protocol).toBe("deco-content"); - }); - - it("checks the token before reading the body", async () => { - const { handler } = setup({ token: "t" }); - const response = await call(handler, "{not json"); - expect(response.status).toBe(401); - }); - - it("maps authorize() outcomes: forbidden is a JSON-RPC error with HTTP 200", async () => { - const { handler } = setup({ - authorize: (r) => (r.headers.get("x-user") === "ok" ? true : "forbidden"), - }); - const denied = await call(handler, rpc(1, "describe")); - expect(denied.status).toBe(200); - expect(denied.body.error.code).toBe(ErrorCode.Forbidden); - expect((await call(handler, rpc(1, "describe"), { "x-user": "ok" })).body.result).toBeDefined(); - const { handler: unauthorized } = setup({ authorize: () => false }); - expect((await call(unauthorized, rpc(1, "describe"))).status).toBe(401); - }); - - it("refuses a body over maxRequestBytes with HTTP 413, whatever Content-Length says", async () => { - const { handler } = setup({ limits: { maxRequestBytes: 1024 } }); - const big = JSON.stringify(rpc(1, "blocks.apply", { set: { a: { t: "x".repeat(2000) } } })); + it("refuses a body over the request size limit with HTTP 413, whatever Content-Length says", async () => { + const { handler } = setup(); + const big = JSON.stringify( + rpc(1, "blocks.apply", { set: { a: { t: "x".repeat(MAX_REQUEST_BYTES) } } }), + ); const response = await call(handler, big); expect(response.status).toBe(413); expect(response.body.error.code).toBe(ErrorCode.LimitExceeded); @@ -140,7 +104,7 @@ describe("HTTP", () => { }); it("decompresses gzip request bodies, and gzip can't bypass the limit", async () => { - const { handler } = setup({ limits: { maxRequestBytes: 4096 } }); + const { handler } = setup(); const gzip = async (text: string) => new Uint8Array( await new Response( @@ -152,9 +116,9 @@ describe("HTTP", () => { "deco-content", ); const bomb = await gzip( - JSON.stringify(rpc(1, "blocks.apply", { set: { a: { t: "x".repeat(100_000) } } })), + JSON.stringify(rpc(1, "blocks.apply", { set: { a: { t: "x".repeat(MAX_REQUEST_BYTES) } } })), ); - expect(bomb.byteLength).toBeLessThan(4096); + expect(bomb.byteLength).toBeLessThan(MAX_REQUEST_BYTES / 100); expect((await call(handler, bomb, { "content-encoding": "gzip" })).status).toBe(413); expect((await call(handler, "{}", { "content-encoding": "br" })).status).toBe(415); expect( @@ -276,35 +240,6 @@ describe("batches", () => { expect(items[1].error.code).toBe(ErrorCode.InvalidBlock); expect(Object.keys(storage.dump().files)).toEqual(["kept.json"]); }); - - it("bound the aggregate response: later reads answer LimitExceeded, writes still run", async () => { - const { handler, storage } = setup({ limits: { maxBatchResponseBytes: 3000 } }); - storage.setFile("big.json", JSON.stringify({ t: "x".repeat(1500) })); - const items = ( - await call(handler, [ - rpc(1, "blocks.list", {}), - rpc(2, "blocks.list", {}), - rpc(3, "blocks.apply", { set: { w: {} } }), - rpc(4, "blocks.list", {}), - rpc(5, "blocks.list", { ifNoneMatch: "stale" }), - ]) - ).body as any[]; - expect(items[0].result.blocks.big).toBeDefined(); - expect(items[1].error.code).toBe(ErrorCode.LimitExceeded); - expect(items[2].result.versions.w).toEqual(expect.any(String)); - expect(items[3].error.code).toBe(ErrorCode.LimitExceeded); - expect(items[4].error.code).toBe(ErrorCode.LimitExceeded); - expect(storage.dump().files["w.json"]).toBeDefined(); - }); - - it("bound a single response too", async () => { - const { handler, storage } = setup({ limits: { maxBatchResponseBytes: 1000 } }); - storage.setFile("big.json", JSON.stringify({ t: "x".repeat(1500) })); - expect((await call(handler, rpc(9, "blocks.list", {}))).body).toMatchObject({ - id: 9, - error: { code: ErrorCode.LimitExceeded }, - }); - }); }); describe("storage failures", () => { @@ -346,18 +281,13 @@ describe("storage failures", () => { }); describe("describe", () => { - it("reports the storage, the effective limits and the features", async () => { + it("reports the storage and the features", async () => { const storage = createMemoryStorage({ state: { secretsPublicKey: PUBLIC_KEY }, - description: { - kind: "working-tree", - root: "apps/storefront", - limits: { maxOpsPerApply: 100 }, - }, + description: { kind: "working-tree", root: "apps/storefront" }, }); const handler = createContentHandler(storage, { server: { name: "deco-cli", version: "8.0.0" }, - limits: { maxBlockBytes: 2048, maxListBytes: DEFAULT_LIMITS.maxListBytes * 2 }, preview: { url: "http://localhost:5173" }, }); const { result } = (await call(handler, rpc(1, "describe"))).body; @@ -369,10 +299,7 @@ describe("describe", () => { readOnly: false, root: "apps/storefront", schemaFormat: "deco-meta@1", - refs: null, - writes: { idempotency: { retentionMs: 86_400_000 }, schemaPreconditions: true }, pollIntervalMs: 2000, - limits: { ...DEFAULT_LIMITS, maxOpsPerApply: 100, maxBlockBytes: 2048 }, preview: { url: "http://localhost:5173" }, assets: { dir: "public/assets", urlPrefix: "/assets/", maxBytes: 25 * 1024 * 1024 }, secrets: { publicKey: PUBLIC_KEY }, @@ -407,10 +334,6 @@ describe("describe", () => { expect(onError).toHaveBeenCalledWith(expect.any(Error)); }); - it("refuses an empty token at construction, so it can't lock every client out", () => { - expect(() => createContentHandler(createMemoryStorage(), { token: "" })).toThrow(TypeError); - }); - it("polls a git storage every 30 seconds by default", async () => { const handler = createContentHandler(createMemoryStorage({ description: { kind: "git" } })); expect((await call(handler, rpc(1, "describe"))).body.result.pollIntervalMs).toBe(30000); @@ -423,11 +346,4 @@ describe("describe", () => { expect(result.assets).toBeNull(); expect(result.secrets).toBeNull(); }); - - it("advertises idempotency only when the storage keeps receipts", async () => { - const storage = createMemoryStorage(); - const noReceipts: ContentStorage = { ...storage, getReceipt: undefined }; - const handler = createContentHandler(noReceipts); - expect((await call(handler, rpc(1, "describe"))).body.result.writes.idempotency).toBeNull(); - }); }); diff --git a/packages/blocks/src/protocol/server/handler.ts b/packages/blocks/src/protocol/server/handler.ts index dabec8eba..0d6ba2c44 100644 --- a/packages/blocks/src/protocol/server/handler.ts +++ b/packages/blocks/src/protocol/server/handler.ts @@ -4,29 +4,21 @@ * * `POST ` with `Content-Type: application/json` and a body that's * one request object or a batch array. Errors come back in the JSON-RPC - * `error` object with HTTP 200, except a missing or invalid bearer token (401) - * and a body over the size limit (413), which apply to the whole batch, and - * requests that aren't the protocol at all: a method other than POST (405), + * `error` object with HTTP 200, except a body over the size limit (413), + * which applies to the whole batch, and requests that aren't the protocol at all: a method other than POST (405), * or a body that isn't JSON or uses an unsupported encoding (415). Those carry * a JSON-RPC error body too, with a `null` id. * Responses are gzip-compressed when the request accepts it. * * The handler is path-agnostic: mount it where the endpoint lives (`/rpc` on - * the local server). Transport security that depends on where it runs — CORS, - * `Host` checks, Chrome's local-network preflight — belongs to the server - * that mounts it (`deco serve`). + * the local server). The handler doesn't authenticate: who may call it, and + * transport security that depends on where it runs — CORS, `Host` checks, + * Chrome's local-network preflight — belong to the server that mounts it + * (`deco serve` on loopback; Studio behind its own session). */ -import { - type ContentProtocolError, - forbidden, - invalidRequest, - limitExceeded, - parseError, - unauthorized, -} from "../errors.ts"; +import { type ContentProtocolError, invalidRequest, limitExceeded, parseError } from "../errors.ts"; import type { ContentStorage } from "../storage.ts"; -import { assertAuthOptions, authenticate } from "./auth.ts"; -import { type ContentHandlerOptions, Core } from "./core.ts"; +import { type ContentHandlerOptions, Core, MAX_REQUEST_BYTES } from "./core.ts"; import { BodyEncodingError, BodyTooLargeError, @@ -45,21 +37,12 @@ export function createContentHandler( storage: ContentStorage, options: ContentHandlerOptions = {}, ): ContentHandler { - assertAuthOptions(options); const core = new Core(storage, options); return async (request) => { if (request.method !== "POST") { return jsonResponse(request, errorBody(invalidRequest("use POST")), 405, { allow: "POST" }); } - const auth = await authenticate(request, options); - if (!auth.ok) { - return auth.reason === "unauthorized" - ? jsonResponse(request, errorBody(unauthorized()), 401, { - "www-authenticate": 'Bearer realm="deco-content"', - }) - : jsonResponse(request, errorBody(forbidden())); - } if (!isJsonContentType(request)) { return jsonResponse( request, @@ -68,16 +51,15 @@ export function createContentHandler( ); } - const limits = core.limits(await core.description()); let bytes: Uint8Array; try { - bytes = await readBody(request, limits.maxRequestBytes); + bytes = await readBody(request, MAX_REQUEST_BYTES); } catch (error) { if (error instanceof BodyTooLargeError) { return jsonResponse( request, errorBody( - limitExceeded(`the request body is over ${limits.maxRequestBytes} bytes`, { + limitExceeded(`the request body is over ${MAX_REQUEST_BYTES} bytes`, { limit: "maxRequestBytes", }), ), @@ -96,7 +78,6 @@ export function createContentHandler( } catch { return jsonResponse(request, errorBody(parseError())); } - const response = await dispatch(core, body, auth.scope, limits.maxBatchResponseBytes); - return jsonResponse(request, response); + return jsonResponse(request, await dispatch(core, body)); }; } diff --git a/packages/blocks/src/protocol/server/http.ts b/packages/blocks/src/protocol/server/http.ts index 972f62ef1..3c6477627 100644 --- a/packages/blocks/src/protocol/server/http.ts +++ b/packages/blocks/src/protocol/server/http.ts @@ -1,11 +1,9 @@ /** * HTTP plumbing for the content handler: bounded body reads (gzip request - * bodies included), gzip responses and bearer-token checks. Web-standard APIs + * bodies included) and gzip responses. Web-standard APIs * only (Request, Response, CompressionStream), so it runs on Node, Bun, * Workers and Deno. */ -import { sha256Hex } from "../canonical.ts"; - /** The body was larger than allowed (after decompression). */ export class BodyTooLargeError extends Error { constructor(readonly limit: number) { @@ -121,19 +119,3 @@ export function isJsonContentType(request: Request): boolean { const type = request.headers.get("content-type"); return type !== null && type.split(";")[0]!.trim().toLowerCase() === "application/json"; } - -/** The bearer token of the request, or `null` without one. */ -export function bearerToken(request: Request): string | null { - const header = request.headers.get("authorization"); - if (!header) return null; - const match = /^Bearer\s+(\S+)\s*$/i.exec(header); - return match?.[1] ?? null; -} - -/** Compares two strings in time independent of where they differ. */ -export async function timingSafeEqualStrings(a: string, b: string): Promise { - const [ha, hb] = await Promise.all([sha256Hex(a), sha256Hex(b)]); - let diff = 0; - for (let i = 0; i < ha.length; i++) diff |= ha.charCodeAt(i) ^ hb.charCodeAt(i); - return diff === 0; -} diff --git a/packages/blocks/src/protocol/server/index.ts b/packages/blocks/src/protocol/server/index.ts index 79604616b..ef83f37b3 100644 --- a/packages/blocks/src/protocol/server/index.ts +++ b/packages/blocks/src/protocol/server/index.ts @@ -6,6 +6,5 @@ * and Deno. (`deco serve`'s asset uploads use `./assets`, which isn't public.) */ -export type { AuthOptions, AuthorizeResult } from "./auth.ts"; export type { ContentHandlerOptions } from "./core.ts"; export { type ContentHandler, createContentHandler } from "./handler.ts"; diff --git a/packages/blocks/src/protocol/server/methods.test.ts b/packages/blocks/src/protocol/server/methods.test.ts index 350b68eae..30434bcee 100644 --- a/packages/blocks/src/protocol/server/methods.test.ts +++ b/packages/blocks/src/protocol/server/methods.test.ts @@ -15,7 +15,7 @@ import { type ContentProtocolError, ErrorCode } from "../errors"; import { serializeBlock } from "../keys"; import type { CommitAttempt, ContentStorage } from "../storage"; import { createMemoryStorage, type MemoryStorageOptions } from "../storage/memory"; -import type { ContentHandlerOptions } from "./core"; +import { type ContentHandlerOptions, MAX_BLOCK_BYTES, MAX_OPS_PER_APPLY } from "./core"; import { createContentHandler } from "./handler"; function setup( @@ -54,7 +54,7 @@ describe("schema.get", () => { resolvedRef: null, schema: schemaFixture, }); - expect(await client.schemaGet({ ifNoneMatch: first.version })).toEqual({ + expect(await client.schemaGet({ ifNoneMatch: first.version ?? undefined })).toEqual({ notModified: true, version: first.version, }); @@ -66,8 +66,38 @@ describe("schema.get", () => { expect(!result.notModified && result.schema).toEqual({ v7: true }); }); - it("is NotFound without a schema", async () => { + it("reports 'no schema yet' as schema: null, not an error", async () => { const { client } = setup({ state: { schema: null } }); + expect(await client.schemaGet()).toEqual({ + notModified: false, + version: null, + resolvedRef: null, + schema: null, + }); + }); + + it("serves the schema once it appears, and null again once it's gone", async () => { + const { client, storage } = setup({ state: { schema: null } }); + expect((await client.schemaGet()).version).toBeNull(); + storage.setSchema(schemaFixture); + const first = await client.schemaGet(); + expect(first).toMatchObject({ notModified: false, schema: schemaFixture }); + expect(typeof first.version).toBe("string"); + storage.setSchema(null); + expect(await client.schemaGet()).toMatchObject({ version: null, schema: null }); + }); + + it("lists, reads and writes blocks without a schema", async () => { + const { client } = setup({ state: { schema: null } }); + const block = { __resolveType: "site/sections/Hero.tsx", title: "Hi", n: 1, ok: true }; + const { versions } = await client.blocksApply({ set: { hero: block } }); + const list = await client.blocksList(); + expect(!list.notModified && list.blocks).toEqual({ hero: block }); + expect(!list.notModified && list.versions.hero).toBe(versions.hero); + }); + + it("is NotFound without a .deco folder, schema or not", async () => { + const { client } = setup({ state: { schema: null, hasDecoFolder: false } }); await rejects(client.schemaGet(), ErrorCode.NotFound); }); @@ -76,14 +106,6 @@ describe("schema.get", () => { const error = await rejects(client.schemaGet(), ErrorCode.Unavailable); expect(error.data).toEqual({ retryAfterMs: expect.any(Number) }); }); - - it("is LimitExceeded over maxSchemaBytes, even though gzip would make it small", async () => { - const { client } = setup( - { state: { schema: JSON.stringify({ pad: "x".repeat(5000) }) } }, - { limits: { maxSchemaBytes: 4096 } }, - ); - await rejects(client.schemaGet(), ErrorCode.LimitExceeded); - }); }); describe("blocks.list", () => { @@ -122,20 +144,17 @@ describe("blocks.list", () => { }); it("reports files it skips: invalid JSON, non-objects and oversized files", async () => { - const { client } = setup( - { - state: { - files: { - "ok.json": file({ fine: true }), - "broken.json": "{ nope", - "array.json": "[]", - "null.json": "null", - "huge.json": file({ t: "x".repeat(3000) }), - }, + const { client } = setup({ + state: { + files: { + "ok.json": file({ fine: true }), + "broken.json": "{ nope", + "array.json": "[]", + "null.json": "null", + "huge.json": file({ t: "x".repeat(MAX_BLOCK_BYTES) }), }, }, - { limits: { maxBlockBytes: 2048 } }, - ); + }); const result = await client.blocksList(); if (result.notModified) throw new Error("expected the map"); expect(result.blocks).toEqual({ ok: { fine: true } }); @@ -202,31 +221,6 @@ describe("blocks.list", () => { expect(Object.getOwnPropertyDescriptor(result.blocks, "__proto__")?.value).toEqual({ x: 1 }); }); - it("is LimitExceeded over maxListBytes, never a partial map", async () => { - const files: Record = {}; - for (let i = 0; i < 10; i++) files[`e${i}.json`] = file({ t: "x".repeat(500) }); - const { client } = setup({ state: { files } }, { limits: { maxListBytes: 4096 } }); - await rejects(client.blocksList(), ErrorCode.LimitExceeded); - }); - - it("counts bytes even when the storage doesn't report sizes", async () => { - const memory = createMemoryStorage({ - state: { files: { "a.json": file({ t: "x".repeat(5000) }) } }, - }); - const sizeless: ContentStorage = { - ...memory, - snapshot: async (o) => { - const snap = await memory.snapshot(o); - return { ...snap, files: snap.files.map(({ size: _size, ...f }) => f) }; - }, - }; - const client = createContentClient({ - endpoint: "http://t/rpc", - fetch: createContentHandler(sizeless, { limits: { maxListBytes: 4096 } }), - }); - await rejects(client.blocksList(), ErrorCode.LimitExceeded); - }); - it("is NotFound when there's no .deco folder at all", async () => { const { client } = setup({ state: { hasDecoFolder: false } }); await rejects(client.blocksList(), ErrorCode.NotFound); @@ -309,10 +303,10 @@ describe("blocks.apply", () => { expect(storage.commits).toBe(0); }); - it("refuses an entry over maxBlockBytes", async () => { - const { client } = setup({}, { limits: { maxBlockBytes: 64 } }); + it("refuses an entry over the block size limit", async () => { + const { client } = setup(); const error = await rejects( - client.blocksApply({ set: { a: { t: "x".repeat(100) } } }), + client.blocksApply({ set: { a: { t: "x".repeat(MAX_BLOCK_BYTES) } } }), ErrorCode.InvalidBlock, ); expect(error.data).toEqual({ @@ -320,14 +314,20 @@ describe("blocks.apply", () => { }); }); - it("refuses more names than maxOpsPerApply", async () => { - const { client } = setup({}, { limits: { maxOpsPerApply: 3 } }); + it("refuses more names than one blocks.apply takes", async () => { + const { client } = setup(); + const names = (prefix: string, n: number) => + Array.from({ length: n }, (_, i) => `${prefix}${i}`); + const set = Object.fromEntries(names("s", MAX_OPS_PER_APPLY / 2).map((n) => [n, {}])); await rejects( - client.blocksApply({ set: { a: {}, b: {} }, delete: ["c", "d"] }), + client.blocksApply({ set, delete: names("d", MAX_OPS_PER_APPLY / 2 + 1) }), ErrorCode.LimitExceeded, ); // set-wins normalization happens first: a name in both counts once. - await client.blocksApply({ set: { a: {}, b: {} }, delete: ["a", "c"] }); + await client.blocksApply({ + set, + delete: [...Object.keys(set), ...names("d", MAX_OPS_PER_APPLY / 2)], + }); }); it("refuses a new name that differs from an existing entry's, or another new name's, only in case", async () => { @@ -416,19 +416,6 @@ describe("blocks.apply", () => { expect(storage.dump().files["a.json"]).toBe(file({ v: 2 })); }); - it("rejects a write when the schema changed (ifSchemaMatch)", async () => { - const { client, storage } = setup(); - const { version } = await client.schemaGet(); - storage.setSchema({ ...schemaFixture, changed: true }); - const error = await rejects( - client.blocksApply({ set: { a: {} }, ifSchemaMatch: version }), - ErrorCode.Conflict, - ); - expect(error.data).toEqual({ schema: { expected: version, actual: expect.any(String) } }); - const fresh = await client.schemaGet(); - await client.blocksApply({ set: { a: {} }, ifSchemaMatch: fresh.version }); - }); - it("rechecks every guard after storage moved under a commit attempt", async () => { let raced = false; const { client, storage } = setup({ @@ -527,14 +514,18 @@ describe("blocks.apply", () => { await rejects(client.blocksApply({ set: { a: {} } }), ErrorCode.ReadOnly); }); - it("refuses a ref without branches", async () => { - const { client } = setup(); - await rejects(client.blocksApply({ ref: "draft", set: { a: {} } }), ErrorCode.Unsupported); - }); - - it("refuses a request key the endpoint doesn't advertise", async () => { - const { client } = setup({ description: { idempotency: null } }); - await rejects(client.blocksApply({ requestKey: "k", set: { a: {} } }), ErrorCode.Unsupported); + it("refuses the removed ref, requestKey and ifSchemaMatch parameters as unknown", async () => { + const { client, storage } = setup(); + for (const param of ["ref", "requestKey", "ifSchemaMatch"]) { + await rejects( + client.call("blocks.apply", { set: { a: {} }, [param]: "x" } as never), + ErrorCode.InvalidParams, + ); + } + for (const method of ["blocks.list", "schema.get"] as const) { + await rejects(client.call(method, { ref: "main" } as never), ErrorCode.InvalidParams); + } + expect(storage.commits).toBe(0); }); it("skips the commit for an empty apply", async () => { @@ -544,95 +535,4 @@ describe("blocks.apply", () => { expect(storage.commits).toBe(0); }); }); - - describe("request keys", () => { - it("replay the original result, even after the content moved", async () => { - const { client, storage } = setup(); - const params = { requestKey: "save-1", set: { a: { v: 1 } } }; - const first = await client.blocksApply(params); - await client.blocksApply({ set: { b: {} } }); - expect(await client.blocksApply(params)).toEqual(first); - expect(storage.commits).toBe(2); - }); - - it("bind the key to every parameter", async () => { - const { client } = setup(); - await client.blocksApply({ requestKey: "k", set: { a: { v: 1 } } }); - await rejects( - client.blocksApply({ requestKey: "k", set: { a: { v: 2 } } }), - ErrorCode.InvalidParams, - ); - await rejects( - client.blocksApply({ requestKey: "k", set: { a: { v: 1 } }, ifMatch: { a: null } }), - ErrorCode.InvalidParams, - ); - }); - - it("resolve before guards are rechecked", async () => { - const { client } = setup(); - const params = { requestKey: "create", set: { a: {} }, ifMatch: { a: null } }; - const first = await client.blocksApply(params); - // The guard would now fail, but the receipt answers first. - expect(await client.blocksApply(params)).toEqual(first); - }); - - it("commit simultaneous duplicates once", async () => { - const { client, storage } = setup(); - const params = { requestKey: "dup", set: { a: { v: 1 } } }; - const results = await Promise.all([ - client.blocksApply(params), - client.blocksApply(params), - client.blocksApply(params), - ]); - expect(results[1]).toEqual(results[0]); - expect(results[2]).toEqual(results[0]); - expect(storage.commits).toBe(1); - }); - - it("survive a restart", async () => { - const first = setup(); - const params = { requestKey: "r", set: { a: { v: 1 } } }; - const result = await first.client.blocksApply(params); - const restarted = createMemoryStorage({ state: first.storage.dump() }); - const client = createContentClient({ - endpoint: "http://t/rpc", - fetch: createContentHandler(restarted), - }); - expect(await client.blocksApply(params)).toEqual(result); - expect(restarted.commits).toBe(0); - }); - - it("expire after the retention window", async () => { - let now = 0; - const { client, storage } = setup({ - now: () => now, - description: { idempotency: { retentionMs: 1000 } }, - }); - const params = { requestKey: "old", set: { a: { v: 1 } } }; - await client.blocksApply(params); - now = 5000; - await client.blocksApply(params); - expect(storage.commits).toBe(2); - }); - - it("are scoped to the tenant", async () => { - const storage = createMemoryStorage(); - const handler = createContentHandler(storage, { - authorize: (r) => ({ scope: r.headers.get("x-tenant") ?? "" }), - }); - const as = (tenant: string) => - createContentClient({ - endpoint: "http://t/rpc", - fetch: handler, - headers: { "x-tenant": tenant }, - }); - await as("a").blocksApply({ requestKey: "k", set: { a: {} } }); - await as("b").blocksApply({ requestKey: "k", set: { b: {} } }); - expect(storage.commits).toBe(2); - await rejects( - as("a").blocksApply({ requestKey: "k", set: { b: {} } }), - ErrorCode.InvalidParams, - ); - }); - }); }); diff --git a/packages/blocks/src/protocol/server/methods/apply.ts b/packages/blocks/src/protocol/server/methods/apply.ts index 90265d087..c251796eb 100644 --- a/packages/blocks/src/protocol/server/methods/apply.ts +++ b/packages/blocks/src/protocol/server/methods/apply.ts @@ -7,27 +7,23 @@ * 3. `set` wins when a name is in both `set` and `delete`. * 4. Validated first: names, value shapes, sizes and the secret guard are * checked before anything is written, and every violation is reported. - * 5. Preconditions are optional: a failed `ifMatch` (or `ifSchemaMatch`) - * writes nothing and returns a Conflict; without them the last writer wins. + * 5. Preconditions are optional: a failed `ifMatch` writes nothing and + * returns a Conflict; without one the last writer wins. * * 6. Two spellings of one name are one entry: a guard compares against the * entry under any spelling, and the result reports `null` for every other * spelling the commit deleted. * - * A commit attempt that finds storage moved (files, the schema, or a request - * key reserved meanwhile) is retried against a new snapshot after a jittered + * A commit attempt that finds storage moved (files or the schema) is retried against a new snapshot after a jittered * backoff, rechecking every guard, up to `maxCommitAttempts` times. */ -import { applyRequestDigest, canonicalJson } from "../../canonical.ts"; import { type BlockViolation, conflict, invalidBlock, - invalidParams, limitExceeded, readOnly, unavailable, - unsupported, type VersionMismatch, } from "../../errors.ts"; import { @@ -41,11 +37,10 @@ import { spellingKey, } from "../../keys.ts"; import { checkSecrets } from "../../secrets.ts"; -import type { StorageDescription, StoredReceipt } from "../../storage.ts"; -import type { BlocksApplyParams, BlocksApplyResult, DecoMeta, Limits } from "../../types.ts"; +import type { BlocksApplyParams, BlocksApplyResult, DecoMeta } from "../../types.ts"; import { type LoadedContent, type LoadedEntry, loadCurrentContent } from "../content.ts"; -import type { Core } from "../core.ts"; -import { parseSchema, readSchema } from "./read.ts"; +import { type Core, MAX_BLOCK_BYTES, MAX_OPS_PER_APPLY } from "../core.ts"; +import { parseSchema } from "./read.ts"; const utf8 = new TextEncoder(); @@ -56,13 +51,13 @@ interface NormalizedApply { ifMatch: Array<[name: string, version: string | null]>; } -function normalize(params: BlocksApplyParams, limits: Limits): NormalizedApply { +function normalize(params: BlocksApplyParams): NormalizedApply { const set = Object.entries(params.set ?? {}); const setNames = new Set(set.map(([name]) => name)); const deleteNames = [...new Set(params.delete ?? [])].filter((name) => !setNames.has(name)); const ops = set.length + deleteNames.length; - if (ops > limits.maxOpsPerApply) { - throw limitExceeded(`${ops} names in one blocks.apply; the limit is ${limits.maxOpsPerApply}`, { + if (ops > MAX_OPS_PER_APPLY) { + throw limitExceeded(`${ops} names in one blocks.apply; the limit is ${MAX_OPS_PER_APPLY}`, { limit: "maxOpsPerApply", }); } @@ -70,7 +65,7 @@ function normalize(params: BlocksApplyParams, limits: Limits): NormalizedApply { } /** Checks everything that doesn't depend on stored content. */ -function validateStatic(apply: NormalizedApply, limits: Limits, meta: DecoMeta | null) { +function validateStatic(apply: NormalizedApply, meta: DecoMeta | null) { const violations: BlockViolation[] = []; const bodies = new Map(); const bySpelling = new Map(); @@ -91,11 +86,11 @@ function validateStatic(apply: NormalizedApply, limits: Limits, meta: DecoMeta | continue; } const body = serializeBlock(value); - if (utf8.encode(body).byteLength > limits.maxBlockBytes) { + if (utf8.encode(body).byteLength > MAX_BLOCK_BYTES) { violations.push({ name, rule: "too-large", - message: `the entry is over ${limits.maxBlockBytes} bytes`, + message: `the entry is over ${MAX_BLOCK_BYTES} bytes`, }); continue; } @@ -231,12 +226,8 @@ function resultFromVersions( return { revision, versions }; } -function receiptResult(apply: NormalizedApply, receipt: StoredReceipt): BlocksApplyResult { - return resultFromVersions(apply, receipt.revision, receipt.versions, receipt.deleted ?? []); -} - -async function loadSchema(core: Core, params: BlocksApplyParams, limits: Limits) { - const stored = await readSchema(core, params.ref, limits.maxSchemaBytes); +async function loadSchema(core: Core) { + const stored = await core.storage.readSchema(); if (stored === null) return { version: null, meta: null }; if (core.parsedSchema?.version !== stored.version) { core.parsedSchema = { version: stored.version, meta: parseSchema(stored.text) }; @@ -244,67 +235,19 @@ async function loadSchema(core: Core, params: BlocksApplyParams, limits: Limits) return { version: stored.version, meta: core.parsedSchema.meta }; } -function checkFeatures(description: StorageDescription, core: Core, params: BlocksApplyParams) { - if (description.readOnly) throw readOnly(); - core.checkRef(description, params.ref); - if (params.requestKey !== undefined && (!description.idempotency || !core.storage.getReceipt)) { - throw unsupported("this endpoint doesn't support request keys; omit requestKey"); - } -} - export async function blocksApply( core: Core, params: BlocksApplyParams, - scope: string, ): Promise { const description = await core.description(); - checkFeatures(description, core, params); - if (params.requestKey === undefined) return applyOnce(core, params, description, null); - - const digest = await applyRequestDigest(params); - const key = canonicalJson([scope, description.root, params.ref ?? null, params.requestKey]); - // Simultaneous duplicates: wait for the first, then find its receipt. - for (let pending = core.inflight.get(key); pending; pending = core.inflight.get(key)) { - await pending.catch(() => {}); - } - const run = applyOnce(core, params, description, { key, digest }); - core.inflight.set(key, run); - try { - return await run; - } finally { - if (core.inflight.get(key) === run) core.inflight.delete(key); - } -} - -async function applyOnce( - core: Core, - params: BlocksApplyParams, - description: StorageDescription, - receipt: { key: string; digest: string } | null, -): Promise { - const limits = core.limits(description); - const apply = normalize(params, limits); + if (description.readOnly) throw readOnly(); + const apply = normalize(params); for (let attempt = 1; attempt <= core.maxCommitAttempts; attempt++) { - // Receipts for completed identical requests resolve before any guard is rechecked. - if (receipt) { - const stored = await core.storage.getReceipt!(receipt.key); - if (stored) { - if (stored.digest !== receipt.digest) { - throw invalidParams("requestKey was already used for a different request"); - } - return receiptResult(apply, stored); - } - } - - const schema = await loadSchema(core, params, limits); - if (params.ifSchemaMatch !== undefined && params.ifSchemaMatch !== schema.version) { - throw conflict({ schema: { expected: params.ifSchemaMatch, actual: schema.version } }); - } - const { violations, bodies } = validateStatic(apply, limits, schema.meta); - const { snapshot, content } = await loadCurrentContent(core.storage, params.ref, { + const schema = await loadSchema(core); + const { violations, bodies } = validateStatic(apply, schema.meta); + const { snapshot, content } = await loadCurrentContent(core.storage, { readAll: false, - limits, cache: core.cache, }); const bySpelling = entriesBySpelling(content!); @@ -315,19 +258,16 @@ async function applyOnce( if (mismatches) throw conflict({ entries: mismatches }); const { put, delete: deletes, expected } = plan(content!, bySpelling, apply, bodies); - if (Object.keys(put).length === 0 && deletes.length === 0 && !receipt) { + if (Object.keys(put).length === 0 && deletes.length === 0) { return resultFromVersions(apply, snapshot.revision, {}, []); } - // The schema the guards and the secret check used must still be current at commit time. - const dependsOnSchema = params.ifSchemaMatch !== undefined || apply.set.length > 0; + // The schema the secret check used must still be current at commit time. const result = await core.storage.commit({ - ref: params.ref, base: snapshot, put, delete: deletes, expected, - ...(dependsOnSchema ? { expectedSchemaVersion: schema.version } : {}), - receipt: receipt ?? undefined, + ...(apply.set.length > 0 ? { expectedSchemaVersion: schema.version } : {}), }); if (result.status === "committed") { return resultFromVersions(apply, result.revision, result.versions, deletes); diff --git a/packages/blocks/src/protocol/server/methods/read.ts b/packages/blocks/src/protocol/server/methods/read.ts index 578990e5c..ec608c4fd 100644 --- a/packages/blocks/src/protocol/server/methods/read.ts +++ b/packages/blocks/src/protocol/server/methods/read.ts @@ -2,7 +2,7 @@ * The three read methods: `describe`, `schema.get` and `blocks.list`. */ import { publicKeyDerFromPem } from "../../ciphertext.ts"; -import { limitExceeded, notFound, unavailable } from "../../errors.ts"; +import { unavailable } from "../../errors.ts"; import { ASSETS_URL_PREFIX, type BlocksListResult, @@ -52,13 +52,7 @@ export async function describe(core: Core): Promise { readOnly, root: description.root, schemaFormat: SCHEMA_FORMAT, - refs: description.refs, - writes: { - idempotency: core.storage.getReceipt ? description.idempotency : null, - schemaPreconditions: true, - }, pollIntervalMs: core.pollIntervalMs(description), - limits: core.limits(description), preview: core.options.preview ?? null, assets: readOnly || description.assets === null @@ -72,20 +66,6 @@ export async function describe(core: Core): Promise { }; } -/** Reads and parses the schema, enforcing the schema byte limit. */ -export async function readSchema( - core: Core, - ref: string | undefined, - maxSchemaBytes: number, -): Promise<{ version: string; resolvedRef: string | null; text: string } | null> { - const stored = await core.storage.readSchema({ ref }); - if (stored === null) return null; - if (new TextEncoder().encode(stored.text).byteLength > maxSchemaBytes) { - throw limitExceeded(`the schema is over ${maxSchemaBytes} bytes`, { limit: "maxSchemaBytes" }); - } - return stored; -} - /** Parses schema text; a file caught mid-write is reported as Unavailable, never served torn. */ export function parseSchema(text: string): DecoMeta { let schema: unknown; @@ -101,21 +81,16 @@ export function parseSchema(text: string): DecoMeta { } export async function schemaGet(core: Core, params: ReadParams): Promise { - const description = await core.description(); - core.checkRef(description, params.ref); - const limits = core.limits(description); - const stored = await core.storage.readSchema({ ref: params.ref }); + const stored = await core.storage.readSchema(); if (stored === null) { - throw notFound("no schema: neither .deco/schema.gen.json nor .deco/meta.gen.json exists"); + // No schema yet is a state, not an error. The snapshot still refuses a + // site with no .deco folder (NotFound) and reports the resolved branch. + const snapshot = await core.storage.snapshot(); + return { notModified: false, version: null, resolvedRef: snapshot.resolvedRef, schema: null }; } if (params.ifNoneMatch !== undefined && params.ifNoneMatch === stored.version) { return { notModified: true, version: stored.version }; } - if (new TextEncoder().encode(stored.text).byteLength > limits.maxSchemaBytes) { - throw limitExceeded(`the schema is over ${limits.maxSchemaBytes} bytes`, { - limit: "maxSchemaBytes", - }); - } return { notModified: false, version: stored.version, @@ -125,13 +100,9 @@ export async function schemaGet(core: Core, params: ReadParams): Promise { - const description = await core.description(); - core.checkRef(description, params.ref); - const limits = core.limits(description); const { snapshot, content } = await loadCurrentContent( core.storage, - params.ref, - { readAll: true, limits, cache: core.cache }, + { readAll: true, cache: core.cache }, (s) => params.ifNoneMatch !== undefined && params.ifNoneMatch === s.revision, ); if (content === null) { diff --git a/packages/blocks/src/protocol/server/rpc.ts b/packages/blocks/src/protocol/server/rpc.ts index e99feaf39..8c2069400 100644 --- a/packages/blocks/src/protocol/server/rpc.ts +++ b/packages/blocks/src/protocol/server/rpc.ts @@ -4,10 +4,6 @@ * - Every request has an `id`; one without is rejected rather than run. * - A batch runs in order, returns results in order, holds at most 10 calls, * and isn't atomic. - * - The aggregate response is bounded: once it would pass - * `maxBatchResponseBytes`, later reads answer LimitExceeded instead. - * - A write's response is never replaced, alone or in a batch: the write has - * landed, and answering LimitExceeded would tell the client it hadn't. */ import { type ContentProtocolError, @@ -22,8 +18,6 @@ import { type Core, toProtocolError } from "./core.ts"; import { blocksApply } from "./methods/apply.ts"; import { blocksList, describe, schemaGet } from "./methods/read.ts"; -const utf8 = new TextEncoder(); - type Envelope = { id: RpcId; method: MethodName; params: unknown }; const isMethod = (method: string): method is MethodName => @@ -52,7 +46,7 @@ function parseEnvelope( return { id, method: raw.method, params: raw.params }; } -async function run(core: Core, envelope: Envelope, scope: string): Promise { +async function run(core: Core, envelope: Envelope): Promise { switch (envelope.method) { case "describe": validateParams("describe", envelope.params); @@ -62,24 +56,18 @@ async function run(core: Core, envelope: Envelope, scope: string): Promise JSON.stringify({ jsonrpc: "2.0", id, error: error.toJSON() }); -async function call(core: Core, value: unknown, scope: string): Promise { +async function call(core: Core, value: unknown): Promise { const envelope = parseEnvelope(value); if ("error" in envelope) return errorResponse(envelope.id, envelope.error); try { - const result = await run(core, envelope, scope); + const result = await run(core, envelope); return JSON.stringify({ jsonrpc: "2.0", id: envelope.id, result }); } catch (error) { const known = toProtocolError(error); @@ -93,17 +81,8 @@ async function call(core: Core, value: unknown, scope: string): Promise * Runs a parsed request body (one request object or a batch array) and * returns the serialized response body. */ -export async function dispatch(core: Core, body: unknown, scope: string, maxResponseBytes: number) { - if (!Array.isArray(body)) { - const response = await call(core, body, scope); - if (isWrite(body) || utf8.encode(response).byteLength <= maxResponseBytes) return response; - return errorResponse( - parseEnvelope(body).id, - limitExceeded(`the response is over ${maxResponseBytes} bytes`, { - limit: "maxBatchResponseBytes", - }), - ); - } +export async function dispatch(core: Core, body: unknown): Promise { + if (!Array.isArray(body)) return call(core, body); if (body.length === 0) return errorResponse(null, invalidRequest("an empty batch")); if (body.length > MAX_BATCH_CALLS) { return errorResponse( @@ -112,25 +91,6 @@ export async function dispatch(core: Core, body: unknown, scope: string, maxResp ); } const responses: string[] = []; - let total = 2; // the brackets - for (const item of body) { - const envelope = parseEnvelope(item); - const id = envelope.id; - const overBudget = () => - errorResponse( - id, - limitExceeded(`the batch response is over ${maxResponseBytes} bytes`, { - limit: "maxBatchResponseBytes", - }), - ); - // Writes always run, so a batch never drops a save; reads stop once over budget. - const write = !("error" in envelope) && envelope.method === "blocks.apply"; - let response = - !write && total > maxResponseBytes ? overBudget() : await call(core, item, scope); - const bytes = utf8.encode(response).byteLength + (responses.length ? 1 : 0); - if (total + bytes > maxResponseBytes && !write) response = overBudget(); - total += utf8.encode(response).byteLength + (responses.length ? 1 : 0); - responses.push(response); - } + for (const item of body) responses.push(await call(core, item)); return `[${responses.join(",")}]`; } diff --git a/packages/blocks/src/protocol/storage.ts b/packages/blocks/src/protocol/storage.ts index ce89c4de9..bb609715a 100644 --- a/packages/blocks/src/protocol/storage.ts +++ b/packages/blocks/src/protocol/storage.ts @@ -10,7 +10,7 @@ * * Browser-safe: types and small error classes only. */ -import type { Limits, StorageKind } from "./types.ts"; +import type { StorageKind } from "./types.ts"; /** What a storage tells `describe`. */ export interface StorageDescription { @@ -18,19 +18,10 @@ export interface StorageDescription { /** The app root, relative to the repository root (`"."` when they're the same). */ root: string; readOnly: boolean; - /** Branch support; `null` when the storage has no refs (`ref` params are then Unsupported). */ - refs: null | { default: string; autoCreate: boolean }; /** Where uploads go; `null` when read-only or uploads go elsewhere. */ assets: null | { dir: string; maxBytes: number }; - /** - * Durable request-key receipts. Advertise only when `getReceipt` is - * implemented and `commit` persists `attempt.receipt` with the mutation. - */ - idempotency: null | { retentionMs: number }; /** Overrides the poll interval (default: 2000 for a working tree, 30000 for git). */ pollIntervalMs?: number; - /** Lowers the protocol's default limits. Higher values are ignored. */ - limits?: Partial; } /** One saved-block file in a snapshot. */ @@ -47,7 +38,7 @@ export interface StorageFile { export interface StorageSnapshot { /** An opaque revision of the whole set, compared only for equality. */ revision: string; - /** The ref actually read (it can differ from the requested one); `null` without refs. */ + /** The branch actually read, for a storage that reads one; `null` for a working tree. */ resolvedRef: string | null; files: StorageFile[]; } @@ -72,21 +63,8 @@ export interface StoredSchema { resolvedRef: string | null; } -/** A durable receipt for one `blocks.apply` request key. */ -export interface StoredReceipt { - /** The canonical digest of the request the key was first used with. */ - digest: string; - /** The revision that commit produced. */ - revision: string; - /** The versions of the files that commit wrote. */ - versions: Record; - /** The files that commit deleted (`attempt.delete`), so a replay reports the same names. */ - deleted?: string[]; -} - /** One atomic commit attempt. */ export interface CommitAttempt { - ref?: string; /** The snapshot the core validated against. */ base: StorageSnapshot; /** Files to write, by file name, with their full content. */ @@ -101,19 +79,11 @@ export interface CommitAttempt { expected: Record; /** * The schema version the core validated against (`null` = no schema), when - * the write depends on it (`ifSchemaMatch`, or the secret guard). When set, + * the write depends on it (the secret guard checks `set` against it). When set, * a storage MUST reject the attempt as stale if the schema's version at * commit time differs, checked atomically with `expected`. */ expectedSchemaVersion?: string | null; - /** - * Persist this receipt atomically with the mutation (only when idempotency - * is advertised). The key also reserves the request: a storage MUST reject - * the attempt as stale when a receipt for `receipt.key` already exists, - * including one written concurrently by another replica, so two requests - * with the same key never both commit. - */ - receipt?: { key: string; digest: string }; } export type CommitResult = @@ -129,9 +99,9 @@ export interface ContentStorage { describe(): StorageDescription | Promise; /** * Lists the saved-block files. Throws `StorageNotFoundError` when there's no - * `.deco` folder at all, or the ref can't be read. + * `.deco` folder at all. */ - snapshot(options: { ref?: string }): Promise; + snapshot(): Promise; /** * Reads file bodies (UTF-8 text) from a snapshot, each with the version of * the bytes read. A file that vanished is left out. A storage that can't @@ -140,18 +110,16 @@ export interface ContentStorage { */ readFiles(snapshot: StorageSnapshot, files: string[]): Promise>; /** Reads `schema.gen.json`, falling back to `meta.gen.json`; `null` when neither exists. */ - readSchema(options: { ref?: string }): Promise; + readSchema(): Promise; /** Reads `/.deco/secrets.pub`; `null` without one. */ readSecretsPublicKey(): Promise; /** One all-or-nothing commit attempt; durable when it resolves. */ commit(attempt: CommitAttempt): Promise; - /** Looks up a receipt by its scoped key; `null` when unknown or expired. */ - getReceipt?(key: string): Promise; /** Stores an uploaded file, never overwriting one; returns the stored file name. */ putAsset?(name: string, body: Uint8Array): Promise<{ name: string }>; } -/** Thrown by a storage when there's no `.deco` folder, or a ref can't be read or created. */ +/** Thrown by a storage when there's no `.deco` folder. */ export class StorageNotFoundError extends Error { constructor(message: string) { super(message); diff --git a/packages/blocks/src/protocol/storage/fs/fs.test.ts b/packages/blocks/src/protocol/storage/fs/fs.test.ts index 6a9966dda..c89d9014e 100644 --- a/packages/blocks/src/protocol/storage/fs/fs.test.ts +++ b/packages/blocks/src/protocol/storage/fs/fs.test.ts @@ -43,7 +43,7 @@ describe("versions", () => { }).trim(); expect(gitBlobHash(content)).toBe(expected); await writeFile(join(blocksDir(), "a.json"), content); - const snapshot = await createFsStorage({ root }).snapshot({}); + const snapshot = await createFsStorage({ root }).snapshot(); expect(snapshot.files).toEqual([ { file: "a.json", version: expected, size: Buffer.byteLength(content) }, ]); @@ -52,10 +52,10 @@ describe("versions", () => { it("change with the content, and the revision with any file", async () => { const storage = createFsStorage({ root }); await writeFile(join(blocksDir(), "a.json"), "{}\n"); - const first = await storage.snapshot({}); - expect((await storage.snapshot({})).revision).toBe(first.revision); + const first = await storage.snapshot(); + expect((await storage.snapshot()).revision).toBe(first.revision); await writeFile(join(blocksDir(), "a.json"), '{"v":2}\n'); - const second = await storage.snapshot({}); + const second = await storage.snapshot(); expect(second.revision).not.toBe(first.revision); expect(second.files[0].version).not.toBe(first.files[0].version); }); @@ -68,21 +68,19 @@ describe("snapshot", () => { await writeFile(join(blocksDir(), "notes.md"), "x"); await writeFile(join(blocksDir(), ".hidden.json"), "{}"); await mkdir(join(blocksDir(), "sub.json")); - const snapshot = await createFsStorage({ root }).snapshot({}); + const snapshot = await createFsStorage({ root }).snapshot(); expect(snapshot.files.map((f) => f.file)).toEqual(["a.json", "b.json"]); expect(snapshot.resolvedRef).toBeNull(); }); it("is empty when .deco has no blocks folder yet", async () => { await rm(blocksDir(), { recursive: true }); - expect((await createFsStorage({ root }).snapshot({})).files).toEqual([]); + expect((await createFsStorage({ root }).snapshot()).files).toEqual([]); }); it("fails with NotFound without a .deco folder", async () => { await rm(join(root, ".deco"), { recursive: true }); - await expect(createFsStorage({ root }).snapshot({})).rejects.toBeInstanceOf( - StorageNotFoundError, - ); + await expect(createFsStorage({ root }).snapshot()).rejects.toBeInstanceOf(StorageNotFoundError); await expect(clientFor().blocksList()).rejects.toMatchObject({ code: ErrorCode.NotFound }); }); }); @@ -90,11 +88,11 @@ describe("snapshot", () => { describe("schema and secrets", () => { it("reads schema.gen.json, falling back to meta.gen.json", async () => { const storage = createFsStorage({ root }); - expect(await storage.readSchema({})).toBeNull(); + expect(await storage.readSchema()).toBeNull(); await writeFile(join(root, ".deco", "meta.gen.json"), '{"v7":true}'); - expect((await storage.readSchema({}))?.text).toBe('{"v7":true}'); + expect((await storage.readSchema())?.text).toBe('{"v7":true}'); await writeFile(join(root, ".deco", "schema.gen.json"), '{"v8":true}'); - const stored = await storage.readSchema({}); + const stored = await storage.readSchema(); expect(stored).toEqual({ text: '{"v8":true}', version: gitBlobHash('{"v8":true}'), @@ -126,8 +124,6 @@ describe("describe", () => { kind: "working-tree", root: "apps/storefront", readOnly: false, - refs: null, - idempotency: null, pollIntervalMs: 2000, assets: { dir: "apps/storefront/public/assets", maxBytes: 25 * 1024 * 1024 }, }); @@ -178,7 +174,7 @@ describe("commits", () => { it("refuse file names that would escape the folder or be hidden, with a typed error", async () => { const storage = createFsStorage({ root }); - const base = await storage.snapshot({}); + const base = await storage.snapshot(); for (const file of ["../x.json", "a/b.json", ".lock.json", "..json"]) { await expect( storage.commit({ base, put: { [file]: "{}" }, delete: [], expected: {} }), @@ -204,8 +200,8 @@ describe("commits", () => { it("are stale when the schema changed since the core read it", async () => { await writeFile(join(root, ".deco", "schema.gen.json"), '{"v":1}'); const storage = createFsStorage({ root }); - const base = await storage.snapshot({}); - const schema = await storage.readSchema({}); + const base = await storage.snapshot(); + const schema = await storage.readSchema(); await writeFile(join(root, ".deco", "schema.gen.json"), '{"v":2}'); const attempt = { base, put: { "a.json": "{}\n" }, delete: [], expected: {} }; expect(await storage.commit({ ...attempt, expectedSchemaVersion: schema!.version })).toEqual({ @@ -215,7 +211,7 @@ describe("commits", () => { status: "stale", }); expect(await list()).toEqual([]); - const current = await storage.readSchema({}); + const current = await storage.readSchema(); expect( (await storage.commit({ ...attempt, expectedSchemaVersion: current!.version })).status, ).toBe("committed"); @@ -224,8 +220,8 @@ describe("commits", () => { it("compare the meta.gen.json fallback when there's no schema.gen.json", async () => { await writeFile(join(root, ".deco", "meta.gen.json"), '{"v":1}'); const storage = createFsStorage({ root }); - const base = await storage.snapshot({}); - const legacy = await storage.readSchema({}); + const base = await storage.snapshot(); + const legacy = await storage.readSchema(); await writeFile(join(root, ".deco", "schema.gen.json"), '{"v":2}'); const result = await storage.commit({ base, @@ -240,7 +236,7 @@ describe("commits", () => { it("read bodies with the version of the bytes actually read", async () => { const storage = createFsStorage({ root }); await writeFile(join(blocksDir(), "a.json"), "{}\n"); - const snapshot = await storage.snapshot({}); + const snapshot = await storage.snapshot(); await writeFile(join(blocksDir(), "a.json"), '{"edited":true}\n'); const bodies = await storage.readFiles(snapshot, ["a.json", "gone.json"]); expect(bodies).toEqual({ @@ -279,7 +275,7 @@ describe("commits", () => { it("are stale when an expected version changed, and write nothing", async () => { const storage = createFsStorage({ root }); await writeFile(join(blocksDir(), "a.json"), "{}\n"); - const base = await storage.snapshot({}); + const base = await storage.snapshot(); await writeFile(join(blocksDir(), "a.json"), '{"edited":true}\n'); const result = await storage.commit({ base, diff --git a/packages/blocks/src/protocol/storage/fs/index.ts b/packages/blocks/src/protocol/storage/fs/index.ts index 56264fc80..2f357b270 100644 --- a/packages/blocks/src/protocol/storage/fs/index.ts +++ b/packages/blocks/src/protocol/storage/fs/index.ts @@ -35,7 +35,6 @@ import { type StoredFileBody, type StoredSchema, } from "../../storage.ts"; -import type { Limits } from "../../types.ts"; import { gitBlobHash, revisionOf } from "./hash.ts"; import { withCommitLock } from "./lock.ts"; import { applyFileChange, sweepStaleTransactions } from "./transaction.ts"; @@ -57,8 +56,6 @@ export interface FsStorageOptions { assetsMaxBytes?: number; /** Overrides the poll interval (default 2000 ms). */ pollIntervalMs?: number; - /** Lowers the protocol's limits. */ - limits?: Partial; /** How long a commit waits for another process's lock (default 10 s). */ lockTimeoutMs?: number; } @@ -227,16 +224,13 @@ export function createFsStorage(options: FsStorageOptions): FsStorage { kind: "working-tree", root: relativeRoot, readOnly, - refs: null, assets: readOnly ? null : { dir: toPosix(relative(base, assetsDir)) || ".", maxBytes: options.assetsMaxBytes ?? DEFAULT_ASSETS_MAX_BYTES, }, - idempotency: null, pollIntervalMs: options.pollIntervalMs ?? 2000, - limits: options.limits, }; }, @@ -270,8 +264,6 @@ export function createFsStorage(options: FsStorageOptions): FsStorage { async commit(attempt): Promise { if (readOnly) throw new StorageUnavailableError("the storage is read-only"); - if (attempt.receipt) - throw new StorageUnavailableError("the filesystem storage keeps no receipts"); for (const file of [ ...Object.keys(attempt.put), ...attempt.delete, diff --git a/packages/blocks/src/protocol/storage/memory.ts b/packages/blocks/src/protocol/storage/memory.ts index 609eac8bd..e1480c8c2 100644 --- a/packages/blocks/src/protocol/storage/memory.ts +++ b/packages/blocks/src/protocol/storage/memory.ts @@ -1,8 +1,7 @@ /** * An in-memory `ContentStorage` (internal; not a package subpath). * - * A reference implementation of the storage contract, including durable - * request-key receipts persisted atomically with each commit, and a fixture + * A reference implementation of the storage contract, and a fixture * for tests: its state can be dumped and restored to simulate a restart, and * hooks let a test change files between a snapshot and a commit. Browser-safe. */ @@ -18,12 +17,9 @@ import { StorageNotFoundError, type StorageSnapshot, type StoredFileBody, - type StoredReceipt, type StoredSchema, } from "../storage.ts"; -const DAY_MS = 24 * 60 * 60 * 1000; - /** Everything a memory storage holds; JSON-serializable, so it survives a "restart". */ export interface MemoryStorageState { /** Saved-block files, by file name, with their content. */ @@ -33,7 +29,6 @@ export interface MemoryStorageState { /** The content of `meta.gen.json`, the fallback schema, or `null`. */ legacySchema?: string | null; secretsPublicKey: string | null; - receipts: Record; assets: Record; /** `false` simulates an app root with no `.deco` folder at all. */ hasDecoFolder: boolean; @@ -43,8 +38,6 @@ export interface MemoryStorageOptions { state?: Partial; /** Overrides parts of what `describe` reports. */ description?: Partial; - /** Clock for receipt retention (default `Date.now`). */ - now?: () => number; /** Runs after each file body is read; a test can change files here, mid-read. */ afterRead?: (file: string, storage: MemoryStorage) => void | Promise; /** Runs before each commit attempt's checks; a test can change files here. */ @@ -65,13 +58,11 @@ export interface MemoryStorage extends ContentStorage { const version = (content: string) => sha256Hex(content); export function createMemoryStorage(options: MemoryStorageOptions = {}): MemoryStorage { - const now = options.now ?? Date.now; const state: MemoryStorageState = structuredClone({ files: {}, schema: null, legacySchema: null, secretsPublicKey: null, - receipts: {}, assets: {}, hasDecoFolder: true, ...options.state, @@ -80,9 +71,7 @@ export function createMemoryStorage(options: MemoryStorageOptions = {}): MemoryS kind: "git", root: ".", readOnly: false, - refs: null, assets: { dir: "public/assets", maxBytes: 25 * 1024 * 1024 }, - idempotency: { retentionMs: DAY_MS }, ...options.description, }; let commits = 0; @@ -118,7 +107,6 @@ export function createMemoryStorage(options: MemoryStorageOptions = {}): MemoryS if (!isBlockFileName(file)) throw new StorageInvalidFileError(file); } if (!state.hasDecoFolder) throw new StorageNotFoundError("no .deco folder"); - if (attempt.receipt && state.receipts[attempt.receipt.key]) return { status: "stale" }; if ( attempt.expectedSchemaVersion !== undefined && (await schemaVersion()) !== attempt.expectedSchemaVersion @@ -138,15 +126,6 @@ export function createMemoryStorage(options: MemoryStorageOptions = {}): MemoryS for (const [file, v] of Object.entries(written)) versions.set(file, v); for (const file of attempt.delete) if (!(file in attempt.put)) versions.delete(file); const revision = await revisionOf(versions); - if (attempt.receipt) { - state.receipts[attempt.receipt.key] = { - digest: attempt.receipt.digest, - revision, - versions: written, - deleted: [...attempt.delete], - createdAt: now(), - }; - } commits++; return { status: "committed", revision, versions: written }; } @@ -201,20 +180,6 @@ export function createMemoryStorage(options: MemoryStorageOptions = {}): MemoryS return run; }, - async getReceipt(key) { - const receipt = state.receipts[key]; - if (!receipt) return null; - if ( - description.idempotency && - now() - receipt.createdAt > description.idempotency.retentionMs - ) { - delete state.receipts[key]; - return null; - } - const { createdAt: _createdAt, ...stored } = receipt; - return stored; - }, - async putAsset(name, body) { let candidate = name; while (candidate in state.assets) candidate = suffixedAssetName(name); diff --git a/packages/blocks/src/protocol/types.ts b/packages/blocks/src/protocol/types.ts index a4517a0e0..d7c5fa7c4 100644 --- a/packages/blocks/src/protocol/types.ts +++ b/packages/blocks/src/protocol/types.ts @@ -10,6 +10,17 @@ export const PROTOCOL_VERSION = { major: 1, minor: 0 } as const; export const SCHEMA_FORMAT = "deco-meta@1"; +/** + * The Blocks major a generated schema declares, as its top-level + * `blocksMajor`. `deco schema` writes it and `deco check` requires it. It is + * the package's major version, not the full version string, and it is the + * only signal a host such as the site editor should read to tell a v8 site + * from a v7 one: the schema's file name (`schema.gen.json` or + * `meta.gen.json`) doesn't tell. A schema without it, or with another value, + * is not a v8 schema. + */ +export const BLOCKS_MAJOR = 8; + /** The path the local server (`deco serve`) mounts the endpoint at. */ export const RPC_PATH = "/rpc"; @@ -19,33 +30,6 @@ export const MAX_BATCH_CALLS = 10; /** The URL prefix every uploaded asset is stored under in a field. */ export const ASSETS_URL_PREFIX = "/assets/"; -export interface Limits { - /** Names (set plus delete) in one `blocks.apply`. */ - maxOpsPerApply: number; - /** Bytes of one stored entry. */ - maxBlockBytes: number; - /** Bytes of one HTTP request body, uncompressed. */ - maxRequestBytes: number; - /** Uncompressed bytes of the whole block list. */ - maxListBytes: number; - /** Uncompressed bytes of the schema. */ - maxSchemaBytes: number; - /** Uncompressed bytes of a whole batch response. */ - maxBatchResponseBytes: number; -} - -const MiB = 1024 * 1024; - -/** The protocol's default limits. A storage or a server can lower them. */ -export const DEFAULT_LIMITS: Readonly = Object.freeze({ - maxOpsPerApply: 500, - maxBlockBytes: 1 * MiB, - maxRequestBytes: 8 * MiB, - maxListBytes: 16 * MiB, - maxSchemaBytes: 16 * MiB, - maxBatchResponseBytes: 32 * MiB, -}); - export type StorageKind = "working-tree" | "git"; export interface DescribeResult { @@ -58,12 +42,8 @@ export interface DescribeResult { /** The app root: the folder that contains `.deco/`, relative to the repository root. */ root: string; schemaFormat: typeof SCHEMA_FORMAT; - /** Branches; `null` on the local server. */ - refs: null | { default: string; autoCreate: boolean }; - writes: { idempotency: null | { retentionMs: number }; schemaPreconditions: boolean }; /** Local: 2000; git: 30000, plus on window focus. */ pollIntervalMs: number; - limits: Limits; /** * The app the site editor shows in its Preview tab (and where "open the real * page" points); the local server reports `deco serve --preview`. `null`: no preview. @@ -83,15 +63,23 @@ export interface DecoMeta { } export interface ReadParams { - ref?: string; ifNoneMatch?: string; } export type SchemaGetParams = ReadParams; +/** + * `schema: null` (with `version: null`) means the site has no schema yet: + * neither `.deco/schema.gen.json` nor `.deco/meta.gen.json` exists. It's a + * normal state, not an error: `blocks.list` and `blocks.apply` work without a + * schema, so a client still lists and edits the blocks (without typed forms) + * and keeps polling until a schema appears. There's no version to send as + * `ifNoneMatch`, so the poll reads it unconditionally; the answer is tiny. + */ export type SchemaGetResult = | { notModified: true; version: string } - | { notModified: false; version: string; resolvedRef: string | null; schema: DecoMeta }; + | { notModified: false; version: string; resolvedRef: string | null; schema: DecoMeta } + | { notModified: false; version: null; resolvedRef: string | null; schema: null }; export type BlocksListParams = ReadParams; @@ -122,11 +110,6 @@ export type BlocksListResult = }; export interface BlocksApplyParams { - ref?: string; - /** Retry the same logical write; only when advertised. */ - requestKey?: string; - /** Reject if the schema changed; only when advertised. */ - ifSchemaMatch?: string; /** Whole-entry replace (create or update). */ set?: Record>; /** A missing name counts as deleted. */ diff --git a/packages/blocks/src/v8/__conformance__/cli.test.ts b/packages/blocks/src/v8/__conformance__/cli.test.ts index f05bd4b77..1a3de6336 100644 --- a/packages/blocks/src/v8/__conformance__/cli.test.ts +++ b/packages/blocks/src/v8/__conformance__/cli.test.ts @@ -3,9 +3,9 @@ * Docs conformance: the `deco` CLI, `deco schema`, `deco check`, `deco serve` * and the content protocol as the local server exposes it. * - * Source of truth: docs/content/next/{cli,schema,checking,site-editor, - * content-protocol,studio-compatibility,studio-implementation}.mdx on the - * blocks-site `docs/blocks-site` branch. Each test is named after the claim it + * Source of truth: src/content/docs/en/storefront/blocks/next/{cli,schema,checking,site-editor, + * content-protocol,studio-compatibility,studio-implementation}.mdx in + * deco-sites/docs-tanstack. Each test is named after the claim it * checks (cli-01, sch-03, …). Tests for claims the code doesn't meet yet are * left failing on purpose: the fix step turns them green. */ @@ -19,6 +19,7 @@ import { gunzipSync } from "node:zlib"; import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { ErrorCode } from "../../protocol/errors"; import { blockFileName, blockNameFromFile } from "../../protocol/keys"; +import { BLOCKS_MAJOR } from "../../protocol/types"; import { createFixture, type Fixture, @@ -101,7 +102,6 @@ function rpcCall(server: RunningServer, body: unknown, headers: Record[0] = {}, ): Promise }> { const log = recorder(); - const s = await startServer({ cwd: root, port: 0, token: "t0k3n", reporter: log, ...options }); + const s = await startServer({ cwd: root, port: 0, reporter: log, ...options }); servers.push(s); return Object.assign(s, { log }); } @@ -172,11 +172,15 @@ export interface Seo { title: string; description: string } export interface Post extends Route { body: string } export interface Hero { title: string } export enum Tone { Light = "light", Dark = "dark" } +/** @format color */ +export type BrandColor = string; +/** @format color */ +export type TextTone = "black" | "white"; `, "src/sections.tsx": ` import type { ReactNode } from "react"; import type { Lazy, Secret } from "@decocms/blocks"; -import type { Hero, Post, Product, Seo, Tone } from "./model"; +import type { BrandColor, Hero, Post, Product, Seo, TextTone, Tone } from "./model"; export interface AllProps { text: string; @@ -222,6 +226,11 @@ export interface AllProps { fTextarea?: string; /** @format color */ fColor?: string; + brand?: BrandColor; + textTone?: TextTone | null; + palette?: BrandColor[]; + /** @format textarea */ + notes?: BrandColor; /** @format image-uri */ fImage?: string; /** @options ["sm", "md", "lg"] */ @@ -625,12 +634,6 @@ describe("cli.mdx", () => { "127.0.0.1", "--preview", "localhost:8001", - "--token", - "t", - "--allow-origin", - "https://a", - "--allow-origin", - "https://b", "--assets", "static", "--read-only", @@ -640,12 +643,10 @@ describe("cli.mdx", () => { port: "1", host: "127.0.0.1", preview: "localhost:8001", - token: "t", - "allow-origin": ["https://a", "https://b"], assets: "static", "read-only": true, }); - for (const flag of ["--watch", "--studio", "--open", "--cors"]) { + for (const flag of ["--watch", "--studio", "--open", "--cors", "--token", "--allow-origin"]) { const r = await deco(["serve", flag], f.root); expect(r.code).toBe(1); expect(r.out).toContain(`unknown flag ${flag}`); @@ -657,7 +658,7 @@ describe("cli.mdx", () => { const log = recorder(); let s: RunningServer; try { - s = await startServer({ cwd: f.root, reporter: log, token: "x" }); + s = await startServer({ cwd: f.root, reporter: log }); } catch (error) { // Something else holds 4545 on this machine: the error names the port. expect(String(error)).toContain("port 4545 is in use"); @@ -665,19 +666,19 @@ describe("cli.mdx", () => { } servers.push(s); expect(s.port).toBe(4545); - expect(s.endpoint).toBe("http://127.0.0.1:4545/rpc"); - expect(log.text()).toContain("http://127.0.0.1:4545/rpc"); + expect(s.endpoint).toBe("http://localhost:4545/rpc"); + expect(log.text()).toContain("http://localhost:4545/rpc"); }); - it("cli-15: --host defaults to loopback; another address warns", async () => { + it("cli-15: --host defaults to loopback, shown as localhost; another address warns", async () => { const f = fixture(); const loop = await serveFixture(f.root); - expect(loop.endpoint.startsWith("http://127.0.0.1:")).toBe(true); + expect(loop.endpoint.startsWith("http://localhost:")).toBe(true); expect(loop.log.lines.filter((l) => l.level === "warn")).toEqual([]); const open = await serveFixture(f.root, { host: "0.0.0.0" }); const warns = open.log.lines.filter((l) => l.level === "warn").map((l) => l.message); - expect(warns.join("\n")).toMatch(/other machines can reach this server/); + expect(warns.join("\n")).toMatch(/other machines on the network can reach this server/); }); it("cli-16: --preview defaults to the Vite config's port, else http://localhost:5173", async () => { @@ -698,47 +699,44 @@ describe("cli.mdx", () => { }); }); - it("cli-17: --token / DECO_SERVE_TOKEN fix the token; otherwise random per run", async () => { + it("cli-17: deco serve has no token: no Authorization, DECO_SERVE_TOKEN ignored, none in the link", async () => { const f = fixture(); - const a = await serveFixture(f.root, { token: "abc" }); - expect(a.token).toBe("abc"); - expect((await call(a, "describe")).result.protocol).toBe("deco-content"); - - const b = await serveFixture(f.root, { token: undefined, env: { DECO_SERVE_TOKEN: "abc" } }); - expect(b.token).toBe("abc"); - const ok = await raw(b.endpoint, { - headers: { "content-type": "application/json", authorization: "Bearer abc" }, + process.env.DECO_SERVE_TOKEN = "abc"; + let a: Awaited>; + try { + a = await serveFixture(f.root); + } finally { + delete process.env.DECO_SERVE_TOKEN; + } + expect("token" in a).toBe(false); + expect(a.siteEditorUrl).toBe( + `https://studio.decocms.com/site-editor#endpoint=${encodeURIComponent(a.endpoint)}`, + ); + expect(a.log.text()).not.toMatch(/token/i); + const plain = await raw(a.endpoint, { + headers: { "content-type": "application/json" }, body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }), }); - expect(ok.json().result.protocol).toBe("deco-content"); - - const c = await serveFixture(f.root, { token: undefined, env: {} }); - const d = await serveFixture(f.root, { token: undefined, env: {} }); - expect(c.token).not.toBe(d.token); - expect(c.token.length).toBeGreaterThanOrEqual(32); + expect(plain.status).toBe(200); + expect(plain.json().result.protocol).toBe("deco-content"); + // An Authorization header is neither needed nor checked. + const stray = await rpcCall( + a, + { jsonrpc: "2.0", id: 1, method: "describe" }, + { authorization: "Bearer anything" }, + ); + expect(stray.status).toBe(200); }); - it("cli-18 / cp-42: --allow-origin is repeatable; other origins are refused", async () => { + it("cli-18 / cp-42: any origin is answered, its Origin reflected", async () => { const f = fixture(); - const s = await serveFixture(f.root, { - allowOrigins: ["https://one.example", "https://two.example"], - }); - for (const origin of [ - "https://one.example", - "https://two.example", - "https://studio.decocms.com", - ]) { + const s = await serveFixture(f.root); + for (const origin of ["https://studio.decocms.com", "https://example.com"]) { const r = await rpcCall(s, { jsonrpc: "2.0", id: 1, method: "describe" }, { origin }); expect(r.status).toBe(200); expect(r.headers["access-control-allow-origin"]).toBe(origin); + expect(r.headers.vary).toBe("Origin"); } - const evil = await rpcCall( - s, - { jsonrpc: "2.0", id: 1, method: "describe" }, - { origin: "https://evil.example" }, - ); - expect(evil.status).toBe(403); - expect(evil.json().error.code).toBe(ErrorCode.Forbidden); }); it("cli-19: --assets is relative to the app root; the field always stores /assets/", async () => { @@ -746,7 +744,7 @@ describe("cli.mdx", () => { const s = await serveFixture(f.root, { assets: "static/img" }); const r = await raw(s.endpoint.replace("/rpc", "/assets/x.png"), { method: "PUT", - headers: { "content-type": "image/png", authorization: `Bearer ${s.token}` }, + headers: { "content-type": "image/png" }, body: Buffer.from([0x89, 0x50, 0x4e, 0x47]), }); expect(r.status).toBeGreaterThanOrEqual(200); @@ -767,7 +765,7 @@ describe("cli.mdx", () => { expect(f.exists(".deco/blocks/A.json")).toBe(false); const up = await raw(s.endpoint.replace("/rpc", "/assets/x.png"), { method: "PUT", - headers: { "content-type": "image/png", authorization: `Bearer ${s.token}` }, + headers: { "content-type": "image/png" }, body: Buffer.from([1, 2, 3]), }); expect(up.status).toBeGreaterThanOrEqual(400); @@ -785,7 +783,6 @@ describe("cli.mdx", () => { expect(d.root).toBe("apps/storefront"); expect(d.server.name).toBe("deco-cli"); expect(d.kind).toBe("working-tree"); - expect(d.refs).toBeNull(); expect(d.pollIntervalMs).toBe(2000); expect(d.assets.dir).toBe("apps/storefront/public/assets"); expect(d.preview).toEqual({ url: "http://localhost:3001" }); @@ -970,6 +967,25 @@ describe("schema.mdx", () => { expect(p.fImage.format).toBe("image-uri"); }); + it("sch-26: @format on a type alias reaches every field of that type, a select included", () => { + const p = propsOf("everything").properties; + expect(p.brand).toMatchObject({ type: "string", format: "color" }); + expect(p.textTone).toMatchObject({ type: "string", enum: ["black", "white"], format: "color" }); + expect(p.palette.items).toMatchObject({ type: "string", format: "color" }); + expect(p.notes.format).toBe("textarea"); // the field's own tag wins + }); + + it('sch-27: the schema declares a top-level "blocksMajor": 8, the package major', () => { + expect(meta.blocksMajor).toBe(8); + expect(meta.blocksMajor).toBe(BLOCKS_MAJOR); + // The major, not the version string. A published build's package.json + // major must agree with it (the source tree sits at 0.0.0). + const version: string = JSON.parse( + fs.readFileSync(path.join(PKG, "package.json"), "utf8"), + ).version; + if (version !== "0.0.0") expect(Number(version.split(".")[0])).toBe(BLOCKS_MAJOR); + }); + it("sch-18 / sch-21: @options, literal unions and enums all become a JSON Schema enum", () => { const p = propsOf("everything").properties; expect(p.picked).toMatchObject({ type: "string", enum: ["sm", "md", "lg"] }); @@ -1236,6 +1252,21 @@ describe("checking.mdx", () => { }).forRelease(); expect(await cms.resolve("H")).toEqual([{ hero: "x" }, null]); }); + + it('chk-16: deco check rejects a schema without "blocksMajor": 8 and says to run deco schema', async () => { + const f = fixture(STORE_FILES); + expect((await deco(["schema"], f.root)).code).toBe(0); + expect(JSON.parse(f.read(".deco/schema.gen.json")).blocksMajor).toBe(8); + expect((await deco(["check"], f.root)).code).toBe(0); + const { blocksMajor: _, ...legacy } = JSON.parse(f.read(".deco/schema.gen.json")); + for (const schema of [legacy, { ...legacy, blocksMajor: 7 }]) { + f.write(".deco/schema.gen.json", schema); + const r = await deco(["check"], f.root); + expect(r.code).toBe(1); + expect(r.out).toContain('expected "blocksMajor": 8'); + expect(r.out).toContain("run deco schema"); + } + }, 60_000); }); // =========================================================================== @@ -1250,23 +1281,20 @@ describe("site-editor.mdx", () => { ".deco/blocks/B.json": { __resolveType: "page", name: "B", path: "/b" }, }); fs.writeFileSync(path.join(appRoot, ".deco/schema.gen.json"), "{}\n"); - const s = await serveFixture(path.join(appRoot, "src"), { - env: {}, - token: "secret-token", - }); + const s = await serveFixture(path.join(appRoot, "src")); expect(repo).toBeTruthy(); const lines = s.log.lines.map((l) => l.message); const port = s.port; expect(lines).toEqual([ - `Deco server http://127.0.0.1:${port}/rpc`, + `Deco server http://localhost:${port}/rpc`, "Root apps/storefront (.deco/schema.gen.json, 2 blocks)", "Assets apps/storefront/public/assets (PUT /assets/)", "Preview http://localhost:5173", - `Site editor https://studio.decocms.com/site-editor#endpoint=${encodeURIComponent(`http://127.0.0.1:${port}/rpc`)}&token=secret-token`, + `Site editor https://studio.decocms.com/site-editor#endpoint=${encodeURIComponent(`http://localhost:${port}/rpc`)}`, ]); const url = new URL(s.siteEditorUrl); expect(url.search).toBe(""); - expect(url.hash).toContain("token=secret-token"); + expect(url.hash).not.toContain("token"); }); it("se-02: the RPC endpoint is /rpc; other paths 404", async () => { @@ -1274,7 +1302,7 @@ describe("site-editor.mdx", () => { const s = await serveFixture(f.root); expect((await call(s, "describe")).result.protocol).toBe("deco-content"); const other = await raw(s.endpoint.replace("/rpc", "/api"), { - headers: { "content-type": "application/json", authorization: `Bearer ${s.token}` }, + headers: { "content-type": "application/json" }, body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }), }); expect(other.status).toBe(404); @@ -1324,10 +1352,10 @@ describe("site-editor.mdx", () => { it("se-05 / cp-45: uploads never reuse a name", async () => { const f = fixture(); const s = await serveFixture(f.root); - const put = (body: Buffer, type = "image/jpeg", token = s.token, name = "summer-banner.jpg") => + const put = (body: Buffer, type = "image/jpeg", name = "summer-banner.jpg") => raw(s.endpoint.replace("/rpc", `/assets/${name}`), { method: "PUT", - headers: { "content-type": type, authorization: `Bearer ${token}` }, + headers: { "content-type": type }, body, }); const ok = (status: number) => status >= 200 && status < 300; @@ -1340,32 +1368,31 @@ describe("site-editor.mdx", () => { expect(second.json().path).toMatch(/^\/assets\/summer-banner.+\.jpg$/); expect(f.read("public/assets/summer-banner.jpg")).toBe("first"); expect(ok((await put(Buffer.from("

"), "text/html")).status)).toBe(false); - expect((await put(Buffer.from("x"), "image/jpeg", "wrong")).status).toBe(401); for (const [type, name] of [ ["video/mp4", "clip.mp4"], ["font/woff2", "brand.woff2"], ["application/pdf", "terms.pdf"], ]) { - expect(ok((await put(Buffer.from("x"), type, s.token, name)).status), type).toBe(true); + expect(ok((await put(Buffer.from("x"), type, name)).status), type).toBe(true); } const max = (await call(s, "describe")).result.assets.maxBytes; expect(typeof max).toBe("number"); }); - it("se-06: answers Chrome's local-network preflight", async () => { + it("se-06: answers Chrome's local-network preflight, from any origin", async () => { const f = fixture(); const s = await serveFixture(f.root); const r = await raw(s.endpoint, { method: "OPTIONS", headers: { - origin: "https://studio.decocms.com", + origin: "https://example.com", "access-control-request-method": "POST", "access-control-request-private-network": "true", }, }); expect(r.status).toBe(204); expect(r.headers["access-control-allow-private-network"]).toBe("true"); - expect(r.headers["access-control-allow-origin"]).toBe("https://studio.decocms.com"); + expect(r.headers["access-control-allow-origin"]).toBe("https://example.com"); }); it("se-07: describe.secrets.publicKey is .deco/secrets.pub", async () => { @@ -1437,7 +1464,6 @@ describe("content-protocol.mdx", () => { expect(batch.json().map((r: Json) => r.id)).toEqual([1, 2]); const get = await raw(s.endpoint, { method: "GET", - headers: { authorization: `Bearer ${s.token}` }, }); expect(get.status).toBeGreaterThanOrEqual(400); }); @@ -1464,25 +1490,23 @@ describe("content-protocol.mdx", () => { expect(app.exists(".deco/blocks/cp06.json")).toBe(false); }); - it("cp-07 / cp-41: 401 for a bad token, 413 for an oversized body, 200 for a conflict", async () => { + it("cp-07 / cp-41: deco serve never answers 401; 413 for an oversized body, 200 for a conflict", async () => { const noAuth = await raw(s.endpoint, { headers: { "content-type": "application/json" }, body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }), }); - expect(noAuth.status).toBe(401); - expect(noAuth.json().error.code).toBe(ErrorCode.Unauthorized); - const bad = await rpcCall( + expect(noAuth.status).toBe(200); + const stray = await rpcCall( s, { jsonrpc: "2.0", id: 1, method: "describe" }, - { - authorization: "Bearer nope", - }, + { authorization: "Bearer nope" }, ); - expect(bad.status).toBe(401); + expect(stray.status).toBe(200); + expect(stray.json().error).toBeUndefined(); // fetch, like a browser: Node's raw http client sees the early 413 as a reset. const huge = await fetch(s.endpoint, { method: "POST", - headers: { "content-type": "application/json", authorization: `Bearer ${s.token}` }, + headers: { "content-type": "application/json" }, body: JSON.stringify({ jsonrpc: "2.0", id: 1, @@ -1549,10 +1573,7 @@ describe("content-protocol.mdx", () => { "readOnly", "root", "schemaFormat", - "refs", - "writes", "pollIntervalMs", - "limits", "preview", "assets", "secrets", @@ -1563,17 +1584,6 @@ describe("content-protocol.mdx", () => { expect(typeof d.version.minor).toBe("number"); expect(Object.keys(d.server).sort()).toEqual(["name", "version"]); expect(d.schemaFormat).toBe("deco-meta@1"); - expect(Object.keys(d.writes).sort()).toEqual(["idempotency", "schemaPreconditions"]); - expect(Object.keys(d.limits).sort()).toEqual( - [ - "maxOpsPerApply", - "maxBlockBytes", - "maxRequestBytes", - "maxListBytes", - "maxSchemaBytes", - "maxBatchResponseBytes", - ].sort(), - ); expect(Object.keys(d.assets).sort()).toEqual(["dir", "maxBytes", "urlPrefix"]); expect(d.assets.urlPrefix).toBe("/assets/"); }); @@ -1638,9 +1648,21 @@ describe("content-protocol.mdx", () => { expect(torn.error).toBeDefined(); }); - it("cp-17: ref on the local server is Unsupported", async () => { - expect((await call(s, "blocks.list", { ref: "main" })).error.code).toBe(ErrorCode.Unsupported); - expect((await call(s, "schema.get", { ref: "main" })).error.code).toBe(ErrorCode.Unsupported); + it('cp-51: schema.get serves "blocksMajor" untouched, and a schema without it as it is', async () => { + const served = (await call(s, "schema.get")).result.schema; + expect(served.blocksMajor).toBe(8); + expect(served).toEqual(JSON.parse(app.read(".deco/schema.gen.json"))); + const f = fixture(); + f.write(".deco/meta.gen.json", { manifest: { blocks: {} }, schema: { definitions: {} } }); + const s1 = await serveFixture(f.root); + expect((await call(s1, "schema.get")).result.schema).not.toHaveProperty("blocksMajor"); + }); + + it("cp-17: the protocol has no ref parameter; one is an unknown parameter", async () => { + expect((await call(s, "blocks.list", { ref: "main" })).error.code).toBe( + ErrorCode.InvalidParams, + ); + expect((await call(s, "schema.get", { ref: "main" })).error.code).toBe(ErrorCode.InvalidParams); }); it("cp-18 / cp-21: validation first, every violation at once, nothing written", async () => { @@ -1693,34 +1715,12 @@ describe("content-protocol.mdx", () => { expect(app.exists(".deco/blocks/cp24new.json")).toBe(true); }); - it("cp-25: unadvertised requestKey / ifSchemaMatch are rejected, not ignored", async () => { - const d = (await call(s, "describe")).result; - if (d.writes.idempotency === null) { - const r = await call(s, "blocks.apply", { set: { cp25: entry("cp25") }, requestKey: "k1" }); - expect(r.error.code).toBe(ErrorCode.Unsupported); - expect(app.exists(".deco/blocks/cp25.json")).toBe(false); - } - if (d.writes.schemaPreconditions === false) { - const r = await call(s, "blocks.apply", { set: { cp25: entry("cp25") }, ifSchemaMatch: "v" }); - expect(r.error.code).toBe(ErrorCode.Unsupported); + it("cp-25: unknown parameters (requestKey, ifSchemaMatch, ref) are rejected, not ignored", async () => { + for (const param of ["requestKey", "ifSchemaMatch", "ref"]) { + const r = await call(s, "blocks.apply", { set: { cp25: entry("cp25") }, [param]: "x" }); + expect(r.error.code).toBe(ErrorCode.InvalidParams); } - }); - - it("cp-27: ifSchemaMatch mismatch is a Conflict with schema versions (when advertised)", async () => { - const d = (await call(s, "describe")).result; - expect(d.writes.schemaPreconditions).toBe(true); - const { version } = (await call(s, "schema.get")).result; - const r = await call(s, "blocks.apply", { - set: { cp27: entry("cp27") }, - ifSchemaMatch: `${version}-old`, - }); - expect(r.error.code).toBe(ErrorCode.Conflict); - expect(r.error.data.schema).toEqual({ expected: `${version}-old`, actual: version }); - const ok = await call(s, "blocks.apply", { - set: { cp27: entry("cp27") }, - ifSchemaMatch: version, - }); - expect(ok.result).toBeDefined(); + expect(app.exists(".deco/blocks/cp25.json")).toBe(false); }); it("cp-28 / cp-29: the poll batch, and adopting the own write's revision", async () => { @@ -1757,10 +1757,18 @@ describe("content-protocol.mdx", () => { }); }); - it("cp-31: no schema file gives NotFound", async () => { + it("cp-31: no schema file gives schema: null, and blocks still list and save", async () => { const f = fixture(); const s1 = await serveFixture(f.root); - expect((await call(s1, "schema.get")).error.code).toBe(ErrorCode.NotFound); + expect((await call(s1, "schema.get")).result).toEqual({ + notModified: false, + version: null, + resolvedRef: null, + schema: null, + }); + const block = { __resolveType: "site/sections/Hero.tsx", title: "Hi", items: [{ n: 1 }] }; + expect((await call(s1, "blocks.apply", { set: { cp31: block } })).result).toBeTruthy(); + expect((await call(s1, "blocks.list")).result.blocks.cp31).toEqual(block); }); it("cp-32: a Secret field holding a plain string is InvalidBlock", async () => { @@ -1771,16 +1779,8 @@ describe("content-protocol.mdx", () => { expect(app.exists(".deco/blocks/cp32.json")).toBe(false); }); - it("cp-33 / si-05 / si-06: default limits", async () => { - const { limits } = (await call(s, "describe")).result; - expect(limits).toEqual({ - maxOpsPerApply: 500, - maxBlockBytes: 1024 * 1024, - maxRequestBytes: 8 * 1024 * 1024, - maxListBytes: 16 * 1024 * 1024, - maxSchemaBytes: 16 * 1024 * 1024, - maxBatchResponseBytes: 32 * 1024 * 1024, - }); + it("cp-33 / si-05 / si-06: blocks.apply takes at most 500 names; describe advertises no limits", async () => { + expect((await call(s, "describe")).result.limits).toBeUndefined(); const set = Object.fromEntries( Array.from({ length: 501 }, (_, i) => [`cp33-${i}`, { __resolveType: "page" }]), ); @@ -1839,25 +1839,9 @@ describe("content-protocol.mdx", () => { expect(read("protocol/storage/fs/index.ts")).toMatch(/from "\.\.\/\.\.\/keys\.ts"/); }); - it("cp-43: Host must be the server's own loopback address", async () => { - const body = JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }); - const headers = (host: string) => ({ - host, - "content-type": "application/json", - authorization: `Bearer ${s.token}`, - }); - expect((await raw(s.endpoint, { headers: headers("attacker.com"), body })).status).toBe(403); - expect((await raw(s.endpoint, { headers: headers(`127.0.0.1:${s.port}`), body })).status).toBe( - 200, - ); - expect((await raw(s.endpoint, { headers: headers(`localhost:${s.port}`), body })).status).toBe( - 200, - ); - }); - it("cp-44: non-JSON Content-Type on /rpc is refused", async () => { const r = await raw(s.endpoint, { - headers: { "content-type": "text/plain", authorization: `Bearer ${s.token}` }, + headers: { "content-type": "text/plain" }, body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }), }); expect(r.status).toBe(415); @@ -1934,12 +1918,8 @@ describe("content-protocol.mdx", () => { "apply/atomic-set-delete", "apply/set-wins", "apply/if-match", - "apply/schema-precondition", - "idempotency/retry", - "idempotency/simultaneous", - "idempotency/restart", - "idempotency/tenant-isolation", - "limits/list-bytes", + "apply/unknown-params", + "schema/absent", "limits/request-bytes", ]) { expect(ids).toContain(id); @@ -1959,7 +1939,6 @@ describe("content-protocol.mdx", () => { const report = await runConformance({ endpoint: s1.endpoint, assetsEndpoint: s1.endpoint.replace("/rpc", "/assets/"), - token: s1.token, secretField: { blockType: "hero", field: "apiKey" }, secretsPublicKey: publicKeyPem, } as never); @@ -2005,7 +1984,7 @@ describe("studio-compatibility.mdx", () => { ); expect(Object.keys(g.pages)).toEqual(expect.arrayContaining(["page", "post"])); expect(Object.keys(g.redirects)).toContain("redirect"); - expect(Object.keys(g.content)).toEqual(expect.arrayContaining(["seo", "hero", "telemetry"])); + expect(Object.keys(g.content)).toEqual(expect.arrayContaining(["seo", "hero", "cms-settings"])); }); it("sc-02: a render-descriptor-returning function is a section", async () => { @@ -2211,14 +2190,8 @@ export default { page } satisfies Blocks; // =========================================================================== describe("studio-implementation.mdx", () => { - it("si-02 / si-03 / si-04: canonical hashing, goldens and a separate request-digest domain", async () => { - const { - canonicalJson, - computeContentRevision, - applyRequestDigest, - APPLY_DIGEST_DOMAIN, - sha256Hex, - } = await import("@decocms/blocks/protocol"); + it("si-02 / si-03 / si-04: canonical hashing and goldens", async () => { + const { canonicalJson, computeContentRevision } = await import("@decocms/blocks/protocol"); expect(canonicalJson({ b: 1, a: [2, 1], "10": 0, "9": 0 })).toBe( '{"10":0,"9":0,"a":[2,1],"b":1}', ); @@ -2226,15 +2199,6 @@ describe("studio-implementation.mdx", () => { for (const bad of [undefined, Number.NaN, () => 1, 1n]) { expect(() => canonicalJson({ x: bad }), String(bad)).toThrow(); } - const blocks = { a: { __resolveType: "x" } }; - expect(await computeContentRevision(blocks)).not.toBe( - await applyRequestDigest({ set: blocks }), - ); - expect(APPLY_DIGEST_DOMAIN).toMatch(/blocks\.apply/); - expect(await applyRequestDigest({ set: blocks })).toBe( - await sha256Hex(APPLY_DIGEST_DOMAIN + canonicalJson({ set: blocks })), - ); - const { contentHashFixtures } = await import("@decocms/blocks/protocol/conformance"); expect(contentHashFixtures.length).toBeGreaterThan(0); for (const fx of contentHashFixtures as any[]) { @@ -2327,14 +2291,26 @@ describe("studio-implementation.mdx", () => { expect(await loader.update!()).toEqual({ updated: true }); }); - it("si-10: an exact draft revision is fetched with If-Match; a failure is an error, not the head", async () => { + it("si-10: a draft's changes apply to the production this server has; no matching base is fetched", async () => { const fallback = await snap("bundled"); - const loader = remoteLoader(fallback, { site: "acme", token: "t" }) as Loader; - await expect(loader.load("delivery.decocms.com/drafts/acme/feat@v1")).rejects.toThrow( - /HTTP 404/, - ); - // Before any release check, the bundled fallback is served. - expect(await loader.load()).toBe(fallback); + const published = await snap("published"); + publish(1, published); + gates.set(`/sites/acme/revisions/${published.revision}.json`, new Promise(() => {})); // never arrives + const studio = "https://studio.decocms.com/api/acme/decofile/store/b/changes"; + const fetch = globalThis.fetch; + const urls: string[] = []; + vi.stubGlobal("fetch", async (input: string | URL | Request) => { + urls.push(String(input)); + return String(input).startsWith(studio) + ? Response.json({ format: 1, set: { New: { __resolveType: "seo" } }, delete: [] }) + : fetch(input); + }); + const cms = createCMS({ blocks: {}, content: fallback, site: "acme", token: "t" }); + const draft = cms.forDraft("studio.decocms.com/api/acme/decofile/store/b/changes?token=x@c1"); + const [entry] = await draft.resolve("S", { run: false }); + expect(entry).toEqual(fallback.blocks.S); // the bundled release, never waiting for the new one + expect(urls.filter((url) => url.startsWith(studio))).toHaveLength(1); + expect(urls.filter((url) => url.includes("/revisions/"))).toHaveLength(0); }); }); }); diff --git a/packages/blocks/src/v8/__conformance__/delivery.cli.test.ts b/packages/blocks/src/v8/__conformance__/delivery.cli.test.ts index 8b9bb54a0..a7436c2b0 100644 --- a/packages/blocks/src/v8/__conformance__/delivery.cli.test.ts +++ b/packages/blocks/src/v8/__conformance__/delivery.cli.test.ts @@ -131,7 +131,6 @@ describe("draft-synchronization", () => { describe("against a running deco serve", () => { let site: Fixture; let server: Awaited>; - const TOKEN = "conformance-token"; beforeAll(async () => { site = createFixture({ ".deco/blocks/Hero.json": hero }); site.write(".deco/schema.gen.json", meta); @@ -139,16 +138,14 @@ describe("draft-synchronization", () => { server = await startServer({ cwd: site.root, port: 0, - token: TOKEN, reporter: recorder(), - env: {}, }); }); afterAll(async () => { await server?.close(); site?.remove(); }); - const client = () => createContentClient({ endpoint: server.endpoint, token: TOKEN }); + const client = () => createContentClient({ endpoint: server.endpoint }); it("DS-5: a save with a stale ifMatch is a Conflict; without ifMatch the last writer wins", async () => { const listed = await client().blocksList(); @@ -205,20 +202,17 @@ describe("hosted-site-editor", () => { fixture = createFixture({ ".deco/blocks/Hero.json": hero }); fixture.write(".deco/schema.gen.json", meta); fs.mkdirSync(path.join(fixture.root, "src", "deep"), { recursive: true }); - const TOKEN = "t"; for (const options of [ { cwd: path.join(fixture.root, "src", "deep") }, { cwd: "/", root: fixture.root, assets: "static/uploads" }, ]) { const server = await startServer({ port: 0, - token: TOKEN, reporter: recorder(), - env: {}, ...options, }); try { - const c = createContentClient({ endpoint: server.endpoint, token: TOKEN }); + const c = createContentClient({ endpoint: server.endpoint }); const listed = await c.blocksList(); expect(listed.notModified ? null : Object.keys(listed.blocks)).toEqual(["Hero"]); const d = await c.describe(); diff --git a/packages/blocks/src/v8/__conformance__/delivery.snippets.test.ts b/packages/blocks/src/v8/__conformance__/delivery.snippets.test.ts index 086605386..9c443482f 100644 --- a/packages/blocks/src/v8/__conformance__/delivery.snippets.test.ts +++ b/packages/blocks/src/v8/__conformance__/delivery.snippets.test.ts @@ -89,16 +89,16 @@ console.log((await loader.load()).revision); `, // HD-7: hosted-drafts.mdx, "Wire drafts into your app" (a plain request handler). - "drafts/handler.ts": `import { draftPointer, draftCookie, type Client } from "@decocms/blocks"; + "drafts/handler.ts": `import type { Client } from "@decocms/blocks"; import { cms } from "../hosted/cms"; declare function render(client: Client, request: Request): Promise; export async function handle(request: Request) { - const pointer = draftPointer(request); // from ?__draft= or the cookie; null on an ordinary request + const pointer = await cms.draftPointer(request); // from ?__draft= or the cookie; null on an ordinary request const client = pointer ? cms.forDraft(pointer) : cms.forRelease(); const response = await render(client, request); // your own function: list, matchRoute and resolve as usual - const cookie = draftCookie(request); + const cookie = await cms.draftCookie(request); if (cookie) response.headers.append("Set-Cookie", cookie); return response; } @@ -106,11 +106,11 @@ export async function handle(request: Request) { // HD-9: hosted-drafts.mdx, Next.js proxy.ts. "next/src/proxy.ts": `import { NextResponse, type NextRequest } from "next/server"; -import { draftCookie } from "@decocms/blocks"; +import { cms } from "../../hosted/cms"; -export function proxy(request: NextRequest) { +export async function proxy(request: NextRequest) { const response = NextResponse.next(); - const cookie = draftCookie(request); // only set on the request the site editor opens; expiring when leaving preview + const cookie = await cms.draftCookie(request); // only set on the request the site editor opens; expiring when leaving preview if (cookie) response.headers.append("Set-Cookie", cookie); return response; } @@ -118,36 +118,36 @@ export function proxy(request: NextRequest) { // HD-10: hosted-drafts.mdx, Next.js client.server.ts. "next/src/client.server.ts": `import "server-only"; -import { cookies } from "next/headers"; -import { DRAFT_COOKIE } from "@decocms/blocks"; +import { headers } from "next/headers"; import { cms } from "../../hosted/cms"; -// The client for this request: a draft when the cookie holds a valid pointer, production otherwise. +// The client for this request: a draft when the cookie holds a valid pointer on an allowed host, production otherwise. export async function client() { - const pointer = (await cookies()).get(DRAFT_COOKIE)?.value; + const h = await headers(); + const pointer = await cms.draftPointer({ url: \`https://\${h.get("host")}/\`, headers: h }); return pointer ? cms.forDraft(pointer) : cms.forRelease(); } `, // HD-11: hosted-drafts.mdx, TanStack Start src/cms.ts (changes) and src/start.ts. - "tanstack/src/cms.ts": `import { createCMS, draftPointer } from "@decocms/blocks"; + "tanstack/src/cms.ts": `import { createCMS } from "@decocms/blocks"; import blocks from "../../.deco"; import content from "../../.deco/blocks.gen"; export const cms = createCMS({ blocks, content }); // The client for this request: the draft it points at, or the current release. -export const client = (request: Request) => { - const pointer = draftPointer(request); +export const client = async (request: Request) => { + const pointer = await cms.draftPointer(request); return pointer ? cms.forDraft(pointer) : cms.forRelease(); }; `, "tanstack/src/start.ts": `import { createMiddleware, createStart } from "@tanstack/react-start"; -import { draftCookie } from "@decocms/blocks"; +import { cms } from "./cms"; const draftCookieMiddleware = createMiddleware().server(async ({ request, next }) => { const result = await next(); - const cookie = draftCookie(request); // only set on the request the site editor opens; expiring when leaving preview + const cookie = await cms.draftCookie(request); // only set on the request the site editor opens; expiring when leaving preview if (cookie) result.response.headers.append("Set-Cookie", cookie); return result; }); @@ -166,14 +166,25 @@ const pointer = new URL(url).searchParams.get("__draft"); setClient(pointer ? cms.forDraft(pointer) : cms.forRelease()); `, - // HD-18: hosted-drafts.mdx, "Who may preview" (preview domain only). - "drafts/preview-domain.ts": `import { draftPointer } from "@decocms/blocks"; -import { cms } from "../hosted/cms"; + // HD-18: hosted-drafts.mdx, "Who may preview" (narrowing previews further). + "drafts/preview-host.ts": `import { cms } from "../hosted/cms"; +declare function isEmployee(request: Request): boolean; declare const request: Request; -const host = new URL(request.url).host; -const pointer = host === "preview.store.example.com" ? draftPointer(request) : null; -export const client = pointer ? cms.forDraft(pointer) : cms.forRelease(); +const pointer = isEmployee(request) ? await cms.draftPointer(request) : null; // isEmployee: your rule +export { pointer }; +`, + + // RD-11: releases-and-drafts.mdx, "Allow previews per host" (code caps the list). + "drafts/preview-cap.ts": `import { createCMS } from "@decocms/blocks"; +import blocks from "../.deco"; +import content from "../.deco/blocks.gen"; + +export const cms = createCMS({ + blocks, + content, + preview: { hosts: ["*.example.com", "localhost:3000"] }, // the most content may allow +}); `, }; @@ -233,7 +244,8 @@ describe("hosted docs examples compile", () => { ["HD-11", "tanstack/src/cms.ts"], ["HD-11", "tanstack/src/start.ts"], ["HD-13", "drafts/native.ts"], - ["HD-18", "drafts/preview-domain.ts"], + ["HD-18", "drafts/preview-host.ts"], + ["RD-11", "drafts/preview-cap.ts"], ])("%s: %s", (_claim, file) => { expect(errorsIn(file)).toEqual([]); }); diff --git a/packages/blocks/src/v8/__conformance__/delivery.test.ts b/packages/blocks/src/v8/__conformance__/delivery.test.ts index 0f0f7066b..00d862d8e 100644 --- a/packages/blocks/src/v8/__conformance__/delivery.test.ts +++ b/packages/blocks/src/v8/__conformance__/delivery.test.ts @@ -2,9 +2,9 @@ /** * Docs conformance: content-delivery, draft-synchronization, * releases-and-deployment, hosted, hosted-publishing, hosted-drafts and - * hosted-releases-internals (docs/content/next/*.mdx on the blocks-site - * branch). Each test names the claim it checks (CD-*, RD-*, H-*, HP-*, HD-*, - * HRI-*). A failing test is a claim the code doesn't meet yet. + * hosted-releases-internals (src/content/docs/en/storefront/blocks/next/*.mdx in + * deco-sites/docs-tanstack). Each test names the claim it checks (CD-*, RD-*, H-*, HP-*, HD-*, + * HRI-*, DP-*). A failing test is a claim the code doesn't meet yet. */ import fs from "node:fs"; import path from "node:path"; @@ -13,11 +13,9 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import * as root from "../../index"; import { computeContentRevision } from "../../protocol/canonical"; import { HOSTED_ANALYTICS_COLLECTOR } from "../builtins/data"; +import { instanceOf } from "../cms"; import { createCMS, - DRAFT_COOKIE, - draftCookie, - draftPointer, formatDraftPointer, matchRoute, parseDraftPointer, @@ -25,16 +23,24 @@ import { resetForTests, } from "../index"; import { resolveDestination } from "../telemetry"; -import { docsBlocks, docsSnapshot } from "../testFixtures"; -import type { Loader, Snapshot } from "../types"; +import { docsBlocks, docsSnapshot, fakeStudio, STUDIO_HOST, STUDIO_TOKEN } from "../testFixtures"; +import type { Loader, RequestLike, Snapshot } from "../types"; + +/** cms.draftPointer and cms.draftCookie on a CMS with no settings: every host may preview. */ +const helpers = () => createCMS({ blocks: {}, content: { revision: "draft-helpers", blocks: {} } }); +const draftPointer = (request: RequestLike) => helpers().draftPointer(request); +const draftCookie = (request: RequestLike) => helpers().draftCookie(request); +const DRAFT_COOKIE = "__deco_draft"; const HERE = path.dirname(fileURLToPath(import.meta.url)); const ORIGIN = "https://delivery.decocms.com"; -const HOST = new URL(ORIGIN).host; const SITE = "acme"; const TOKEN = "site-token"; const MANIFEST_URL = `${ORIGIN}/sites/acme/channels/production.json`; -const POINTER = `${HOST}/drafts/acme/feat-summer?token=abc@9f3c1a`; +/** A pointer string, for the cookie and parsing claims; drafts that load come from api.draft(). */ +const POINTER = `${STUDIO_HOST}/api/acme/decofile/store/summer-sale/changes?token=abc@9f3c1a`; +/** Where the fake Studio answers drafts. */ +const STUDIO = `https://${STUDIO_HOST}`; const remote = (fallback: Snapshot | Loader) => remoteLoader(fallback, { site: SITE, token: TOKEN }) as Loader; @@ -79,34 +85,37 @@ function deferred(): Deferred { return { promise, resolve }; } -/** A fake delivery API: a channel manifest, immutable revision assets and drafts by version. */ +/** A saved seo entry, as a draft changes it. */ +const seoEntry = (title: string) => ({ __resolveType: "seo", title, description: "d" }); + +/** + * A fake delivery API (a channel manifest and immutable revision assets) and + * a fake Studio API answering draft pointers (see testFixtures' fakeStudio). + */ function deliveryApi() { + const studio = fakeStudio(); const assets = new Map(); - const drafts = new Map(); const gates = new Map(); let manifest: Record | undefined; let failAll = false; - const requests: { url: string; headers: Record }[] = []; + const requests: { url: string; headers: Record; init?: RequestInit }[] = []; const fetch = vi.fn(async (input: string | URL | Request, init?: RequestInit) => { const url = String(input); const headers = Object.fromEntries(new Headers(init?.headers).entries()); - requests.push({ url, headers }); + requests.push({ url, headers, init }); if (failAll) throw new TypeError("fetch failed"); + await gates.get(new URL(url).pathname)?.promise; + if (url.startsWith(`${STUDIO}/`)) return studio.fetch(url, init); + if (!url.startsWith(ORIGIN)) return new Response("wrong host", { status: 599 }); + const p = url.slice(ORIGIN.length).split("?")[0] ?? ""; if (url === MANIFEST_URL) { if (!manifest) return new Response("missing", { status: 404 }); return Response.json(manifest); } - if (!url.startsWith(ORIGIN)) return new Response("wrong host", { status: 599 }); - const pathname = url.slice(ORIGIN.length); - if (pathname.startsWith("/drafts/")) { - const body = drafts.get(headers["if-match"] ?? ""); - return body === undefined ? new Response("gone", { status: 404 }) : Response.json(body); - } - const p = pathname.split("?")[0] ?? ""; - await gates.get(p)?.promise; if (assets.has(p)) { const body = assets.get(p); - return body instanceof Response ? body : Response.json(body); + if (body instanceof Response) return body; + return Response.json(body); } return new Response("not found", { status: 404 }); }); @@ -134,9 +143,11 @@ function deliveryApi() { asset(p: string, body: unknown) { assets.set(p, body); }, - draft(version: string, snapshot: Snapshot) { - drafts.set(version, snapshot); - }, + /** Saves a draft branch's changes on the fake Studio; returns the pointer it would mint. */ + draft: studio.draft, + /** Answers a draft branch with this response instead. */ + respond: studio.respond, + draftFetches: () => requests.filter((r) => r.url.startsWith(`${STUDIO}/`)), gate(p: string) { const d = deferred(); gates.set(p, d); @@ -161,17 +172,25 @@ const titleOf = async (client: ReturnType["forRelea // --------------------------------------------------------------------------- describe("content-delivery", () => { - it("CD-1: the SDK never calls GitHub; every fetch goes to the delivery origin", async () => { - const source = fs.readFileSync(path.join(HERE, "../remoteLoader.ts"), "utf8"); - expect(source).not.toMatch(/github/i); + it("CD-1: the SDK never calls GitHub; releases come from the delivery origin, drafts from the pointer's Studio host", async () => { + for (const file of ["../remoteLoader.ts", "../draftChanges.ts"]) { + expect(fs.readFileSync(path.join(HERE, file), "utf8")).not.toMatch(/github/i); + } const api = deliveryApi(); api.publish(1, await hashed("Published")); - api.draft("9f3c1a", await hashed("Draft")); - const loader = remote(docsSnapshot()); - await loader.update?.(); - await loader.load(POINTER); - expect(api.requests.length).toBeGreaterThan(0); - for (const { url } of api.requests) expect(url.startsWith(`${ORIGIN}/`)).toBe(true); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); + const cms = createCMS({ + blocks: docsBlocks(), + content: docsSnapshot(), + site: SITE, + token: TOKEN, + }); + await cms.update(); + expect(await titleOf(cms.forDraft(pointer))).toBe("Draft"); + const urls = api.requests.map((r) => r.url); + expect(urls.filter((url) => url.startsWith(`${ORIGIN}/`))).toHaveLength(2); // manifest, release + expect(urls.filter((url) => url.startsWith(`${STUDIO}/`))).toHaveLength(1); + expect(urls).toHaveLength(3); }); it("CD-3: one canonical hash, insertion-order independent, shared by the CLI and remoteLoader", async () => { @@ -270,22 +289,6 @@ describe("content-delivery", () => { expect(await titleOf(cms.forRelease())).toBe("A"); }); - it("CD-14: a draft is fetched at the exact version the pointer names, from the delivery origin", async () => { - const api = deliveryApi(); - api.draft("9f3c1a", await hashed("Draft 9f3c1a")); - api.draft("aaaaaa", await hashed("Newer head")); - const cms = createCMS({ - blocks: docsBlocks(), - content: docsSnapshot(), - site: SITE, - token: TOKEN, - }); - expect(await titleOf(cms.forDraft(POINTER))).toBe("Draft 9f3c1a"); - const draftRequest = api.requests.find((r) => r.url.includes("/drafts/")); - expect(draftRequest?.url).toBe(`${ORIGIN}/drafts/acme/feat-summer?token=abc`); - expect(draftRequest?.headers["if-match"]).toBe("9f3c1a"); - }); - it("CD-16: the SDK bounds the snapshot size it accepts (stops reading an oversized asset)", async () => { const api = deliveryApi(); const CHUNK = new Uint8Array(1024 * 1024).fill(0x20); // JSON whitespace @@ -309,21 +312,484 @@ describe("content-delivery", () => { // Rejected by a size bound, not after buffering and failing to parse all of it. expect(pulled).toBeLessThan(TOTAL_MB); }, 60_000); +}); + +// --------------------------------------------------------------------------- +// content-delivery.mdx#draft-previews, hosted-drafts.mdx#who-may-preview: +// a draft pointer names Studio's API, which answers only the changes +// --------------------------------------------------------------------------- + +/** A body that streams `mb` MiB of JSON whitespace after `[`, counting what was pulled. */ +function hugeBody(mb: number) { + const chunk = new Uint8Array(1024 * 1024).fill(0x20); + const state = { pulled: 0 }; + const body = new ReadableStream({ + pull(controller) { + if (state.pulled === 0) controller.enqueue(new TextEncoder().encode("[")); + if (state.pulled >= mb) return controller.close(); + state.pulled++; + controller.enqueue(chunk); + }, + }); + return { body, state }; +} - it("CD-16: drafts kept in memory are bounded (old versions are evicted)", async () => { +describe("draft previews", () => { + const cmsOf = (options: Partial[0]> = {}) => + createCMS({ blocks: docsBlocks(), content: docsSnapshot(), ...options }); + const failed = async (client: ReturnType["forDraft"]>) => { + const [value, error] = await client.resolve("SummerSEO"); + const [list, listError] = await client.list("seo"); + return { value, code: error?.code, list, listCode: listError?.code }; + }; + const FAILED = { value: null, code: "LOADER_FAILED", list: null, listCode: "LOADER_FAILED" }; + + it("DP-1: GET https://&v=: forced variants removed, no cookies or credentials, fetched once", async () => { const api = deliveryApi(); - const versions = Array.from({ length: 40 }, (_, i) => `v${i}`); - for (const v of versions) api.draft(v, await hashed(`Draft ${v}`)); - const cms = createCMS({ - blocks: docsBlocks(), - content: docsSnapshot(), - site: SITE, - token: TOKEN, + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); + const withVariant = formatDraftPointer({ + ...parseDraftPointer(pointer)!, + variants: [{ block: "Home", path: "", index: 1 }], + }); + const cms = cmsOf({ site: SITE, token: TOKEN }); + expect(await titleOf(cms.forDraft(withVariant))).toBe("Draft"); + expect(await titleOf(cms.forDraft(pointer))).toBe("Draft"); + const [request, ...more] = api.draftFetches(); + expect(more).toEqual([]); + expect(request!.url).toBe( + `${STUDIO}/api/acme/decofile/store/summer-sale/changes?token=${STUDIO_TOKEN}&v=9f3c1a`, + ); + expect(request!.headers).not.toHaveProperty("cookie"); + expect(request!.headers).not.toHaveProperty("authorization"); // never the site token + expect(request!.init?.credentials).not.toBe("include"); + expect(request!.init?.redirect).toBe("manual"); + }); + + it("DP-2: set replaces production's block whole, delete removes it, every other block is production's", async () => { + const api = deliveryApi(); + const production = docsSnapshot(); + const before = JSON.stringify(production); + const pointer = api.draft({ + set: { + SummerSEO: { __resolveType: "seo", title: "Draft" }, // no description: never merged + NewPost: { __resolveType: "post", name: "New", path: "/blog/new", date: "2026-10-01" }, + }, + delete: ["HelloWorld", "LegacySummer"], + }); + const draft = cmsOf({ content: production }).forDraft(pointer); + expect(await draft.resolve("SummerSEO", { run: false })).toEqual([ + { __resolveType: "seo", title: "Draft" }, + null, + ]); + expect((await draft.resolve("HelloWorld"))[1]?.code).toBe("NOT_FOUND"); + expect((await draft.resolve("LegacySummer"))[1]?.code).toBe("NOT_FOUND"); + const [card] = await draft.resolve("SummerCard", { run: false }); + expect(card).toEqual({ + __resolveType: "product-card", + title: "Summer collection", + product: { __resolveType: "catalog-product", slug: "summer-shirt" }, + }); + // list: production's names plus set's, minus delete. + const [posts] = await draft.list<{ name: string }>("post"); + expect(posts?.map((post) => post.name)).toEqual(["New"]); + expect(await draft.list("redirect")).toEqual([[], null]); + const [pages] = await draft.list<{ name: string }>("page"); + expect(pages?.map((page) => page.name).sort()).toEqual(["Home", "Summer campaign"]); + expect(JSON.stringify(production)).toBe(before); // never mutated + }); + + it("DP-3: a client captures production once; the next client gets the new release and fetches the draft again", async () => { + const api = deliveryApi(); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); + const cms = cmsOf({ site: SITE, token: TOKEN }); + const first = cms.forDraft(pointer); + expect(await titleOf(first)).toBe("Draft"); + api.publish( + 1, + await hashed("Published", { + HelloWorld: { __resolveType: "post", name: "Republished", path: "/x", date: "2026-10-02" }, + }), + ); + await cms.update(); + const [oldPosts] = await first.list<{ name: string }>("post"); + expect(oldPosts?.map((p) => p.name)).toEqual(["Hello, world"]); // its captured base + const next = cms.forDraft(pointer); + expect(await titleOf(next)).toBe("Draft"); + const [newPosts] = await next.list<{ name: string }>("post"); + expect(newPosts?.map((p) => p.name)).toEqual(["Republished"]); + expect(api.draftFetches()).toHaveLength(2); // dropped when the release changed + expect(await next.revision()).not.toBe(await first.revision()); + }); + + it("DP-4: the draft's revision is an opaque identity of the production revision and the version", async () => { + const api = deliveryApi(); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); + const draft = cmsOf().forDraft(pointer); + const revision = await draft.revision(); + expect(revision).toContain("rev-1"); + expect(revision).toContain("9f3c1a"); + expect(revision).not.toBe("rev-1"); + }); + + it("DP-5: only *.decocms.com and loopback hosts by default; any other host, or a look-alike, is refused before anything is fetched", async () => { + const api = deliveryApi(); + const cms = cmsOf(); + const changes = "/api/acme/decofile/store/summer-sale/changes?token=t"; + for (const pointer of [ + `evil.example${changes}@1`, + `studio.decocms.com.evil.example${changes}@1`, + `evil.example/studio.decocms.com${changes}@1`, + `evil-decocms.com${changes}@1`, + `decocms.com${changes}@1`, + `studio.decocms.com:8443${changes}@1`, + `169.254.169.254/latest/meta-data?x=1@1`, + `studio.decocms.com@evil.example${changes}@1`, + `https://studio.decocms.com${changes}@1`, + `studio.decocms.com//evil.example/x@1`, + `studio.decocms.com\\@evil.example/x@1`, + ]) { + expect({ pointer, ...(await failed(cms.forDraft(pointer))) }).toEqual({ pointer, ...FAILED }); + } + expect(api.fetch).not.toHaveBeenCalled(); + // The host is compared lowercase, so an uppercase pointer still reaches Studio. + const upper = api + .draft({ set: { SummerSEO: seoEntry("Draft") } }) + .replace(STUDIO_HOST, STUDIO_HOST.toUpperCase()); + expect(await titleOf(cms.forDraft(upper))).toBe("Draft"); + }); + + it("DP-6: DECO_PREVIEW_API_DOMAINS replaces the default list; content can't widen it", async () => { + const api = deliveryApi(); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); + vi.stubEnv("DECO_PREVIEW_API_DOMAINS", "studio.example.com"); + expect(await failed(cmsOf().forDraft(pointer))).toEqual(FAILED); + expect(api.fetch).not.toHaveBeenCalled(); + resetForTests(); + vi.stubEnv("DECO_PREVIEW_API_DOMAINS", " .decocms.com , studio.example.com "); + expect(await titleOf(cmsOf().forDraft(pointer))).toBe("Draft"); + // Content can't widen it: the CMS block has no say over where drafts come from. + resetForTests(); + vi.unstubAllEnvs(); + const release = docsSnapshot(); + release.blocks.CMS = { __resolveType: "cms-settings", preview: { hosts: ["*"] } }; + const fromContent = cmsOf({ content: release }); + expect( + await failed( + fromContent.forDraft("evil.example/api/acme/decofile/store/x/changes?token=t@1"), + ), + ).toEqual(FAILED); + expect(api.requests.filter((r) => r.url.includes("evil.example"))).toEqual([]); + }); + + it("DP-7: HTTPS, except plain HTTP for localhost, *.localhost and 127.0.0.1; a port only on those and local.studio.decocms.com; [::1] only when configured", async () => { + const urls: string[] = []; + vi.stubGlobal( + "fetch", + vi.fn(async (input: string | URL | Request) => { + urls.push(String(input)); + return Response.json({ format: 1, set: {}, delete: [] }); + }), + ); + const cms = cmsOf(); + for (const host of [ + "localhost:4000", + "studio.localhost", + "127.0.0.1:4000", + "local.studio.decocms.com:4000", + "studio.decocms.com", + "pr-12.pr.studio.decocms.com", + ]) { + expect((await cms.forDraft(`${host}/changes?token=t@v1`).revision()).endsWith("~v1")).toBe( + true, + ); + } + expect(urls).toEqual([ + "http://localhost:4000/changes?token=t&v=v1", + "http://studio.localhost/changes?token=t&v=v1", + "http://127.0.0.1:4000/changes?token=t&v=v1", + "https://local.studio.decocms.com:4000/changes?token=t&v=v1", + "https://studio.decocms.com/changes?token=t&v=v1", + "https://pr-12.pr.studio.decocms.com/changes?token=t&v=v1", + ]); + + // The defaults are v7's, which never listed the IPv6 loopback. + urls.length = 0; + expect(await failed(cms.forDraft("[::1]:4000/changes?token=t@v2"))).toEqual(FAILED); + expect(urls).toEqual([]); + // Configured, it is a loopback domain like the others: plain HTTP, any port. + vi.stubEnv("DECO_PREVIEW_API_DOMAINS", "[::1]"); + expect( + (await cmsOf().forDraft("[::1]:4000/changes?token=t@v3").revision()).endsWith("~v3"), + ).toBe(true); + expect(urls).toEqual(["http://[::1]:4000/changes?token=t&v=v3"]); + }); + + it("DP-8: only a 200 is accepted: an error status, or a redirect, is a failed draft", async () => { + for (const status of [201, 204, 301, 302, 400, 401, 403, 404, 413, 500, 502]) { + resetForTests(); + const api = deliveryApi(); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); + api.respond("summer-sale", () => + status === 204 + ? new Response(null, { status }) + : new Response(JSON.stringify({ format: 1, set: {}, delete: [] }), { + status, + headers: status >= 300 && status < 400 ? { location: "https://evil.example/" } : {}, + }), + ); + const cms = cmsOf(); + await cms.forRelease().revision(); + expect({ status, ...(await failed(cms.forDraft(pointer))) }).toEqual({ status, ...FAILED }); + expect(api.requests.filter((r) => r.url.includes("evil.example"))).toEqual([]); + } + }); + + it("DP-9: a response over 16 MiB is refused while it downloads", async () => { + const api = deliveryApi(); + const pointer = api.draft({}); + const { body, state } = hugeBody(64); + api.respond("summer-sale", () => new Response(body)); + expect(await failed(cmsOf().forDraft(pointer))).toEqual(FAILED); + expect(state.pulled).toBeLessThanOrEqual(18); + // A declared length over the limit isn't read at all. + resetForTests(); + const declared = deliveryApi(); + const big = declared.draft({}); + const huge = hugeBody(64); + declared.respond( + "summer-sale", + () => new Response(huge.body, { headers: { "content-length": String(17 * 1024 * 1024) } }), + ); + expect(await failed(cmsOf().forDraft(big))).toEqual(FAILED); + expect(huge.state.pulled).toBeLessThanOrEqual(1); + }, 60_000); + + it("DP-10: a response that doesn't arrive within 10 seconds is a failed draft", async () => { + const timeouts: number[] = []; + const controller = new AbortController(); + vi.spyOn(AbortSignal, "timeout").mockImplementation((ms: number) => { + timeouts.push(ms); + return controller.signal; + }); + let started!: () => void; + const fetching = new Promise((resolve) => { + started = resolve; + }); + vi.stubGlobal( + "fetch", + vi.fn( + (_input: unknown, init?: RequestInit) => + new Promise((_resolve, reject) => { + started(); + init?.signal?.addEventListener("abort", () => reject(init.signal?.reason)); + }), + ), + ); + const client = cmsOf().forDraft(POINTER); + const result = failed(client); + await fetching; + expect(timeouts).toEqual([10_000]); + controller.abort(new DOMException("timed out", "TimeoutError")); + expect(await result).toEqual(FAILED); + }); + + it("DP-11: the shape is checked: exactly format 1, set and delete, disjoint", async () => { + const bodies: unknown[] = [ + { format: 2, set: {}, delete: [] }, + { set: {}, delete: [] }, + { format: 1, set: {}, delete: [], baseRevision: "x" }, + { format: 1, set: [], delete: [] }, + { format: 1, set: null, delete: [] }, + { format: 1, set: {} }, + { format: 1, set: {}, delete: "HelloWorld" }, + { format: 1, set: {}, delete: [1] }, + { format: 1, set: {}, delete: ["A", "A"] }, + { format: 1, set: { A: seoEntry("x") }, delete: ["A"] }, + [{ format: 1, set: {}, delete: [] }], + "changes", + null, + ]; + for (const body of [...bodies, "not json"]) { + resetForTests(); + const api = deliveryApi(); + const pointer = api.draft({}); + api.respond("summer-sale", () => + body === "not json" ? new Response("{oops") : Response.json(body), + ); + expect({ body, ...(await failed(cmsOf().forDraft(pointer))) }).toEqual({ body, ...FAILED }); + } + }); + + it("DP-12: a failed draft is LOADER_FAILED on every call, never published content, and isn't reused", async () => { + const api = deliveryApi(); + api.publish(1, await hashed("Release")); + const cms = cmsOf({ site: SITE, token: TOKEN }); + await cms.update(); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); + api.respond("summer-sale", () => new Response("bad gateway", { status: 502 })); + const client = cms.forDraft(pointer); + expect(await failed(client)).toEqual(FAILED); + await expect(client.revision()).rejects.toMatchObject({ code: "LOADER_FAILED" }); + const [, error] = await client.resolve("SummerSEO"); + expect(String((error?.cause as Error | undefined)?.message)).toContain("HTTP 502"); + api.respond("summer-sale", () => + Response.json({ format: 1, set: { SummerSEO: seoEntry("Back") }, delete: [] }), + ); + expect(await titleOf(cms.forDraft(pointer))).toBe("Back"); + expect(api.draftFetches()).toHaveLength(2); + }); + + it("DP-13: an expired or wrong token gets no content", async () => { + const api = deliveryApi(); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }, { token: "expired" }); + expect(await failed(cmsOf().forDraft(pointer))).toEqual(FAILED); + expect(api.draftFetches()).toHaveLength(1); + }); + + it("DP-14: a fetched draft is reused for up to a minute per pointer, at most three per CMS", async () => { + let now = 1_000_000; + vi.spyOn(Date, "now").mockImplementation(() => now); + const api = deliveryApi(); + const pointer = (branch: string) => + api.draft({ set: { SummerSEO: seoEntry(branch) } }, { branch }); + const cms = cmsOf(); + expect(await titleOf(cms.forDraft(pointer("a")))).toBe("a"); + now += 59_000; + expect(await titleOf(cms.forDraft(pointer("a")))).toBe("a"); + expect(api.draftFetches()).toHaveLength(1); + now += 2_000; + await titleOf(cms.forDraft(pointer("a"))); + expect(api.draftFetches()).toHaveLength(2); // a minute passed: fetched (and its token checked) again + for (const branch of ["b", "c", "d"]) await titleOf(cms.forDraft(pointer(branch))); + expect(api.draftFetches()).toHaveLength(5); + await titleOf(cms.forDraft(pointer("a"))); // the fourth evicted the oldest + expect(api.draftFetches()).toHaveLength(6); + await titleOf(cms.forDraft(pointer("d"))); + expect(api.draftFetches()).toHaveLength(6); + }); + + it("DP-15: drafts need no site or token, and never send them", async () => { + const api = deliveryApi(); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); + expect(await titleOf(cmsOf().forDraft(pointer))).toBe("Draft"); + expect(api.requests.map((r) => new URL(r.url).host)).toEqual([STUDIO_HOST]); + resetForTests(); + const hosted = deliveryApi(); + const hostedPointer = hosted.draft({ set: { SummerSEO: seoEntry("Draft") } }); + expect(await titleOf(cmsOf({ site: SITE, token: TOKEN }).forDraft(hostedPointer))).toBe( + "Draft", + ); + for (const r of hosted.draftFetches()) expect(r.headers.authorization).toBeUndefined(); + }); + + it("DP-16: a pointer whose version is local names no draft: nothing is fetched, forced variants apply", async () => { + const api = deliveryApi(); + const content = docsSnapshot(); + content.blocks.Banner = { + __resolveType: "multivariate", + variants: [ + { rule: { __resolveType: "always" }, value: "spring" }, + { rule: { __resolveType: "never" }, value: "summer" }, + ], + }; + const cms = cmsOf({ content }); + const local = formatDraftPointer({ + host: "localhost:4547", + path: "/", + version: "local", + variants: [{ block: "Banner", path: "", index: 1 }], }); - for (const v of versions) await titleOf(cms.forDraft(`${HOST}/drafts/acme/x@${v}`)); - const before = api.requests.length; - await titleOf(cms.forDraft(`${HOST}/drafts/acme/x@v0`)); - expect(api.requests.length).toBe(before + 1); // v0 was evicted and fetched again + expect(await cms.forDraft(local).resolve("Banner")).toEqual(["summer", null]); + expect(await titleOf(cms.forDraft("localhost:4547/@local"))).toBe("Sunny!"); + expect(await cms.forDraft("localhost:4547/@local").revision()).toBe("rev-1"); + expect(api.fetch).not.toHaveBeenCalled(); + }); + + it("DP-17: forced variants apply on top of the draft's changes; every variant shares one fetch", async () => { + const api = deliveryApi(); + const banner = { + __resolveType: "multivariate", + variants: [ + { rule: { __resolveType: "never" }, value: "draft fallback" }, + { rule: { __resolveType: "never" }, value: "draft summer" }, + ], + }; + const pointer = api.draft({ set: { Banner: banner } }); + const parsed = parseDraftPointer(pointer)!; + const cms = cmsOf(); + const forced = (index: number) => + formatDraftPointer({ ...parsed, variants: [{ block: "Banner", path: "", index }] }); + expect(await cms.forDraft(forced(1)).resolve("Banner")).toEqual(["draft summer", null]); + expect(await cms.forDraft(forced(0)).resolve("Banner")).toEqual(["draft fallback", null]); + expect(await cms.forDraft(pointer).resolve("Banner")).toEqual([undefined, null]); // the rules + expect(api.draftFetches()).toHaveLength(1); + }); + + it("DP-18: empty changes (a branch before its first save) show production", async () => { + const api = deliveryApi(); + const pointer = api.draft({}, { branch: "not-saved-yet" }); + const cms = cmsOf(); + expect(await titleOf(cms.forDraft(pointer))).toBe("Sunny!"); + const [pages] = await cms.forDraft(pointer).list("page"); + expect(pages).toHaveLength(2); + }); + + it("DP-19: release checks don't depend on drafts: a draft client schedules the same check, never in front of it", async () => { + const api = deliveryApi(); + api.publish( + 1, + await hashed("Published", { + HelloWorld: { __resolveType: "post", name: "Republished", path: "/x", date: "2026-10-02" }, + }), + ); + const channel = api.gate("/sites/acme/channels/production.json"); // the background check hangs + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); + const cms = cmsOf({ site: SITE, token: TOKEN }); + const first = cms.forDraft(pointer); + expect((await first.list<{ name: string }>("post"))[0]?.map((p) => p.name)).toEqual([ + "Hello, world", + ]); // cold: the fallback is the base + expect(api.assetFetches()).toBe(0); + expect(api.manifestFetches()).toBeLessThanOrEqual(1); // scheduled beside the draft, still pending + channel.resolve(); + await flush(); + const next = cms.forDraft(pointer); + expect(await titleOf(next)).toBe("Draft"); + expect((await next.list<{ name: string }>("post"))[0]?.map((p) => p.name)).toEqual([ + "Republished", + ]); + }); + + it("DP-20: an entry named __proto__ in set stays an entry", async () => { + const api = deliveryApi(); + const pointer = api.draft({ + set: JSON.parse( + '{"__proto__": {"__resolveType": "seo", "title": "Odd", "description": "d"}}', + ), + }); + const [entry] = await cmsOf().forDraft(pointer).resolve("__proto__", { run: false }); + expect(entry).toEqual({ __resolveType: "seo", title: "Odd", description: "d" }); + expect(({} as Record).title).toBeUndefined(); + }); + + it("DP-21: a response that was redirected anyway (a fetch polyfill ignoring redirect: manual) is refused", async () => { + const changes = { format: 1, set: { SummerSEO: seoEntry("Elsewhere") }, delete: [] }; + const redirected = (url: string, flag: boolean) => { + const response = Response.json(changes); + Object.defineProperty(response, "redirected", { value: flag }); + Object.defineProperty(response, "url", { value: url }); + return response; + }; + for (const [url, flag] of [ + ["https://evil.example/changes", false], + [`https://${STUDIO_HOST}/elsewhere`, true], + ] as const) { + resetForTests(); + const api = deliveryApi(); + const pointer = api.draft({}); + api.respond("summer-sale", () => redirected(url, flag)); + expect({ url, ...(await failed(cmsOf().forDraft(pointer))) }).toEqual({ url, ...FAILED }); + } }); }); @@ -383,16 +849,16 @@ describe("releases-and-deployment", () => { it("RD-7: a revision unlocks nothing: forRevision() never reaches the draft", async () => { const api = deliveryApi(); - const draft = await hashed("Secret draft"); - api.draft("9f3c1a", draft); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Secret draft") } }); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot(), site: SITE, token: TOKEN, }); - expect(await titleOf(cms.forDraft(POINTER))).toBe("Secret draft"); - const byRevision = cms.forRevision(draft.revision); + const draft = cms.forDraft(pointer); + expect(await titleOf(draft)).toBe("Secret draft"); + const byRevision = cms.forRevision(await draft.revision()); expect(await titleOf(byRevision)).toBe("Sunny!"); }); }); @@ -448,7 +914,7 @@ describe("hosted", () => { vi.stubEnv("NODE_ENV", "development"); const api = deliveryApi(); api.publish(1, await hashed("Published")); - api.draft("9f3c1a", await hashed("Draft")); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot(), @@ -460,7 +926,7 @@ describe("hosted", () => { await cms.update(); expect(api.manifestFetches()).toBe(0); expect(await titleOf(cms.forRelease())).toBe("Sunny!"); - expect(await titleOf(cms.forDraft(POINTER))).toBe("Draft"); + expect(await titleOf(cms.forDraft(pointer))).toBe("Draft"); }); it("H-6: site and token alone never send telemetry", async () => { @@ -486,10 +952,8 @@ describe("hosted", () => { expect(destination?.endpoint).toMatch(/^https:\/\/[^/]*decocms\.com/); expect(destination?.headers.authorization).toBe(`Bearer ${TOKEN}`); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); - const [analytics] = await cms - .forRelease() - .resolve<{ collector: string }>({ __resolveType: "analytics" }); - expect(analytics?.collector).toBe(HOSTED_ANALYTICS_COLLECTOR); + const { analytics } = await cms.settings(); + expect(analytics.collector).toBe(HOSTED_ANALYTICS_COLLECTOR); }); }); @@ -563,10 +1027,21 @@ describe("hosted-publishing", () => { }); describe("check schedule", () => { + // Background work goes through the host hook so a tick awaits exactly the + // checks it scheduled: a timer-based flush can return before a loaded + // machine runs them, and the counts below would read one short. + const HOOK = Symbol.for("decocms.blocks.background"); + const g = globalThis as { [HOOK]?: (task: () => Promise) => void }; + let background: Promise[] = []; let now = 1_000_000; beforeEach(() => { now = 1_000_000; vi.spyOn(Date, "now").mockImplementation(() => now); + background = []; + g[HOOK] = (task) => void background.push(task()); + }); + afterEach(() => { + delete g[HOOK]; }); /** Creates a CMS over an updatable loader and returns a probe of when checks run. */ @@ -579,7 +1054,7 @@ describe("hosted-publishing", () => { async tick(ms: number) { now += ms; cms.forRelease(); - await flush(); + await Promise.all(background.splice(0)); return update.mock.calls.length; }, }; @@ -785,7 +1260,7 @@ describe("hosted-publishing", () => { // --------------------------------------------------------------------------- describe("hosted-drafts", () => { - it("HD-1: pointer is @, version after the last @; format round-trips", () => { + it("HD-1: pointer is @, version after the last @; format round-trips", async () => { const raw = "api.deco.example/drafts/acme/feat-summer?token=abc@9f3c1a"; const parsed = parseDraftPointer(raw); expect(parsed).toEqual({ @@ -795,40 +1270,48 @@ describe("hosted-drafts", () => { }); expect(formatDraftPointer(parsed!)).toBe(raw); const link = new URL(`https://store.example.com/summer?__draft=${encodeURIComponent(raw)}`); - expect(draftPointer(new Request(link))).toBe(raw); + expect(await draftPointer(new Request(link))).toBe(raw); }); - it("HD-2: forDraft fetches that version once and serves it from memory afterwards", async () => { + it("HD-2: forDraft fetches the pointer once and serves it from memory afterwards", async () => { const api = deliveryApi(); - api.draft("9f3c1a", await hashed("Draft")); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot(), site: SITE, token: TOKEN, }); - expect(await titleOf(cms.forDraft(POINTER))).toBe("Draft"); - expect(await titleOf(cms.forDraft(POINTER))).toBe("Draft"); - expect(api.requests.filter((r) => r.url.includes("/drafts/"))).toHaveLength(1); + expect(await titleOf(cms.forDraft(pointer))).toBe("Draft"); + expect(await titleOf(cms.forDraft(pointer))).toBe("Draft"); + expect(api.draftFetches()).toHaveLength(1); }); - it("HD-3: a draft replaces the whole revision, deletions included", async () => { + it("HD-3: a draft holds only changed blocks and deletions; every other block is inherited", async () => { const api = deliveryApi(); - api.draft("9f3c1a", await hashed("Draft")); // only SummerSEO + const pointer = api.draft({ + set: { SummerSEO: seoEntry("Draft") }, + delete: ["HelloWorld"], + }); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot(), site: SITE, token: TOKEN, }); - const [pages] = await cms.forDraft(POINTER).list("page"); - expect(pages).toEqual([]); + const draft = cms.forDraft(pointer); + expect(await titleOf(draft)).toBe("Draft"); + const [pages] = await draft.list("page"); const [released] = await cms.forRelease().list("page"); - expect(released?.length).toBeGreaterThan(0); + const names = (list: unknown) => (list as { name: string }[]).map((page) => page.name); + expect(names(pages)).toEqual(names(released)); // inherited from production + expect(names(pages).length).toBeGreaterThan(0); + expect((await draft.resolve("HelloWorld"))[1]?.code).toBe("NOT_FOUND"); + expect(await draft.list("post")).toEqual([[], null]); }); it("HD-4: a draft that can't be fetched is an error, never published content", async () => { - deliveryApi(); // no such draft: 404 + deliveryApi(); // the token in POINTER isn't the one the fake Studio signed: 401 const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot(), @@ -844,31 +1327,31 @@ describe("hosted-drafts", () => { expect(listError?.code).toBe("LOADER_FAILED"); }); - it("HD-5: draftPointer reads ?__draft= or the cookie; draftCookie only answers ?__draft=", () => { + it("HD-5: draftPointer reads ?__draft= or the cookie; draftCookie only answers ?__draft=", async () => { const fromUrl = new Request(`https://s.example/?__draft=${encodeURIComponent(POINTER)}`); - expect(draftPointer(fromUrl)).toBe(POINTER); - expect(draftCookie(fromUrl)).toMatch(new RegExp(`^${DRAFT_COOKIE}=`)); + expect(await draftPointer(fromUrl)).toBe(POINTER); + expect(await draftCookie(fromUrl)).toMatch(new RegExp(`^${DRAFT_COOKIE}=`)); const fromCookie = new Request("https://s.example/", { headers: { cookie: `${DRAFT_COOKIE}=${encodeURIComponent(POINTER)}` }, }); - expect(draftPointer(fromCookie)).toBe(POINTER); - expect(draftCookie(fromCookie)).toBeNull(); + expect(await draftPointer(fromCookie)).toBe(POINTER); + expect(await draftCookie(fromCookie)).toBeNull(); const plain = new Request("https://s.example/"); - expect(draftPointer(plain)).toBeNull(); - expect(draftCookie(plain)).toBeNull(); + expect(await draftPointer(plain)).toBeNull(); + expect(await draftCookie(plain)).toBeNull(); }); - it("HD-6: ?__draft=off expires the cookie and reads no pointer", () => { + it("HD-6: ?__draft=off expires the cookie and reads no pointer", async () => { const off = new Request("https://s.example/?__draft=off", { headers: { cookie: `${DRAFT_COOKIE}=${encodeURIComponent(POINTER)}` }, }); - expect(draftPointer(off)).toBeNull(); - expect(draftCookie(off)).toMatch(/Max-Age=0/); + expect(await draftPointer(off)).toBeNull(); + expect(await draftCookie(off)).toMatch(/Max-Age=0/); }); it("HD-7: the plain request handler example runs as written", async () => { const api = deliveryApi(); - api.draft("9f3c1a", await hashed("Draft")); + const POINTER = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot(), @@ -878,10 +1361,10 @@ describe("hosted-drafts", () => { const render = async (client: ReturnType, _request: Request) => new Response(await titleOf(client)); async function handle(request: Request) { - const pointer = draftPointer(request); + const pointer = await cms.draftPointer(request); const client = pointer ? cms.forDraft(pointer) : cms.forRelease(); const response = await render(client, request); - const cookie = draftCookie(request); + const cookie = await cms.draftCookie(request); if (cookie) response.headers.append("Set-Cookie", cookie); return response; } @@ -899,22 +1382,22 @@ describe("hosted-drafts", () => { expect(await visitor.text()).toBe("Sunny!"); }); - it("HD-8: without site and token, forDraft reads the content module (no fetch, no error)", async () => { + it("HD-8: without site and token, forDraft layers the draft over the content module; only Studio is asked", async () => { const api = deliveryApi(); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); - expect(await titleOf(cms.forDraft(POINTER))).toBe("Sunny!"); - expect(api.fetch).not.toHaveBeenCalled(); + expect(await titleOf(cms.forDraft(pointer))).toBe("Draft"); + expect(api.requests.map((r) => new URL(r.url).host)).toEqual([STUDIO_HOST]); }); - it("HD-10: DRAFT_COOKIE is the cookie draftCookie writes and draftPointer reads", () => { - expect(DRAFT_COOKIE).toBe("deco-draft"); - const set = draftCookie( + it("HD-10: __deco_draft is the cookie draftCookie writes and draftPointer reads", async () => { + const set = await draftCookie( new Request(`https://s.example/?__draft=${encodeURIComponent(POINTER)}`), ); expect(set?.startsWith(`${DRAFT_COOKIE}=`)).toBe(true); }); - it("HD-12: examples never spell __draft or deco-draft", () => { + it("HD-12: examples never spell __draft or __deco_draft", () => { const examples = path.resolve(HERE, "../../../../../examples"); const hits: string[] = []; const walk = (dir: string) => { @@ -924,7 +1407,7 @@ describe("hosted-drafts", () => { if (entry.isDirectory()) walk(full); else if (/\.(ts|tsx)$/.test(entry.name)) { const text = fs.readFileSync(full, "utf8"); - if (/__draft|deco-draft/.test(text)) hits.push(full); + if (/__draft|__deco_draft/.test(text)) hits.push(full); } } }; @@ -934,7 +1417,7 @@ describe("hosted-drafts", () => { it("HD-13: forDraft takes only the pointer string (the React Native example)", async () => { const api = deliveryApi(); - api.draft("9f3c1a", await hashed("Mobile draft")); + const POINTER = api.draft({ set: { SummerSEO: seoEntry("Mobile draft") } }); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot(), @@ -975,7 +1458,7 @@ describe("hosted-drafts", () => { site: SITE, token: TOKEN, }); - for (const pointer of ["garbage", "https://x/y@1", "@1", `${HOST}/x@`]) { + for (const pointer of ["garbage", "https://x/y@1", "@1", `${STUDIO_HOST}/x@`]) { const [value, error] = await cms.forDraft(pointer).list("seo"); expect(value).toBeNull(); expect(error?.code).toBe("LOADER_FAILED"); @@ -983,25 +1466,89 @@ describe("hosted-drafts", () => { expect(api.fetch).not.toHaveBeenCalled(); }); - it("HD-18: the preview-domain example only reads the pointer on the preview host", async () => { + it("HD-18: preview hosts from the release's CMS block: a draft previews on staging only, every other host serves the release", async () => { const api = deliveryApi(); - api.draft("9f3c1a", await hashed("Preview")); - const cms = createCMS({ - blocks: docsBlocks(), - content: docsSnapshot(), - site: SITE, - token: TOKEN, + // The draft tries to allow its own host: settings come from the release, so it can't. + const POINTER = api.draft({ + set: { + SummerSEO: seoEntry("Preview"), + CMS: { __resolveType: "cms-settings", preview: { hosts: ["*"] } }, + }, }); - const pick = (request: Request) => { - const host = new URL(request.url).host; - const pointer = host === "preview.store.example.com" ? draftPointer(request) : null; - return pointer ? cms.forDraft(pointer) : cms.forRelease(); + const release = docsSnapshot(); + release.blocks.CMS = { + __resolveType: "cms-settings", + preview: { hosts: ["staging.store.example.com"] }, + }; + const cms = createCMS({ blocks: docsBlocks(), content: release, site: SITE, token: TOKEN }); + const pick = async (request: Request) => { + const pointer = await cms.draftPointer(request); + const cookie = await cms.draftCookie(request); + return { client: pointer ? cms.forDraft(pointer) : cms.forRelease(), cookie }; }; const q = `?__draft=${encodeURIComponent(POINTER)}`; - expect(await titleOf(pick(new Request(`https://preview.store.example.com/${q}`)))).toBe( - "Preview", + const staging = await pick(new Request(`https://staging.store.example.com/${q}`)); + expect(await titleOf(staging.client)).toBe("Preview"); + expect(staging.cookie).not.toBeNull(); + // A preview of that draft doesn't change the settings either. + expect((await cms.settings()).preview.hosts).toEqual(["staging.store.example.com"]); + const live = await pick(new Request(`https://store.example.com/${q}`)); + expect(await titleOf(live.client)).toBe("Sunny!"); + expect(live.cookie).toBeNull(); + // A cookie set on staging is ignored on the public host too, without an error. + const withCookie = await pick( + new Request("https://store.example.com/", { + headers: { cookie: `${DRAFT_COOKIE}=${encodeURIComponent(POINTER)}` }, + }), ); - expect(await titleOf(pick(new Request(`https://store.example.com/${q}`)))).toBe("Sunny!"); + expect(await titleOf(withCookie.client)).toBe("Sunny!"); + }); +}); + +// --------------------------------------------------------------------------- +// api-reference.mdx › cms.settings() with hosted releases +// --------------------------------------------------------------------------- + +describe("CMS settings from hosted releases", () => { + const bundled = (): Snapshot => { + const snapshot = docsSnapshot(); + snapshot.blocks.CMS = { + __resolveType: "cms-settings", + preview: { hosts: ["staging.example.com"] }, + analytics: { collector: "https://bundled.example/events" }, + }; + return snapshot; + }; + + it("S-1: offline, at boot, settings come from the bundled content module, with no fetch", async () => { + const api = deliveryApi(); + api.fail(true); + const cms = createCMS({ blocks: docsBlocks(), content: bundled(), site: SITE, token: TOKEN }); + const settings = await cms.settings(); + expect(settings.preview.hosts).toEqual(["staging.example.com"]); + expect(settings.analytics.collector).toBe("https://bundled.example/events"); + expect(api.fetch).not.toHaveBeenCalled(); + // Drafts are gated by them even while the Deco API can't be reached. + const link = `?__draft=${encodeURIComponent("delivery.decocms.com/x@v")}`; + expect(await cms.draftPointer(new Request(`https://www.example.com/${link}`))).toBeNull(); + expect(api.fetch).not.toHaveBeenCalled(); + // A request serves the bundled release; the failed background check changes nothing. + expect(await titleOf(cms.forRelease())).toBe("Sunny!"); + await flush(50); + expect((await cms.settings()).preview.hosts).toEqual(["staging.example.com"]); + }); + + it("S-2: a newer release's settings arrive with the background check, never before", async () => { + const api = deliveryApi(); + const next = await hashed("Published", { + CMS: { __resolveType: "cms-settings", preview: { hosts: ["www.example.com"] } }, + }); + api.publish(1, next); + const cms = createCMS({ blocks: docsBlocks(), content: bundled(), site: SITE, token: TOKEN }); + expect((await cms.settings()).preview.hosts).toEqual(["staging.example.com"]); + expect(api.fetch).not.toHaveBeenCalled(); + expect(await cms.update()).toEqual({ updated: true }); + expect((await cms.settings()).preview.hosts).toEqual(["www.example.com"]); }); }); @@ -1168,7 +1715,9 @@ describe("hosted-releases-internals", () => { token: TOKEN, }); await cms.update(); - const [value, error] = await cms.forDraft(POINTER).resolve("SummerSEO"); + const pointer = api.draft({ set: { SummerSEO: seoEntry("Draft") } }); + api.respond("summer-sale", () => new Response("no such project", { status: 404 })); + const [value, error] = await cms.forDraft(pointer).resolve("SummerSEO"); expect(value).toBeNull(); expect(error?.code).toBe("LOADER_FAILED"); }); @@ -1182,7 +1731,7 @@ describe("hosted-releases-internals", () => { }; const a = createCMS(config); const b = createCMS({ ...config, content: { ...docsSnapshot(), root: ".deco" } }); - expect(b).toBe(a); + expect(instanceOf(b)).toBe(instanceOf(a)); const keys = Object.getOwnPropertySymbols(globalThis) .map((s) => Symbol.keyFor(s)) .filter((k): k is string => typeof k === "string" && k.startsWith("decocms.blocks")); @@ -1228,7 +1777,7 @@ describe("hosted-releases-internals", () => { const content = { ...docsSnapshot(), root: ".deco" }; const a = createCMS({ blocks: docsBlocks(), content, interval: 60_000 }); const b = createCMS({ blocks: docsBlocks(), content, interval: 120_000 }); - expect(b).toBe(a); + expect(instanceOf(b)).toBe(instanceOf(a)); expect(warn).toHaveBeenCalledWith(expect.stringContaining("interval")); }); diff --git a/packages/blocks/src/v8/__conformance__/extra.test.ts b/packages/blocks/src/v8/__conformance__/extra.test.ts index 6bb65d92e..4872a891c 100644 --- a/packages/blocks/src/v8/__conformance__/extra.test.ts +++ b/packages/blocks/src/v8/__conformance__/extra.test.ts @@ -1,7 +1,7 @@ // @vitest-environment node /** - * Conformance, second pass: claims from the docs (blocks-site - * docs/content/next) that the other conformance files test weakly or not at + * Conformance, second pass: claims from the docs (deco-sites/docs-tanstack + * src/content/docs/en/storefront/blocks/next) that the other conformance files test weakly or not at * all. Each `it` quotes the page and the claim it checks. */ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; @@ -10,15 +10,9 @@ import { readResponseJson } from "../../protocol/client"; import { blockFileName, blockNameFromFile } from "../../protocol/keys"; import { createContentHandler } from "../../protocol/server/handler"; import { createMemoryStorage } from "../../protocol/storage/memory"; +import { instanceOf } from "../cms"; import { createInstrumentedFetch } from "../fetch"; -import { - createCMS, - draftCookie, - draftPointer, - matchRoute, - parseDraftPointer, - resetForTests, -} from "../index"; +import { createCMS, matchRoute, parseDraftPointer, resetForTests } from "../index"; import { resolveDestination, setCurrentTelemetry } from "../telemetry"; import { docsBlocks, docsSnapshot } from "../testFixtures"; import type { Blocks, Lazy, Loader, Snapshot } from "../types"; @@ -380,9 +374,11 @@ describe("built-ins (built-in-blocks.mdx, lazy-blocks.mdx, matchers-and-variants .forRelease() .resolve({ __resolveType: "page", name: "n", path: "/", sections: [] }); expect(page).toMatchObject({ wrapped: true }); - const [analytics] = await cms.forRelease().resolve({ __resolveType: "analytics" }); - expect(analytics).toMatchObject({ enabled: true }); - expect(typeof (analytics as { collector: string }).collector).toBe("string"); + const [settings] = await cms.forRelease().resolve<{ analytics: { collector: string } }>({ + __resolveType: "cms-settings", + }); + expect(settings?.analytics).toMatchObject({ enabled: true }); + expect(typeof settings?.analytics.collector).toBe("string"); }); }); @@ -414,18 +410,22 @@ describe("clients and content (api-reference.mdx, content.mdx)", () => { it("X21 api-reference: forRevision pins to a served revision; an unknown revision (or a draft's) behaves like the release", async () => { let current = docsSnapshot("rev-1"); const loader: Loader = { - load: async (pointer) => (pointer ? docsSnapshot("draft-rev") : current), + load: async () => current, update: async () => ({ updated: true }), }; + vi.stubGlobal("fetch", async () => Response.json({ format: 1, set: {}, delete: [] })); const cms = createCMS({ blocks: docsBlocks(), content: loader }); expect(await cms.forRelease().revision()).toBe("rev-1"); - expect(await cms.forDraft("api.deco.example/drafts/x@v1").revision()).toBe("draft-rev"); + const draftRevision = await cms + .forDraft("studio.decocms.com/api/acme/decofile/store/x/changes?token=t@v1") + .revision(); + expect(draftRevision).toBe("rev-1~v1"); current = docsSnapshot("rev-2"); await cms.update(); expect(await cms.forRelease().revision()).toBe("rev-2"); expect(await cms.forRevision("rev-1").revision()).toBe("rev-1"); expect(await cms.forRevision("nope").revision()).toBe("rev-2"); - expect(await cms.forRevision("draft-rev").revision()).toBe("rev-2"); + expect(await cms.forRevision(draftRevision).revision()).toBe("rev-2"); }); it("X22 api-reference › Loaders: a loader without update() is asked on every client; update() never throws", async () => { @@ -454,17 +454,22 @@ describe("clients and content (api-reference.mdx, content.mdx)", () => { const content = snap({}); const a = createCMS({ blocks: {}, content, interval: 1000 }); const b = createCMS({ blocks: {}, content, interval: 120_000 }); - expect(b).toBe(a); + expect(instanceOf(b)).toBe(instanceOf(a)); expect(warn.mock.calls.flat().join("\n")).toMatch(/interval/); }); - it("X24 api-reference › Draft pointers: ?__draft=off wins over the cookie; parse rejects schemes, stray @, unrooted paths", () => { + it("X24 api-reference › Draft pointers: ?__draft=off wins over the cookie; parse rejects schemes, stray @, unrooted paths", async () => { const req = (url: string, cookie?: string) => new Request(url, { headers: cookie ? { cookie } : {} }); - expect(draftPointer(req("https://s.example/?__draft=off", "deco-draft=h/p@v"))).toBeNull(); - expect(draftPointer(req("https://s.example/", "deco-draft=h%2Fp%40v"))).toBe("h/p@v"); - expect(draftCookie(req("https://s.example/?__draft=off"))).toMatch(/Max-Age=0/); - expect(draftCookie(req("https://s.example/"))).toBeNull(); + const cms = createCMS({ blocks: {}, content: { revision: "x24", blocks: {} } }); + expect( + await cms.draftPointer(req("https://s.example/?__draft=off", "__deco_draft=h/p@v")), + ).toBeNull(); + expect(await cms.draftPointer(req("https://s.example/", "__deco_draft=h%2Fp%40v"))).toBe( + "h/p@v", + ); + expect(await cms.draftCookie(req("https://s.example/?__draft=off"))).toMatch(/Max-Age=0/); + expect(await cms.draftCookie(req("https://s.example/"))).toBeNull(); expect(parseDraftPointer("https://h.example/p@v")).toBeNull(); expect(parseDraftPointer("h.example/p@x@v")).toBeNull(); expect(parseDraftPointer("h.example@v")).toBeNull(); @@ -599,20 +604,23 @@ describe("the content protocol (content-protocol.mdx)", () => { }); describe("telemetry settings are content (telemetry.mdx)", () => { - it("X31 the Telemetry block's rates are capped by telemetry.limits (trace default cap 0); enabled: false switches it off", async () => { - const { TelemetryPipeline } = await import("../telemetry"); - const block = (props: Record) => - snap({ Telemetry: { __resolveType: "telemetry", ...props } }, JSON.stringify(props)); - vi.spyOn(Math, "random").mockReturnValue(0.5); - const capped = new TelemetryPipeline(resolveDestination({ endpoint: "https://otel.example" })!); - capped.useRelease(block({ traceSampleRate: 1 })); - expect(capped.sampleTrace()).toBe(false); - const open = new TelemetryPipeline( - resolveDestination({ endpoint: "https://otel.example", limits: { traceSampleRate: 1 } })!, - ); - open.useRelease(block({ traceSampleRate: 1 })); - expect(open.sampleTrace()).toBe(true); - open.useRelease(block({ traceSampleRate: 1, enabled: false })); - expect(open.sampleTrace()).toBe(false); + it("X31 the telemetry section's rates are capped by telemetry.limits (trace default cap 0); enabled: false switches it off", async () => { + const settings = (telemetry: Record, limits?: { traceSampleRate: number }) => + createCMS({ + blocks: {}, + content: snap( + { CMS: { __resolveType: "cms-settings", telemetry } }, + JSON.stringify(telemetry), + ), + telemetry: { endpoint: "https://otel.example", ...(limits ? { limits } : {}) }, + }).settings(); + expect((await settings({ traceSampleRate: 1 })).telemetry.traceSampleRate).toBe(0); + expect( + (await settings({ traceSampleRate: 1 }, { traceSampleRate: 1 })).telemetry.traceSampleRate, + ).toBe(1); + expect( + (await settings({ traceSampleRate: 1, enabled: false }, { traceSampleRate: 1 })).telemetry + .enabled, + ).toBe(false); }); }); diff --git a/packages/blocks/src/v8/__conformance__/guides.cli.test.ts b/packages/blocks/src/v8/__conformance__/guides.cli.test.ts index 7a0bd99e6..537580912 100644 --- a/packages/blocks/src/v8/__conformance__/guides.cli.test.ts +++ b/packages/blocks/src/v8/__conformance__/guides.cli.test.ts @@ -257,17 +257,19 @@ const _b: Blocks = blocks; ); it( - "qs-06/tr-16: deco serve prints a site editor link with a fresh token; saves write .deco/blocks", + "qs-06/tr-16: deco serve prints a site editor link with no token; saves write .deco/blocks", async () => { const p = quickstart(); fs.mkdirSync(path.join(p.root, ".git")); expect(p.deco("schema").code).toBe(0); const first = await serve(p, ["--port", "0"]); - const second = await serve(p, ["--port", "0"]); - expect(first.token).not.toBe(second.token); + expect(first.out).toContain( + `https://studio.decocms.com/site-editor#endpoint=${encodeURIComponent(first.endpoint)}\n`, + ); + expect(first.out).not.toMatch(/token/i); const reply = await fetch(first.endpoint, { method: "POST", - headers: { authorization: `Bearer ${first.token}`, "content-type": "application/json" }, + headers: { "content-type": "application/json" }, body: JSON.stringify({ jsonrpc: "2.0", id: 1, @@ -339,28 +341,23 @@ const _b: Blocks = blocks; async function serve( p: Project, args: string[], -): Promise<{ endpoint: string; token: string; out: string; child: ChildProcess }> { +): Promise<{ endpoint: string; out: string; child: ChildProcess }> { const child = spawn(process.execPath, [BIN, "serve", ...args], { cwd: p.root, - env: (() => { - const env: NodeJS.ProcessEnv = { ...process.env, NO_COLOR: "1" }; - delete env.DECO_SERVE_TOKEN; - return env; - })(), + env: { ...process.env, NO_COLOR: "1" }, }); children.push(child); let out = ""; child.stdout?.on("data", (d) => (out += d)); child.stderr?.on("data", (d) => (out += d)); const started = Date.now(); - while (!/token=/.test(out)) { + while (!/site-editor#endpoint=\S+\n/.test(out)) { if (child.exitCode !== null) throw new Error(`serve exited: ${out}`); if (Date.now() - started > 30_000) throw new Error(`serve never printed a link: ${out}`); await new Promise((r) => setTimeout(r, 50)); } const endpoint = /(http:\/\/[^\s]+\/rpc)/.exec(out)![1]; - const token = decodeURIComponent(/token=([^\s&]+)/.exec(out)![1]); - return { endpoint, token, out, child }; + return { endpoint, out, child }; } describe("serve flags", () => { @@ -379,12 +376,12 @@ describe("serve flags", () => { ); it( - "tr-16: --allow-origin admits another origin, and local-network preflights are answered", + "tr-16: any origin is answered, and local-network preflights too", async () => { const p = project(); fs.mkdirSync(path.join(p.root, ".git")); p.write(".deco/schema.gen.json", {}); - const s = await serve(p, ["--port", "0", "--allow-origin", "https://editor.example.com"]); + const s = await serve(p, ["--port", "0"]); const reply = await fetch(s.endpoint, { method: "OPTIONS", headers: { @@ -881,7 +878,7 @@ export default { real: (input: { zzRealField: string }) => input } satisfies Blo const s = await serve(p, ["--port", "0"]); const reply = await fetch(s.endpoint, { method: "POST", - headers: { authorization: `Bearer ${s.token}`, "content-type": "application/json" }, + headers: { "content-type": "application/json" }, body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }), }); const body = await reply.json(); diff --git a/packages/blocks/src/v8/__conformance__/guides.test.ts b/packages/blocks/src/v8/__conformance__/guides.test.ts index cdaf9ca40..e6715b4c3 100644 --- a/packages/blocks/src/v8/__conformance__/guides.test.ts +++ b/packages/blocks/src/v8/__conformance__/guides.test.ts @@ -25,6 +25,7 @@ import { import { AnalyticsScript } from "@decocms/blocks/analytics"; import { createInstrumentedFetch } from "@decocms/blocks/fetch"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { instanceOf } from "../cms"; import { resolveDestination } from "../telemetry"; const SRC = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); @@ -143,14 +144,13 @@ describe("quickstart", () => { expect(spy.mock.calls[0][0]).toEqual({ title: "hi" }); }); - it("qs-17/hiw-06/dd-03: exactly ten built-ins, the design-decisions nine plus secret, all resolvable", async () => { + it("qs-17/hiw-06/dd-03: exactly nine built-ins, all resolvable", async () => { const { builtIns } = await import("../builtins/index"); expect(Object.keys(builtIns).sort()).toEqual( [ "page", "redirect", - "telemetry", - "analytics", + "cms-settings", "always", "never", "date", @@ -166,8 +166,7 @@ describe("quickstart", () => { ["date", {}], ["page", { name: "x", path: "/", sections: [] }], ["redirect", { from: "/a", to: "/b", permanent: false }], - ["telemetry", {}], - ["analytics", {}], + ["cms-settings", {}], ["multivariate", { variants: [] }], ] as const) { const [, error] = await client.resolve({ __resolveType: type, ...inputs }); @@ -619,19 +618,16 @@ describe("renames and migrations", () => { ).not.toThrow(); }); - it("mig-10/dd-16: the analytics block fills defaults; AnalyticsScript renders nothing when disabled", async () => { - const client = createCMS({ + it("mig-10/dd-16: the analytics section fills defaults; AnalyticsScript renders nothing when disabled", async () => { + const cms = createCMS({ blocks: {}, - content: snap({ Analytics: { __resolveType: "analytics" } }), - }).forRelease(); - const [settings, error] = await client.resolve<{ enabled: boolean; collector: string }>( - "Analytics", - ); - expect(error).toBeNull(); - expect(settings?.enabled).toBe(true); - expect(settings?.collector).toMatch(/^https:\/\//); - expect(AnalyticsScript({ ...settings, enabled: false })).toBeNull(); - expect(AnalyticsScript(settings ?? {})).not.toBeNull(); + content: snap({ CMS: { __resolveType: "cms-settings" } }), + }); + const { analytics } = await cms.settings(); + expect(analytics.enabled).toBe(true); + expect(analytics.collector).toMatch(/^https:\/\//); + expect(AnalyticsScript({ ...analytics, enabled: false })).toBeNull(); + expect(AnalyticsScript(analytics)).not.toBeNull(); }); }); @@ -715,7 +711,7 @@ describe("troubleshooting", () => { const content = { ...snap({}), root: ".deco" }; const first = createCMS({ blocks: {}, content, telemetry: false }); const second = createCMS({ blocks: {}, content, telemetry: { endpoint: "https://x.example" } }); - expect(second).toBe(first); + expect(instanceOf(second)).toBe(instanceOf(first)); expect(warn).toHaveBeenCalledWith(expect.stringMatching(/different options/)); }); @@ -792,7 +788,7 @@ describe("internals", () => { const copy = (await import( /* @vite-ignore */ `../cms?copy=${Date.now()}` )) as typeof import("../cms"); - expect(copy.createCMS({ blocks: {}, content })).toBe(first); + expect(instanceOf(copy.createCMS({ blocks: {}, content }))).toBe(instanceOf(first)); const r1 = remoteLoader(content, { site: "s", token: "t" }); const copyRemote = (await import( /* @vite-ignore */ `../remoteLoader?copy=${Date.now()}` @@ -888,19 +884,18 @@ describe("design decisions", () => { expect(load).toHaveBeenCalledTimes(1); }); - it("dd-11: content can be a Loader with load(pointer?) and optional update()", async () => { - const pointers: unknown[] = []; + it("dd-11: content can be a Loader with load() and optional update()", async () => { + const calls: unknown[][] = []; const loader: Loader = { - load: async (pointer) => { - pointers.push(pointer); + load: async (...args: unknown[]) => { + calls.push(args); return snap({}); }, }; const cms = createCMS({ blocks: {}, content: loader }); await cms.forRelease().revision(); - await cms.forDraft("localhost:8000/x@v1").revision(); - expect(pointers[0] ?? null).toBeNull(); - expect(pointers[1]).toBe("localhost:8000/x@v1"); + await cms.forDraft("localhost:8000/x@local").revision(); // deco serve's pointer: no draft + expect(calls).toEqual([[], []]); }); it("dd-12: a request never waits for update()", async () => { @@ -942,7 +937,9 @@ describe("design decisions", () => { try { const capped = createCMS({ blocks: { broken }, - content: snap({ Telemetry: { __resolveType: "telemetry", errorSampleRate: 1 } }), + content: snap({ + CMS: { __resolveType: "cms-settings", telemetry: { errorSampleRate: 1 } }, + }), telemetry: { endpoint: "https://otel.example.com", limits: { errorSampleRate: 0.1 } }, }); vi.spyOn(Math, "random").mockReturnValue(0.5); // inside the editor's 1, above code's 0.1 @@ -953,7 +950,7 @@ describe("design decisions", () => { const off = createCMS({ blocks: { broken }, content: snap({ - Telemetry: { __resolveType: "telemetry", enabled: false, errorSampleRate: 1 }, + CMS: { __resolveType: "cms-settings", telemetry: { enabled: false, errorSampleRate: 1 } }, }), telemetry: { endpoint: "https://otel2.example.com", limits: { errorSampleRate: 1 } }, }); diff --git a/packages/blocks/src/v8/__conformance__/observability.analytics.test.tsx b/packages/blocks/src/v8/__conformance__/observability.analytics.test.tsx index 7b3ec6126..b88af34f4 100644 --- a/packages/blocks/src/v8/__conformance__/observability.analytics.test.tsx +++ b/packages/blocks/src/v8/__conformance__/observability.analytics.test.tsx @@ -82,18 +82,19 @@ function decode(src: string): any { return JSON.parse(new TextDecoder().decode(bytes)); } -const resolveAnalytics = async (block: Record) => { +const resolveAnalytics = async (section: Record) => { const cms = createCMS({ blocks: {}, - content: { revision: "r", blocks: { Analytics: { __resolveType: "analytics", ...block } } }, + content: { + revision: "r", + blocks: { CMS: { __resolveType: "cms-settings", analytics: section } }, + }, }); - const [value, error] = await cms.forRelease().resolve("Analytics"); - expect(error).toBeNull(); - return value as Required; + return (await cms.settings()).analytics; }; -describe("the analytics block (analytics.mdx)", () => { - it("ana-01/ana-02/htel-04: {__resolveType: 'analytics'} resolves to settings with the hosted collector and enabled: true", async () => { +describe("the analytics section of the CMS settings (analytics.mdx)", () => { + it("ana-01/ana-02/htel-04: the analytics section resolves to settings with the hosted collector and enabled: true", async () => { expect(await resolveAnalytics({})).toEqual({ collector: HOSTED_ANALYTICS_COLLECTOR, enabled: true, @@ -104,7 +105,7 @@ describe("the analytics block (analytics.mdx)", () => { }); }); - it("ana-02: the analytics block's props are collector and enabled only (no site ID)", async () => { + it("ana-02: the analytics section's fields are collector and enabled only (no site ID)", async () => { // The resolved settings carry nothing else either. expect(Object.keys(await resolveAnalytics({})).sort()).toEqual(["collector", "enabled"]); }); @@ -114,15 +115,10 @@ describe("the analytics block (analytics.mdx)", () => { expect(typeof analyticsModule.track).toBe("function"); }); - it('ana-04: resolve("Analytics") returns a tuple whose first element spreads into ', async () => { - const cms = createCMS({ - blocks: {}, - content: { revision: "r", blocks: { Analytics: { __resolveType: "analytics" } } }, - }); - const result = await cms.forRelease().resolve("Analytics"); - expect(Array.isArray(result)).toBe(true); - const [analytics] = result; - const html = renderToString(); + it("ana-04: cms.settings().analytics spreads into , with no CMS block at all", async () => { + const cms = createCMS({ blocks: {}, content: { revision: "r", blocks: {} } }); + const { analytics } = await cms.settings(); + const html = renderToString(); expect(html).toContain(" { expect(renderToStaticMarkup()).toBe(""); }); - it("ana-10: variants pick between analytics settings like any block", async () => { + it("ana-10: the section can have variants, picked when cms.settings() is called", async () => { const cms = createCMS({ blocks: {}, content: { revision: "r", blocks: { - Analytics: { - __resolveType: "multivariate", - variants: [ - { - rule: { __resolveType: "never" }, - value: { - __resolveType: "lazy", - value: { __resolveType: "analytics", collector: "https://a.example" }, + CMS: { + __resolveType: "cms-settings", + analytics: { + __resolveType: "multivariate", + variants: [ + { + rule: { __resolveType: "never" }, + value: { __resolveType: "lazy", value: { collector: "https://a.example" } }, }, - }, - { - rule: { __resolveType: "always" }, - value: { - __resolveType: "lazy", - value: { __resolveType: "analytics", enabled: false }, + { + rule: { __resolveType: "always" }, + value: { __resolveType: "lazy", value: { enabled: false } }, }, - }, - ], + ], + }, }, }, }, }); - const [settings] = await cms.forRelease().resolve("Analytics"); - expect(settings).toEqual({ collector: HOSTED_ANALYTICS_COLLECTOR, enabled: false }); + const { analytics } = await cms.settings(); + expect(analytics).toEqual({ collector: HOSTED_ANALYTICS_COLLECTOR, enabled: false }); }); - it("up-04: the framework ships always, never, date, multivariate and analytics built-ins", () => { - for (const name of ["always", "never", "date", "multivariate", "analytics"]) { + it("up-04: the framework ships always, never, date, multivariate and cms-settings built-ins", () => { + for (const name of ["always", "never", "date", "multivariate", "cms-settings"]) { expect(typeof builtIns[name]).toBe("function"); } }); diff --git a/packages/blocks/src/v8/__conformance__/observability.test.ts b/packages/blocks/src/v8/__conformance__/observability.test.ts index 58d3b0bef..610bf092f 100644 --- a/packages/blocks/src/v8/__conformance__/observability.test.ts +++ b/packages/blocks/src/v8/__conformance__/observability.test.ts @@ -20,7 +20,7 @@ import { createFixture } from "../cli/__tests__/fixture"; import { decoPaths } from "../cli/root"; import { generateSchema } from "../cli/schema/generate"; import { toBase64 } from "../cli/schema/typeToSchema"; -import { createCMS, resetForTests } from "../cms"; +import { createCMS, instanceOf, resetForTests } from "../cms"; import { createInstrumentedFetch } from "../fetch"; import { currentTelemetry } from "../telemetry"; import { docsBlocks, docsSnapshot } from "../testFixtures"; @@ -121,7 +121,7 @@ function withTelemetryBlock(block: Record, revision = "rev-1"): const snapshot = docsSnapshot(revision); return { ...snapshot, - blocks: { ...snapshot.blocks, Telemetry: { __resolveType: "telemetry", ...block } }, + blocks: { ...snapshot.blocks, CMS: { __resolveType: "cms-settings", telemetry: block } }, }; } @@ -190,7 +190,7 @@ describe("the docs' snippets typecheck (conformance/observability)", () => { it("htel-01: the hosted-telemetry cms.ts compiles with site/token possibly undefined", () => { expect(errorsIn("hosted-cms.ts")).toBe(""); }); - it('ana-04: the analytics layout (`const [analytics] = …resolve("Analytics")`, ``) compiles', () => { + it("ana-04: the analytics layout (`const { analytics } = await cms.settings()`, ``) compiles", () => { expect(errorsIn("layout.tsx")).toBe(""); }); it("ana-08: the track() button compiles", () => { @@ -419,8 +419,8 @@ describe("what's sent (telemetry.mdx)", () => { expect(sent.some((s) => s.url.endsWith("/v1/traces"))).toBe(false); }); - // The first client of a release must already honour that release's Telemetry - // block: no warm-up request before the one that should be traced. + // The first client of a release must already honour that release's telemetry + // section: no warm-up request before the one that should be traced. it("tel-12: with limits and block traceSampleRate 1, block resolution and upstream spans are sent", async () => { vi.spyOn(Math, "random").mockReturnValue(0); const { sent } = collector(); @@ -506,15 +506,19 @@ describe("what's sent (telemetry.mdx)", () => { // telemetry.mdx › Telemetry settings are content // --------------------------------------------------------------------------- -describe("the Telemetry block (telemetry.mdx)", () => { - it("tel-16/tel-21/ana-02: `telemetry` is always in the schema, with enabled, metrics, errorSampleRate, traceSampleRate and no destination fields", async () => { +describe("the telemetry section of the CMS block (telemetry.mdx)", () => { + it("tel-16/tel-21/ana-02: `cms-settings` is always in the schema, its telemetry section with enabled, metrics, errorSampleRate, traceSampleRate and no destination fields", async () => { const fixture = createFixture({ ".deco/index.ts": "export default {};\n", }); try { const { meta } = await generateSchema(decoPaths(fixture.root)); - expect(meta.manifest.blocks.content.telemetry).toBeDefined(); - const definition = meta.schema.definitions[toBase64("telemetry")] as any; + expect(meta.manifest.blocks.content["cms-settings"]).toBeDefined(); + const settings = meta.schema.definitions[toBase64("cms-settings")] as any; + expect(Object.keys(settings.properties).sort()).toEqual( + ["__resolveType", "analytics", "preview", "telemetry"].sort(), + ); + const definition = settings.properties.telemetry; const props = definition.properties ?? definition; const fields = Object.keys( JSON.stringify(definition).includes("errorSampleRate") ? props : {}, @@ -527,19 +531,18 @@ describe("the Telemetry block (telemetry.mdx)", () => { expect(JSON.stringify(definition)).not.toContain(`"${forbidden}"`); } expect(fields).not.toContain("endpoint"); - // ana-02: the analytics form has collector and enabled only (no site ID). - const analytics = meta.schema.definitions[toBase64("analytics")] as any; - expect( - Object.keys(analytics.properties ?? {}) - .filter((k) => k !== "__resolveType") - .sort(), - ).toEqual(["collector", "enabled"]); + // ana-02: the analytics section has collector and enabled only (no site ID). + const analytics = settings.properties.analytics; + expect(Object.keys(analytics.properties ?? {}).sort()).toEqual(["collector", "enabled"]); + // The old built-ins are gone, with no alias. + expect(meta.schema.definitions[toBase64("telemetry")]).toBeUndefined(); + expect(meta.schema.definitions[toBase64("analytics")]).toBeUndefined(); } finally { fixture.remove(); } }, 60_000); - it("tel-17: defaults apply with no Telemetry block: enabled, metrics on, errorSampleRate 0.05, traces off", async () => { + it("tel-17: defaults apply with no CMS block: enabled, metrics on, errorSampleRate 0.05, traces off", async () => { const { sent } = collector(); const cms = createCMS({ blocks: { ...docsBlocks(), broken }, @@ -558,10 +561,18 @@ describe("the Telemetry block (telemetry.mdx)", () => { expect(spans(sent)).toHaveLength(0); }); - it("tel-17: {__resolveType: 'telemetry'} resolves to its arguments as saved (built-in-blocks)", async () => { + it("tel-17: {__resolveType: 'cms-settings'} returns its input with the telemetry defaults filled in (built-in-blocks)", async () => { const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); - const [value] = await cms.forRelease().resolve({ __resolveType: "telemetry", metrics: false }); - expect(value).toEqual({ metrics: false }); + const [value] = await cms.forRelease().resolve<{ telemetry: unknown }>({ + __resolveType: "cms-settings", + telemetry: { metrics: false }, + }); + expect(value?.telemetry).toEqual({ + enabled: true, + metrics: false, + errorSampleRate: 0.05, + traceSampleRate: 0, + }); }); it("tel-18: enabled: false switches all telemetry off", async () => { @@ -593,7 +604,7 @@ describe("the Telemetry block (telemetry.mdx)", () => { expect(sent.some((s) => s.url.endsWith("/v1/logs"))).toBe(true); }); - it("tel-20: deco content doesn't create .deco/blocks/Telemetry.json", () => { + it("tel-20: deco content doesn't create .deco/blocks/CMS.json", () => { const fixture = createFixture({ ".deco/index.ts": "export default {};\n", ".deco/blocks/Home.json": { __resolveType: "page", name: "Home", path: "/", sections: [] }, @@ -605,13 +616,13 @@ describe("the Telemetry block (telemetry.mdx)", () => { stdio: "pipe", }); expect(fixture.exists(".deco/blocks.gen.ts")).toBe(true); - expect(fixture.exists(".deco/blocks/Telemetry.json")).toBe(false); + expect(fixture.exists(".deco/blocks/CMS.json")).toBe(false); } finally { fixture.remove(); } }, 60_000); - it("tel-21: a Telemetry block can't redirect telemetry (an endpoint field is ignored)", async () => { + it("tel-21: the telemetry section can't redirect telemetry (an endpoint field is ignored)", async () => { const { sent } = collector(); const cms = createCMS({ blocks: docsBlocks(), @@ -997,7 +1008,7 @@ describe("caching (caching.mdx)", () => { const loader = { load, update: async () => ({ updated: false }) }; const cmsA = a.createCMS({ blocks: docsBlocks(), content: loader }); const cmsB = b.createCMS({ blocks: docsBlocks(), content: loader }); - expect(cmsB).toBe(cmsA); + expect(instanceOf(cmsB)).toBe(instanceOf(cmsA)); await cmsA.forRelease().resolve("SummerSEO"); await cmsB.forRelease().resolve("SummerSEO"); expect(load).toHaveBeenCalledTimes(1); diff --git a/packages/blocks/src/v8/__conformance__/sdk.test.ts b/packages/blocks/src/v8/__conformance__/sdk.test.ts index 3a16e3e1b..2d87f68f1 100644 --- a/packages/blocks/src/v8/__conformance__/sdk.test.ts +++ b/packages/blocks/src/v8/__conformance__/sdk.test.ts @@ -1,6 +1,6 @@ // @vitest-environment node /** - * Conformance: the docs (blocks-site docs/content/next: api-reference, + * Conformance: the docs (deco-sites/docs-tanstack src/content/docs/en/storefront/blocks/next: api-reference, * content, releases-and-drafts, routing, router-internals, rendering) are the * source of truth. Each `describe` names the claim it checks. A failing test * here is a gap between the docs and the code, to be fixed in one of them. @@ -14,14 +14,12 @@ import * as root from "../../index"; import { generateSecretsKeyPair } from "../../protocol/__tests__/fixtures"; import * as analyticsModule from "../analytics"; import { AnalyticsScript, track } from "../analytics"; +import { instanceOf } from "../cms"; import * as fetchModule from "../fetch"; import { createInstrumentedFetch } from "../fetch"; import * as v8 from "../index"; import { createCMS, - DRAFT_COOKIE, - draftCookie, - draftPointer, formatDraftPointer, matchRoute, parseDraftPointer, @@ -31,10 +29,27 @@ import { import * as secretsModule from "../secrets"; import { encryptSecret } from "../secrets"; import { resolveDestination, setCurrentTelemetry, TelemetryPipeline } from "../telemetry"; -import { docsBlocks, docsSnapshot, hero, seo } from "../testFixtures"; -import type { Blocks, Loader, Redirect, Route, Snapshot } from "../types"; +import { docsBlocks, docsSnapshot, fakeStudio, hero, seo } from "../testFixtures"; +import type { Blocks, CMS, Redirect, RequestLike, Route, Snapshot } from "../types"; import { typecheck } from "./typecheck"; +/** cms.draftPointer and cms.draftCookie, on a CMS with no settings: every host may preview. */ +const helpers = () => createCMS({ blocks: {}, content: { revision: "draft-helpers", blocks: {} } }); +const draftPointer = (request: RequestLike) => helpers().draftPointer(request); +const draftCookie = (request: RequestLike) => helpers().draftCookie(request); +const DRAFT_COOKIE = "__deco_draft"; + +/** A fake Studio holding one draft that retitles SummerSEO; returns its pointer and the stubbed fetch. */ +function studioDraft(title = "Draft!") { + const studio = fakeStudio(); + const fetch = vi.fn(studio.fetch); + vi.stubGlobal("fetch", fetch); + const pointer = studio.draft({ + set: { SummerSEO: { __resolveType: "seo", title, description: "Light layers for long days." } }, + }); + return { studio, fetch, pointer }; +} + const here = path.dirname(fileURLToPath(import.meta.url)); const pkgRoot = path.resolve(here, "../../.."); const readSource = (rel: string) => fs.readFileSync(path.join(pkgRoot, rel), "utf8"); @@ -110,10 +125,7 @@ function exportedNames(source: string, from: string): string[] { // --------------------------------------------------------------------------- const DOCUMENTED_ROOT_VALUES = [ - "DRAFT_COOKIE", "createCMS", - "draftCookie", - "draftPointer", "formatDraftPointer", "matchRoute", "parseDraftPointer", @@ -136,6 +148,9 @@ const DOCUMENTED_ROOT_TYPES = [ "Variant", "Telemetry", "Analytics", + "CMSSettings", + "EffectiveSettings", + "RequestLike", "Match", "Result", "CMSError", @@ -214,7 +229,8 @@ describe("AR-04 / AR-10 / AR-14 / AR-48 / AR-57 / AR-58 / AR-62 documented signa "sig.ts": ` import { createCMS, type Blocks, type CMS, type Client, type Snapshot, type Loader, type ListOptions, type Result, type CMSError, type Block, type BlockFunction, type Route, type Seo, type Page, - type Redirect, type Secret, type Lazy, type Variant, type TelemetryConfig, type Telemetry, type Analytics } from "@decocms/blocks"; + type Redirect, type Secret, type Lazy, type Variant, type TelemetryConfig, type Telemetry, type Analytics, + type CMSSettings, type EffectiveSettings, type RequestLike } from "@decocms/blocks"; import type { ReactNode } from "react"; type Equal = (() => T extends A ? 1 : 2) extends (() => T extends B ? 1 : 2) ? true : false; @@ -227,10 +243,19 @@ createCMS({ blocks, content }); createCMS({ blocks, content: loader, interval: 60_000, telemetry: false, secrets: { key: "k" }, site: "s", token: "t" }); createCMS({ blocks, content, telemetry: { site: "s", token: "t", limits: { errorSampleRate: 0.1, traceSampleRate: 0 } } }); createCMS({ blocks, content, telemetry: { endpoint: "https://otel.example", headers: { a: "b" } } }); +createCMS({ blocks, content, preview: { hosts: ["*.example.com", "localhost:3000"] } }); +// @ts-expect-error draft hosts come from DECO_PREVIEW_API_DOMAINS, not code +createCMS({ blocks, content, preview: { sources: ["studio.example.com"] } }); // @ts-expect-error not a documented option createCMS({ blocks, content, ignoreCase: true }); -assert>(); +assert>(); +assert Promise>>(); +assert Promise>>(); +assert Promise>>(); +assert>(); +assert>(); +assert; analytics: Required }>>(); assert Client>>(); assert Client>>(); assert Promise<{ updated: boolean }>>>(); @@ -245,7 +270,7 @@ assert, { where?: (entry: number) => boolean; sort?: ( assert>(); assert unknown | Promise>>(); assert>>(); -assert Promise>>(); +assert Promise>>(); assert>(); assert>(); assert>(); @@ -279,8 +304,8 @@ assert; alia }); describe("AR-05 interval: default DECO_CONTENT_INTERVAL or 60 000, minimum 60 000", () => { - const intervalOf = (cms: unknown) => - (cms as { fingerprint: { interval: number } }).fingerprint.interval; + const intervalOf = (cms: CMS) => + (instanceOf(cms) as { fingerprint: { interval: number } }).fingerprint.interval; it("clamps a value below the minimum", () => { vi.spyOn(console, "warn").mockImplementation(() => {}); @@ -341,12 +366,19 @@ describe("AR-08 site and token load hosted content only, never telemetry", () => const fetch = vi.fn(async () => new Response("{}")); vi.stubGlobal("fetch", fetch); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot(), site: "acme" }); - const [seo] = await cms.forDraft("delivery.decocms.com/drafts/acme/x@v1").resolve("SummerSEO"); + const [seo] = await cms.forRelease().resolve("SummerSEO"); expect(seo).toEqual({ title: "Sunny!", description: "Light layers for long days." }); expect(await cms.update()).toEqual({ updated: false }); expect(fetch).not.toHaveBeenCalled(); }); + it("drafts don't need site or token", async () => { + const { pointer } = studioDraft(); + const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); + const [seo] = await cms.forDraft(pointer).resolve<{ title: string }>("SummerSEO"); + expect(seo?.title).toBe("Draft!"); + }); + it("site and token with telemetry omitted and no OTEL env: no telemetry", async () => { const { currentTelemetry } = await import("../telemetry"); createCMS({ blocks: docsBlocks(), content: docsSnapshot(), site: "acme", token: "t" }); @@ -374,38 +406,38 @@ describe("AR-09 / AR-56 secrets", () => { }); describe("AR-11 / AR-20 / AR-26 / RD-03 / CT-09 drafts", () => { - it("a snapshot source ignores the pointer: forDraft reads the release", async () => { + it("forDraft layers the pointer's changes over the content module", async () => { + const { pointer } = studioDraft(); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); - const draft = cms.forDraft("api.deco.example/drafts/acme/main@9f3c1a"); - expect(await draft.resolve("SummerSEO")).toEqual(await cms.forRelease().resolve("SummerSEO")); - expect(await draft.revision()).toBe("rev-1"); + const draft = cms.forDraft(pointer); + expect((await draft.resolve<{ title: string }>("SummerSEO"))[0]?.title).toBe("Draft!"); + expect((await draft.resolve("HomePage"))[1]).toBeNull(); + expect(await draft.revision()).toBe("rev-1~9f3c1a"); }); - it("a draft whose load(pointer) fails makes every call return LOADER_FAILED with the cause", async () => { - const cause = new Error("storage down"); - const loader: Loader = { - load: async (pointer) => { - if (pointer) throw cause; - return docsSnapshot(); - }, - }; - const cms = createCMS({ blocks: docsBlocks(), content: loader }); - const client = cms.forDraft("api.deco.example/drafts/acme/main@9f3c1a"); + it("a draft whose changes can't be fetched makes every call return LOADER_FAILED with the cause", async () => { + const { studio, pointer } = studioDraft(); + studio.respond("summer-sale", () => new Response("down", { status: 502 })); + const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); + const client = cms.forDraft(pointer); const [value, error] = await client.resolve("SummerSEO"); expect(value).toBeNull(); - expect(error).toMatchObject({ code: "LOADER_FAILED", cause }); + expect(error?.code).toBe("LOADER_FAILED"); + expect(String((error?.cause as Error | undefined)?.message)).toContain("HTTP 502"); const [list, listError] = await client.list("page"); expect(list).toBeNull(); expect(listError?.code).toBe("LOADER_FAILED"); }); - it("a pointer that doesn't parse makes every call return LOADER_FAILED; load(pointer) is never called", async () => { - const load = vi.fn(async () => docsSnapshot()); - const cms = createCMS({ blocks: docsBlocks(), content: { load } }); - const client = cms.forDraft("garbage"); - expect((await client.resolve("SummerSEO"))[1]?.code).toBe("LOADER_FAILED"); - expect((await client.list("page"))[1]?.code).toBe("LOADER_FAILED"); - expect(load).not.toHaveBeenCalledWith("garbage"); + it("a pointer that doesn't parse, or names a host outside the preview API domains, is LOADER_FAILED with no fetch", async () => { + const { fetch } = studioDraft(); + const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); + for (const pointer of ["garbage", "api.deco.example/drafts/acme/main@9f3c1a"]) { + const client = cms.forDraft(pointer); + expect((await client.resolve("SummerSEO"))[1]?.code).toBe("LOADER_FAILED"); + expect((await client.list("page"))[1]?.code).toBe("LOADER_FAILED"); + } + expect(fetch).not.toHaveBeenCalled(); }); it("a snapshot source has no scheduled checks", async () => { @@ -484,21 +516,18 @@ describe("AR-66 a draft pointer's forced variants (releases-and-drafts#preview-a expect([value, error]).toEqual(["spring", null]); }); - it("a loader gets the pointer without them, once for every variant of one draft", async () => { - const load = vi.fn(async (_pointer?: string | null) => content()); - const cms = createCMS({ blocks: docsBlocks(), content: { load } }); + it("a draft is fetched without them, once for every variant of one draft", async () => { + const { fetch, pointer } = studioDraft(); + const cms = createCMS({ blocks: docsBlocks(), content: content() }); const draft = (index: number) => formatDraftPointer({ - host: "api.deco.example", - path: "/drafts/acme/main?token=t", - version: "9f3c1a", + ...parseDraftPointer(pointer)!, variants: [{ block: "Banner", path: "", index }], }); expect((await cms.forDraft(draft(1)).resolve("Banner"))[0]).toBe("summer"); expect((await cms.forDraft(draft(0)).resolve("Banner"))[0]).toBe("spring"); - expect(load.mock.calls.filter(([p]) => p)).toEqual([ - ["api.deco.example/drafts/acme/main?token=t@9f3c1a"], - ]); + expect(fetch).toHaveBeenCalledTimes(1); + expect(String(fetch.mock.calls[0]![0])).not.toContain("__variant"); }); }); @@ -596,17 +625,17 @@ describe("AR-17 clients are cheap: the content cache lives in the CMS", () => { }); describe("AR-18 / AR-19 content: a snapshot or any loader", () => { - it("accepts both; load() for the release, load(pointer) for a draft", async () => { + it("accepts both; load() takes no argument, and a draft is layered over what it returned", async () => { const fromSnapshot = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); expect((await fromSnapshot.forRelease().resolve("SummerSEO"))[1]).toBeNull(); - const load = vi.fn(async (_pointer?: string | null) => docsSnapshot()); + const { pointer } = studioDraft(); + const load = vi.fn(async () => docsSnapshot()); const cms = createCMS({ blocks: docsBlocks(), content: { load } }); await cms.forRelease().resolve("SummerSEO"); - expect(load).toHaveBeenLastCalledWith(); - const pointer = "api.deco.example/drafts/acme/main@9f3c1a"; - await cms.forDraft(pointer).resolve("SummerSEO"); - expect(load).toHaveBeenLastCalledWith(pointer); + const [seo] = await cms.forDraft(pointer).resolve<{ title: string }>("SummerSEO"); + expect(seo?.title).toBe("Draft!"); + for (const call of load.mock.calls) expect(call).toEqual([]); }); }); @@ -617,8 +646,10 @@ describe("AR-22 remoteLoader", () => { expect(await remoteLoader(fallback, { site: "", token: "" }).load()).toBe(fallback); }); - it("createCMS builds it when site and token are set: forDraft reaches the Deco API", async () => { - const fetch = vi.fn(async () => new Response("nope", { status: 404 })); + it("createCMS builds it when site and token are set: update() reaches the Deco API", async () => { + const fetch = vi.fn( + async (_input: string | URL | Request) => new Response("nope", { status: 404 }), + ); vi.stubGlobal("fetch", fetch); const cms = createCMS({ blocks: docsBlocks(), @@ -626,11 +657,10 @@ describe("AR-22 remoteLoader", () => { site: "acme", token: "t", }); - const [, error] = await cms - .forDraft("delivery.decocms.com/drafts/acme/x@v1") - .resolve("SummerSEO"); - expect(error?.code).toBe("LOADER_FAILED"); - expect(fetch).toHaveBeenCalled(); + expect(await cms.update()).toEqual({ updated: false }); + expect(String(fetch.mock.calls[0]?.[0])).toBe( + "https://delivery.decocms.com/sites/acme/channels/production.json", + ); }); }); @@ -687,8 +717,10 @@ describe("AR-28 / AR-29 / AR-30 / CT-10 one instance per process", () => { it("same content module object -> same instance; different -> different", () => { const content = docsSnapshot(); const a = createCMS({ blocks: docsBlocks(), content }); - expect(createCMS({ blocks: docsBlocks(), content })).toBe(a); - expect(createCMS({ blocks: docsBlocks(), content, site: "acme", token: "t" })).not.toBe(a); + expect(instanceOf(createCMS({ blocks: docsBlocks(), content }))).toBe(instanceOf(a)); + expect( + instanceOf(createCMS({ blocks: docsBlocks(), content, site: "acme", token: "t" })), + ).not.toBe(instanceOf(a)); }); it("a hot reload that hands in a new content module (new revision) keeps the instance", () => { @@ -696,7 +728,7 @@ describe("AR-28 / AR-29 / AR-30 / CT-10 one instance per process", () => { const before = { revision: "rev-1", blocks: {}, aliases: {}, root: "apps/site/.deco" }; const after = { revision: "rev-2", blocks: {}, aliases: {}, root: "apps/site/.deco" }; const a = createCMS({ blocks: {}, content: before }); - expect(createCMS({ blocks: {}, content: after })).toBe(a); + expect(instanceOf(createCMS({ blocks: {}, content: after }))).toBe(instanceOf(a)); }); it("the key is a Symbol.for('decocms.blocks…') on globalThis", () => { @@ -711,7 +743,7 @@ describe("AR-28 / AR-29 / AR-30 / CT-10 one instance per process", () => { const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); const content = docsSnapshot(); const a = createCMS({ blocks: {}, content }); - expect(createCMS({ blocks: {}, content, interval: 120_000 })).toBe(a); + expect(instanceOf(createCMS({ blocks: {}, content, interval: 120_000 }))).toBe(instanceOf(a)); expect(warn).toHaveBeenCalledWith(expect.stringContaining("interval")); }); @@ -778,38 +810,45 @@ if (pointer) console.log(\`previewing \${pointer.version} from \${pointer.host}\ expect(errors).toEqual([]); }, 60_000); - it("AR-35 draftPointer: URL first, then the cookie; null for neither or ?__draft=off", () => { + it("AR-35 draftPointer: URL first, then the cookie; null for neither or ?__draft=off", async () => { const cookie = { cookie: `${DRAFT_COOKIE}=${encodeURIComponent("h/c@v1")}` }; - expect(draftPointer(new Request("https://s/x?__draft=h/u@v1", { headers: cookie }))).toBe( + expect(await draftPointer(new Request("https://s/x?__draft=h/u@v1", { headers: cookie }))).toBe( "h/u@v1", ); - expect(draftPointer(new Request("https://s/x", { headers: cookie }))).toBe("h/c@v1"); - expect(draftPointer(new Request("https://s/x"))).toBeNull(); - expect(draftPointer(new Request("https://s/x?__draft=off", { headers: cookie }))).toBeNull(); + expect(await draftPointer(new Request("https://s/x", { headers: cookie }))).toBe("h/c@v1"); + expect(await draftPointer(new Request("https://s/x"))).toBeNull(); + expect( + await draftPointer(new Request("https://s/x?__draft=off", { headers: cookie })), + ).toBeNull(); }); - it("AR-36 draftCookie: HttpOnly; Secure; SameSite=None; Partitioned; Path=/; off expires it; null otherwise", () => { - const set = draftCookie(new Request("https://s/x?__draft=h/u@v1")); - expect(set).toContain("deco-draft="); + it("AR-36 draftCookie: HttpOnly; Secure; SameSite=None; Partitioned; Path=/; off expires it; null otherwise", async () => { + const set = await draftCookie(new Request("https://s/x?__draft=h/u@v1")); + expect(set).toContain("__deco_draft="); for (const attribute of ["HttpOnly", "Secure", "SameSite=None", "Partitioned", "Path=/"]) { expect(set).toContain(attribute); } expect(set).not.toContain("SameSite=Lax"); - const off = draftCookie(new Request("https://s/x?__draft=off")); + const off = await draftCookie(new Request("https://s/x?__draft=off")); expect(off).toMatch(/Max-Age=0|Expires=/); - expect(draftCookie(new Request("https://s/x"))).toBeNull(); + expect(await draftCookie(new Request("https://s/x"))).toBeNull(); // An invalid ?__draft= gives null too. - expect(draftCookie(new Request("https://s/x?__draft=not-a-pointer"))).toBeNull(); + expect(await draftCookie(new Request("https://s/x?__draft=not-a-pointer"))).toBeNull(); }); - it("AR-37 DRAFT_COOKIE is deco-draft", () => { - expect(DRAFT_COOKIE).toBe("deco-draft"); + it("AR-37 the helpers are CMS methods; the package root has no free draftPointer, draftCookie or DRAFT_COOKIE", () => { + for (const name of ["draftPointer", "draftCookie", "DRAFT_COOKIE"]) { + expect(Object.keys(root)).not.toContain(name); + } + const cms = helpers(); + expect(typeof cms.draftPointer).toBe("function"); + expect(typeof cms.draftCookie).toBe("function"); }); - it("AR-38 both helpers take anything with url and headers", () => { + it("AR-38 both helpers take anything with url and headers", async () => { const like = { url: "https://s/x?__draft=h/u@v1", headers: new Headers() }; - expect(draftPointer(like)).toBe("h/u@v1"); - expect(draftCookie(like)).not.toBeNull(); + expect(await draftPointer(like)).toBe("h/u@v1"); + expect(await draftCookie(like)).not.toBeNull(); }); }); @@ -1165,10 +1204,10 @@ describe("AR-55 / AR-61 analytics", () => { expect(() => track("signup", { plan: "pro" })).not.toThrow(); }); - it("the analytics block fills the defaults", async () => { + it("the analytics section of cms.settings() fills the defaults", async () => { const cms = createCMS({ blocks: {}, content: { revision: "r", blocks: {} } }); - const [value] = await cms.forRelease().resolve({ __resolveType: "analytics" }); - expect(value).toEqual({ collector: expect.stringMatching(/^https:\/\//), enabled: true }); + const { analytics } = await cms.settings(); + expect(analytics).toEqual({ collector: expect.stringMatching(/^https:\/\//), enabled: true }); }); }); @@ -1211,7 +1250,7 @@ describe("AR-59 multivariate and lazy", () => { }); }); -describe("AR-60 the Telemetry block: defaults and caps", () => { +describe("AR-60 the telemetry section: defaults and caps", () => { it("defaults 0.05 / 0, capped by limits", async () => { const sent: string[] = []; vi.stubGlobal( @@ -1223,15 +1262,11 @@ describe("AR-60 the Telemetry block: defaults and caps", () => { ); const pipeline = (limits?: { errorSampleRate?: number; traceSampleRate?: number }) => new TelemetryPipeline(resolveDestination({ endpoint: "https://otel.example", limits })!); - const withBlock = (block: Record): Snapshot => ({ - revision: `r-${JSON.stringify(block)}`, - blocks: { Telemetry: { __resolveType: "telemetry", ...block } }, - }); const random = vi.spyOn(Math, "random"); // errorSampleRate 0.5 under the default limit 0.1: a draw of 0.2 is dropped, 0.09 kept. const capped = pipeline(); - capped.useRelease(withBlock({ errorSampleRate: 0.5 })); + capped.apply({ errorSampleRate: 0.5 }); random.mockReturnValue(0.2); capped.error("e", {}); await capped.flush(); @@ -1244,7 +1279,7 @@ describe("AR-60 the Telemetry block: defaults and caps", () => { // Default errorSampleRate 0.05: 0.06 dropped. sent.length = 0; const defaults = pipeline({ errorSampleRate: 1 }); - defaults.useRelease({ revision: "none", blocks: {} }); + defaults.apply({}); random.mockReturnValue(0.06); defaults.error("e", {}); await defaults.flush(); @@ -1253,12 +1288,14 @@ describe("AR-60 the Telemetry block: defaults and caps", () => { // traceSampleRate defaults to 0 and is capped at limit 0. random.mockReturnValue(0); const traces = pipeline(); - traces.useRelease(withBlock({ traceSampleRate: 1 })); + traces.apply({ traceSampleRate: 1 }); expect(traces.sampleTrace()).toBe(false); }); it("is always in the schema", () => { - expect(readSource("src/v8/cli/schema/builtinSchemas.ts")).toMatch(/telemetry/); + expect(readSource("src/v8/cli/schema/builtinSchemas.ts")).toMatch( + /"cms-settings"[\s\S]*telemetry/, + ); }); }); @@ -1343,33 +1380,32 @@ describe("AR-64 / AR-65 error codes and paths", () => { // --------------------------------------------------------------------------- describe("RD-02 / RD-04 the draft client works like the release client", () => { - it("forDraft passes the pointer to load(pointer); draftPointer picks the client", async () => { - const draft = docsSnapshot("draft-1"); - (draft.blocks.SummerSEO as { title: string }).title = "Draft!"; - const load = vi.fn(async (pointer?: string | null) => (pointer ? draft : docsSnapshot())); - const cms = createCMS({ blocks: docsBlocks(), content: { load } }); - const pick = (request: Request) => { - const pointer = draftPointer(request); + it("forDraft reads the draft the pointer names; draftPointer picks the client", async () => { + const { pointer } = studioDraft(); + const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); + const pick = async (request: Request) => { + const pointer = await cms.draftPointer(request); return pointer ? cms.forDraft(pointer) : cms.forRelease(); }; - const pointer = "api.deco.example/drafts/acme/main@9f3c1a"; - const [d] = await pick(new Request(`https://s/?__draft=${pointer}`)).resolve<{ title: string }>( + const link = `https://s/?__draft=${encodeURIComponent(pointer)}`; + const [d] = await (await pick(new Request(link))).resolve<{ + title: string; + }>("SummerSEO"); + expect(d?.title).toBe("Draft!"); + const [r] = await (await pick(new Request("https://s/"))).resolve<{ title: string }>( "SummerSEO", ); - expect(load).toHaveBeenCalledWith(pointer); - expect(d?.title).toBe("Draft!"); - const [r] = await pick(new Request("https://s/")).resolve<{ title: string }>("SummerSEO"); expect(r?.title).toBe("Sunny!"); }); it("RD-04 snippet typechecks", () => { const errors = typecheck({ "preview.ts": ` -import { createCMS, draftPointer, type Snapshot } from "@decocms/blocks"; +import { createCMS, type Snapshot } from "@decocms/blocks"; declare const content: Snapshot; declare const request: Request; const cms = createCMS({ blocks: {}, content }); -const pointer = draftPointer(request); +const pointer = await cms.draftPointer(request); const client = pointer ? cms.forDraft(pointer) : cms.forRelease(); export { client }; `, @@ -1602,13 +1638,24 @@ export { views }; expect(errors).toEqual([]); }, 60_000); - it("RN-06 the SDK never sees the request: no requestScope, no Request/signal in createCMS or Client", () => { + it("RN-06 resolution never sees the request: no requestScope, no Request/signal in createCMS or Client; only the draft helpers read one", () => { const r = root as Record; expect(Object.keys(r).filter((k) => /requestScope/i.test(k))).toEqual([]); expect(Object.keys(v8).filter((k) => /request/i.test(k))).toEqual([]); const types = readSource("src/v8/types.ts"); - const cmsAndClient = types.slice(types.indexOf("export interface Client")); - expect(cmsAndClient).not.toMatch(/Request|signal/); + const section = (start: string) => { + const from = types.indexOf(start); + return types.slice(from, types.indexOf("\n}\n", from)); + }; + expect(section("export interface CMSConfig")).not.toMatch(/Request|signal/); + expect(section("export interface Client")).not.toMatch(/Request|signal/); + const cmsMethods = section("export interface CMS {") + .split("\n") + .filter((line) => /^\s{2}\w+\(/.test(line) && /Request|signal/.test(line)); + expect(cmsMethods.map((line) => line.trim().split("(")[0])).toEqual([ + "draftPointer", + "draftCookie", + ]); }); }); diff --git a/packages/blocks/src/v8/__conformance__/syntax.cli.test.ts b/packages/blocks/src/v8/__conformance__/syntax.cli.test.ts index 76073539c..d7c8c17d0 100644 --- a/packages/blocks/src/v8/__conformance__/syntax.cli.test.ts +++ b/packages/blocks/src/v8/__conformance__/syntax.cli.test.ts @@ -20,7 +20,7 @@ import { startServer } from "../cli/serve/server"; const here = path.dirname(fileURLToPath(import.meta.url)); const PACKAGE = path.resolve(here, "../../.."); -const TEN = [ +const BUILT_INS = [ "lazy", "multivariate", "always", @@ -28,8 +28,7 @@ const TEN = [ "date", "page", "redirect", - "telemetry", - "analytics", + "cms-settings", "secret", ]; @@ -213,13 +212,13 @@ describe("the .deco folder", () => { }); describe("deco schema", () => { - it("blocks-13/builtin-01: all ten built-ins are in the schema, even with an empty block map", async () => { + it("blocks-13/builtin-01: all nine built-ins are in the schema, even with an empty block map", async () => { const empty = createFixture({ ".deco/index.ts": "export default {};\n" }); try { const { code, out } = await deco(["schema"], empty.root); expect(code, out).toBe(0); const emptyMeta = JSON.parse(empty.read(".deco/schema.gen.json")); - for (const name of TEN) { + for (const name of BUILT_INS) { expect(emptyMeta.schema.definitions[toBase64(name)], name).toBeDefined(); } } finally { @@ -306,7 +305,7 @@ describe("deco check", () => { }); it("builtin-26: no saved block can take a built-in's name", async () => { - for (const name of TEN) { + for (const name of BUILT_INS) { const { code, out } = await checkWith({ [name]: { __resolveType: "always" } }); expect(code, name).toBe(1); expect(out).toContain(`saved block "${name}" has the name of a built-in block`); @@ -412,9 +411,7 @@ describe("deco serve", () => { const server = await startServer({ cwd: site.root, port: 0, - token: "t", reporter: recorder(), - env: {}, }); try { const body: any = await new Promise((resolve, reject) => { @@ -426,7 +423,6 @@ describe("deco serve", () => { path: "/rpc", headers: { host: `127.0.0.1:${server.port}`, - authorization: "Bearer t", "content-type": "application/json", }, }, diff --git a/packages/blocks/src/v8/__conformance__/syntax.test.ts b/packages/blocks/src/v8/__conformance__/syntax.test.ts index 5900c5ed7..1d2c885e4 100644 --- a/packages/blocks/src/v8/__conformance__/syntax.test.ts +++ b/packages/blocks/src/v8/__conformance__/syntax.test.ts @@ -49,7 +49,7 @@ import { } from "./syntaxExamples"; const here = path.dirname(fileURLToPath(import.meta.url)); -const TEN = [ +const BUILT_INS = [ "lazy", "multivariate", "always", @@ -57,8 +57,7 @@ const TEN = [ "date", "page", "redirect", - "telemetry", - "analytics", + "cms-settings", "secret", ]; @@ -270,10 +269,10 @@ describe("blocks", () => { expect(error?.code).toBe("UNKNOWN_BLOCK"); }); - it("blocks-13/builtin-01: exactly ten built-ins, always in the registry, no UNKNOWN_BLOCK with an empty map", async () => { - expect(Object.keys(builtIns).sort()).toEqual([...TEN].sort()); + it("blocks-13/builtin-01: exactly nine built-ins, always in the registry, no UNKNOWN_BLOCK with an empty map", async () => { + expect(Object.keys(builtIns).sort()).toEqual([...BUILT_INS].sort()); const client = clientWith({}); - for (const name of TEN) { + for (const name of BUILT_INS) { const [, error] = await client.resolve({ __resolveType: name }); expect(error?.code, name).not.toBe("UNKNOWN_BLOCK"); } @@ -711,17 +710,13 @@ describe("built-in blocks", () => { ]); }); - it("builtin-08: telemetry returns its arguments as saved", async () => { - const args = { enabled: true, metrics: false, errorSampleRate: 0.5, traceSampleRate: 0 }; - expect(await clientWith({}).resolve({ __resolveType: "telemetry", ...args })).toEqual([ - args, - null, - ]); - }); - - it("builtin-09: analytics fills its defaults (the hosted collector, enabled)", async () => { - expect(await clientWith({}).resolve({ __resolveType: "analytics" })).toEqual([ - { collector: HOSTED_ANALYTICS_COLLECTOR, enabled: true }, + it("builtin-08/builtin-09: cms-settings returns its input with the telemetry and analytics defaults filled in", async () => { + const telemetry = { enabled: true, metrics: false, errorSampleRate: 0.5, traceSampleRate: 0 }; + const preview = { hosts: ["staging.example.com"] }; + expect( + await clientWith({}).resolve({ __resolveType: "cms-settings", preview, telemetry }), + ).toEqual([ + { preview, telemetry, analytics: { collector: HOSTED_ANALYTICS_COLLECTOR, enabled: true } }, null, ]); expect(HOSTED_ANALYTICS_COLLECTOR).toMatch(/^https:\/\//); @@ -976,7 +971,10 @@ describe("secrets", () => { content: { revision: "r", blocks: { - Telemetry: { __resolveType: "telemetry", traceSampleRate: 1, errorSampleRate: 1 }, + CMS: { + __resolveType: "cms-settings", + telemetry: { traceSampleRate: 1, errorSampleRate: 1 }, + }, }, }, secrets: { key: priv }, diff --git a/packages/blocks/src/v8/analytics.test.ts b/packages/blocks/src/v8/analytics.test.ts index d4941312b..caf72c951 100644 --- a/packages/blocks/src/v8/analytics.test.ts +++ b/packages/blocks/src/v8/analytics.test.ts @@ -7,7 +7,7 @@ import type { ReactElement } from "react"; import { renderToStaticMarkup } from "react-dom/server"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { AnalyticsScript, track } from "./analytics"; -import { analytics, HOSTED_ANALYTICS_COLLECTOR } from "./builtins/data"; +import { analyticsSection as analytics, HOSTED_ANALYTICS_COLLECTOR } from "./builtins/data"; const COLLECTOR = "https://stats.example.com/events"; diff --git a/packages/blocks/src/v8/boundedJson.ts b/packages/blocks/src/v8/boundedJson.ts new file mode 100644 index 000000000..8ac4606c2 --- /dev/null +++ b/packages/blocks/src/v8/boundedJson.ts @@ -0,0 +1,47 @@ +/** + * Network reads the SDK makes on its own (hosted releases, draft changes): + * a timeout on the whole request, and a JSON body refused as soon as it + * would exceed a size, before it's buffered whole (a Worker isolate has + * 128 MB). + */ + +/** An abort signal that fires after `ms`, where the runtime has `AbortSignal.timeout`. */ +export function timeoutSignal(ms: number): AbortSignal | undefined { + return typeof AbortSignal !== "undefined" && typeof AbortSignal.timeout === "function" + ? AbortSignal.timeout(ms) + : undefined; +} + +/** Parses a JSON body, refusing it once it passes `maxBytes`. */ +export async function readBoundedJson( + response: Response, + label: string, + maxBytes: number, +): Promise { + const tooLarge = () => new Error(`${label}: larger than ${maxBytes} bytes`); + if (Number(response.headers.get("content-length")) > maxBytes) { + await response.body?.cancel(); + throw tooLarge(); + } + if (response.body === null) { + const text = await response.text(); + if (text.length > maxBytes) throw tooLarge(); + return JSON.parse(text); + } + const reader = response.body.getReader(); + const decoder = new TextDecoder(); + const parts: string[] = []; + let size = 0; + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + size += value.byteLength; + if (size > maxBytes) { + await reader.cancel(); + throw tooLarge(); + } + parts.push(decoder.decode(value, { stream: true })); + } + parts.push(decoder.decode()); + return JSON.parse(parts.join("")); +} diff --git a/packages/blocks/src/v8/builtins.test.ts b/packages/blocks/src/v8/builtins.test.ts index 22b224955..88a99e1b4 100644 --- a/packages/blocks/src/v8/builtins.test.ts +++ b/packages/blocks/src/v8/builtins.test.ts @@ -22,11 +22,11 @@ function client(blocks: Blocks = docsBlocks(), content: Snapshot = docsSnapshot( const lazyValue = (value: unknown) => ({ __resolveType: "lazy", value }); describe("the built-in list", () => { - it("has the ten documented names", () => { + it("has the nine documented names", () => { expect(Object.keys(builtIns).sort()).toEqual( [ "always", - "analytics", + "cms-settings", "date", "lazy", "multivariate", @@ -34,7 +34,6 @@ describe("the built-in list", () => { "page", "redirect", "secret", - "telemetry", ].sort(), ); }); @@ -43,8 +42,7 @@ describe("the built-in list", () => { for (const name of [ "page", "redirect", - "telemetry", - "analytics", + "cms-settings", "always", "never", "date", @@ -367,7 +365,7 @@ describe("page", () => { }); }); -describe("redirect, telemetry, analytics", () => { +describe("redirect, cms-settings", () => { it("redirect returns its arguments as saved, optional fields included", async () => { const saved = { from: "/old/:slug", @@ -386,30 +384,34 @@ describe("redirect, telemetry, analytics", () => { ]); }); - it("telemetry returns its arguments as saved", async () => { - const settings = { enabled: false, metrics: true, errorSampleRate: 0.05, traceSampleRate: 0 }; - expect(await client({}).resolve({ __resolveType: "telemetry", ...settings })).toEqual([ - settings, + it("cms-settings fills in the telemetry and analytics defaults and keeps preview as saved", async () => { + expect(await client({}).resolve({ __resolveType: "cms-settings" })).toEqual([ + { + telemetry: { enabled: true, metrics: true, errorSampleRate: 0.05, traceSampleRate: 0 }, + analytics: { collector: HOSTED_ANALYTICS_COLLECTOR, enabled: true }, + }, null, ]); - expect(await client({}).resolve({ __resolveType: "telemetry" })).toEqual([{}, null]); - }); - - it("analytics fills in the defaults: the hosted collector and enabled", async () => { - expect(await client({}).resolve({ __resolveType: "analytics" })).toEqual([ - { collector: HOSTED_ANALYTICS_COLLECTOR, enabled: true }, + const saved = { + preview: { hosts: ["staging.example.com"] }, + telemetry: { enabled: false, errorSampleRate: 0.5 }, + analytics: { collector: "https://stats.example.com/events", enabled: false }, + }; + expect(await client({}).resolve({ __resolveType: "cms-settings", ...saved })).toEqual([ + { + preview: { hosts: ["staging.example.com"] }, + telemetry: { enabled: false, metrics: true, errorSampleRate: 0.5, traceSampleRate: 0 }, + analytics: { collector: "https://stats.example.com/events", enabled: false }, + }, null, ]); }); - it("analytics keeps a collector and enabled: false", async () => { - expect( - await client({}).resolve({ - __resolveType: "analytics", - collector: "https://stats.example.com/events", - enabled: false, - }), - ).toEqual([{ collector: "https://stats.example.com/events", enabled: false }, null]); + it("telemetry and analytics are no longer built-ins (folded into cms-settings, no alias)", async () => { + for (const type of ["telemetry", "analytics"]) { + const [, error] = await client({}).resolve({ __resolveType: type }); + expect(error?.code, type).toBe("UNKNOWN_BLOCK"); + } }); it("the hosted collector is an https URL", () => { diff --git a/packages/blocks/src/v8/builtins/data.ts b/packages/blocks/src/v8/builtins/data.ts index 3082f4f5f..86c57e809 100644 --- a/packages/blocks/src/v8/builtins/data.ts +++ b/packages/blocks/src/v8/builtins/data.ts @@ -1,4 +1,4 @@ -import type { Analytics, Redirect, Telemetry } from "../types.ts"; +import type { Analytics, CMSSettings, Redirect, Telemetry } from "../types.ts"; /** The hosted Deco CMS collector: where page views go when `collector` is left out. */ export const HOSTED_ANALYTICS_COLLECTOR = "https://d.lilstts.com/events"; @@ -12,12 +12,61 @@ export const page =

(props: P): P => props; /** `redirect`: returns its arguments as saved. */ export const redirect = (props: Redirect): Redirect => props; -/** `telemetry`: the type of the well-known `Telemetry` saved block; returns its arguments. */ -export const telemetry = (props: Telemetry = {}): Telemetry => props; +/** The `telemetry` section's defaults (sample rates before code's caps). */ +export const TELEMETRY_DEFAULTS: Readonly> = Object.freeze({ + enabled: true, + metrics: true, + errorSampleRate: 0.05, + traceSampleRate: 0, +}); + +/** The well-known saved block that holds the CMS settings, and its built-in type. */ +export const SETTINGS_BLOCK = "CMS"; +export const SETTINGS_TYPE = "cms-settings"; + +/** The `telemetry` section with its defaults filled in; rates as saved, before any cap. */ +export function telemetrySection(section: unknown): Required { + const t = isRecord(section) ? section : {}; + return { + enabled: typeof t.enabled === "boolean" ? t.enabled : TELEMETRY_DEFAULTS.enabled, + metrics: typeof t.metrics === "boolean" ? t.metrics : TELEMETRY_DEFAULTS.metrics, + errorSampleRate: rate(t.errorSampleRate, TELEMETRY_DEFAULTS.errorSampleRate), + traceSampleRate: rate(t.traceSampleRate, TELEMETRY_DEFAULTS.traceSampleRate), + }; +} + +/** The `analytics` section with its defaults filled in. */ +export function analyticsSection(section: unknown): Required { + const a = isRecord(section) ? section : {}; + return { + collector: + typeof a.collector === "string" && a.collector !== "" + ? a.collector + : HOSTED_ANALYTICS_COLLECTOR, + enabled: typeof a.enabled === "boolean" ? a.enabled : true, + }; +} -/** `analytics`: returns its input with the defaults filled in. */ -export const analytics = (props: Analytics = {}): Required => ({ +/** + * `cms-settings`: the type of the well-known saved block `CMS`. Returns its + * input with the telemetry and analytics defaults filled in; `preview` stays + * as saved, since its default depends on code (`cms.settings()` applies it). + */ +export const cmsSettings = ( + props: CMSSettings = {}, +): CMSSettings & { telemetry: Required; analytics: Required } => ({ ...props, - collector: props.collector ?? HOSTED_ANALYTICS_COLLECTOR, - enabled: props.enabled ?? true, + telemetry: telemetrySection(props.telemetry), + analytics: analyticsSection(props.analytics), }); + +/** A sample rate between 0 and 1, or `fallback` when it isn't a number. */ +export function rate(value: unknown, fallback: number): number { + return typeof value === "number" && Number.isFinite(value) + ? Math.min(1, Math.max(0, value)) + : fallback; +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} diff --git a/packages/blocks/src/v8/builtins/index.ts b/packages/blocks/src/v8/builtins/index.ts index 4841945c3..188c76eb2 100644 --- a/packages/blocks/src/v8/builtins/index.ts +++ b/packages/blocks/src/v8/builtins/index.ts @@ -1,5 +1,5 @@ import type { BlockFunction } from "../types.ts"; -import { analytics, page, redirect, telemetry } from "./data.ts"; +import { cmsSettings, page, redirect, SETTINGS_TYPE } from "./data.ts"; import { lazy } from "./lazy.ts"; import { always, date, never } from "./matchers.ts"; import { multivariate } from "./multivariate.ts"; @@ -17,8 +17,7 @@ export const builtIns: Readonly> = Object.freeze({ date, page, redirect, - telemetry, - analytics, + [SETTINGS_TYPE]: cmsSettings, secret: secretBlock(), }); diff --git a/packages/blocks/src/v8/canonical.ts b/packages/blocks/src/v8/canonical.ts index a3a5bddef..2135758f8 100644 --- a/packages/blocks/src/v8/canonical.ts +++ b/packages/blocks/src/v8/canonical.ts @@ -104,14 +104,3 @@ export const CONTENT_HASH_FORMAT = 1; export async function computeContentRevision(blocks: Record): Promise { return sha256Hex(canonicalJson(blocks)); } - -/** The domain prefix of `blocks.apply` request digests, so they never equal another hash. */ -export const APPLY_DIGEST_DOMAIN = "deco-content/blocks.apply@1\n"; - -/** - * The digest a request key is bound to: SHA-256 over the domain prefix and - * the canonical JSON of every supplied parameter. - */ -export function applyRequestDigest(params: object): Promise { - return sha256Hex(APPLY_DIGEST_DOMAIN + canonicalJson(params)); -} diff --git a/packages/blocks/src/v8/cli/check/check.test.ts b/packages/blocks/src/v8/cli/check/check.test.ts index 428a2c4a8..7b05c1bd7 100644 --- a/packages/blocks/src/v8/cli/check/check.test.ts +++ b/packages/blocks/src/v8/cli/check/check.test.ts @@ -671,6 +671,29 @@ describe("deco check", () => { } }); + it('requires "blocksMajor": 8 at the top of the schema, with a guided message', () => { + expect(meta.blocksMajor).toBe(8); + const fixture = createFixture({ ".deco/schema.gen.json": meta }); + try { + const { blocksMajor: _, ...missing } = meta; + for (const [schema, found] of [ + [missing, 'no top-level "blocksMajor"'], + [{ ...meta, blocksMajor: 7 }, '"blocksMajor": 7'], + [{ ...meta, blocksMajor: "8" }, '"blocksMajor": "8"'], + [{ ...meta, blocksMajor: null }, '"blocksMajor": null'], + ] as const) { + fixture.write(".deco/schema.gen.json", schema); + expect(() => check({ cwd: fixture.root, reporter: recorder() })).toThrow( + new RegExp(`${found}; expected "blocksMajor": 8\\. .*run deco schema to regenerate it`), + ); + } + fixture.write(".deco/schema.gen.json", meta); + expect(check({ cwd: fixture.root, reporter: recorder() })).toBe(0); + } finally { + fixture.remove(); + } + }); + it("formats warnings with a prefix", () => { expect( formatProblems([ @@ -679,3 +702,63 @@ describe("deco check", () => { ).toBe(".deco/blocks/A.json\n warning: x: m"); }); }); + +describe("the CMS settings block", () => { + it("passes a CMS block with every section, variants included", () => { + expect( + run({ + CMS: { + __resolveType: "cms-settings", + preview: { hosts: ["staging.example.com", "*.preview.example.com"] }, + telemetry: { enabled: true, metrics: true, errorSampleRate: 0.05, traceSampleRate: 0 }, + analytics: { + __resolveType: "multivariate", + variants: [ + { + rule: { __resolveType: "always" }, + value: { __resolveType: "lazy", value: { enabled: false } }, + }, + ], + }, + }, + }), + ).toEqual([]); + }); + + it("reports out-of-range rates and unknown fields", () => { + expect( + lines( + run({ + CMS: { __resolveType: "cms-settings", telemetry: { errorSampleRate: 2 }, nope: true }, + }), + ), + ).toEqual(expect.arrayContaining([expect.stringContaining("telemetry.errorSampleRate")])); + }); + + it("a leftover telemetry or analytics block fails, pointing at the CMS block", () => { + const out = lines( + run({ + Telemetry: { __resolveType: "telemetry", enabled: false }, + Analytics: { __resolveType: "analytics" }, + }), + ); + expect(out).toHaveLength(2); + for (const line of out) { + expect(line).toMatch( + /unknown block type "(telemetry|analytics)": the built-in is now the \w+ section of the "CMS" block/, + ); + } + }); + + it("warns about a CMS block of another type, which is ignored as settings", () => { + expect(lines(run({ CMS: hero }))).toEqual([ + 'CMS.json warning "CMS" is the name of the CMS settings block (type "cms-settings"); this one is ignored as settings, so rename it', + ]); + }); + + it("cms-settings is a reserved name", () => { + expect(lines(run({ "cms-settings": { __resolveType: "cms-settings" } }))).toEqual([ + 'cms-settings.json saved block "cms-settings" has the name of a built-in block', + ]); + }); +}); diff --git a/packages/blocks/src/v8/cli/check/index.ts b/packages/blocks/src/v8/cli/check/index.ts index 69cbc80a0..c63b63ec0 100644 --- a/packages/blocks/src/v8/cli/check/index.ts +++ b/packages/blocks/src/v8/cli/check/index.ts @@ -20,6 +20,8 @@ import fs from "node:fs"; import { Ajv, type ValidateFunction } from "ajv"; import { isWellFormedCiphertext } from "../../../protocol/secrets.ts"; +import { BLOCKS_MAJOR } from "../../../protocol/types.ts"; +import { SETTINGS_BLOCK, SETTINGS_TYPE } from "../../builtins/data.ts"; import { isBlock, own } from "../../json.ts"; import { findRouteConflicts } from "../../matchRoute.ts"; import { isBuiltIn, storesPlainVariants } from "../builtins.ts"; @@ -70,6 +72,12 @@ function blockDefinitionName(schema: Json): string | null { return e && e.length === 1 && typeof schema.title === "string" && !schema.format ? e[0] : null; } +/** Prerelease built-ins folded into the CMS settings block, with no alias. */ +const FOLDED_BUILT_INS: Record = { + telemetry: `the built-in is now the telemetry section of the "${SETTINGS_BLOCK}" block (type "${SETTINGS_TYPE}"); see /next/renames-and-migrations#site-settings`, + analytics: `the built-in is now the analytics section of the "${SETTINGS_BLOCK}" block (type "${SETTINGS_TYPE}"); see /next/renames-and-migrations#site-settings`, +}; + // --------------------------------------------------------------------------- // The validation document: the schema, with every field also accepting a block // --------------------------------------------------------------------------- @@ -481,7 +489,13 @@ class Checker { if (type === null) { if (!Object.hasOwn(this.saved.blocks, name)) { - this.report(ctx, path, `unknown block type "${name}"`); + this.report( + ctx, + path, + Object.hasOwn(FOLDED_BUILT_INS, name) + ? `unknown block type "${name}": ${FOLDED_BUILT_INS[name]}` + : `unknown block type "${name}"`, + ); return; } // A reference to a saved block: check what it resolves to against the field. @@ -650,6 +664,15 @@ class Checker { this.report(ctx, "", "missing __resolveType: a saved block names the function it calls"); continue; } + if (name === SETTINGS_BLOCK && entry.__resolveType !== SETTINGS_TYPE) { + this.report( + ctx, + "", + `"${SETTINGS_BLOCK}" is the name of the CMS settings block (type "${SETTINGS_TYPE}"); ` + + "this one is ignored as settings, so rename it", + "warning", + ); + } this.checkBlock(ctx, entry, "", null); } problems.push(...this.routeProblems()); @@ -745,11 +768,34 @@ function readSchemaFile(file: string): DecoMeta { } catch { throw new CliError(`no ${file}; run deco schema first`); } + let meta: DecoMeta; try { - return JSON.parse(text) as DecoMeta; + meta = JSON.parse(text) as DecoMeta; } catch (error) { throw new CliError(`${file}: invalid JSON: ${(error as Error).message}`); } + assertBlocksMajor(meta, file); + return meta; +} + +/** + * A schema must declare `"blocksMajor": 8`: the site editor treats a site as + * v8 only on that field, so a schema without it would edit as a v7 site. + */ +export function assertBlocksMajor(meta: unknown, file: string): void { + const found = + typeof meta === "object" && meta !== null && !Array.isArray(meta) + ? (meta as { blocksMajor?: unknown }).blocksMajor + : undefined; + if (found === BLOCKS_MAJOR) return; + const what = + found === undefined + ? 'has no top-level "blocksMajor"' + : `has "blocksMajor": ${JSON.stringify(found)}`; + throw new CliError( + `${file} ${what}; expected "blocksMajor": ${BLOCKS_MAJOR}. ` + + "It was written by an older or different deco schema: run deco schema to regenerate it, then commit it.", + ); } /** `deco check`. Returns the exit code: 0 when everything fits (warnings aside), 1 otherwise. */ diff --git a/packages/blocks/src/v8/cli/index.ts b/packages/blocks/src/v8/cli/index.ts index 3d8116d15..4aff9150a 100644 --- a/packages/blocks/src/v8/cli/index.ts +++ b/packages/blocks/src/v8/cli/index.ts @@ -6,8 +6,9 @@ * * The four commands and their options, plus what the one-time migration * (the `deco-v7-to-v8-migration` skill) reads: the built-in block names, the alias - * table and the saved-blocks reader. + * table, the saved-blocks reader and the host-pattern parser. */ +export { formatHostPattern, type HostPattern, parseHostPattern } from "../hosts.ts"; export { BUILT_IN_BLOCKS, LEGACY_ALIASES } from "./builtins.ts"; export { type CheckOptions, check } from "./check/index.ts"; export { type ContentOptions, content, readSavedBlocks, type SavedBlocks } from "./content.ts"; diff --git a/packages/blocks/src/v8/cli/run.test.ts b/packages/blocks/src/v8/cli/run.test.ts index 42b80ec73..6ef485ae3 100644 --- a/packages/blocks/src/v8/cli/run.test.ts +++ b/packages/blocks/src/v8/cli/run.test.ts @@ -20,15 +20,11 @@ describe("flags", () => { root: "apps/a", watch: true, }); - expect( - parseFlags("serve", [ - "--port=5000", - "--allow-origin", - "http://a", - "--allow-origin=http://b", - "--read-only", - ]), - ).toEqual({ port: "5000", "allow-origin": ["http://a", "http://b"], "read-only": true }); + expect(parseFlags("serve", ["--port=5000", "--host", "0.0.0.0", "--read-only"])).toEqual({ + port: "5000", + host: "0.0.0.0", + "read-only": true, + }); }); it.each([ @@ -167,7 +163,7 @@ describe("runCli", () => { stop = resolve; }); const out = recorder(); - const done = runCli(["serve", "--port", "0", "--token", "t"], { + const done = runCli(["serve", "--port", "0"], { cwd: fixture.root, reporter: out, until, diff --git a/packages/blocks/src/v8/cli/run.ts b/packages/blocks/src/v8/cli/run.ts index 18ff61c9e..70ab15ec2 100644 --- a/packages/blocks/src/v8/cli/run.ts +++ b/packages/blocks/src/v8/cli/run.ts @@ -10,7 +10,7 @@ import { reportSchema, schema, writeSchema } from "./schema/index.ts"; import { serve, startServer } from "./serve/server.ts"; import { watchFiles, watchTree } from "./watch.ts"; -type FlagKind = "string" | "boolean" | "list"; +type FlagKind = "string" | "boolean"; const COMMANDS: Record }> = { schema: { @@ -24,14 +24,12 @@ const COMMANDS: Record check: { usage: "deco check [--root

]", flags: { root: "string" } }, serve: { usage: - "deco serve [--root ] [--port ] [--host ] [--preview ] [--token ] [--allow-origin ]… [--assets ] [--read-only]", + "deco serve [--root ] [--port ] [--host ] [--preview ] [--assets ] [--read-only]", flags: { root: "string", port: "string", host: "string", preview: "string", - token: "string", - "allow-origin": "list", assets: "string", "read-only": "boolean", }, @@ -44,7 +42,7 @@ Commands: schema turn the types in .deco/index.ts into .deco/schema.gen.json content turn .deco/blocks into the content module, .deco/blocks.gen.ts check check that every saved block fits the schema (writes nothing) - serve serve .deco to the site editor on this machine + serve serve .deco to the site editor on this machine, at http://localhost:4545/rpc ${Object.values(COMMANDS) .map((c) => ` ${c.usage}`) @@ -78,8 +76,7 @@ export function parseFlags(command: string, args: string[]): ParsedFlags { throw new CliError(`--${name} needs a value`); i++; } - if (kind === "list") flags[name] = [...((flags[name] as string[]) ?? []), value]; - else flags[name] = value; + flags[name] = value; } return flags; } @@ -213,8 +210,6 @@ export async function runCli(argv: string[], options: RunOptions = {}): Promise< port, host: flags.host as string | undefined, preview: flags.preview as string | undefined, - token: flags.token as string | undefined, - allowOrigins: flags["allow-origin"] as string[] | undefined, assets: flags.assets as string | undefined, readOnly: Boolean(flags["read-only"]), }; diff --git a/packages/blocks/src/v8/cli/schema/builtinSchemas.ts b/packages/blocks/src/v8/cli/schema/builtinSchemas.ts index 2274cd08c..c8341088b 100644 --- a/packages/blocks/src/v8/cli/schema/builtinSchemas.ts +++ b/packages/blocks/src/v8/cli/schema/builtinSchemas.ts @@ -1,5 +1,5 @@ /** - * The forms of the ten built-in blocks (spec: built-in-blocks, and the types + * The forms of the nine built-in blocks (spec: built-in-blocks, and the types * in api-reference › Types). They're always in the schema, unless the block * map declares the same key, which replaces the built-in. */ @@ -15,8 +15,7 @@ export const BUILT_IN_GROUPS: Record = { date: "matchers", page: "pages", redirect: "redirects", - telemetry: "content", - analytics: "content", + "cms-settings": "content", secret: "loaders", }; @@ -36,8 +35,11 @@ const BUILT_IN_DOCS: Record { root, fitting(type) { const target = nonNullable(type); - if (target.isAny() || target.isUnknown()) return []; + if (target.isAny() || target.isUnknown() || isFreeFormMap(target)) return []; const cached = fitCache.get(target.compilerType); if (cached) return cached; const keys = candidates @@ -423,6 +426,7 @@ export async function generateSchema(paths: DecoPaths): Promise { const meta: DecoMeta = { major: 1, + blocksMajor: BLOCKS_MAJOR, version: packageVersion(), namespace: "site", site: siteName(paths.root), diff --git a/packages/blocks/src/v8/cli/schema/schema.test.ts b/packages/blocks/src/v8/cli/schema/schema.test.ts index 04210ee13..7959e7b95 100644 --- a/packages/blocks/src/v8/cli/schema/schema.test.ts +++ b/packages/blocks/src/v8/cli/schema/schema.test.ts @@ -79,7 +79,7 @@ describe("groups", () => { expect(meta.manifest.blocks.pages.post).toBeDefined(); expect(meta.manifest.blocks.pages.page).toBeDefined(); expect(meta.manifest.blocks.content.menu).toBeDefined(); - expect(meta.manifest.blocks.content.telemetry).toBeDefined(); + expect(meta.manifest.blocks.content["cms-settings"]).toBeDefined(); }); it("puts every other function in loaders, multivariate and lazy included", () => { @@ -282,7 +282,7 @@ describe("interchangeable blocks", () => { }); describe("built-ins and aliases", () => { - it("adds the ten built-ins", () => { + it("adds the nine built-ins", () => { for (const name of [ "lazy", "multivariate", @@ -291,8 +291,7 @@ describe("built-ins and aliases", () => { "date", "page", "redirect", - "telemetry", - "analytics", + "cms-settings", "secret", ]) { expect(def(name), name).toBeDefined(); @@ -387,6 +386,85 @@ export const page = (props: StorePage) => props;`, }, 30_000); }); +describe("v7 field fidelity", () => { + it("keeps literal selects in declared order, maps @format datetime, and keeps free-form maps plain", async () => { + const fixture = createFixture({ + "src/card.ts": ` +// Literals the checker meets first get lower ids; a union of them would +// otherwise come out in that order instead of the order written below. +export type Earlier = "page" | "product"; +export type Kind = "search" | "collection" | "page" | "product"; +export interface Badge { label: string } +export interface Props { + earlier?: Earlier; + kind: "search" | "collection" | "page" | "product"; + aliased?: Kind; + /** @format datetime */ + endsAt?: string; + entities?: Record; +} +export const card = (props: Props) => ({ component: "card", props }); +export const badge = (): Badge => ({ label: "" });`, + ".deco/index.ts": `import { badge, card } from "../src/card"; export default { badge, card };`, + }); + try { + const { meta: m } = await generateSchema(decoPaths(fixture.root)); + const p = m.schema.definitions[`${b64("card")}@Props`]?.properties; + expect(p, Object.keys(m.schema.definitions).join(" ")).toBeDefined(); + expect(p.kind.enum).toEqual(["search", "collection", "page", "product"]); + expect(p.aliased.enum).toEqual(["search", "collection", "page", "product"]); + expect(p.endsAt.format).toBe("date-time"); + expect(p.entities).toEqual({ + type: "object", + additionalProperties: {}, + nullable: true, + title: "Entities", + }); + } finally { + fixture.remove(); + } + }, 30_000); +}); + +describe("@format on a type alias", () => { + it("gives every field of an imported alias its format: plain, optional, nullable, listed, a select", async () => { + const fixture = createFixture({ + "src/widgets.ts": ` +/** @format color */ +export type Paint = string; +/** @format color */ +export type TextTone = "black" | "white"; +export type Plain = string;`, + "src/banner.ts": ` +import type { Paint, Plain, TextTone } from "./widgets"; +export interface Props { + background: Paint; + border?: Paint | null; + stripes?: Paint[]; + titleTone?: TextTone; + /** @format textarea */ + note?: Paint; + plain?: Plain; +} +export const banner = (props: Props) => ({ component: "banner", props });`, + ".deco/index.ts": `import { banner } from "../src/banner"; export default { banner };`, + }); + try { + const { meta: m } = await generateSchema(decoPaths(fixture.root)); + const p = m.schema.definitions[`${b64("banner")}@Props`]?.properties; + expect(p, Object.keys(m.schema.definitions).join(" ")).toBeDefined(); + expect(p.background).toEqual({ type: "string", format: "color", title: "Background" }); + expect(p.border).toMatchObject({ type: "string", format: "color", nullable: true }); + expect(p.stripes.items).toEqual({ type: "string", format: "color" }); + expect(p.titleTone).toMatchObject({ enum: ["black", "white"], format: "color" }); + expect(p.note.format).toBe("textarea"); + expect(p.plain.format).toBeUndefined(); + } finally { + fixture.remove(); + } + }, 30_000); +}); + describe("imported maps", () => { it("follows a spread app map, the later key winning", async () => { const fixture = createFixture({ diff --git a/packages/blocks/src/v8/cli/schema/tsProgram.ts b/packages/blocks/src/v8/cli/schema/tsProgram.ts index 511582ade..97c3f7138 100644 --- a/packages/blocks/src/v8/cli/schema/tsProgram.ts +++ b/packages/blocks/src/v8/cli/schema/tsProgram.ts @@ -321,6 +321,50 @@ export class TsNode { return t && typeof t === "object" && "kind" in t ? this.ctx.node(t) : undefined; } + /** + * The declarations of the type aliases a type annotation names, as written: + * `Kind` and `Kind | null` both give `Kind`'s. The checker drops the alias + * from `Kind | undefined`, so this is how a field finds its alias. + */ + getAliasDeclarationTexts(): string[] { + const ts = this.ctx.ts; + const node = this.compilerNode as TS.TypeNode; + const members = ts.isUnionTypeNode(node) ? node.types : [node]; + const texts: string[] = []; + for (const member of members) { + const decl = this.ctx.checker.getTypeFromTypeNode(member).aliasSymbol?.declarations?.[0]; + if (decl) texts.push(decl.getText(decl.getSourceFile())); + } + return texts; + } + + /** + * The `@format` on the type alias this annotation names: `Color`, + * `Color | null` and `Color[]` (then `items` is set) all give `Color`'s. + * Read from the alias's declaration, since the checker drops an alias of a + * primitive (`type Color = string` is just `string`). + */ + getAliasFormat(): { format: string; items: boolean } | undefined { + const ts = this.ctx.ts; + const checker = this.ctx.checker; + const node = this.compilerNode as TS.TypeNode; + for (const member of ts.isUnionTypeNode(node) ? node.types : [node]) { + const items = ts.isArrayTypeNode(member); + const target = items ? member.elementType : member; + if (!ts.isTypeReferenceNode(target)) continue; + let symbol = checker.getSymbolAtLocation(target.typeName); + if (symbol && symbol.flags & ts.SymbolFlags.Alias) symbol = checker.getAliasedSymbol(symbol); + const decl = symbol?.declarations?.find(ts.isTypeAliasDeclaration); + if (!decl) continue; + for (const tag of ts.getJSDocTags(decl)) { + if (tag.tagName.text !== "format") continue; + const format = ts.getTextOfJSDocComment(tag.comment)?.trim(); + if (format) return { format, items }; + } + } + return undefined; + } + /** Whether `other` is this node or inside it. */ contains(other: TsNode): boolean { const a = this.compilerNode; diff --git a/packages/blocks/src/v8/cli/schema/typeToSchema.ts b/packages/blocks/src/v8/cli/schema/typeToSchema.ts index 812ab57d7..9b76a9c6f 100644 --- a/packages/blocks/src/v8/cli/schema/typeToSchema.ts +++ b/packages/blocks/src/v8/cli/schema/typeToSchema.ts @@ -123,9 +123,20 @@ function parseOptions(raw: string): { values: (string | number)[]; labels?: stri return labelled ? { values, labels } : { values }; } +/** + * `@format` spellings that name a standard format under another name, kept + * from v7: `@format datetime` (Eitri's vocabulary) is the `date-time` picker, + * and `deco check` validates it as one. + */ +const FORMAT_ALIASES: Record = { datetime: "date-time" }; + function applyJsDocToSchema(schema: any, tags: Record): void { for (const [tag, value] of Object.entries(tags)) { if (tag === "ignore") continue; + if (tag === "format") { + schema.format = FORMAT_ALIASES[value] ?? value; + continue; + } if (tag === "hide") { schema.hide = "true"; continue; @@ -220,6 +231,20 @@ function applyWidgetFormat(schema: any, typeHint: string): void { } } +/** + * `@format` written on a type alias (`/** @format color *\/ type Color = string`) + * reaches every field of that type, a literal select included; a field's own + * `@format` still wins, since its tags apply afterwards. + */ +function applyAliasFormat( + schema: any, + alias: { format: string; items: boolean } | undefined, +): void { + if (!alias) return; + const target = alias.items ? schema.items : schema; + if (target?.type === "string") target.format ??= FORMAT_ALIASES[alias.format] ?? alias.format; +} + // --------------------------------------------------------------------------- // Recognizing the framework's own types // --------------------------------------------------------------------------- @@ -324,6 +349,19 @@ export function nonNullable(type: Type): Type { return parts.length === 1 ? parts[0] : type.getNonNullableType(); } +/** + * `Record` and friends: a map of anything, which every block's + * output fits. It's a free-form value, not a place for a block, so the field + * doesn't offer the saved-block picker or list every block as a choice. + */ +export function isFreeFormMap(type: Type): boolean { + const t = nonNullable(type); + if (!t.isObject() || t.getProperties().length > 0 || t.getCallSignatures().length > 0) + return false; + const value = t.getStringIndexType() ?? t.getNumberIndexType(); + return value !== undefined && (value.isAny() || value.isUnknown()); +} + /** Types a block can fill: objects, arrays and intersections of objects, not primitives. */ function isObjectLike(type: Type): boolean { const t = nonNullable(type); @@ -434,10 +472,51 @@ function fieldSchema( } const schema = typeToJsonSchema(propType, new Set(visited), ctx); + orderEnumByHint(schema, typeHint); applyWidgetFormat(schema, typeHint); return schema; } +/** + * Literal values in the order `text` (the declaration as written) spells them. + * The checker orders a union's members by when it first met each literal, + * which another file can change, so `"a" | "b"` could come out as `["b", "a"]`. + * Values `text` doesn't spell all of keep the checker's order. + */ +export function inDeclaredOrder(values: T[], text: string): T[] { + const at = values.map((value) => { + const spelled = + typeof value === "string" + ? [`"${value}"`, `'${value}'`, `\`${value}\``] + .map((q) => text.indexOf(q)) + .filter((i) => i >= 0) + : [ + ...text.matchAll( + new RegExp(`(? m.index ?? -1); + return spelled.length > 0 ? Math.min(...spelled) : -1; + }); + if (at.some((i) => i < 0)) return values; + return values + .map((value, i) => ({ value, at: at[i] })) + .sort((a, b) => a.at - b.at) + .map((e) => e.value); +} + +/** Reorder a literal select (or a list of them) by the field's annotation. */ +function orderEnumByHint(schema: any, typeHint: string): void { + if (Array.isArray(schema?.enum)) schema.enum = inDeclaredOrder(schema.enum, typeHint); + else if (Array.isArray(schema?.items?.enum)) { + schema.items.enum = inDeclaredOrder(schema.items.enum, typeHint); + } +} + +/** The declaration text of a type alias, for ordering its literal members. */ +function aliasDeclarationText(type: Type): string { + return type.getAliasSymbol()?.getDeclarations()[0]?.getText() ?? ""; +} + /** The "plain value" branch of a field that blocks can fill. */ function inlineSchema(type: Type, ctx: SchemaContext, visited: Set): any { if (type.isArray()) { @@ -514,12 +593,20 @@ export function typeToJsonSchema( } // String-literal unions and string enums: a select, written into the schema. if (nonNull.every((t) => t.isStringLiteral())) { - const result: any = { type: "string", enum: nonNull.map((t) => t.getLiteralValue()) }; + const values = nonNull.map((t) => t.getLiteralValue() as string); + const result: any = { + type: "string", + enum: inDeclaredOrder(values, aliasDeclarationText(type)), + }; if (isNullable) result.nullable = true; return result; } if (nonNull.every((t) => t.isNumberLiteral())) { - const result: any = { type: "number", enum: nonNull.map((t) => t.getLiteralValue()) }; + const values = nonNull.map((t) => t.getLiteralValue() as number); + const result: any = { + type: "number", + enum: inDeclaredOrder(values, aliasDeclarationText(type)), + }; if (isNullable) result.nullable = true; return result; } @@ -630,6 +717,10 @@ export function typeToJsonSchema( } const schema = fieldSchema(propType, typeHint, ctx, visited); + if (typeNode) { + orderEnumByHint(schema, typeNode.getAliasDeclarationTexts().join("\n")); + applyAliasFormat(schema, typeNode.getAliasFormat()); + } if (schema.anyOf && schema.anyOf[0]?.$ref === resolvableRef().$ref) { // A block-ref field: nullability lives on the wrapper. if (isNullableHint(optional, typeHint)) schema.nullable = true; diff --git a/packages/blocks/src/v8/cli/serve/serve.test.ts b/packages/blocks/src/v8/cli/serve/serve.test.ts index f8e6475e3..d377b5e90 100644 --- a/packages/blocks/src/v8/cli/serve/serve.test.ts +++ b/packages/blocks/src/v8/cli/serve/serve.test.ts @@ -36,7 +36,6 @@ afterEach(async () => { fixture?.remove(); }); -const TOKEN = "test-token"; const STUDIO = "https://studio.decocms.com"; const hero = { __resolveType: "hero", title: "Summer", size: "md" }; @@ -51,9 +50,7 @@ async function start( server = await startServer({ cwd: fixture.root, port: 0, - token: TOKEN, reporter: out, - env: {}, ...options, }); return server; @@ -69,17 +66,22 @@ interface Reply { function request( method: string, pathname: string, - { body, headers = {} }: { body?: string | Buffer; headers?: Record } = {}, + { + body, + headers = {}, + host = "127.0.0.1", + }: { body?: string | Buffer; headers?: Record; host?: string } = {}, ): Promise { const port = server!.port; + const hostHeader = host.includes(":") ? `[${host}]:${port}` : `${host}:${port}`; return new Promise((resolve, reject) => { const req = http.request( { - host: "127.0.0.1", + host, port, method, path: pathname, - headers: { host: `127.0.0.1:${port}`, ...headers }, + headers: { host: hostHeader, ...headers }, }, (res) => { const chunks: Buffer[] = []; @@ -103,7 +105,7 @@ function request( }); } -const auth = { authorization: `Bearer ${TOKEN}`, "content-type": "application/json" }; +const json = { "content-type": "application/json" }; let nextId = 1; async function rpc(method: string, params?: unknown, headers: Record = {}) { const reply = await request("POST", "/rpc", { @@ -113,7 +115,7 @@ async function rpc(method: string, params?: unknown, headers: Record { it("prints the address, root, assets, app and site editor link", async () => { await start({ preview: "http://localhost:3001" }); const text = out.text(); - expect(text).toContain(`Deco server http://127.0.0.1:${server!.port}/rpc`); + expect(text).toContain(`Deco server http://localhost:${server!.port}/rpc`); expect(text).toContain("Root . (.deco/schema.gen.json, 0 blocks)"); expect(text).toContain("Assets public/assets (PUT /assets/)"); expect(text).toContain("Preview http://localhost:3001"); expect(server!.siteEditorUrl).toBe( - `https://studio.decocms.com/site-editor#endpoint=${encodeURIComponent(server!.endpoint)}&token=${TOKEN}`, + `https://studio.decocms.com/site-editor#endpoint=${encodeURIComponent(server!.endpoint)}`, ); + expect(text).toContain(`Site editor ${server!.siteEditorUrl}`); }); - it("uses DECO_SERVE_TOKEN, else a random token per run", async () => { - await start({ token: undefined, env: { DECO_SERVE_TOKEN: "from-env" } }); - expect(server!.token).toBe("from-env"); - await server!.close(); - server = await startServer({ cwd: fixture.root, port: 0, reporter: out, env: {} }); - expect(server.token).toMatch(/^[A-Za-z0-9_-]{43}$/); + it("has no token: nothing about one in its output, and DECO_SERVE_TOKEN is ignored", async () => { + process.env.DECO_SERVE_TOKEN = "from-env"; + try { + await start(); + } finally { + delete process.env.DECO_SERVE_TOKEN; + } + expect("token" in server!).toBe(false); + expect(server!.siteEditorUrl).not.toMatch(/token/i); + expect(out.text()).not.toMatch(/token/i); + const reply = await request("POST", "/rpc", { + body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }), + headers: { ...json, authorization: "Bearer from-env" }, + }); + expect(reply.status).toBe(200); + expect(reply.body.result.protocol).toBe("deco-content"); }); it("warns when listening beyond loopback", async () => { await start({ host: "0.0.0.0" }); - expect( - out.lines.some((l) => l.level === "warn" && /other machines can reach/.test(l.message)), - ).toBe(true); + const warnings = out.lines.filter((l) => l.level === "warn"); + expect(warnings).toHaveLength(1); + expect(warnings[0].message).toMatch(/other machines on the network can reach this server/); + expect(warnings[0].message).not.toMatch(/token/i); + // The site editor connects through loopback, never the wildcard address. + expect(server!.siteEditorUrl).toBe( + `https://studio.decocms.com/site-editor#endpoint=${encodeURIComponent(`http://localhost:${server!.port}/rpc`)}`, + ); + }); + + it("listens on 127.0.0.1 and ::1 on one port, so localhost reaches it either way", async () => { + await start(); + const body = JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }); + const v4 = await request("POST", "/rpc", { body, headers: json }); + expect(v4.status).toBe(200); + let v6: Reply; + try { + v6 = await request("POST", "/rpc", { body, headers: json, host: "::1" }); + } catch (error) { + // No IPv6 on this machine: the server skipped ::1. + expect((error as NodeJS.ErrnoException).code).toMatch( + /EADDRNOTAVAIL|EAFNOSUPPORT|ENETUNREACH/, + ); + return; + } + expect(v6.status).toBe(200); + expect(v6.body.result.protocol).toBe("deco-content"); + }); + + it("listens only on --host when given", async () => { + await start({ host: "127.0.0.1" }); + expect(server!.endpoint).toBe(`http://127.0.0.1:${server!.port}/rpc`); + await expect( + request("POST", "/rpc", { body: "{}", headers: json, host: "::1" }), + ).rejects.toThrow(); + }); + + it("reads --host case-insensitively", async () => { + await start({ host: "LOCALHOST" }); + expect(out.lines.filter((l) => l.level === "warn")).toEqual([]); }); it("defaults the app to the Vite config's port, read as text, else 5173", async () => { @@ -167,66 +217,51 @@ describe("starting", () => { await expect(start({ preview: bad })).rejects.toThrow(/--preview must be a local address/); } }); - - it("refuses an empty token", async () => { - await expect(start({ token: "" })).rejects.toThrow(/--token can't be empty/); - await expect(start({ token: undefined, env: { DECO_SERVE_TOKEN: " " } })).rejects.toThrow( - /DECO_SERVE_TOKEN can't be empty/, - ); - }); }); -describe("the security checks", () => { - it("accepts only the site editor's origins and --allow-origin", async () => { - await start({ allowOrigins: ["http://localhost:8000"] }); - const evil = await request("POST", "/rpc", { - body: "{}", - headers: { ...auth, origin: "https://evil.example" }, - }); - expect(evil.status).toBe(403); - expect(evil.headers["access-control-allow-origin"]).toBeUndefined(); - for (const origin of [STUDIO, "http://localhost:8000"]) { - const ok = await request("POST", "/rpc", { - body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }), - headers: { ...auth, origin }, +describe("CORS", () => { + const describeCall = JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }); + + it("answers any origin, reflecting it, and logs nothing", async () => { + await start(); + for (const origin of [STUDIO, "http://localhost:4000", "https://example.com"]) { + const reply = await request("POST", "/rpc", { + body: describeCall, + headers: { ...json, origin }, }); - expect(ok.status).toBe(200); - expect(ok.headers["access-control-allow-origin"]).toBe(origin); + expect(reply.status, origin).toBe(200); + expect(reply.headers["access-control-allow-origin"]).toBe(origin); + expect(reply.headers.vary).toBe("Origin"); } + expect(out.lines.filter((l) => l.level === "warn")).toEqual([]); + }); + + it("accepts a request without Origin (a local process)", async () => { + await start(); + const reply = await request("POST", "/rpc", { body: describeCall, headers: json }); + expect(reply.status).toBe(200); + expect(reply.headers["access-control-allow-origin"]).toBeUndefined(); }); - it("answers the CORS and local-network preflight without a token", async () => { + it("answers a foreign origin's CORS and private-network preflight", async () => { await start(); const reply = await request("OPTIONS", "/rpc", { headers: { - origin: STUDIO, + origin: "https://example.com", "access-control-request-method": "POST", "access-control-request-private-network": "true", }, }); expect(reply.status).toBe(204); - expect(reply.headers["access-control-allow-origin"]).toBe(STUDIO); + expect(reply.headers["access-control-allow-origin"]).toBe("https://example.com"); expect(reply.headers["access-control-allow-private-network"]).toBe("true"); - expect(reply.headers["access-control-allow-headers"]).toContain("authorization"); - }); - - it("rejects any Host but its own address (DNS rebinding)", async () => { - await start(); - const reply = await request("POST", "/rpc", { - body: "{}", - headers: { ...auth, host: "evil.example" }, - }); - expect(reply.status).toBe(403); - const localhost = await request("POST", "/rpc", { - body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "describe" }), - headers: { ...auth, host: `localhost:${server!.port}` }, - }); - expect(localhost.status).toBe(200); + expect(reply.headers["access-control-allow-headers"]).toBe("content-type"); + expect(reply.headers.vary).toBe("Origin"); }); }); describe("the content protocol", () => { - it("describes the endpoint: working tree, root, limits, assets and the secrets key", async () => { + it("describes the endpoint: working tree, root, assets and the secrets key", async () => { const { publicKeyPem } = await sealSecret("x"); await start({ preview: "http://localhost:3000/" }, { ".deco/secrets.pub": publicKeyPem }); const { result } = await rpc("describe"); @@ -238,10 +273,7 @@ describe("the content protocol", () => { readOnly: false, root: ".", schemaFormat: "deco-meta@1", - refs: null, - writes: { idempotency: null, schemaPreconditions: true }, pollIntervalMs: 2000, - limits: { maxOpsPerApply: 500, maxBlockBytes: 1048576, maxRequestBytes: 8388608 }, preview: { url: "http://localhost:3000" }, assets: { dir: "public/assets", urlPrefix: "/assets/", maxBytes: 25 * 1024 * 1024 }, secrets: { publicKey: publicKeyPem }, @@ -257,9 +289,7 @@ describe("the content protocol", () => { cwd: fixture.root, root: "apps/storefront", port: 0, - token: TOKEN, reporter: out, - env: {}, }); const { result } = await rpc("describe"); expect(result.root).toBe("apps/storefront"); @@ -306,7 +336,7 @@ describe("uploads", () => { const put = (name: string, type = "image/png", body: Buffer = png) => request("PUT", `/assets/${name}`, { body, - headers: { authorization: `Bearer ${TOKEN}`, "content-type": type }, + headers: { "content-type": type }, }); it("writes the file into public/assets and answers with the path the field stores", async () => { @@ -358,7 +388,7 @@ describe("the content protocol's conformance suite, against deco serve", () => { beforeAll(async () => { ({ publicKeyPem } = await sealSecret("x")); site = createFixture({ ".deco/schema.gen.json": meta, ".deco/secrets.pub": publicKeyPem }); - const options = { cwd: site.root, port: 0, token: TOKEN, reporter: recorder(), env: {} }; + const options = { cwd: site.root, port: 0, reporter: recorder() }; running = await startServer(options); readOnly = await startServer({ ...options, readOnly: true }); }); @@ -371,7 +401,6 @@ describe("the content protocol's conformance suite, against deco serve", () => { const optionsFor = (server: () => RunningServer) => () => ({ endpoint: server().endpoint, assetsEndpoint: server().endpoint.replace(/\/rpc$/, "/assets/"), - token: TOKEN, secretField: { blockType: "hero", field: "apiKey" }, secretsPublicKey: publicKeyPem, }); diff --git a/packages/blocks/src/v8/cli/serve/server.ts b/packages/blocks/src/v8/cli/serve/server.ts index 04848c229..cf4daf15c 100644 --- a/packages/blocks/src/v8/cli/serve/server.ts +++ b/packages/blocks/src/v8/cli/serve/server.ts @@ -4,17 +4,19 @@ * * The content protocol is `@decocms/blocks/protocol`'s `createContentHandler` * (at `/rpc`) and `createAssetHandler` (at `/assets/`) over the - * filesystem storage; they own the bearer token, `Content-Type`, size limits - * and upload rules. This file is the Node HTTP layer and the checks that - * depend on where it runs: + * filesystem storage; they own `Content-Type`, size limits and upload rules. + * `deco serve` has no authentication and answers any origin: any website open + * in the browser can read and write the content through it, so it is meant + * to run only while editing. This file is the Node HTTP layer: * - * - Listens on 127.0.0.1 unless `--host` says otherwise (with a warning). - * - Browser requests are accepted only from the site editor's origins and - * `--allow-origin`; CORS and Chrome's local-network preflights are answered. - * - Any `Host` other than the server's own address is refused (DNS rebinding). + * - Listens on loopback, 127.0.0.1 and ::1 on one port, so + * `http://localhost:` reaches it whichever address the OS gives + * localhost; `--host` listens on that one address instead (with a warning + * beyond loopback: the network can reach it then). + * - CORS is answered for any `Origin` (reflected, with `Vary: Origin`), and + * so are Chrome's Private/Local Network Access preflights. * - Every save regenerates `.deco/blocks.gen.ts`. */ -import { randomBytes } from "node:crypto"; import fs from "node:fs"; import http from "node:http"; import type { AddressInfo } from "node:net"; @@ -30,18 +32,12 @@ import { readSavedBlocks, writeContent } from "../content.ts"; import { consoleReporter, type Reporter } from "../log.ts"; import { CliError, decoPaths, findDecoRoot, packageVersion } from "../root.ts"; -/** The site editor's origins: the browser origins allowed by default. */ -const STUDIO_ORIGINS = [ - "https://studio.decocms.com", - "https://admin.decocms.com", - "https://admin.deco.cx", -]; - /** Where the site editor link points. */ -const STUDIO_ORIGIN = STUDIO_ORIGINS[0]; +const STUDIO_ORIGIN = "https://studio.decocms.com"; const DEFAULT_PORT = 4545; -const DEFAULT_HOST = "127.0.0.1"; +/** Where it listens without `--host`; ::1 is skipped where IPv6 is unavailable. */ +const DEFAULT_HOSTS = ["127.0.0.1", "::1"]; const LOOPBACK = new Set(["127.0.0.1", "localhost", "::1", "[::1]"]); @@ -52,19 +48,15 @@ export interface ServeOptions { host?: string; /** The local app the site editor previews: `localhost:8001` or a full loopback URL. */ preview?: string; - token?: string; - allowOrigins?: string[]; /** The upload folder, relative to the folder that contains `.deco`. */ assets?: string; readOnly?: boolean; reporter?: Reporter; - env?: NodeJS.ProcessEnv; } export interface RunningServer { - /** The content protocol endpoint, `http://127.0.0.1:4545/rpc`. */ + /** The content protocol endpoint, `http://localhost:4545/rpc`. */ endpoint: string; - token: string; siteEditorUrl: string; port: number; close(): Promise; @@ -111,24 +103,11 @@ function previewUrl(input: string): string { /** Start the server; resolves once it listens. */ export async function startServer(options: ServeOptions = {}): Promise { const reporter = options.reporter ?? consoleReporter; - const env = options.env ?? process.env; const paths = decoPaths(findDecoRoot(options)); - const host = options.host ?? DEFAULT_HOST; + const host = options.host?.toLowerCase(); const requestedPort = options.port ?? DEFAULT_PORT; const readOnly = options.readOnly ?? false; - if (options.token !== undefined && options.token.trim() === "") { - throw new CliError("--token can't be empty: pass a token, or leave it out for a random one"); - } - if (options.token === undefined && env.DECO_SERVE_TOKEN?.trim() === "") { - throw new CliError( - "DECO_SERVE_TOKEN can't be empty: set a token, or unset it for a random one", - ); - } - const token = options.token ?? env.DECO_SERVE_TOKEN ?? randomBytes(32).toString("base64url"); const preview = previewUrl(options.preview ?? defaultPreviewUrl(paths.root)); - const allowedOrigins = new Set( - [...STUDIO_ORIGINS, ...(options.allowOrigins ?? [])].map((o) => o.replace(/\/+$/, "")), - ); if (!Number.isInteger(requestedPort) || requestedPort < 0 || requestedPort > 65535) { throw new CliError(`--port must be a port number, got ${requestedPort}`); @@ -161,21 +140,13 @@ export async function startServer(options: ServeOptions = {}): Promise reporter.warn(String((error as Error)?.message ?? error)), }); - const assets = createAssetHandler(storage, { token }); - - let port = requestedPort; - const allowedHosts = () => { - const hosts = new Set([`127.0.0.1:${port}`, `localhost:${port}`, `[::1]:${port}`]); - if (!LOOPBACK.has(host)) hosts.add(`${host.includes(":") ? `[${host}]` : host}:${port}`); - return hosts; - }; + const assets = createAssetHandler(storage); - const server = http.createServer(async (req, res) => { + const handle: http.RequestListener = async (req, res) => { const origin = req.headers.origin; const cors: Record = {}; const refuse = (status: number, code: number, message: string) => { @@ -184,21 +155,14 @@ export async function startServer(options: ServeOptions = {}): Promise = { "access-control-allow-methods": "POST, PUT, OPTIONS", - "access-control-allow-headers": "authorization, content-type", + "access-control-allow-headers": "content-type", "access-control-max-age": "600", }; // Chrome's Local Network Access / Private Network Access preflight. @@ -220,34 +184,30 @@ export async function startServer(options: ServeOptions = {}): Promise((resolve, reject) => { - server.once("error", (error: NodeJS.ErrnoException) => - reject( - error.code === "EADDRINUSE" - ? new CliError(`port ${requestedPort} is in use; pass --port`) - : error, - ), - ); - server.listen(requestedPort, host, () => resolve()); - }); - port = (server.address() as AddressInfo).port; + const servers = await listen(handle, host ? [host] : DEFAULT_HOSTS, requestedPort); + const port = (servers[0].address() as AddressInfo).port; - const displayHost = host.includes(":") ? `[${host}]` : host; + const displayHost = !host ? "localhost" : host.includes(":") ? `[${host}]` : host; const endpoint = `http://${displayHost}:${port}/rpc`; - const siteEditorUrl = `${STUDIO_ORIGIN}/site-editor#endpoint=${encodeURIComponent(endpoint)}&token=${encodeURIComponent(token)}`; + // The site editor connects only through loopback; a wildcard address + // listens there too, so its link uses localhost. + const wildcard = host === "0.0.0.0" || host === "::"; + const linkEndpoint = wildcard ? `http://localhost:${port}/rpc` : endpoint; + const siteEditorUrl = `${STUDIO_ORIGIN}/site-editor#endpoint=${encodeURIComponent(linkEndpoint)}`; - if (!LOOPBACK.has(host)) { + if (host && !LOOPBACK.has(host)) { reporter.warn( - `warning: listening on ${host}: other machines can reach this server (the token still applies)`, + `warning: listening on ${host}: other machines on the network can reach this server and read and write the content` + + (wildcard ? "" : "; the site editor connects only through localhost"), ); } const schemaFile = fs.existsSync(paths.schema) ? ".deco/schema.gen.json" : fs.existsSync(paths.legacySchema) ? ".deco/meta.gen.json" - : "no schema: run deco schema"; + : "no schema yet: run deco schema for typed forms"; const count = Object.keys(readSavedBlocks(paths.blocks).blocks).length; const description = await storage.describe(); const label = (name: string) => name.padEnd(21); @@ -263,17 +223,58 @@ export async function startServer(options: ServeOptions = {}): Promise - new Promise((resolve) => { - server.closeAllConnections?.(); - server.close(() => resolve()); - }), + close: () => Promise.all(servers.map(close)).then(() => undefined), }; } +/** + * One server per address, all on the port the first one gets. An address + * after the first that the machine doesn't have (::1 without IPv6) is + * skipped; with port 0, a port the next address can't take is tried again. + */ +async function listen( + handle: http.RequestListener, + hosts: string[], + requestedPort: number, +): Promise { + for (let attempt = 0; ; attempt++) { + const servers: http.Server[] = []; + let port = requestedPort; + try { + for (const [i, host] of hosts.entries()) { + const server = http.createServer(handle); + try { + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(port, host, () => resolve()); + }); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (i > 0 && (code === "EADDRNOTAVAIL" || code === "EAFNOSUPPORT")) continue; + throw error; + } + servers.push(server); + port = (server.address() as AddressInfo).port; + } + return servers; + } catch (error) { + await Promise.all(servers.map(close)); + if ((error as NodeJS.ErrnoException).code !== "EADDRINUSE") throw error; + if (requestedPort === 0 && attempt < 5) continue; + throw new CliError(`port ${port} is in use; pass --port`); + } + } +} + +function close(server: http.Server): Promise { + return new Promise((resolve) => { + server.closeAllConnections?.(); + server.close(() => resolve()); + }); +} + /** A Node request as a fetch `Request`, its body streamed. */ function toRequest(req: http.IncomingMessage): Request { const headers = new Headers(); diff --git a/packages/blocks/src/v8/cms.test.ts b/packages/blocks/src/v8/cms.test.ts index 0d3b32e80..f03e091c9 100644 --- a/packages/blocks/src/v8/cms.test.ts +++ b/packages/blocks/src/v8/cms.test.ts @@ -4,39 +4,40 @@ * clients, loaders, drafts, revisions, update(), and one instance per process. */ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { createCMS, resetForTests } from "./cms"; -import { docsBlocks, docsSnapshot } from "./testFixtures"; +import { createCMS, instanceOf, resetForTests } from "./cms"; +import { docsBlocks, docsSnapshot, fakeStudio } from "./testFixtures"; import type { Loader, Snapshot } from "./types"; -const POINTER = "api.deco.example/drafts/acme/main?token=abc@9f3c1a"; - beforeEach(() => resetForTests()); afterEach(() => { vi.useRealTimers(); vi.restoreAllMocks(); vi.unstubAllEnvs(); + vi.unstubAllGlobals(); }); -function draftSnapshot(): Snapshot { - return { - revision: "draft-9f3c1a", - blocks: { - SummerSEO: { __resolveType: "seo", title: "Draft title", description: "Draft" }, - }, - }; +const DRAFT_SEO = { __resolveType: "seo", title: "Draft title", description: "Draft" }; + +/** A fake Studio with one saved draft: its pointer, and the fetch the CMS makes. */ +function studioDraft() { + const studio = fakeStudio(); + const fetch = vi.fn(studio.fetch); + vi.stubGlobal("fetch", fetch); + const pointer = studio.draft({ set: { SummerSEO: DRAFT_SEO }, delete: ["HelloWorld"] }); + return { studio, fetch, pointer }; } -/** A loader serving the release and one draft, recording every call. */ -function draftLoader(overrides: Partial = {}) { - const calls: (string | null | undefined)[] = []; +/** A loader recording every load. */ +function countingLoader(overrides: Partial = {}) { + let loads = 0; const loader: Loader = { - async load(pointer) { - calls.push(pointer); - return pointer ? draftSnapshot() : docsSnapshot(); + async load() { + loads++; + return docsSnapshot(); }, ...overrides, }; - return { loader, calls }; + return { loader, loads: () => loads }; } describe("createCMS with the content module", () => { @@ -49,13 +50,16 @@ describe("createCMS with the content module", () => { expect(await client.revision()).toBe("rev-1"); }); - it("forDraft acts like forRelease: the content module has no drafts and ignores the pointer", async () => { + it("forDraft layers the draft's changes over the content module, with no site or token", async () => { + const { pointer } = studioDraft(); const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); - for (const pointer of [POINTER, "not a pointer", ""]) { - const [seo, error] = await cms.forDraft(pointer).resolve("SummerSEO"); - expect(error).toBeNull(); - expect(seo).toEqual({ title: "Sunny!", description: "Light layers for long days." }); - } + const draft = cms.forDraft(pointer); + expect(await draft.resolve("SummerSEO")).toEqual([ + { title: "Draft title", description: "Draft" }, + null, + ]); + expect((await draft.resolve("HelloWorld"))[1]?.code).toBe("NOT_FOUND"); + expect((await draft.resolve("HomePage"))[1]).toBeNull(); // inherited }); it("update() reports nothing new: a snapshot never changes", async () => { @@ -72,44 +76,26 @@ describe("createCMS with the content module", () => { }); describe("loaders", () => { - it("load() with no argument is the release", async () => { - const { loader, calls } = draftLoader(); + it("load() is the release; a draft is layered over what it returned", async () => { + const { pointer } = studioDraft(); + const { loader, loads } = countingLoader(); const cms = createCMS({ blocks: docsBlocks(), content: loader }); const [seo] = await cms.forRelease().resolve("SummerSEO"); expect(seo).toEqual({ title: "Sunny!", description: "Light layers for long days." }); - expect(calls).toEqual([undefined]); - }); - - it("forDraft passes the pointer to load(pointer) and reads the draft", async () => { - const { loader, calls } = draftLoader(); - const cms = createCMS({ blocks: docsBlocks(), content: loader }); - const client = cms.forDraft(POINTER); - const [seo, error] = await client.resolve("SummerSEO"); - expect(error).toBeNull(); - expect(seo).toEqual({ title: "Draft title", description: "Draft" }); - expect(calls).toEqual([POINTER]); - expect(await client.revision()).toBe("draft-9f3c1a"); - }); - - it("a draft client never mixes in published entries", async () => { - const { loader } = draftLoader(); - const cms = createCMS({ blocks: docsBlocks(), content: loader }); - const [, error] = await cms.forDraft(POINTER).resolve("HomePage"); - expect(error?.code).toBe("NOT_FOUND"); + const client = cms.forDraft(pointer); + expect((await client.resolve("SummerSEO"))[0]).toEqual({ + title: "Draft title", + description: "Draft", + }); + expect(await client.revision()).toBe("rev-1~9f3c1a"); + expect(loads()).toBe(2); // a loader without update() is asked per client }); it("a failing draft makes every call on that client return [null, LOADER_FAILED]", async () => { - const boom = new Error("403 from the draft host"); - const cms = createCMS({ - blocks: docsBlocks(), - content: { - async load(pointer) { - if (pointer) throw boom; - return docsSnapshot(); - }, - }, - }); - const client = cms.forDraft(POINTER); + const { studio, pointer } = studioDraft(); + studio.respond("summer-sale", () => new Response("down", { status: 502 })); + const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); + const client = cms.forDraft(pointer); const [v1, e1] = await client.resolve("SummerSEO"); const [v2, e2] = await client.list("page"); const [v3, e3] = await client.resolve({ title: "literal" }); @@ -119,7 +105,8 @@ describe("loaders", () => { [v3, e3], ] as const) { expect(value).toBeNull(); - expect(error).toMatchObject({ code: "LOADER_FAILED", cause: boom }); + expect(error).toMatchObject({ code: "LOADER_FAILED" }); + expect(String((error?.cause as Error)?.message)).toContain("HTTP 502"); } await expect(client.revision()).rejects.toMatchObject({ code: "LOADER_FAILED" }); // …and the release is unaffected. @@ -127,14 +114,14 @@ describe("loaders", () => { }); it("a pointer that doesn't parse is LOADER_FAILED, never a silent fallback to the release", async () => { - const { loader, calls } = draftLoader(); - const cms = createCMS({ blocks: docsBlocks(), content: loader }); + const { fetch } = studioDraft(); + const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); for (const bad of ["https://evil.example/x@1", "no-path@1", "host/path", "", "a b/x@1"]) { const [value, error] = await cms.forDraft(bad).resolve("SummerSEO"); expect(value, bad).toBeNull(); expect(error?.code, bad).toBe("LOADER_FAILED"); } - expect(calls).toEqual([]); + expect(fetch).not.toHaveBeenCalled(); }); it("a loader that returns something other than a snapshot is LOADER_FAILED", async () => { @@ -204,34 +191,39 @@ describe("loaders", () => { expect(update).toHaveBeenCalledTimes(1); }); - it("an update drops cached drafts, so a loader that ignores the pointer can't serve a stale release", async () => { + it("an update drops fetched drafts, and the next draft client follows the new release", async () => { + const { fetch, pointer } = studioDraft(); let revision = "r1"; - const load = vi.fn(async () => ({ revision, blocks: { Name: revision } })); const cms = createCMS({ blocks: {}, - content: { load, update: async () => ({ updated: true }) }, + content: { + load: async () => ({ revision, blocks: { Name: revision } }), + update: async () => ({ updated: true }), + }, }); - expect(await cms.forDraft(POINTER).resolve("Name")).toEqual(["r1", null]); + expect(await cms.forDraft(pointer).resolve("Name")).toEqual(["r1", null]); revision = "r2"; await cms.update(); - expect(await cms.forDraft(POINTER).resolve("Name")).toEqual(["r2", null]); + expect(await cms.forDraft(pointer).resolve("Name")).toEqual(["r2", null]); + expect(fetch).toHaveBeenCalledTimes(2); }); - it("an update that finds nothing new keeps cached drafts", async () => { - const { loader, calls } = draftLoader({ update: async () => ({ updated: false }) }); + it("an update that finds nothing new keeps fetched drafts", async () => { + const { fetch, pointer } = studioDraft(); + const { loader } = countingLoader({ update: async () => ({ updated: false }) }); const cms = createCMS({ blocks: docsBlocks(), content: loader }); - await cms.forDraft(POINTER).resolve("SummerSEO"); + await cms.forDraft(pointer).resolve("SummerSEO"); await cms.update(); - await cms.forDraft(POINTER).resolve("SummerSEO"); - expect(calls).toEqual([POINTER]); + await cms.forDraft(pointer).resolve("SummerSEO"); + expect(fetch).toHaveBeenCalledTimes(1); }); - it("drafts are cached per pointer (the version is immutable)", async () => { - const { loader, calls } = draftLoader(); - const cms = createCMS({ blocks: docsBlocks(), content: loader }); - await cms.forDraft(POINTER).resolve("SummerSEO"); - await cms.forDraft(POINTER).resolve("SummerSEO"); - expect(calls).toEqual([POINTER]); + it("drafts are fetched once per pointer within a minute", async () => { + const { fetch, pointer } = studioDraft(); + const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); + await cms.forDraft(pointer).resolve("SummerSEO"); + await cms.forDraft(pointer).resolve("SummerSEO"); + expect(fetch).toHaveBeenCalledTimes(1); }); }); @@ -273,10 +265,10 @@ describe("one revision per client", () => { }); it("forRevision never reaches a draft: a draft revision behaves like the release", async () => { - const { loader } = draftLoader(); - const cms = createCMS({ blocks: docsBlocks(), content: loader }); - expect(await cms.forDraft(POINTER).revision()).toBe("draft-9f3c1a"); - const client = cms.forRevision("draft-9f3c1a"); + const { pointer } = studioDraft(); + const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); + expect(await cms.forDraft(pointer).revision()).toBe("rev-1~9f3c1a"); + const client = cms.forRevision("rev-1~9f3c1a"); expect(await client.revision()).toBe("rev-1"); expect(await client.resolve("SummerSEO")).toEqual([ { title: "Sunny!", description: "Light layers for long days." }, @@ -295,8 +287,22 @@ describe("one revision per client", () => { }); describe("update() checks on an interval", () => { - /** Background work runs at an idle moment (here, a timer): give it a turn. */ - const idle = () => new Promise((resolve) => setTimeout(resolve, 20)); + // Background work goes through the host hook, deferred to a timer so a + // request never runs it; `idle` waits for exactly what was scheduled (a + // fixed sleep could return first on a loaded machine and read a count short). + const HOOK = Symbol.for("decocms.blocks.background"); + const g = globalThis as { [HOOK]?: (task: () => Promise) => void }; + let background: Promise[] = []; + beforeEach(() => { + background = []; + g[HOOK] = (task) => { + background.push(new Promise((resolve) => setTimeout(resolve, 0)).then(task)); + }; + }); + afterEach(() => { + delete g[HOOK]; + }); + const idle = () => Promise.all(background.splice(0)); /** A CMS over a loader with update(), with the clock and the jitter under test control. */ function scheduled(interval?: number, random = 0.5) { @@ -418,10 +424,30 @@ describe("update() checks on an interval", () => { }); describe("one instance per process", () => { + it("two calls on one content root share the content but each resolves with its own block map", async () => { + // Next runs the proxy bundle in the app's process: its createCMS must not replace the app's map. + const content = { revision: "r1", root: ".deco", blocks: { Hero: { __resolveType: "hero" } } }; + const app = createCMS({ blocks: { hero: () => "app" }, content }); + const proxy = createCMS({ blocks: { hero: () => "proxy" }, content }); + expect(instanceOf(proxy)).toBe(instanceOf(app)); + expect(await app.forRelease().resolve("Hero")).toEqual(["app", null]); + expect(await proxy.forRelease().resolve("Hero")).toEqual(["proxy", null]); + expect(await app.forRevision("r1").resolve("Hero")).toEqual(["app", null]); + expect(await app.forDraft("localhost:4547/@local").resolve("Hero")).toEqual(["app", null]); + // Still one store: an update through either is seen by both. + createCMS({ + blocks: { hero: () => "proxy" }, + content: { revision: "r2", root: ".deco", blocks: content.blocks }, + }); + expect(await app.forRelease().revision()).toBe("r2"); + }); + it("the same config returns the same instance", () => { const blocks = docsBlocks(); const content = docsSnapshot(); - expect(createCMS({ blocks, content })).toBe(createCMS({ blocks, content })); + expect(instanceOf(createCMS({ blocks, content }))).toBe( + instanceOf(createCMS({ blocks, content })), + ); }); it("is keyed by the content's identity, never its revision (a hot reload keeps the instance and serves the new content)", async () => { @@ -432,7 +458,7 @@ describe("one instance per process", () => { const next = { ...docsSnapshot("rev-2"), root: ".deco" }; (next.blocks.SummerSEO as Record).title = "Reloaded"; const second = createCMS({ blocks: docsBlocks(), content: next }); - expect(second).toBe(first); + expect(instanceOf(second)).toBe(instanceOf(first)); expect(await first.forRelease().revision()).toBe("rev-2"); expect((await first.forRelease().resolve<{ title: string }>("SummerSEO"))[0]?.title).toBe( "Reloaded", @@ -448,7 +474,7 @@ describe("one instance per process", () => { blocks: docsBlocks(), content: { ...docsSnapshot(), root: "sites/b/.deco" } as Snapshot, }); - expect(a).not.toBe(b); + expect(instanceOf(a)).not.toBe(instanceOf(b)); }); it("two content modules without a root never share an instance", async () => { @@ -457,10 +483,10 @@ describe("one instance per process", () => { const modB: Snapshot = { revision: "same", blocks: { Home: "B" } }; const a = createCMS({ blocks: {}, content: modA }); const b = createCMS({ blocks: {}, content: modB }); - expect(a).not.toBe(b); + expect(instanceOf(a)).not.toBe(instanceOf(b)); expect(await a.forRelease().resolve("Home")).toEqual(["A", null]); expect(await b.forRelease().resolve("Home")).toEqual(["B", null]); - expect(createCMS({ blocks: {}, content: modA })).toBe(a); + expect(instanceOf(createCMS({ blocks: {}, content: modA }))).toBe(instanceOf(a)); expect(warn).not.toHaveBeenCalled(); }); @@ -480,21 +506,29 @@ describe("one instance per process", () => { }); it("a loader you write is identified by the loader object", () => { - const one = draftLoader().loader; - const two = draftLoader().loader; - expect(createCMS({ blocks: {}, content: one })).toBe(createCMS({ blocks: {}, content: one })); - expect(createCMS({ blocks: {}, content: one })).not.toBe( - createCMS({ blocks: {}, content: two }), + const one = countingLoader().loader; + const two = countingLoader().loader; + expect(instanceOf(createCMS({ blocks: {}, content: one }))).toBe( + instanceOf(createCMS({ blocks: {}, content: one })), + ); + expect(instanceOf(createCMS({ blocks: {}, content: one }))).not.toBe( + instanceOf(createCMS({ blocks: {}, content: two })), ); }); it("with the hosted Deco CMS, the site ID and token are part of the key", () => { const content = docsSnapshot(); const a = createCMS({ blocks: {}, content, site: "acme", token: "t1" }); - expect(createCMS({ blocks: {}, content, site: "acme", token: "t1" })).toBe(a); - expect(createCMS({ blocks: {}, content, site: "other", token: "t1" })).not.toBe(a); - expect(createCMS({ blocks: {}, content, site: "acme", token: "t2" })).not.toBe(a); - expect(createCMS({ blocks: {}, content })).not.toBe(a); + expect(instanceOf(createCMS({ blocks: {}, content, site: "acme", token: "t1" }))).toBe( + instanceOf(a), + ); + expect(instanceOf(createCMS({ blocks: {}, content, site: "other", token: "t1" }))).not.toBe( + instanceOf(a), + ); + expect(instanceOf(createCMS({ blocks: {}, content, site: "acme", token: "t2" }))).not.toBe( + instanceOf(a), + ); + expect(instanceOf(createCMS({ blocks: {}, content }))).not.toBe(instanceOf(a)); }); it("never puts the token in the global symbol registry", () => { @@ -514,7 +548,7 @@ describe("one instance per process", () => { interval: 120_000, telemetry: { endpoint: "https://otel.example.com" }, }); - expect(second).toBe(first); + expect(instanceOf(second)).toBe(instanceOf(first)); expect(warn).toHaveBeenCalledTimes(1); expect(warn.mock.calls[0]?.[0]).toContain("interval"); expect(warn.mock.calls[0]?.[0]).toContain("telemetry"); @@ -545,14 +579,27 @@ describe("one instance per process", () => { Symbol.keyFor(s)?.startsWith("decocms.blocks.cms:"), ); expect(symbols).toHaveLength(1); - expect((globalThis as unknown as Record)[symbols[0]!]).toBe(cms); + expect((globalThis as unknown as Record)[symbols[0]!]).toBe(instanceOf(cms)); + }); + + it("an instance an older copy of the package stored (no handles) is replaced, not adopted", async () => { + const content = { revision: "r1", root: ".deco-old", blocks: { Home: "home" } }; + createCMS({ blocks: {}, content }); + const key = Object.getOwnPropertySymbols(globalThis).find((s) => + Symbol.keyFor(s)?.startsWith("decocms.blocks.cms:"), + )!; + const store = globalThis as unknown as Record; + store[key] = { adopt: () => {}, fingerprint: {} }; + const cms = createCMS({ blocks: {}, content }); + expect(await cms.forRelease().resolve("Home")).toEqual(["home", null]); + expect(store[key]).toBe(instanceOf(cms)); }); it("resetForTests clears every stored instance", () => { const content = docsSnapshot(); const first = createCMS({ blocks: {}, content }); resetForTests(); - expect(createCMS({ blocks: {}, content })).not.toBe(first); + expect(instanceOf(createCMS({ blocks: {}, content }))).not.toBe(instanceOf(first)); expect( Object.getOwnPropertySymbols(globalThis).filter((s) => Symbol.keyFor(s)?.startsWith("decocms.blocks.cms:"), diff --git a/packages/blocks/src/v8/cms.ts b/packages/blocks/src/v8/cms.ts index 8af087580..1dc381b1c 100644 --- a/packages/blocks/src/v8/cms.ts +++ b/packages/blocks/src/v8/cms.ts @@ -7,30 +7,62 @@ * cache. See /next/api-reference#one-instance-per-process. */ import { runInBackground } from "./background.ts"; +import { SETTINGS_BLOCK, SETTINGS_TYPE } from "./builtins/data.ts"; import { builtIns } from "./builtins/index.ts"; import { secretBlock } from "./builtins/secret.ts"; import { CMSClient } from "./client.ts"; import { ContentStore, isLoader, isSnapshot } from "./content.ts"; -import { parseDraftPointer } from "./draft.ts"; +import { draftCookieFor, endsPreview, parseDraftPointer, readDraftPointer } from "./draft.ts"; +import { allowsHost, type HostPattern, parseHostPattern } from "./hosts.ts"; import { clearGlobals, contentIdentity, fnv1a, readEnv } from "./identity.ts"; +import { isPlainObject } from "./json.ts"; import { remoteLoader, resetRemoteLoaders } from "./remoteLoader.ts"; -import { resolveDestination, setCurrentTelemetry, TelemetryPipeline } from "./telemetry.ts"; -import type { Blocks, Client, CMS, CMSConfig, Loader, Snapshot } from "./types.ts"; +import { + defaultSettings, + isStatic, + parseCodeHosts, + readSettings, + type SettingsCaps, +} from "./settings.ts"; +import { + resolveDestination, + setCurrentTelemetry, + TelemetryPipeline, + telemetryLimits, +} from "./telemetry.ts"; +import type { + Blocks, + Client, + CMS, + CMSConfig, + EffectiveSettings, + Loader, + RequestLike, + Snapshot, +} from "./types.ts"; import { forceVariants } from "./variants.ts"; const INSTANCE_PREFIX = "decocms.blocks.cms:"; const MIN_INTERVAL = 60_000; /** Each check runs one interval, plus or minus up to this much, after the previous one. */ const JITTER = 10_000; +/** How often telemetry reads its settings again when the release hasn't changed (date rules). */ +const TELEMETRY_SETTINGS_MS = 60_000; /** Options that must agree between two `createCMS` calls sharing an instance. */ interface Fingerprint { interval: number; telemetry: string; + preview: string; secrets: string; } -class CMSInstance implements CMS { +/** + * The state one content root shares across `createCMS` calls: the content + * store, caps, telemetry and secret key. Each call gets its own handle + * holding its own block map (see `CMSHandle`). + */ +class CMSInstance { readonly config: Readonly; readonly fingerprint: Fingerprint; readonly #store: ContentStore; @@ -39,19 +71,28 @@ class CMSInstance implements CMS { /** The built-ins with a `secret` that holds this instance's key (the first call's). */ readonly #builtIns: Readonly; readonly #telemetry: TelemetryPipeline | undefined; - #blocks: Blocks; + readonly #caps: SettingsCaps; + /** Settings of a `CMS` block with nothing to run, per release snapshot. */ + #staticSettings = new WeakMap(); + /** The release and time telemetry last read its section. */ + #telemetryRead: { snapshot: Snapshot; at: number } | undefined; + /** One handle per block map, so the same config returns the same object. */ + readonly #handles = new WeakMap(); /** - * When the next check is due. The first is on the first `forRelease()`, not + * When the next check is due. The first is on the first `forRelease()` or `forDraft()`, not * at construction: Workers read `Date.now()` as 0 at module scope. */ #nextCheck: number | undefined; constructor(config: CMSConfig, interval: number) { this.config = config; - this.#blocks = config.blocks; this.#interval = interval; this.#store = new ContentStore(contentOf(config)); this.fingerprint = fingerprintOf(config, interval); + this.#caps = { + hosts: parseCodeHosts(config.preview), + limits: telemetryLimits(config.telemetry), + }; const destination = resolveDestination(config.telemetry, config.site); if (destination !== null) { this.#telemetry = new TelemetryPipeline(destination); @@ -63,29 +104,57 @@ class CMSInstance implements CMS { this.#builtIns = Object.freeze({ ...builtIns, secret }); } - forRelease(): Client { + forRelease(blocks: Blocks): Client { this.#scheduleUpdate(); const telemetry = this.#telemetry; - if (telemetry === undefined) return this.#client(() => this.#store.release()); - return this.#client(() => - this.#store.release().then((snapshot) => { + if (telemetry === undefined) return this.#client(blocks, () => this.#store.release()); + return this.#client(blocks, () => + this.#store.release().then(async (snapshot) => { telemetry.useRelease(snapshot); + await this.#followTelemetrySettings(blocks, telemetry, snapshot); return snapshot; }), ); } - /** The draft, with the variants the pointer forces (even over a source with no drafts). */ - forDraft(pointer: string): Client { + /** + * The release's settings, from memory: never a draft, never a fetch. The + * `CMS` block resolves on each call, so a field with variants is decided + * where it's called; one with nothing to run is resolved once per release. + */ + settings(blocks: Blocks): Promise { + return this.#settingsOf(blocks, this.#store.current()); + } + + async draftPointer(blocks: Blocks, request: RequestLike): Promise { + const pointer = readDraftPointer(request); + if (pointer === null) return null; + return (await this.#previewAllowed(blocks, request)) ? pointer : null; + } + + async draftCookie(blocks: Blocks, request: RequestLike): Promise { + const cookie = draftCookieFor(request); + if (cookie === null) return null; + // Ending a preview is allowed anywhere, so a cookie set before a host was removed still goes. + if (endsPreview(request)) return cookie; + return (await this.#previewAllowed(blocks, request)) ? cookie : null; + } + + /** + * The draft, with the variants the pointer forces (even for a `local` + * pointer, which names no draft). A draft client is a production client + * reading a draft pointer, so it schedules the release check like any other. + */ + forDraft(blocks: Blocks, pointer: string): Client { + this.#scheduleUpdate(); + const load = () => this.#store.draft(pointer); const variants = parseDraftPointer(pointer)?.variants; - if (variants === undefined) return this.#client(() => this.#store.draft(pointer)); - return this.#client(() => - this.#store.draft(pointer).then((snapshot) => forceVariants(snapshot, variants)), - ); + if (variants === undefined) return this.#client(blocks, load); + return this.#client(blocks, () => load().then((snapshot) => forceVariants(snapshot, variants))); } - forRevision(revision: string): Client { - return this.#client(() => this.#store.revision(revision)); + forRevision(blocks: Blocks, revision: string): Client { + return this.#client(blocks, () => this.#store.revision(revision)); } update(): Promise<{ updated: boolean }> { @@ -93,16 +162,85 @@ class CMSInstance implements CMS { return this.#store.update(); } - /** A later `createCMS` call with the same key hands in its (possibly hot-reloaded) map and content. */ + /** The handle for a block map: this instance, resolving with that map. */ + handle(blocks: Blocks): CMS { + let handle = this.#handles.get(blocks); + if (handle === undefined) { + handle = new CMSHandle(this, blocks); + this.#handles.set(blocks, handle); + } + return handle; + } + + /** + * A later `createCMS` call with the same key hands in its (possibly + * hot-reloaded) content. Its block map stays with its own handle: another + * bundle in the same process (Next's proxy) must not replace the app's. + */ adopt(config: CMSConfig): void { - this.#blocks = config.blocks; this.#store.replace(contentOf(config)); + this.#staticSettings = new WeakMap(); + this.#telemetryRead = undefined; + } + + async #settingsOf(blocks: Blocks, snapshot: Snapshot | undefined): Promise { + if (snapshot === undefined) return defaultSettings(this.#caps); + const cached = this.#staticSettings.get(snapshot); + if (cached !== undefined) return cached; + const settings = await readSettings( + snapshot, + () => + new CMSClient({ + load: () => Promise.resolve(snapshot), + blocks, + builtIns: this.#builtIns, + onCollision: (name) => this.#warnCollision(name), + }), + this.#caps, + ); + // A block map's own cms-settings function may answer differently per call. + const entry = snapshot.blocks[SETTINGS_BLOCK]; + const fixed = !isPlainObject(entry) || isStatic(entry); + if (fixed && !Object.hasOwn(blocks, SETTINGS_TYPE)) { + this.#staticSettings.set(snapshot, settings); + } + return settings; + } + + /** Whether the release's preview hosts allow this request's host. */ + async #previewAllowed(blocks: Blocks, request: RequestLike): Promise { + const { hosts } = (await this.settings(blocks)).preview; + const patterns = hosts + .map((raw) => parseHostPattern(raw)) + .filter((pattern): pattern is HostPattern => pattern !== null); + return allowsHost(patterns, request.url); + } + + /** + * Telemetry follows the release's `telemetry` section, read outside any + * request's choices: when the release changes (before its first client + * runs) and then at most once a minute, so date rules take effect. + */ + async #followTelemetrySettings( + blocks: Blocks, + telemetry: TelemetryPipeline, + snapshot: Snapshot, + ): Promise { + const last = this.#telemetryRead; + const now = Date.now(); + if (last?.snapshot === snapshot && now - last.at < TELEMETRY_SETTINGS_MS) return; + this.#telemetryRead = { snapshot, at: now }; + const read = this.#settingsOf(blocks, snapshot).then((settings) => { + if (this.#telemetryRead?.snapshot === snapshot) telemetry.apply(settings.telemetry); + }); + // A new release waits for its rates; a periodic re-read doesn't hold up the request. + if (last?.snapshot !== snapshot) await read; } - #client(load: () => Promise): Client { + #client(blocks: Blocks, load: () => Promise): Client { return new CMSClient({ load, - blocks: this.#blocks, + blocks, builtIns: this.#builtIns, onCollision: (name) => this.#warnCollision(name), telemetry: this.#telemetry?.forClient(), @@ -134,9 +272,67 @@ class CMSInstance implements CMS { } } +/** `Symbol.for`, so a second copy of this module still reads it. */ +const INSTANCE = Symbol.for("decocms.blocks.cms.instance"); + +/** + * What `createCMS` returns: the shared instance plus the block map of this + * call. Every handle on one content root shares its content, caps, telemetry + * and secret key; each resolves with the block map it was created with. + */ +class CMSHandle implements CMS { + readonly #instance: CMSInstance; + readonly #blocks: Blocks; + + constructor(instance: CMSInstance, blocks: Blocks) { + this.#instance = instance; + this.#blocks = blocks; + } + + /** The shared instance (tests use it to tell whether two handles share one). */ + get [INSTANCE](): CMSInstance { + return this.#instance; + } + + forRelease(): Client { + return this.#instance.forRelease(this.#blocks); + } + + forDraft(pointer: string): Client { + return this.#instance.forDraft(this.#blocks, pointer); + } + + forRevision(revision: string): Client { + return this.#instance.forRevision(this.#blocks, revision); + } + + update(): Promise<{ updated: boolean }> { + return this.#instance.update(); + } + + settings(): Promise { + return this.#instance.settings(this.#blocks); + } + + draftPointer(request: RequestLike): Promise { + return this.#instance.draftPointer(this.#blocks, request); + } + + draftCookie(request: RequestLike): Promise { + return this.#instance.draftCookie(this.#blocks, request); + } +} + +/** The shared instance behind a `createCMS` result; `undefined` for anything else. */ +export function instanceOf(cms: CMS): object | undefined { + const instance = (cms as unknown as Record)[INSTANCE]; + return instance instanceof Object ? instance : undefined; +} + /** - * Creates the CMS, or returns the instance this configuration already has. - * Create it once, at module scope, and ask it for a client per request. + * Creates the CMS: a handle on the one instance this content root has in the + * process, holding this call's block map. Create it once, at module scope, and + * ask it for a client per request. */ export function createCMS(config: CMSConfig): CMS { validate(config); @@ -146,14 +342,20 @@ export function createCMS(config: CMSConfig): CMS { const key = Symbol.for(INSTANCE_PREFIX + identityOf(config)); const store = globalThis as unknown as Record; const existing = store[key]; - if (existing instanceof Object && typeof existing.adopt === "function") { + // An instance an older copy of this package stored (a dev reload across + // versions) has no handles: it's replaced rather than adopted. + if ( + existing instanceof Object && + typeof existing.adopt === "function" && + typeof existing.handle === "function" + ) { warnOnConflict(existing.fingerprint, fingerprintOf(config, interval)); existing.adopt(config); - return existing; + return existing.handle(config.blocks); } const instance = new CMSInstance(config, interval); store[key] = instance; - return instance; + return instance.handle(config.blocks); } /** Clears every stored CMS instance, so a test starts clean. */ @@ -163,7 +365,7 @@ export function resetForTests(): void { setCurrentTelemetry(undefined); } -/** With `site` and `token`, the content is the fallback of hosted releases and drafts. */ +/** With `site` and `token`, the content is the fallback of hosted releases. */ function contentOf(config: CMSConfig): Snapshot | Loader { if (!config.site || !config.token) return config.content; return remoteLoader(config.content, { site: config.site, token: config.token }); @@ -181,6 +383,7 @@ function validate(config: CMSConfig): void { "createCMS: `content` must be the content module ({ revision, blocks }) or a loader with load()", ); } + parseCodeHosts(config.preview); } function resolveInterval(configured: number | undefined): number { @@ -216,6 +419,7 @@ function fingerprintOf(config: CMSConfig, interval: number): Fingerprint { return { interval, telemetry: stableJson(config.telemetry ?? null), + preview: stableJson(config.preview ?? null), secrets: config.secrets?.key ? fnv1a(config.secrets.key) : "", }; } diff --git a/packages/blocks/src/v8/content.ts b/packages/blocks/src/v8/content.ts index 85630ccc5..3f367c992 100644 --- a/packages/blocks/src/v8/content.ts +++ b/packages/blocks/src/v8/content.ts @@ -3,17 +3,30 @@ * the caches every client shares, and the release revisions this process has * served. Clients only ever see whole `{ revision, blocks }` snapshots. * - * Drafts are never recorded as served: `forRevision(revision)` takes a - * revision string a client may hand back, so it must only ever reach published - * content, never a draft someone loaded with a pointer. + * A draft is the release with a draft's changes layered over it (see + * ./draftChanges.ts). Drafts are never recorded as served: + * `forRevision(revision)` takes a revision string a client may hand back, so + * it must only ever reach published content, never a draft someone loaded + * with a pointer. */ import { formatDraftPointer, parseDraftPointer } from "./draft.ts"; +import { type DraftChanges, fetchDraftChanges, LOCAL_VERSION, layerDraft } from "./draftChanges.ts"; import { errors, isResolutionError } from "./errors.ts"; import { isPlainObject } from "./json.ts"; import type { Loader, Snapshot } from "./types.ts"; const SERVED_REVISIONS = 16; -const CACHED_DRAFTS = 32; +/** Fetched drafts kept per CMS. */ +const CACHED_DRAFTS = 3; +/** How long a fetched draft is reused before it's fetched again, so its token is checked again. */ +const DRAFT_TTL_MS = 60_000; + +/** A fetched draft, and the views of it over each release it was layered on. */ +interface CachedDraft { + changes: Promise; + at: number; + views: WeakMap; +} export function isLoader(content: unknown): content is Loader { return ( @@ -23,6 +36,20 @@ export function isLoader(content: unknown): content is Loader { ); } +/** + * An internal hook a built-in loader (`remoteLoader`) has: the release it + * would return right now, from memory, or `undefined` when that takes a load. + * `cms.settings()` reads through it, so it never fetches. + */ +export const PEEK_RELEASE: unique symbol = Symbol.for("decocms.blocks.peekRelease"); + +/** The release a source holds in memory: the snapshot itself, or what a loader can peek. */ +export function peekRelease(source: Snapshot | Loader): Snapshot | undefined { + if (!isLoader(source)) return source; + const peek = (source as { [PEEK_RELEASE]?: () => Snapshot | undefined })[PEEK_RELEASE]; + return typeof peek === "function" ? peek.call(source) : undefined; +} + export function isSnapshot(content: unknown): content is Snapshot { return ( isPlainObject(content) && @@ -36,8 +63,10 @@ export class ContentStore { #source: Snapshot | Loader; #release: Promise | undefined; #updating: Promise<{ updated: boolean }> | undefined; - readonly #drafts = new BoundedMap>(CACHED_DRAFTS); + readonly #drafts = new BoundedMap(CACHED_DRAFTS); readonly #served = new BoundedMap(SERVED_REVISIONS); + /** The release this store last handed a client, for a loader that can't peek. */ + #latest: Snapshot | undefined; constructor(source: Snapshot | Loader) { this.#source = source; @@ -56,9 +85,9 @@ export class ContentStore { release(): Promise { const source = this.#source; if (!isLoader(source)) return Promise.resolve(this.#serve(source)); - if (!this.updatable) return this.#load(source, null).then((snapshot) => this.#serve(snapshot)); + if (!this.updatable) return this.#load(source).then((snapshot) => this.#serve(snapshot)); if (this.#release === undefined) { - const pending = this.#load(source, null).then((snapshot) => this.#serve(snapshot)); + const pending = this.#load(source).then((snapshot) => this.#serve(snapshot)); this.#release = pending; // A failed load isn't cached: the next client tries again. pending.catch(() => { @@ -69,31 +98,55 @@ export class ContentStore { } /** - * The draft a pointer names. A snapshot has no drafts and ignores the - * pointer. A loader gets `load(pointer)` only for a pointer that parses, - * formatted again without its `__variant` parameters (so every variant of - * one draft shares one load); anything else is `LOADER_FAILED`, never a - * silent fallback to the release. + * The draft a pointer names: its changes, fetched from a preview API domain, + * layered over the release. A pointer that doesn't parse, or any failure, is + * `LOADER_FAILED`, never a silent fallback to the release. A pointer whose + * version is `local` names no draft: the release, with nothing fetched. The + * changes are keyed by the pointer without its `__variant` parameters (every + * variant of one draft shares one fetch) and reused for a minute. */ draft(pointer: string): Promise { - const source = this.#source; - if (!isLoader(source)) return this.release(); const parsed = parseDraftPointer(pointer); if (parsed === null) { return Promise.reject(errors.loaderFailed(`invalid draft pointer "${truncate(pointer)}"`)); } - // The draft itself, without the variants a preview forces: those apply per client. - const key = formatDraftPointer({ - host: parsed.host, - path: parsed.path, - version: parsed.version, + if (parsed.version === LOCAL_VERSION) return this.release(); + const { host, path, version } = parsed; + const key = formatDraftPointer({ host, path, version }); + let entry = this.#drafts.get(key); + if (entry === undefined || Date.now() - entry.at >= DRAFT_TTL_MS) { + const fetched: CachedDraft = { + changes: fetchDraftChanges(parsed).catch((error: unknown) => { + throw errors.loaderFailed("the draft's changes couldn't be fetched", error); + }), + at: Date.now(), + views: new WeakMap(), + }; + this.#drafts.set(key, fetched); + // A failure isn't reused: the next client fetches again. + fetched.changes.catch(() => { + if (this.#drafts.get(key) === fetched) this.#drafts.delete(key); + }); + entry = fetched; + } + const { changes, views } = entry; + return Promise.all([this.release(), changes]).then(([base, draft]) => { + let view = views.get(base); + if (view === undefined) { + view = layerDraft(base, draft, version); + views.set(base, view); + } + return view; }); - const cached = this.#drafts.get(key); - if (cached !== undefined) return cached; - const pending = this.#load(source, key); - this.#drafts.set(key, pending); - pending.catch(() => this.#drafts.delete(key)); - return pending; + } + + /** + * The current release as it is in memory, without loading anything: the + * content module, what a built-in loader holds, or the release this store + * last loaded. `undefined` before a custom loader's first load. + */ + current(): Snapshot | undefined { + return peekRelease(this.#source) ?? this.#latest; } /** A release revision this store has served, or the release when it's unknown (drafts included). */ @@ -114,8 +167,9 @@ export class ContentStore { const updated = result?.updated === true; if (updated) { this.#release = undefined; - // A loader that ignores the pointer hands back the release as the - // draft; that copy is as stale as the release now. + // `#latest` stays: settings keep the release that was serving until + // the next one loads, never the defaults (which may allow every host). + // Drafts are fetched again, so a preview follows the release too. this.#drafts.clear(); } return { updated }; @@ -139,24 +193,24 @@ export class ContentStore { if (source === previous) { // The same loader (a hosted remoteLoader that adopted new fallback content): re-read it. this.#release = undefined; + this.#latest = undefined; + this.#drafts.clear(); return; } if (isSnapshot(previous)) this.#served.delete(previous.revision); this.#source = source; this.#release = undefined; + this.#latest = undefined; this.#drafts.clear(); } - async #load(loader: Loader, pointer: string | null): Promise { + async #load(loader: Loader): Promise { let snapshot: unknown; try { - snapshot = pointer === null ? await loader.load() : await loader.load(pointer); + snapshot = await loader.load(); } catch (error) { if (isResolutionError(error)) throw error; - throw errors.loaderFailed( - pointer === null ? "the content loader failed" : "the draft couldn't load", - error, - ); + throw errors.loaderFailed("the content loader failed", error); } if (!isSnapshot(snapshot)) { throw errors.loaderFailed("the content loader returned something other than a snapshot"); @@ -165,6 +219,7 @@ export class ContentStore { } #serve(snapshot: Snapshot): Snapshot { + this.#latest = snapshot; this.#served.set(snapshot.revision, snapshot); return snapshot; } @@ -174,29 +229,46 @@ function truncate(value: string): string { return value.length > 80 ? `${value.slice(0, 80)}…` : value; } -/** A Map that forgets its least recently set entry past `limit`. */ -class BoundedMap { +/** + * A Map that forgets its least recently used entries past `limit`, counted by + * `size` (one per entry unless given). + */ +export class BoundedMap { readonly #map = new Map(); - constructor(readonly limit: number) {} + #total = 0; + constructor( + readonly limit: number, + readonly size: (value: V) => number = () => 1, + ) {} get(key: K): V | undefined { - return this.#map.get(key); + const value = this.#map.get(key); + if (value !== undefined) { + this.#map.delete(key); + this.#map.set(key, value); + } + return value; } set(key: K, value: V): void { - this.#map.delete(key); + this.delete(key); this.#map.set(key, value); - if (this.#map.size > this.limit) { - const oldest = this.#map.keys().next(); - if (!oldest.done) this.#map.delete(oldest.value); + this.#total += this.size(value); + for (const [oldest, old] of this.#map) { + if (this.#total <= this.limit || oldest === key) break; + this.#map.delete(oldest); + this.#total -= this.size(old); } } delete(key: K): void { + if (!this.#map.has(key)) return; + this.#total -= this.size(this.#map.get(key) as V); this.#map.delete(key); } clear(): void { this.#map.clear(); + this.#total = 0; } } diff --git a/packages/blocks/src/v8/draft.test.ts b/packages/blocks/src/v8/draft.test.ts index 77a5f6b82..5557dda88 100644 --- a/packages/blocks/src/v8/draft.test.ts +++ b/packages/blocks/src/v8/draft.test.ts @@ -1,13 +1,15 @@ // @vitest-environment node /** Draft pointers (api-reference#draft-pointers, hosted-drafts.mdx). */ import { describe, expect, it } from "vitest"; -import { - DRAFT_COOKIE, - draftCookie, - draftPointer, - formatDraftPointer, - parseDraftPointer, -} from "./draft"; +import { createCMS } from "./cms"; +import { formatDraftPointer, parseDraftPointer } from "./draft"; +import type { RequestLike } from "./types"; + +/** The helpers as a site calls them: methods of a CMS with no settings, so every host may preview. */ +const cms = createCMS({ blocks: {}, content: { revision: "r", blocks: {} } }); +const draftPointer = (request: RequestLike) => cms.draftPointer(request); +const draftCookie = (request: RequestLike) => cms.draftCookie(request); +const DRAFT_COOKIE = "__deco_draft"; const POINTER = "api.deco.example/drafts/acme/main?token=abc.def-ghi@9f3c1a"; @@ -28,6 +30,15 @@ describe("parseDraftPointer", () => { }); }); + it("accepts a bracketed IPv6 host, with or without a port, in a URL's form", () => { + expect(parseDraftPointer("[::1]:4000/changes?token=t@v1")).toEqual({ + host: "[::1]:4000", + path: "/changes?token=t", + version: "v1", + }); + expect(parseDraftPointer("[0:0:0:0:0:0:0:1]/x@v1")?.host).toBe("[::1]"); + }); + it("lowercases the host and keeps path and version as they are", () => { expect(parseDraftPointer("API.Deco.Example/Drafts/X?T=Y@AbC_1.2-3")).toEqual({ host: "api.deco.example", @@ -64,6 +75,10 @@ describe("parseDraftPointer", () => { ["whitespace", "api.deco.example/drafts/x y@v1"], ["a fragment", "api.deco.example/drafts/x#frag@v1"], ["a double slash path", "api.deco.example//evil.example/x@v1"], + ["an unclosed IPv6 bracket", "[::1/x@v1"], + ["a name in brackets", "[evil.example]/x@v1"], + ["junk after an IPv6 bracket", "[::1]evil.example/x@v1"], + ["an IPv6 port out of range", "[::1]:70000/x@v1"], ["something absurdly long", `api.deco.example/${"a".repeat(5000)}@v1`], ]; for (const [label, raw] of invalid) { @@ -179,10 +194,10 @@ describe("forced variants (the __variant parameters)", () => { ); }); - it("draftCookie stores a pointer with forced variants", () => { + it("draftCookie stores a pointer with forced variants", async () => { const pointer = `localhost:4547/?__variant=${forced}@local`; expect( - draftCookie(request(`https://s.example/?__draft=${encodeURIComponent(pointer)}`)), + await draftCookie(request(`https://s.example/?__draft=${encodeURIComponent(pointer)}`)), ).toContain(`${DRAFT_COOKIE}=${encodeURIComponent(pointer)};`); }); }); @@ -194,72 +209,82 @@ function request(url: string, cookie?: string): Request { describe("draftPointer", () => { const encoded = encodeURIComponent(POINTER); - it("reads ?__draft= from the URL first", () => { - expect(draftPointer(request(`https://store.example.com/summer?__draft=${encoded}`))).toBe( + it("reads ?__draft= from the URL first", async () => { + expect(await draftPointer(request(`https://store.example.com/summer?__draft=${encoded}`))).toBe( POINTER, ); }); - it("prefers the URL over the cookie", () => { + it("prefers the URL over the cookie", async () => { const other = encodeURIComponent("api.deco.example/drafts/other@v2"); expect( - draftPointer( + await draftPointer( request(`https://store.example.com/?__draft=${encoded}`, `${DRAFT_COOKIE}=${other}`), ), ).toBe(POINTER); }); - it("then reads the deco-draft cookie", () => { + it("then reads the __deco_draft cookie", async () => { expect( - draftPointer( + await draftPointer( request("https://store.example.com/summer", `theme=dark; ${DRAFT_COOKIE}=${encoded}; x=1`), ), ).toBe(POINTER); }); - it("returns null when neither is present", () => { - expect(draftPointer(request("https://store.example.com/summer"))).toBeNull(); - expect(draftPointer(request("https://store.example.com/summer", "other=1"))).toBeNull(); + it("returns null when neither is present", async () => { + expect(await draftPointer(request("https://store.example.com/summer"))).toBeNull(); + expect(await draftPointer(request("https://store.example.com/summer", "other=1"))).toBeNull(); }); - it("returns null when the URL says ?__draft=off, even with a cookie", () => { + it("returns null when the URL says ?__draft=off, even with a cookie", async () => { expect( - draftPointer(request("https://store.example.com/?__draft=off", `${DRAFT_COOKIE}=${encoded}`)), + await draftPointer( + request("https://store.example.com/?__draft=off", `${DRAFT_COOKIE}=${encoded}`), + ), ).toBeNull(); }); - it("an empty ?__draft= falls back to the cookie", () => { + it("an empty ?__draft= falls back to the cookie", async () => { expect( - draftPointer(request("https://store.example.com/?__draft=", `${DRAFT_COOKIE}=${encoded}`)), + await draftPointer( + request("https://store.example.com/?__draft=", `${DRAFT_COOKIE}=${encoded}`), + ), ).toBe(POINTER); }); - it("returns the value as is: forDraft validates it", () => { - expect(draftPointer(request("https://store.example.com/?__draft=garbage"))).toBe("garbage"); + it("returns the value as is: forDraft validates it", async () => { + expect(await draftPointer(request("https://store.example.com/?__draft=garbage"))).toBe( + "garbage", + ); }); - it("takes anything with url and headers (a framework wrapper)", () => { + it("takes anything with url and headers (a framework wrapper)", async () => { expect( - draftPointer({ + await draftPointer({ url: `https://store.example.com/?__draft=${encoded}`, headers: new Headers(), }), ).toBe(POINTER); - expect(draftPointer({ url: "/relative?__draft=x", headers: new Headers() })).toBe("x"); + expect(await draftPointer({ url: "/relative?__draft=x", headers: new Headers() })).toBe("x"); }); - it("ignores a cookie whose name only ends with deco-draft", () => { - expect(draftPointer(request("https://s.example/", `not-deco-draft=${encoded}`))).toBeNull(); + it("ignores a cookie whose name only ends with __deco_draft", async () => { + expect( + await draftPointer(request("https://s.example/", `not__deco_draft=${encoded}`)), + ).toBeNull(); }); - it("returns null on an undecodable cookie", () => { - expect(draftPointer(request("https://s.example/", `${DRAFT_COOKIE}=%E0%A4%A`))).toBeNull(); + it("returns null on an undecodable cookie", async () => { + expect( + await draftPointer(request("https://s.example/", `${DRAFT_COOKIE}=%E0%A4%A`)), + ).toBeNull(); }); }); describe("draftCookie", () => { - it("stores the pointer when the URL carries ?__draft= (Secure; SameSite=None; Partitioned — D4)", () => { - const cookie = draftCookie( + it("stores the pointer when the URL carries ?__draft= (Secure; SameSite=None; Partitioned — D4)", async () => { + const cookie = await draftCookie( request(`https://store.example.com/summer?__draft=${encodeURIComponent(POINTER)}`), ); expect(cookie).toBe( @@ -267,41 +292,37 @@ describe("draftCookie", () => { ); }); - it("round-trips: the stored cookie is read back by draftPointer", () => { - const cookie = draftCookie( + it("round-trips: the stored cookie is read back by draftPointer", async () => { + const cookie = await draftCookie( request(`https://s.example/?__draft=${encodeURIComponent(POINTER)}`), - )!; - const pair = cookie.split(";")[0]!; - expect(draftPointer(request("https://s.example/next-page", pair))).toBe(POINTER); + ); + const pair = cookie!.split(";")[0]!; + expect(await draftPointer(request("https://s.example/next-page", pair))).toBe(POINTER); }); - it("expires the cookie on ?__draft=off (how the site editor ends a preview)", () => { - expect(draftCookie(request("https://store.example.com/?__draft=off"))).toBe( + it("expires the cookie on ?__draft=off (how the site editor ends a preview)", async () => { + expect(await draftCookie(request("https://store.example.com/?__draft=off"))).toBe( `${DRAFT_COOKIE}=; Path=/; Max-Age=0; HttpOnly; Secure; SameSite=None; Partitioned`, ); }); - it("is null on every other request", () => { - expect(draftCookie(request("https://store.example.com/summer"))).toBeNull(); + it("is null on every other request", async () => { + expect(await draftCookie(request("https://store.example.com/summer"))).toBeNull(); expect( - draftCookie( + await draftCookie( request( "https://store.example.com/summer", `${DRAFT_COOKIE}=${encodeURIComponent(POINTER)}`, ), ), ).toBeNull(); - expect(draftCookie(request("https://store.example.com/?__draft="))).toBeNull(); + expect(await draftCookie(request("https://store.example.com/?__draft="))).toBeNull(); }); - it("never stores a pointer that doesn't parse", () => { + it("never stores a pointer that doesn't parse", async () => { expect( - draftCookie(request("https://store.example.com/?__draft=https://evil.example/x@1")), + await draftCookie(request("https://store.example.com/?__draft=https://evil.example/x@1")), ).toBeNull(); - expect(draftCookie(request("https://store.example.com/?__draft=garbage"))).toBeNull(); - }); - - it("names the cookie DRAFT_COOKIE", () => { - expect(DRAFT_COOKIE).toBe("deco-draft"); + expect(await draftCookie(request("https://store.example.com/?__draft=garbage"))).toBeNull(); }); }); diff --git a/packages/blocks/src/v8/draft.ts b/packages/blocks/src/v8/draft.ts index 2746aca70..2f05b4e33 100644 --- a/packages/blocks/src/v8/draft.ts +++ b/packages/blocks/src/v8/draft.ts @@ -1,16 +1,18 @@ /** * Draft pointers (see /next/api-reference#draft-pointers): the string - * `@` that names a draft, and the two - * helpers that carry one from a `?__draft=` link into a cookie. + * `@` that names a draft, and what + * `cms.draftPointer` and `cms.draftCookie` read and write to carry one from a + * `?__draft=` link into a cookie. The host check against the preview hosts is + * the CMS's: these read the request only. * * The query's reserved `__variant` parameters are the variants a preview * forces (`@=`, URL-encoded, one per multivariate); the * parser lifts them out of `path` into `variants`. */ -import type { DraftPointer, ForcedVariant } from "./types.ts"; +import type { DraftPointer, ForcedVariant, RequestLike } from "./types.ts"; -/** The draft cookie's name, for frameworks whose cookie API has no Request (Next.js `cookies()`). */ -export const DRAFT_COOKIE = "deco-draft"; +/** The draft cookie's name. Not exported from the package: reading it directly would skip the host check. */ +const DRAFT_COOKIE = "__deco_draft"; const DRAFT_PARAM = "__draft"; const DRAFT_OFF = "off"; @@ -126,6 +128,7 @@ function plainVariant({ block, path, index }: ForcedVariant): ForcedVariant { } function normalizeHost(authority: string): string | null { + if (authority.startsWith("[")) return normalizeIPv6Host(authority); const parts = authority.toLowerCase().split(":"); if (parts.length > 2) return null; const [host, port] = parts; @@ -135,15 +138,29 @@ function normalizeHost(authority: string): string | null { return `${host}:${port}`; } -/** Anything with a `url` and `headers`: a fetch `Request`, a `NextRequest`, a framework wrapper. */ -export type RequestLike = Request | { url: string; headers: Headers }; +/** `[addr]` or `[addr]:port`, the address in the compressed, lowercase form a URL gives it. */ +function normalizeIPv6Host(authority: string): string | null { + const end = authority.indexOf("]"); + const address = authority.slice(0, end + 1); + const rest = authority.slice(end + 1); + if (end === -1 || !/^\[[0-9a-fA-F:.]+\]$/.test(address)) return null; + if (rest !== "" && (!rest.startsWith(":") || !PORT_RE.test(rest.slice(1)))) return null; + if (rest !== "" && Number(rest.slice(1)) > 65535) return null; + let host: string; + try { + host = new URL(`http://${address}/`).hostname; + } catch { + return null; + } + return `${host}${rest}`; +} /** * The draft pointer a request carries: `?__draft=` from the URL first, then - * the `deco-draft` cookie. `null` when neither is present, or when the URL + * the `__deco_draft` cookie. `null` when neither is present, or when the URL * says `?__draft=off`. The value is returned as is; `cms.forDraft` validates it. */ -export function draftPointer(request: RequestLike): string | null { +export function readDraftPointer(request: RequestLike): string | null { const param = draftParam(request); if (param === DRAFT_OFF) return null; if (param) return param; @@ -161,7 +178,7 @@ export function draftPointer(request: RequestLike): string | null { * is `Secure; SameSite=None; Partitioned` (CHIPS): `SameSite=Lax` would never * be sent there. `HttpOnly` keeps page scripts from reading the pointer. */ -export function draftCookie(request: RequestLike): string | null { +export function draftCookieFor(request: RequestLike): string | null { const param = draftParam(request); if (!param) return null; if (param === DRAFT_OFF) return serializeCookie("", ["Max-Age=0"]); @@ -169,6 +186,11 @@ export function draftCookie(request: RequestLike): string | null { return serializeCookie(encodeURIComponent(param), []); } +/** Whether the request's URL says `?__draft=off`: the cookie that ends a preview is set on any host. */ +export function endsPreview(request: RequestLike): boolean { + return draftParam(request) === DRAFT_OFF; +} + function serializeCookie(value: string, extra: string[]): string { return [ `${DRAFT_COOKIE}=${value}`, diff --git a/packages/blocks/src/v8/draftChanges.ts b/packages/blocks/src/v8/draftChanges.ts new file mode 100644 index 000000000..4b8305444 --- /dev/null +++ b/packages/blocks/src/v8/draftChanges.ts @@ -0,0 +1,172 @@ +/** + * Draft changes (see /next/content-delivery#draft-previews): what a draft + * pointer's address answers, and how the CMS layers it over production. + * + * - The pointer's host must fall under one of the preview API domains (the + * same rule as v7): by default `*.decocms.com` (Studio, its PR previews, + * `local.studio.decocms.com`) and the loopback hosts; `DECO_PREVIEW_API_DOMAINS` + * replaces the list. Any other host is refused before anything is fetched. + * - `GET ://&v=`: the scheme comes from the + * domain that admitted the host, never from the pointer: plain `http` only + * for loopback hosts. The pointer's `__variant` parameters are already gone (the parser + * lifts them out), no cookies or credentials are sent, and redirects aren't + * followed (a response that was redirected anyway is refused). + * - Only a 200 is accepted, within 10 s and up to 16 MiB, shaped exactly + * `{ format: 1, set: { : }, delete: […] }` with + * `set` and `delete` disjoint. + */ +import { readBoundedJson, timeoutSignal } from "./boundedJson.ts"; +import { readEnv } from "./identity.ts"; +import { isPlainObject } from "./json.ts"; +import type { DraftPointer, Snapshot } from "./types.ts"; + +/** The version a pointer has when it names no draft: `deco serve`'s working tree. */ +export const LOCAL_VERSION = "local"; + +const FORMAT = 1; +const TIMEOUT_MS = 10_000; +const MAX_BYTES = 16 * 1024 * 1024; + +/** What a draft changed compared with production: whole blocks, and deletions. */ +export interface DraftChanges { + format: typeof FORMAT; + set: Record; + delete: string[]; +} + +/** + * The domains a draft pointer's host may fall under, the defaults v7 shipped. + * Deco operates every one of them, so the defaults add no SSRF surface. An + * entry starting with a dot matches any host that ends with it, on a label + * boundary (`evil-decocms.com` doesn't pass `.decocms.com`); any other entry + * matches that exact host. The first entry that matches decides whether a + * port and plain `http` are allowed. + */ +export const DEFAULT_PREVIEW_API_DOMAINS: readonly string[] = [ + "local.studio.decocms.com", // the Studio dev origin (https, with a port) + "localhost", + "127.0.0.1", + ".localhost", + ".decocms.com", // Studio and its preview deployments +]; + +/** `DECO_PREVIEW_API_DOMAINS` (a comma list) when set, else the defaults. */ +function previewApiDomains(): readonly string[] { + const configured = (readEnv("DECO_PREVIEW_API_DOMAINS") ?? "") + .split(",") + .map((domain) => domain.trim().toLowerCase()) + .filter(Boolean); + return configured.length > 0 ? configured : DEFAULT_PREVIEW_API_DOMAINS; +} + +/** + * The origin a pointer's host is fetched from, or `null` when no preview API + * domain admits it. Loopback domains (and `local.studio.decocms.com`) may + * carry a port; a public domain may not, so a pointer can't aim the fetch at + * an odd port. Loopback hosts are `http`; everything else is `https`. + */ +export function previewApiOrigin(authority: string): string | null { + const lower = authority.toLowerCase(); + const end = lower.startsWith("[") ? lower.indexOf("]") + 1 : -1; + const host = end > 0 ? lower.slice(0, end) : lower.split(":")[0]!; + const rest = end > 0 ? lower.slice(end) : lower.slice(host.length); + const port = rest.startsWith(":") ? rest.slice(1) : undefined; + if (!host) return null; + const domain = previewApiDomains().find((d) => + d.startsWith(".") ? host.length > d.length && host.endsWith(d) : host === d, + ); + if (domain === undefined) return null; + const insecure = isLoopbackDomain(domain); + if (port !== undefined && !insecure && domain !== "local.studio.decocms.com") return null; + return `${insecure ? "http" : "https"}://${host}${port === undefined ? "" : `:${port}`}`; +} + +function isLoopbackDomain(domain: string): boolean { + return ( + domain === "localhost" || + domain === "127.0.0.1" || + domain === "[::1]" || + domain.endsWith(".localhost") + ); +} + +/** + * The URL the SDK fetches for a pointer: the origin its host is admitted + * under, the version as `v`. `null` when no preview API domain admits the host. + */ +export function draftChangesUrl(pointer: DraftPointer): string | null { + const origin = previewApiOrigin(pointer.host); + if (origin === null) return null; + const separator = pointer.path.includes("?") ? "&" : "?"; + return `${origin}${pointer.path}${separator}v=${encodeURIComponent(pointer.version)}`; +} + +/** Fetches a draft's changes; rejects on a host no preview API domain admits (without fetching) or on any failure. */ +export async function fetchDraftChanges(pointer: DraftPointer): Promise { + const url = draftChangesUrl(pointer); + if (url === null) { + throw new Error( + `draft pointer names "${pointer.host}", which isn't under a preview API domain ` + + "(DECO_PREVIEW_API_DOMAINS); nothing was fetched", + ); + } + const response = await fetch(url, { + headers: { accept: "application/json" }, + redirect: "manual", + signal: timeoutSignal(TIMEOUT_MS), + }); + // Some fetch polyfills (React Native, whatwg-fetch) ignore `redirect: "manual"` + // and follow anyway: refuse a response that came from anywhere else. + if ( + response.redirected || + (response.url && new URL(response.url).origin !== new URL(url).origin) + ) { + await response.body?.cancel(); + throw new Error("draft changes: the pointer's address redirected; nothing was read"); + } + if (response.status !== 200) { + await response.body?.cancel(); + throw new Error(`draft changes: HTTP ${response.status}`); + } + return parseDraftChanges(await readBoundedJson(response, "draft changes", MAX_BYTES)); +} + +/** Validates a draft changes response; throws on an unknown format or any unexpected shape. */ +export function parseDraftChanges(body: unknown): DraftChanges { + const invalid = (why: string) => new Error(`draft changes: ${why}`); + if (!isPlainObject(body) || body.format !== FORMAT) throw invalid("unknown format"); + if (Object.keys(body).some((key) => key !== "format" && key !== "set" && key !== "delete")) { + throw invalid("unexpected field"); + } + const { set, delete: deleted } = body; + if (!isPlainObject(set)) throw invalid("set must be an object"); + if (!Array.isArray(deleted) || deleted.some((name) => typeof name !== "string")) { + throw invalid("delete must be a list of names"); + } + if (new Set(deleted).size !== deleted.length) throw invalid("delete repeats a name"); + if (deleted.some((name) => Object.hasOwn(set, name))) throw invalid("set and delete overlap"); + return body as unknown as DraftChanges; +} + +/** + * The draft as a snapshot: production's entries, with the changed ones + * replaced whole and the deleted ones gone. A shallow copy: every unchanged + * entry is production's own object, and production is never mutated. Its + * revision is an opaque identity of the pair, never a release's. + */ +export function layerDraft(base: Snapshot, changes: DraftChanges, version: string): Snapshot { + const blocks: Record = { ...base.blocks }; + for (const name of changes.delete) delete blocks[name]; + for (const [name, entry] of Object.entries(changes.set)) { + // defineProperty, so an entry named "__proto__" stays an entry. + Object.defineProperty(blocks, name, { + value: entry, + enumerable: true, + writable: true, + configurable: true, + }); + } + const view: Snapshot = { revision: `${base.revision}~${version}`, blocks }; + if (base.aliases !== undefined) view.aliases = base.aliases; + return view; +} diff --git a/packages/blocks/src/v8/dualInstance.test.ts b/packages/blocks/src/v8/dualInstance.test.ts index e14a22e42..d34a2e86d 100644 --- a/packages/blocks/src/v8/dualInstance.test.ts +++ b/packages/blocks/src/v8/dualInstance.test.ts @@ -5,6 +5,7 @@ * This is the module-duplication bug the package split exists to prevent. */ import { afterEach, describe, expect, it, vi } from "vitest"; +import { instanceOf } from "./cms"; import { docsBlocks, docsSnapshot } from "./testFixtures"; type CMSModule = typeof import("./index"); @@ -31,7 +32,7 @@ describe("two copies of the module", () => { const content = docsSnapshot(); const fromA = a.createCMS({ blocks: docsBlocks(), content }); const fromB = b.createCMS({ blocks: docsBlocks(), content }); - expect(fromB).toBe(fromA); + expect(instanceOf(fromB)).toBe(instanceOf(fromA)); }); it("share one instance and one content cache for the same loader", async () => { @@ -41,7 +42,7 @@ describe("two copies of the module", () => { const loader = { load, update: async () => ({ updated: false }) }; const cmsA = a.createCMS({ blocks: docsBlocks(), content: loader }); const cmsB = b.createCMS({ blocks: docsBlocks(), content: loader }); - expect(cmsB).toBe(cmsA); + expect(instanceOf(cmsB)).toBe(instanceOf(cmsA)); await cmsA.forRelease().resolve("SummerSEO"); await cmsB.forRelease().resolve("SummerSEO"); expect(load).toHaveBeenCalledTimes(1); diff --git a/packages/blocks/src/v8/hosts.test.ts b/packages/blocks/src/v8/hosts.test.ts new file mode 100644 index 000000000..a26f43fbd --- /dev/null +++ b/packages/blocks/src/v8/hosts.test.ts @@ -0,0 +1,214 @@ +// @vitest-environment node +/** Host patterns (api-reference#host-patterns): parsing, matching and "within". */ +import { describe, expect, it } from "vitest"; +import { + allowsHost, + formatHostPattern, + type HostPattern, + parseHostPattern, + patternWithin, +} from "./hosts"; + +const parse = (raw: string): HostPattern => { + const pattern = parseHostPattern(raw); + if (pattern === null) throw new Error(`not a pattern: ${raw}`); + return pattern; +}; +const allows = (patterns: string[], url: string) => allowsHost(patterns.map(parse), url); +const within = (inner: string, outer: string) => patternWithin(parse(inner), parse(outer)); + +describe("parseHostPattern", () => { + it.each([ + ["*", "*"], + ["staging.example.com", "staging.example.com"], + ["Staging.Example.COM.", "staging.example.com"], + ["*.example.com", "*.example.com"], + ["*.Example.com.", "*.example.com"], + ["localhost", "localhost"], + ["localhost:3000", "localhost:3000"], + ["localhost:03000", "localhost:3000"], + ["127.0.0.1", "127.0.0.1"], + ["127.0.0.1:8080", "127.0.0.1:8080"], + ["[::1]", "[::1]"], + ["[0:0:0:0:0:0:0:1]:5173", "[::1]:5173"], + ["bücher.example", "xn--bcher-kva.example"], + ["*.bücher.example", "*.xn--bcher-kva.example"], + ])("%s is a pattern (normal form %s)", (raw, normal) => { + expect(formatHostPattern(parse(raw))).toBe(normal); + }); + + it.each([ + "", + " ", + "staging.example.com ", + " staging.example.com", + "https://staging.example.com", + "staging.example.com/path", + "staging.example.com/", + "a..example.com", + ".example.com", + "example.com..", + "a*.example.com", + "*a.example.com", + "*.*.com", + "x.*.example.com", + "*.com", + "*.", + "**", + "*.example.com*", + "*:3000", + "*.1.2.3.4", + "*.[::1]", + "1.2.3", + "01.2.3.4", + "256.1.1.1", + "0x7f.0.0.1", + "::1", + "[::1", + "[zz::1]", + "example.com:", + "example.com:abc", + "example.com:70000", + "example.com:80:80", + "user@example.com", + "example.com?x", + "example.com#x", + "exa mple.com", + "exa\tmple.com", + "-", + ])("%j is not a pattern", (raw) => { + expect(parseHostPattern(raw)).toBeNull(); + }); + + it("rejects anything that isn't a string", () => { + for (const raw of [null, undefined, 1, {}, ["a.com"], true]) { + expect(parseHostPattern(raw)).toBeNull(); + } + }); +}); + +describe("matching a request's host", () => { + it('"*" matches every host, and even a URL that can\'t be read', () => { + expect(allows(["*"], "https://anything.example/")).toBe(true); + expect(allows(["*"], "/relative")).toBe(true); + expect(allows(["*"], "not a url")).toBe(true); + }); + + it("an exact name matches only itself, on any port", () => { + const list = ["staging.example.com"]; + expect(allows(list, "https://staging.example.com/x")).toBe(true); + expect(allows(list, "http://staging.example.com:8080/")).toBe(true); + expect(allows(list, "https://STAGING.Example.com/")).toBe(true); + expect(allows(list, "https://staging.example.com./")).toBe(true); + expect(allows(list, "https://example.com/")).toBe(false); + expect(allows(list, "https://www.staging.example.com/")).toBe(false); + expect(allows(list, "https://staging.example.com.attacker.com/")).toBe(false); + expect(allows(list, "https://xstaging.example.com/")).toBe(false); + }); + + it("*.name matches one or more labels in front, label by label from the right", () => { + const list = ["*.example.com"]; + expect(allows(list, "https://a.example.com/")).toBe(true); + expect(allows(list, "https://a.b.example.com/")).toBe(true); + expect(allows(list, "https://A.EXAMPLE.COM./")).toBe(true); + expect(allows(list, "https://example.com/")).toBe(false); + expect(allows(list, "https://badexample.com/")).toBe(false); + expect(allows(list, "https://a.badexample.com/")).toBe(false); + expect(allows(list, "https://a.example.com.attacker.com/")).toBe(false); + expect(allows(list, "https://example.com.attacker.com/")).toBe(false); + expect(allows(list, "https://a.example.co/")).toBe(false); + expect(allows(list, "https://a.example.com../")).toBe(false); + }); + + it("a pattern with a port matches only a URL naming that port; default ports are never named", () => { + expect(allows(["localhost:3000"], "http://localhost:3000/")).toBe(true); + expect(allows(["localhost:3000"], "http://localhost/")).toBe(false); + expect(allows(["localhost:3000"], "http://localhost:3001/")).toBe(false); + expect(allows(["*.example.com:8443"], "https://a.example.com:8443/")).toBe(true); + expect(allows(["*.example.com:8443"], "https://a.example.com/")).toBe(false); + expect(allows(["example.com:443"], "https://example.com:443/")).toBe(false); + }); + + it("punycode: a non-ASCII host matches its pattern in either spelling", () => { + expect(allows(["bücher.example"], "https://xn--bcher-kva.example/")).toBe(true); + expect(allows(["xn--bcher-kva.example"], "https://bücher.example/")).toBe(true); + expect(allows(["*.bücher.example"], "https://shop.bücher.example/")).toBe(true); + }); + + it("IP addresses match exactly, in the form a URL gives them", () => { + expect(allows(["127.0.0.1"], "http://127.0.0.1:5173/")).toBe(true); + expect(allows(["127.0.0.1"], "http://0x7f.0.0.1/")).toBe(true); // the URL normalizes it + expect(allows(["127.0.0.1"], "http://127.0.0.2/")).toBe(false); + expect(allows(["[::1]"], "http://[0:0::1]:3000/")).toBe(true); + expect(allows(["*.example.com"], "http://127.0.0.1/")).toBe(false); + }); + + it("a URL that can't be read matches no pattern but *", () => { + expect(allows(["localhost"], "/relative?__draft=x")).toBe(false); + expect(allows(["*.example.com"], "")).toBe(false); + }); + + it("userinfo can't smuggle a host in", () => { + expect(allows(["staging.example.com"], "https://staging.example.com@evil.com/")).toBe(false); + expect(allows(["*.example.com"], "https://a.example.com:x@evil.com/")).toBe(false); + }); + + it("an empty list matches nothing", () => { + expect(allowsHost([], "https://example.com/")).toBe(false); + }); + + it("a URL with a username or password counts as unreadable (a forged Host header)", () => { + const list = ["staging.example.com"]; + expect(allows(list, "https://public.com@staging.example.com/")).toBe(false); + expect(allows(list, "https://user:pass@staging.example.com/")).toBe(false); + expect(allows(list, "https://:pass@staging.example.com/")).toBe(false); + expect(allows(["*"], "https://public.com@staging.example.com/")).toBe(true); + expect(allows(list, "https://staging.example.com/")).toBe(true); + }); + + it("a hostname with an empty label matches no wildcard or name", () => { + expect(allows(["*.example.com"], "https://a..example.com/")).toBe(false); + expect(allows(["*.example.com"], "https://a.example.com/")).toBe(true); + expect(allows(["*"], "https://a..example.com/")).toBe(true); + }); +}); + +describe("a content entry within code's entry", () => { + it("names and wildcards", () => { + expect(within("staging.example.com", "*.example.com")).toBe(true); + expect(within("a.b.example.com", "*.example.com")).toBe(true); + expect(within("*.a.example.com", "*.example.com")).toBe(true); + expect(within("*.example.com", "*.example.com")).toBe(true); + expect(within("*.example.com", "*.a.example.com")).toBe(false); + expect(within("example.com", "*.example.com")).toBe(false); + expect(within("*.example.com", "example.com")).toBe(false); + expect(within("staging.example.com", "staging.example.com")).toBe(true); + expect(within("other.example.com", "staging.example.com")).toBe(false); + expect(within("store.attacker.com", "*.example.com")).toBe(false); + expect(within("a.example.com.attacker.com", "*.example.com")).toBe(false); + expect(within("*.badexample.com", "*.example.com")).toBe(false); + }); + + it('"*" is within only "*"; everything is within "*"', () => { + expect(within("*", "*")).toBe(true); + expect(within("*", "*.example.com")).toBe(false); + expect(within("*.example.com", "*")).toBe(true); + expect(within("localhost:3000", "*")).toBe(true); + }); + + it("ports: no port is within only no port; port P is within no port or P", () => { + expect(within("localhost", "localhost:3000")).toBe(false); + expect(within("localhost:3000", "localhost")).toBe(true); + expect(within("localhost:3000", "localhost:3000")).toBe(true); + expect(within("localhost:3001", "localhost:3000")).toBe(false); + expect(within("a.example.com:8080", "*.example.com")).toBe(true); + expect(within("*.example.com", "*.example.com:8080")).toBe(false); + }); + + it("IP addresses: only the same address", () => { + expect(within("127.0.0.1", "127.0.0.1")).toBe(true); + expect(within("127.0.0.1:3000", "127.0.0.1")).toBe(true); + expect(within("127.0.0.2", "127.0.0.1")).toBe(false); + expect(within("[::1]", "[::1]")).toBe(true); + }); +}); diff --git a/packages/blocks/src/v8/hosts.ts b/packages/blocks/src/v8/hosts.ts new file mode 100644 index 000000000..0e42f56cc --- /dev/null +++ b/packages/blocks/src/v8/hosts.ts @@ -0,0 +1,184 @@ +/** + * Host patterns (see /next/api-reference#host-patterns): the format of + * `preview.hosts`, in `createCMS` and in the `CMS` block. + * + * - Hostnames and patterns are compared lowercase, with one trailing dot + * removed; names outside ASCII in their punycode form. + * - A pattern without `*` matches that one hostname. + * - `*.name` matches a hostname that ends in `.name` with at least one more + * label in front, compared label by label from the right. `*` is only ever + * the whole leading label, and at least two labels follow it. + * - `"*"` alone matches every host. + * - A pattern without a port matches any port; one with a port matches only + * a URL that names that port. + * - IPv4, and IPv6 in brackets, match exactly and never take a wildcard. + * - Anything else (a scheme, a path, a space, an empty label) isn't a pattern. + */ + +/** A parsed pattern. `port` is absent when the pattern names none. */ +export type HostPattern = + | { kind: "any" } + | { kind: "name"; host: string; port?: string } + | { kind: "wildcard"; suffix: string; port?: string } + | { kind: "ip"; host: string; port?: string }; + +/** The list that allows every host. */ +export const EVERY_HOST = "*"; + +const LABEL_RE = /^[a-z0-9_]([a-z0-9_-]{0,61}[a-z0-9_])?$/; +const PORT_RE = /^[0-9]{1,5}$/; +const IPV4_RE = + /^(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])(\.(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])){3}$/; +const NUMERIC_LABEL_RE = /^(0x[0-9a-f]*|[0-9]+)$/; + +/** Parses one pattern; `null` when it isn't one. */ +export function parseHostPattern(raw: unknown): HostPattern | null { + if (typeof raw !== "string" || raw.length === 0 || raw.length > 300) return null; + if (raw === EVERY_HOST) return { kind: "any" }; + if (/[\s/\\?#@]/.test(raw) || raw.includes("://")) return null; + const split = splitPort(raw); + if (split === null) return null; + const { port } = split; + let host = split.host; + if (host.startsWith("[")) { + const ip = normalizeIPv6(host); + return ip === null ? null : withPort({ kind: "ip", host: ip }, port); + } + host = host.toLowerCase(); + if (host.endsWith(".")) host = host.slice(0, -1); + if (host.startsWith("*.")) { + const suffix = normalizeName(host.slice(2)); + if (suffix === null || suffix.split(".").length < 2) return null; + return withPort({ kind: "wildcard", suffix }, port); + } + if (IPV4_RE.test(host)) return withPort({ kind: "ip", host }, port); + const name = normalizeName(host); + return name === null ? null : withPort({ kind: "name", host: name }, port); +} + +/** Formats a parsed pattern back into its normal form. */ +export function formatHostPattern(pattern: HostPattern): string { + if (pattern.kind === "any") return EVERY_HOST; + const host = pattern.kind === "wildcard" ? `*.${pattern.suffix}` : pattern.host; + return pattern.port === undefined ? host : `${host}:${pattern.port}`; +} + +/** + * Whether a request URL's host is one a pattern matches. A URL that can't be + * read matches only `"*"`, and neither does a hostname with an empty label. + */ +export function hostMatches(pattern: HostPattern, url: URL | null): boolean { + if (pattern.kind === "any") return true; + if (url === null) return false; + if (pattern.port !== undefined && pattern.port !== url.port) return false; + let host = url.hostname.toLowerCase(); + if (host.endsWith(".")) host = host.slice(0, -1); + // An empty label (`a..example.com`) isn't a hostname any pattern names. + if (host.split(".").includes("")) return false; + switch (pattern.kind) { + case "name": + case "ip": + return host === pattern.host; + case "wildcard": + return host.length > pattern.suffix.length + 1 && host.endsWith(`.${pattern.suffix}`); + } +} + +/** + * Whether a request may preview under this list: some pattern matches its URL's + * host. A URL with a username or password (`https://public.com@staging.example.com/`, + * which a forged Host header can produce) counts as unreadable. + */ +export function allowsHost(patterns: readonly HostPattern[], requestUrl: string): boolean { + let url: URL | null; + try { + url = new URL(requestUrl); + } catch { + url = null; + } + if (url !== null && (url.username !== "" || url.password !== "")) url = null; + return patterns.some((pattern) => hostMatches(pattern, url)); +} + +/** + * Whether every host `inner` matches is matched by `outer`: content's entry + * `inner` is within code's entry `outer`. + */ +export function patternWithin(inner: HostPattern, outer: HostPattern): boolean { + if (outer.kind === "any") return true; + if (inner.kind === "any") return false; + if (outer.port !== undefined && inner.port !== outer.port) return false; + switch (outer.kind) { + case "name": + case "ip": + return inner.kind === outer.kind && inner.host === outer.host; + case "wildcard": + if (inner.kind === "name") return inner.host.endsWith(`.${outer.suffix}`); + if (inner.kind === "wildcard") { + return inner.suffix === outer.suffix || inner.suffix.endsWith(`.${outer.suffix}`); + } + return false; + } +} + +function withPort(pattern: T, port: string | undefined): T { + return port === undefined ? pattern : { ...pattern, port }; +} + +/** `host[:port]`, with the port checked; `null` on an empty host or a bad port. */ +function splitPort(raw: string): { host: string; port?: string } | null { + let host = raw; + let port: string | undefined; + if (raw.startsWith("[")) { + const end = raw.indexOf("]"); + if (end === -1) return null; + host = raw.slice(0, end + 1); + const rest = raw.slice(end + 1); + if (rest !== "") { + if (!rest.startsWith(":")) return null; + port = rest.slice(1); + } + } else { + const colon = raw.indexOf(":"); + if (colon !== -1) { + if (raw.indexOf(":", colon + 1) !== -1) return null; // an IPv6 address without brackets + host = raw.slice(0, colon); + port = raw.slice(colon + 1); + } + } + if (host === "") return null; + if (port !== undefined && (!PORT_RE.test(port) || Number(port) > 65535)) return null; + // The form a URL gives it: no leading zeros. + return port === undefined ? { host } : { host, port: String(Number(port)) }; +} + +/** A DNS name in its ASCII (punycode) form, lowercase; `null` if it isn't one. */ +function normalizeName(name: string): string | null { + if (name === "" || name.includes("*")) return null; + let ascii = name; + // biome-ignore lint/suspicious/noControlCharactersInRegex: the ASCII range is the point + if (/[^\x00-\x7f]/.test(name)) { + try { + ascii = new URL(`http://${name}/`).hostname; + } catch { + return null; + } + } + const labels = ascii.split("."); + if (labels.some((label) => !LABEL_RE.test(label))) return null; + // A name ending in a numeric label is an IPv4 address to a URL parser: only + // the canonical dotted form above is one. + if (NUMERIC_LABEL_RE.test(labels[labels.length - 1])) return null; + return ascii.length > 253 ? null : ascii; +} + +/** `[addr]` in the compressed, lowercase form a URL's hostname has. */ +function normalizeIPv6(bracketed: string): string | null { + if (!/^\[[0-9a-fA-F:.]+\]$/.test(bracketed)) return null; + try { + const host = new URL(`http://${bracketed}/`).hostname; + return host.startsWith("[") ? host : null; + } catch { + return null; + } +} diff --git a/packages/blocks/src/v8/index.ts b/packages/blocks/src/v8/index.ts index 75a4da093..f2cdb181f 100644 --- a/packages/blocks/src/v8/index.ts +++ b/packages/blocks/src/v8/index.ts @@ -1,16 +1,11 @@ /** * The next-major (v8) core SDK: `createCMS`, the clients, the built-in blocks, - * `matchRoute` and the draft helpers. Re-exported from the package root, where + * `matchRoute` and the draft pointer helpers (`cms.draftPointer` and + * `cms.draftCookie` are methods: they check the preview hosts). Re-exported from the package root, where * these names are the documented ones; see /next/api-reference. */ export { createCMS, resetForTests } from "./cms.ts"; -export { - DRAFT_COOKIE, - draftCookie, - draftPointer, - formatDraftPointer, - parseDraftPointer, -} from "./draft.ts"; +export { formatDraftPointer, parseDraftPointer } from "./draft.ts"; export { matchRoute } from "./matchRoute.ts"; export { remoteLoader } from "./remoteLoader.ts"; export type { @@ -21,13 +16,16 @@ export type { Client, CMS, CMSError, + CMSSettings, DraftPointer, + EffectiveSettings, Lazy, ListOptions, Loader, Match, Page, Redirect, + RequestLike, Result, Route, Secret, diff --git a/packages/blocks/src/v8/remoteLoader.test.ts b/packages/blocks/src/v8/remoteLoader.test.ts index 19cde9251..9e0070ceb 100644 --- a/packages/blocks/src/v8/remoteLoader.test.ts +++ b/packages/blocks/src/v8/remoteLoader.test.ts @@ -1,8 +1,9 @@ // @vitest-environment node /** * remoteLoader (hosted-releases-internals.mdx, hosted-publishing.mdx, - * content-delivery.mdx, hosted-drafts.mdx): requests read memory, the - * background check follows the channel manifest, drafts are fetched exactly. + * content-delivery.mdx): requests read memory, and the background check + * follows the channel manifest. Drafts aren't the loader's (see + * draftChanges.test.ts). */ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { computeContentRevision } from "./canonical"; @@ -17,7 +18,6 @@ const remote = (...args: Parameters) => remoteLoader(...arg const HOSTED_DELIVERY_ORIGIN = "https://delivery.decocms.com"; const SITE = "acme"; const TOKEN = "site-token"; -const HOST = new URL(HOSTED_DELIVERY_ORIGIN).host; const MANIFEST_URL = `${HOSTED_DELIVERY_ORIGIN}/sites/acme/channels/production.json`; beforeEach(() => resetForTests()); @@ -224,44 +224,6 @@ describe("remoteLoader: releases", () => { }); }); -describe("remoteLoader: drafts", () => { - it("fetches exactly the pointer's version from the delivery host, with the site token", async () => { - const api = deliveryApi(); - const draft = await hashed("Draft"); - api.asset("/drafts/acme/feat", draft); - const loader = remote(docsSnapshot(), { site: SITE, token: TOKEN }); - - expect(await loader.load(`${HOST}/drafts/acme/feat?token=signed@9f3c1a`)).toEqual(draft); - expect(api.requests[0]).toMatchObject({ - url: `https://${HOST}/drafts/acme/feat?token=signed`, - headers: { authorization: `Bearer ${TOKEN}`, "if-match": "9f3c1a" }, - }); - }); - - it("refuses a pointer to any other host without fetching (LOADER_FAILED)", async () => { - const api = deliveryApi(); - const cms = createCMS({ - blocks: docsBlocks(), - content: docsSnapshot(), - site: SITE, - token: TOKEN, - }); - const [value, error] = await cms.forDraft("evil.example/steal@v1").resolve("SummerSEO"); - expect(value).toBeNull(); - expect(error?.code).toBe("LOADER_FAILED"); - expect(api.fetch).not.toHaveBeenCalled(); - }); - - it("loads drafts in development too", async () => { - vi.stubEnv("NODE_ENV", "development"); - const api = deliveryApi(); - const draft = await hashed("Draft"); - api.asset("/drafts/acme/feat", draft); - const loader = remote(docsSnapshot(), { site: SITE, token: TOKEN }); - expect(await loader.load(`${HOST}/drafts/acme/feat@v1`)).toEqual(draft); - }); -}); - describe("remoteLoader with createCMS", () => { it("site and token wrap the content: a check swaps the release in for the next client", async () => { const api = deliveryApi(); diff --git a/packages/blocks/src/v8/remoteLoader.ts b/packages/blocks/src/v8/remoteLoader.ts index d9d8a65be..0ee18ebff 100644 --- a/packages/blocks/src/v8/remoteLoader.ts +++ b/packages/blocks/src/v8/remoteLoader.ts @@ -1,7 +1,8 @@ /** - * `remoteLoader`: hosted releases and drafts from the Deco API, over a - * fallback (the content module, or any loader). See - * /next/hosted-releases-internals and /next/hosted-publishing. + * `remoteLoader`: hosted releases from the Deco API, over a fallback (the + * content module, or any loader). See /next/hosted-releases-internals and + * /next/hosted-publishing. Drafts aren't a loader's job: the CMS layers a + * draft's changes over what `load()` returns (see ./draftChanges.ts). * * - `load()` never touches the network: it returns the newest release this * process fetched, or the fallback's content until one is fetched. @@ -11,31 +12,28 @@ * ignored; a revision equal to the fallback's is served from the fallback * without a download; anything else is fetched, verified against its * content hash and swapped in whole. Any error keeps memory as it was. - * - `load(pointer)` fetches the draft a pointer names and waits for it. Only - * pointers to the delivery host are fetched; anything else is refused. * - In development (`NODE_ENV=development`), releases stay on the fallback so - * local files win; drafts still load. - * - A release or draft larger than `MAX_SNAPSHOT_BYTES` is refused while it - * downloads, before it's buffered whole (a Worker isolate has 128 MB). - * - Without `site` or `token` it's a plain loader over the fallback: no - * releases, no drafts beyond what the fallback serves. + * local files win. + * - A release larger than `MAX_SNAPSHOT_BYTES` is refused while it downloads, + * before it's buffered whole (a Worker isolate has 128 MB). + * - Without `site` or `token` it's a plain loader over the fallback. * * Instances are process-wide singletons, like `createCMS`'s. */ +import { readBoundedJson, timeoutSignal } from "./boundedJson.ts"; import { computeContentRevision } from "./canonical.ts"; -import { isSnapshot } from "./content.ts"; -import { parseDraftPointer } from "./draft.ts"; +import { isSnapshot, PEEK_RELEASE, peekRelease } from "./content.ts"; import { clearGlobals, contentIdentity, fnv1a } from "./identity.ts"; import { isPlainObject } from "./json.ts"; import type { Loader, Snapshot } from "./types.ts"; -/** The hosted delivery origin: channel manifests, release assets and drafts. */ +/** The hosted delivery origin: channel manifests and release assets. */ const HOSTED_DELIVERY_ORIGIN = "https://delivery.decocms.com"; const INSTANCE_PREFIX = "decocms.blocks.remote:"; const MANIFEST_FORMAT = 1; const FETCH_TIMEOUT_MS = 10_000; -/** The largest release or draft accepted, in bytes of JSON. */ +/** The largest release accepted, in bytes of JSON. */ const MAX_SNAPSHOT_BYTES = 64 * 1024 * 1024; interface RemoteLoaderOptions { @@ -58,6 +56,8 @@ class RemoteLoader implements Loader { readonly interval: number | undefined; #fallback: Snapshot | Loader; #fallbackRevision: string | undefined; + /** The fallback's content as last loaded, for a fallback that is itself a loader. */ + #fallbackSnapshot: Snapshot | undefined; #current: Snapshot | undefined; #generation = -1; @@ -76,10 +76,15 @@ class RemoteLoader implements Loader { if (fallback === this.#fallback) return; this.#fallback = fallback; this.#fallbackRevision = undefined; + this.#fallbackSnapshot = undefined; } - load(pointer?: string | null): Promise { - if (pointer) return this.#draft(pointer); + /** What `load()` would return now, from memory only (see `peekRelease`). */ + [PEEK_RELEASE](): Snapshot | undefined { + return this.#current ?? peekRelease(this.#fallback) ?? this.#fallbackSnapshot; + } + + load(): Promise { if (this.#current !== undefined) return Promise.resolve(this.#current); return this.#loadFallback(); } @@ -114,7 +119,10 @@ class RemoteLoader implements Loader { async #loadFallback(): Promise { const fallback = this.#fallback; const snapshot = isSnapshot(fallback) ? fallback : await (fallback as Loader).load(); - if (fallback === this.#fallback) this.#fallbackRevision = snapshot.revision; + if (fallback === this.#fallback) { + this.#fallbackRevision = snapshot.revision; + this.#fallbackSnapshot = snapshot; + } return snapshot; } @@ -152,7 +160,11 @@ class RemoteLoader implements Loader { this.#auth(), ); if (!response.ok) throw new Error(`release ${manifest.revision}: HTTP ${response.status}`); - const snapshot = await readBoundedJson(response, `release ${manifest.revision}`); + const snapshot = await readBoundedJson( + response, + `release ${manifest.revision}`, + MAX_SNAPSHOT_BYTES, + ); if ( !isSnapshot(snapshot) || snapshot.revision !== manifest.revision || @@ -163,23 +175,6 @@ class RemoteLoader implements Loader { return snapshot; } - async #draft(raw: string): Promise { - const pointer = parseDraftPointer(raw); - if (pointer === null) throw new Error("invalid draft pointer"); - if (pointer.host !== new URL(HOSTED_DELIVERY_ORIGIN).host) { - throw new Error(`draft pointer names an unexpected host "${pointer.host}"`); - } - // If-Match: serve exactly this version or fail, never the branch's newer head. - const response = await fetchWithTimeout(`https://${pointer.host}${pointer.path}`, { - ...this.#auth(), - "if-match": pointer.version, - }); - if (!response.ok) throw new Error(`draft: HTTP ${response.status}`); - const snapshot = await readBoundedJson(response, "draft"); - if (!isSnapshot(snapshot)) throw new Error("draft: not a snapshot"); - return snapshot; - } - #auth(): Record { return { authorization: `Bearer ${this.#token}` }; } @@ -197,14 +192,18 @@ class LocalLoader implements Loader { this.#fallback = fallback; } - load(pointer?: string | null): Promise { + [PEEK_RELEASE](): Snapshot | undefined { + return peekRelease(this.#fallback); + } + + load(): Promise { const fallback = this.#fallback; - return isSnapshot(fallback) ? Promise.resolve(fallback) : fallback.load(pointer); + return isSnapshot(fallback) ? Promise.resolve(fallback) : fallback.load(); } } /** - * Hosted releases and drafts over a fallback. `createCMS` builds it for you + * Hosted releases over a fallback. `createCMS` builds it for you * when `site` and `token` are set. With either unset (a dev or test * environment without the variables), it's a loader over the fallback alone. */ @@ -238,38 +237,8 @@ export function resetRemoteLoaders(): void { clearGlobals(INSTANCE_PREFIX); } -/** Parses a JSON body, refusing it as soon as it's larger than `MAX_SNAPSHOT_BYTES`. */ -async function readBoundedJson(response: Response, label: string): Promise { - const tooLarge = () => new Error(`${label}: larger than ${MAX_SNAPSHOT_BYTES} bytes`); - if (Number(response.headers.get("content-length")) > MAX_SNAPSHOT_BYTES) { - await response.body?.cancel(); - throw tooLarge(); - } - if (response.body === null) return response.json(); - const reader = response.body.getReader(); - const decoder = new TextDecoder(); - const parts: string[] = []; - let size = 0; - for (;;) { - const { done, value } = await reader.read(); - if (done) break; - size += value.byteLength; - if (size > MAX_SNAPSHOT_BYTES) { - await reader.cancel(); - throw tooLarge(); - } - parts.push(decoder.decode(value, { stream: true })); - } - parts.push(decoder.decode()); - return JSON.parse(parts.join("")); -} - function fetchWithTimeout(url: string, headers: Record): Promise { - const signal = - typeof AbortSignal !== "undefined" && typeof AbortSignal.timeout === "function" - ? AbortSignal.timeout(FETCH_TIMEOUT_MS) - : undefined; - return fetch(url, { headers, signal }); + return fetch(url, { headers, signal: timeoutSignal(FETCH_TIMEOUT_MS) }); } function isDevelopment(): boolean { diff --git a/packages/blocks/src/v8/resolve.test.ts b/packages/blocks/src/v8/resolve.test.ts index 8fd4d681c..2c7cb7a3a 100644 --- a/packages/blocks/src/v8/resolve.test.ts +++ b/packages/blocks/src/v8/resolve.test.ts @@ -233,8 +233,7 @@ describe("the lookup rule — { ...savedBlocks, ...builtIns, ...blocks }", () => "date", "page", "redirect", - "telemetry", - "analytics", + "cms-settings", "multivariate", "lazy", ]) { diff --git a/packages/blocks/src/v8/rootExports.test.ts b/packages/blocks/src/v8/rootExports.test.ts index 0197ce771..fb5e10997 100644 --- a/packages/blocks/src/v8/rootExports.test.ts +++ b/packages/blocks/src/v8/rootExports.test.ts @@ -13,13 +13,17 @@ describe("@decocms/blocks root", () => { expect(root.resetForTests).toBe(v8.resetForTests); expect(root.matchRoute).toBe(v8.matchRoute); expect(root.remoteLoader).toBe(v8.remoteLoader); - expect(root.draftPointer).toBe(v8.draftPointer); - expect(root.draftCookie).toBe(v8.draftCookie); - expect(root.DRAFT_COOKIE).toBe(v8.DRAFT_COOKIE); expect(root.parseDraftPointer).toBe(v8.parseDraftPointer); expect(root.formatDraftPointer).toBe(v8.formatDraftPointer); }); + it("has no free draft helpers: draftPointer and draftCookie are CMS methods that check the host", () => { + const names = Object.keys(root); + for (const name of ["draftPointer", "draftCookie", "DRAFT_COOKIE"]) { + expect(names).not.toContain(name); + } + }); + it("exports nothing outside the v8 API", () => { const extra = Object.keys(root).filter((name) => !(name in v8)); expect(extra).toEqual([]); diff --git a/packages/blocks/src/v8/settings.test.ts b/packages/blocks/src/v8/settings.test.ts new file mode 100644 index 000000000..1c09aea8e --- /dev/null +++ b/packages/blocks/src/v8/settings.test.ts @@ -0,0 +1,531 @@ +// @vitest-environment node +/** + * The CMS settings (built-in-blocks#cms-settings, api-reference#cms-settings, + * releases-and-drafts#allow-previews-per-host): `cms.settings()`, its caps, + * and the host check of `cms.draftPointer` / `cms.draftCookie`. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { HOSTED_ANALYTICS_COLLECTOR } from "./builtins/data"; +import { createCMS, instanceOf, resetForTests } from "./cms"; +import { currentTelemetry } from "./telemetry"; +import { docsBlocks, docsSnapshot, fakeStudio } from "./testFixtures"; +import type { EffectiveSettings, Snapshot } from "./types"; + +const POINTER = "studio.decocms.com/api/acme/decofile/store/main/changes?token=abc@9f3c1a"; +const DEFAULTS: EffectiveSettings = { + preview: { hosts: ["*"] }, + telemetry: { enabled: true, metrics: true, errorSampleRate: 0.05, traceSampleRate: 0 }, + analytics: { collector: HOSTED_ANALYTICS_COLLECTOR, enabled: true }, +}; + +beforeEach(() => resetForTests()); +afterEach(() => { + vi.useRealTimers(); + vi.restoreAllMocks(); + vi.unstubAllGlobals(); + resetForTests(); +}); + +/** The docs' release with a `CMS` block. */ +function withSettings(settings: Record | undefined, revision = "rev-1"): Snapshot { + const snapshot = docsSnapshot(revision); + if (settings !== undefined) snapshot.blocks.CMS = { __resolveType: "cms-settings", ...settings }; + return snapshot; +} + +const lazy = (value: unknown) => ({ __resolveType: "lazy", value }); +const request = (url: string, cookie?: string) => + new Request(url, { headers: cookie ? { cookie } : {} }); + +describe("cms.settings(): defaults", () => { + it("with no CMS block, every default; every host may preview", async () => { + const cms = createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); + expect(await cms.settings()).toEqual(DEFAULTS); + }); + + it("the settings are deep-frozen: one caller can't change them for every request", async () => { + const cms = createCMS({ + blocks: docsBlocks(), + content: withSettings({ preview: { hosts: ["staging.example.com"] } }), + }); + const settings = await cms.settings(); + expect(Object.isFrozen(settings)).toBe(true); + expect(Object.isFrozen(settings.preview)).toBe(true); + expect(Object.isFrozen(settings.preview.hosts)).toBe(true); + expect(Object.isFrozen(settings.telemetry)).toBe(true); + expect(Object.isFrozen(settings.analytics)).toBe(true); + expect(() => { + (settings.analytics as { enabled: boolean }).enabled = false; + }).toThrow(TypeError); + expect((await cms.settings()).analytics.enabled).toBe(true); + expect( + Object.isFrozen(await createCMS({ blocks: {}, content: docsSnapshot("r9") }).settings()), + ).toBe(true); + }); + + it("an empty CMS block is the defaults too", async () => { + const cms = createCMS({ blocks: docsBlocks(), content: withSettings({}) }); + expect(await cms.settings()).toEqual(DEFAULTS); + }); + + it("each field left out gets its default; the others are kept", async () => { + const cms = createCMS({ + blocks: docsBlocks(), + content: withSettings({ + telemetry: { metrics: false }, + analytics: { collector: "https://stats.example.com/events" }, + }), + }); + expect(await cms.settings()).toEqual({ + preview: { hosts: ["*"] }, + telemetry: { enabled: true, metrics: false, errorSampleRate: 0.05, traceSampleRate: 0 }, + analytics: { collector: "https://stats.example.com/events", enabled: true }, + }); + }); + + it("a CMS block of another type is ignored: the defaults apply", async () => { + const snapshot = docsSnapshot(); + snapshot.blocks.CMS = { __resolveType: "seo", title: "t", description: "d" }; + const cms = createCMS({ blocks: docsBlocks(), content: snapshot }); + expect(await cms.settings()).toEqual(DEFAULTS); + }); + + it("garbage in a section is ignored field by field", async () => { + const cms = createCMS({ + blocks: docsBlocks(), + content: withSettings({ + preview: { hosts: "staging.example.com" }, + telemetry: { enabled: "no", errorSampleRate: "high" }, + analytics: { collector: 42, enabled: null }, + }), + }); + expect(await cms.settings()).toEqual(DEFAULTS); + }); + + it("never rejects, even when the content loader fails", async () => { + const cms = createCMS({ + blocks: {}, + content: { + load: async () => { + throw new Error("down"); + }, + }, + }); + await cms + .forRelease() + .revision() + .catch(() => {}); + expect(await cms.settings()).toEqual(DEFAULTS); + }); +}); + +describe("cms.settings(): caps from code", () => { + it("telemetry rates are the lower of content's and telemetry.limits (defaults 0.1 and 0)", async () => { + const content = withSettings({ telemetry: { errorSampleRate: 0.5, traceSampleRate: 0.5 } }); + const defaults = await createCMS({ blocks: {}, content }).settings(); + expect(defaults.telemetry).toMatchObject({ errorSampleRate: 0.1, traceSampleRate: 0 }); + resetForTests(); + const limited = await createCMS({ + blocks: {}, + content, + telemetry: { + endpoint: "https://otel.example", + limits: { errorSampleRate: 1, traceSampleRate: 0.2 }, + }, + }).settings(); + expect(limited.telemetry).toMatchObject({ errorSampleRate: 0.5, traceSampleRate: 0.2 }); + }); + + it("telemetry: false still caps with the default limits", async () => { + const cms = createCMS({ + blocks: {}, + content: withSettings({ telemetry: { errorSampleRate: 1 } }), + telemetry: false, + }); + expect((await cms.settings()).telemetry.errorSampleRate).toBe(0.1); + }); + + it("without a preview field, code's list applies (in normal form)", async () => { + const cms = createCMS({ + blocks: {}, + content: withSettings({ telemetry: {} }), + preview: { hosts: ["*.Example.com", "localhost:3000"] }, + }); + expect((await cms.settings()).preview.hosts).toEqual(["*.example.com", "localhost:3000"]); + }); + + it("content keeps only its entries that fall within code's list", async () => { + const cms = createCMS({ + blocks: {}, + content: withSettings({ + preview: { + hosts: [ + "staging.example.com", + "*.preview.example.com", + "localhost:3000", + "localhost", + "store.attacker.com", + "staging.example.com.attacker.com", + "*", + "*.com", + "https://staging.example.com", + ], + }, + }), + preview: { hosts: ["*.example.com", "localhost:3000"] }, + }); + expect((await cms.settings()).preview.hosts).toEqual([ + "staging.example.com", + "*.preview.example.com", + "localhost:3000", + ]); + }); + + it("without a cap, content may allow any valid pattern; invalid entries are left out", async () => { + const cms = createCMS({ + blocks: {}, + content: withSettings({ preview: { hosts: ["Staging.Example.com.", "a b", "*.com", "*"] } }), + }); + expect((await cms.settings()).preview.hosts).toEqual(["staging.example.com", "*"]); + }); + + it("an empty list turns previews off", async () => { + const cms = createCMS({ blocks: {}, content: withSettings({ preview: { hosts: [] } }) }); + expect((await cms.settings()).preview.hosts).toEqual([]); + const pointer = encodeURIComponent(POINTER); + expect(await cms.draftPointer(request(`https://localhost/?__draft=${pointer}`))).toBeNull(); + expect(await cms.draftCookie(request(`https://localhost/?__draft=${pointer}`))).toBeNull(); + }); + + it("createCMS throws on a pattern code can't read, naming it", () => { + for (const bad of ["https://x.example.com", "*.com", "a*.example.com", "x.example.com/"]) { + expect( + () => createCMS({ blocks: {}, content: docsSnapshot(), preview: { hosts: [bad] } }), + bad, + ).toThrow(new RegExp(JSON.stringify(bad).replace(/[.*+?^${}()|[\]\\/]/g, "\\$&"))); + } + expect(() => + createCMS({ + blocks: {}, + content: docsSnapshot(), + preview: { hosts: "a.example.com" as unknown as string[] }, + }), + ).toThrow(TypeError); + }); +}); + +describe("cms.settings(): variants", () => { + it("a section can vary on its own, picked on each call", async () => { + let campaign = false; + const cms = createCMS({ + blocks: { campaign: () => campaign }, + content: withSettings({ + telemetry: { metrics: false }, + analytics: { + __resolveType: "multivariate", + variants: [ + { rule: { __resolveType: "campaign" }, value: lazy({ enabled: true }) }, + { rule: { __resolveType: "always" }, value: lazy({ enabled: false }) }, + ], + }, + }), + }); + expect((await cms.settings()).analytics.enabled).toBe(false); + campaign = true; + expect((await cms.settings()).analytics.enabled).toBe(true); + // The other sections are unaffected. + expect((await cms.settings()).telemetry.metrics).toBe(false); + }); + + it("a single field can have variants", async () => { + const cms = createCMS({ + blocks: {}, + content: withSettings({ + preview: { + hosts: { + __resolveType: "multivariate", + variants: [ + { rule: { __resolveType: "never" }, value: lazy(["never.example.com"]) }, + { rule: { __resolveType: "always" }, value: lazy(["staging.example.com"]) }, + ], + }, + }, + }), + }); + expect((await cms.settings()).preview.hosts).toEqual(["staging.example.com"]); + }); + + it("the whole block can be a reference to a saved block's values", async () => { + const snapshot = withSettings({ analytics: { __resolveType: "Collector" } }); + snapshot.blocks.Collector = { + __resolveType: "multivariate", + variants: [{ rule: { __resolveType: "always" }, value: lazy({ enabled: false }) }], + }; + const cms = createCMS({ blocks: {}, content: snapshot }); + expect((await cms.settings()).analytics.enabled).toBe(false); + }); + + it("a section that fails to resolve gets its defaults, still capped; the others are kept", async () => { + const cms = createCMS({ + blocks: { + broken: () => { + throw new Error("no"); + }, + }, + content: withSettings({ + preview: { hosts: ["staging.example.com"] }, + telemetry: { __resolveType: "broken" }, + analytics: { enabled: false }, + }), + preview: { hosts: ["*.example.com"] }, + }); + expect(await cms.settings()).toEqual({ + preview: { hosts: ["staging.example.com"] }, + telemetry: DEFAULTS.telemetry, + analytics: { collector: HOSTED_ANALYTICS_COLLECTOR, enabled: false }, + }); + }); + + it("a preview section that fails gets code's list, never every host", async () => { + const cms = createCMS({ + blocks: { + broken: () => { + throw new Error("no"); + }, + }, + content: withSettings({ preview: { __resolveType: "broken" } }), + preview: { hosts: ["staging.example.com"] }, + }); + expect((await cms.settings()).preview.hosts).toEqual(["staging.example.com"]); + }); +}); + +describe("cms.settings(): always the release in memory", () => { + it("a custom loader with nothing in memory yet gives the defaults, without a load", async () => { + const load = vi.fn(async () => withSettings({ analytics: { enabled: false } })); + const cms = createCMS({ + blocks: {}, + content: { load, update: async () => ({ updated: false }) }, + }); + expect(await cms.settings()).toEqual(DEFAULTS); + expect(load).not.toHaveBeenCalled(); + await cms.forRelease().revision(); + expect((await cms.settings()).analytics.enabled).toBe(false); + expect(load).toHaveBeenCalledTimes(1); + }); + + it("after a custom loader's update, the old release's settings hold until the next one loads", async () => { + let current = withSettings({ preview: { hosts: ["staging.example.com"] } }, "rev-1"); + const cms = createCMS({ + blocks: docsBlocks(), + content: { load: async () => current, update: async () => ({ updated: true }) }, + }); + const link = `?__draft=${encodeURIComponent(POINTER)}`; + await cms.forRelease().revision(); + current = withSettings({ preview: { hosts: ["next.example.com"] } }, "rev-2"); + expect(await cms.update()).toEqual({ updated: true }); + // Not the defaults (every host) in between: still the old release's hosts. + expect((await cms.settings()).preview.hosts).toEqual(["staging.example.com"]); + expect(await cms.draftPointer(request(`https://www.example.com/${link}`))).toBeNull(); + expect(await cms.forRelease().revision()).toBe("rev-2"); + expect((await cms.settings()).preview.hosts).toEqual(["next.example.com"]); + }); + + it("a draft that changes CMS changes neither the settings, nor its own preview gating", async () => { + const release = withSettings({ + preview: { hosts: ["staging.example.com"] }, + analytics: { enabled: true }, + }); + const studio = fakeStudio(); + vi.stubGlobal("fetch", vi.fn(studio.fetch)); + const pointer = studio.draft({ + set: { + CMS: { + __resolveType: "cms-settings", + preview: { hosts: ["*"] }, + analytics: { enabled: false }, + }, + }, + }); + const cms = createCMS({ blocks: docsBlocks(), content: release }); + await cms.forRelease().revision(); + const [saved] = await cms + .forDraft(pointer) + .resolve<{ preview: unknown }>("CMS", { run: false }); + expect(saved?.preview).toEqual({ hosts: ["*"] }); // the draft has it… + const settings = await cms.settings(); // …the settings don't + expect(settings.preview.hosts).toEqual(["staging.example.com"]); + expect(settings.analytics.enabled).toBe(true); + const link = `?__draft=${encodeURIComponent(pointer)}`; + expect(await cms.draftPointer(request(`https://store.example.com/${link}`))).toBeNull(); + expect(await cms.draftPointer(request(`https://staging.example.com/${link}`))).toBe(pointer); + }); + + it("a draft that changes CMS doesn't change telemetry", async () => { + const studio = fakeStudio(); + vi.stubGlobal( + "fetch", + vi.fn(async (input: string | URL | Request, init?: RequestInit) => + String(input).startsWith("https://studio.decocms.com/") + ? studio.fetch(input, init) + : new Response(null), + ), + ); + vi.spyOn(Math, "random").mockReturnValue(0); + const release = withSettings({ telemetry: { traceSampleRate: 0 } }); + const pointer = studio.draft({ + set: { CMS: { __resolveType: "cms-settings", telemetry: { traceSampleRate: 1 } } }, + }); + const cms = createCMS({ + blocks: docsBlocks(), + content: release, + telemetry: { endpoint: "https://otel.example", limits: { traceSampleRate: 1 } }, + }); + await cms.forRelease().revision(); + expect((await cms.forDraft(pointer).resolve("SummerSEO"))[1]).toBeNull(); + expect(currentTelemetry()?.sampleTrace()).toBe(false); + }); + + it("telemetry follows the release's section before the release's first client runs", async () => { + vi.stubGlobal( + "fetch", + vi.fn(async () => new Response(null)), + ); + vi.spyOn(Math, "random").mockReturnValue(0); + const cms = createCMS({ + blocks: docsBlocks(), + content: withSettings({ telemetry: { traceSampleRate: 1 } }), + telemetry: { endpoint: "https://otel.example", limits: { traceSampleRate: 1 } }, + }); + expect(currentTelemetry()?.sampleTrace()).toBe(false); // the defaults, before any client + await cms.forRelease().revision(); + expect(currentTelemetry()?.sampleTrace()).toBe(true); + }); + + it("a telemetry section with a date rule takes effect within a minute, outside any request", async () => { + vi.stubGlobal( + "fetch", + vi.fn(async () => new Response(null)), + ); + vi.spyOn(Math, "random").mockReturnValue(0); + let now = Date.parse("2026-10-05T12:00:00Z"); + vi.spyOn(Date, "now").mockImplementation(() => now); + const cms = createCMS({ + blocks: docsBlocks(), + content: withSettings({ + telemetry: { + traceSampleRate: { + __resolveType: "multivariate", + variants: [ + { + rule: { __resolveType: "date", start: "2026-10-05T12:00:30Z" }, + value: lazy(1), + }, + { rule: { __resolveType: "always" }, value: lazy(0) }, + ], + }, + }, + }), + telemetry: { endpoint: "https://otel.example", limits: { traceSampleRate: 1 } }, + }); + await cms.forRelease().revision(); + expect(currentTelemetry()?.sampleTrace()).toBe(false); + now += 45_000; + await cms.forRelease().revision(); // within the minute: not read again + expect(currentTelemetry()?.sampleTrace()).toBe(false); + now += 30_000; + await cms.forRelease().revision(); // read again, without holding up the request + await new Promise((resolve) => setTimeout(resolve, 0)); + expect(currentTelemetry()?.sampleTrace()).toBe(true); + }); + + it("a hot reload with a new CMS block takes effect", async () => { + const root = "/tmp/settings-hot-reload/.deco"; + const first = { ...withSettings({ analytics: { enabled: true } }), root }; + const cms = createCMS({ blocks: {}, content: first }); + expect((await cms.settings()).analytics.enabled).toBe(true); + const second = { ...withSettings({ analytics: { enabled: false } }), root }; + expect(instanceOf(createCMS({ blocks: {}, content: second }))).toBe(instanceOf(cms)); + expect((await cms.settings()).analytics.enabled).toBe(false); + }); +}); + +describe("cms.draftPointer and cms.draftCookie on a host outside the list", () => { + const settings = withSettings({ + preview: { hosts: ["staging.example.com", "*.preview.example.com"] }, + }); + const pointer = encodeURIComponent(POINTER); + const cookie = `__deco_draft=${pointer}`; + + it("ignore the parameter and the cookie: the request gets the release, never an error", async () => { + const cms = createCMS({ blocks: docsBlocks(), content: settings }); + for (const url of [ + `https://store.example.com/?__draft=${pointer}`, + `https://preview.example.com/?__draft=${pointer}`, + `https://staging.example.com.attacker.com/?__draft=${pointer}`, + `https://a.preview.example.com.attacker.com/?__draft=${pointer}`, + `https://xstaging.example.com/?__draft=${pointer}`, + ]) { + expect(await cms.draftPointer(request(url)), url).toBeNull(); + expect(await cms.draftCookie(request(url)), url).toBeNull(); + } + // A URL whose host can't be read is outside the list. + const relative = { url: `/relative?__draft=${pointer}`, headers: new Headers() }; + expect(await cms.draftPointer(relative)).toBeNull(); + expect(await cms.draftCookie(relative)).toBeNull(); + expect(await cms.draftPointer(request("https://store.example.com/", cookie))).toBeNull(); + const client = cms.forRelease(); + expect((await client.resolve<{ title: string }>("SummerSEO"))[0]?.title).toBe("Sunny!"); + }); + + it("allow it on a listed host, in any case, with a trailing dot or a port", async () => { + const cms = createCMS({ blocks: docsBlocks(), content: settings }); + for (const url of [ + `https://staging.example.com/?__draft=${pointer}`, + `https://STAGING.example.com./x?__draft=${pointer}`, + `http://staging.example.com:8080/?__draft=${pointer}`, + `https://pr-12.preview.example.com/?__draft=${pointer}`, + ]) { + expect(await cms.draftPointer(request(url)), url).toBe(POINTER); + expect(await cms.draftCookie(request(url)), url).toMatch(/^__deco_draft=/); + } + expect(await cms.draftPointer(request("https://staging.example.com/", cookie))).toBe(POINTER); + }); + + it("?__draft=off still expires the cookie on any host", async () => { + const cms = createCMS({ blocks: docsBlocks(), content: settings }); + expect(await cms.draftCookie(request("https://store.example.com/?__draft=off"))).toMatch( + /Max-Age=0/, + ); + expect( + await cms.draftPointer(request("https://store.example.com/?__draft=off", cookie)), + ).toBeNull(); + }); + + it("a cap in code keeps content from widening the list", async () => { + const cms = createCMS({ + blocks: docsBlocks(), + content: withSettings({ preview: { hosts: ["*", "store.example.com"] } }), + preview: { hosts: ["staging.example.com"] }, + }); + expect((await cms.settings()).preview.hosts).toEqual([]); + expect( + await cms.draftPointer(request(`https://store.example.com/?__draft=${pointer}`)), + ).toBeNull(); + expect( + await cms.draftPointer(request(`https://staging.example.com/?__draft=${pointer}`)), + ).toBeNull(); + }); + + it("a cap in code applies on its own when content has no preview field", async () => { + const cms = createCMS({ + blocks: docsBlocks(), + content: docsSnapshot(), + preview: { hosts: ["localhost:3000"] }, + }); + expect(await cms.draftPointer(request(`http://localhost:3000/?__draft=${pointer}`))).toBe( + POINTER, + ); + expect(await cms.draftPointer(request(`http://localhost:3001/?__draft=${pointer}`))).toBeNull(); + }); +}); diff --git a/packages/blocks/src/v8/settings.ts b/packages/blocks/src/v8/settings.ts new file mode 100644 index 000000000..09364246d --- /dev/null +++ b/packages/blocks/src/v8/settings.ts @@ -0,0 +1,164 @@ +/** + * The CMS settings (see /next/built-in-blocks#cms-settings and + * /next/api-reference#cms-settings): the saved block `CMS`, of the built-in + * type `cms-settings`, read from the release, with the defaults filled in and + * code's caps applied. + * + * - The caller hands in the release snapshot already in memory; nothing here + * loads content, so reading settings never fetches. + * - A missing block, a block of another type, or a section that fails to + * resolve gives that section's defaults, still capped. Nothing rejects. + * - Telemetry rates are the lower of content's and `telemetry.limits`. + * - Preview hosts: content's entries that fall within code's `preview.hosts` + * (all of them when code sets none); without the field, code's list, or + * every host. + */ +import { + analyticsSection, + SETTINGS_BLOCK, + SETTINGS_TYPE, + telemetrySection, +} from "./builtins/data.ts"; +import { + EVERY_HOST, + formatHostPattern, + type HostPattern, + parseHostPattern, + patternWithin, +} from "./hosts.ts"; +import { isPlainObject } from "./json.ts"; +import type { Client, CMSConfig, EffectiveSettings, Snapshot } from "./types.ts"; + +/** What code allows: the caps every section's content is held to. */ +export interface SettingsCaps { + /** Code's `preview.hosts`, parsed; `undefined` when code sets none. */ + hosts: readonly HostPattern[] | undefined; + limits: { errorSampleRate: number; traceSampleRate: number }; +} + +const SECTIONS = ["preview", "telemetry", "analytics"] as const; + +/** + * Code's `preview.hosts`, parsed. Throws a `TypeError` on anything that isn't + * a list of host patterns: that's a bug in the app, not in content. + */ +export function parseCodeHosts(preview: CMSConfig["preview"]): HostPattern[] | undefined { + return parseCodePatterns(preview, "hosts"); +} + +function parseCodePatterns( + preview: CMSConfig["preview"], + field: "hosts", +): HostPattern[] | undefined { + if (preview === undefined || preview === null) return undefined; + if (typeof preview !== "object") { + throw new TypeError("createCMS: `preview` must be an object, such as { hosts: [...] }"); + } + const list = preview[field]; + if (list === undefined) return undefined; + if (!Array.isArray(list)) { + throw new TypeError(`createCMS: \`preview.${field}\` must be a list of host patterns`); + } + return list.map((raw) => { + const pattern = parseHostPattern(raw); + if (pattern === null) { + throw new TypeError( + `createCMS: ${JSON.stringify(raw)} in preview.${field} isn't a host pattern ` + + "(see /next/api-reference#host-patterns)", + ); + } + return pattern; + }); +} + +/** The settings with nothing saved: every default, capped. */ +export function defaultSettings(caps: SettingsCaps): EffectiveSettings { + return effectiveSettings({}, caps); +} + +/** + * The settings a release holds. `client` reads that same release; it resolves + * the `CMS` block whole and, if that fails, each section on its own, so one + * failing section falls back to its defaults alone. + */ +export async function readSettings( + snapshot: Snapshot | undefined, + client: () => Client, + caps: SettingsCaps, +): Promise { + try { + const entry = snapshot?.blocks[SETTINGS_BLOCK]; + if (!isPlainObject(entry) || entry.__resolveType !== SETTINGS_TYPE) { + return defaultSettings(caps); + } + const reader = client(); + const [whole] = await reader.resolve>(SETTINGS_BLOCK); + if (isPlainObject(whole)) return effectiveSettings(whole, caps); + const sections: Record = {}; + await Promise.all( + SECTIONS.map(async (name) => { + if (entry[name] === undefined) return; + const [value] = await reader.resolve(entry[name]); + sections[name] = value ?? undefined; // a section that fails gets its defaults + }), + ); + return effectiveSettings(sections, caps); + } catch { + return defaultSettings(caps); + } +} + +/** Whether a saved block holds nothing to run, so its settings are the same on every call. */ +export function isStatic(entry: Record): boolean { + return Object.entries(entry).every( + ([key, value]) => key === "__resolveType" || !holdsBlock(value), + ); +} + +function holdsBlock(value: unknown): boolean { + if (Array.isArray(value)) return value.some(holdsBlock); + if (!isPlainObject(value)) return false; + return "__resolveType" in value || Object.values(value).some(holdsBlock); +} + +/** + * The settings, deep-frozen: one object per release is handed to every caller, + * so a caller that writes to it would change it for every request. + */ +function effectiveSettings(value: Record, caps: SettingsCaps): EffectiveSettings { + const telemetry = telemetrySection(value.telemetry); + return deepFreeze({ + preview: { hosts: effectiveHosts(value.preview, caps.hosts) }, + telemetry: { + ...telemetry, + errorSampleRate: Math.min(telemetry.errorSampleRate, caps.limits.errorSampleRate), + traceSampleRate: Math.min(telemetry.traceSampleRate, caps.limits.traceSampleRate), + }, + analytics: analyticsSection(value.analytics), + }); +} + +function deepFreeze(value: T): T { + if (value !== null && typeof value === "object" && !Object.isFrozen(value)) { + Object.freeze(value); + for (const child of Object.values(value)) deepFreeze(child); + } + return value; +} + +function effectiveHosts(preview: unknown, code: readonly HostPattern[] | undefined): string[] { + const saved = isPlainObject(preview) && Array.isArray(preview.hosts) ? preview.hosts : undefined; + if (saved === undefined) return code ? unique(code.map(formatHostPattern)) : [EVERY_HOST]; + const kept: string[] = []; + for (const raw of saved) { + const pattern = parseHostPattern(raw); + if (pattern === null) continue; // not a pattern: left out + if (code && !code.some((cap) => patternWithin(pattern, cap))) continue; + kept.push(formatHostPattern(pattern)); + } + return unique(kept); +} + +function unique(values: string[]): string[] { + return [...new Set(values)]; +} diff --git a/packages/blocks/src/v8/telemetry.test.ts b/packages/blocks/src/v8/telemetry.test.ts index 0d2b14357..19d750ae5 100644 --- a/packages/blocks/src/v8/telemetry.test.ts +++ b/packages/blocks/src/v8/telemetry.test.ts @@ -74,7 +74,7 @@ function withTelemetryBlock(block: Record): Snapshot { const snapshot = docsSnapshot(); return { ...snapshot, - blocks: { ...snapshot.blocks, Telemetry: { __resolveType: "telemetry", ...block } }, + blocks: { ...snapshot.blocks, CMS: { __resolveType: "cms-settings", telemetry: block } }, }; } @@ -114,6 +114,38 @@ describe("where telemetry goes", () => { }); }); + it("reads v7's DECO_OTEL_* names as aliases; the standard name wins when both are set", () => { + vi.stubEnv("DECO_OTEL_METRICS_ENDPOINT", `${ENDPOINT}/v1/metrics`); + vi.stubEnv("DECO_OTEL_LOGS_ENDPOINT", `${ENDPOINT}/v1/logs`); + vi.stubEnv("DECO_OTEL_TRACES_ENDPOINT", `${ENDPOINT}/v1/traces`); + vi.stubEnv("DECO_OTEL_HEADERS", "x-team=v7"); + vi.stubEnv("DECO_OTEL_AUTH_TOKEN", "Bearer v7"); + expect(resolveDestination(undefined)).toMatchObject({ + endpoint: "", + signals: { + metrics: `${ENDPOINT}/v1/metrics`, + logs: `${ENDPOINT}/v1/logs`, + traces: `${ENDPOINT}/v1/traces`, + }, + headers: { "x-team": "v7", authorization: "Bearer v7" }, + }); + vi.stubEnv("OTEL_EXPORTER_OTLP_METRICS_ENDPOINT", "https://std.example/m"); + vi.stubEnv("OTEL_EXPORTER_OTLP_HEADERS", "x-team=std,authorization=Bearer%20std"); + expect(resolveDestination(undefined)).toMatchObject({ + signals: { metrics: "https://std.example/m", logs: `${ENDPOINT}/v1/logs` }, + headers: { "x-team": "std", authorization: "Bearer std" }, + }); + }); + + it("sends a signal to its own URL, and drops a signal with no destination", async () => { + const { sent } = collector(); + vi.stubEnv("DECO_OTEL_METRICS_ENDPOINT", "https://ingest.example/v1/metrics"); + createCMS({ blocks: docsBlocks(), content: docsSnapshot() }); + await upstream("https://search.example/q"); + await runBackground(); + expect(sent.map((s) => s.url)).toEqual(["https://ingest.example/v1/metrics"]); + }); + it("top-level site and token never turn telemetry on", async () => { const { fetch } = collector(); createCMS({ blocks: docsBlocks(), content: docsSnapshot(), site: "acme", token: "tok" }); diff --git a/packages/blocks/src/v8/telemetry.ts b/packages/blocks/src/v8/telemetry.ts index a4e7fe0a3..84c1057c9 100644 --- a/packages/blocks/src/v8/telemetry.ts +++ b/packages/blocks/src/v8/telemetry.ts @@ -3,10 +3,14 @@ * /next/telemetry and /next/telemetry-internals. * * - **Where** comes from code: `createCMS({ telemetry })`, or the standard - * `OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_EXPORTER_OTLP_HEADERS` when it's left - * out. `false` sends nothing. - * - **How much** comes from content: the `Telemetry` saved block of the - * release being served, capped by `telemetry.limits`. + * `OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_EXPORTER_OTLP_HEADERS` (and the + * per-signal `OTEL_EXPORTER_OTLP_{METRICS,LOGS,TRACES}_ENDPOINT`) when it's + * left out. v7's `DECO_OTEL_*` names are read as aliases; the standard name + * wins when both are set. `false` sends nothing. + * - **How much** comes from content: the `telemetry` section of the CMS + * settings (`cms.settings()`, the release's `CMS` block), capped by + * `telemetry.limits`. The CMS reads it outside any request, when the + * release changes and then at most once a minute. * - Metrics are aggregated in memory (delta histograms); error logs and traces * are sampled. Batches leave in the background (after the response on * Workers, on an unref'd timer elsewhere); a send that fails is retried @@ -15,10 +19,10 @@ * cookies or authorization values, and no value a secret block decrypted. */ import { hasBackgroundHook, later, runInBackground } from "./background.ts"; +import { rate, TELEMETRY_DEFAULTS } from "./builtins/data.ts"; import { isResolutionError } from "./errors.ts"; import { readEnv } from "./identity.ts"; -import { isPlainObject } from "./json.ts"; -import type { CMSError, Snapshot, TelemetryConfig } from "./types.ts"; +import type { CMSError, Snapshot, Telemetry, TelemetryConfig } from "./types.ts"; /** The hosted Deco CMS collector, for `telemetry: { site, token }`. */ const HOSTED_TELEMETRY_ENDPOINT = "https://otel.decocms.com"; @@ -31,13 +35,29 @@ const MAX_SECRETS = 1_000; const RETRY_STATUSES = new Set([429, 502, 503, 504]); /** OpenTelemetry's recommended `http.*.request.duration` buckets, in seconds. */ const BOUNDS = [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10]; -const DEFAULTS = { enabled: true, metrics: true, errorSampleRate: 0.05, traceSampleRate: 0 }; const DEFAULT_LIMITS = { errorSampleRate: 0.1, traceSampleRate: 0 }; +/** The caps on content's sample rates: `telemetry.limits`, defaults 0.1 and 0. */ +export function telemetryLimits(config: false | TelemetryConfig | undefined): { + errorSampleRate: number; + traceSampleRate: number; +} { + const limits = config ? config.limits : undefined; + return { + errorSampleRate: rate(limits?.errorSampleRate, DEFAULT_LIMITS.errorSampleRate), + traceSampleRate: rate(limits?.traceSampleRate, DEFAULT_LIMITS.traceSampleRate), + }; +} + type Attributes = Record; +type Signal = "metrics" | "logs" | "traces"; + interface Destination { + /** The base URL each signal's `/v1/` is appended to; `""` with per-signal URLs only. */ endpoint: string; + /** Full per-signal URLs, used as they are (the `OTEL_EXPORTER_OTLP__ENDPOINT` form). */ + signals?: Partial>; headers: Record; site?: string; limits: { errorSampleRate: number; traceSampleRate: number }; @@ -86,16 +106,8 @@ export function resolveDestination( site?: string, ): Destination | null { if (config === false) return null; - if (config === undefined || config === null) { - const endpoint = readEnv("OTEL_EXPORTER_OTLP_ENDPOINT"); - if (!endpoint) return null; - const headers = parseKeyValues(readEnv("OTEL_EXPORTER_OTLP_HEADERS")); - return { endpoint, headers, site, limits: DEFAULT_LIMITS }; - } - const limits = { - errorSampleRate: rate(config.limits?.errorSampleRate, DEFAULT_LIMITS.errorSampleRate), - traceSampleRate: rate(config.limits?.traceSampleRate, DEFAULT_LIMITS.traceSampleRate), - }; + if (config === undefined || config === null) return destinationFromEnv(site); + const limits = telemetryLimits(config); if ("endpoint" in config) { if (typeof config.endpoint !== "string" || !config.endpoint) return null; return { endpoint: config.endpoint, headers: { ...config.headers }, site, limits }; @@ -109,10 +121,49 @@ export function resolveDestination( }; } +/** v7's names for the standard OTLP variables, read when the standard one is unset. */ +const V7_ALIASES = { + OTEL_EXPORTER_OTLP_HEADERS: "DECO_OTEL_HEADERS", + OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: "DECO_OTEL_METRICS_ENDPOINT", + OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: "DECO_OTEL_LOGS_ENDPOINT", + OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "DECO_OTEL_TRACES_ENDPOINT", +} as const; + +/** The standard variable, else its v7 alias. */ +function readOtlpEnv(name: keyof typeof V7_ALIASES | "OTEL_EXPORTER_OTLP_ENDPOINT") { + return ( + readEnv(name) || + (name in V7_ALIASES ? readEnv(V7_ALIASES[name as keyof typeof V7_ALIASES]) : undefined) + ); +} + +/** + * The destination the environment names: the standard OTLP variables, with + * v7's `DECO_OTEL_*` names as aliases. `DECO_OTEL_AUTH_TOKEN` (v7's secret + * for the collector) becomes the `authorization` header unless the headers + * set one. `null` when no endpoint is set. + */ +function destinationFromEnv(site: string | undefined): Destination | null { + const endpoint = readOtlpEnv("OTEL_EXPORTER_OTLP_ENDPOINT") ?? ""; + const signals: Partial> = {}; + for (const signal of ["metrics", "logs", "traces"] as const) { + const url = readOtlpEnv( + `OTEL_EXPORTER_OTLP_${signal.toUpperCase() as Uppercase}_ENDPOINT`, + ); + if (url) signals[signal] = url; + } + if (!endpoint && Object.keys(signals).length === 0) return null; + const token = readEnv("DECO_OTEL_AUTH_TOKEN"); + const headers = { + ...(token ? { authorization: token } : {}), + ...parseKeyValues(readOtlpEnv("OTEL_EXPORTER_OTLP_HEADERS")), + }; + return { endpoint, signals, headers, site, limits: DEFAULT_LIMITS }; +} + export class TelemetryPipeline { readonly #destination: Destination; - #settings = { ...DEFAULTS }; - #settingsRevision: string | undefined; + #settings: Required = { ...TELEMETRY_DEFAULTS }; #release: string | undefined; #histograms = new Map(); #logs: LogRecord[] = []; @@ -124,7 +175,7 @@ export class TelemetryPipeline { constructor(destination: Destination) { this.#destination = destination; - this.#apply(undefined); + this.apply(TELEMETRY_DEFAULTS); } /** Never send `value`: it's replaced with `[redacted]` in every log, span and label. */ @@ -132,13 +183,29 @@ export class TelemetryPipeline { if (value.length > 0 && this.#secrets.size < MAX_SECRETS) this.#secrets.add(value); } - /** The release being served: its revision labels batches and its `Telemetry` block sets the rates. */ + /** The release being served: its revision labels batches. */ useRelease(snapshot: Snapshot): void { this.#release = snapshot.revision; - if (this.#settingsRevision === snapshot.revision) return; - this.#settingsRevision = snapshot.revision; - const entry = snapshot.blocks.Telemetry; - this.#apply(isPlainObject(entry) && entry.__resolveType === "telemetry" ? entry : undefined); + } + + /** + * The `telemetry` section to follow (from `cms.settings()`). Rates are + * capped by the destination's limits here too, whatever the caller passed. + */ + apply(section: Telemetry): void { + const { limits } = this.#destination; + this.#settings = { + enabled: typeof section.enabled === "boolean" ? section.enabled : TELEMETRY_DEFAULTS.enabled, + metrics: typeof section.metrics === "boolean" ? section.metrics : TELEMETRY_DEFAULTS.metrics, + errorSampleRate: Math.min( + rate(section.errorSampleRate, TELEMETRY_DEFAULTS.errorSampleRate), + limits.errorSampleRate, + ), + traceSampleRate: Math.min( + rate(section.traceSampleRate, TELEMETRY_DEFAULTS.traceSampleRate), + limits.traceSampleRate, + ), + }; } histogram(name: string, attributes: Attributes, seconds: number): void { @@ -188,7 +255,7 @@ export class TelemetryPipeline { /** * The hooks one client reports through. The trace decision is made once per * client, on its first block, after its release is loaded, so the first - * client of a release already uses that release's `Telemetry` block. + * client of a release already uses that release's `telemetry` section. */ forClient(): ClientTelemetry { let decided = false; @@ -248,7 +315,7 @@ export class TelemetryPipeline { const byName = new Map(); for (const h of histograms) byName.set(h.name, [...(byName.get(h.name) ?? []), h]); sends.push( - this.#send("/v1/metrics", { + this.#send("metrics", { resourceMetrics: [ { resource, @@ -282,7 +349,7 @@ export class TelemetryPipeline { } if (logs.length > 0) { sends.push( - this.#send("/v1/logs", { + this.#send("logs", { resourceLogs: [ { resource, @@ -306,7 +373,7 @@ export class TelemetryPipeline { if (spans.length > 0) { const kinds = { internal: 1, server: 2, client: 3 }; sends.push( - this.#send("/v1/traces", { + this.#send("traces", { resourceSpans: [ { resource, @@ -340,22 +407,6 @@ export class TelemetryPipeline { return scrubText(out); } - #apply(block: Record | undefined): void { - const { limits } = this.#destination; - this.#settings = { - enabled: typeof block?.enabled === "boolean" ? block.enabled : DEFAULTS.enabled, - metrics: typeof block?.metrics === "boolean" ? block.metrics : DEFAULTS.metrics, - errorSampleRate: Math.min( - rate(block?.errorSampleRate, DEFAULTS.errorSampleRate), - limits.errorSampleRate, - ), - traceSampleRate: Math.min( - rate(block?.traceSampleRate, DEFAULTS.traceSampleRate), - limits.traceSampleRate, - ), - }; - } - #schedule(): void { if (this.#scheduled) return; this.#scheduled = true; @@ -392,8 +443,11 @@ export class TelemetryPipeline { }; } - async #send(path: string, payload: unknown): Promise { - const url = this.#destination.endpoint.replace(/\/+$/, "") + path; + async #send(signal: Signal, payload: unknown): Promise { + const { endpoint, signals } = this.#destination; + const url = + signals?.[signal] ?? (endpoint ? `${endpoint.replace(/\/+$/, "")}/v1/${signal}` : ""); + if (!url) return; // this signal has no destination const { body, gzipped } = await gzip(JSON.stringify(payload)); const headers: Record = { ...this.#destination.headers, @@ -483,12 +537,6 @@ function randomHex(bytes: number): string { return Array.from(values, (b) => b.toString(16).padStart(2, "0")).join(""); } -function rate(value: unknown, fallback: number): number { - return typeof value === "number" && Number.isFinite(value) - ? Math.min(1, Math.max(0, value)) - : fallback; -} - /** `k1=v1,k2=v2` with URL-encoded values, the format of the OTEL_* environment variables. */ function parseKeyValues(raw: string | undefined): Record { const out: Record = {}; diff --git a/packages/blocks/src/v8/testFixtures.ts b/packages/blocks/src/v8/testFixtures.ts index 91b0edeb9..98d6cf19b 100644 --- a/packages/blocks/src/v8/testFixtures.ts +++ b/packages/blocks/src/v8/testFixtures.ts @@ -84,3 +84,54 @@ export function docsSnapshot(revision = "rev-1"): Snapshot { }, }; } + +/** The host and token of the fake Studio API below. */ +export const STUDIO_HOST = "studio.decocms.com"; +export const STUDIO_TOKEN = "signed-token"; + +/** + * A fake Studio API answering draft pointers the way the docs describe + * (/next/content-delivery#draft-previews): per draft branch, the changes + * compared with production, behind the token Studio signed. Hand `fetch` to + * `vi.stubGlobal("fetch", …)`. + */ +export function fakeStudio() { + const branches = new Map; delete?: string[] }>(); + const overrides = new Map Response>(); + const requests: { url: string; init: RequestInit | undefined }[] = []; + const PATH = /^\/api\/acme\/decofile\/store\/([^/]+)\/changes$/; + const fetch = async (input: string | URL | Request, init?: RequestInit): Promise => { + const url = new URL(String(input)); + requests.push({ url: url.href, init }); + const branch = PATH.exec(url.pathname)?.[1]; + if (url.host !== STUDIO_HOST || branch === undefined) { + return new Response("not found", { status: 404 }); + } + const override = overrides.get(branch); + if (override) return override(); + if (url.searchParams.get("token") !== STUDIO_TOKEN) { + return Response.json({ error: "invalid token" }, { status: 401 }); + } + const changes = branches.get(branch) ?? {}; + return Response.json( + { format: 1, set: changes.set ?? {}, delete: changes.delete ?? [] }, + { headers: { "cache-control": "no-store", "access-control-allow-origin": "*" } }, + ); + }; + return { + fetch, + requests, + /** Saves a draft branch's changes and returns the pointer Studio would mint for them. */ + draft( + changes: { set?: Record; delete?: string[] }, + { branch = "summer-sale", version = "9f3c1a", token = STUDIO_TOKEN } = {}, + ): string { + branches.set(branch, changes); + return `${STUDIO_HOST}/api/acme/decofile/store/${branch}/changes?token=${token}@${version}`; + }, + /** Answers a branch with this response instead. */ + respond(branch: string, response: () => Response) { + overrides.set(branch, response); + }, + }; +} diff --git a/packages/blocks/src/v8/types.ts b/packages/blocks/src/v8/types.ts index ff7d0323b..5620c1a67 100644 --- a/packages/blocks/src/v8/types.ts +++ b/packages/blocks/src/v8/types.ts @@ -37,19 +37,22 @@ export type Snapshot = { */ export type ContentModule = Snapshot & { root?: string }; -/** A source of content. `load()` is the release; `load(pointer)` is a draft. */ +/** + * A source of content: `load()` returns it. Drafts aren't a loader's job: + * `forDraft` layers a draft's changes over what `load()` returned. + */ export interface Loader { - load(pointer?: string | null): Promise; + load(): Promise; update?(): Promise<{ updated: boolean }>; } /** The parts of a draft pointer, `@`. */ export interface DraftPointer { - /** `host[:port]` of the content source that holds the draft. */ + /** `host[:port]` of the Studio API that serves the draft's changes. */ host: string; /** Starts with `/`; opaque to the app. Never carries the `__variant` parameters. */ path: string; - /** Opaque and immutable: the branch head or ETag. */ + /** Opaque: the commit of the editor's last save. `"local"` names no draft (`deco serve`). */ version: string; /** The variants this preview forces (the query's `__variant` parameters); absent when none. */ variants?: ForcedVariant[]; @@ -130,7 +133,28 @@ export interface Variant { value: Lazy; } -/** The `Telemetry` saved block's type (built-in `telemetry`). */ +/** + * The built-in `cms-settings` block's props: the type of the well-known saved + * block `CMS`. Every field is optional, and any field can have variants. Read + * it through `cms.settings()`, which fills in the defaults and applies code's + * caps (see /next/built-in-blocks#cms-settings). + */ +export interface CMSSettings { + /** Host patterns previews are allowed on, within `createCMS`'s `preview.hosts`. */ + preview?: { hosts?: string[] }; + telemetry?: Telemetry; + analytics?: Analytics; +} + +/** What `cms.settings()` returns: every section, defaults filled in, caps applied. */ +export interface EffectiveSettings { + /** `["*"]` means every host; an empty list, none. */ + preview: { hosts: string[] }; + telemetry: Required; + analytics: Required; +} + +/** The `telemetry` section of the CMS settings. */ export interface Telemetry { /** Default `true`; `false` switches telemetry off. */ enabled?: boolean; @@ -142,7 +166,7 @@ export interface Telemetry { traceSampleRate?: number; } -/** The built-in `analytics` block's props. Separate from telemetry. */ +/** The `analytics` section of the CMS settings, and `AnalyticsScript`'s props. */ export interface Analytics { /** An endpoint that accepts the One Dollar Stats format; default: the hosted Deco CMS collector. */ collector?: string; @@ -150,6 +174,9 @@ export interface Analytics { enabled?: boolean; } +/** Anything with a `url` and `headers`: a fetch `Request`, a `NextRequest`, a framework wrapper. */ +export type RequestLike = Request | { url: string; headers: { get(name: string): string | null } }; + // --------------------------------------------------------------------------- // Results and errors // --------------------------------------------------------------------------- @@ -201,11 +228,18 @@ export interface CMSConfig { interval?: number; /** Where telemetry goes; see /next/telemetry. */ telemetry?: false | TelemetryConfig; + preview?: { + /** + * The most content may allow previews on, in the host pattern format + * (/next/api-reference#host-patterns). Without it, content may allow any host. + */ + hosts?: string[]; + }; /** The private key that decrypts `secret` blocks. */ secrets?: { key?: string }; - /** Your site's ID, for hosted releases and drafts. */ + /** Your site's ID, for hosted releases. Drafts don't need it. */ site?: string; - /** Your site token (secret), for hosted releases and drafts. */ + /** Your site token (secret), for hosted releases. Drafts don't need it. */ token?: string; } @@ -244,10 +278,32 @@ export interface Client { export interface CMS { /** A client reading the current release. */ forRelease(): Client; - /** A client reading the draft a pointer names. */ + /** + * A client reading the draft a pointer names: its changes, fetched from a + * preview API domain (`*.decocms.com` and loopback by default; + * `DECO_PREVIEW_API_DOMAINS` replaces the list), over this server's + * production content. + */ forDraft(pointer: string): Client; /** A client pinned to a revision this CMS has served; an unknown revision behaves like the release. */ forRevision(revision: string): Client; /** Ask the content source for newer content now; never throws. */ update(): Promise<{ updated: boolean }>; + /** + * The release's CMS settings (the saved block `CMS`), defaults filled in and + * code's caps applied. Reads the release already in memory, never a draft; + * never fetches, never rejects. + */ + settings(): Promise; + /** + * The request's draft pointer: `?__draft=` first, then the `__deco_draft` + * cookie. `null` when neither is present, for `?__draft=off`, and on a host + * outside `settings().preview.hosts`, where the request gets the release. + */ + draftPointer(request: RequestLike): Promise; + /** + * The `Set-Cookie` value that starts a preview (a valid `?__draft=` on an + * allowed host) or ends one (`?__draft=off`, on any host); `null` otherwise. + */ + draftCookie(request: RequestLike): Promise; }