Read .zarr folders and plain SpatialData stores in the serverless viewer

DESIGN.md

@@ -1343,9 +1343,10 @@ A checkpoint is **directly readable by a browser** over HTTP Range, with no back
 (§14.2). `ZIP_STORED` is what makes that possible — a zarr chunk, and equally a shape
 parquet, is a contiguous byte span — and four write-time steps in
 `_write_browser_reader_support` serve it. The first three are what the serverless viewer
-needs to exist at all, not an optimization: a checkpoint written before them is rejected
-on open (§14.2), because a Zarr v3 store carries no child index and the reader could not
-even name the table.
+needs to exist at all, not an optimization: without them the reader falls back to reading
+the store as plain SpatialData (§14.2), which draws no boundaries and reads genes from the
+whole matrix, and without consolidated metadata it could not even name the table, because
+a Zarr v3 store carries no child index.
 
 - **`_shard_rasters`** rewrites image/label arrays with the Zarr v3 sharding codec
   (inner chunk `_SHARD_INNER` 512, shard `_SHARD_SIZE` 4096), region-by-region so
@@ -1435,6 +1436,18 @@ call `api.ts` directly — `useArrowField`, `useVivImageLayer`, `usePolygonBbox`
 Everything downstream (palettes, point styling, channel shaders, legends, minimap,
 invert axes, backdrop) already worked off plain typed arrays and is untouched.
 
+The same reader opens a `.zarr/` folder (a URL whose path ends in `/`), and a store this
+app never saved. A folder is read one object per key — `FetchStore` for a public folder,
+or `HostSignedFolderStore` under an embed host whose bucket needs a presigned URL per
+object (docs/EMBED_PROTOCOL.md "Folder stores"). A store without the `viewer/` sidecar
+(nf-core/sopa or spatialdata-io output) gets one derived in the browser by
+`plainSpatialData.deriveSidecar`: the first table with `obsm/spatial`, and each image's
+manifest built from its OME metadata, placed against the cells the way `imaging.pixel_to_world`
+reconciles them. Contrast defaults come from the coarsest level, as `_channel_norm` computes
+them. Its default displays follow `manager.auto_displays`. What the backend bakes and
+the browser cannot cheaply derive stays missing: the shapes spatial index (so no boundaries)
+and the CSC gene mirror (so a gene reads the whole CSR matrix).
+
 Details that make it work:
 
 - `checkpointSource` materializes the **same Arrow schemas** `transport/arrow.py`

DEVELOPMENT.md

@@ -119,7 +119,11 @@ packages/viewer/  @cirrobio/spatial-viewer — the deck.gl canvases and the chec
                   helpers a host's own controls need
   src/data/       the DataSource contract the canvas renders through, the DataSourceProvider,
                   and checkpointSource (a .zarr.zip read directly with zarrita over HTTP
-                  Range — the serverless viewer, DESIGN §14.2). parquetShapes.ts +
+                  Range, or a .zarr/ folder — the serverless viewer, DESIGN §14.2).
+                  folderStore.ts reads a folder whose objects an embed host signs one by
+                  one (docs/EMBED_PROTOCOL.md "Folder stores"); plainSpatialData.ts derives
+                  the viewer sidecar and default displays for a store this app did not
+                  save (sopa, spatialdata-io output). parquetShapes.ts +
                   wkbGeoArrow.ts are the boundary half: the shape file is GeoParquet, not
                   zarr, so it is range-queried with hyparquet against its covering index
   src/types.ts    the display model (DisplaySpec/DisplayEncoding/SessionFields/ImageInfo)
@@ -181,6 +185,8 @@ Component-level notes: [`backend/README.md`](backend/README.md),
 | Change where a save writes, what the file is named, or what the session is called | `backend/app/main.py` (`_validated_destination`, `_validated_name`) + `deps.py` (`default_save_path`) + `sessions/session.py` (`rename`, `_run_load`) + `frontend/src/components/SaveCheckpointDialog.tsx` | below |
 | Change how rendered plot figures are stored, served or shown | `backend/app/persistence/store.py` (`_write_figures`, `read_figure`, `figure_index`) + `sessions/session.py` (`figure`, `figure_index`, `figures_to_persist`) + `frontend/src/lib/figures.ts` + `components/PlotGallery.tsx` / `FigureLightbox.tsx` / `PlotDetail.tsx` | below |
 | Change what the serverless viewer can read from a checkpoint | `backend/app/persistence/store.py` (`_write_viewer_sidecar`, the writer half) + `packages/viewer/src/data/checkpointSource.ts` (the reader half) — the two must move together | [DESIGN.md](DESIGN.md) §14.1–14.2, [docs/CHECKPOINT_FORMAT.md](docs/CHECKPOINT_FORMAT.md) §4 |
+| Change how a store without the sidecar (plain SpatialData) is read — which table, where images sit, the default displays | `packages/viewer/src/data/plainSpatialData.ts` (its image placement ports `imaging.pixel_to_world`, its displays `manager.auto_displays`; keep them agreeing) + `plainSpatialData.test.ts` | [docs/CHECKPOINT_FORMAT.md](docs/CHECKPOINT_FORMAT.md) §9 |
+| Change how a `.zarr/` folder is read under an embed host (signing, listing) | `packages/viewer/src/data/folderStore.ts` + `frontend/src/data/embedBridge.ts` (`embedFolderAccess`) + [docs/EMBED_PROTOCOL.md](docs/EMBED_PROTOCOL.md) | [docs/EMBED_PROTOCOL.md](docs/EMBED_PROTOCOL.md) |
 | Change how cell boundaries are indexed or range-queried | `backend/app/persistence/store.py` (`_index_shapes`, `_row_group_rows`, `_selectivity` — the writer half) + `packages/viewer/src/data/parquetShapes.ts` and `wkbGeoArrow.ts` (the reader half). A change to the on-disk index must keep `test_e2e.run_shape_index_check` passing: it re-derives the pruning from the file and compares it against a brute-force row scan | [DESIGN.md](DESIGN.md) §14.1–14.2, [docs/CHECKPOINT_FORMAT.md](docs/CHECKPOINT_FORMAT.md) §4.4 |
 | Change the shape of `app_state`, the `viewer/` sidecar, `X_csc`, or `index.json` | `backend/app/schemas/checkpoint/*.schema.json` (the JSON Schema is validated against on every write) + [docs/CHECKPOINT_FORMAT.md](docs/CHECKPOINT_FORMAT.md) in the same commit — `sds-governance/checks/check_checkpoint_schema_docs.py` fails the build otherwise | [docs/CHECKPOINT_FORMAT.md](docs/CHECKPOINT_FORMAT.md) |
 | Add a render-path call the canvas makes | `packages/viewer/src/data/types.ts` (the `DataSource` interface), then **both** `frontend/src/data/apiSource.ts` and `packages/viewer/src/data/checkpointSource.ts` | [DESIGN.md](DESIGN.md) §14.2 |

docs/CHECKPOINT_FORMAT.md

@@ -647,11 +647,14 @@ granularities:
   a backend-less reader has no data to fall back to if the sidecar it finds is
   newer than the version it understands, so this app's own reader **refuses**
   to open a checkpoint whose `sidecar_version` exceeds what it was built
-  against, rather than guessing at an unknown shape. A checkpoint with **no**
-  `viewer/` group at all (written before the sidecar existed) is refused
-  outright by a backend-less reader for the same reason — Zarr v3 has no child
-  index, so without the sidecar's `table_keys` a reader can't even enumerate
-  what tables exist. Additive keys do **not** bump this version: `figures`
+  against, rather than guessing at an unknown shape. A store with **no**
+  `viewer/` group at all (written before the sidecar existed, or not written by
+  this app) is read as plain SpatialData: the backend-less reader derives what the
+  sidecar would have said from the consolidated metadata (the first table with
+  `obsm/spatial`, each image's manifest, identity `coords_transform`; see
+  `packages/viewer/src/data/plainSpatialData.ts`), and refuses the store only when
+  that finds nothing to draw. Consolidated metadata is then required, because Zarr
+  v3 has no child index to enumerate tables from otherwise. Additive keys do **not** bump this version: `figures`
   (§4.3) was added without one, because an older reader ignoring a key it does
   not know still renders the file correctly, while a bump would make it refuse
   a file it could have read.

docs/EMBED_PROTOCOL.md

@@ -60,11 +60,17 @@ config (`SpatialDataDisplay` in the dashboard package) — same field names
     displays: Array<{ id: string; name: string } & DisplayPayload>, // saved displays in app_state order
     obsColumns: Array<{ name: string; kind: 'categorical' | 'numeric' }>,
     images: Array<{ element: string; channelNames: string[]; isRgb: boolean;
-                    contrastRange: [number, number][] }>,
+                    contrastRange: [number, number][];     // per channel [min, max] of the data
+                    contrastLimits: [number, number][] }>, // per channel default contrast (1.1.0+)
     obsmKeys: Array<{ key: string; nComponents: number }>,
+    shapes: string[],  // polygon shape elements the canvas can draw as boundaries (1.1.0+)
   }
 }
 ```
+   `contrastLimits` is the contrast a channel shows when the display sets none, so a
+   host's contrast control can start where the canvas does. `shapes` lists the boundary
+   sets a display's `shapes_layer` may name. Both were added in 1.1.0; a host should treat
+   them as optional when it may talk to an older viewer.
 2. `display-changed` — debounced (<=500ms) whenever the ACTIVE display's
    encoding or viewport changes in-iframe (user pans/zooms or uses in-canvas
    controls):
@@ -124,6 +130,41 @@ not retried forever. Concurrent reads that all expire at the same moment share
 one re-sign rather than each asking for their own, and a request that goes
 unanswered for 15s rejects.
 
+### Folder stores
+
+A checkpoint URL whose **path** ends in `/` names a `.zarr/` folder rather than a
+`.zarr.zip` (1.1.0+). An object store has no single presigned URL for a folder, so in embed
+mode the viewer asks the host to sign each object key and to list the folder. Keys and
+prefixes are relative to the store root. The folder URL itself is never fetched, so any
+URL under the folder with a trailing-slash path works (a presign of the folder key is
+convenient).
+
+Viewer -> parent:
+```ts
+{ source: 'sds-embed', version: 1, type: 'sign-keys', requestId: string, keys: string[] }
+{ source: 'sds-embed', version: 1, type: 'list-keys', requestId: string, prefix: string }
+```
+
+Parent -> viewer:
+```ts
+{ source: 'cirro-dashboard', version: 1, type: 'signed-keys',
+  requestId: string, urls: string[] | null }   // same order as keys; null = failed
+{ source: 'cirro-dashboard', version: 1, type: 'listed-keys',
+  requestId: string, keys: string[] | null }   // every key under prefix; null = failed
+```
+
+The viewer lists the whole folder once (`prefix: ''`) when it opens, and answers a key
+absent from that listing as missing without fetching it. That keeps S3's 403-for-a-missing-key
+(a presigner without ListBucket) from reading as an expired signature. It batches the
+keys requested in one tick into one `sign-keys`, reuses a signature for four minutes, and
+re-signs a listed key once if a GET answers 401/403. Requests unanswered for 15s reject.
+
+A folder need not have been saved by this app. The viewer opens any consolidated Zarr v3
+SpatialData store and derives the table, image manifests and default displays itself
+(`packages/viewer/src/data/plainSpatialData.ts`). When the store holds nothing it can show
+(no table with `obsm/spatial`, Zarr v2, no consolidated metadata, not zarr at all), it
+posts `error` with a message saying which.
+
 ## Handshake order
 
 1. Parent creates iframe with `embed=1`.
@@ -136,26 +177,41 @@ unanswered for 15s rejects.
 
 ## Dashboard node contract (implemented in @cirrobio/dashboard)
 
-- Node type id: `'spatialdata'` (NODE_TYPE.spatialdata).
+The host side lives in Cirro-portal's `packages/dashboard/src/views/spatialdata/`. The
+viewer itself is deployed as the Cirro-tools `spatialdata` tool (this repo's release
+`viewer-dist.tar.gz`, served unmodified at `/tools/spatialdata/`).
+
+- Node type id: `'spatialdata'` (`NODE_TYPE.spatialData`).
 - Config type `SpatialDataConfig`:
 ```ts
 interface SpatialDataConfig {
+  title: string;
   datasetId: string;
-  datasetName?: string;
-  path: string;            // dataset-relative path to the .zarr.zip
-  sizeBytes?: number;
-  title?: string;
+  datasetName: string;
+  path: string;            // dataset-relative path to the .zarr.zip or .zarr folder
   display?: DisplayPayload & { id?: string };  // persisted display settings
 }
 ```
-- New OPTIONAL host capability in SqlHostCapabilities:
+  A node added from the dashboard's "+ Spatial" picker has no `display` until the viewer
+  first opens, and then saves the one it starts on.
+- Optional host capability in `SqlHostCapabilities`:
 ```ts
-/** Base URL of a deployed Spatial Data Studio serverless viewer build
- *  (directory containing index.html). When absent, spatialdata nodes render
- *  an explanatory placeholder instead of an iframe. */
-spatialViewerUrl?: () => Promise<string> | string;
+/** Directory of a deployed Spatial Data Studio viewer (holding index.html). When
+ *  absent, or when datasetFileUrl is absent, spatialdata nodes render an explanatory
+ *  placeholder instead of an iframe. */
+readonly spatialViewerUrl?: string;
 ```
-- File detection predicate (do NOT widen isTabularFile):
+  The checkpoint URL, and every `refresh-checkpoint-url` answer, is signed through the
+  existing `datasetFileUrl(projectId, datasetId, path)` capability.
+- File and folder detection (do NOT widen isTabularFile):
 ```ts
 isSpatialDataFile(path) === /\.zarr\.zip$/i.test(path)   // covers .sdata.zarr.zip
+isSpatialDataFolder(path) === /\.zarr\/?$/i.test(path)   // any .zarr folder
 ```
+  A `.zarr` folder is offered whether or not it is SpatialData; the viewer's `error` says
+  when it is not something it can show. The host signs a folder's objects through
+  `datasetFileUrl` and lists them from the dataset's file manifest.
+- Persistence: settings-panel edits are always saved to the node. Viewer-side
+  `display-changed` events (camera moves, display switches) are saved only while the
+  tile's settings panel is open, so exploring a tile that is not locked
+  (`lock_view`) leaves its saved framing alone.

docs/USER_GUIDE.md

@@ -209,6 +209,17 @@ and save it again to add them. Any host that serves the file with HTTP range req
 will do — put the built app, your `.zarr.zip` files, and a small `index.json` listing
 them in one folder and the page becomes a browsable collection you can switch between.
 
+The viewer also opens SpatialData stores it did not save: a `.zarr.zip`, or a `.zarr/`
+folder (point `?checkpoint=` at the folder with a trailing `/`), such as nf-core/sopa or
+spatialdata-io output. It finds the table, places the images against the cells, and starts
+on the displays a new session would: the spatial view colored by the first categorical
+column over the first image, and a UMAP (or other embedding) view when the table has one.
+Such a store must be Zarr v3 with consolidated metadata (what spatialdata 0.3 and later
+write), and its table needs `obsm["spatial"]`; when one of those is missing the viewer
+says which instead of opening an empty view. Two things need a store saved by this app:
+cell boundaries are not drawn, and coloring by a gene reads the table's whole expression
+matrix rather than one gene's slice (a few seconds on a 2 GB sopa store).
+
 The **Plots** view works here too: the figures saved with the checkpoint are in the file,
 so the grid, the fullscreen view and the SVG/PDF/PNG downloads all work with no backend.
 The left panel opens collapsed and holds one thing: the history of the analysis that

frontend/package.json

@@ -1,7 +1,7 @@
 {
   "name": "spatial-data-studio-frontend",
   "private": true,
-  "version": "1.0.2",
+  "version": "1.1.0",
   "type": "module",
   "scripts": {
     "dev": "vite",

frontend/src/App.tsx

@@ -8,7 +8,7 @@ import { useCheckpointSession } from './data/useCheckpointSession';
 import {
   checkpointUrlFromLocation, fetchCheckpointIndex, isEmbedMode, openCheckpointPath,
 } from './data/checkpointIndex';
-import { requestFreshCheckpointUrl, useEmbedBridge } from './data/embedBridge';
+import { embedFolderAccess, requestFreshCheckpointUrl, useEmbedBridge } from './data/embedBridge';
 import CheckpointIndexPage from './components/CheckpointIndexPage';
 import { useSSE } from './hooks/useSSE';
 import { useUrlViewSync } from './hooks/useUrlViewSync';
@@ -53,7 +53,12 @@ export default function App() {
   const embed = useMemo(isEmbedMode, []) && checkpointUrl !== null;
   // An embed host signs checkpoint URLs for minutes at a time, so the reader
   // re-signs through the host rather than dying partway through a long session.
-  const checkpoint = useCheckpointSession(checkpointUrl, embed ? requestFreshCheckpointUrl : undefined);
+  // A `.zarr/` folder has no single URL to sign, so the host signs each of its objects.
+  const checkpoint = useCheckpointSession(
+    checkpointUrl,
+    embed ? requestFreshCheckpointUrl : undefined,
+    embed ? embedFolderAccess : undefined,
+  );
   useEmbedBridge(embed, checkpoint);
   // Shareable view links, serverless only and never under an embed host — there the
   // dashboard owns display state over postMessage and a URL writer would race it.