Feat/ome consolidated metadata

USERGUIDE-OME-ZARR.md

@@ -149,3 +149,48 @@ written.createScaleLevel(
 - `ome.v0_6.MultiscaleImage.create(...)`
 
 Use the corresponding metadata classes for each version package.
+
+## Consolidated metadata
+
+A Zarr v3 group can hold a copy of the metadata of all of its descendants in its own `zarr.json`, so
+that the whole hierarchy opens with a single read. See the
+[consolidated metadata section](USERGUIDE.md#consolidated-metadata-v3) of the main guide.
+
+The OME-Zarr nodes use it. Consolidate at the root of the hierarchy, then open there:
+
+```java
+Group.consolidateMetadata(plateHandle);          // once, after writing
+
+Plate plate = Plate.open(plateHandle);           // one request
+Well well = plate.openWell("A/1");               // no request
+MultiscaleImage image = well.openImage("0");     // no request
+Array level0 = (Array) image.openScaleLevel(0);  // no request
+```
+
+Walking down asks the group that is already open, so every descendant is answered from the cache the
+plate loaded. Without a cache the same calls read one `zarr.json` per node, as before.
+
+`getLabels()` and `openLabel(name)` use the cache as well, and so does the image-node discovery of
+`v0_6.Scene`, which otherwise lists the store and opens every child.
+
+Notes:
+
+- This applies to OME-Zarr **0.5 and 0.6**, which are backed by Zarr v3. OME-Zarr 0.4 is backed by
+  Zarr v2, and the v2 `.zmetadata` file is not supported, so 0.4 reads one node at a time.
+- The benefit only exists if you open at the group that holds the cache. Pointing directly at
+  `plate/A/1/0` reads that node from the store, because nothing above it was opened.
+- The cache is a snapshot. A well added to a plate after consolidating is not found until the plate is
+  consolidated again; the error says so. Open the group with `UseConsolidated.IGNORE` to bypass it.
+
+### Building a node from a group you already have
+
+Every version class has a `fromGroup(...)` next to its `openX(StoreHandle)`:
+
+```java
+Plate plate = ome.v0_5.Plate.fromGroup(group);   // no store request
+Plate plate = Plate.fromGroup(group);            // picks 0.5 or 0.6 from the attributes
+```
+
+`openX(StoreHandle)` is now `fromGroup(Group.open(handle))`, so opening a node reads its `zarr.json`
+exactly once. Previously the version-detecting entry points probed and read it, then let the version
+class read it again — three requests for one file.

USERGUIDE.md

@@ -288,6 +288,61 @@ Array array = group.createArray(
         .build()
 );
 ```
+### Consolidated Metadata (v3)
+A group can keep a copy of the metadata of all of its descendants inside its own `zarr.json`, so that
+the whole hierarchy can be opened with a single read instead of one read per node. This matters most
+over HTTP and S3, where every node otherwise costs a request.
+
+```java
+// Write the cache. This walks the hierarchy once and stores the metadata of every
+// descendant in the metadata of this group.
+Group root = Group.consolidateMetadata(storeHandle);
+Group root = Group.consolidateMetadata("/data/my.zarr");   // or a path
+
+// Consolidate a subtree instead of the whole hierarchy
+Group.consolidateMetadata(storeHandle.resolve("sub"));
+
+// The same, on a group that is already open
+Group root = Group.open(storeHandle).consolidateMetadata();
+
+// Reads and listings are answered from the cache, without touching the store.
+Group sub = (Group) root.get("sub");
+Array array = (Array) sub.get("nested");
+Node[] everything = root.listAsArray();
+
+// Remove the cache again
+root.dropConsolidatedMetadata();
+```
+
+On opening, the cache is used if the group has one. Pass `UseConsolidated` to say otherwise:
+
+```java
+Group auto = Group.open(storeHandle);                                   // use it if present
+Group required = Group.open(storeHandle, UseConsolidated.REQUIRE);      // fail if absent
+Group required = Group.openConsolidated(storeHandle);                   // the same, shorter
+Group fresh = Group.open(storeHandle, UseConsolidated.IGNORE);          // read every node instead
+```
+
+`UseConsolidated.IGNORE` drops the cache from the metadata held in memory, so writing that metadata
+again - with `setAttributes()` for example - removes the cache from the store as well. The choice
+applies to the group being opened only: a subgroup reached through `get()` uses a cache of its own if
+it has one.
+
+The cache is written in the same format as `zarr.consolidate_metadata()` in zarr-python, and behaves
+the same way on reading, so both libraries can read each other's output.
+
+**The cache is authoritative.** While a group is using its cache, `get()` and `list()` answer from it
+alone and never fall back to the store. A key the cache does not hold is reported as absent.
+
+**The cache is a snapshot.** Nothing invalidates it when a node is added, removed or changed
+afterwards, so it has to be written again after modifying the hierarchy. Until then a node added
+afterwards is invisible, and a node changed afterwards is served as it was. Open the group with
+`UseConsolidated.IGNORE` if in doubt.
+
+Consolidated metadata is a Zarr v3 feature here; the v2 `.zmetadata` file is not supported. Note that
+consolidated metadata is not part of the Zarr v3 specification, so other implementations may not
+support it; `consolidateMetadata()` logs a warning saying so, as zarr-python does.
+
 ### Hierarchical Example
 ```java
 Group root = Group.create(

src/main/java/dev/zarr/zarrjava/experimental/ome/MultiscaleImage.java

@@ -67,6 +67,25 @@ default List<String> getAxisNames() throws ZarrException {
      * Returns all label names from the {@code labels/} sub-group, or an empty list if none exist.
      */
     default List<String> getLabels() throws IOException, ZarrException {
+        dev.zarr.zarrjava.v3.Group v3Group = asV3Group();
+        if (v3Group != null) {
+            // Walk down through the group, so that a consolidated ancestor answers without a request.
+            dev.zarr.zarrjava.core.Node labelsNode = v3Group.get(new String[]{"labels"});
+            if (!(labelsNode instanceof dev.zarr.zarrjava.v3.Group)) {
+                return Collections.emptyList();
+            }
+            dev.zarr.zarrjava.core.Attributes labelsAttributes =
+                    ((dev.zarr.zarrjava.v3.Group) labelsNode).metadata.attributes;
+            if (labelsAttributes == null || !labelsAttributes.containsKey("labels")) {
+                return Collections.emptyList();
+            }
+            List<String> result = new ArrayList<>();
+            for (Object item : labelsAttributes.getList("labels")) {
+                result.add(String.valueOf(item));
+            }
+            return result;
+        }
+
         StoreHandle labelsHandle = getStoreHandle().resolve("labels");
 
         // Try v0.5: labels/zarr.json with {"attributes": {"labels": [...]}}
@@ -109,6 +128,10 @@ default List<String> getLabels() throws IOException, ZarrException {
      * Opens the named label image from the {@code labels/} sub-group.
      */
     default MultiscaleImage openLabel(String name) throws IOException, ZarrException {
+        dev.zarr.zarrjava.v3.Group v3Group = asV3Group();
+        if (v3Group != null) {
+            return fromGroup(OmeNodes.childGroup(v3Group, "labels/" + name));
+        }
         return MultiscaleImage.open(getStoreHandle().resolve("labels").resolve(name));
     }
 
@@ -118,34 +141,43 @@ default MultiscaleImage openLabel(String name) throws IOException, ZarrException
      * <p>Tries v0.6 (zarr.json with version "0.6"), then v0.5 (zarr.json with "ome" key), then v0.4 (.zattrs with "multiscales" key).
      */
     static MultiscaleImage open(StoreHandle storeHandle) throws IOException, ZarrException {
-        // Try version >= 0.5: zarr.json with "ome" key
-        StoreHandle zarrJson = storeHandle.resolve(Node.ZARR_JSON);
-        if (zarrJson.exists()) {
-            com.fasterxml.jackson.databind.ObjectMapper mapper = OmeObjectMappers.makeV3Mapper();
-            byte[] bytes = Utils.toArray(zarrJson.readNonNull());
-            com.fasterxml.jackson.databind.JsonNode root = mapper.readTree(bytes);
-            com.fasterxml.jackson.databind.JsonNode attrs = root.get("attributes");
-            if (attrs != null && attrs.has("ome")) {
-                com.fasterxml.jackson.databind.JsonNode omeNode = attrs.get("ome");
-                String version = omeNode.has("version") ? omeNode.get("version").asText() : "";
-                if (version.startsWith("0.6")) {
-                    return dev.zarr.zarrjava.experimental.ome.v0_6.MultiscaleImage.openMultiscaleImage(storeHandle);
-                }
-                return dev.zarr.zarrjava.experimental.ome.v0_5.MultiscaleImage.openMultiscaleImage(storeHandle);
-            }
+        // Zarr v3 (OME-Zarr 0.5 and 0.6): a zarr.json holding an "ome" attribute. The group is read once
+        // here and handed to the version class, which does not read it again.
+        dev.zarr.zarrjava.v3.Group v3Group = OmeNodes.openV3GroupOrNull(storeHandle);
+        if (v3Group != null && OmeNodes.omeAttributes(v3Group.metadata.attributes) != null) {
+            return fromGroup(v3Group);
         }
 
-        // Try v0.4: .zattrs with "multiscales" key
-        StoreHandle zattrs = storeHandle.resolve(Node.ZATTRS);
-        if (zattrs.exists()) {
-            com.fasterxml.jackson.databind.ObjectMapper mapper = OmeObjectMappers.makeV2Mapper();
-            byte[] bytes = Utils.toArray(zattrs.readNonNull());
-            com.fasterxml.jackson.databind.JsonNode root = mapper.readTree(bytes);
-            if (root.has("multiscales")) {
-                return dev.zarr.zarrjava.experimental.ome.v0_4.MultiscaleImage.openMultiscaleImage(storeHandle);
-            }
+        // Zarr v2 (OME-Zarr 0.4): a .zattrs holding a "multiscales" key.
+        dev.zarr.zarrjava.v2.Group v2Group = OmeNodes.openV2GroupOrNull(storeHandle);
+        if (v2Group != null && v2Group.metadata.attributes != null
+                && v2Group.metadata.attributes.containsKey("multiscales")) {
+            return dev.zarr.zarrjava.experimental.ome.v0_4.MultiscaleImage.fromGroup(v2Group);
         }
 
         throw new ZarrException("No OME-Zarr multiscale metadata found at " + storeHandle);
     }
+
+    /**
+     * Builds a multiscale image from a Zarr v3 group that is already open, picking the OME-Zarr version
+     * from its attributes. No store request is made, so a group that came from {@code Group.get()} is
+     * served from the consolidated metadata of its ancestor.
+     */
+    static MultiscaleImage fromGroup(dev.zarr.zarrjava.v3.Group group) throws IOException, ZarrException {
+        String version = OmeNodes.omeVersion(group.metadata.attributes);
+        if (version != null && version.startsWith("0.6")) {
+            return dev.zarr.zarrjava.experimental.ome.v0_6.MultiscaleImage.fromGroup(group);
+        }
+        return dev.zarr.zarrjava.experimental.ome.v0_5.MultiscaleImage.fromGroup(group);
+    }
+
+    /**
+     * The Zarr v3 group backing this image, or null for OME-Zarr 0.4, which is backed by a Zarr v2
+     * group. Used by {@link #getLabels()} and {@link #openLabel(String)} to walk down through the group
+     * - and therefore through its consolidated metadata - rather than by re-reading the store.
+     */
+    @Nullable
+    default dev.zarr.zarrjava.v3.Group asV3Group() {
+        return null;
+    }
 }

src/main/java/dev/zarr/zarrjava/experimental/ome/OmeNodes.java

@@ -0,0 +1,166 @@
+package dev.zarr.zarrjava.experimental.ome;
+
+import dev.zarr.zarrjava.ZarrException;
+import dev.zarr.zarrjava.core.Attributes;
+import dev.zarr.zarrjava.store.StoreHandle;
+import dev.zarr.zarrjava.v3.Array;
+import dev.zarr.zarrjava.v3.Group;
+import dev.zarr.zarrjava.v3.Node;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.file.NoSuchFileException;
+
+/**
+ * Helpers shared by the OME-Zarr nodes for opening a node once and for walking down a hierarchy
+ * through the group that is already open.
+ * <p>
+ * Walking down through {@link Group#get} instead of resolving a fresh {@link StoreHandle} is what
+ * lets an OME-Zarr hierarchy benefit from consolidated metadata: a group that has a cache answers
+ * for all of its descendants without touching the store. Resolving a handle and opening it again
+ * always costs a request per node, because a handle is only a path and carries no metadata.
+ */
+public final class OmeNodes {
+
+    private OmeNodes() {
+    }
+
+    /**
+     * Opens the Zarr v3 group at {@code storeHandle}, or returns null if the node is not a Zarr v3
+     * group: either there is no {@code zarr.json} there, or it describes an array. Reads
+     * {@code zarr.json} once.
+     */
+    @Nullable
+    public static Group openV3GroupOrNull(@Nonnull StoreHandle storeHandle)
+            throws IOException, ZarrException {
+        try {
+            Node node = Node.open(storeHandle);
+            return node instanceof Group ? (Group) node : null;
+        } catch (NoSuchFileException e) {
+            return null;
+        }
+    }
+
+    /**
+     * Opens the Zarr v2 group at {@code storeHandle}, or returns null if there is no {@code .zgroup}
+     * there, meaning the node is not a Zarr v2 group.
+     */
+    @Nullable
+    public static dev.zarr.zarrjava.v2.Group openV2GroupOrNull(@Nonnull StoreHandle storeHandle)
+            throws IOException {
+        try {
+            return dev.zarr.zarrjava.v2.Group.open(storeHandle);
+        } catch (NoSuchFileException e) {
+            return null;
+        }
+    }
+
+    /**
+     * Returns the {@code version} of the {@code ome} attribute, or null if the attributes hold no
+     * usable {@code ome} entry. Used to pick the OME-Zarr version of a node whose metadata has
+     * already been read.
+     */
+    @Nullable
+    public static String omeVersion(@Nullable Attributes attributes) {
+        Attributes ome = omeAttributes(attributes);
+        if (ome == null) {
+            return null;
+        }
+        Object version = ome.get("version");
+        return version == null ? null : version.toString();
+    }
+
+    /**
+     * Whether the {@code ome} attribute holds {@code key}, for example {@code "plate"},
+     * {@code "well"} or {@code "multiscales"}.
+     */
+    public static boolean omeHas(@Nullable Attributes attributes, @Nonnull String key) {
+        Attributes ome = omeAttributes(attributes);
+        return ome != null && ome.get(key) != null;
+    }
+
+    /**
+     * The {@code ome} attribute as {@link Attributes}, or null if it is absent or not a mapping.
+     */
+    @Nullable
+    public static Attributes omeAttributes(@Nullable Attributes attributes) {
+        if (attributes == null || !attributes.containsKey("ome")) {
+            return null;
+        }
+        try {
+            return attributes.getAttributes("ome");
+        } catch (IllegalArgumentException e) {
+            return null;
+        }
+    }
+
+    /**
+     * Returns the child group at {@code path} below {@code parent}, using the consolidated metadata of
+     * {@code parent} if it has any.
+     *
+     * @param path the path of the child relative to {@code parent}, {@code "/"}-separated
+     * @throws ZarrException if there is no node at {@code path}, or if it is not a group
+     */
+    @Nonnull
+    public static Group childGroup(@Nonnull Group parent, @Nonnull String path)
+            throws IOException, ZarrException {
+        Node child = child(parent, path);
+        if (!(child instanceof Group)) {
+            throw new ZarrException(
+                    "'" + path + "' below " + parent.storeHandle + " is not a group.");
+        }
+        return (Group) child;
+    }
+
+    /**
+     * Returns the child array at {@code path} below {@code parent}, using the consolidated metadata of
+     * {@code parent} if it has any.
+     *
+     * @param path the path of the child relative to {@code parent}, {@code "/"}-separated
+     * @throws ZarrException if there is no node at {@code path}, or if it is not an array
+     */
+    @Nonnull
+    public static Array childArray(@Nonnull Group parent, @Nonnull String path)
+            throws IOException, ZarrException {
+        Node child = child(parent, path);
+        if (!(child instanceof Array)) {
+            throw new ZarrException(
+                    "'" + path + "' below " + parent.storeHandle + " is not an array.");
+        }
+        return (Array) child;
+    }
+
+    @Nonnull
+    private static Node child(@Nonnull Group parent, @Nonnull String path)
+            throws IOException, ZarrException {
+        Node child = parent.get(path.split("/"));
+        if (child == null) {
+            throw new ZarrException("No node at '" + path + "' below " + parent.storeHandle
+                    + ". If the group has consolidated metadata, that cache is a snapshot and may be"
+                    + " stale; consolidate it again, or open the group with UseConsolidated.IGNORE.");
+        }
+        return child;
+    }
+
+    /**
+     * Returns the names of the direct children of {@code group} without listing the store if
+     * {@code group} has consolidated metadata, and by listing the store otherwise.
+     */
+    @Nonnull
+    public static java.util.List<String> childNames(@Nonnull Group group) {
+        java.util.List<String> names = new java.util.ArrayList<>();
+        if (group.metadata.consolidatedMetadata != null) {
+            for (String key : group.metadata.consolidatedMetadata.metadata.keySet()) {
+                if (key.indexOf('/') < 0) {
+                    names.add(key);
+                }
+            }
+            return names;
+        }
+        try (java.util.stream.Stream<String> children = group.storeHandle.listChildren()) {
+            children.forEach(names::add);
+        }
+        return names;
+    }
+}