Repository navigation
feat: open images from cloud storage URIs (s3://, gs://) - #168
Open
davehorsfall wants to merge 2 commits into
Open
davehorsfall wants to merge 2 commits into
davehorsfall wants to merge 2 commits into
Conversation
`s3://bucket/key` is how object storage is named nearly everywhere outside a
browser -- the AWS and gcloud CLIs, boto3, fsspec, zarr-python, and most papers'
data availability statements -- and vizarr could not open one. Worse, it said
something untrue when it failed: an unknown scheme throws the same TypeError a
blocked request does, so `s3://...` was reported as "most likely a CORs issue",
sending the reader to look at a bucket policy for a request that never left the
browser. Both are fixed here, because the honest message is the one that names
the schemes that do work.
There is no S3 protocol to implement. S3's wire protocol is HTTPS and the URI is
a naming convention; every client that accepts one parses it and issues an HTTPS
request, so this does the same, once, before any store is built. A browser could
not do otherwise in any case: `fetch` has a fixed set of schemes with no registry
to extend, a service worker only ever sees http(s), and `registerProtocolHandler`
is for navigation rather than fetching.
Four forms:
s3://bucket/key -> bucket.s3.amazonaws.com
s3+https://endpoint/bucket/key -> any S3-compatible server
s3+http://endpoint/bucket/key -> a local MinIO, or plain HTTP
gs://bucket/key -> storage.googleapis.com
`s3+https://` follows neuroglancer so a link means the same thing in both, but
not its ambiguity: neuroglancer's host may itself be a bucket, which it resolves
by probing. Here **the first path segment after the host is always the bucket**.
Nothing is lost -- where the host is the bucket, the plain https url already
works and there is nothing for an S3 scheme to add -- and the probe goes away.
No region, deliberately. A URI carries none, the global endpoint routes to the
right one, and a guess fails hard: `janelia-cosem-datasets` is served by
`s3.amazonaws.com` and 301s on `s3.us-west-2.amazonaws.com`, because despite the
institute it belongs to the bucket is in `us-east-1`.
Resolution happens where a store is built, which is also before the ZipFileStore
attempt, so `s3://bucket/image.zarr.zip` is range-read like any other zip. The
address bar keeps whichever form the user gave, so a shared link stays readable
by the tools that produced the URI.
The resolver is exported and used by both plugins. A table and the image beside
it come from the same bucket; `s3://` working for one and not the other would be
a worse bug than not supporting it at all.
Anonymous only, and CORS remains the bucket owner's decision. Both are limits of
the browser rather than of this change, and both are written down on the docs
page rather than left for a reader to discover.
Closes #165
Co-Authored-By: Claude Opus 5 <[email protected]>
✅ Deploy Preview for biongff-vizarr ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
`?source=s3+https://host/bucket/key` is how anyone writes the link, and a `+` in a query string means a space, so `URLSearchParams` hands the viewer `s3 https://` -- not a scheme at all, and no warning anywhere. The form documented on the feature page did not work when typed. Neuroglancer does not meet this because it keeps its state in the fragment. Vizarr takes a query parameter, so it does. Undoing it is unambiguous: no scheme contains a space, so a source beginning `s3 http` can only be this. Links vizarr writes itself were already correct, since `URLSearchParams` escapes the plus on the way out; this is for the ones people paste. Co-Authored-By: Claude Opus 5 <[email protected]>
There was a problem hiding this comment.
🟡 Changes recommended
Bioformats2raw XML metadata fetches still use unresolved cloud URI schemes, preventing cloud images and label sources from loading.
2 open issues
1 file not reviewed
pnpm-lock.yaml: Generated file
What changed in this PR
Adds cloud-storage URI support to Vizarr and its plugins by translating supported schemes into HTTP(S) URLs at fetch boundaries, addressing #165 while preserving users’ original source links.
Changes:
- Adds a shared resolver for Amazon S3, S3-compatible endpoints, and Google Cloud Storage.
- Integrates resolution into viewer and plugin loading, with clearer unsupported-scheme errors.
- Adds offline and remote tests plus a new Features documentation section.
| File | Description |
|---|---|
| viewer/tests/remote/cloud-uri.test.ts | Adds real-bucket cloud URI loading tests. |
| viewer/tests/error-handling.test.ts | Tests unsupported-scheme error reporting. |
| viewer/tests/cloud-uri.test.ts | Tests URI mappings offline. |
| viewer/src/utils.ts | Resolves cloud URIs before constructing stores. |
| viewer/src/services/http.ts | Adds scheme diagnostics and resolves failure-probe URLs. |
| viewer/src/index.tsx | Exports the resolver and supported schemes. |
| viewer/src/cloud-uri.ts | Implements cloud URI translation. |
| typedoc.json | Registers Features documentation. |
| roi-selector/src/importRois.ts | Resolves cloud URIs for ROI imports. |
| pnpm-lock.yaml | Records the AnnData plugin’s viewer dependency. |
| docs/features/index.md | Introduces the Features section. |
| docs/features/cloud-storage-uris.md | Documents URI formats, examples, and limitations. |
| anndata-zarr/vite.config.js | Externalizes the viewer dependency. |
| anndata-zarr/src/components/AnndataController.tsx | Resolves cloud URIs for AnnData sources. |
| anndata-zarr/package.json | Adds the shared resolver’s package dependency. |
Review effort: Balanced
| // url here, before any store sees it. Resolving once at the front rather than inside | ||
| // each branch is also what lets `s3://bucket/image.zarr.zip` reach ZipFileStore as a | ||
| // url it can make range requests against. | ||
| const href = resolveCloudUri(source); |
|
|
||
| ``` | ||
| https://biongff.github.io/vizarr/?source=s3://janelia-cosem-datasets/jrc_hela-2/jrc_hela-2.zarr/recon-1/em/fibsem-uint8 | ||
| https://biongff.github.io/vizarr/?source=s3+https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Closes #165.
Vizarr could not open
s3://bucket/key, which is how object storage is named nearlyeverywhere outside a browser. It also said something untrue when it failed — an unknown
scheme throws the same
TypeErrora blocked request does, so the one thing a user had togo on was "this is most likely a CORs issue" for a request that never left the browser.
s3://bucket/keyhttps://bucket.s3.amazonaws.com/keys3+https://endpoint/bucket/keys3+http://endpoint/bucket/keygs://bucket/keyhttps://storage.googleapis.com/bucket/keyThere is no S3 protocol, and that is the whole point
S3's wire protocol is HTTPS; the URI is a naming convention the AWS CLI introduced.
Every client that accepts one — boto3, s3fs, zarr-python, the CLI itself — parses it and
issues an HTTPS request. Translating is not declining to do the real thing; it is the real
thing, in every client there is.
A browser could not do otherwise regardless:
fetchhas a fixed scheme set with noregistry to extend, a service worker only ever sees
http(s), andregisterProtocolHandleris for navigation rather than fetching. This is also why zarritacannot help —
FetchStoreis a thin wrapper overfetchwith no scheme handling, and itsown documentation reaches S3 by being handed an
https://URL.Two decisions worth review
The first path segment after the host is always the bucket. Neuroglancer has the same
s3+https://scheme and documents it as ambiguous — the host may itself be a bucket — andresolves that by probing. We don't need the ambiguity: where the host is the bucket, the
plain
https://URL already works, so there is nothing for an S3 scheme to add. Declaringone reading removes both the ambiguity and the round trip.
No region, and no way to name one. A URI carries none, the global endpoint routes
correctly, and a guess is a hard failure rather than a near miss:
janelia-cosem-datasetsis served bys3.amazonaws.comand answers with a 301 ons3.us-west-2.amazonaws.com, because despite the institute it belongs to the bucket is inus-east-1.Where it is wired in
Resolution happens where a store is built, and before the
ZipFileStoreattempt — sos3://bucket/image.zarr.zipis range-read like any other zip rather than falling through.The resolver is exported from
@biongff/vizarrand used by both plugins. A table and theimage beside it live in the same bucket, and
s3://working for one but not the otherwould be a worse bug than not supporting it at all. That adds
@biongff/vizarrtoanndata-zarr, externalised in its build exactly asroi-selectoralready does.The address bar keeps whichever form the user gave. Resolution is for fetching, not for
links, so a shared vizarr link stays readable by the tool that produced the URI.
What it deliberately cannot do
https://URL isthe answer for private data.
error message now distinguishes this from a URI we could not parse, which is the case it
used to be confused with.
gcp://— not a scheme any Google tool emits. No Azure, whose URI does notcarry the storage account name the HTTPS host needs.
Checks
pnpm test— 64 passed, up from 46 (18 new; the mapping is pure, so it is tested offline)pnpm test:remote— 3 new, opening real images from AWS and from the EBI Embassypnpm lint,pnpm check,pnpm build:docs— cleans3://janelia-cosem-datasets/jrc_hela-2/…ands3+https://uk1s3.embassy.ebi.ac.uk/idr/…both load, andaz://…now reports"az:// is not an address a browser can open. Vizarr understands https://, http://,
s3://, s3+https://, s3+http://, gs://."
Documentation
A new Features section — the non-RFC counterpart to the RFC pages in #162 — with
docs/features/index.mdand a page on cloud storage URIs covering the forms, the limitsand the reasons.
🤖 Generated with Claude Code