Rewrite the Chung lab demo notebook for the OME-Zarr layout - #208
Rewrite the Chung lab demo notebook for the OME-Zarr layout#208bendichter wants to merge 2 commits into
Conversation
Dandiset 000108 was reorganized after this notebook was written: the image data moved from microscopy/*.SPIM.h5 to micr/*.SPIM.ome.zarr, and the JSON sidecars that carried each chunk's extent and offset were replaced by the OME-NGFF coordinateTransformations inside each store. The notebook had been excluded from CI and from the Colab badge because of it, and this rewrites the data access against the current layout. The narrative is unchanged: find the chunks of one slab, look at how they tile, read a whole-slab overview from the resolution pyramid, read a region at full resolution across a chunk boundary, and segment blood vessels in the LEC channel. The stitching is now driven by the voxel size and position each store publishes, so it is much shorter than the sidecar arithmetic it replaces, and the seam figure compares cosine blending against a plain average to show what the blending buys. Two changes of substance beyond the port. The interactive neuroglancer viewer is now a printed link, since a viewer bound to a local port is of no use in Colab or CI. And a cell reports the slab z origins per stain, because they are not always recorded: in sample 41 the LEC chunks disagree, five reporting 0 um and four 82000 um, and one chunk of sample 40 carries no coordinateTransformations at all. Stitching uses only the y positions, which are consistent, so the notebook is unaffected, but the gap is worth surfacing rather than papering over. The directory's requirements.in gains the packages the rewrite needs, so dashboard.ipynb re-locks with it and the two continue to share one container image. Both notebooks were run end to end on Python 3.13. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_011NuStoQykizhQCx2KBrMCN
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Index page previewA preview of the generated Last updated: commit |
Executed notebooksCI executes the notebooks changed in this PR through a Jupyter kernel and publishes each one, with its outputs, as soon as it finishes. Links appear below as notebooks complete; an hourglass means the notebook is still running. 2 of 2 finished. Commit |
|
could we set the overlay stitched slab to respect axis ratio of the underlying pixels? also after the blending demo, it would be nice to do a demo of the overlay with and without blending. |
Two changes from Satra's review of #208. The voxels are not cubic (3.625 um along y against 2.564 um along x), so every image drawn on its voxel grid was stretched by about 1.4 along x. Each imshow now passes an extent in micrometers, which puts both axes in physical units and lets the default aspect keep a micrometer the same length on both. The whole-slab overview now matches the proportions of the photograph beside it. The blending demonstration covered one seam at full resolution. A new section repeats it across the whole slab: the two overviews side by side, and below them the profiles averaged along the scan axis over a few chunk boundaries, where the plain average steps at each boundary and the blended profile crosses smoothly. The two overviews differ by up to 34% of the peak, and only within the 7% of the slab that lies in an overlapping margin. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_011NuStoQykizhQCx2KBrMCN
|
Done. thanks for looking at this @satra . I'm not an expert in this modality, I was merely using AI to adjust for the data reorg. I'll defer to you on the scientific content. |
|
thanks @bendichter - i can't see this in the updated deploy (although is see the code changes in the notebook). is that automatically done? |
|
@satra the preview is here: https://notebooks.dandiarchive.org/pr-preview/208/notebooks/000108_chunglab_demo_2021_09_27_dandi_demo_ipynb.html This is still a PR and it won't be reflected on notebooks.dandiarchive.org until it is merged. Do you think it's ready to merge now? |
2 participants
Dandiset 000108 was reorganized after
2021-09-27_dandi-demo.ipynbwas written. The image data moved frommicroscopy/*.SPIM.h5tomicr/*.SPIM.ome.zarr, and the JSON sidecars that gave each chunk's extent and offset were replaced by the OME-NGFFcoordinateTransformationsinside each store. The notebook has been on both exclusion lists since then, so it has neither badge and is not tested. This rewrites its data access for the current layout and removes both exclusions.Draft because the dataset is not ours: I would like the Chung lab or DANDI staff to confirm the sample I picked is a reasonable one to show, and to look at the metadata gaps below.
What the notebook does now
The structure follows the original: find the chunks of one slab, show how they tile, read a whole-slab overview from the resolution pyramid, read a region at full resolution across a chunk boundary, and segment blood vessels in the LEC channel. What changed is how the data are found and placed.
Chunk geometry now comes from the OME-NGFF metadata rather than from sidecars, which makes the stitching considerably shorter than the code it replaces. Each store publishes its voxel size (2.564, 3.625, 2.564 um) and its position, and the chunks turn out to step 6649 um along y and overlap by 775 um, about 214 voxels, which matches the description in the original text. Reads are in micrometers, and the overlap is combined with a raised-cosine weight. The seam figure shows blending against a plain average side by side: the average leaves two visible intensity steps at the edges of the overlap and the blend does not.
The overview at level 6 is a couple of seconds of streaming, and it is displayed next to the photograph of the slab, which is a good check that the reconstruction matches the tissue.
Two changes go beyond a port. The interactive neuroglancer viewer is now a printed link, since a viewer bound to a local port is no use in Colab or CI, and the link is built the same way
neuroglancer-stitched.pybuilds its links. And there is a short section reporting the slab z origins per stain, for the reason below.Metadata gaps I ran into
These are in the dandiset, not in the notebook, and someone who knows the pipeline should probably look at them:
0, chunks 6 to 9 report82000. The YO and NN chunks of the same sample all report82000, and samples 40 and 42 sit at80000and84000, so the slab spacing is otherwise regular and the five zeros look wrong.coordinateTransformationsat all, so neither its voxel size nor its position is recorded.Stitching a slab only needs the y positions, which are consistent everywhere I checked, so the notebook works regardless; it reads the geometry defensively and falls back to the voxel size implied by the pyramid when a level omits its transform. I added the reporting cell so a reader assembling slabs into a whole brain does not silently trust the z values. Happy to file this as a helpdesk issue if that is the better venue.
Verification
Both notebooks in the directory were run end to end through the CI harness on Python 3.13, in the same fresh-environment mode CI uses: the rewritten demo in 20 s,
dashboard.ipynbin 162 s. The directory'srequirements.ingains zarr, fsspec, aiohttp, pillow, requests and scikit-image, sodashboard.ipynbre-locks alongside it and the two stay in one image group, which means the published000108-chunglab-demoimage keeps its name and simply gains the second notebook.🤖 Generated with Claude Code
https://claude.ai/code/session_011NuStoQykizhQCx2KBrMCN