Repository navigation
Conversation
Integrates the training recipe of the 'mitochondria2' model, which so far only existed outside this repository, as a first-class part of the package. - synapse_net/training/mitochondria.py: get_mitochondria_paths for the recursive data discovery and the train/val split (optionally from a split file), and mitochondria_training, which fixes the hyperparameters of the published model. Registered as the CLI synapse_net.run_mitochondria_training. - supervised_training: expose initial_features, mixed_precision, early_stopping and log_image_interval. The defaults keep the previous behaviour for all other callers. - scripts/cooper/training/train_mitochondria_tomo.py plus the split file of the published run, so it can be reproduced file for file. - test/training/__init__.py, without which unittest discovery skipped all of test/training. Discovery goes from 106 to 121 tests with this commit. The recipe was taken from the init of the deployed checkpoint, not from the training script. Verified by training on the real tomograms and comparing the resulting checkpoint against the deployed one: model kwargs, loss, optimizer, scheduler, mixed_precision, early_stopping, log_image_interval and the loader kwargs all agree, and both models have the same state dict and the same 18 InstanceNorm3d layers. Note that the percentile normalization is part of the model: the same preprocessing has to be passed to segment_mitochondria at inference time. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The previous commit added test/training/__init__.py so that unittest discovery would pick
up that folder. That also enabled the eight tests in test/training/test_training_cli.py,
which had never run in CI before and do not pass there: all three CI jobs failed, while
main is green. Windows failed within seconds, which matches the skip that test/test_cli.py
already carries ("CLI does not work on Windows") for the same kind of subprocess test.
Fixing those tests is not this change's job, so restore test/training to the state it was
in and move the new mitochondria tests to test/, where discovery finds them without
enabling anything else. CI now runs 113 tests, the 106 from main plus these 7.
Also skip the CLI test on Windows, following test/test_cli.py.
The tests in test/training/test_training_cli.py are still invisible to CI. That is
pre-existing and worth a separate look.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Integrates the training recipe of the 'cristae4' model, which so far only existed outside this repository. Cristae training needs machinery that the package did not have, so this adds it rather than only wiring up a recipe. - synapse_net/training/loss.py: MaskedDiceLoss and MaskedDiceLossPerSample, which read a per-voxel loss mask from the second half of the target channels. This is how the voxels of mitochondria without cristae annotations are kept out of the loss. - synapse_net/training/transform.py: the joint transforms that build that mask, the membrane proximity weighting it can carry, and standardize_channel / normalize_channel, which normalize the tomogram channel while leaving the mitochondria state untouched. - synapse_net/training/cristae.py: get_cristae_paths and cristae_training, plus the CLI synapse_net.run_cristae_training. - supervised_training: expose 'norm' (the cristae model has no normalization layers, and until now the model was always built with InstanceNorm) and let the joint 'transform' be overridden, which the cristae mask transform needs. The defaults keep the previous behaviour for all other callers, and the model now records its norm in the checkpoint. - scripts/cooper/training/train_cristae.py plus the split file of the published run. Verified against the originals in the separate synapse repository: the losses (binary and weighted, several shapes), the membrane weighting, both mask transforms including the batched path, and the channel normalizations are all bit-identical. A checkpoint trained with this code agrees with the deployed one on every model kwarg (including norm=None and the two input channels), the optimizer, the scheduler, mixed precision, early stopping, the log interval and the loader kwargs. get_cristae_paths reproduces the published 135 / 14 split file for file, in order. Note that the published checkpoint could not be loaded without the synapse package on the path, because its loss class was pickled into the checkpoint. Checkpoints written by this code no longer depend on anything outside synapse_net. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The volume EM mitochondria training needs the same multi-root split files as the cristae training, so '_normalize_roots' and '_resolve_split' move out of cristae.py unchanged. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
'supervised_training' accepted every other model argument but built the U-Net with the default scale factors, so a recipe for strongly anisotropic data could not be expressed through it. The default stays None, so the model that the existing recipes build is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Volume EM blocks that were cut out of a larger FIB-SEM volume carry white filler borders where the cutout extends past the imaged region. Percentile normalization maps them to the bright end of the value range, so the network sees a strong edge that does not exist in the sample. 'RemoveWhitePatchesAndNormalize' is a class rather than a function because torch-em pickles the raw transform into the checkpoint by reference: the published volume EM model used a transform that was local to '__main__', which makes its checkpoint impossible to load. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Reproduces the recipe of the model that was trained on the FIB-SEM datasets 4007 and 4009 at a voxel size of 25 nm in z and 5 nm in xy. The defaults were read back from the checkpoint of that run, so the module builds the same architecture: two anisotropic downsampling levels for the 5:1 anisotropy and no normalization layers. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Besides the recipe and the path discovery, this pins the two things that would silently build a different network than the published one: the scale factors have to reach the model, and the model has to be built without normalization layers. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…odel The split file pins the split of the published run, which was checked against the file lists in its checkpoint. The segmentation script removes the white filler from the whole volume before predicting, because 'get_prediction' standardizes a numpy input volume before it runs the per-block 'preprocess', and the filler is identified by its literal value. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Also ignore the folders that the training tests write, so that their output does not end up in the repository. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Checked on 4009_raw_z0-128_y0-1600_x0-1600_0, which is 14% filler: removing it gives Dice 0.84 against the annotations, leaving it in gives 0.79. The effect is on the surrounding tissue, via the percentile normalization -- the model does not segment the filler itself either way, so the earlier wording claimed the wrong mechanism. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Diffs the trainer that 'vol_em_mitochondria_training' builds against the setup stored in the published checkpoint, without training. Every field matches, including the model kwargs, both loader kwargs, the sampler parameters and the per-file dataset lengths. The two places where the port deviates structurally are checked rather than assumed: the raw transform is compared numerically against a transcription of the published one and agrees exactly on real patches (including patches containing filler), and the PadIfNecessary that the port adds to the joint transform is confirmed never to fire, because no block is smaller than the patch. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Three clean runs, seeds 42/43/44, one job each. They write to a separate save_root: the published checkpoint was produced by two jobs that shared one and raced, and this module's overwrite guard now refuses that outright. The manifests live here rather than in the synapse repo so that the reproduction configuration sits with the code it reproduces; the runner there reads them from any path. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
torch-em pickles the training dataset into the checkpoint, and the published volume EM mitochondria model was trained by a script whose raw transform was a local function, so its checkpoint references '__main__._raw_transform_with_fix_white_patches' and cannot be loaded anywhere else. This rewrites that reference to the equivalent importable class and writes a new copy; the weights, the optimizer state and every hyperparameter are copied through unchanged, verified bit-identical over all 46 weight tensors. The original is not modified. Models trained with this module do not need it, since their raw transform is a module-level class. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The run script claimed it stopped on wall time at 14,500 of 50,000 iterations. It did not. Two jobs for that experiment ran concurrently for ~23 h against the same checkpoint folder, the second warm-started weights-only from the first's epoch-15 snapshot, and both ended on early stopping with 8-10 h of their limit unused. The surviving checkpoint has a lineage of ~16,500 steps with an optimizer reset at 2,000, and the counters in it describe only the second job. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Segments every test block with every model through the shipped inference path, with identical preprocessing and identical post-processing, and reports instance matching, mean segmentation accuracy, symmetric best dice, semantic dice and instance counts per block, alongside each run's best validation metric. Segmentations are cached so a model can be added without redoing the rest. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Records what the published checkpoint actually is, what to expect from a reproduction run, and the provenance of the two held-out test blocks, including the content check that rules out overlap for the two training cutouts whose filenames carry no coordinates. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Running the setup check under a clean environment (Python 3.14, torch 2.13, torch_em 0.10.6) showed that newer torch-em serializes a 'mixed_precision_dtype' field the published run did not have. Every recipe field is identical there, so the check now separates version additions from real differences, and tolerates only the ones listed with a reason why they cannot change this recipe -- an unknown addition is still reported as a difference, because it may well change what is trained. 'mixed_precision_dtype' is read only by the mixed-precision train and validate loops, and this recipe trains in single precision. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Instance F1 0.6905 averaged over the two blocks, semantic dice 0.8490. The reproduction runs are queued; their rows are added to this file when they finish. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The published post-processing over-segments under this watershed: 48 and 57 predicted instances against 30 and 43 annotated ones, precision 0.585 against recall 0.847, semantic dice 0.849. Raising min_size from 1,000 to 20,000 takes F1 from 0.69 to 0.85 with recall unchanged, so it removes only false positives -- on these two blocks. That threshold is not adopted. It was read off the test data, and the two test blocks happen to hold only large mitochondria: of the 717 mitochondria annotated in the training blocks, 18% are smaller than 20,000 voxels, so such a filter would discard nearly a fifth of the real objects. The inherited min_size of 1,000 is consistent with the annotation instead, since 99.7% of annotated mitochondria exceed it. The sweep stays available as a diagnostic for how much of the error is fragments. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
… caveat into the report synapse_net is installed editable, so a queued job trains whatever is checked out when it starts, not what was checked out when it was submitted. Switching branches while runs are queued silently changes what they train. The generated report now also states why a min_size must not be read off the sweep, so the caveat travels with the numbers instead of living only in the README. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The .log file that the job writes holds only the four startup lines; the progress bar and the per-epoch metrics go to the .err file, so watching the .log makes a healthy run look stalled. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
tmp_cristae_training/ was committed in ac296c8, before tmp_*/ was ignored. The cristae test fixture then failed on datasets that already existed, and since setUp failed, tearDown never cleaned up, so every test of the class errored. The fixtures now overwrite their files, so a folder left behind by an interrupted run cannot do that again. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The entries of a split file always use '/', and were joined onto the root as they were, so on
Windows the paths came out mixed ('root\cutout_0/block-0.h5'). Besides failing the path tests,
that broke the exclusion of test volumes from a random cristae split: 'exclude' matches
substrings against the paths found by glob, which use '\' throughout, so the test volumes
were never excluded. The paths are unchanged on Linux and macOS.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…mparison The baselines (MitoNet, micro-sam) run in environments without synapse_net, so their segmentations are made elsewhere; '-s name=folder' scores them with the same metrics as the checkpoints, and '--title' names the report. '--split_file/--split_key' scores the blocks of a split file, e.g. the validation blocks. Those are named like the test blocks, so their cached segmentations go into a folder of their own, or a cached test segmentation would be scored against validation labels. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
MitoNet_v1 is resampled to the target voxel size, segmented (ortho-plane consensus and xy stack), and resized back to the native grid, where the untouched ground truth scores it. The size filter is the same physical volume as synapse-net's 1000 native voxels; empanada's default of 500 voxels would be ~21,600 native ones at 30 nm. Segmentations are stored as uint32: the earlier runs saved empanada's 1000 + n ids as uint8, which wraps. voxel_sweep.py holds the lookup and resampling that the baselines share, since their environments have no synapse_net. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The voxel size is selected on the validation blocks, which MitoNet never saw, and only the selected setting of each approach is run once on the test blocks: A keeps z at 25 nm and sweeps xy, B is isotropic, C holds single settings such as 30/8/8. The scorer caches its metrics per folder, says so when no setting clears F1 0.05, and flags a selection on the edge of its grid. MitoNet works only at 8-15 nm in xy and reaches test F1 0.09-0.12; the inverted contrast scores ~0 everywhere, so the 30 nm failure was scale, not contrast. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
SAM resizes every image and tile to 1024 px, so a resampled slice of these blocks would reach the encoder at ~7.8 nm/px whatever the voxel size. Every image or tile that holds data is padded to exactly 1024 px instead, so the sweep controls the scale the encoder sees. micro-sam (vit_b_em_organelles, AIS) works only at the native 5 nm, where it reaches val F1 0.089 and test F1 0.073, below MitoNet; 3.75 nm, the step past the edge of the grid, scores lower. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
How each baseline is run and scored, why the voxel size is selected on the validation blocks, why the earlier February runs are not comparable, and that micro-sam 1.7.7 fails in a fresh environment, where it gets elf 0.9, while 1.8.x works. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…ia-training Main moved the U-Net builders to training/models.py and the loader to training/dataloaders.py, so the norm, initial_features, scale_factors and joint transform options of this branch now live there and _get_model forwards them. Picks up the torch_em decoder fix from main that the CI failures were caused by. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.