DynaCell v1.1 by alxndrkalinin · Pull Request #494 · mehta-lab/VisCy

alxndrkalinin · 2026-07-29T20:53:25Z

No description provided.

_final_metrics_cache_valid checked that pixel_metrics.npy exists, which is not enough now that the bare PSNR/SSIM/NRMSE names mean the scale-sensitive form. A cache written between the 2026-07-29 scale-invariant switch and the dual-reporting change holds scale-INVARIANT values under those names, and is byte-indistinguishable from a genuinely pre-switch cache. Reusing either would publish one scaling under the other's name -- silently, since the column count and dtypes are identical. Require the SI_* columns instead. Their absence is the only on-disk signal that a row predates dual reporting, so it has to force the recompute; a condition then self-heals on its next eval without anyone having to know when it was last run. The AP-column test's pixel fixture gains both families, since it now has to represent a *valid* cache rather than just a present one. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

… done eval Every eval CSV on disk carries one scaling of PSNR/SSIM/NRMSE, and which one depends on whether it predates 2026-07-29. Re-running the leaves with force_recompute.final_metrics=true would fix that, but it is the wrong tool: pixel metrics are ~4% of a warm-cache condition (eval_timing.csv: 10.6 s of 253 s on an iPSC nucleus leaf, against 107 s of per-timepoint feature pairwise and 71 s of MicroSSIM), and a full final-metrics pass also rewrites mask_metrics.csv and feature_metrics.csv -- moving already-published KID and MicroSSIM numbers for no reason. This pass is a strict column add. It takes the same `leaf=` override the grouped eval does, so pred/GT paths, channels and spacing come from the one authoritative source instead of being re-derived; it loads no models; and it leaves every other column and both sibling CSVs untouched. The work list comes from the cached rows' own (FOV, Timepoint) pairs rather than a fresh plate walk, so limit_positions / exclude_fov_names / a partial store cannot cause it to score a different set than the eval did. Recomputing both families also audits provenance for free: each cached bare column is compared against both, classifying every condition as scale_sensitive (pre-switch), scale_invariant (post-switch) or mismatch. PCC is the control -- affine-invariant, so it must reproduce, and a MOVED verdict means the prediction store was rewritten under the cache. That case raises rather than writing, because a mixed-provenance row is exactly what this is meant to prevent. Idempotent: a condition already carrying SI_* is skipped, so shards can be killed and relaunched. Measured 155 rows/min on an A40. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

PIXEL_METRICS listed one SSIM, NRMSE and PSNR each, so `dynacell report` renders whichever scaling the CSV happens to hold without saying which. Now that compute_pixel_metrics writes both, list both -- adjacent, so the pair reads as one comparison -- and mark SI_SSIM / SI_PSNR as higher-is-better alongside their bare forms. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

_conditions_of returned the top-level config unchanged for a leaf with no `conditions` list, leaving apply_dataset_ref to the CLI. Any other caller then got a config with `pixel_metrics.spacing` still `???` -- it comes from `benchmark.dataset_ref`, not from the leaf -- and died on MissingMandatoryValue. That took out 14 of the legacy floor/ablation leaves (vscyto3d_randinit, vscyto3d_cytoland, vscyto3d_infectionft, phase) in the backfill campaign, which are exactly the dirs the metric-reliability table's Rand-init column and the ablation tables read. Apply the splice inside the helper instead, so it returns configs that are usable on their own and the CLI keeps no separate responsibility for it. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

The module documented that a moved PCC "raises rather than writing", but the check lived in the CLI *after* backfill_condition had already written the CSV -- and any other driver (the campaign's shard runner) skipped it entirely. So 13 conditions got recomputed pixel columns merged into caches whose Spectral_PCC, MicroMS3IM, mask and feature values came from different arrays: a mixed-provenance row, which is the one outcome this module exists to prevent. PCC is affine-invariant, so it cannot move under a scaling relabel. When it moves, the cache was not produced from the arrays on disk now, and no column add can fix that -- the leaf needs force_recompute.final_metrics=true. Report the condition `stale_cache` with the worst PCC delta and write nothing; the CLI's exit now keys off that status rather than re-deriving it, so every driver gets the same behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

_classify decides which of the two freshly computed families a cached column reproduces, and the write gate, the audit tally and the "needs re-eval" verdict all key off it. The cases worth fixing are the ones a looser implementation gets wrong: reduction-order drift must be absorbed while a whole-family swap must not be, an all-NaN column carries no provenance rather than matching by default, and two families that agree are reported as a collapse instead of arbitrarily attributed -- otherwise every "both scalings" table would silently show one number twice. Also pins that the work list is grouped from the cached rows, which is what keeps the pass from covering a different position set than the eval did. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

The write gate compared cached and freshly recomputed PCC row by row at rtol=1e-3. That is the wrong scale for PCC: its numerator is a difference of nearly-cancelling sums, so on a near-degenerate prediction float32 cancellation alone moves a row by ~1e-3. FNet3D on HEK mitochondria sits at PCC 0.003 -- no correlation at all -- where 1e-3 is 30% of the value. 19 of 648 conditions in the 2026-08-03 pass tripped the per-row check while their means agreed to four decimals. Three were cross-checked against independent ground truth and all three showed the arrays were unchanged: pix2pix3d mito iPSC-trained -> A549 mock reproduced the published 0.540 as 0.5396, and the two HEK cells agreed to 8.6e-4 and 1.2e-2 against the pre-campaign CSV snapshot. Their prediction and GT stores are both provably older than the evals and untouched since, so there was no data change to catch. Gate on the mean instead, at 5e-4 -- tighter than the 3-decimal rounding the tables publish, so a mean inside it cannot change a cell, while a real re-predict moves it well outside. A per-row failure that clears the mean is reported as reproduced_mean rather than silently collapsed into reproduced, so the audit still surfaces it. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Writes down the trap that nearly cost GPU-days on 2026-08-03. PCC is the right signal for "did this cache come from the arrays on disk now" because it is affine-invariant, but a per-row relative tolerance is the wrong test for it: its numerator is a difference of nearly-cancelling sums, so on a near-degenerate prediction (FNet3D on HEK mito sits at PCC 0.003) float32 cancellation alone moves a row by 30%. rtol=1e-3 per row flagged 19 of 648 conditions as stale and every one that fed a table was a false positive. Also records the four checks that proved them false -- comparing the mean against an independently published number, full-scan chunk mtimes on both prediction and GT, checking for a live writer, and git log on the config surface -- and the rule they add up to: a staleness verdict is a hypothesis, so verify it before spending compute. A gate that fails closed is safe to leave over-sensitive; a false positive must never drive a re-run. Notes in passing that focus.* / slice_selection do not affect pixel metrics, since reaching for them to explain a moved pixel number was one of the wrong turns taken. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

W7a of the brightfield-input ablation, and a hard prerequisite for writing any *__bf* prediction to disk. discover_predictions calls parse_zarr_name unguarded, so an unregistered *__bf* store raises ValueError and breaks grouped-eval regeneration for EVERY bucket, not just the brightfield ones. ipsc__bf -> ipsc_bf_trained and a549__bf -> a549_bf_trained, each its own bucket. The a549__deconv -> a549_trained entry is a misleading precedent: deconv provenance marks how the target was produced from the same Phase3D input, so folding it groups like with like. __bf swaps the input channel itself -- raw Brightfield instead of the waveorder reconstruction derived from it -- so folding would collide a brightfield and a phase prediction on one canonical_identity and let one silently shadow the other. test_walk_predictions_yields_known_buckets asserted equality against a hardcoded 4x3. The brightfield arms cover nucleus and ER only and their predictions land incrementally, so equality would fail throughout the predict drain without ever indicating a problem. Reformulated as "the core grid is present" plus "no bucket label outside the table" -- strictly stronger, since it also catches an unregistered label the old check would have accepted had it been in the hardcoded set. test_parse_zarr_name_out_of_scope_train_set_raises used a549__bf as its out-of-scope example, which this commit makes in-scope. Repointed at a549__bf__deconv, which paths.py documents as a grammar-ready follow-up with no fits behind it. Deliberately NOT regenerating the grouped leaves, which the W7a sketch called for: the prediction tree has grown since they were last generated, so a blanket regeneration folds the 2D in-focus track into all 12 3D buckets -- scoring per-z 2D models on full-volume metrics they never targeted -- and flips force_recompute.final_metrics to true, defeating cache-skip. Verified by regenerating, diffing (2246 insertions across 14 files) and reverting to byte-identical. Brightfield eval will get filtered sibling buckets instead, which is an eval-time concern and no longer blocks the predicts. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

W7b. Four predict sets per arm (own domain plus the three A549 conditions) across 2 train sets x 2 organelles x 2 models, so the brightfield arms get the same 2x2 cross-domain grid the phase models have. Every field is copied from the arm's own PHASE baseline leaf and verified against it: each brightfield leaf's fully composed config differs from its baseline in exactly 9 fields, all intended (source_channel, the normalization key, ckpt_path, output_store, run_root, job_name, experiment_id, trained_on, dataset_ref). A paired ablation must not also vary the predict recipe, so this is asserted rather than assumed. Checkpoint selection follows each model's own fit configuration: * fnet3d arms take the loss/validate-monitored callback's best_model_path, read out of last.ckpt. That file carries TWO ModelCheckpoint callbacks and the unmonitored one's best_model_path is just the final step (latest-epoch=N-step=200000) with best_model_score None -- taking the last callback found silently picks it. The monitored bests are nucleus 0.7935/0.1787 and ER 0.5222/0.5071 for iPSC/A549. * celldiff arms are save_top_k=-1 with no monitor, so the final epoch (epoch=19) is pinned explicitly. Verified that dataset_ref.source_channel resolves to Brightfield in the composed config for all four corners of the grid rather than being dropped -- DatasetRef sets extra="forbid" precisely because a silently ignored source_channel would train and predict on Phase3D while every log claimed brightfield, and a no-op ablation is worse than a failed one. Also confirmed the A549 nucleus leaves still resolve their gene-keyed h2b target and the iPSC-test leaves reach test_cropped. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Each __2d bucket goes from 12 to 16 conditions, adding celldiff_2d x {ipsc, a549 mock/denv/zikv} -- 48 new conditions in total. celldiff_2d was held out of these leaves on purpose: the generator's only freshness gate is pred_path.is_dir(), which passes on a partially-written store (4813 of 5787 chunk files, measured on a live one). Its A549-plate predicts have now finished, so the exclusion is retired. The gate applied by hand in place of the generator's: - sacct: all 48 CELLDiff2D_PRED jobs COMPLETED, ExitCode 0:0. - Per-store chunk counts for all 48 celldiff_2d prediction stores match the same organelle/pool/bucket fnet2d store exactly. The complete size is a property of (organelle, bucket) and not a constant: 4270-4300 on the iPSC buckets, 5211-5787 on the A549 ones. - er/a549__denv is a low outlier at 5211 in every pool, so it was cross-checked against two further fully-evaluated models -- both FCMAE-2D arms, also 5211 over the same 12 FOVs. That closes the one blind spot of an fnet2d-relative gate, which by construction cannot see a shortfall it shares with its own reference. Blocks were lifted verbatim from a generate_grouped_eval_configs.py --out-root run into scratch and inserted immediately before each test set's vscyto2d block, which reproduces the generator's own within-test-set ordering (celldiff_2d, vscyto2d, unext2_2d, fnet2d). The integrity gate is the load-bearing part: after inserting, deleting the 4 new blocks again had to restore every file byte for byte, which passed on all 12 and proves the already-evaluated conditions were not disturbed. Across the 12 buckets the 192 conditions carry 192 distinct pred_cache_dir and 192 distinct save_dir, and all 64 distinct pred_paths exist on disk. force_recompute.final_metrics stays false, against the generator's default of true, so the 12 pre-existing conditions cache-skip -- fnet2d and both FCMAE-2D arms have pixel and feature metrics in all 48 of their save dirs -- and only the 4 celldiff_2d conditions pay the cold deep-feature extraction. That extraction still needs --constraint a100_80|h100_80|h200; bare a100 matches the 40 GB nodes and OOMs. Headers updated in the same pass: scope 12 -> 16, the exclusion paragraph replaced with the gate above, and the cache-skip note corrected to name the three arms that are now evaluated rather than only fnet2d. Tests: generate_grouped_eval_configs_test.py 41/41; test_benchmark_config_composition.py + test_paths.py 1786 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01KWqdbgQhTbBqxnwPBzKHhp

PatchGAN3D layers 1-4 used a cubic kernel_size=4 with padding=1, so a size-1 Z axis pads to 3 against a size-4 kernel and torch raises "Kernel size can't be greater than actual input size" at layer 1. Only layer 5 had been written Z-safe as (1, 4, 4)/(0, 1, 1). That made the discriminator the sole blocker for running pix2pix Z-preserving: the UNetViT generator already builds at [1, 512, 512] with a (1, 4, 4) patch, because it shares ViTBottleneck3D with CellDiff and 0126fe6 gave that per-axis patch extents. preserve_z collapses the kernel and padding on Z for layers 1-4 and drops the Z stride of layers 3-4, so Z passes through untouched rather than merely being tolerated -- a 5-plane slab comes out 5 planes, where the strided 3D topology would halve it twice. MultiScalePatchGAN3D forwards the flag; its inter-scale pooling was already YX-only. Verified: at Z=1 the two scales emit (1,1,1,31,31) and (1,1,1,15,15), the same YX patch grid the 3D config produces, and every conv weight receives a gradient. preserve_z=False reproduces the published 3D topology bit-for-bit -- same kernels, strides and forward output -- which matters because these are the convs all 14 pix2pix3d leaves train. 11/11 tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Fifth model in the 2D-vs-3D benchmark track, after fnet2d, the two fcmae_vscyto2d arms and celldiff_2d. pix2pix2d_unetvit is the same GAN with both halves run Z-preserving -- generator input_spatial_size [1,512,512] and patch_size [1,4,4], discriminator preserve_z: true -- not a distinct architecture, matching how celldiff_2d relates to celldiff. M1 had excluded pix2pix for "fixed 3D input size / 3D patch-embed". Half of that is now stale: the generator shares ViTBottleneck3D with CellDiff, so 0126fe6 already gave it per-axis patch extents, and it builds and round-trips [1,512,512] at 65.3M params (3D: 78.1M) with no new code. The discriminator was the real blocker and needed c2d0481. ⚠ Mirrors Run D, NOT the LSGAN baseline that the 3D train.yml leaves compose. The published pix2pix3d weights came from the 12 train_4gpu_modernized.yml leaves: their checkpoints carry loss_type='nonsat', lr 2e-4/2e-4, r1_gamma=10, ema_kimg=10, lecam_gamma=0.3, lambda_l1=10, 201 generator_ema.* tensors and max_epochs 40. Mirroring the LSGAN overlay would have pitted a 2D LSGAN arm against a 3D Run-D arm and confounded geometry with the recipe. The three Run-D deltas are baked into the overlay rather than repeated in every leaf, and checkpoints monitor loss/validate_ema as in 3D. Twelve leaves: 4 organelles x {ipsc_confocal, a549_mantis, joint}. All read the offline 5-plane `_focus.zarr` slab stores at z_window_size=1, so the manifest resolver is disabled and the data fields are inline. The target fragment's [13,624,624] weighted crop is list-replaced to [1,624,624]; NormalizeSampled(fov_statistics) is inherited unchanged, including in the joint leaves' inline children, where celldiff diverges to MinMaxSampled. Z-flattening follows the fcmae_vscyto2d convention: rotate_range -> [0,0,0] (rotation about axis 0 mixes Y and Z, degenerate at Z=1), scale_range Z -> [1,1], sigma_z -> [0,0], crops -> [1,512,512]. XY augmentation policy is untouched so 2D-vs-3D isolates geometry. Global batch is held at the 3D arm's 16 samples/step: 4-GPU DDP x batch_size 4 single-set, and batch_size 2 x num_samples 2 per joint child since BatchedConcatDataModule does not divide. A GAN cannot absorb a batch change quietly -- DynacellGAN derives the G-EMA decay per-step from the global batch size -- and collapsing onto one rank was measured and rejected: on an A40 (44.7 GiB) at [1,512,512] with bf16 + two AdamW states, peak is 13.3 GiB at bs=4 and 25.0 GiB at bs=8, and bs=16 OOMs. That 13.3 GiB/rank is also why the leaves narrow the hardware_4gpu constraint OFF h100|h200, the opposite of the 3D leaf's h200 pin, and cut mem 512G -> 128G for the 5-plane stores. Both go in the leaf body, since launcher_profiles compose after model overlays. The predict overlay carries both documented pix2pix predict traps: predict_overlap Z must be 0 because window_starts requires overlap < patch, and ema_kimg must stay unstated so engine.py rebuilds the EMA shadow from the checkpoint's own keys instead of silently dropping 201 tensors. sliding_window is still needed at Z=1 -- the reason is lateral, the ViT is fixed at 512x512 and A549 is 640x960. All 12 leaves resolve through submit_benchmark_job.py and were checked on the resolved config, not just the exit code: Z=1 geometry, every Run-D knob, 4x4=16 global batch, focus-store data paths, flattened crops and augs, the ema checkpoint monitor, and the rendered sbatch directives. generate_grouped_eval_configs_test.py 44 passed, 1 failure pre-existing (the 6 HEK leaves' cross_condition_probe key). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…y wall All four TIMEOUTed at hardware_4gpu.yml's 4-day cap and were never resumed, leaving them at 102/200, 86/200, 114/200 and 118/200 epochs: jobs 31822521 (nucl scratch), 33631899 (nucl pretrained, third attempt after 31822355 CANCELLED and 31822498/33557666 FAILED), 31822536 (memb scratch) and 31822529 (memb pretrained) — every one TIMEOUT with elapsed 4-00:00:11 to 4-00:00:29 against a 4-00:00:00 limit. This corrects the campaign's own record, which had these four as "no wall hit — cause unestablished" and gated their resubmission on diagnosing an unexplained stop. There is nothing to diagnose: they are the same failure as the ER/mito four, which already carry this profile. Measured 1.22-1.40 epochs/h, i.e. 143-164 h to reach max_epochs=200. That makes the 7-day wall only 1.02x the slowest requirement, so the profile header now says explicitly that these four fit inside it only because they are resumed (80/81/69/66 h from their current epochs) and that a from-scratch nucleus/membrane joint fit needs a longer wall than this profile provides. _LONG_WALL_TRAIN_LEAVES grows from four leaves to eight with the measured rate attached, as test_only_measured_slow_fits_get_the_long_wall requires; the test guards both directions, so this is the only way the list can move. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…d peak The 12 pix2pix2d_unetvit fit leaves set constraint="a40|a6000|l40s|h100|h200", which test_4gpu_train_leaves_inherit_a100_exclude rejected outright — the leaves shipped in de9ad32 without this suite being run, so it has been red since. The constraint is deliberate and the guardrail's stated reason does not cover it: A100 is excluded for repeat NCCL coordination hangs, and this widening still excludes A100. The >=80 GB floor is a proxy for "cannot OOM", and pix2pix2d peaks at 13.3 GiB/rank at batch_size 4 (measured on the interactive A40, 44.7 GiB usable) — 3.4x headroom on the smallest card admitted. All three added cards can satisfy --nodes=1 --gpus=4 in the gpu partition: gpu-c-1 has 8 a40, gpu-b-[1-6] 4 a6000 each, gpu-g-2 4 l40s (gpu-g-1 has 3 and is simply never selected). The payoff is backfill — each fit is ~35 min (job 35148927 COMPLETED 0:0 in 34:38 at 40/40 epochs), so restricting 12 of them to Hopper queues them behind the 4-day and 7-day fits competing for the same cards. Rather than loosen the rule for everyone, the exception is an explicit allowlist carrying that measurement, mirroring _LONG_WALL_TRAIN_LEAVES. Every other 4-GPU leaf stays pinned to h100|h200, A100 stays excluded for the listed leaves too (no measurement can license a NCCL hang), and a widening that traded Hopper away instead of adding to it now fails. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…finished fits First 8 of the 48 pix2pix2d predict leaves: iPSC-trained and A549-trained nucleus, each against 3 A549 conditions + iPSC. The remaining 10 arms are still fitting and get theirs as each one lands, so the unblocked majority predicts alongside the tail instead of behind it. An arm contributes leaves only once its run log carries `Trainer.fit` stopped: — never a checkpoint filename, which names a best-by-monitor epoch and on the fnet2d fits lagged the true final epoch by 73 (project_checkpoint_write_halt). Both arms here logged max_epochs=40 reached. The baked ckpt is the monitored callback's own best_model_path out of last.ckpt, so it is the epoch the recipe selected on loss/validate_ema: epoch=39-step=20000-loss=0.724 (iPSC) and epoch=39-step=12000-loss=2.340 (A549). Source normalization is NormalizeSampled(Phase3D, fov_statistics, mean/std), reproducing what the fit inherits from targets/nucleus.yml — a MinMax source here (the celldiff_2d idiom) would silently degrade every prediction. dataset_ref overrides target to h2b on the A549 sets only; the iPSC manifest has no h2b and raises TargetNotFoundError if it leaks in. No z_reduction: at z_window_size=1 each plane is its own window, so there is nothing to reduce across, unlike the 3D leaves' 8-plane windows. Composition-verified: all 8 resolve to the intended test store and their own arm's checkpoint at z_window_size=1 with predict_method=sliding_window. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Third arm to finish (`Trainer.fit` stopped: max_epochs=40 reached), so its 4 leaves go out now rather than waiting on the remaining 9 fits. Baked checkpoint is the run's best loss/validate_ema epoch, epoch=39-step=12000-loss=0.648, read from the monitored callback's best_model_path inside last.ckpt. Composition-verified: all 4 resolve target_channel=Membrane against the combined dual_nucl_memb test stores (and iPSC cell.zarr) at z_window_size=1. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Fourth arm to finish. Emitted and submitted by the auto-submitter rather than by hand: it re-runs the leaf generator every 5 min and fires an arm's 4 predicts as soon as that arm's run log carries `Trainer.fit` stopped:, so no arm waits on the slowest of the 12 fits. Baked checkpoint is the run's best loss/validate_ema epoch from the monitored callback's best_model_path inside last.ckpt. Jobs 35150143, 35150144, 35150146, 35150147. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…eaves Fifth and sixth arms to finish, both emitted and submitted by the auto-submitter in the same pass, so they land as one commit rather than two identical ones. Jobs 35150155-158 (joint nucleus) and 35150171-175 (A549-trained ER). The joint arm is the first pix2pix2d leaf set whose fit pooled iPSC + A549, and it needs no batch-semantics adjustment at predict time: the joint/single-set batch_size asymmetry is a BatchedConcatDataModule train_dataloader property, and predict runs a plain HCSDataModule at batch_size 1 from the predict overlay. ER uses targets/er_sec61b.yml and carries no dataset_ref target override — unlike nucleus/membrane, A549 does not re-key ER by gene, so adding one would raise TargetNotFoundError against the iPSC manifest. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Sixth of the twelve pix2pix2d arms to finish its fit. The mito/a549_mantis run reached `Trainer.fit` stopped: `max_epochs=40` reached, so its four predict leaves are now generatable: the three A549 conditions plus the iPSC out-of-domain transfer. Completion is read from the run log, never from a checkpoint filename -- Lightning writes the stop line to stderr, so slurm/*.err and wandb/run-*/files/output.log are searched alongside slurm/*.out. The baked checkpoint is the run's best-by-monitor epoch (epoch=35-step=7848-loss=1.245), not last.ckpt: save_top_k means the highest retained epoch tracks the monitor, not progress. The iPSC leaf carries no dataset_ref override. Only the A549 test sets need one, and leaking the h2b/caax target into an iPSC leaf raises TargetNotFoundError: target not found in dataset 'aics-hipsc'. No z_reduction, unlike the 3D pix2pix leaves: at z_window_size=1 each plane comes from exactly one window, so there is nothing to reduce. predict_method=sliding_window stays, because the ViT generator is fixed at 512x512 by its positional embeddings while A549 is 640x960. test_benchmark_config_composition.py: 1028 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…rse them 5a25142 registered ipsc__bf/a549__bf in _CANONICAL_TRAIN_SET_TO_BUCKET so the brightfield-input ablation would get its own buckets instead of colliding with the phase predictions. But main() and emit_readme() iterate _ORGANELLES x _TRAIN_SETS, and that tuple still listed only the three phase labels -- so the 32 bf conditions parsed cleanly, landed in `buckets`, and were then never written. No error, not even the "skipping empty bucket" line: just four silently absent leaves. Adding both labels to _TRAIN_SETS makes them emittable. A dry-run against the live tree now reports er/nucleus x {ipsc,a549}_bf_trained at 8 conditions each (2 models x 4 test sets = the full 32), and mitochondria/membrane at 0, correctly skipped -- the ablation only covers nucleus and ER. No collision with the phase arms, verified rather than assumed: save_dir_for and pred_cache_dir_for both key on train_set_canonical, so bf lands under ipsc__bf/a549__bf while phase stays at ipsc/a549; condition_name keys on the bucket label, so the two are distinct there too. benchmark_dataset_ref and _gt_cache_dir_for key only on test_set/organelle, which is correct -- __bf swaps the model *input* channel while the fluorescence GT is the same store. Also drops the now-false docstring line on parse_zarr_name still naming a549__bf as an out-of-scope token; the ablation dirs are the live example. test_every_registered_bucket_label_is_emittable pins the invariant that was missing: every label in the bucket table must appear in _TRAIN_SETS. Asserted as a subset relation, so a future half-registered bucket fails rather than disappearing. Suite: 42 passed, 4 deselected. This is capability only -- no leaves emitted here. 12 of the 32 bf predicts are still running, and the generator's only freshness gate is pred_path.is_dir(), which passes on a half-written store, so emission and eval wait on a chunk-count check. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Eighth of the twelve pix2pix2d arms to finish its fit: mito/ipsc_confocal reached `Trainer.fit` stopped: `max_epochs=40` reached, so its in-domain iPSC predict plus the three A549 transfer conditions are generatable. Baked checkpoint is the run's best-by-monitor epoch (epoch=35-step=18000-loss=1.045), identical across all four leaves -- one fit, four test sets. No dataset_ref override in any of the four, which is correct rather than an omission: only nucleus and membrane need one, because their A549 target token (h2b/caax) differs from the iPSC one. TOMM20 is spelled the same in both manifests, so the target resolves without help. Leaking an override into an iPSC leaf is what raised TargetNotFoundError earlier in this arm's generation. test_benchmark_config_composition.py: 1032 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Ninth of the twelve pix2pix2d arms to finish its fit. er/joint reached `Trainer.fit` stopped: `max_epochs=40` reached, so its in-domain iPSC predict plus the three A549 conditions are generatable. Baked checkpoint is that run's best-by-monitor epoch (epoch=34-step=22960-loss=0.671), from the joint model dir, and all four leaves share it. Its step count is the joint tell: 22960 steps over 35 epochs against the iPSC-trained arm's 18000 over 36, because BatchedConcatDataModule pools iPSC + A549 and so runs more steps per epoch. Prediction stores land under er/pix2pix2d_unetvit/joint/<test>/, the canonical joint path -- not the CellDiff-R2 joint_predictions/ layout, which is that family's own exception. test_benchmark_config_composition.py: 1036 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Tenth of the twelve pix2pix2d arms to finish its fit. mito/joint reached `Trainer.fit` stopped: `max_epochs=40` reached, so its in-domain iPSC predict plus the three A549 conditions are generatable. Baked checkpoint is that run's best-by-monitor epoch (epoch=39-step=29280-loss=1.072), shared by all four leaves. Unlike the other joint arms its best epoch is the last one (39/40), so save_top_k retained the final epoch here -- still read from the monitored callback rather than assumed from the filename. Prediction stores land under mito/pix2pix2d_unetvit/joint/<test>/, the canonical joint path. test_benchmark_config_composition.py: 1040 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

2cffa41 fixed the brightfield buckets going unemitted by growing _TRAIN_SETS to match _CANONICAL_TRAIN_SET_TO_BUCKET, then added a test to assert the two agree. That treated a duplication as something to guard rather than something to remove: the same set of bucket labels was written out twice, and only a human (or the new test) kept them in sync. Derive it instead -- tuple(dict.fromkeys(table.values())) -- so a label the table registers cannot be omitted from emission at all. dict.fromkeys dedupes the many-to-one labels (a549/a549__deconv both map to a549_trained) while preserving first-occurrence order. Verified behavior-neutral: the derived tuple is ('ipsc_trained', 'joint', 'a549_trained', 'ipsc_bf_trained', 'a549_bf_trained') -- byte-identical to the hand-written one, so the generated README and bucket emission order do not move. Achieving that needed the two joint rows moved above the a549 rows in the table literal, which is safe for every consumer: the table is read only via .get(), .values() and sorted(), never iterated for meaning. Since literal order is now the only source of README row order, that is called out in a comment above the table. _TRAIN_SETS moves below the table because it now depends on it. test_every_registered_bucket_label_is_emittable is kept rather than deleted: the subset relation is now true by construction, so the test's job becomes failing loudly if anyone re-hardcodes the tuple, which is what would reintroduce the bug class. It also now pins the order, so a table reorder is a deliberate choice rather than a surprise README diff. generate_grouped_eval_configs_test.py 42 passed; full config suite 1082 passed, 119 skipped, 0 failed. --dry-run against the live tree emits the same 5 buckets per organelle in the same order. Found by a reuse/simplification/efficiency/altitude review pass. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…w rationale All 40 pix2pix2d predict leaves restated, in a 3-line header paragraph, why the overlay sets predict_method=sliding_window and predict_overlap Z=0. The overlay already documents it -- and documents it better, because it carries the caveat the copies drop: "(the iPSC test is a single 512x512 tile)". Without that caveat the restatement is wrong in the 10 predict__ipsc_confocal.yml leaves, where it asserts the reason is that "the A549 test set is 640x960" for a leaf that never touches A549. The real reason sliding_window is still needed there is overlay-level, not leaf-level, which is exactly why the overlay owns it. Replaced with a pointer at model_overlays/pix2pix2d_unetvit_predict.yml. This also matches the sibling ViT family: the pix2pix3d_unetvit predict leaves never restate this overlay-owned fact either. Comment-only -- verified no changed line lacks a leading '#', so every composed value is untouched. The scratch generator that emits these leaves was fixed in the same pass, so re-running it now reproduces all 40 files byte-identically (md5 of the set unchanged across a regeneration) instead of reverting them on the next poll. Full config suite: 1082 passed, 119 skipped, 0 failed. Found by a reuse/simplification/efficiency/altitude review pass. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

… NaN the gradient Three joint FCMAE resumes died with OptimizerHealthError 109-169 steps after restoring a healthy checkpoint: finite forward loss, every one of the model's 32.1 M parameter gradients NaN, zero inf, recurring roughly 1 step in 6-27. Resuming from earlier checkpoints does not help and is not even monotonic in checkpoint age (119 -> 169 steps going back 6 epochs on one arm, 117 -> 109 going back 3 on another), which rules out a poisoned weight state; the source checkpoints are clean (0 non-finite across 252 tensors and the optimizer state). detect_anomaly on the real run named MulBackward0 for the sigma_x * sigma_y product feeding the Cauchy-Schwarz bound. That bound was added 2026-07-23 in a556bc9 and did not exist when these checkpoints trained, which is why the same data and config diverge now. The trigger is precision, not structure. On the captured failing batch the input gradient is 0.02% NaN under 16-mixed and exactly 0.00% in fp32: the floored SSIM denominator produces gradients around 1e8, which overflow fp16 to inf, and a later multiply turns those into NaN. _compute_ssim_and_cs_bf16 already selects its own precision on purpose -- bf16 convolutions, fp32 for everything after -- so an enclosing autocast re-entering it defeats that choice. Disable autocast around the call. Two alternatives were measured and rejected: - detaching the bound clears the NaN but drops gradient agreement with the fp32 monai reference from cosine 0.99 to 0.79, because the clamp binds often enough on ordinary inputs that the bound carries real gradient. - splitting into _safe_sqrt(sigma_x) * _safe_sqrt(sigma_y) leaves the NaN entirely, confirming the large product is not the mechanism. The forward moves by ~3e-6 on the captured batch (it now always takes the fp32 path). ms_ssim_25d is not read by the dynacell evaluation pipeline, so no reported metric changes. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Completes the 12-arm pix2pix2d family at 48/48 predict leaves. This arm was the last fit to finish (35151606, COMPLETED 0:0 in 6:53:57), so its four leaves are the ones the auto-submitter emitted last. The fit is verified complete on the Lightning stop reason rather than sacct alone: `Trainer.fit` stopped: `max_epochs=40` reached, epoch 39 / step 32000, and the baked checkpoint carries all 201 generator_ema.* tensors with 0 of 456 non-finite. The EMA tensor count matters here -- a recipe without ema_kimg silently drops them and the predict then runs the raw generator. It took 6:53:57 against 1:13-1:23 for the other joint arms only because it landed on gpu-c-1, an A40 without NVLINK, while the others got h100/h200 nodes. That spread is expected: these leaves deliberately narrow off h100|h200 to backfill better, since the 2D GAN needs just 13.3 GiB/rank. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…uckets Takes each bucket from 16 to 20 conditions (48 new conditions), completing the in-focus 2D track at 5 families. pix2pix2d was held out until now for the documented reason: the generator's only freshness gate is pred_path.is_dir(), which passes on a partially-written store. Store completeness gate (.tmp/pix2pix2d/verify_predict_stores.py), a stronger one than celldiff_2d got. Per array, expected chunk count = prod(ceil(shape / chunks)) read analytically from that array's own zarr.json, compared against the files actually present -- this catches a truncated write inside a position without needing a reference family, which a fnet2d-relative count could not do, since it cannot detect a shortfall shared with its own reference. Paired with a position-set comparison against a family already known complete, because a store missing whole FOVs is internally self-consistent and its own metadata can never reveal the gap. All 48 stores passed both: 100 positions per iPSC bucket, 12 per A549 bucket, every array fully chunked. sacct: all 48 predicts COMPLETED 0:0. Condition blocks are lifted verbatim from a generate_grouped_eval_configs.py --out-root <scratch> run, never hand-written. Each goes immediately after its test set's fnet2d block, which is where the generator itself puts it: its within-test-set order is celldiff_2d, celldiff_r2, vscyto2d, unext2_2d, vscyto3d, unext2, fnet2d, fnet3d, pix2pix2d, pix2pix3d, so among the 2D-only models pix2pix2d sorts last. An integrity gate asserted per file that deleting the 4 inserted blocks restores the original bytes exactly. That is what proves the 16 already-evaluated conditions were untouched -- they cache-skip only if their text is unchanged. Header and condition edits are gated separately so a header rewrite cannot stand in for a condition change. Also verified: 240 conditions across the 12 buckets have 240 unique pred_cache_dir and 240 unique save_dir, all 240 pred_paths exist on disk, and each bucket has 20 unique condition names. force_recompute.final_metrics stays false, so the 16 pre-existing conditions cache-skip -- fnet2d, both FCMAE-2D arms and celldiff_2d all have feature_metrics.csv in 48/48 of their save dirs (checked, not assumed; pix2pix2d is 0/48). Only the 4 pix2pix2d conditions pay the cold deep-feature extraction, which needs --constraint a100_80|h100_80|h200. Headers updated in the same files: scope 16 -> 20, the gate actually applied, and the corrected cache note. That is one concern with the fold. Tests: generate_grouped_eval_configs_test.py + test_benchmark_config_composition.py + test_paths.py = 1898 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

``_final_metrics_cache_valid`` has required the SI_* columns since 36cbfdf ("reject a pixel cache that carries only one metric scaling"), but this fixture's pixel row stayed ``{"metric": "pixel"}``. The gate therefore invalidated the cache it was built to prove reusable, and ``test_evaluate_model_reuses_cache_without_feature_metrics`` has been failing since 2026-08-03. It went unnoticed because test_evaluation_pipeline.py is one of the files the default ``pytest`` invocation does not collect. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

The A549 mock leg of the second pix2pix3d membrane segaux draw (fit 37115317, --ckpt last with the EMA generator), scored while its denv and zikv legs are still predicting. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

The A549 denv leg of the second pix2pix3d membrane segaux draw, whose predict finished after wave3g; zikv is still predicting. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

fnet3d_vscyto3daug_segaux_seed1_e76 is the mito segaux second draw re-predicted from latest-epoch=76-step=200000: its best-val checkpoint is epoch 18, 44-47 epochs before the baselines (62, 65), the same stage gap draw 1's segaux_e76 control covers. Without its own entry the token would collapse onto segaux_seed1. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

The second FNet-3D vscyto3daug mito segaux draw (fit 37115309) on all four legs at its best-val checkpoint (epoch 18, wave3k), and the same draw re-predicted from latest-epoch=76 (ctrl5), since epoch 18 is 44-47 epochs before the baselines. Same protocol as draw 1's wave2w + ctrl3. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

The A549 zikv leg of the second pix2pix3d membrane segaux draw, the last leg of the Spotlight v2 second-draw fits. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

…Cooldown callback (#528) * feat(dynacell): add opt-in channels_last and compile_model flags to DynacellUNet Both default off, so existing leaves and checkpoints behave as before. configure_model applies them before the strategy's DDP wrap: channels_last converts 5-D weights to channels_last_3d and 4-D weights to channels_last (UNeXt2 mixes 2-D and 3-D layers), and compile_model compiles the backbone in place for fit only, keeping state-dict keys. Neither is saved in the hyperparameters, so a resume takes the leaf's value and older checkouts can still resume checkpoints written with the flags on. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * feat(viscy-utils): add LinearLRCooldown callback Ramps every param group's LR linearly from initial_lr to a final fraction between two global steps, for cooldown branches of constant-LR fits. It is stateless (depends only on trainer.global_step), so a resumed fit continues the ramp, and it refuses to start unless the only scheduler is ConstantLR with factor 1, since a step-wise scheduler would overwrite or compound it. The test runs in the dynacell CI job, which has no separate viscy-utils run. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(dynacell): keep the compiled UNet backbone fit-scoped An in-place Module.compile() stayed on the backbone after Trainer.fit, so running predict on the same DynacellUNet afterwards went through the compiled path that the flag promised to keep out of predict (sliding- window shapes vary). The compiled callable now lives in a plain-dict cache, not the module tree, so the state-dict keys stay unchanged. It is used only while fitting and dropped in on_fit_end, or by a callback when a fit raises (Lightning skips on_fit_end then), mirroring the CELL-Diff compiled inference copy. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J --------- Co-authored-by: Alexandr Kalinin <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>

The pix2pix3d membrane SAUNA arm (Phase B1): fit epoch 39 / step 128000, predicted from last.ckpt with EMA like both segaux draws, all four legs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

…> 16/64G Both predict profiles inherited their CPU and memory shape from hardware_h200_single on the stated grounds that "resource accounting is identical". Nothing measured ever justified it, and it is wrong by more than an order of magnitude. Across every COMPLETED predict in Slurm's retention window (2026-07-15 onward, ~1500 jobs, 46 job-name families): peak RSS of any family was 3.58 GiB, and every family but one sat at or below 2.75 GiB, against a 256G request. Peak CPU was 13.6 cores (TotalCPU/Elapsed; fnet2d job 35038580 = 6030 s / 443 s), with family medians of 1.0-4.1 cores. All 1204 predict leaves in the tree compose num_workers: 0, so dataloading is in-process by construction and cannot use the other 16 cores. 16 cores clears the observed maximum; 64G is ~18x the observed peak RSS. Memory was the binding constraint on throughput, not the GPU: a 256G request caps a 500G node at 2 concurrent predicts, so a 4-GPU node stranded two of its GPUs. hardware_predict_celldiff_lite.yml already sizes itself this way from measurement for the same reason. hardware_h200_single is deliberately left alone -- it is shared with 40 fit leaves, so it must not be re-tuned from predict data. The 41 predict leaves on it keep the fat request until they get their own profile. `time` is untouched, so test_predict_leaf_wall_limit_matches_its_family still holds. test_benchmark_config_composition.py and the submit-tool tests both exit 0. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_017dKLDsYRpC1YWJhWNi4sdN

Each *__fcmae3d_rescore bucket was a hand-written fork of its parent, holding the subset of conditions whose predictions had been rewritten by the Phase 18 W4 re-predict. They existed because adding conditions to a finished bucket used to mean rescoring all of it. only_conditions makes that unnecessary: selection happens before anything is scored, cached or loaded, so the unselected conditions cost nothing. Verified equivalent rather than assumed. For all six, every child condition block is byte-identical to its parent's, all non-conditions globals match, and every save_dir is unchanged, so no output moves. _merge_condition rendered to sorted YAML is identical child-standalone vs parent-narrowed, for every condition, in the same order. _with_forced_probe_siblings adds nothing in all six cases -- checked by running it, with a known-positive control (narrowing er_joint to a single mock condition does pull in its DENV and ZIKV siblings, so the empty result is a real pass and not a dead code path). All six parents resolve cleanly through the CLI with their narrowing applied. The parent and the condition list each bucket replaces: er_joint, membrane_joint, nucleus_joint, mitochondria_joint only_conditions=[vscyto3d__joint__a549_denv,unext2__joint__a549_denv, vscyto3d__joint__a549_mock,unext2__joint__a549_mock, vscyto3d__joint__a549_zikv,unext2__joint__a549_zikv, vscyto3d__joint__ipsc,unext2__joint__ipsc] membrane_hek_a549xy, mitochondria_hek_a549xy only_conditions=[vscyto3d__joint__hek] Provenance the headers carried, recorded here because it is nowhere else in the repo: the joint ER fits were re-run to epoch 199/200 and re-predicted 2026-08-06 (8 of er_joint's 24 conditions); the joint membrane fits likewise, re-predicted 2026-08-07 from epoch=148-step=78970-loss=0.374 (pretrained) and epoch=172-step=91690-loss=0.381 (scratch), 8 of 20; the HEK joint VSCyto3D store was rewritten the same day, 1 of 7. The operational rule those headers taught is now in the evaluation README instead of six copies: a re-predict does not invalidate the eval caches, so pass every pred_* force_recompute flag, not final_metrics alone, or the run rescores stale masks and embeddings built from the old predictions. GT-side caches stay warm. nucleus_ipsc_bf_trained__patchaug is deliberately NOT touched. It looks like the same pattern but is not a narrowing: all 8 of its conditions are absent from its parent, so folding it in means regenerating that parent from 8 to 16 conditions, which is a separate change. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_017dKLDsYRpC1YWJhWNi4sdN

Three generators emit grouped eval buckets and all stamp an "Auto-generated by" line: generate_grouped_eval_configs.py, generate_spotlight_v2_eval_configs.py and generate_lite_benchmark_configs.py. Hand-written buckets carry no stamp, and both kinds live in one directory under one filename, so nothing in the path tells them apart. At present that is 238 generated and 65 hand-written. The risk is asymmetric regeneration: a regen rewrites only the stamped set, so a hand-written bucket beside them is left untouched and can go stale against a parent that moved, with nothing failing. Registering the hand-written ones makes adding another a deliberate act instead of the default path, and makes a generated leaf that lost its stamp fail loudly rather than quietly becoming unmaintained. Two tests, in both directions: every unstamped bucket must be registered, and every registered name must still exist and must still be unstamped (so the registry cannot outlive its buckets or shadow a generated one). Verified to discriminate rather than merely pass -- each failure mode was induced and seen to fail: - drop a real hand-written bucket from the registry -> first test fails naming it - register a bucket that does not exist -> second test fails - give a registered bucket an "Auto-generated by" line -> second test fails telling you to drop it The registry is a record, not a budget; adding to it is fine. The comment points at only_conditions as the cheaper alternative to forking a leaf, which is what the six *__fcmae3d_rescore buckets turned out to need. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_017dKLDsYRpC1YWJhWNi4sdN

41 predict leaves composed hardware_h200_single.yml, a FIT profile that 48 fit leaves (44 train.yml + 4 train_smoke.yml) also compose. It therefore cannot be re-tuned from predict data, so those predicts were stuck requesting the fit shape of 32 cores and 256G. New hardware_predict_h200.yml breaks the coupling. Sized from measurement. Across every COMPLETED predict of these leaves that sacct retains -- 19 jobs, all fcmae_vscyto3d_pretrained -- peak RSS was 2.02 GiB and peak CPU 2.92 cores (TotalCPU/Elapsed, job 34880997 = 4628 s / 1586 s), median 2.41 cores. All 41 compose num_workers: 0 and batch_size: 1, so dataloading is in-process and cannot use more. 16 cores and 64G match the other predict profiles, leaving ~5x and ~30x headroom. Affected: 25 fcmae_vscyto3d_pretrained, 8 vscyto3d_cytolandft, 8 vscyto3d_infectionft_dynacellft. Two things deliberately NOT changed, both recorded in the profile: constraint stays h200, and is not a measured requirement -- predict is full_image at 512x512x15 fp32, and a sibling leaf of the same model dir (constraint a40|...|h200) COMPLETED on gpu-f-5. The pin stays because GPU type perturbs outputs here (cross-GPU mask floors, reduction-order PCC), so unpinning risks moving published numbers on a re-predict. That is a separate decision, not a side effect of this one. time stays at 4 days. 16 of the 41 leaves still compose ckpt_path: REPLACE_ME_WITH_PRODUCTION_CHECKPOINT_PATH and have never completed a run, so their wall need is unmeasured; the tree's convention is that an unmeasured family inherits a conservative cap rather than a speculative one. The 25 that do run top out at 0.65 h. hardware_h200_single is now absent from _PREDICT_PROFILE_TIME rather than listed, so a predict leaf pointed back at the fit profile fails loudly instead of inheriting an unmeasured wall. Verified to discriminate: repointing one leaf at it fails with "unknown hardware profile 'hardware_h200_single.yml'". Also corrects the fit-leaf count I wrote in the previous profile commit and in the test comment: 48, measured, not 40. test_benchmark_config_composition.py exits 0. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_017dKLDsYRpC1YWJhWNi4sdN

Speedup pilot P2: the nucleus FNet-3D vscyto3daug v2 baseline branched at 38 / 60 / 100% of its 200k-step constant-LR budget and cooled linearly to zero LR, each predicted from its final weights. Registers the three PAPER_KEY tokens so they do not collapse onto v2, and adds the ctrl6 wave (cool38 and cool60 all legs, cool100 three legs; its a549__denv store is still predicting). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

Speedup pilot P3: the UNeXt2-3D membrane segaux_seed1 arm forked from the v2_seed1 baseline's epoch-123 checkpoint instead of trained from scratch, predicted from its best-val epoch 149 (inside the e123-e178 range of the runs it is compared with). Registers its PAPER_KEY token and adds the ctrl7 wave, all four legs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

…into dynacell-spotlight-v2-eval-waves

wave3n: the Phase B2 FNet-3D vscyto3daug nucleus loss-weight sweep (segaux at 0.5x and 2x seg_aux_weight), both at --ckpt best epoch 51, inside the baselines' epoch 64 / 74-control range; all four legs. ctrl6b: the last P2 budget-control leg, cool100 a549__denv. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

pix2pix3d_unetvit_segaux_sauna_seed1 re-draws the membrane SAUNA arm with seed_everything: 1 (SEED_SOURCES), at the same calibrated seg_aux_weight 2.9. Draw 1 passed the B1 rules with an A549 FG_PCC gain over both baselines and both segaux draws, which rule R3 does not decide on a single draw. Registers the PAPER_KEY and grouped-eval tokens and generates the train leaf and four predict leaves. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

…ed flags (#529) trainer.benchmark: true on the FCMAE (UNeXt2-3D) fit overlays and trainer.benchmark + model.init_args.channels_last on the FNet3D fit overlays. Measured on H200 with the real dataloader: UNeXt2-3D 4-GPU DDP 0.930 s/step vs 4.322 1-GPU default (compile slower under DDP, so off); FNet-3D 0.493 vs 0.918 s/step; training-loss numerics within the replicate floor. Predict overlays are untouched, so predict leaves resolve unchanged. A data-driven test keyed on the composed architecture pins the flags on every 3-D FCMAE/FNet3D train leaf, keeps them off every other train leaf and off these families' predict leaves. Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J Co-authored-by: Alexandr Kalinin <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>

The four fit overlays carried two copies of the same 4-5 line rationale (found by /simplify on #529). Keep it in the primary overlay of each family, cite the 1-GPU measurement that isolates the flag (0.900 vs 4.322 s/step; the 4-GPU row mixed in the GPU count), and point the copy overlays at it. The fit-only reason is the user's decision to keep predicts on the defaults their stores were made with; "kernel choice only" argued the opposite. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

From /simplify on #529: list only the flags each family sets (no None placeholders or default dict), have the helper return the expected flags directly, and name the family FCMAE (a separate timm UNeXt2 architecture is not covered). Drop the predict-leaf test: no predict leaf composes a *_fit.yml overlay (0 of 1219), so it could not fail, and it skipped 722 of its 1219 cases. Mutation check: benchmark false in fcmae_vscyto3d_fit.yml fails 35 train leaves; composition file 2141 passed, 190 skipped. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

/code-review on #529: the FCMAE overlay said the cudnn.benchmark train loss was within the replicate floor, but the measured gap is 1.25x the floor, from one replicate pair over 400 steps inside the LR warm-up (speedup RECOMMENDATIONS.md s7). Only FNet3D (0.92-0.95 over 3,000 steps) supports "within the floor". Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

… focus2d eval (#526) * feat(viscy-data): predict only listed Z planes per position and timepoint A 2D model predicts every plane of the full-Z test store, while the 2D benchmark scores only the focus slab, so most of its predict time and storage goes to planes nothing reads. HCSDataModule.predict_z_planes takes a JSON plane file (position -> per-timepoint plane list) and restricts the predict windows to it (z_window_size=1). The writer allocates the full source extent so the store stays aligned plane by plane with its source (skipped planes keep fill value 0), records the predicted planes per channel (viscy_predicted_z_planes, cleared again by a full run) so a reader can refuse to score zeros, and carries z_planes_sha256_12 in the completion marker so a resume never mixes two plane selections. The launcher's --resume-predict hashes the same file, read as JSON to keep the launcher free of torch. Measured on fnet2d ER, A549-trained, A549 mock (A40): the slab3 store equals the existing full-Z store on all 360 slab planes (max |diff| 0) and holds no other plane; 0.78 vs 12.2 GB on disk; predict loop 60 s vs 930 s at the same ~6 windows/s. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * feat(dynacell): score the 2D benchmark on one focus plane (focus2d) The 2D benchmark scored only ER/mito Dice on the focus plane (segmentation.semantic_focus_halfwidth); every other metric still integrated the 2D model's per-plane predictions over the full volume, and a 2D model that predicts only its focus slab could not be scored at all. focus2d.halfwidth replaces that key and moves every metric to one GT-anchored plane per FOV/timepoint (segmentation.focus_anchor): - pixel metrics, MicroMS3IM (and its leaf calibration pool) and CP on the plane itself; 2D SSIM / Spectral PCC, FRC replaces the XY/Z FSC - deep features and per-cell similarity on the 2h+1 slab around it - semantic masks segmented on the slab and scored on the plane; instance targets must use the same slab (validated) GT/pred caches move to <cache_dir>/focus2d_h<h>/ and record the plane recipe, refusing another: the CP cache key carries no z geometry, so a shared dir would silently score 2D prediction rows against cached 3D GT rows. CP uses its own <target>__focus2d_h<h> reference, built by build_cp_reference.py --focus2d from the focus2d leaves' per-dataset plane recipes. The recipe is stamped in metrics_provenance.json and compared both ways, and a grouped bucket cannot mix modes. precompute-gt build.focus_planes writes the plane file a 2D model predicts with, resolved by the same function the eval uses. The eval refuses a plane-restricted store that lacks its slab (e.g. another anchor) and any plane-restricted store under full-volume scoring. The eight ER/mito slab3 buckets move to focus2d.halfwidth=1; their masks are unchanged, and their pixel rows are now on the plane. Measured on fnet2d ER, A549-trained, A549 mock: focus2d eval of the full-Z store and of its slab3-only re-predict agree on all 120 pixel and mask rows (max |diff| 0). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * feat(dynacell): predict 2D leaves on the focus2d slab only Every 2D predict leaf (the five benchmark families and their ablation arms) now sets data.init_args.predict_z_planes to focus2d_planes/<organelle>/<test>__h1.json and writes prediction_focus2d_h1.zarr beside the existing full-Z prediction.zarr, which stays untouched. A 2D model then predicts only the 3-plane slab the focus2d eval reads: ~1/16 of the planes on A549 (48 -> 3), ~1/13 on iPSC. Setting predict_z_planes to null (and the store back to prediction.zarr) predicts every plane as before. The 150 Spotlight-v2 arm leaves are regenerated from their edited baselines by tools/generate_spotlight_v2_leaves.py (--check: 407 leaves, 0 stale), so the ccond arm reads its FNet segaux mask from that arm's slab3 store on the same planes. The plane files are written by precompute-gt build.focus_planes; the predict and eval stores they index are the same file for all 16 organelle/test pairs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * feat(dynacell): add focus2d grouped eval buckets for the 2D benchmark grouped/<organelle>_<pool>__2d_focus2d holds the 20 conditions of each canonical <organelle>_<pool>__2d bucket (5 2D families x 4 test sets), scored with focus2d.halfwidth=1 and written under eval_focus2d_h1/; <organelle>_phase_floor__2d_focus2d scores the raw Phase3D input on the same planes. Anchors: nucleus_area for nucleus/membrane and iPSC ER/mito, phase_midband for A549 ER/mito (no nuclei channel). They read the existing full-Z prediction stores, which hold the slab plane for plane (a slab3 re-predict scored identically), so no re-predict is needed. Generated by experiments/2026-10-07_2d-vs-3d-cost/make_focus2d_buckets.py (gitignored). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): let focus_plane_stack pass the 3D benchmark through focus_plane_stack(vol, None) now returns vol, so the four "full volume or focus-plane stack" ternaries and the duplicated fov_cp_features call in precompute-gt become one unconditional call each. The plane cut reuses slab_mip with halfwidth 0, the module's documented single-plane slicer, instead of a third slicing idiom. The focus helpers move to the module-level import in pipeline.py. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): read the eval's GT nuclei through gt_nucleus_volume _process_one_fov kept its own copy of "which store and channel hold the GT nuclei" next to the gt_nucleus_volume helper the calibration and precompute-gt use, plus two inline "the nucleus target is its own nucleus volume" expressions. _gt_nuclei now memoizes the helper, so the eval, its MicroMS3IM calibration and the predict plane file place the plane from one definition. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): build the uncovered-slab map in one comprehension The coverage check built the per-timepoint differences and then filtered the empty ones in a second dict; one filtered comprehension does both. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): resolve the CP reference halfwidth via read_focus2d_config resolve_cp_reference_path read focus2d.halfwidth with a raw select, a second unvalidated reading of the key: a negative halfwidth named a '__focus2d_h-1' reference instead of raising like the rest of the eval. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): carry focus estimator params into the focus2d CP reference build read_focus2d_config folds the focus.* estimator params into the phase_midband plane recipe, but build_cp_reference only merged three segmentation keys from the leaves, so a leaf overriding focus.* would build a config whose plane recipe differs from the eval's cache. No focus2d leaf sets focus: today, so the built references are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * test(dynacell): fold the hand-built plane-store check into the predict test Both tests asserted that a plane-restricted store scores like the full one; the real HCSDataModule predict already produces such a store, so the two refusal checks move onto it and the zero-filled copy goes, saving two end-to-end evaluate_predictions runs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(viscy-utils): canonicalize predicted planes in one helper z_planes_sha256_12 and record_z_planes each spelled out the same sorted, de-duplicated int conversion; _canonical_planes now serves both, so the run hash and the recorded planes cannot drift apart. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(viscy-utils): derive the writer's position name once write_on_batch_end resolved the same position twice on consecutive lines by two expressions, one through a single-use _position_name helper. One name now indexes both the plate and the plane map, and the skipped-window mask indexes planes directly (fancy indexing needs no sort or de-dup). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): hash the predict plane file through read_z_planes The launcher parsed the plane file's positions itself, a second reader of the schema, on the stated grounds of keeping torch out. Importing the launcher already loads viscy_data.sliding_window (measured: the extra import costs 0.0 s), so it now calls read_z_planes. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(viscy-utils): refuse a plane-restricted predict into another run's store overwrite rewrites chunks in place, so a slab predict into a store a full-Z (or other) run wrote left that run's planes around the slab: the store kept its old size and voxels from other weights, and an existing short array was never grown to the source extent. Run positions now must hold no prediction channel another run wrote; rerunning the same plane run (resume) still passes. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): keep prediction.zarr as the 2D predict leaves' store The slab3 leaves wrote prediction_focus2d_h1.zarr, a name the store grammar (key_from_prediction_store), both eval-config generators and the focus2d buckets never read, so a re-predict would have been invisible to every eval. The store name stays prediction.zarr: the writer records the planes, the eval refuses a full-volume score of a slab store, and the writer refuses a slab predict over a full-Z one. The leaf comment no longer says null switches the store name. The 150 Spotlight-v2 leaves are regenerated (--check: 407 leaves, 0 stale). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): refuse build.focus_planes under limit_positions The plane file holds only the positions the loop visits, so a smoke run with limit_positions atomically replaced the full file and every predict on it then failed on the missing positions. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): refuse a full-volume eval of a slab store before calibration The refusal ran per FOV, after the leaf-level MicroMS3IM calibration had already fit on the store's zero planes. evaluate_predictions now scans the recorded planes of every prediction position up front. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): keep the plane file's path out of the predict settings hash prediction_settings_sha256_12 hashed predict_z_planes as a setting, so moving or copying an unchanged plane file changed settings_sha256_12 and --resume-predict refused the store, although z_planes_sha256_12 already identifies the plane selection by content. The path is now loading-only, like data_path. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): name --focus2d in the 2D CP reference's rebuild hints A focus2d eval loads <target>__focus2d_h<h>.json, but the missing-file, recipe, dataset and GT-cell-mismatch errors all suggested the 3D build command, so following the hint built <target>.json and the eval still failed. _build_command now takes the reference path and adds --focus2d <h> for a 2D-benchmark reference, read back from the name cp_reference_path gives it. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * docs(dynacell): trim the slab3 predict leaves' comment to what the leaf decides The two lines on removing a full-Z store repeated the writer's refusal and the dynacell CLAUDE.md note in 268 copies; the leaf keeps only the switch it owns (null predicts every plane). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(viscy-utils): fold the plane-run refusal into the writer's mixed check The plane-restricted refusal was a second list, close and raise for the rule _cannot_share already expresses; a plane run simply extends that check to its own FOVs, so one condition and one message cover both. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): resolve focus2d once per evaluate_predictions call Three branches of evaluate_predictions each re-read the focus2d recipe from the config; one local now serves the up-front refusal, the MicroMS3IM calibration and the warm-pass skip. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): parse the CP reference's focus2d name beside its builder _build_command recovered the halfwidth with a regex that restated the <target>__focus2d_h<h>.json format cp_reference_path defines; the inverse now lives next to it in paths.py, so a rename of the suffix cannot silently drop --focus2d from the rebuild hints. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * docs(dynacell): drop a comment that restates the limit_positions error Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(viscy-utils): let a slab predict replace another run's same-slab store The plane-run refusal fired on any marker mismatch, so re-predicting a slab store with a new checkpoint (--ckpt best) on the same plane file was refused although it rewrites every plane the store holds. Refuse only when another run's recorded planes differ from this run's (a full-Z store records none), with its own message naming the remedy (remove that store); out-of-run FOVs keep the unchanged mixed check. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): refuse a full-volume eval of a slab store before writing The refusal ran inside the with block that opens the eval dir's segmentation_results.zarr with mode="w", so a mistaken full-volume eval truncated that store before refusing. The recorded planes are now checked on their own read of the prediction before anything is opened for writing. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): refuse an unfinished plane-restricted prediction in the eval A plane run writes its started marker before its first batch but records its planes only on completion, so an interrupted slab predict read as a full-volume store: the focus2d coverage check saw no planes to check and the full-volume refusal saw no plane restriction, and both scored unwritten planes as zeros. plane_predict_unfinished flags a marker that names a plane selection with no planes recorded, and evaluate_predictions refuses such a store up front in both modes. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(viscy-utils): tell an unfinished plane predict by its marker's source_shape plane_predict_unfinished keyed on missing recorded planes, but a re-predict of the same slab replaces only the marker on start and keeps the previous run's planes until it completes, so an interrupted one looked finished and a focus2d eval could score a mix of two runs' planes. A plane run's marker gains source_shape only on completion, so its absence is the signal in every case. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc --------- Co-authored-by: Alexandr Kalinin <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>

…ckets Phase B2: the nucleus FNet-3D vscyto3daug segaux_halfw arm lost A549 Dice on all three legs when predicted from its best-val epoch 51. Its re-prediction from latest-epoch=76-step=200000 (the e74 stage-control floor) tests whether that loss is a checkpoint-stage artifact. Registers its PAPER_KEY token and adds the ctrl8 wave, all four legs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

* feat(dynacell): score interior-only foreground metrics by default Spotlight v2 judged models on foreground-limited pixel metrics, because whole-image pixel metrics are dominated by background voxels. Turn the FG_* columns on in eval.yaml with a hard GT interior mask (feather_sigma_um: 0.0); feather_sigma_um: null keeps the per-target feathered recipe from metrics.FOREGROUND_SIGMAS_UM. The weight map is built from the GT alone and only weights the scoring; neither image is zeroed, so whole-image columns are untouched. FOREGROUND_METRICS_VERSION is not bumped: no FG_* value moves for a given recipe, and the recipe is already stamped in metrics_provenance.json, so an existing save dir recomputes its final metrics once on its next run. Hand-built test configs omit the block, so a compose test of the real eval.yaml pins the new default. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(dynacell): give nucleoli and lysosomes foreground sigma defaults With FG metrics on by default, an eval whose target has no FOREGROUND_SIGMAS_UM entry raises before scoring, and nucleoli and lysosomes are accepted targets without one. They take mitochondria's sigmas (a region mask around compact or punctate bodies), marked untuned because DynaCell has no GT for them. A composition test now resolves the default FG settings for every accepted target; the null-sigma error test moves to a target outside the table. New entries move no existing FG value, so FOREGROUND_METRICS_VERSION stays. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * test(dynacell): keep the otsu sigma test on a target with no default test_otsu_source_resolves_no_smoothing_sigma switched to lysosomes to prove the otsu source needs no smooth_sigma_um default. This PR gives lysosomes a default, so the check passed whether or not otsu looks one up. Use golgi, as the sibling resolution test already does. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * test(dynacell): pin the FG default for every target from the evaluator's list Merge the two new FG-default tests into one parametrized test that checks the exact resolved settings per target. The expected smoothing sigma comes from FOREGROUND_SIGMAS_UM, and the target list from the new segmentation.EVAL_TARGETS, which prepare_segmentation_model now uses too, so a target added to the evaluator is covered automatically. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * refactor(dynacell): derive the nucleoli and lysosomes FG sigmas from mito They take mitochondria's sigmas by design; copying the literal would let them drift silently if mitochondria is retuned. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * docs(dynacell): state the FG default once eval.yaml named FOREGROUND_SIGMAS_UM twice in one comment, and the tables.py comment copied the default, which is how its old "off by default" went stale. The default now lives in eval.yaml and the README defaults table only. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(dynacell): copy, not alias, the nucleoli and lysosomes FG sigmas The previous derivation bound both entries to mitochondria's dict, so an in-place edit of one (a test monkeypatch or a sensitivity sweep) changed all three. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * docs(dynacell): describe the default FG weight as the hard interior mask eval.yaml and the README metrics table still called the weight a soft map; under the new default (feather_sigma_um 0) it is binary, feathered only when feather_sigma_um is positive or null. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J --------- Co-authored-by: Alexandr Kalinin <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>

The *__2d_focus2d buckets from #526 carry no "Auto-generated by" stamp: they were emitted by an experiment script outside the tracked generators (experiments/2026-10-07_2d-vs-3d-cost/make_focus2d_buckets.py), so a regen of the generated set never rewrites them. #526 branched before the guard landed in 3ae322e and CI has no dynacell job, so the merge left test_every_grouped_leaf_is_generated_or_registered_hand_written failing on dynacell-models, naming exactly these 16. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

…gistry 3ae322e built _HAND_WRITTEN_BUCKETS from the main checkout's directory listing, which holds 26 grouped buckets that have never been committed (*__a549only, *__ipsconly, *_celldiff_r2_*_later, the 2D sample- and voxel-matched finals, er_a549_trained__temporal_rescore, ...). In any clean checkout they do not exist, so test_registered_hand_written_buckets_all_exist_and_are_unstamped failed on dynacell-models naming all 26. The registry records hand-written buckets in the repo. A checkout that still carries those untracked dirs now fails the other test, naming them, which is the guard doing its job: commit the ones worth keeping, or remove them. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

…uckets The nucleus FNet-3D vscyto3daug segaux arm (draw 1) gained A549 FG_PCC when predicted from its best-val epoch 48, but ctrl8's halfw arm lost that gain by epoch 76 and draw 2 (epoch 57) never showed it. Its re-prediction from latest-epoch=76-step=200000 tests whether the gain is a checkpoint-stage effect. Adds the ctrl9 wave for the three legs whose predicts finished; a549__zikv follows in ctrl9b. The PAPER_KEY token already exists from the mito control. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

The a549__zikv leg of ctrl9 (nucleus fnet3d_vscyto3daug_segaux re-predicted from latest-epoch=76-step=200000), split out because its predict finished after the other three legs' evals had started. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

…into dynacell-spotlight-v2-eval-waves

B3b of Spotlight v2. The membrane segaux arm's Dice target is a CLAHE+Otsu mask that fills 40-66% of the plane, so the term rewards thick blobs rather than walls; filter-based thin masks (B3) failed validation. This arm keeps the segaux recipe and swaps only the target: cpdino whole-cell outer outlines on the GT membrane, per plane, inside a wall band around the nucleus-area anchor, canonical mask elsewhere. The mask is a new array in a derived training store that mirrors the iPSC cell.zarr (plate metadata and position order identical, image arrays symlinked), so the fit leaf disables the manifest resolver and declares data_path and both channels inline. The thin mask changes the Dice term's gradient scale, so the weight is calibrated on its own: on the same April checkpoint and val batches, read from the derived store, the canonical mask reproduces 0.8609 and fg_mask_cpout gives 0.5533 [0.397-0.886] -> 0.55. The arm keeps its segaux source's 7-day wall. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

) * feat(viscy-utils): add a segmenter-consistency loss SegConsistencyLoss matches a frozen segmenter's dense outputs on the prediction and the target: MSE on the two flow channels plus binary KL (BCE with logits minus the target entropy) on the cell probability, so the loss is 0 at pred == target. S(target) runs under no_grad on the same crop every step. A 3-D crop is reduced to the slab MIP around the target's nuclear-area plane, as the eval segments it; an optional Ctrl-U weight down-weights pixels where S disagrees with its flipped view. The segmenter stays out of the module tree, so checkpoints and strict ckpt_path loads are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(dynacell): wire the segmenter-consistency term into the UNet and GAN engines DynacellUNet and DynacellGAN take seg_consistency + seg_consistency_weight, added next to seg_aux (training loss / generator objective). loss/validate stays base-only; the term is logged as loss/segcons_* (UNet) and loss/g_segcons_* (GAN) in training and loss/validate[_ema]_segcons in validation. The weight has no default: a term with weight None (the leaf placeholder) is refused at build, as is a weight without the term. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(dynacell): add a differentiable Cellpose-SAM wrapper CellposeDense returns cellpose flows and cell-probability logits that are differentiable w.r.t. the input, ported from the G4 gate: normalize99 with no-grad statistics, 256-px tiles, taper blend, loaded by absolute path from cellpose's model dir (no silent fallback or download), cpdino refused. With a gradient, tiles run under activation checkpointing (checkpoint_tiles, default 1): a stored fwd/bwd holds ~5.6 GB per tile. Tests: parity with CellposeModel.eval on identical normalized input, finite-difference gradient, checkpointing parity, and one DynacellUNet step through SegConsistencyLoss (GPU + cpsam only). Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(dynacell): add P1 segmenter-consistency fine-tune arms (nucleus) Spotlight-v2 generator arms segcons / segcons_ctrl on FNet-3D vscyto3daug, UNeXt2-3D and pix2pix2d nucleus: both fork the baseline recipe's final checkpoint weights-only (model.init_args.ckpt_path) and fine-tune for 10% of its budget at 0.1x its peak LR; segcons adds SegConsistencyLoss with cpsam (G4's S_train). seg_consistency_weight is a null placeholder until calibrated, which the engines refuse. Predict leaves read last.ckpt. Registers the six tokens in PAPER_KEY and the grouped parser, and admits the pix2pix2d forks to the <80 GB cards like their baseline. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(dynacell): calibrate the P1 segcons weights seg_consistency_weight per arm is the median over 20 seeded val batches of ||grad base|| / ||grad seg_consistency|| at the fork checkpoint (calibrate_variant.py segcons variant, A40): FNet-3D 0.48 [0.25-0.93], UNeXt2-3D 0.050 [0.024-0.088], pix2pix2d 0.57 [0.32-0.67]. The tool's plain-Dice row reproduces the recorded FNet-3D vscyto3daug 4.5 on its July checkpoint (4.504 [2.58-6.72]). Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(dynacell): reject seg_consistency combined with seg_aux With both terms the logged loss/base_train would be the seg_aux base, not the loss before the consistency term. No arm combines them. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(viscy-utils): drop unused SegConsistencyLoss options and the 4-D path No arm uses max_samples or the Ctrl-U uncertainty weight, and every engine hands the loss 5-D (B, 1, Z, Y, X) crops (Z = 1 for 2-D models), so the loss now requires 5-D. The z <= 1 early return in nucleus_area_planes is redundant: the guard band of a one-plane stack is plane 0. Co-Authored-By: Claude Opus 5.5 <[email protected]> * perf(viscy-utils): build the Otsu histogram without a host sync torch.bincount sizes its output on the host, so each focus-plane pick synced the GPU. scatter_add_ into a fixed (rows, 256) histogram does not. Thresholds are bit-identical to the bincount version (new test), and a CUDA sync-debug test pins the no-sync claim; it trips on the old version. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(dynacell): simplify CellposeDense normalize99 is hard-coded to 1/99 with float quantiles (no per-call host-to-device tensor), the taper uses BSIZE, checkpointing keys on the tiles' requires_grad alone, and the train() override and requires_grad_ call go: SegConsistencyLoss freezes its segmenter, and the cpsam test now checks that it stays in eval with no grads inside the loss. The net is still put in eval once at construction: cellpose builds it in train mode, whose flows differ from stock inference by up to 1.7 (parity test). Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(dynacell): pin SegConsistencyLoss's focus port to the eval nucleus_area_planes / _slab_mip against dynacell.evaluation.focus nucleus_area_plane / slab_mip on 50 blob volumes (Z = 1..40, a constant volume, random scale and offset): planes and slabs (halfwidth 0-2) equal. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(dynacell): drop the segcons null-weight placeholder path All three weights are calibrated, so SEG_CONSISTENCY_WEIGHTS is typed float and the placeholder header branch goes. The generator test no longer rebuilds each arm with a null weight (3 extra ViT-L loads); the engine refusal is covered by test_seg_consistency_training. --check: 467 leaves, 0 stale, no leaf changes. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(dynacell): skip cellpose tests without cellpose; fix a docstring cellpose is an optional dynacell extra: test_cellpose_dense importorskips it at module top, and the generator's cpsam build test does so in-test so the rest of that module still runs. The gradient test's docstring now says what it asserts. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(dynacell): crop the segcons projection to 240 px, one cellpose tile cellpose get_pad_yx pads a 256-px image to 272, which make_tiles cuts into 2 x 2 tiles: 4x the segmenter compute per sample. A 240-px crop pads to exactly 256, one tile. SEG_CONSISTENCY_ARGS and the loss default move to 240; CellposeDense.tiling exposes the padding/tile layout, and a test pins one tile at the configured crop (and 4 at 256). Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(dynacell): shorten the pix2pix2d fork's EMA half-life Both pix2pix2d arms predict from generator_ema, which starts as the fork checkpoint's shadow. At the baseline's ema_kimg 10, the 1000-step x 16-sample fine-tune leaves 0.5^(16000/10000) = 33% of that start shadow in it; ema_kimg 2 leaves 0.5^8 = 0.4%. Set for both arms of the pair. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(dynacell): keep CellposeDense in eval under train() Restores the train() override next to the construction-time eval(): the network's train mode changes its outputs (flows off by up to 1.7 vs stock inference), so a parent's train() must not reach it if the wrapper is ever registered as a submodule. Tested through a ModuleDict parent. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(dynacell): reject a non-positive seg_consistency_weight A zero weight builds the segmenter and pays its cost for no term; turning the term off means dropping seg_consistency. The gradient test now checks the step's backbone gradient equals the base gradient plus weight x the term's own gradient, instead of comparing against a zero-weight arm. Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(dynacell): drop the stale seg_consistency weight placeholder wording The weights are calibrated, so the engine's docstring and error describe a missing weight (no default) rather than a null leaf placeholder, and the generator docstring no longer mentions a null weight. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(dynacell): check every fork-checkpoint tensor in the control build The strict-load test compared only the first state_dict tensor; it now requires every tensor of the built control to equal the fork checkpoint's. Co-Authored-By: Claude Opus 5.5 <[email protected]> * feat(dynacell): recalibrate the segcons weights at the 240-px crop Same protocol at the leaves' one-tile crop (results/*__segcons_crop240): FNet-3D 0.37 [0.14-0.65], UNeXt2-3D 0.032 [0.024-0.066], pix2pix2d 0.19 [0.067-0.34], from 0.48 / 0.050 / 0.57 at the 4-tile 256-px crop. The plain-Dice instrument rows are unchanged to every printed digit. The comment records the convention's known limitation: segaux-style base (pix2pix2d: generator_ema, lambda_l1 * L1, no adversarial term) and no training-batch averaging, kept for comparability with the segaux weights. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(dynacell): keep the global RNG unchanged when building CellposeDense CellposeModel random-initializes its ViT-L before loading the checkpoint, which advanced the global torch RNG between seed_everything and HCSDataModule's torch.randperm train/val split. A segcons arm therefore split its FOVs differently from its same-seed segcons_ctrl, confounding the comparison (measured: seed 42 randperm differs after construction). Construction now runs inside torch.random.fork_rng; a test pins the torch state (it fails on the previous code). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J --------- Co-authored-by: Alexandr Kalinin <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>

alxndrkalinin and others added 30 commits August 3, 2026 12:46

alxndrkalinin and others added 30 commits October 10, 2026 05:25

Merge branch 'dynacell-models' of https://github.com/mehta-lab/VisCy …

cb981ce

…into dynacell-spotlight-v2-eval-waves

Merge branch 'dynacell-models' of https://github.com/mehta-lab/VisCy …

7a224b8

…into dynacell-spotlight-v2-eval-waves

Navigation Menu

Repository navigation

DynaCell v1.1 - #494

DynaCell v1.1#494
alxndrkalinin wants to merge 782 commits into
mainfrom
dynacell-models

alxndrkalinin commented Jul 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

alxndrkalinin commented Jul 29, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants