Repository navigation

Version 3 with cached cross chunk edges - #454

Draft

akhileshh wants to merge 368 commits into

mainfrom

This pull request is big! We're only showing the most recent 250 commits

Commits on Feb 25, 2026
Commits on Feb 26, 2026
- add buildkit env for cloudbuild
  akhileshh
  committed
  df96222
Commits on Feb 28, 2026
- remove bigtable client, reorganize tests, improve redis+rq ops
  akhileshh
  committed
  45dddf5
Commits on Mar 4, 2026
- upgrade kvdbclient
  akhileshh
  committed
  803acd6
Commits on Mar 18, 2026
- remove multiwrapper from ingest module
  akhileshh
  committed
  e758d68
Commits on Mar 20, 2026
- feat(sv_split): track max sv id to create new ids; convert ws seg to ocdbt
  akhileshh
  committed
  90e8258
- feat(sv_split): metadata changes to support ocdbt seg
  akhileshh
  committed
  a2a51f9
- feat(sv_split): split sv, update seg and edges, read and write new edges from pcg
  akhileshh
  committed
  ea78867
- feat(sv_split): sv split in frontend
  akhileshh
  committed
  14338c6
- fix(sv_split): update multicut tests, add other tests
  akhileshh
  committed
  fb50497
- use kvdbclient, organize tests
  akhileshh
  committed
  17c5577
- regenrate requirements
  akhileshh
  committed
  0c1cd1f
- fix: use registered attributes module
  akhileshh
  committed
  deff2c0
- feat(ingest): make ocdbt seg optional
  akhileshh
  committed
  328f98e
- add ocdbt flag to ingest cli
  akhileshh
  committed
  9c0778a
- bugfix: use labels for cx edges after split with inf affinity
  akhileshh
  committed
  18d3198
- optimize whole sv lookup, use rep sv from source
  akhileshh
  committed
  5e82105
- fix: use relative ocdbt path in info
  akhileshh
  committed
  b659906
- fix: inf edges to only closest partner; mesh worker pubsub version, always sv lookup from seg for ocdbt
  akhileshh
  committed
  7b27599
- remove multiwrapper usage and references
  akhileshh
  committed
  621a811
- use a common logger, cleanup old mess
  akhileshh
  committed
  174156b
Commits on Mar 21, 2026
Commits on Mar 22, 2026
Commits on Apr 16, 2026
Commits on Apr 21, 2026
- ocdbt: write_seg writes base scale only
  akhileshh
  committed
  5ddd4d1
- edits: forward SV-split bbox to publish_edit for async downsample
  akhileshh
  committed
  ff8caff
- downsample: async pubsub worker updates coarser mips
  akhileshh
  committed
  d936782
- deps: add tinybrain
  akhileshh
  committed
  8a269ea
Commits on Apr 22, 2026
Commits on Apr 23, 2026
- feat(ocdbt): pinned-at reads
  akhileshh
  committed
  affa672
- refactor(sv-split): pure planner + consolidated persist
  akhileshh
  committed
  f216de7
- feat(locks): IndefiniteL2ChunkLock + exception-path hold
  akhileshh
  committed
  984634d
- feat(repair): stuck_ops — cleanup-then-replay for SV-split ops
  akhileshh
  committed
  ff6f215
- deps: kvdbclient>=0.7.0
  akhileshh
  committed
  a7fae00
- refactor(sv-split): thread operation_ts + named structs
  akhileshh
  committed
  4b7aa1d
- feat(stuck_ops): surface exception-path + verify cells before replay
  akhileshh
  committed
  f19eb4d
- include kvstack spec in info
  akhileshh
  committed
  56923af
Commits on Apr 24, 2026
- feat(meta): require fork to be pre-created; assert fork_exists
  akhileshh
  committed
  7672ceb
- docs: scope-set stuck signal; pre-replay cell verification
  akhileshh
  committed
  ed7f9f0
Commits on Apr 28, 2026
- sv-split: drive bbox from user seeds, narrow cut set to bbox
  akhileshh
  committed
  0eace5e
- docs: edge update after sv split
  akhileshh
  committed
  8eec8ca
Commits on May 7, 2026
- sv discrepancy error message details
  akhileshh
  committed
  4216ba6
- update gitignore, add hybrid ws seg plan
  akhileshh
  committed
  d7fcd2e
Commits on May 13, 2026
- unify ocdbt sv lookup for 2d and 3d clicks
  akhileshh
  committed
  df4ac6a
Commits on May 19, 2026
Commits on May 20, 2026
Commits on May 21, 2026
Commits on May 22, 2026
- feat(ingest): consolidate OCDBT lifecycle under `ingest layer`; add task-body mode flags
  akhileshh
  committed
  6b373a6
Commits on May 23, 2026
- tmp: disable ocdbt seg read for meshing
  akhileshh
  committed
  8ca9743
Commits on May 24, 2026
- fix(ocdbt): self-heal stale fork manifests; re-enable OCDBT seg reads
  akhileshh
  committed
  0cb09c1
Commits on May 25, 2026
Commits on May 26, 2026
- feat(debug): per-block memory + IO tracking on HierarchicalProfiler
  akhileshh
  committed
  6d8caa7
- feat(graph): PCG_DRY_RUN env var to suppress edit-flow writes
  akhileshh
  committed
  fcf37a7
Commits on May 27, 2026
Commits on May 28, 2026
Commits on May 29, 2026
- perf(sv-split): write fresh IDs into the seg crop in place
  akhileshh
  committed
  05b4373
- fix(sv-split): align chunk lattice to voxel offset, drop redundant root mask
  akhileshh
  committed
  3309803
Commits on May 31, 2026
- Add scheduling package for weight-balanced work partitioning
  akhileshh
  committed
  cfea904
- Make HierarchicalProfiler fold-on-record, fork-safe, and tree-reported
  akhileshh
  committed
  d129f44
- Add cgroup peak-memory reader for pod sizing
  akhileshh
  committed
  c2c7405
- Add parallel sharded mesh stitching, mesh-equivalent to the serial path
  akhileshh
  committed
  d245d88
- Add profiling + correctness harness for the parallel stitch
  akhileshh
  committed
  538eb3c
- Decode draco buffers as zero-copy views; deprecate the serial stitch fn
  akhileshh
  committed
  3f729c8
- Dispatch the parallel stitch from the mesh task
  akhileshh
  committed
  63d43ef
- Remove unused ocdbt edge put/get helpers
  akhileshh
  committed
  1a1025b
- Harden parallel stitch for heavy chunks
  akhileshh
  committed
  6bc0152
Commits on Jun 1, 2026
- Run each initial-mesh chunk in a forked child that exits
  akhileshh
  committed
  caaa067
- Reclaim mesh-buffer heap after the stitch shard write
  akhileshh
  committed
  4182aed
Commits on Jun 2, 2026
Commits on Jun 3, 2026
Commits on Jun 4, 2026
Commits on Jun 6, 2026
Commits on Jun 7, 2026
Commits on Jun 8, 2026
Commits on Jun 11, 2026
- feat(pipeline): cross-branch chunk-batch core (ingest + meshing)
  akhileshh
  committed
  1fd457f
- feat(pipeline): pcgv3 ingest dispatch + migration workload
  akhileshh
  committed
  1a8c48f
Commits on Jun 12, 2026
- fix(pipeline): force-exit workers to dodge bigtable thread
  
  akhileshh
  and
  claude
  committed
  d124757
- fix(pipeline): port chunk lock to kvdbclient primitives
  
  akhileshh
  and
  claude
  committed
  d5b225b
- fix(logging): stop duplicate log lines, print chunk coords as ints
  
  akhileshh
  and
  claude
  committed
  a913ae2
- feat(pipeline): verify hierarchy after the root chunk is built
  
  akhileshh
  and
  claude
  committed
  3eab490
Commits on Jun 13, 2026
- fix(pipeline): route every entrypoint through one run_and_exit helper
  
  akhileshh
  and
  claude
  committed
  1d8e26b
- feat(ingest): --exist-ok skips create when the graph table exists
  
  akhileshh
  and
  claude
  committed
  d1c3697
Commits on Jun 15, 2026
- refactor(pipeline): adopt cave-pipeline for the distribution core
  
  akhileshh
  and
  claude
  committed
  a1ebe1c
- build: bump cave-pipeline to 0.0.3
  
  akhileshh
  and
  claude
  committed
  750f4d5
Commits on Jun 16, 2026
- perf(graph): read watershed via tensorstore
  
  akhileshh
  and
  claude
  committed
  abfa8c7
- perf(graph): defer graph_tool/pandas/networkx/cloudfiles off import path
  
  akhileshh
  and
  claude
  committed
  07036ec
- perf(sv-split): adopt fastremap.point_cloud(shell=True) in build_coords_by_label
  
  akhileshh
  and
  claude
  committed
  19cd3d2
- fix(meshing): read watershed at requested mip; derive chunk_size
  
  akhileshh
  and
  claude
  committed
  30f9487
- fix(pipeline): delegate mesh-meta setup to the meshing library
  
  akhileshh
  and
  claude
  committed
  1e90ad0
- build: derive version from git tags via setuptools_scm
  
  akhileshh
  and
  claude
  committed
  4feda63
- perf(imports): confine cloudvolume to the meshing package
  
  akhileshh
  and
  claude
  committed
  2756be1
Commits on Jun 17, 2026
- feat(ingest): stamp mesh ingest boundary at root-layer build
  
  akhileshh
  and
  claude
  committed
  fd8633c
- fix(build): set version only from version-like tags
  
  akhileshh
  and
  claude
  committed
  383797f
Commits on Jun 18, 2026
- fix(meta): isolate copied-table mesh dirs by layout
  
  akhileshh
  and
  claude
  committed
  335e657
- fix(version): stamp tables from a committed _version.py literal
  
  akhileshh
  and
  claude
  committed
  3f13889
Commits on Jun 20, 2026
- fix(meshing): handle top-layer chunks in calculate_stop_layer
  
  akhileshh
  and
  claude
  committed
  72ccfda
Commits on Jun 21, 2026
- Merge pull request #534 from CAVEconnectome/akhilesh/sv-splitting-locks
  akhileshh
  authored
  3918789
- fix(deps): bump scipy>=1.18.0 to dodge broken cp314 1.17.1 wheel
  
  akhileshh
  and
  claude
  committed
  530bfe6
Commits on Jun 25, 2026
- fix(deps): make conda the sole numpy source, drop it from the pip lock
  
  akhileshh
  and
  claude
  committed
  10a1b70
Commits on Jun 27, 2026
- docs(readme): document the release workflow
  
  akhileshh
  and
  claude
  committed
  3c00b0b
Commits on Jun 29, 2026
- test: configure pytest discovery in pyproject
  
  akhileshh
  and
  claude
  committed
  99943c4
- test: drop unused gen_graph_with_edges fixture
  
  akhileshh
  and
  claude
  committed
  d1123b1
- test: extract fake_timestamp helper for the shared past timestamp
  
  akhileshh
  and
  claude
  committed
  bfedd19
- test: add build_graph helper deriving the parent hierarchy
  
  akhileshh
  and
  claude
  committed
  62bbed8
- test: build_graph spec uses named SV coordinates
  akhileshh
  committed
  e08bbe2
- test: build_graph returns BuiltGraph(cg, sv)
  akhileshh
  committed
  d345c14
- test: migrate graph-test builders to the build_graph factory
  
  akhileshh
  and
  claude
  committed
  55c65f0
- test: eliminate hardcoded node coordinates from the suite
  
  akhileshh
  and
  claude
  committed
  c529db7
- test: group edit-operation tests into an edits package
  
  akhileshh
  and
  claude
  committed
  57653c6
- test: keep stale-edge tests on a between-chunk edge
  
  akhileshh
  and
  claude
  committed
  b340ac5
- test: cut real-time sleeps from the slowest lock/error tests
  
  akhileshh
  and
  claude
  committed
  3ef7798
- refactor: discover test backends from kvdbclient (>=0.8.0)
  
  akhileshh
  and
  claude
  committed
  8677404
Commits on Aug 10, 2026
- fix(ingest): size worker pools by the pod's cpu, not the node's
  
  akhileshh
  and
  claude
  committed
  7d8a643
- fix(meshing): use the parallel stitch, bounded by the pod's cpu
  
  akhileshh
  and
  claude
  committed
  228516a
- fix(deps): require cave-pipeline 0.0.4 and pin the conda solve
  
  akhileshh
  and
  claude
  committed
  20e3f1e
Commits on Aug 12, 2026
- fix(meshing): size shard upload parts to bound compose calls
  
  akhileshh
  and
  claude
  committed
  851ed8c