Problem
fl_valley_attribute() (#40) is measured only on the bundled 518,400-cell tile at k = 5 groups.
The driver (NewGraphEnvironment/floodplains#40) wants to run it on a watershed group — MORR is
~27M cells, k = 33 by gnis_name and k = 340 by blue_line_key. Nothing establishes that the
k = 5 timing extrapolates, and there is a specific reason to think it may not.
What is measured
bundled tile, 648 x 800 = 518,400 cells
fl_valley_confine() 1.36 s
fl_valley_attribute() by gnis_name (k=5) 0.74 s
fl_valley_attribute() by blue_line_key 0.68 s
Attribution is cheaper than the delineation at this scale.
Why that may not hold
The saving comes from cropping each group's cost distance to its own bounding box plus
crop_margin. Two things degrade that:
- A bbox is a poor proxy for a long sinuous mainstem. The Morice's bounding box is close to
its whole AOI, so the largest and most important group gets little or no crop benefit. On the
bundled tile the per-group crops are already 39-74% of the full grid, and at k = 5 they sum
to ~276% of a full-grid pass.
- Cost is per group. k = 340 near-full-grid
costDist() passes on 27M cells is a different
order of magnitude from five passes on 0.5M — potentially hours, not the fraction of a
delineation the small tile suggests.
The docs and both issue bodies currently say the k = 5 number and explicitly decline to claim
k = 340. This issue is to replace that hedge with a measurement.
What to do
If it is too slow
Options, roughly in order of effort:
- Buffer the geometry instead of the bbox. Crop to a
crop_margin buffer around the group's
lines rather than their bounding box — a sinuous mainstem's buffer is far smaller than its bbox.
Likely the single biggest win and contained entirely within fl_group_cells().
- Coarsen the default grouping.
gnis_name (k = 33 on MORR) may be all the driver needs;
blue_line_key at k = 340 can be opt-in.
- One multi-source pass carrying seed identity — approach C in floodplains#40 as literally
written. A single cost accumulation propagating a seed label, rather than k passes. Loses exact
per-group thresholds and cannot represent overlap natively, so it is a real trade, not a
drop-in.
Do not optimize before measuring — option 1 is cheap but pointless if k = 340 is already fine.
Relationship to #35
#35 is the WSG-scale pipeline (multi-AOI iteration, memory, on-disk intermediates). This issue is
narrower: does attribution's per-group cost pass hold up at WSG scale at all? The answer determines
whether #35's iterator can call fl_valley_attribute() per AOI as a normal step, or whether
attribution needs its own memory and chunking story inside that pipeline.
Sequence it before #35's design settles: #35's option (b) is "no new in-flooded chunking, leverage
terra's facilities", which is a reasonable call for fl_valley_confine() on one AOI and an untested
one for k=340 cost passes plus a cells_by_group list that holds cell indices for every group at
once. If the measurement here says attribution is the memory ceiling, that changes #35(b).
Keep them separate: this is a measurement with a yes/no outcome, #35 is a design.
Problem
fl_valley_attribute()(#40) is measured only on the bundled 518,400-cell tile at k = 5 groups.The driver (NewGraphEnvironment/floodplains#40) wants to run it on a watershed group — MORR is
~27M cells, k = 33 by
gnis_nameand k = 340 byblue_line_key. Nothing establishes that thek = 5 timing extrapolates, and there is a specific reason to think it may not.
What is measured
Attribution is cheaper than the delineation at this scale.
Why that may not hold
The saving comes from cropping each group's cost distance to its own bounding box plus
crop_margin. Two things degrade that:its whole AOI, so the largest and most important group gets little or no crop benefit. On the
bundled tile the per-group crops are already 39-74% of the full grid, and at k = 5 they sum
to ~276% of a full-grid pass.
costDist()passes on 27M cells is a differentorder of magnitude from five passes on 0.5M — potentially hours, not the fraction of a
delineation the small tile suggests.
The docs and both issue bodies currently say the k = 5 number and explicitly decline to claim
k = 340. This issue is to replace that hedge with a measurement.
What to do
fl_valley_attribute()on one real watershed group (MORR is the motivating case) atk by
gnis_nameand k byblue_line_key, against thefl_valley_confine()time for thesame AOI
cells_by_groupholds cell indices for every group simultaneously, andthe fallback allocates two full-extent rasters
fl_valley_attribute()and the claim inDelineate and attribute floodplains per watercourse/reach, not only per watershed group floodplains#40 with real numbers
If it is too slow
Options, roughly in order of effort:
crop_marginbuffer around the group'slines rather than their bounding box — a sinuous mainstem's buffer is far smaller than its bbox.
Likely the single biggest win and contained entirely within
fl_group_cells().gnis_name(k = 33 on MORR) may be all the driver needs;blue_line_keyat k = 340 can be opt-in.written. A single cost accumulation propagating a seed label, rather than k passes. Loses exact
per-group thresholds and cannot represent overlap natively, so it is a real trade, not a
drop-in.
Do not optimize before measuring — option 1 is cheap but pointless if k = 340 is already fine.
Relationship to #35
#35 is the WSG-scale pipeline (multi-AOI iteration, memory, on-disk intermediates). This issue is
narrower: does attribution's per-group cost pass hold up at WSG scale at all? The answer determines
whether #35's iterator can call
fl_valley_attribute()per AOI as a normal step, or whetherattribution needs its own memory and chunking story inside that pipeline.
Sequence it before #35's design settles: #35's option (b) is "no new in-flooded chunking, leverage
terra's facilities", which is a reasonable call for
fl_valley_confine()on one AOI and an untestedone for k=340 cost passes plus a
cells_by_grouplist that holds cell indices for every group atonce. If the measurement here says attribution is the memory ceiling, that changes #35(b).
Keep them separate: this is a measurement with a yes/no outcome, #35 is a design.