Skip to content

Measure fl_valley_attribute() at watershed-group scale before the driver relies on it #44

Description

@NewGraphEnvironment

Problem

fl_valley_attribute() (#40) is measured only on the bundled 518,400-cell tile at k = 5 groups.
The driver (NewGraphEnvironment/floodplains#40) wants to run it on a watershed group — MORR is
~27M cells, k = 33 by gnis_name and k = 340 by blue_line_key. Nothing establishes that the
k = 5 timing extrapolates, and there is a specific reason to think it may not.

What is measured

bundled tile, 648 x 800 = 518,400 cells
  fl_valley_confine()                      1.36 s
  fl_valley_attribute() by gnis_name (k=5) 0.74 s
  fl_valley_attribute() by blue_line_key   0.68 s

Attribution is cheaper than the delineation at this scale.

Why that may not hold

The saving comes from cropping each group's cost distance to its own bounding box plus
crop_margin. Two things degrade that:

  1. A bbox is a poor proxy for a long sinuous mainstem. The Morice's bounding box is close to
    its whole AOI, so the largest and most important group gets little or no crop benefit. On the
    bundled tile the per-group crops are already 39-74% of the full grid, and at k = 5 they sum
    to ~276% of a full-grid pass.
  2. Cost is per group. k = 340 near-full-grid costDist() passes on 27M cells is a different
    order of magnitude from five passes on 0.5M — potentially hours, not the fraction of a
    delineation the small tile suggests.

The docs and both issue bodies currently say the k = 5 number and explicitly decline to claim
k = 340. This issue is to replace that hedge with a measurement.

What to do

  • Time fl_valley_attribute() on one real watershed group (MORR is the motivating case) at
    k by gnis_name and k by blue_line_key, against the fl_valley_confine() time for the
    same AOI
  • Record peak memory — cells_by_group holds cell indices for every group simultaneously, and
    the fallback allocates two full-extent rasters
  • Report the crop efficiency actually achieved: per-group crop area as a fraction of the AOI
  • Update the Performance section of fl_valley_attribute() and the claim in
    Delineate and attribute floodplains per watercourse/reach, not only per watershed group floodplains#40 with real numbers

If it is too slow

Options, roughly in order of effort:

  1. Buffer the geometry instead of the bbox. Crop to a crop_margin buffer around the group's
    lines rather than their bounding box — a sinuous mainstem's buffer is far smaller than its bbox.
    Likely the single biggest win and contained entirely within fl_group_cells().
  2. Coarsen the default grouping. gnis_name (k = 33 on MORR) may be all the driver needs;
    blue_line_key at k = 340 can be opt-in.
  3. One multi-source pass carrying seed identity — approach C in floodplains#40 as literally
    written. A single cost accumulation propagating a seed label, rather than k passes. Loses exact
    per-group thresholds and cannot represent overlap natively, so it is a real trade, not a
    drop-in.

Do not optimize before measuring — option 1 is cheap but pointless if k = 340 is already fine.

Relationship to #35

#35 is the WSG-scale pipeline (multi-AOI iteration, memory, on-disk intermediates). This issue is
narrower: does attribution's per-group cost pass hold up at WSG scale at all? The answer determines
whether #35's iterator can call fl_valley_attribute() per AOI as a normal step, or whether
attribution needs its own memory and chunking story inside that pipeline.

Sequence it before #35's design settles: #35's option (b) is "no new in-flooded chunking, leverage
terra's facilities", which is a reasonable call for fl_valley_confine() on one AOI and an untested
one for k=340 cost passes plus a cells_by_group list that holds cell indices for every group at
once. If the measurement here says attribution is the memory ceiling, that changes #35(b).

Keep them separate: this is a measurement with a yes/no outcome, #35 is a design.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions