diff --git a/README.md b/README.md index 7f69ecc9..d32e6959 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # pyphoenix-project +> A beginning is the time for taking the most delicate care that the balances are correct.
— Princess Irulan + ![pay no attention](pay_no_attention.jpg) we're refactoring, promise diff --git a/docs/_config.yml b/docs/_config.yml index acacbfe3..10b55880 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -2,17 +2,6 @@ title: flopy4 / pyphoenix author: MODFLOW Development Team logo: "" -execute: - execute_notebooks: "off" - -html: - use_issues_button: true - use_repository_button: true - -repository: - url: https://github.com/modflowpy/pyphoenix-project - branch: develop - path_to_book: docs - -sphinx: - extra_extensions: [] +# plans and design notes are read in the source tree (see dev/plans/README.md) +exclude_patterns: + - dev/plans/** diff --git a/docs/_toc.yml b/docs/_toc.yml index 66d493ec..60d63856 100644 --- a/docs/_toc.yml +++ b/docs/_toc.yml @@ -15,6 +15,8 @@ parts: - caption: Guide chapters: + - file: dev/concepts + title: Concepts and terminology - file: examples/period-data title: Period data - file: mf6-versions @@ -29,6 +31,6 @@ parts: - file: dev/srs - file: dev/sdd - file: dev/map - - file: dev/protocols-plan - - file: dev/array-converter-design + - file: dev/mmp-plan - file: dev/test + - file: dev/load-benchmarks diff --git a/docs/dev/concepts.md b/docs/dev/concepts.md new file mode 100644 index 00000000..a7a793ae --- /dev/null +++ b/docs/dev/concepts.md @@ -0,0 +1,254 @@ +# Concepts and terminology + +flopy4 draws its vocabulary from MODFLOW 6, xarray, scientific Python and +database design. This page says what its words mean and how they map to MF6 and +flopy3. The first half is for anyone using flopy4. The second is for people +working on it, or reading its design docs. + +## Names in transition + +This page uses the names flopy4 is moving to. Some of them haven't reached the +code yet, so until then use the name on the right. + +| This page | In the code today | +|---|---| +| `to_xarray()` gives a `DataTree` for every component | a package gives a `Dataset` | +| `dense()` on the `grid` accessor | `to_xarray()`'s default for a list package | + +## For users + +### What a simulation is made of + +| Term | Meaning | MF6 | flopy3 | flopy4 | +|---|---|---|---|---| +| **simulation** | The unit of work: one or more models, and how they're timed, solved and coupled | simulation, `mfsim.nam` | `MFSimulation` | `Simulation` | +| **model** | One simulated process on one grid, such as groundwater flow | model | `ModflowGwf` | `Gwf` | +| **package** | One input file's worth of a model or simulation, such as a stress or the grid | package | `ModflowGwfwel` | `Wel` | +| **utility** | A package that provides a capability rather than a hydrologic process, like observations or time-varying conductivity | utility | `ModflowUtlobs` | `Obs` | +| **exchange** | A connection between two models | exchange | `ModflowGwfgwf` | `Gwfgwf` | +| **solution** | The numerical solver for a set of models | solution, such as IMS or EMS | `ModflowIms` | `Ims` | + +MF6 divides stress packages into two kinds. **Simple stress packages** (CHD, WEL, +DRN, RIV, GHB, RCH and EVT) can change their boundaries between periods. +**Advanced packages** (MAW, SFR, LAK and UZF) define their features once. + +### Time + +| Term | Meaning | MF6 | flopy3 | +|---|---|---|---| +| **stress period** | A division of a simulation's time. Period data is given by stress period. | stress period | stress period | +| **time step** | A division of a stress period that MF6 solves for | time step | time step, `kstp` | +| **`kper`** | A stress period's number in Python, starting at 0 | `iper` in a file | `kper`, also from 0 | +| **`iper`** | A stress period's number in an input file, starting at 1 | `iper` | | + +So `stress_period_data={0: ...}` in Python writes `BEGIN PERIOD 1` in the file. MF6's +source also has a `kper`, which starts at 1. You'd only meet it reading the Fortran. + +### How a package takes period data + +MF6 has three **input forms**, plus the advanced packages' period blocks. flopy4 +has a class for each form, since RCH and RCHA take different arguments. The file +type is the same (both are `RCH6`), and an option in the file says which form it +uses. + +| Form | Option | Examples | flopy3 | +|---|---|---|---| +| **list-based** | none | `Wel`, `Chd`, `Rch` | `ModflowGwfwel`, `stress_period_data` | +| **layer array-based** | `READASARRAYS` | `Rcha`, `Evta` | `ModflowGwfrcha` | +| **grid array-based** | `READARRAYGRID` | `Welg`, `Chdg` | `ModflowGwfchdg` | + +Once a block has been given, a period with no block carries on from the last one, in +every form. Blocks that are present behave differently, so changing a package's form +can change what your input means: + +| Form | A new block | Not given in the block | An empty block | +|---|---|---|---| +| list-based, and MVR | replaces all entries | removed | clears all entries | +| layer array-based | replaces the arrays it names | keeps its value | doesn't turn anything off, so give a rate of zero | +| grid array-based | replaces the arrays it names | keeps its value, by position | clears all entries | +| advanced | changes only what it names | keeps its setting | changes nothing, so deactivate through settings | + +### What a row refers to + +| Term | Meaning | Addressed by | Stays the same across periods? | +|---|---|---|---| +| **feature** | A thing an advanced package defines once, such as an SFR reach, a lake, a multi-aquifer well or a UZF cell | its number, `ifno`, declared in PACKAGEDATA | yes | +| **list entry** | A row in a list-based period block | its position in that block | no | +| **grid entry** | A cell with a stress in a grid array-based block | its position among the block's cells with a value, in node order | no | +| **boundname** | A label on a row, enabled by `BOUNDNAMES` | not an address | see below | + +Entry numbers can change meaning between periods. Boundary 2 can be one well in +period 1 and another in period 2, so MVR and SPC rows that use a number have to +match that period's order. If you insert, delete or reorder a period's rows, check +anything that refers to them by number. + +A boundname isn't an ID. Several rows can share one, and a flow observation of a +boundname reports their sum. + +A layer array has no entries at all. It has one value per vertical column of the +grid (`ncpl`), and nothing refers to one by number. + +### Xarray + +| Term | Meaning | +|---|---| +| **`to_xarray()`** | Exports a component to xarray as it's held: what you gave, in the periods you gave it. A list stays a table of rows. `from_xarray()` reads it back. | +| **`dense()`** | Computes the array you'd usually want to plot or do arithmetic on: values placed on the grid, for every period. It's computed from the export, and isn't stored. | +| **`kper`** | The period dim of a dense array. In a table of rows, a coordinate saying which period each row is in. | +| **`entry`** | A period list's row dim, one entry per row, with `kper` as a coordinate on it. | +| **`_kper`** | The periods given for one array, as in `recharge_kper`. Each array has its own, since each can be given in different periods. | + +### Other terms you'll meet + +| Term | Meaning | Note | +|---|---|---| +| **name file** | The file listing a simulation's or a model's components | Two words, as in mf6io. | +| **cellid** | A cell's address: layer, row and column on DIS, layer and cell on DISV, node on DISU | Written from 1 in a file. | +| **user node**, **reduced node** | A cell's number counting every cell, and counting only cells left in the grid by `IDOMAIN` | Inputs use user nodes. | +| **`x_world`**, **`y_world`** | A cell's coordinates after the grid's origin and rotation are applied. `x` and `y` are local to the grid. | MF6 gives `XORIGIN`, `YORIGIN` and `ANGROT`. flopy3 calls local coordinates "model coordinates". | + +## For developers + +### Choosing a term + +Prefer, in order: + +1. **MF6's term,** if it has one. Users read mf6io. +2. **A domain standard** (CF, UGRID, xarray, SQL), at that domain's boundary. +3. **An established theory term,** in design docs only, never in user docs. +4. **A new word,** only with a definition here. + +Lead with a plain statement, and give a theory term once in parentheses. Drop it if +it doesn't help settle a real question. + +### Glossary + +| Term | Meaning | MF6 | flopy3 | Theory | +|---|---|---|---|---| +| **DFN** | A definition file: the spec for one kind of input file, such as `gwf-wel` | DFN | DFN | | +| **component** | A node in a simulation's tree. Each kind has one DFN. | differs: in IDM, `component` is `GWF` and `subcomponent` is `RCHA`, and SPC is the "Stress Package Component" | | | +| **`WorkspaceComponent`** | A component with a workspace, namely a simulation or model. A model under a simulation uses the simulation's. | | | | +| **field** | A variable on an attrs class or in a DFN. For a value spread over cells, say "gridded quantity". | variable | variable | | +| **`ftype`** | flopy4's one attribute for a component's file type | `ftype`, `mtype`, `exgtype`, `slntype` | | | +| **`bound`** | The role of a package whose rows are boundaries. In prose say feature, list entry or grid entry. | `bound`, `nbound` in source | | | +| **binding** | The row or option in one file that attaches a component to the simulation: a `PACKAGES` row, or `OBS6 FILEIN` | | | | +| **binding arguments** | What the binding knows and the file doesn't, like which two models an exchange connects | | | | +| **layout** | How an MF6 file writes things down: blocks, keywords, arrays or lists, `CONSTANT`, `OPEN/CLOSE`, unit factors | | | physical schema | +| **logical** | What the data is: tables, columns, keys, and settings that change what the model computes | | | logical schema | +| **runtime memory** | MF6's own variables, as listed in the DFN memory catalog. A level, but not the logical one. | memory | | | +| **label** | What an option is. **Layout**: re-encoding the data to match leaves the model unchanged. **Logical**: describes the model. **Directive**: changes only what's run, printed, saved or checked. **Mixed**: only CSUB's `HEAD_BASED`. | | | | +| **`schema` option** | An option that decides which logical columns or tables exist, or what other components can refer to. `AUXILIARY` adds columns, `BOUNDNAMES` adds a name column others can refer to, and `SORPTION` decides which arrays exist. It's an attribute some options carry, separate from their label. | | | row polymorphism, dependent record | +| **layout preservation** | The rule that the object model holds input as given, layout included, and logical forms are views over it | | | concrete vs. abstract syntax (an analogy) | +| **view** | Something computed from held data and never stored. Say whether it's read-only or two-way. | | | external schema | +| **lens** | A two-way mapping between a layout and the logical relation. Its **get** reads the view, and its **put** writes an edit back. Its **complement** is what the view lacks and `put` needs. It's **partial** if some logical content has no form in that layout, as with two RCH rows in one vertical column under RCHA. | | | lens, with its laws | +| **observation** | What logical content is judged against: results, what can be referred to, metadata a conversion must keep, and results after a permitted edit | | | observational equivalence | +| **stage** | When a derivation's result is fixed: **constant** (unit factors), **edit** (`STORAGECOEFFICIENT`, safe to compute at write time only), or **run** (`AUXMULTNAME`, TS, TAS). Say "fixed at", not "bound at", since binding means something else here. | | | binding-time analysis | +| **snapshot**, **delta** | A list block is a snapshot, carried until replaced. An advanced block is a delta. | | | step function at change points | +| **sparse table** | A union record's rows read as one logical row, as with `1 STATUS INACTIVE` and `1 RATE 0.1` | keystring | | entity-attribute-value | +| **representation** | flopy4's xarray form of a simulation. Files and other libraries' conventions are adapters over it. | | | | +| **NetCDF profile** | One way the representation is written to NetCDF. The **grouped archive** has a group per component, and MF6 doesn't read it. **MF6 NetCDF** is ungrouped, and MF6 does. | | | | +| **type tag** | The marker saying what kind of component an exported node holds | | | | +| **discriminator** | What selects among variants: a union row's keyword, or `READASARRAYS` for a class | | | sum type, with arms | +| **key domain** | The set of values a key can take, such as `<=maxbound` | | | bounded index | +| **sync** | `flopy4 mf6 sync`, which regenerates the classes from an MF6 version's DFNs. **Sync contract**: the `_contract.py` it writes, recording the version, DFN commit and generated files. | | `flopy.mf6.utils.generate_classes` | | + +### One word, one sense + +These words have carried several meanings. Use the bare word only for its main +sense, and the other wording for the rest. + +| Word | Bare, it means | Otherwise say | +|---|---|---| +| layout | MF6 file layout | "NetCDF profile"; "cell indexing"; "row format"; "memory arrangement" | +| package | an MF6 package | `mf6xr` (working name) for the planned Python package | +| spec | never used bare | "DFN spec"; "representation spec" | +| view | computed from held data | "SQL view"; "row proxy" for a row object over `ItemTable` | +| context | never used bare | "binding arguments"; `WorkspaceComponent`; "`RowContext`"; "memory context" | +| rule | never used bare | "named rule" in a DFN; "layout preservation"; "rewrite rule" in the codemod; "constraint" for what a valid simulation must satisfy | +| tag | never used bare | "tagged" for the DFN key; "type tag"; "discriminator"; "option attribute" for `binding` and `schema` | +| contract | never used bare | "sync contract"; "data contract"; "import-linter contract"; "input constraints" | +| schema | never used bare | "DFN schema version"; "`schema` option"; "representation" for the export; "database schema" | +| level | layout, logical or runtime | "site" for where a discriminator sits | +| dynamic | the DFN's `dynamic` key, for a keyword whose value comes from its `fk` | stage words for derivations (see above) | +| kind | qualified only | "key domain" for keys; "constraint kind"; "lens kind" | +| field | an attrs or DFN field | "gridded quantity" | +| layered | use the keyword, `LAYERED` | "cell indexing" for layered or flat | +| row, entry | row for tables; entry for an MF6 boundary row, as mf6io says | `Item` and `Record` keep their class names; a "control record" is MF6's array header | +| fill forward | what happens for a period with no block | for what a new block does, say replace or update | + +### Equivalence + +These names say exactly what a gate or test checks, so use the one you mean. + +| Name | Holds when | +|---|---| +| **object-equal** | `==` on components | +| **text-equal** | files are the same bytes, perhaps up to whitespace | +| **layout-equal** | same blocks, options, control records and external-file choices | +| **logically equal** | same after normalizing by the option labels | +| **result-equal** | MF6's heads and budgets agree within a stated tolerance | + +"Round trip" alone is ambiguous, since it has meant both load, write and load, and +load, write, run and compare. Say which. + +### Type vocabulary + +The design docs use these for what varies in MF6 input, and what decides it. + +| Construct | Example | Theory | +|---|---|---| +| a reference whose target depends on another value | MVR's `(pname1, id1)` | dependent pair | +| a component that depends on its binding | SPC's target package | dependent function, ML functor | +| array shape that depends on the grid | griddata | shape polymorphism | +| wrapping a value type | periodic data, a value or a time series name | parametric type constructor | +| accepting many forms of one thing | a row as a tuple or a dict | coercion | +| a numeric range | `k > 0` | refinement type | +| missing values | inapplicable vs. not given | Codd's marks | + +## Acronyms + +| Acronym | Meaning | +|---|---| +| MF6 | MODFLOW 6 | +| DFN | definition file | +| IDM | MF6's Input Data Model, which loads input into memory | +| MVP, MMP, GA | minimum viable product, minimum marketable product, general availability: flopy4's [milestones](map.md) | +| SRS, SDD | [software requirements specification](srs.md), [software design description](sdd.md) | +| DIS, DISV, DISU | structured, vertex and unstructured discretization | +| TDIS | time discretization | +| IMS, EMS | iterative and explicit model solution | +| GRB | binary grid file | +| HDS | binary head file | +| CBC | cell-by-cell budget file | +| ATS | adaptive time step utility | +| TS, TAS | time series, time-array series | +| TVK, TVS | time-varying hydraulic conductivity, time-varying storage | +| OBS | observations utility | +| NCF | model NetCDF utility | +| HPC | high performance computing utility | +| MVR | water mover | +| GNC | ghost-node correction | +| SSM | source and sink mixing | +| SPC, SPCA | stress package component, list-based and array-based | +| BMI, XMI | Basic Model Interface, and MF6's extended form of it | +| CF, UGRID | Climate and Forecast conventions, the unstructured grid conventions, both for NetCDF | + +## Project shorthand + +| Term | Meaning | +|---|---| +| `flopy4` | The Python package in this repo, code name pyphoenix | +| FloPy 4 | The product the [SRS](srs.md) and [SDD](sdd.md) describe | +| flopy 4.x | The flopy release series `flopy4` is to merge into | +| flopy3 | flopy 3.x, including its `flopy.mf6` | +| legacy engine | flopy3's MF6 data layer, if it's kept as `flopy.mf6.legacy`. It would be frozen and wouldn't sync. Whether to keep it is open. | +| compat layer | What keeps flopy3 code working on the new engine in 4.x: generated `ModflowXxx` names, and handwritten pieces for the simulation and model | +| `Legacy` | A flopy3 constructor keyword flopy4 still takes, with a warning | +| codemod | `flopy4 migrate`, which rewrites flopy3 code | +| registry corpus | The models in modflow-devtools' registry, used as test input | +| tier A-E | The test tiers for 4.x. A: shared code. B: old engine internals. C: compat scoreboard. D: new engine. E: equivalence between the engines. | +| scoreboard | Tier C: how much flopy3 code and how many MF6 autotests and examples pass on the compat layer. It reports and doesn't gate. | +| Track A-G | The work in the [MMP plan](mmp-plan.md): engine correctness, API decisions, data and xarray, docs, migration, 3.x prep, packaging | +| dev3 | The DFN schema version `2.0.0.dev3` from modflow-devtools | +| materialize | In benchmarks, reading everything a load deferred. In the design docs, computing a derivation and storing the result. Say which. | +| FUNC-*n*, NFR-*n*, SYS-*n* | Requirements in the [SRS](srs.md) | diff --git a/docs/dev/map.md b/docs/dev/map.md index 64fb567a..fcf35ad7 100644 --- a/docs/dev/map.md +++ b/docs/dev/map.md @@ -33,10 +33,16 @@ All core and most peripheral functionality, and may involve: - Finalizing maintenance/release plans - Finalizing MF6 version support plan -Release to wider test audience at USGS and Deltares via `pip install` from github URL. Begin beta versioning. +Release to wider test audience at USGS and Deltares via `pip install` from github URL. This is the last beta; beta versioning begins before it. + +What's left to reach the MMP, and in what order, is in [mmp-plan.md](mmp-plan.md). All plans and design notes are listed in [plans/README.md](plans/README.md). + +### Release candidates + +Integrated into the existing repository on a `4.x` branch, cut once MMP feedback has settled the API. Released as `flopy==4.0.0rcN` via PyPI and conda-forge. See [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md). ### GA -Feature-parity with 3.x. Integrated into existing repository. Generally available via standard channels (PyPI, Conda). +Feature-parity with 3.x. The last release candidate. Generally available via standard channels (PyPI, Conda). 3.x enters maintenance-only mode for a limited time after which support will be dropped and all effort moved to 4.x. diff --git a/docs/dev/mmp-plan.md b/docs/dev/mmp-plan.md new file mode 100644 index 00000000..68125d76 --- /dev/null +++ b/docs/dev/mmp-plan.md @@ -0,0 +1,193 @@ +# Road to MMP + +What's left between today and the MMP, in what order, and what comes +after it. Written 2026-11-03 against `develop` (`8d25e4b7`). This doc +sequences the work and links to the plans that hold the detail, so it +stays short and the plans stay the source of truth. Update the status +column as things land. Every plan and design note, scheduled or not, +is listed in [plans/README.md](plans/README.md). + +## Release shape + +Decided 2026-11-03. + +| stage | released from | installed by | gate | +|---|---|---|---| +| alphas | this repo | `pip install` from GitHub | as needed | +| betas, MMP the last of them | this repo | `pip install` from GitHub | the first: open, below; MMP: "MMP gate" below | +| release candidates `flopy==4.0.0rcN` | flopy's `4.x` branch | PyPI, conda-forge | the integration plan's "Before the first release candidate" | +| GA | flopy, the last release candidate | PyPI, conda-forge | the integration plan's "Before GA" | + +Why this order, and not integrating into flopy before MMP: + +- **GitHub installs don't need the packaging work.** PyPI rejects git + dependencies, `pip install git+...` doesn't. The devtools release, the + xugrid range and Python 3.14 gate the release candidates, not MMP. +- **Testers can run both engines side by side.** `flopy4` and `flopy` + are different packages, so testers can compare them on their own + models in one environment. +- **Feedback lands at this repo's pace,** without flopy's review and CI + on every beta fix. +- **The `4.x` branch stays short.** It's cut after MMP feedback has + settled the API, so it carries a stable codebase, not beta churn. + +The costs: integration issues (the shared-code protocol, tier A tests +against both engines) surface after MMP, not before; testers change +`import flopy4` to `import flopy` at the release candidates (a scripted +rename, documented in [`migrating.md`](../migrating.md)). The integration plan rates the +code move as low risk, and the engine risk is exercised by the betas +either way. + +**Open: version numbers.** `flopy4` is at `0.0.1.dev0`. Versioning the +pre-releases `4.0.0aN`/`4.0.0bN` lines them up with `flopy==4.0.0` and +avoids a jump from 0.x. + +**Open: what starts the betas.** The MMP is the last beta, so the first +comes before it, but no gate is set for the first. + +## What MMP means + +From [`map.md`](map.md): all core and most peripheral functionality, released to a +wider test audience at USGS and Deltares, beta versioning. Its list +includes rough feature parity with 3.x, user-testing feedback, adapted +3.x docs and examples, comprehensive testing, and finalized maintenance, +release and MF6 version support plans. + +The SRS requirements due at MMP, plus the one MVP requirement still open: + +| req | MoSCoW | status (2026-11-03) | remaining | +|---|---|---|---| +| FUNC-7 round-trip with identical results (MVP) | C | Not done. [`test.md`](test.md) lists flopy3/4 comparison and round-trip tests as unchecked. | Track A | +| FUNC-8 larger-than-memory | M | Partial. Only open/close text writes are bounded by chunk size; period arrays and NetCDF compute everything; loads are eager (#516); no xarray IO backend (#368); lists have no lazy path. | Track C, [`out-of-core.md`](plans/xarray/out-of-core.md) | +| FUNC-18 lazy input everywhere | M | Griddata and dict-form period arrays only. | Track C | +| FUNC-19 lazy load, unchanged write-through | M | Not started. | Track C | +| FUNC-20 faster text writer | S | Not started. | Track C | +| NFR-8 memory bounded by chunk size | M | Not met, except as under FUNC-8. | Track C | +| NFR-9 same results whatever the array type | M | Not tested. | Track C | +| NFR-10 writes keep lazy data's sources safe | M | Not started; only arises with lazy load. | Track C | +| FUNC-10 wide range of MF6 versions | S | Mostly. `flopy4 mf6 sync`, [`mf6-versions.md`](../mf6-versions.md); tested against the latest release and MF6 `develop` only. | Decide whether "wide range" needs a test matrix of older releases. | +| FUNC-11 spatial and temporal units | C | Not audited. Grid adapters carry length units. | Audit, or defer past MMP (it's a C). | +| FUNC-14 reproduce the spec | S | Partial. `_contract.py` records the DFN commit; #299 open. | Track B | +| NFR-1 public vs internal API in docs | M | Not done. No API reference in the docs. | Track D | +| NFR-6 documented, type-hinted public API | S | Not done, same reason. | Track D | +| NFR-7 consolidated entry points | S | Partial. #342 (unified output loading) and #232 open. | Track B | + +The GitHub MMP milestone has 24 open issues. Many are questions +(#229, #240, #258, #259, #436, #523) that are either decisions for +Track B or can move past MMP. Triage it early, so the milestone matches +this doc. + +## MMP gate + +- [ ] Tier E (equivalence harness) passes on the registry corpus, with + any accepted gaps listed. FUNC-7 falls out of this. +- [ ] Round-trip load/write tests over the corpus. +- [ ] The bare-list `stress_period_data` bug fixed. +- [ ] The user-facing API decisions in Track B made and implemented. + Testers write code against them, so changing them after MMP breaks + them. +- [ ] Datetime period keys (#518) in, or explicitly deferred. +- [ ] The FUNC-8 target model created, written and reloaded under a memory + limit, and the rest of [`out-of-core.md`](plans/xarray/out-of-core.md)'s + phases done (FUNC-18 to FUNC-20, NFR-8 to NFR-10). +- [ ] API reference in the docs, public vs internal marked (NFR-1, NFR-6). +- [ ] Concepts doc (#217), and [`migrating.md`](../migrating.md) covering the compat layer. +- [ ] Migration scoreboard running here against MF6's autotests and the + examples, so the release notes can say how much flopy3 code runs with + only imports and class names changed, and how much after + `flopy4 migrate`. +- [ ] Install from GitHub works on Windows, Linux and macOS (SYS-2), with + the command documented. +- [ ] MMP milestone triaged: each open issue fixed, decided, or moved. + +## Tracks + +All run at once. The table says what each needs to deliver by MMP; work +after MMP is under "After MMP". + +| track | items | by MMP | detail | +|---|---|---|---| +| **A. Engine correctness** | bare-list bug; equivalence harness (tier E) over the registry; round-trip tests; fix what they find | everything | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) "Tests", [`test.md`](test.md). **Longest and least predictable. Start first.** | +| **B. API decisions** | `output.head` (I3); `get_package`/`modelgrid` on Model; `name` vs `sim_name`; accessor naming and `flatten()`/`node`; field naming, name or longname (#258); parent/child init (#259); unified output loading (#342); the data-shape questions, decided together by one rule: STO types (#229), UZF placeholders (#523), SFR `connectiondata` (#345), packages without a grid (#226) | decided and implemented | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) "Open questions"; [`overview.md`](plans/xarray/overview.md) "Open questions"; [`layout-logical-decisions.md`](plans/dfn-spec/layout-logical-decisions.md) for #226, #229, #259, #345, #523 | +| **C. Data and xarray** | `to-xarray-constants` fix; datetime period keys (#518, #226); lossless `to_xarray`/`from_xarray` and `dense()` (export phase 1), including an Options section in the representation spec (logical options as data, layout options as encoding, output options off the data) and opt-in views for dynamic derivations (`AUXMULTNAME`, TAS); `simplify-converter` rebased and merged; out-of-core phases 0 to 5: streaming writes, lazy `dense()`, lazy load (#516, #368), lazy lists, a faster text writer, the FUNC-8 target | the fix, #518 and export phase 1; `simplify-converter`; out-of-core phases 0 to 5 | [`export.md`](plans/xarray/export.md), [`datetime-periods.md`](plans/xarray/datetime-periods.md), [`representation.md`](plans/xarray/representation.md), [`layout-logical-decisions.md`](plans/dfn-spec/layout-logical-decisions.md) "Track C", [`converter-rebase.md`](plans/io/converter-rebase.md), [`out-of-core.md`](plans/xarray/out-of-core.md) | +| **D. Docs** | API reference; concepts (#217); 3.x examples adapted; [`migrating.md`](../migrating.md) for compat; SDD update once export phase 1 lands | all | [`map.md`](map.md), [`srs.md`](srs.md), [`sdd.md`](sdd.md) | +| **E. Migration** | classify every flopy3 difference (accept, deprecate, break); runtime aliases in flopy4 itself (positional parent, `register_ims_package`, `write_simulation`, `get_package`, `modelgrid`, `*_filerecord`, ...); the shared `Legacy` table; the `flopy4 migrate` codemod; the corpus scoreboard over MF6 autotests and examples. Generated `ModflowXxx` class names stay in flopy 4.x's compat layer | classification, aliases and scoreboard; the codemod if time allows | [`aliases-and-codemod.md`](plans/integration/aliases-and-codemod.md); [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) "The compat layer" | +| **F. 3.x prep** (flopy `develop`) | 3.x items 1 to 5: utilities out of `flopy/mf6`, the shared-code protocol, upstream patches, deprecations, forward-compatible keywords | nothing required; items 1 and 2 by the `4.x` cut | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) "3.x work" | +| **G. Packaging** | devtools release; drop the git dependency; xugrid range; Python 3.14 and the 3.11 question; numpy floor; lazy xugrid import | nothing required (GitHub installs); all by the first release candidate | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) Phase 0 item 3; [`overview.md`](plans/xarray/overview.md) (lazy xugrid) | + +Tracks F and G aren't needed for MMP, but start them now: they're small, +each piece is releasable on its own, and they decide how quickly the +release candidates follow MMP. + +### Dependencies between tracks + +- Track A's harness finds engine bugs, and fixing them may touch Track C + (period data, arrays). Keep Track C's changes small until the harness + is running. +- #226 (packages without a grid), #259 (parent/child init) and #287's + binding-graph navigation (`Spc.target`, `Model.flow_model`, ...) need + the same attach operation. Design them together, or the relation ends + up held in two places. +- Track A's two comparisons check different things: round-trip + (FUNC-7) preserves layout as given, while the flopy3/flopy4 file + comparison is logical, normalized by the option labels + ([`layout-logical-decisions.md`](plans/dfn-spec/layout-logical-decisions.md) "Track A"). +- Track E's aliases and codemod rules wait for Track B's naming + decisions (and default file naming), or they get regenerated. +- Track D's API reference waits for Track B, and the SDD update waits + for export phase 1 (Track C). +- Out-of-core's writer work can start now, if it stays byte-identical for + lazy arrays while Track A's harness comes up. Lazy load and lazy lists + change ingress, so they wait for `simplify-converter`. Its random-chunk + test is tier E with a wrapper, so it needs the harness too. +- Track B reserves `load(chunks=...)` before testers write code against + `load`. +- Export phase 2's `dense()` is built lazy from the start, so it goes with + out-of-core phase 2. + +### The longest chain + +Equivalence harness → engine fixes → tier E passes → MMP. The other +tracks should finish inside it. If one starts to hold up the gate, +that's the time to add people to it. + +FUNC-8 at MMP makes a second long chain: `simplify-converter` → lazy load → +lazy lists → the target model. Its first unknown is the construction peak +([`out-of-core.md`](plans/xarray/out-of-core.md) phase 0), which could put the +target out of reach until it's fixed. + +## After MMP + +From [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md), Phases 2 and 3: + +1. Work through MMP feedback. Decide I2 (compat data access), informed by + the scoreboard numbers and the feedback. +2. Cut `4.x` once the API has settled and the integration plan's "Before + cutting `4.x`" list is done. +3. Move the old engine to the legacy path, bring `flopy4` in as + `flopy.mf6`, run tiers A to E there. +4. Release candidates `flopy==4.0.0rcN` from PyPI and conda-forge. +5. GA, then 3.x maintenance for a set time. + +## Decisions to make + +From the plans this doc links to, ordered by when they're needed. I1 to +I7 are the integration plan's open questions: + +| decision | needed by | where | +|---|---|---| +| Pre-release version numbers (`4.0.0bN`?) | first alpha | here | +| What starts the betas | first beta | here | +| `output.head`: which side renames (I3) | MMP | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) | +| Remaining Track B API questions | MMP | Track B | +| attrs or pydantic (#282, #367) | MMP, possibly put off to GA | [`pydantic-object-model-plan.md`](pydantic-object-model-plan.md) | +| The layout/logical rule, and with it #226, #229, #259, #345, #523 | MMP; early, since Track B's other data questions follow from it | [`layout-logical-decisions.md`](plans/dfn-spec/layout-logical-decisions.md) | +| Post #287 before the layout/logical decisions, so Deltares can weigh in on the rule? | before the layout/logical rule | [`layout-logical-decisions.md`](plans/dfn-spec/layout-logical-decisions.md) | +| Default file names (`m.chd` vs `m.chd0.chd`) | the Part 1 aliases; decide early, it changes flopy4's own tests | [`aliases-and-codemod.md`](plans/integration/aliases-and-codemod.md) | +| FUNC-10 test matrix; FUNC-11 in or out of MMP | MMP | SRS | +| Compat data access (I2) | `4.x` cut | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) | +| Legacy engine and its namespace (I1) | `4.x` cut | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) | +| Where the top-level modules go (I6) | `4.x` cut | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md), [`overview.md`](plans/xarray/overview.md) | +| numpy floor and Python range (I5) | first release candidate | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) | +| 3.x keywords in 3.x (I4) | 3.x item 5 | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) | +| 3.x support window (I7) | GA | [`replacing-flopy-mf6.md`](plans/integration/replacing-flopy-mf6.md) | diff --git a/docs/dev/plans/README.md b/docs/dev/plans/README.md new file mode 100644 index 00000000..a6c16c55 --- /dev/null +++ b/docs/dev/plans/README.md @@ -0,0 +1,90 @@ +# Plans and design notes + +Plans with work left, research reports and reference notes, by topic. They +reflect current understanding and change as it does. A plan moves to the +[archive](archive/README.md) once nothing in it is left to do. +Everything under `docs/dev/plans/` is read in the source tree and isn't +part of the built docs; the published developer docs (SRS, SDD, roadmap, +testing) sit one level up. For what's scheduled, and in what order, see +[mmp-plan.md](../mmp-plan.md); for milestones, see [map.md](../map.md). + +## Integration with flopy + +| Doc | What | +|---|---| +| [integration/replacing-flopy-mf6.md](integration/replacing-flopy-mf6.md) | replacing flopy's mf6 engine with flopy4: 3.x prep, the 4.x layout, the compat layer, test tiers, release order | +| [integration/aliases-and-codemod.md](integration/aliases-and-codemod.md) | runtime aliases for flopy3 idioms in flopy4, then a `flopy4 migrate` codemod | +| [integration/period-list-values.md](integration/period-list-values.md) | not started: flopy3's `get_data`, `set_data`, `.array` and `.data` on `PeriodRows`, if unchanged 3.x scripts become a goal | +| [integration/modflowapi.md](integration/modflowapi.md) | speculative, not for 4.x: folding modflowapi into flopy4 as an XMI plugin, with modflowapi's API | + +## xarray + +[xarray/overview.md](xarray/overview.md) is the umbrella doc for this area. + +| Doc | What | +|---|---| +| [xarray/overview.md](xarray/overview.md) | layering an MF6 xarray package inside flopy4 so it can be extracted later | +| [xarray/representation.md](xarray/representation.md) | the spec: grid dims and coordinates, the tree, period data, how it relates to netCDF, CF, UGRID and MF6 NetCDF | +| [xarray/export.md](xarray/export.md) | lossless `to_xarray`/`from_xarray`, `dense()`, `flatten()`, cell layouts | +| [xarray/out-of-core.md](xarray/out-of-core.md) | larger-than-memory models (FUNC-8): streaming writes, lazy load, lazy lists, a faster text writer, the tests that prove them | +| [xarray/datetime-periods.md](xarray/datetime-periods.md) | datetime period keys (#518), then `create_tdis` (#226) | +| [xarray/rotated-grid-index.md](xarray/rotated-grid-index.md) | DIS coordinates, rotation, `CellIndex`, z selection, the accessor, and what's next | +| [xarray/rch-evt-layer.md](xarray/rch-evt-layer.md) | z coordinates for RCHA/EVTA arrays | +| [xarray/tas-interpolation.md](xarray/tas-interpolation.md) | evaluating time-array series per time step | +| [xarray/combined-index.md](xarray/combined-index.md) | a combined grid index on `CellIndex`, and why not yet | + +## Components and code generation + +| Doc | What | +|---|---| +| [components/mixin-fields.md](components/mixin-fields.md) | what mixins rely on in the generated classes, and how sync checks it | +| [components/user-dir-classes.md](components/user-dir-classes.md) | generating component classes into a per-user directory instead of the installed package | +| [components/constructor-stubs.md](components/constructor-stubs.md) | generated `.pyi` stubs so type checkers accept what constructors accept, then `py.typed` | +| [../pydantic-object-model-plan.md](../pydantic-object-model-plan.md) | attrs vs. pydantic for the object model; a complete migration on draft #367, undecided | + +## Lists + +| Doc | What | +|---|---| +| [lists/list-fields.md](lists/list-fields.md) | list fields: representation, `ItemTable` storage, row layout | +| [lists/record-unify.md](lists/record-unify.md) | merging `Item` into `Record` | +| [lists/binary-list-shapes.md](lists/binary-list-shapes.md) | which blocks MF6 can read binary list input into | + +## Reading and writing + +| Doc | What | +|---|---| +| [io/converter-simplification.md](io/converter-simplification.md) | one schema-driven structure/unstructure path, with the classes as the only runtime schema, typed loader moved off to `typed-loader` (unmerged `simplify-converter` branch) | +| [io/converter-rebase.md](io/converter-rebase.md) | rebasing `simplify-converter` onto develop, with the typed loader kept current on a stacked `typed-loader` branch | +| [io/loader-comparison.md](io/loader-comparison.md) | the basic and typed loaders measured against each other | + +## Maintenance + +| Doc | What | +|---|---| +| [maintenance/backlog.md](maintenance/backlog.md) | small open items that don't belong to a plan | +| [maintenance/simplification-survey.md](maintenance/simplification-survey.md) | where hand-written `flopy4/mf6` could be simpler | +| [maintenance/dev-docs.md](maintenance/dev-docs.md) | turning the reference notes here into published developer docs | + +## The DFN spec + +Research on whether the DFN spec should describe MF6 input at two levels, +logical and layout, and what that means for flopy4 (discussion #287). +Start with [dfn-spec/overview.md](dfn-spec/overview.md), which defines the +terms the others use. + +| Doc | What | +|---|---| +| [dfn-spec/overview.md](dfn-spec/overview.md) | concepts, hypotheses, evidence, next steps | +| [dfn-spec/factoring.md](dfn-spec/factoring.md) | the design rule: what goes in the DFN, the classes, the codec and named checks; three structural changes | +| [dfn-spec/option-classification.md](dfn-spec/option-classification.md) | every DFN option labelled layout, logical or output | +| [dfn-spec/constraint-audit.md](dfn-spec/constraint-audit.md) | kinds of input constraint, where each is enforced, and what could express it | +| [dfn-spec/dfn-gaps.md](dfn-spec/dfn-gaps.md) | rules MF6 enforces that the DFNs don't state, and memory catalog gaps; for MF6 and devtools developers | +| [dfn-spec/layout-logical-decisions.md](dfn-spec/layout-logical-decisions.md) | the layout/logical rule applied to MMP issues #226, #229, #259, #345, #523 | +| [dfn-spec/schema-polymorphism.md](dfn-spec/schema-polymorphism.md) | schema polymorphism (#287) vs. component key roles (devtools #405) | +| [dfn-spec/ddl-export.md](dfn-spec/ddl-export.md) | working plan, not started: run the DDL slice's rules on real models loaded with flopy4 | + +## Archive + +Finished, superseded and abandoned plans are in [archive/](archive/README.md), +as they were last written. diff --git a/docs/dev/plans/archive/2024-first-meeting.md b/docs/dev/plans/archive/2024-first-meeting.md new file mode 100644 index 00000000..7e5963a9 --- /dev/null +++ b/docs/dev/plans/archive/2024-first-meeting.md @@ -0,0 +1,13 @@ +# plan for first meeting + +- short slideset with conceptual overview +- topics + - user stories: how users will interact with it + - getting/setting package data + - loading/writing input files + - developer stories: how developers will interact with it + - generating source code from DFNs + - write python directly + - templating engine + - fully dynamic + - debugging diff --git a/docs/dev/plans/archive/2024-idm.md b/docs/dev/plans/archive/2024-idm.md new file mode 100644 index 00000000..1a669d59 --- /dev/null +++ b/docs/dev/plans/archive/2024-idm.md @@ -0,0 +1,174 @@ +# Input Data Model + +MODFLOW 6 accepts input via **input files**. Each MODFLOW 6 input file corresponds to a **simulation**, a **model**, or a **package**. A package can be associated with a simulation, a model, or another package (in the latter case the package is called a **subpackage**). Each MODFLOW 6 input file contains one or more **blocks**. A block contains zero or more **parameters**. A parameter can be a **scalar**, **array**, or **list**. + +This 10k-foot view in mind, the rest of this document describes the MODFLOW 6 input data model and the corresponding FloPy interface layer from the bottom up. + +## Parameters + +MODFLOW 6 accepts three kinds of parameters: + +- scalars +- arrays +- lists (tables) + +The `MFParameter` class is the parent of all parameter classes. + +**Properties** + +- `value` + +**Instance Methods** + +- `write()` + +**Class Methods** + +- `load()` + +### Scalars + +MODFLOW 6 accepts scalar inputs of several kinds: + +- keywords +- integers +- reals +- paths + +The `MFScalar` abstract class is the parent of all scalar parameter classes. + +#### Keywords + +The `MFKeyword` class represents boolean switches. + +#### Integers + +The `MFInteger` class represents integer parameters. + +#### Reals + +The `MFDouble` class represents real parameters. + +#### Paths + +The `MFFilename` class represents external file paths. + +#### Repeating scalars + +todo + +#### Records + +todo + +### Arrays + +MODFLOW 6 accepts 1D, 2D, and 3D input arrays. + +The `MFArray` class represents input arrays. +The array is stored as a flat NumPy `ndarray` +until written or viewed/returned. The class +emulates `ndarray` and works with standard +NumPy functions. + +**Properties** + +- `control_line`: returns the control line +- `value`: returns a copy of the `ndarray` in the proper shape +- `raw`: returns a copy of the array in the proper shape, without the scale factor applied +- `how`: identifies the array's input format +- `factor`: return the scale factor value + +**Instance Methods** + +- `__getitem__()`: slice the array +- `__setitem__()`: set/update the array +- `__str__()`: brief summary of array: control line +- `__repr__()`: array as will be written to input file + - control line for external/constant + - control line + data lines for internal +- `write()`: write the array to file + +**Class Methods** + +- `load()`: load the array from file + +### Lists + +MODFLOW 6 accepts lists of space-delimited input records. + +The `MFList` class represents list (tabular) data, e.g. stress +period data. The class handles reading, dynamic dtype handling +and dtype expansion for boundnames and auxvars. + +**Properties** + +- `value`: return a copy of the underlying `DataFrame` + +**Instance Methods** + +- `__getitem__()`: get a column, dict syntax +- `__setitem__()`: set a column, dict syntax +- `__getattr__()`: get a column, attr syntax +- `__setattr__()`: set a column, attr syntax +- `__str__()`: brief summary of list +- `__repr__()`: list as will be written to input file +- `write()`: write the list to file + +**Class Methods** + +- `load()`: load the list from file + +## Blocks + +A block is a collection of parameters in a MODFLOW 6 input file. A block is delimited by lines containing `BEGIN ` and `END `. + +A block contains zero or more parameters. If a block contains a list, the list must be the block's only parameter. Parameters are identified by keywords or indices — scalars and arrays are identified by keyword, lists by the index of the block. A block index is an integer appended to the block's name. + +The `MFBlock` class represents an input block. + +**Properties** + +- `name`: block name +- `index`: only set if period block +- `type`: options, dimensions, package data + +**Instance Methods** + +- `__getitem__()`: get a param, dict syntax +- `__setitem__()`: set a param, dict syntax +- `__getattr__()`: get a param, attr syntax +- `__setattr__()`: set a param, attr syntax +- `__str__()`: brief summary of block +- `__repr__()`: block as will be written to input file +- `write()`: write the list to file + +**Class Methods** + +- `load()`: load the list from file + +## Packages + +A package configures either a model or a simulation. A package input file contains one or more related blocks. + +The `MFPackage` class represents a MODFLOW 6 package. The class reads/writes package input files, delegating to constituent blocks. + +**Properties** + +- `options`: options block (always) +- `periods`: period blocks (only if relevant to package) +- `package`: + +**Methods** + +## Models + +A model is a physical process included in a simulation. A model is associated with a namefile and a number of package input files. + +The `MFModel` class represents a MODFLOW 6 model. The class reads/writes model name files and delegates reading/writing of package input files to constituent packages. + +## Simulations + +A simulation is MODFLOW 6's main entry point. A simulation contains one or more models. A simulation is associated a namefile and a number of model and package input files. + +The `MFSimulation` class represents a MODFLOW 6 simulation. The class reads/writes simulation name files and delegates reading/writing of model and package input files to constituent models and packages. diff --git a/docs/dev/plans/archive/2024-notes.md b/docs/dev/plans/archive/2024-notes.md new file mode 100644 index 00000000..2f0e08dc --- /dev/null +++ b/docs/dev/plans/archive/2024-notes.md @@ -0,0 +1,172 @@ +# General goals + +- easy to read and debug from developer standpoint +- maintain the existing flopy.mf6 functionality +- fast read/write routines +- minimize codebase size + - avoid manually generating python source as we did previously +- interact with flopy with standard python data types + - minimize knowledge needed of MF6 input files + - distinguish conceptual model from configuration language + +# Design principles + +- references should only point downwards (sim -> model -> pkg ...) + - not upwards, as in current flopy (e.g. packages knowing which model is their parent) +- maintain inversion of control + - interfaces don't need to know .. + - data representation: could be numpy, pandas, xarray, etc + - configuration language: could be standard input files, netcdf, toml, etc + +# Implementation ideas + +- use mixins for cross-cutting concerns (e.g. plotting, exporting) +- can mf6 components emulate iterable/mapping? + - in the spirit of pre-existing `SimulationDict` + - simulation as iterable of models/exchanges/packages + - also expose `.models`, `.exchanges`, `.packages` properties separately + - model as iterable of packages + - package as iterable of blocks + - block as iterable of scalars/arrays/lists + - resources: + - https://docs.python.org/3/reference/datamodel.html?emulating-container-types=#emulating-container-types + - https://stackoverflow.com/questions/3387691/how-to-perfectly-override-a-dict +- can input types (mfarray, mflist) emulate numpy arrays and pandas dataframes? + - arrays: numpy mixin implementing ufuncs etc + - https://numpy.org/doc/stable/user/basics.dispatch.html#basics-dispatch + - https://numpy.org/doc/stable/user/basics.interoperability.html#basics-interoperability + - https://stackoverflow.com/questions/55386602/how-to-overide-numpy-ufunc-with-array-ufunc + - lists: inherit from DataFrame like GeoPandas does? + - https://github.com/geopandas/geopandas/blob/main/geopandas/geodataframe.py#L78 +- string representations + - https://stackoverflow.com/questions/1436703/what-is-the-difference-between-str-and-repr + - `__str__()` of model/package/block/input **classes** as human-readable DFN specification + - `__repr__()` of model/package/block/input **classes** as exact specification as in DFN file + - `__str__()` of model/package/block/input **instances** as abbreviated summary of what/how input will be written + - mfarray: control line + - mflist: block name? what to do if inline? headers? + - `__repr__()` of model/package/block/input **instances** as exactly what would be written to input file + - `write()` to text buffer and return as `str` +- metaclass magic? + - motivation + - convenient hooks to mf6 input/output file data + - hide details/complexity of the configuration backend + - user doesn't need to know details of MF6 input file format + - just sim/model/package/etc conceptual model and standard python data structures + - create classes dynamically without manually generating Python source + - case study: Django dynamic API generation + - define Model in terms of IntegerField, CharField, etc + - access model instance properties as builtin types (int, str) + - https://code.djangoproject.com/wiki/DevModelCreation + - https://code.djangoproject.com/wiki/DynamicModels + - https://github.com/django/django/blob/86e13843c2ab510fba1de84588efe7fd03555531/django/db/models/base.py#L95 + - our case: dynamically define components based on DFNs + - define MFBlock in terms of MFScalar (MFKeyword, MFInteger, etc), MFArray, MFList + - define MFPackage in terms of MFBlock + - define MFModel in terms of MFPackage + - access package/model instance data as bool, int, ndarray, recarray/dataframe + - unnecessary for user to interact with MFScalar, MFArray, MFList directly? + - misc notes: + - set `__doc__` dynamically in `__new__()` + - override `__setattr__()` so `__init__()` on model/package/block classes + - waterfall down to parameter level + - proper initialization of parameter instances instead of native types + - discussions: + - https://stackoverflow.com/a/6581949/6514033 + - https://stackoverflow.com/questions/2005878/what-are-python-metaclasses-useful-for + - https://stackoverflow.com/questions/15247075/how-can-i-dynamically-create-derived-classes-from-a-base-class + - https://stackoverflow.com/questions/17929543/how-can-i-dynamically-create-class-methods-for-a-class-in-python +- defer MF6 component generation until install time? + - no need to version DFNs and generated files or decide when to sync them from MF6 + - include latest MF6's DFNs with each corresponding release of flopy + - https://setuptools.pypa.io/en/latest/userguide/datafiles.html#subdirectory-for-data-files + - at first install time, flopy uses the DFNs packaged with it to generate components + - https://setuptools.pypa.io/en/latest/userguide/datafiles.html#accessing-data-files-at-runtime + - split out a new command from generate_classes just to retrieve latest DFNs: get_dfns? + - update components with get_dfns then generate_classes or just reinstall + - maybe worth bundling get_dfns + generate_classes into command: update_classes? + - questions: + - preserve intellisense in VSCode and similar in PyCharm? + - does this make debugging harder? + - autogenerated file could still be stepped into + - just not quite as accessible (in site-packages not local source dir) + - links: + - https://setuptools.pypa.io/en/latest/userguide/extension.html#setuptools.command.build.SubCommand.build_lib + - https://github.com/modflowpy/flopy/pull/1970#issuecomment-1743794710 + - https://github.com/modflowpy/flopy/pull/1970#issuecomment-1745730335 + - https://code.visualstudio.com/docs/editor/intellisense +- use https://docs.python.org/3/reference/datamodel.html#slots for memory savings? +- implement signal architecture: https://code.djangoproject.com/wiki/Signals + - motivation: package/model can signal another package/model when something happens + - e.g. dis can signal when it loads grid info, so other components can update themselves or use it +- use similar file access API as django? https://code.djangoproject.com/wiki/FileStorageRefactor +- if a block has just one parameter and the parameter and the block share the same name, shortcut attribute? + +questions + +- how does flopy deal with list (table) columns conditional on an options block setting? + - mf6.coordinates.modelcoordinates has an input shape resolver + - computes the shape of an mflist based on available package options + - mf6.data.mfdatastorage.resolve_typelist() determines column types for list based input + - core of the issue: tabular data shape varies depending on options + - also: how to deal with jagged list data? will mf6 ever drop these? + +misc + +- MF6 input file format as a DSL (mini-language) for configuring hydrologic simulations + - framework-specific modeling language(s) + - https://en.wikipedia.org/wiki/Modeling_language + - https://gsd.uwaterloo.ca/sites/default/files/models06.pdf + - modeler "completes" simulation framework by specifying: + - components (models and packages) via input files + - settings (blocks/options) within input files + - input file (1 per component instance) as metamodel: specifies a model (simulation) + - bespoke format + - DFN (1 per component class) as meta-metamodel: specifies a metamodel (input file) + - formal grammar: defines the language subset each component supports + - different bespoke format + - programs implement some combination of parser/generator/interpreter for input files and/or DFNs + - roles + - parser: read + - generator: write + - interpreter: run + - for input files, this means running the simulation + - for DFNs, this means generating source code + - data access layer as well as user-facing interface layer + - in mf6: `dfn2f90.py` + - in flopy: `createpackages.py`, `flopy.mf6.utils.generate_classes` + - current capabilities + - `dfn2f90.py` + - DFNs: read and run but not write + - input files: none + - flopy + - DFNs: read and run but not write + - input files: read and write and run (indirectly, by invoking MF6) + - MF6 + - DFNs: none + - input files: read and run but not write + - modflowapi + - DFNs: none + - input files: run (directly, by binding to MF6) but not read or write + - flopy has the broadest scope! + - needs to understand DFNs and input files, and use the former to speak the latter + - future MF6 may support other configuration languages (e.g. YAML, TOML, NetCDF) for input + - what do we want people to [be able to / have to] write? + - currently, developers write DFNs and modelers write either Python or input files by hand + - in future, developers write TOML, modelers write Python or TOML or NetCDF or etc? + - makes sense to use same configuration language for model input and input spec? + - removes the need for flopy and dfn2f90 to parse a custom language + - writing/reading input files and/or DFNs as a form of serialization/deserialization +- IDM as analogous to object-relational mapping (ORM)... object-input mapping (OIM)? + - map between input files and Python objects + - any impedance mismatch? + - https://en.wikipedia.org/wiki/Object%E2%80%93relational_impedance_mismatch + - https://en.wikipedia.org/wiki/Object%E2%80%93relational_impedance_mismatch#Mismatches + - maybe not? most of these seem irrelevant for input files + - active record pattern: https://en.wikipedia.org/wiki/Active_record_pattern + - alternative to full-fledged OIM: data access object pattern + - https://en.wikipedia.org/wiki/Data_access_object + - encapsulate input data without exposing input format details +- how to implement multiple backends (input file formats) + - define blocks in terms of native python types? + - dynamically bind diff --git a/docs/dev/plans/archive/2024-ofm.md b/docs/dev/plans/archive/2024-ofm.md new file mode 100644 index 00000000..6039376b --- /dev/null +++ b/docs/dev/plans/archive/2024-ofm.md @@ -0,0 +1,11 @@ +# Object-File Mapping + +FloPy aims to provide an object-oriented interface to MODFLOW 6. This involves translating between user-facing Python objects and MODFLOW 6 input files. This task is analogous to that performed by an object-relational mapping (ORM) library for a relational database. + +ORM software, [Django](https://www.djangoproject.com/) for instance, aims to hide the details of database access from the user: + +> When some (user) code creates a subclass of `models.Model`, a lot goes on behind the scenes in order to present a reasonably normal looking Python class to the user, whilst providing the necessary database support when it comes time to save or load instances of the model.[^1] + + + +[^1]: https://code.djangoproject.com/wiki/DevModelCreation. diff --git a/docs/dev/plans/archive/README.md b/docs/dev/plans/archive/README.md new file mode 100644 index 00000000..5011a54e --- /dev/null +++ b/docs/dev/plans/archive/README.md @@ -0,0 +1,98 @@ +# Archived plans + +Finished, superseded or abandoned plans, kept for their rationale. They're +committed as they were last written: statuses, branch names, line numbers +and paths are as of then, and many name files at the repo root, where the +notes used to live. Live plans are in the [index](../README.md). + +## Object model, codegen and fields + +| Doc | What | Outcome | +|---|---|---| +| [2024-idm.md](2024-idm.md), [2024-ofm.md](2024-ofm.md), [2024-notes.md](2024-notes.md), [2024-first-meeting.md](2024-first-meeting.md) | the 2024 input data model, object-file mapping, goals and first-meeting outline (`backup` branch) | historical | +| [mf6-object-model-plan.md](mf6-object-model-plan.md) | the MF6 object model | done | +| [backend-refactor-plan.md](backend-refactor-plan.md) | attrs components with on-demand xarray views instead of xattree proxying | done, as the xattree removal | +| [xattree-removal-plan.md](xattree-removal-plan.md), [xattree-child-naming-plan.md](xattree-child-naming-plan.md) | dropping xattree | done (#356); child naming superseded | +| [protocols-plan.md](protocols-plan.md) | dimension resolution protocols and mixin | done | +| [on-representations.md](on-representations.md) | DFN-vs-class representation mismatches the parser handled (`load` branch) | historical | +| [codegen-comparison.md](codegen-comparison.md), [codegen-recommendation.md](codegen-recommendation.md), [pydantic-prototype-summary.md](pydantic-prototype-summary.md), [model-rebuild-explained.md](model-rebuild-explained.md) | attrs vs. pydantic for generated classes (`plan-codegen` branch) | superseded by the plan below | +| [field-functions-plan.md](field-functions-plan.md) | one field function per DFN field type | done (#463) | +| [record-values-plan.md](record-values-plan.md) | plain values for every record field | done | +| [mypy-converter-plan.md](mypy-converter-plan.md) | letting mypy see field converters | abandoned (#433 closed); replaced by [constructor-stubs.md](../components/constructor-stubs.md) | +| [nested-keystring-union-plan.md](nested-keystring-union-plan.md) | typing OC's nested `ocsetting` unions | done | +| [array-equality-spec.md](array-equality-spec.md) | array-aware component equality | done (#394) | +| [codegen-base-class-removal-plan.md](codegen-base-class-removal-plan.md) | generating TDIS and NCF instead of hand-written bases | done (#385) | +| [dis-base-class-removal-plan.md](dis-base-class-removal-plan.md) | generating the discretization packages | done (#391) | +| [disv-plan.md](disv-plan.md) | what #391 delivered for DISV, and what was left | historical | +| [codegen-role-removal-plan.md](codegen-role-removal-plan.md) | removing the codegen "role" layer | proposal, not taken up | +| [model-codegen-plan.md](model-codegen-plan.md) | generating the model classes | done (#411) | +| [child-fields-plan.md](child-fields-plan.md), [links-from-dfns-plan.md](links-from-dfns-plan.md) | child fields and links from DFN component links | done (#395, #396) | +| [package-binding-plan.md](package-binding-plan.md) | packages that hold the components their file records name (e.g. `TS6 FILEIN`) | see doc | +| [reference-fields-plan.md](reference-fields-plan.md) | generic resolve and check for reference fields | done (#499) | +| [reference-fields-notes.md](reference-fields-notes.md) | fields that refer to components without owning them (solution models, exchange models) | done | +| [solution-binding.md](solution-binding.md) | how solutions and exchanges are tied to their models (#285) | done (#496, #497) | +| [binding-rows.md](binding-rows.md) | generating name-file binding rows from the DFNs | done (#526) | +| [package-naming.md](package-naming.md) | default package names and filenames, and the one-file-per-component check | done (#495) | +| [cellid-node-arrays.md](cellid-node-arrays.md) | devtools' cellid arrays and what changes in flopy4 | see doc | +| [dev3-migration-gaps.md](dev3-migration-gaps.md) | gaps hit migrating to DFN schema dev3 | resolved | +| [develop-shared-fixes.md](develop-shared-fixes.md) | fixes to carry from `pydantic-plan` to develop | obsolete | + +## Dimensions, lists and periods + +| Doc | What | Outcome | +|---|---|---| +| [dimension-issues.md](dimension-issues.md) | dimension follow-ups from #167, as issue drafts | done except one item | +| [list-dim-default-problem.md](list-dim-default-problem.md) | linked dimensions: explicit value vs. default | deferred | +| [boundname-plan.md](boundname-plan.md) | boundnames and token text end to end | done (#470, #478) | +| [period-lists.md](period-lists.md) | an xarray form for list packages' period data | done (#476) | +| [period-storage-plan.md](period-storage-plan.md) | per-period storage for period arrays | implemented on a branch; see doc | +| [repeating-block-header-generalization.md](repeating-block-header-generalization.md) | repeating blocks via `Block.header` | partly adopted | +| [flopy3-period-names-plan.md](flopy3-period-names-plan.md) | flopy3's period-data argument names | done (#503) | +| [tas-shorthand-typing-plan.md](tas-shorthand-typing-plan.md) | typing the TAS name shorthand | proposal | + +## Loading, writing and performance + +| Doc | What | Outcome | +|---|---|---| +| [plan.md](plan.md) | the Track A / Track B development plan (input loading) | done | +| [loader-roadmap.md](loader-roadmap.md) | the input loader roadmap | superseded | +| [class-driven-reader-plan.md](class-driven-reader-plan.md) | dropping the reader's runtime DFN dependency | done (#455) | +| [load-context-plan.md](load-context-plan.md) | load context | done (#418) | +| [model-relpath-plan.md](model-relpath-plan.md) | model directories | implemented on a branch; see #421 | +| [load-corpus-gaps.md](load-corpus-gaps.md) | `Simulation.load()` gaps on the test corpus | done | +| [oc-format-load-bug.md](oc-format-load-bug.md) | `test060_gms_ets` failing to load an OC print format with no format word | fixed by correcting the DFNs (MODFLOW-ORG/modflow6#3129), in #549; #548 closed | +| [trailing-text-load-fix.md](trailing-text-load-fix.md) | `Simulation.load` dropping a file's period blocks when text followed `END PERIOD n` (`test021_twri`'s recharge) | fixed in #545, with the warning; text between blocks now raises, and whether to accept it as MF6 does is open | +| [load-branch-todo.md](load-branch-todo.md) | TODO list from the `load` branch | historical | +| [typed-grammar-issues.md](typed-grammar-issues.md) | generated typed grammars falling back to the basic parser (`load` branch) | historical | +| [typed-grammar-remarks.md](typed-grammar-remarks.md), [typed-grammar-trailing-remarks.md](typed-grammar-trailing-remarks.md) | whether the typed grammar keeps trailing remarks, and its speed | answered: keep remarks | +| [typed-loader-wiring.md](typed-loader-wiring.md) | making the typed loader a selectable alternative | superseded by [converter-simplification.md](../io/converter-simplification.md) (unmerged `simplify-converter`) | +| [typed-loader-perf.md](typed-loader-perf.md) | load performance work | done (#474-#484); paused | +| [optimization-plan.md](optimization-plan.md) | input-load optimizations, basic loader first | superseded by [typed-loader-perf.md](typed-loader-perf.md) | +| [load-baseline-plan.md](load-baseline-plan.md) | baseline load benchmarks, flopy3 vs. flopy4 | folded into #482 | +| [benchmark-infra-plan.md](benchmark-infra-plan.md) | benchmark infrastructure | done (#482) | +| [io-benchmark-results.md](io-benchmark-results.md) | flopy4 vs. flopy3 IO benchmark, 2026-10-16 | stale | +| [write-options-plan.md](write-options-plan.md) | write options | implemented locally; see #454 | +| [dead-code-audit.md](dead-code-audit.md) | unused code in hand-written `flopy4/` | done (#509, #510, #528, #530, #531); one item in the [backlog](../maintenance/backlog.md) | +| [writer-dead-code.md](writer-dead-code.md) | the writer's dead xarray-Dataset list path | done (#505) | + +## xarray and ecosystem + +| Doc | What | Outcome | +|---|---|---| +| [spatial-index-design.md](spatial-index-design.md) | the first spatial index design (`xarray-index`, #278) | superseded by [rotated-grid-index.md](../xarray/rotated-grid-index.md) | +| [netcdf-spec-plan.md](netcdf-spec-plan.md) | what of a NetCDF spec belongs in devtools vs. flopy4 | superseded by [representation.md](../xarray/representation.md) | +| [ecosystem-sync-plan.md](ecosystem-sync-plan.md) | syncing DFNs and tooling across the MF6 ecosystem | done | +| [python-models.md](python-models.md) | a shared generic layer and executors for MF6 over XMI, pywatershed and timflow | abandoned: the generic layer is small (`spec.py`, `attrs_xarray.py`, `dimensions.py`, the last tied to the DFNs) and everything else is MF6-specific, and the codes run too differently for one executor interface. Interoperate through xarray datasets instead; MF6 over XMI is [modflowapi.md](../integration/modflowapi.md) | + +## DFN spec research + +| Doc | What | Outcome | +|---|---|---| +| [polymorphism-hypotheses.md](polymorphism-hypotheses.md) | working notes on three hypotheses from the #287 findings | superseded by [dfn-spec/overview.md](../dfn-spec/overview.md) | + +## Follow-up lists + +| Doc | What | Outcome | +|---|---|---| +| [followups-after-row-layout.md](followups-after-row-layout.md) | follow-ups after row layout, boundnames and conditional fields | items 1-3 done; 4 and 5 moved to the [backlog](../maintenance/backlog.md) | +| [open-followups.md](open-followups.md) | leftovers from the plans archived by 2026-10-19 | nothing left open | diff --git a/docs/dev/plans/archive/array-equality-spec.md b/docs/dev/plans/archive/array-equality-spec.md new file mode 100644 index 00000000..71989cd0 --- /dev/null +++ b/docs/dev/plans/archive/array-equality-spec.md @@ -0,0 +1,63 @@ +# Spec: array-aware equality for components (item 6) + +**Status: implemented and merged as #394 (2026-10-03).** Kept for porting to +`pydantic-plan`; see the Design and `array_eq` sections. + +Same design on `develop` (attrs) and `pydantic-plan`. Only the mechanism for +attaching a comparer to a field differs. + +## Problem + +Generated `__eq__` compares field tuples, so ndarray fields use numpy's +elementwise `==` and raise `ValueError: truth value ... is ambiguous`. +Any `Dis(...) == Dis(...)` fails; `Ims() == Ims()` works. + +## Design + +Per-field comparers, not a hand-written `Component.__eq__`. The framework's +own generated `__eq__` stays in charge, so a subclass decorator (`eq=True`) +can't silently shadow a base-class override. + +- attrs: `eq=attrs.cmp_using(eq=array_eq)` on array fields. +- pydantic-plan: the equivalent per-field mechanism (to be decided there). +- Applied by default in `spec.field()` (the shared field helper) when the + annotation is `NDArray[...]`, `IntArrayLike` or `FloatArrayLike`, so + generated classes and hand-written subclasses using `field()` are covered + with no template changes. Fields declared with bare `attrs.field()` must opt + in explicitly. + +## `array_eq(a, b)` semantics + +1. Shape must match. +2. dtype must match. +3. Values equal, with NaN == NaN at the same position. `equal_nan` applies + only to float/complex dtypes (`np.isnan` fails on object/str arrays). +4. dask-backed arrays are never computed. Equal only if shape, dtype, chunks + and the dask graph `name` match (same name means the same computation). + Otherwise unequal. +5. Mixed dask/numpy: unequal (no implicit compute). + +## Fields included + +- `parent` is excluded (already `eq=False`). +- `dims` is excluded (already `eq=False` on develop; an `InitVar` on + pydantic-plan). It is a construction-time hint used to derive array shapes, + so equal arrays already capture what it determined. This keeps + `test_eq_ignores_dims_and_parent` valid on both branches. + +## Tests (real generated subclass, not `Component`) + +- Two equal `Dis` compare equal. +- Changing one element of `idomain` makes them unequal. +- Same values but different dtype or shape: unequal. +- NaNs in the same place: equal. +- `parent` and `dims` do not affect the result. +- Two dask-backed fields: equal when they are the same graph, unequal otherwise, + and no compute is triggered. +- A subclass decorated with `@attrs.define` still gets array-aware equality + (guards against the override gotcha). + +## Prerequisite (done) + +The TDIS change that dropped the separate `perlen`/`nstp`/`tsmult` (#379) +arrays first, so no hand-written array fields remain to opt in. diff --git a/docs/dev/plans/archive/backend-refactor-plan.md b/docs/dev/plans/archive/backend-refactor-plan.md new file mode 100644 index 00000000..19502e64 --- /dev/null +++ b/docs/dev/plans/archive/backend-refactor-plan.md @@ -0,0 +1,135 @@ +# Backend Refactoring Plan + +Move from xattree's magic attribute proxying to explicit attrs-based components with xarray views. + +- `attrs` fields as primary storage (don't empty Python `__dict__` to populate `xr.Dataset`) +- `xr.Dataset/DataTree` views built on-demand, no syncing/proxying, single source of truth +- protocols define capabilities (e.g, dims, dataset/tree conversion), implemented by mixins +- mixin classes support `xr.Dataset/DataTree` conversion (autogenerated classes stay minimal) +- don't rely on `xr.DataTree` for tree structure, a `parent` attr and child fields are enough + +## Design + +### Components + +Leaf nodes (packages with no children) get a `to_dataset()` method provided by a mixin which implements a runtime-checkable protocol. Internal nodes (packages with subpackages, models, simulations) get a `to_datatree()` method provided by a mixin. Both expose a temporary `.data` property during migration for backward compatibility with xattree (`.data` will be removed once migration is complete). + +Base classes implement `MutableMapping` to provide dict-like access to children (e.g., `model["dis"]` accesses `model.dis`). + +#### Field decorators/metadata + +The `flopy4/spec.py` module currently wraps `xattree` field decorators. These are fairly simple, they just encode metadata creation which will need to move here. I think we can move the base field decorator definitions to the `flopy4.spec` module largely unchanged: + +- `field(...)` +- `dim(...)` +- `coord(...)` +- `array(...)` +- `path(...)` + +These set metadata keys like: `"kind"`, `"dims"`, `"dtype"`, `"coord"`, `"scope"`, `"child"` (for component relationships). + +And continue wrapping them in the `flopy4.mf6.spec` module with any MF6-specific metadata, like block name. The only difference will be that metadata attributes can live at the top level of the metadata dictionary, instead of inside a top-level "xattree" entry. + +### Protocols + +#### Dimension resolution + +- `DimensionProvider`: provides dimensions for arrays to reference + - `get_dimensions() -> dict[str, int]` + - `get_dimension_scope() -> str | type` +- `DimensionRegistry`: registers dimensions and allows lookups + +#### Xarray type conversion + +- `DatasetConvertible`: provides an `xr.Dataset` view + - `to_dataset() -> xr.Dataset` method + - `from_dataset(ds) -> Self` classmethod + - Temporary `.data` property for backward compatibility (alias for `to_dataset()`) +- `DataTreeConvertible`: provides an `xr.DataTree` view + - `to_datatree() -> xr.DataTree` method + - `from_datatree(dt) -> Self` classmethod + - Temporary `.data` property for backward compatibility (alias for `to_datatree()`) + +### Mixin classes + +#### Dimension resolution + +`DimensionRegistryMixin`: containers that aggregate dimensions at a specific scope (model/sim) + - Each registry handles a single scope (set via `_dimension_scope` class attribute) + - `register_dimension_provider(provider: DimensionProvider)`: accepts providers matching its scope, forwards others to parent + - `resolve_dimension(dim_name: str) -> int | None`: checks local cache, walks up parent chain if not found + - `get_all_dimensions() -> dict[str, int]`: returns locally registered dimensions + - Auto-registers dimension providers in `__setattr__` when components are assigned + - Sets `parent` field automatically in `__setattr__` (raises error if component already has different parent) + +Design notes: +- Components implement `DimensionProvider` protocol directly (no mixin needed) +- Model registries handle "model"-scoped dimensions (nlay, nrow, ncol) +- Simulation registries handle "global"-scoped dimensions (nper, time steps) +- Scope enforcement happens naturally via registry hierarchy (no explicit checking needed) +- Resolution walks up parent chain, allowing packages to access both model and global dimensions +- Dimension fields are immutable after initialization (`on_setattr=attrs.setters.frozen`) + +#### Xarray conversion + +- `DatasetConvertibleMixin`: leaf node components + - `to_dataset()` method calls `attrs_to_dataset(self)` + - `from_dataset(ds)` classmethod calls `dataset_to_attrs(cls, ds)` + - Temporary `.data` property (alias for `to_dataset()`) +- `DataTreeConvertibleMixin`: internal node components + - `to_datatree()` method calls `attrs_to_datatree(self)` + - `from_datatree(dt)` classmethod calls `datatree_to_attrs(cls, dt)` + - Temporary `.data` property (alias for `to_datatree()`) + +These `xr.Dataset`/`DataTree` conversion mixins can be implemented with some generic conversion functions under the hood: + +- `attrs_to_dataset(obj) -> xr.Dataset` + - Introspects attrs fields reading metadata directly + - Arrays → data_vars (shared references) + - Scalars → attrs (copied) + - Handles dims, coords via field metadata + - Validates array shapes lazily by resolving dimensions from parent registry +- `dataset_to_attrs(cls, ds) -> Self` + - Reconstructs leaf node `attrs` component from `xr.Dataset` +- `attrs_to_datatree(obj) -> xr.DataTree` + - Recursively builds tree from object + children (identified via `"child"` metadata) +- `datatree_to_attrs(cls, dt) -> Self` + - Reconstructs internal node `attrs` component from `xr.DataTree` + +### Module organization + +- `flopy4/protocols.py`: Protocol definitions (`DimensionProvider`, `DimensionRegistry`, `DatasetConvertible`, `DataTreeConvertible`) +- `flopy4/mixins.py`: Mixin implementations (`DimensionRegistryMixin`, `DatasetConvertibleMixin`, `DataTreeConvertibleMixin`) +- `flopy4/xarray.py`: Generic conversion functions (`attrs_to_dataset`, `attrs_to_datatree`, etc.) +- `flopy4/spec.py`: Field decorator functions (`dim`, `array`, `coord`, `field`, `path`) +- `flopy4/mf6/spec.py`: MF6-specific field decorator extensions (adds `"block"` metadata, etc.) + +## Migration Strategy + +Go top down. Start with internal nodes (simulation, model), then migrate leaf nodes (packages). + +Containers continue to define the existing `xattree` interface contract (an `xr.DataTree` under a `.data` property) allowing gradual migration without adapters. + +### 1. Implement foundations + +Move field decorator metadata system from `xattree` to `flopy4.spec` module (remove top-level "xattree" key, store metadata directly). Add `"child"` metadata for component relationships. + +Implement protocols in `flopy4/protocols.py` (`DimensionProvider`, `DimensionRegistry`, `DatasetConvertible`, `DataTreeConvertible`). + +Implement mixins in `flopy4/mixins.py` (`DimensionRegistryMixin`, `DatasetConvertibleMixin`, `DataTreeConvertibleMixin`). Auto-set `parent` field and auto-register dimension providers in `__setattr__` (with error if reassigning to different parent). + +Implement generic conversion functions in `flopy4/xarray.py` (`attrs_to_dataset`, `attrs_to_datatree`, etc.) with lazy dimension validation. + +### 2. Update internal nodes + +Refactor `Context` base class, use `@define` instead of `@xattree`, inherit `DimensionRegistryMixin`, `DataTreeConvertibleMixin`, and `MutableMapping`. + +Refactor `Simulation` class, use `@define` instead of `@xattree`, set `_dimension_scope = "global"`, continue exposing `xr.DataTree` under `.data` (temporary, calls `to_datatree()`). + +Refactor base `Model` class and `Gwf` concrete class, use `@define` instead of `@xattree`, set `_dimension_scope = "model"`, inherit `DataTreeConvertibleMixin`. Add `parent` field linking to `Simulation` (set automatically in parent's `__setattr__`). Continue exposing `xr.DataTree` under `.data` (temporary). + +### 3. Update leaf nodes + +Refactor `Package` base class, using `@define` instead of `@xattree` and inheriting `DatasetConvertibleMixin`. Continue exposing `xr.Dataset` under `.data` (temporary, calls `to_dataset()`). Add a `parent` field linking to containing model (set automatically in parent's `__setattr__`). + +Migrate packages one at a time, starting with dimension providers (`Tdis`, `Dis`), then dimension consumers (`Npf`, `Sto`, etc.). Dimension-providing packages implement `DimensionProvider` protocol directly (no mixin needed) and have immutable dimension fields (`on_setattr=attrs.setters.frozen`). They auto-register when assigned to parent via parent's `__setattr__`. diff --git a/docs/dev/plans/archive/benchmark-infra-plan.md b/docs/dev/plans/archive/benchmark-infra-plan.md new file mode 100644 index 00000000..3f70d8d2 --- /dev/null +++ b/docs/dev/plans/archive/benchmark-infra-plan.md @@ -0,0 +1,194 @@ +# Benchmark infrastructure plan + +**Archived 2026-10-19.** Done: merged as #482 ("benchmarking infra"), with the suite in `benchmarks/`, tiers, `bench-memory`, `bench-profile` and `bench-report`; `docs/profile/` is gone. The "Later" items (history across commits, CI regression checks, typed-loader benchmarks) are not started. + +This plan merges our three benchmark approaches into one suite that can be checked in: + +- the `origin/load-baseline` pytest-benchmark suite; +- `docs/profile/`; +- the scratchpad harness behind [`io-benchmark-results.md`](io-benchmark-results.md). + +The suite will live in a top-level `benchmarks/` directory, run on pytest-benchmark. + +## Decisions + +- **pytest-benchmark is the engine, not asv.** asv is too heavyweight. + - pytest-benchmark is already a dependency, and `load-baseline` uses it. + - Its JSON output is what other tools read, so we can add these later without touching the benchmarks: + - history across commits (github-action-benchmark or Bencher); + - regression checks in CI (pytest-codspeed accepts the same `benchmark` fixture). + - The fair-comparison logic is specific to this project, so we write it either way. +- **Memory comes from pytest-memray, not RSS.** + - memray counts only what is allocated during the test, numpy included, so it gives each model's own footprint without the import baseline. + - It slows things down, so it runs in its own session (`bench-memory`) and never alongside the timing runs. +- **The subprocess worker is kept only for:** + - cold start; + - the memory cost of importing each library, which memray can't see. +- **Large models are their own tier.** They run only with `--bench-tier large`, and the registry downloads them on first use. +- **Results stay out of git.** + - `.benchmarks/` is gitignored. + - A Markdown report generated from a saved run is committed only when it's worth keeping. + - `docs/profile/results/` goes. +- **No CI gating for now.** Shared runners are too noisy for absolute times. Revisit later with codspeed or a manual or nightly workflow. + +## What to keep from each source + +### `origin/load-baseline` (this is the base) + +The branch is unmerged and dated 2026-09-24. Its commits are b00ac6ba, 5bae03a6, 8f7e4c25 and e4d095e3, plus the parse-benchmark commits af3b2ff0 and c101c031. + +Files: + +- `test/mf6/test_mf6_load_benchmark.py` +- `test/mf6/test_mf6_parse_benchmark.py` +- `test/mf6/load_benchmark_models.txt` (~180 models) +- `docs/dev/load-benchmarks.md` +- the `bench-load` task, defined in `pyproject.toml` on that branch + +What to keep: + +- **Fixed flopy3 options:** `FLOPY3_OPTIONS = {verbosity_level: 0, lazy_io: False, verify_data: False, use_pandas: True}`. +- **`load_only_for(dfn_names)`:** limits flopy3's `load_only` to the package types flopy4 actually loaded. + - It finds those types with `load_flopy4_recording`, which patches `Package.load`. + - It handles the cases where several flopy4 components share one flopy3 type: `rcha`/`evta`, the `*g` packages and exchanges. +- **`check`:** in a subprocess, an audit hook confirms both libraries opened the same set of files. A model that fails the check is left out. +- **`touch(x)`:** the materialize step, which sums every array and list. + - It's needed because flopy3 doesn't read OPEN/CLOSE arrays at load time. + - flopy3 then rereads them on every `get_data()`, without caching. +- **Benchmarks:** + - `test_load` + - `test_load_materialize` (the fair one) + - `test_load_materialize_largest` + - `test_cold_start` + - the basic vs. typed parse benchmark +- **The opt-in skip, and the pinned manifest** (`--bench-load-manifest`). The manifest fails loudly if a listed model stops loading or stops doing the same work in both libraries. +- **The module layout.** + - The top of the module imports only the standard library, and the script entry point exits before pytest is imported. + - This keeps the cold-start timing free of pytest's import cost (~0.1 s). + +### Scratchpad harness (this session) + +The scripts are in `/private/tmp/claude-502/-Users-wbonelli-dev-pyphoenix-project/0c2187ab-eda9-4492-9ae6-11e169356e97/scratchpad/`: + +- `bench/worker.py` +- `bench/run.sh` +- `bench/summarize.py` +- `size_corpus.py` + +What to keep: + +- Write benchmarks that write into a fresh empty directory each round. + - flopy4: set `sim.workspace = out`, then call `sim.write()`. + - flopy3: `set_sim_path(out)`, then `write_simulation(silent=True)`. +- Record bytes written, and keep a round-trip check: reload the written output and compare arrays. +- Keep the size-ranged, tagged model selection. It's the basis for the tiers below. +- Drop its flopy3 setup (default options, no `load_only`) and `run.sh`. + +### `docs/profile/` (on develop) + +Files: + +- `run_all.py` +- `ff_read.py`, `ff_write.py` +- `test1000_write.py`, `test1005_write.py` +- `chunked_profile.py` +- `diag_list_scaling.py` +- `_timer.py` +- `test_profile_scripts.py` +- `results/` + +What to keep and what to drop: + +- Keep the scenario scripts as profiling tools, moved to `benchmarks/profile/`. +- Keep the git commit and timestamp in every result, and Markdown report generation. +- Drop `run_all.py`, `_timer.py` and `results/`. +- Check whether `test_profile_scripts.py` is still worth keeping after the move. + +## Layout + +``` +benchmarks/ + conftest.py # --bench-tier, --bench-manifest, model fixtures, opt-in skip + models.toml # tiers + tags per model + _ops.py # per library: load / materialize / write; stdlib-only at module level + _check.py # file-equivalence (audit hook), round-trip compare + _worker.py # subprocess entry: cold start, import memory + test_load.py # load, load+materialize (from load-baseline) + test_write.py # write; bytes written in extra_info + test_parse.py # basic vs. typed (from load-baseline) + test_memory.py # pytest-memray peak per (library, model, op); run in its own session + report.py # pytest-benchmark JSON (one run or two to compare) -> Markdown + profile/ # scenario scripts moved from docs/profile/ +``` + +- `_ops.py` is the only place load, materialize and write are defined. Every benchmark and the worker go through it, so flopy3's options can't drift between benchmarks. +- The equivalence and round-trip checks run once per model per session, as session fixtures. A model that fails is reported and left out, so it can't skew the comparison silently. +- Each model's tags, input size and bytes written go into `extra_info`, so `report.py` can group results by data shape. + +## Tiers + +Models come from the modflow-devtools registry. All are in `KNOWN_PASSING`. + +- **smoke** (about 5 tiny models, under a minute): + - test001a_Tharmonic + - test006_gwf3_disv_ext + - test028_sfr + - test045_lake4ss_dev + - test120_mv_dis-lgr +- **standard** (the default): the smoke tier, plus: + - test016_Keating_dev + - test033_wtdecay + - test019_VilhelmsenGF +- **large** (opt-in, downloaded on first use): + - test1000_751x751 + - test016_Keating_disu_mfusg + - test1005_secp + - test1002_biscqtg_disv_dev + - test1001_Peterson +- **full**: the ~180-model manifest from `load-baseline`. + +Each model gets tags for its data shape: + +- `array`: secp, Peterson, Keating_disu_mfusg, 751x751 +- `list`: biscqtg +- `exg`: dis-lgr +- `openclose`: disv_ext, VilhelmsenGF, wtdecay +- `sfr`, `lak`: package-specific cases + +## pixi tasks + +- `bench`: the standard tier, with autosave. +- `bench-large`: the large tier. +- `bench-full`: the full manifest. +- `bench-compare`: compare against the last saved run. +- `bench-memory`: a pytest-memray session. +- `bench-report`: write the Markdown report. + +## Steps + +1. **Port `load-baseline`.** On a local branch off develop: + - bring in the benchmark files from `load-baseline` (cherry-pick, or check them out); + - move them into `benchmarks/`; + - split out `_ops.py` and `_check.py`; + - check that `bench` runs. +2. **Write benchmarks.** Add `test_write.py`, bytes written and the round-trip check. +3. **Tiers.** Add `models.toml` and the `--bench-tier` option. +4. **Memory.** + - Add pytest-memray to the dev environment, then add `test_memory.py` and the `bench-memory` task. + - Have `_worker.py` handle cold start and import memory. +5. **Report.** + - Add `report.py`. + - Rerun the standard and large tiers with the fair flopy3 options. + - Regenerate [`io-benchmark-results.md`](io-benchmark-results.md) from those runs. +6. **Profiling scripts.** + - Move the `docs/profile/` scripts into `benchmarks/profile/`. + - Remove `run_all.py`, `_timer.py` and `results/`. + - Update any docs that point to them. + +Run checks through `pixi run -e dev`: the smoke tests, plus the benchmark files that changed. Don't run the full test suite. + +## Later + +- History across commits with github-action-benchmark or Bencher. Both read pytest-benchmark JSON. +- CI regression checks with pytest-codspeed. Note that its instruction counts mostly ignore time spent waiting on IO. +- Typed-loader benchmarks once the typed loader is wired into structuring: add `flopy4-typed` to `LOADERS`. diff --git a/docs/dev/plans/archive/binding-rows.md b/docs/dev/plans/archive/binding-rows.md new file mode 100644 index 00000000..d7acb91d --- /dev/null +++ b/docs/dev/plans/archive/binding-rows.md @@ -0,0 +1,176 @@ +# Binding rows: generate them once, interpret them generically + +Follow-up to #501 (reference fields generated from the DFNs' model-name +keys). Goal: one generated description of each name-file binding row, and +one generic piece of code that reads and writes rows from it, instead of +row shape being reconstructed by heuristics in several places. + +Objectives, in tension: +- derive as much as possible from the DFNs +- keep the implementation simple +- generated classes reflect their DFNs accurately + +## Problem + +A binding row registers a child component in its parent's name file +(sim-nam `models`/`exchanges`/`solutiongroup`, a model nam's `packages`). +The DFNs describe every row completely (v2 schema `2.0.0.dev3`, devtools +`b7c43566`): + +| role | marked by | sim `exchanges` | sim `models` | sim `solutiongroup` | `-nam` `packages` | +|---|---|---|---|---|---| +| type | column named by the File's `component_ftype` | `exgtype` | `mtype` | `slntype` | `ftype` | +| file | `File` with `component_ftype` | `exgfile` | `mfname` | `slnfname` | `fname` | +| child's own name | `pk` | — | `mname` | — | `pname` (**not marked yet**, see devtools plan) | +| references | `fk` (to `models.mname`) | `exgmnamea`, `exgmnameb` | — | `slnmnames` (string Array) | — | + +Each component DFN also has its `ftype` (`GWF6-GWT6`, `IMS6`, `CHD6`; +`gwf-chd` and `gwf-chdg` are both `CHD6`). + +Codegen discards the roles: a parent gets `child(block="exchanges")` and +nothing more. Runtime then rebuilds row shape in: + +1. `converter/binding.py` `component_ftype(cls)`: ftype guessed from the + Python class name (exchange name slicing, 4-letter `g`/`a` suffix, + `slntype`). `component.py` `get_ftypes()` builds the token registry from + those guesses, qualifying collisions by model prefix. `package.py` + derives the default file extension from it. +2. `converter/binding.py` `Binding.from_component._get_binding_terms` + (egress): `isinstance` cascade (Solution -> `_solved_models`; reference + fields; Model/Package -> name). +3. `converter/ingress/structure.py` `_resolve_bindings` (ingress): + `row[2]` is the pname unless Exchange/Solution; `_exchange_models` + separately gives an exchange its models for its load context; + `_apply_binding_terms` sets references after the load. +4. `utils/codegen/make.py` (#501): `_binding_row` searches back from the + child to its row to generate reference fields; `_REFERENCE_NAMES` + renames `slnmnames` -> `models`. + +Nothing ties these together. + +## Design + +### 1. Generated `ftype` + +Codegen emits the DFN's ftype on every component class: + +```python +class GwfGwt(Exchange): + ftype: ClassVar[str] = "GWF6-GWT6" +``` + +- Delete `component_ftype`'s heuristics; callers read `cls.ftype` + (`get_ftypes`, `Package` default extension, `_resolve_bindings` + matching, `attrs_xarray`'s G/A note). +- Handwritten bases (`Model`, `Exchange`, `Solution`, `Package`) declare + `ftype: ClassVar[str]` without a value. `Solution.slntype` likely goes + too (derive from `ftype` if still needed). +- The G/A pairs keep sharing an ftype (as in MF6); `get_ftypes`' collision + handling and `_disambiguate_ga_variant` (peek at READASARRAYS / + READARRAYGRID) stay. That's MF6 behavior, not a heuristic. +- Check: models' ftype (`GWF6`) comes from `-nam`'s `ftype`, and + exchange/solution tokens match what `component_ftype` produced, so no + name file changes. The round-trip snapshot tests (#498) will tell. + +### 2. Generated `row=` on child fields + +Codegen, building a parent's child field for a list with a +`component_ftype` file column (`_child_field_spec`'s `component_ftype` +branch), emits the row as the child attribute each column maps to, in +column order: + +```python +models: list[Model] = child(block="models", row=("ftype", "filename", "name")) +exchanges: list[Exchange] = child( + block="exchanges", row=("ftype", "filename", "exgmnamea", "exgmnameb") +) +solutiongroup: list[Solution] = child( + block="solutiongroup", row=("ftype", "filename", "slnmnames") +) +packages-style fields: child(block="packages", row=("ftype", "filename", "name")) +``` + +Role -> attribute: type column -> `ftype`; file column -> `filename`; +`pk` column -> `name`; `fk` column -> the reference field of the same +(safe) name. A column with no role: error at codegen (forces the DFN +annotation rather than a guess). + +Keyword file records (`TDIS6 FILEIN`, `HPC6`), which aren't rows, are +untouched (`_keyword` children). + +`flopy4.spec.child` gains `row: tuple[str, ...] | None`, kept in metadata. + +### 3. One generic `Binding` + +`converter/binding.py` becomes the single row <-> child mapping, driven by +the field's `row`: + +- egress: `[value(child, a) for a in row]`; `ftype` upper, `filename` as + posix, references as names (`reference_names`), list values spread + (only the last column may be a list); trailing empty name dropped as now. +- ingress: given the row and the target class (matched by `cls.ftype`, + G/A disambiguation as now), assign columns in order: `name` -> `name=` + load kwarg, reference columns resolved against the models loaded so far + (`resolve`, case-insensitive), a list column takes the rest. +- References are resolved **before** the child loads and passed in, so an + exchange's load context comes from its own `exgmnamea`/`exgmnameb`. + Delete `_exchange_models`, the pname-unless-Exchange/Solution branch and + `_apply_binding_terms`. +- `_make_binding_blocks` (egress) and `_resolve_bindings` (ingress) call + into `Binding` and keep only traversal (blocks, numbered + SOLUTIONGROUP, dims providers first, unknown-row warnings). + +Handwritten semantics that stay (the DFNs can't say them): +- `_reference_problem` (Exchange: model types per side; Solution: + `solves`). +- A lone solution listing no models solves them all: a hook for a + reference column's written value when empty, e.g. + `Component._row_value(attr)` overridden by `Solution` to return + `_solved_models()` for its models column. Keep it the only override. + +### 4. Reference generation folded into the row pass + +Build one index over the DFNs, component name -> (parent, row record), +once per codegen run (in `make_modules` / `build_component_spec`'s +caller), from the same lists `_child_link` resolves. Both the parent's +`row=` and the child's reference fields come from it. Delete +`_binding_row`. + +## Open decisions + +- **`slnmnames` vs `models`.** Generating names from the DFN makes the + field `slnmnames` (consistent with `exgmnamea`), and `_REFERENCE_NAMES` + goes. Recommended: DFN name, optionally a read-only `models` alias + property on `Solution`. Decide before step 4; touches + `Solution`/`simulation_methods`/`model.py` (`Model.solution`), tests, + docs/examples using `Ims(models=...)`. +- **`pname` as `pk`.** Resolved: devtools `7f72ae21` (#400) marks + `-nam.packages.pname` as `pk` (plan: + `~/dev/modflow-devtools/archive/pname-pk-plan.md`). Step 2 needs flopy4's + devtools pin moved past it (currently `07d1273a` in `pixi.lock`) and a + DFN re-sync. No hardcoded name-column rule is needed. + +## Steps + +Each a separate commit/PR, each verified with targeted tests plus the +round-trip snapshot tests (#498) touching name files. + +1. Generated `ftype`; delete `component_ftype`. Sync at the contract commit + (`flopy4 mf6 sync MODFLOW-ORG/modflow6@ --mf6-version + `, not bare `sync`, which takes the latest release when no + binary is on PATH). +2. `row=` on generated child fields; generic `Binding` for egress and + ingress; delete the `isinstance` branches, `_exchange_models`, + `_apply_binding_terms`. +3. Row index in codegen; reference fields from it; delete `_binding_row`; + settle `slnmnames`. + +Expected net: less handwritten code than today; every row's shape in one +generated place, one interpreter. + +## Notes + +- Run tests through `pixi run -e dev`, targeted (`-k`), not whole files or + the full smoke suite. +- `_mixin_fields.py` now looks for attributes a base class lacks on its + leaf subclasses (#501); generated `ftype`/fields on subclasses rely on it. diff --git a/docs/dev/plans/archive/boundname-plan.md b/docs/dev/plans/archive/boundname-plan.md new file mode 100644 index 00000000..0ca72939 --- /dev/null +++ b/docs/dev/plans/archive/boundname-plan.md @@ -0,0 +1,241 @@ +# Boundnames and token text, end to end + +**Archived 2026-10-19.** Done: #470 (boundname fixes) and #478 (boundnames on the `when` mechanism). Step 3 (keep number-like text) is open as item 2 of [`followups-after-row-layout.md`](followups-after-row-layout.md); step 4's `begin`/`end` check was dropped. + +Plan for one piece of work covering how flopy4 builds, writes and reads +boundnames (and other free-text values like obsnames), so that a value +round-trips the way MF6 itself treats it. Pulls together #459, #465, the +quoting in #462, and the reader bugs found while reviewing #462 and #464. + +Checked against develop `e73d059d`, #462 at `3c0a4706` (merged as `0b2bb73d`, +trimmed: no `_RESERVED_PREFIX`/`_LOOKS_LIKE_NUMBER`), #464 at `0c2f451d`, +and MF6 6.7.0 (`~/dev/modflow6/bin/downloaded/mf6`) with hand-written input +files. + +## Status (2026-10-15) + +Open as #470 ("boundname fixes", supersedes #465), branch `boundname-text` +on the fork at `7069b861`, off develop `e8f2b939`, four commits: + +- Step 1: `spec.string(empty_is_none=...)`; the generator sets it on the + last column when optional, untagged and a string (the 28 generated + boundnames). The name file's `pname` isn't a generated item, so the + loader maps `''` to None. +- Step 2: done on develop by #467's row layout (presence from the + `boundnames` option, per [`record-unify.md`](../lists/record-unify.md) rule 4); #470 only trims + the docstrings it left. +- Step 5: other string fields and files reject `""`; the writer raises on + an empty value as a backstop. +- Step 6: boundname round-trip tests, including numeric and keyword-like + names. + +Not in #470: + +- Step 3 (keep number-like tokens' text). Without it, an obsname `1.50` + still reloads as `1.5`. +- Step 4's `begin`/`end` check (`tokens2line`), dropped before the PR. + +Follow-ups: + +- #470 conflicts with `option-columns` (`item.py`, `package.py`, + `test_mf6_io_roundtrip.py`); rebase whichever merges second. +- After both merge, fold boundnames into the option mechanism + (`modflow-devtools/when-attribute-plan.md`, flopy4 step 3): boundname + gets `when="boundnames"` from the DFN, `RowContext.boundnames` becomes + `"boundnames" in options`, and `Package._coerce_item_list`'s boundname + branch joins the option loop. + +## What MF6 does + +Verified by running MF6 on small GHB + OBS models, and in the source +(`BlockParser.f90`, `urword` in `InputOutput.f90`): + +- **A row's columns are decided by position, from the package's options.** + A boundname is whatever comes after the fixed and aux columns. Its text + can be anything: `5`, `1.50`, `end`, `endpoint` all load. +- **Quotes only group a word.** `urword` returns the text inside the quotes, + so a quoted word is otherwise treated exactly like a bare one. +- **Only a line's first word can be BEGIN/END.** `end` later in a line is + data (a boundname `end`, or an OBS id `end`, both work). An obsname `end` + at the start of a line ends the block, **quoted or not** (`'end'` fails + the same way). +- **OBS can't refer to a numeric boundname.** An OBS id that looks like a + number is read as a cell location (`5` -> layer 5), quoted or not. A + boundname `5a` works. +- `''` as a boundname loads without error. + +## What flopy4 does now + +Built in Python, written, reloaded with `Simulation.load`, and run through +MF6 (all MF6 runs succeeded): + +| Case | develop | #462 | +|---|---|---| +| boundnames `end`, `begin`, `endpoint` | round-trips, written bare | round-trips, written quoted | +| boundnames `5`, `1.50` | **reload as `None`** | written `'5'`, `'1.50'`, **still reload as `None`** | +| boundname `5` with one aux column | **boundname `None`**, aux kept | same | +| obsname `1.50` | **reloads as `1.5`** | written `'1.50'`, round-trips | +| boundname `""` | written as no boundname, reloads `None` | same | +| boundname `None` mixed with named rows | round-trips | same | + +## Problems + +1. **The reader decides boundname presence by type, not position** + (`item.py:720-722`, `Item.from_tokens`). A trailing token is taken as a + boundname only if it's a string that doesn't parse as a number. So a + numeric boundname is silently dropped, whether or not it was quoted. + (#462's comment that it also shifts columns doesn't hold: with an aux + column, the aux value is kept and only the boundname is lost.) +2. **The reader loses the original text of numeric-looking values.** The + basic transformer turns every number-like token into a number before the + schema is known, so a string field gets `str(1.5)`: obsname `1.50` comes + back as `1.5`, which also renames its CSV column. #462's quoting hides + this for files flopy4 writes, but not for files written elsewhere. +3. **Python-side construction is inconsistent about `""`.** In + `construct_item`, `((0,0,0), 1.0, "")` gives boundname `None` but + `((0,0,0), 1.0, (), "")` gives `""` (#465). The writer already treats + `""` as no boundname (`Item.to_tokens`: `if boundname:`), and MF6 has no + use for an empty name, so `""` is a third "no boundname" state that can't + survive a round trip. +4. **The writer quotes things MF6 doesn't need quoted** (#462's + `_RESERVED_PREFIX` and `_LOOKS_LIKE_NUMBER`), working around problems 1 + and 2 and the typed grammar's begin/end bugs (fixed in #464). It also + can't help in the one case MF6 can't represent: a `begin`/`end` value at + the start of a line. + +## Plan + +1. **Normalize `""` to `None` for trailing optional labels whenever an item + is built.** Which fields get this comes from the DFNs, not a name list + (see "Deriving the rule from the DFNs" below): it's boundnames and the + model name files' `pname`. The rest of this step says "boundname" for + short. + This makes `None` the only "no boundname" value. Writing needs no change: it + already never writes an empty boundname. Put the normalization in a + converter on the `boundname` field itself, not in `construct_item`: + user input (tuples, dicts, dataframes) and loading (`from_tokens`) both + build items through the class, so one converter covers both. That + matters after #462, where a hand-written `''` boundname (which MF6 + accepts) would otherwise load as `""`. (Need to check that the field + helper accepts a converter.) Supersedes #465. +2. **Read boundname presence from position.** When the row's widths are + known (aux count from `sizes`, no optional columns whose presence is + inferred from the token count), a boundname is there if and only if a + token is left over. Keep the type check only as a tiebreaker for rows + that are truly ambiguous, i.e. when an inferred optional group and a + boundname would use the same token. Even then, the options that gate + that group (MF6 decides from them) are the better source where we have + them. + **Superseded by [`record-unify.md`](../lists/record-unify.md), step 4, rule 4** (agreed + 2026-10-14): presence comes from the `boundnames` option (or, for Python + input with the option unset, from one surplus value, which sets it), so + no type-check tiebreaker is needed. Whichever lands first should do it + the way that rule says. +3. **Keep the original text of number-like tokens**, so a string field gets + exactly what was in the file. Smallest change: have the basic + transformer return number subclasses that carry the source text (e.g. a + `float`/`int` subclass with `.text`), and have string fields and the + boundname take `.text`. The alternative, keeping all tokens as strings + until the schema is known, is cleaner but touches far more code. +4. **Writer: quote only for grouping, and raise on what MF6 can't + represent.** `quote_if_needed` quotes only to keep a value one word + (#462's `_BARE_WORD` rule). Raise for a `begin`/`end` value that would + start a line (an obsname), since MF6 rejects it quoted or not, and keep + #462's error for values containing both quote types. (#462 merged + without `_RESERVED_PREFIX`/`_LOOKS_LIKE_NUMBER`, so there's nothing to + remove.) +5. **Reject empty strings in other free-text fields; don't convert them.** + Outside boundnames, MF6 fails on `''` or does something meaningless + with it (MF6 6.7.0): + + | Field | MF6 with `''` | + |---|---| + | obsname | fails, misleadingly ("Unexpected end of file reached") | + | OBS id | accepted silently, an observation of nothing useful | + | `AUXMULTNAME` | fails ("no AUX variable found with this name") | + | `TS6 FILEIN`, `OBS6 FILEIN` | fails (`Could not open ""`) | + + Required fields have no `None` to convert to, and converting an optional + option's `""` to `None` would silently drop something the user set, + usually a bug (e.g. a filename built from an empty variable), which MF6 + would have caught. So add a non-empty validator on free-text string + fields, applied wherever items/components are built (API and loader + alike; a file with `''` there is one MF6 rejects anyway). Step 1's + trailing optional labels (boundname, nam `pname`) are the only + exception: there `""` naturally means "no label" (e.g. a partly-filled + dataframe column), and MF6 accepts an absent one. Both rules are derived + from the DFNs; see below. `quote_if_needed("")` + then raises instead of writing `''`, as a backstop. During + implementation, list the DFNs' string fields and confirm none gives an + empty value a meaning. +6. **Tests.** A round-trip test over the matrix above (build -> write -> + reload) for each case: boundnames `5`, `1.50`, `end`, `begin`, + `endpoint`, `""`, `None`, numeric boundname with aux, obsname `1.50`. Plus + reader tests loading hand-written files (not just flopy4 output), so + problem 2 is covered for files written elsewhere. If CI has an `mf6` + binary (the UZF test suggests so), also run the written files through it. + +## Deriving the rule from the DFNs + +Steps 1 and 5 shouldn't name `boundname`. Classifying every string, file and +string-array field in the v2 spec (devtools `LocalDfnRegistry`, +`~/dev/modflow6-dfns` at `6943c633`, schema `2.0.0.dev3`) shows one +structural class that matches: + +**optional, untagged, and the last column of a list row.** Exactly 46 +fields, all `boundname` (40) or the model name files' `pname` (6, +`ftype fname pname`, where MF6 picks a default name if it's absent). +Nothing else in the spec, of any type, is in this class. + +Why it's principled: a trailing, optional, untagged column is the only kind +of optional column MF6 handles per row by simply leaving it off. Nothing +after it shifts, and no keyword or option controls it, so it's the only +place where empty and absent mean the same thing. Fields that look similar +fall outside it, as they should: + +| Field(s) | Why not in the class | Rule | +|---|---|---| +| MVR `mname`, `mname1`, `mname2` | optional and untagged but not last; all present or all absent, controlled by `MODELNAMES` | non-empty | +| OBS id union's `boundname` arm | required, not last; a reference to a boundname, not a label | non-empty | +| tagged optional strings (95, e.g. `AUXMULTNAME`, `CRS`) | follow a keyword; empty means nothing | non-empty | +| SSM `pname` (required), MVR `pname` (key) | required | non-empty | + +Implement both rules in code generation: the generator already emits each +field's `optional`/`tagged` metadata, so it can attach the `""` -> `None` +converter to fields in this class and a non-empty validator to every other +free-text string field. Nothing hand-maintained, and new packages pick it +up on sync. (`pname` gains from it too: today an empty `pname` would be +written as `''` under #462, since `to_tokens` only skips `None`.) + +## How this relates to the PRs + +- **#462 (quoting), merged as `0b2bb73d`.** Landed trimmed: the shared + `QUOTED` terminal, `unquote`, the merged `quote_if_needed` (`_BARE_WORD` + rule) and quoting in `record`/`data2lines`. No `_RESERVED_PREFIX` or + `_LOOKS_LIKE_NUMBER`, so step 4's removal is already done; what's left of + step 4 is raising on a line-leading `begin`/`end` and on `""`. Without the + numeric quoting, obsname `1.50` reloads as `1.5` again, as on develop + before; step 3 fixes it. +- **#464 (begin/end terminals), merged.** Grammar only, no boundname + code. The `BARE_WORD`/`QUOTED_WORD` rename follows it. +- **#465 (`""` boundname), closed.** Superseded by #470. +- **#459 (merged).** Made a trailing `None` an absent boundname in + `construct_item`. Step 1 builds on it. + +## Sequencing + +1. ~~#462~~, ~~#464~~, ~~#467~~, merged. +2. `option-columns` (option-gated columns, the OBS numeric-boundname + warning, codegen reading the DFN's `when`), then #470 rebased onto it, + or the other way round. +3. The boundname fold-in, after both. +4. Step 3, if still wanted, as its own PR. + +## Open questions + +- ~~Should flopy4 warn when an OBS id refers to a numeric boundname?~~ Yes: + `option-columns` warns. +- ~~Are trailing optional groups and a boundname ambiguous with the options + known?~~ No: with the package options in `RowContext.options`, columns + an option turns on are placed by the option (`option-columns`), so no + type check is needed. diff --git a/docs/dev/plans/archive/cellid-node-arrays.md b/docs/dev/plans/archive/cellid-node-arrays.md new file mode 100644 index 00000000..e5c6e458 --- /dev/null +++ b/docs/dev/plans/archive/cellid-node-arrays.md @@ -0,0 +1,117 @@ +# Cellid arrays: what changes for flopy4 + +modflow-devtools branch `cellid-node-arrays` (wpbonelli fork) +changes how every cellid column is represented in the dev2/dev3 DFN spec. Not +yet merged upstream. flopy4 depends on `modflow-devtools` from MODFLOW-ORG +`main` (`pyproject.toml`), so nothing changes here until it merges. To try it +earlier, install that branch into the env, after which `LocalDfnRegistry` +migrates v1 DFNs with the new rules. This replaces the locally patched spec +that put `node` on `cellidsj`. + +## The schema change + +There's now one encoding for a cellid: + +``` +Array(dtype="integer", shape=["ncelldim", ...], index=True, cellid=True) +``` + +- The first shape axis is always `ncelldim`: one cellid's components (3 for DIS, + 2 for DISV, 1 for DISU). Shapes list the fastest-varying axis first, as grid + arrays already do (`botm` is `["ncol", "nrow", "nlay"]`). +- Any further axes count cellids. These columns are still written inline on the + row, one cellid after another. +- `Integer.node` is **removed**. The attribute is now `cellid`, on `Array` only, + and requires integer dtype, `index=True`, a leading `ncelldim`, and a + list-item context. +- The leading `ncelldim` isn't resolved as a dim in the component's scope. Its + value depends on the grid the column refers to. Exchanges don't see + `ncelldim` at all, since their parent is the simulation. + +| column | before | after | +|---|---|---| +| stress `cellid`, HFB `cellid1`/`cellid2`, CSUB, PRP, ... | `Array(shape=["ncelldim"])` | same, plus `index`, `cellid` | +| exchange `cellidm1`/`cellidm2` | `Integer(index, node)` | `Array(shape=["ncelldim"], index, cellid)` | +| GNC `cellidn`/`cellidm` | `Integer(index, node)` | `Array(shape=["ncelldim"], index, cellid)` | +| GNC `cellidsj` | `Array(shape=["numalphaj"], index)` | `Array(shape=["ncelldim", "numalphaj"], index, cellid)` | + +44 components change. Every stress-package `cellid` gains `index` and `cellid`; +the exchanges and GNC change type. + +## What happens in flopy4 + +From reading the code. Not yet run against the branch. + +### Breaks: GNC `cellidsj` (codegen `filters.py` and `make.py`) + +Even with cellid detection fixed (next section), the column-role chain in +`make.py` (~L227-256) only treats a 1-element shape as +`counted`. `cellidsj` now has a 2-element shape, so it falls through to +`elif col.is_cellid` and becomes `role="cellid"` with +`shape="ncelldim,numalphaj"`. In other words, it's read as **one** cellid. The +generated `gwf/gnc.py`, now +`cellidsj: ... = field(array=True, cellid=True, count="numalphaj")`, would +regress. The GNC tests in `test/mf6/test_mf6_codec.py` (~L420-443) should catch +it. + +Fix: for a `cellid` column, use `shape[1:]` as the count shape. If it's empty, +the column is one cellid (`role="cellid"`). If it's one dim, the column is +`role="counted"` with `count=shape[1]` and `cellid=True`, i.e. the existing +counted-cellid path, unchanged downstream. Read and write (`item.py` +`from_tokens`/`to_tokens`, `cellid` + `array` metadata) already handle a +counted cellid column. + +### Breaks: cellid detection (`filters.py` `_fields_to_columns`, ~L952) + +```python +is_cellid=(isinstance(col, Array) and list(col.shape or []) == ["ncelldim"]) +or bool(getattr(col, "node", False)), +``` + +The shape check still catches the single cellids, which are now `["ncelldim"]` +arrays. But the `node` fallback matches nothing now, so `cellidsj` (shape +`["ncelldim", "numalphaj"]`) is **not detected as a cellid at all**. It has to +become `isinstance(col, Array) and col.cellid`, which covers every cellid. The +`getattr` and its comment ("only Integer has `node` yet") go away, and +`ColumnSpec.is_cellid`'s comment needs updating. This fix has to land together +with the `make.py` fix above, or `cellidsj` falls through to a non-cellid role. + +### Unaffected + +- **`index` on cellid columns.** It's newly set, but `is_cellid` is checked + before `is_index` in `make.py`, and `cellid` metadata is checked before + `index` in `item.py`. So there's no double 1-based offset. Worth a quick look + at any other place that reads `is_index` without checking `is_cellid` first. +- **Stress packages.** Their shape didn't change, so their generated classes + should come out the same. +- **ncelldim inference** (`item.ncelldim_from_dims` / `infer_ncelldim`) is + unchanged. Cellid width is still taken from the grid at read time. + +### Still open (not addressed by this change) + +- **Exchanges** (`EXG_TIER` in `test_mf6_codegen.py` excludes gwfgwf etc.). + Their cellid columns now look like any other cellid, which helps. But each + column needs its own model's `ncelldim`, and the spec still doesn't say which + model a column refers to. flopy4 has to hard-code it: in `exg-*`, `cellidm1` + is model 1 and `cellidm2` is model 2. In GNC under an exchange, `cellidn` and + `cellidsj` are model 1 and `cellidm` is model 2. A later schema addition + (e.g. `cellid: "1" | "2"`) may make this explicit. +- **PRP** (`test_prt_basic` docstring). PRP's `cellid` column is a plain + `["ncelldim"]` cellid in a list item, now marked `cellid`. The + "(nreleasepts, ncelldim)" limitation is flopy4-side, and this change doesn't + touch it. +- **Multi-dimensional inline arrays.** `cellidsj` is the first array with more + than one dimension inside a list item. Other code that assumes inline arrays + are 1-D (e.g. `len(shape) == 1` checks) should be audited, not just the role + chain. + +## Checklist after the devtools change lands + +1. Bump or refresh `modflow-devtools`, then regenerate (`make.py`). +2. Switch `is_cellid` to `Array.cellid` (required, not optional), and drop the + stale `getattr`/comments. +3. Fix the `make.py` role chain for `cellid` arrays (count from `shape[1:]`). +4. Diff the regenerated packages. Only `gwf/gnc.py`, and the exchange classes + if generated, should change. Anything else is unexpected. +5. Run the codec/codegen tests, especially GNC. +6. Drop the locally patched spec. diff --git a/docs/dev/plans/archive/child-fields-plan.md b/docs/dev/plans/archive/child-fields-plan.md new file mode 100644 index 00000000..3e11398f --- /dev/null +++ b/docs/dev/plans/archive/child-fields-plan.md @@ -0,0 +1,328 @@ +# Child fields from DFN component links + +Replace codegen's hardcoded `BINDINGS` and `SUBPACKAGES` tables +(`flopy4/mf6/utils/codegen/make.py`) with one rule driven by the DFN, and +`spec.subpackage()` with one `spec.child()` decorator. Builds on +`../modflow-devtools/subpackage-links-plan.md` and supersedes parts of +[`package-binding-plan.md`](package-binding-plan.md) (see "Relation to other plans"). + +## Background + +Some fields name the input file of another component. There are three +syntaxes, but one concept: + +| Case | Syntax | Target | +|---|---|---| +| Subpackage (`ncf`, `ts`, `obs`, `tvk`, ...) | record `NCF6 FILEIN ` | fixed (`utl-ncf`) | +| Simulation TDIS | tagged scalar `TDIS6 ` | fixed (`sim-tdis`) | +| Name-file rows: models, exchanges, solutions, model packages | list row ` [terms]` | picked by the row's type column | + +flopy4 holds the child component instead of the filename, and generates the +record or row from it. + +- **Contexts** (simulation, models): already work this way at runtime. + - Egress: `_make_binding_blocks`, `converter/egress/unstructure.py:35`. + - Ingress: `_resolve_bindings`, `converter/ingress/structure.py:454`. + - `converter/binding.py:Binding`. +- **Packages:** `subpackage(file_field=...)`, `spec.py:166`, emits a child + next to a path field (`ncf` beside `ncf_file`). + - `Package._sync_subpackage_files` (`package.py:360`) copies the child's + filename into the path field. + - Nothing loads the child. + +The dev3 DFNs don't say which fields are links, so codegen hardcodes them: + +- `SUBPACKAGES`: record name → component. Only `ncf_filerecord` today. +- `BINDINGS`: sim-nam field → `ChildField(py_name, type, factory, imports, + converter)` for `tdis6`, `models`, `exchanges`, `solutiongroup`. + +In dev3, the filename columns are plain `String`: `tdis6`, `mfname`, +`exgfile`, `slnfname`, `*-nam` `fname`. The file record filenames are +`File(direction="in")` with no link. + +The DFN already has everything else needed to pick the targets: + +| Components | Parent | Type / subtype | +|---|---|---| +| `{gwf,gwt,gwe,prt,chf,olf}-nam` | sim-nam | type `model` | +| `exg-*` | sim-nam | subtype `exchange` | +| `sln-ims`, `sln-ems` | sim-nam | subtype `solution` | +| `sim-tdis` | sim-nam | — | + +## Decisions already made + +- **Follow the DFN for names.** No `solutiongroup` → `solutions` override: + the field becomes `Simulation.solutiongroup`. This is a breaking rename; + update its callers (see step 3). +- **Fixed-target names come from the target's module name** + (`filters.module_name`): `sim-tdis` → `tdis`, `utl-ncf` → `ncf`, + `utl-ts` → `ts`. +- **Conversions live on the child class, not in codegen.** + - `convert_time` (`simulation_methods.py`) becomes `Tdis.coerce(Time)`. + - The four copies of `convert_grid` (in `gwf`, `gwt`, `gwe` and `prt` + `__init__.py`) become `Dis.coerce` / `Disv.coerce` (from a grid). + - `from_time` and `from_grid` already live on those classes. +- **The child is the source of truth for the filename.** A linked field is + replaced by the child field, with no parallel path field, the same way + binding lists already work. + - The path field stays only when the target has no generated flopy4 class + (e.g. `obs_filerecord` until `utl-obs` is generated). + - So "point at a file flopy4 doesn't manage" still works for those. + +## DFN side (devtools, done by the user; do not implement here) + +Add to `File` in `modflow_devtools/dfns/schema.py`: + +```python +class File(InputFieldBase): + direction: Literal["in", "out"] + component: str | list[str] | None = None # component(s) this file is input for + component_ref: str | None = None # sibling field naming the instance's type +``` + +**`component`** is a selector in the vocabulary of `parent`: + +- It takes concrete names or type names, extended to subtypes (`exchange`, + `solution`). +- It matches only components that can be children of the field's own + component. So `"package"` in `gwf-nam` means gwf-nam's packages. + +**`component_ref`** works like `fk_ref`: it names a sibling column whose +value (the ftype token) picks the concrete component. Without it, the +selector must resolve to exactly one component. + +**Annotations:** + +| Field | `component` | `component_ref` | +|---|---|---| +| `sim-nam.timing.tdis6` | `sim-tdis` | — | +| `sim-nam.models.mfname` | `model` | `mtype` | +| `sim-nam.exchanges.exgfile` | `exchange` | `exgtype` | +| `sim-nam.solutiongroup.slnfname` | `solution` | `slntype` | +| `*-nam.packages.fname` | `package` | `ftype` | +| `*_filerecord` filenames | from legacy `# flopy subpackage` lines | — | + +**Migration:** + +- Retype the five name-file filename columns from `String` to + `File(direction="in")`. +- Subpackage links come from the legacy annotations, as in + `subpackage-links-plan.md`. +- Name-file links come from a small table in `migrate_to_v2_0_0_dev2.py`, + next to `_OC_RTYPE_VALID`. + +**Also:** + +- `tdis6` should be required: mf6 errors with "TIMING block variable TDIS6 + is unset" (`SimulationCreate.f90:191`), but the DFN marks it `optional`. +- Optional: mark `mname` as pk, and `exgmnamea`, `exgmnameb` and + `slnmnames` as fk to `models.mname`. + +## flopy4 side + +### 1. `spec.child()` + +In `flopy4/mf6/spec.py`, replace `subpackage()` with: + +```python +def child(block: str | None = None, default=None, converter=None): + return attrs.field( + default=default, + converter=converter or coerce_child, + metadata={"child": True, "block": block} if block else {"child": True}, + ) +``` + +**`coerce_child`** passes `None` and component instances through. Otherwise +it tries each candidate class's `coerce` classmethod, in the attribute's +type order. It needs the field's type, so do it as an attrs `field_transformer`, +or have it resolve candidates lazily via `child_field_candidates`. Pick +whichever is simpler. + +- Values are bare, list or dict; it coerces each element. +- It raises a `TypeError` naming the accepted types. + +**Children are still detected by type.** `child_field_candidates`, used +across `component.py`, `unstructure.py` and `structure.py`, keeps doing +that. `metadata["child"]` makes it explicit for codegen and readers. Don't +rewrite the tree code to key on it unless it simplifies something. + +**Migrate the hand-written uses:** + +- Model package fields: `field(block="packages", ...)` in `gwf`, `gwt`, + `gwe` and `prt` `__init__.py` become `child(block="packages", ...)`. +- Delete `convert_grid` in favour of `coerce`. +- Add `Tdis.coerce` in `tdis_methods.py`, and `Dis.coerce` / `Disv.coerce` + on their method mixins. Generated classes get methods only through + mixins (`MIXINS` in make.py), so check that DIS and DISV have one. + +### 2. Loading and writing package-level children + +Dropping the path field means a package's linked children have to load, or +loading a DIS with `NCF6 FILEIN` loses the reference. + +- **Load:** in `structure_component`, for each record replaced by a child, + load the file as the child's class and set `child.filename`. Reuse + `_resolve_bindings`'s pieces rather than adding a second loader: + - class lookup by ftype, including G/A-variant disambiguation; + - `Package.load(..., dims=..., name=...)`; + - workspace handling. + - Paths are relative to the simulation workspace, not the parent file's + directory ([`package-binding-plan.md`](package-binding-plan.md) §1). +- **Write:** emit ` FILEIN ` from the child. The + keyword and direction come from the DFN record. The path field's current + `_keyword` / `direction` metadata (`spec.path`) moves onto the child + field's metadata. + - Delete `_sync_subpackage_files` and the `file_field` metadata. +- **Repeating records** (`ts_filerecord`, `tas_filerecord`) give + `list[Ts]`. Each child gets its own default filename + ([`package-binding-plan.md`](package-binding-plan.md) §2). + +### 3. Codegen + +In `make.py`, replace the following with one link-driven pass: + +- `BINDINGS`, `ChildField`, `_child_field_spec`; +- `SUBPACKAGES`, `_subpackage_field_spec`; +- the `skip=frozenset(bindings)` plumbing. + +**Overlay until devtools lands.** The installed devtools `File` has no +`component` field, and pydantic may reject unknown attributes. So add a +flopy4-side overlay, plus one accessor that reads the DFN first and falls +back to it: + +```python +# (component, dotted field path) -> (component selector, component_ref) +# Remove once the dev3 DFNs carry File.component (modflow-devtools). +LINKS: dict[tuple[str, str], tuple[str | list[str], str | None]] = { + ("sim-nam", "tdis6"): ("sim-tdis", None), + ("sim-nam", "models.mfname"): ("model", "mtype"), + ("sim-nam", "exchanges.exgfile"): ("exchange", "exgtype"), + ("sim-nam", "solutiongroup.slnfname"): ("solution", "slntype"), + ("*", "ncf_filerecord.ncf6_filename"): ("utl-ncf", None), +} + +def field_link(component, path, field): ... +``` + +`field_link` checks `getattr(field, "component", None)` first. It must not +care whether the field is `String` or `File`: in dev3 today, the +name-file columns are `String`. + +**Selector resolution:** + +- Candidates are components that match the selector by name, type or + subtype, and whose `parent` includes this component. +- Error if it resolves to nothing. +- Error if a link without `component_ref` resolves to more than one. +- Put the check in `check_mixins`, or rename that to `check_tables`. + +**Emit rules** (the field is the one holding the link): + +| Shape | Emitted field | +|---|---| +| Fixed target, scalar or non-repeating record | `: Optional[X] = child(block=...)` | +| Fixed target, repeating record | `: list[X] = child(block=..., default=Factory(list))` | +| Fixed target, required field | `: X = child(block=..., default=Factory(X))` | +| `component_ref` list whose candidates share a flopy4 base (Model, Exchange, Solution) | `: dict[str, Base] = child(block=..., default=Factory(dict))` | +| `component_ref` list of packages | fan out, one field per package (see below) | + +- The "required" row is `tdis` once `tdis6` is required. In the overlay, + record `tdis6` as required: until devtools fixes it, an overlay entry + may override `optional`. +- Choose the base class with `_base_class` logic. Import the base from + `flopy4.mf6.{model,exchange,solution}`. +- The whole list block (or record) is replaced by the child field. Its + other columns (`mtype`, `mname`, `exgmnamea`, ...) are derived from the + child by `Binding.from_component`. So skip the list block property and + its column attributes. +- **Package fan-out** (model name files) is not needed until gwf-nam etc. + are generated. Implement it then. Known wrinkles: + - DIS/DISV/DISU share one field `dis` (one grid per model). + - G/A variants share an ftype and become `list[Union[Chd, Chdg]]`. + - `multi` decides `list` vs `Optional`. + - Mirror the hand-written `Gwf` (`gwf/__init__.py:183`). + +**Imports:** collect child classes' imports the way `extra_imports` does +today. Use `_component_module` (`make.py:1035`) for the module path. + +**Regenerate**, then check the generated `Simulation`: + +- it has `tdis`, `models`, `exchanges`, `solutiongroup`; +- `dis.py` / `disv.py` (gwf, gwt, gwe, prt) have + `ncf: Optional[Ncf] = child(...)` and no `ncf_file`. + +**Rename `solutions` → `solutiongroup`.** + +- About 47 references under `flopy4/`, `test/`, `docs/examples` and + `docs/profile`. +- Find them with: + `grep -rn '\.solutions\b\|solutions=' --include='*.py' --include='*.ipynb'`. +- The `.ipynb` files in `docs/examples` pair with the `.py` files. Check + whether they're jupytext-synced before editing both. +- `Solution` objects are keyed by name in a dict; only the attribute name + changes. + +### 4. Tests + +- **Codegen:** + - `test_simulation_spec` (`test/mf6/test_mf6_codegen.py`) expects + `solutiongroup`. + - The subpackage test at `test_mf6_codegen.py:827` monkeypatches + `SUBPACKAGES`; switch it to `LINKS`. + - New: an unknown selector, and an ambiguous selector without + `component_ref`, both raise. +- **Coercion:** + - `Simulation(tdis=Time(...))` still works. + - `Gwf(dis=StructuredGrid(...))` still works. + - A wrong type raises `TypeError`. +- **Subpackage round trip:** + - Load a DIS with `NCF6 FILEIN`: `dis.ncf` is loaded and named. + - Write it back and compare the records. + - Use `../modflow6/autotest` or an existing NCF test model, if any. +- **mf6 heads:** use the round-trip script in the session scratchpad, or + the integration tests. Check test001a_Tharmonic, test020_mawconfined, + test032_sfr and test028_sfr_rewet: heads must be identical. + +## Order + +1. `child()` + `coerce` (`Tdis`, `Dis`, `Disv`), hand-written models + switched, `convert_grid`/`convert_time` deleted. No codegen change yet; + the smoke suite passes. +2. Codegen: `LINKS` overlay + `field_link` + emit rules. Delete `BINDINGS`, + `ChildField`, `SUBPACKAGES`. Regenerate. `solutions` → `solutiongroup` + rename. +3. Package-level child load/write (§2), with NCF. Drop the `ncf_file` path + fields. +4. Extend `LINKS` to `ts_filerecord` and `tas_filerecord` (list children). +5. Later: devtools lands `File.component` → delete `LINKS`. Package fan-out + when model name files are generated. + +Steps 1–2 make a PR; 3–4 make another. + +## Relation to other plans + +- [`package-binding-plan.md`](package-binding-plan.md): + - §1 (loading), §2 (list children) and §5 (write) still apply. Here they + are step 3–4. + - §3: this plan picks option (b), deriving the path from the child, for + linked records whose target has a flopy4 class. Unlinked records keep + their path field. + - §4's table: becomes `LINKS` entries, later `File.component`. +- `../modflow-devtools/subpackage-links-plan.md`: this plan extends its + `File.component` with selectors and `component_ref`. Update that plan + rather than writing a second devtools one. + +## Working notes + +- Use pixi for everything: `pixi run -e dev `. Not uv or `.venv`. +- **Regenerate:** + `pixi run -e dev python -W ignore -m flopy4.mf6.utils.codegen.dfn2py -d ../modflow6/doc/mf6io/mf6ivar/dfn --existing-only` + - Codegen must be idempotent: a second run leaves no diff. +- **Smoke:** + `MPLBACKEND=Agg pixi run -e dev pytest -q -n auto --smoke -p no:cacheprovider` + - Run only the smoke suite; CI runs the full suite. +- **Lint:** `pixi run -e dev ruff check . && pixi run -e dev ruff format --check . && pixi run -e dev mypy flopy4` +- Keep comments light. +- Don't commit `.vscode/settings.json`. +- Never push, pull or comment on GitHub without an explicit request. diff --git a/docs/dev/plans/archive/class-driven-reader-plan.md b/docs/dev/plans/archive/class-driven-reader-plan.md new file mode 100644 index 00000000..67dbfe83 --- /dev/null +++ b/docs/dev/plans/archive/class-driven-reader-plan.md @@ -0,0 +1,191 @@ +# Class-driven reader: drop the runtime DFN dependency + +**Archived 2026-10-19.** Done: steps 1-3 merged (#455 dropped `flopy4/mf6/dfns/` and the reader's DFN dependency); step 4 deferred as not needed. The devtools items are tracked in that repo. + +## Goal + +Make the generated classes (and grammars) the only runtime artifacts of +`flopy4 mf6 sync`, so flopy4 doesn't ship `flopy4/mf6/dfns/`, a `__dfn__` +attribute, or read the devtools cache at runtime. As it turned out (see +Steps), the typed reader gets what it needs from the grammars, not the +classes. + +Provenance becomes `DFN_COMMIT` (plus the developmode flag, and optionally a +spec digest) in `_contract.py`. mf6 stays the owner of the spec. What flopy4 +ships is a generated projection of it, like the `.lark` grammars, never a +hand-edited copy. + +## Background + +- `flopy4/mf6/dfns/` holds the spec so the typed reader agrees with the + classes. That fixed the drift from reading the devtools cache at runtime + (#429), but it ships a copy of the spec and churns about 129 files on + every develop sync. +- `../modflow-devtools/dfn-cache-for-flopy4-plan.md` proposed making the + devtools cache fit for runtime use: content-addressed entries, `refs/` + pointers, `put()` for local checkouts, and so on. If the runtime stops + needing DFNs, most of that has no consumer. See "devtools" below. +- The writer (`flopy4/mf6/codec/writer`) imports no DFNs. It runs from the + classes. The reader is the only part that's spec-driven. + +## What the reader reads from the DFN today + +From `flopy4/mf6/codec/reader/transformer/typed.py` (via +`codec/reader/dfns.py:get_component_dfn`): + +- `dfn.blocks`, and `get_fields(recurse=True)` as `_flat_fields` (L29-32) +- block headers (L249-290): + - record headers (OBS `CONTINUOUS FILEOUT [BINARY]`), using + `Keyword` vs `File`, `mode_keyword` and `direction` + - index type (`Double` for `utl-tas` time blocks) +- list element names, and the arms of union items (`_list_elements`, and the + `ocsetting_all` branch around L326) +- each field's type, used to convert tokens (Keyword → `True`, and so on) +- hyphen vs underscore name matching + +## The classes already carry almost all of it + +When the type hints are accurate, the annotation plus existing metadata +identifies the DFN type: + +| Class side | DFN side | +|---|---| +| `bool` | keyword | +| `int` (`index=True` marks an index) | integer | +| `float` | double | +| `str` | string | +| `Path`, via `path(direction=, keyword=)` | file (`FILEIN`/`FILEOUT` from `direction`; the writer already does this in `item.py:511`) | +| `tuple` + `cellid=` | cellid array | +| `Union[float, str]` + `time_series=True` | time-series double | +| `Union[...]` + `union=("cellid", "index", "string")` | untagged union (OBS `id`/`id2`) | +| arm `Item` classes with `_keyword` and no fields (`Oc.All`, `Oc.First`, ...) | keyword arms of a union | +| `dict[int, ...]` vs `dict[float, ...]` on a block field | block index type (`PERIOD` vs `utl-tas` `TIME`) | +| `_header: ClassVar[str]` on a `Block` (`Obs.ContinuousBlock`) | header record | +| `block=`, `fill_forward=`, `shape=`, `dim=` | block membership, period fill-forward, input dims | + +**Principle:** the reader should infer from the classes using the same +conventions the writer uses to emit from them. Format rules live once, in +`Item` / `Record` / `Block` (`Item.from_tokens` already parses union columns +this way). Add metadata to the `field()` / `path()` / `child()` vocabulary +only when the writer needs it too. Don't add any for the reader alone. + +## The hints that are wrong + +The annotations are only ambiguous in two places. Both are codegen bugs: + +1. **Untagged strings are widened to `Union[float, str]`.** + `codegen/filters.py:850-852`, `value_column`: + ```python + object_dtype = object_dtype or time_series + py_type = "Union[float, str]" if object_dtype else _DFN_PY.get(dfn_type, "float") + ``` + and `make.py` (around L300) passes `object_dtype=isinstance(f, String)`. + This confuses the storage dtype (`np.object_` columns) with the type of + a value. It hits every untagged string column in list items: OBS `obsname` + and `obstype` (`utl/obs.py`), OC `rtype` (`gwf/oc.py`), and others. + `boundname` avoids it only because it's special-cased. The line was last + touched in ee0f593 (#414), and nothing explains the string case. + **Fix:** plain strings become `str`. Time-series columns stay + `Union[float, str]`. +2. **Keywords inside records become `Optional[str]` + `tagged=True`.** + `make.py`, the `col.is_row_keyword` branch: + `attr_column(name, "str", {"tagged": True}, ...)`. OBS `binary` is the + example. That looks the same as a tagged string field. Option keywords + already become `bool`. **Fix:** record keywords become `bool` too. + +Expect some fallout from both: item conversion, readers or tests that relied +on the wide hint (numeric-looking obs names, for example), and how the writer +emits record keywords. + +## Scope of invertibility + +The target is `invert(cls) == normalize(dfn)`, restricted to flopy4's scope: + +- **In scope:** + - I/O structure (everything in the table above); + - input dims (`shape=`, `dim=`); + - developmode fields, when codegen runs with `--developmode` (already a + flag: `make.py:1939`, `dfn2py.py`). Record the flag in `_contract.py` + and run the check under the same filter. +- **Descriptions** go into docstrings. The DFN text has markup + (``` ``BINARY'' ```, `\texttt`), so a docstring is a rendered form. + Compare normalized text, or leave descriptions out of the check. Don't + parse docstrings back. +- **Out of scope:** the memory section and runtime dims. Those are for + modflowapi, which should consume the devtools spec directly. +- **Record shapes aren't encoded.** Each reshaping codegen does is a named, + deterministic rule: filerecord → `child(keyword=...)`, cellid array → + `cellid=True`, `iper` header → `fill_forward`, `maxbound` → computed + property. `normalize(Component)` applies the same rules to the DFN side. + If a rule isn't injective (two DFN shapes collapse into one class form), the + check fails. Fix it case by case, and only if mf6 reads the two shapes + differently. + +## Steps + +Status as of `grammar-driven-reader` (3c7dbe9, 5e5e47c) and +`fix-codegen-hints` (1faebc1): + +1. **Fix the hints in codegen.** Done (1faebc1). LAK `bedleak` and MAW + `head_limit` are `string` in the DFN but hold a number or a word, so + `str` fields accept numbers for now. See + `../modflow-devtools/number-or-word-unions-plan.md`. +2. **Make the typed reader DFN-free.** Done, but not as first planned + (3c7dbe9). The transformer only needed DFN *syntax* (record part names, + keyword vs value, union arms, record headers, a float block index), and + the generated grammars already encode that. So every generated rule is + now named for what it is (`grammar/names.py`), and the transformer reads + the parse tree alone. `codec/reader/dfns.py` and `dfn_path` are gone. + The classes don't need to encode record shapes; the grammar keeps them. + Across the model corpus the output matches the old reader's, apart from + bugs fixed in the old flat DFN lookup (MAW/LAK/SFR `auxiliary`, union + arms with hyphens). +3. **Drop `flopy4/mf6/dfns/`.** Done, merged in #455. Sync stops storing + DFNs and removes the copy earlier syncs left. It also regenerates the + `.lark` grammars in its staged swap, building each one first and + removing stale ones. +4. **The `invert(cls) == normalize(dfn)` check.** Deferred, probably not + needed. With the reader grammar-driven, nothing at runtime relies on the + classes being invertible, and `structure_component` (already + class-driven) is covered by the load and round-trip tests. Worth adding + only if codegen starts losing information those tests can't catch. + +Optional later step: if `mf6 --spec` lands upstream, use it to check that the +user's mf6 binary matches the commit the classes came from. The binary +already reports `+`. This checks compatibility with the installed mf6, +not reader correctness. + +## devtools + +The devtools cache becomes a **dev-time cache of upstream data**: + +- `flopy4 mf6 sync` and codegen read it, run by developers and CI. +- It holds what mf6 publishes (a release's or the nightly's DFN zip), keyed + by `source@tag`, as in the original DFNs API design. +- Someone who installs a flopy4 wheel never touches it. + +So `../modflow-devtools/dfn-cache-for-flopy4-plan.md` shrinks to these items: + +- atomic, verified `sync()`: + - an asset-name template per source in the bootstrap `dfns.toml` + (`mf{tag}_dfns.zip`); + - extract into a temp dir under a `FileLock`, reusing the pattern from + models #364; + - a manifest file marks an entry as complete; + - note that `os.replace` onto an existing non-empty directory fails; +- a manifest recording source, tag, asset sha256, the tag's commit (so flopy4 + can drop `_remote_commit`), devtools version and fetch time; +- offline fallback for `latest`, using `cached_tag()` with a warning; +- optionally, `Dfns.digest()`. It must be canonical (`exclude_defaults`, + sorted keys, fixed JSON separators). It depends on the devtools version, + because migration runs at load time and `schema_version` doesn't change + with each migration fix. + +Dropped: content-addressed entries, `refs/` pointers, `put()` for local +checkouts, and a `develop` → nightly alias. Use +`modflow6-nightly-build@latest`, or `LocalDfnRegistry` on a checkout when +you need HEAD. A single-file `Dfns.dump` / `load` is nice to have now, +because there's no stored copy to diff. + +Upstream asks stay the same: a manifest inside the DFN zip that names the +commit, a published sha256, and later `mf6 --spec`. diff --git a/docs/dev/plans/archive/codegen-base-class-removal-plan.md b/docs/dev/plans/archive/codegen-base-class-removal-plan.md new file mode 100644 index 00000000..ee20a376 --- /dev/null +++ b/docs/dev/plans/archive/codegen-base-class-removal-plan.md @@ -0,0 +1,414 @@ +# Removing hand-written base classes from TDIS and NCF + +Follow-up to the TDIS list-shape refactor (#379). + +**Status:** implemented on branch `codegen-mixins`. The design below is +what was planned. Where the implementation departs from it, see +[Departures and follow-ups](#departures-and-follow-ups). + +## Resume here + +**Branch state:** `codegen-mixins`, off develop at `957ee21`. All changes +are staged; nothing is committed or pushed. +- Verified: `pixi run -e dev smoke` gives 1407 passed, 16 skipped, 10 + xfailed. `ruff check`, `ruff format` and `mypy flopy4` are clean. +- The full suite was last run before the final mypy/`to_time` fixes; the + only failures then were those two, since fixed. Rerun it before + committing. + +**Next steps, in order:** +1. Commit this branch (after the user reviews), then open a PR only when + asked. +2. ~~`layered` fix (departure 8).~~ Done on this branch (unstaged; see + departure 8 for what changed). +3. The devtools list-shape work (`../modflow-devtools/list-shape-bounds-plan.md`), + then the flopy4 follow-up in departure 2 (`<` bounds, exact checks, + drop the `list_col_dim` fallback). +4. Leftovers: pyright check of the mixin `self` types (departure 1); + confirm the one-row default-rows interpretation (departure 3). + +**Regeneration gotcha:** codegen imports `flopy4`, so if generated modules +use a `spec.field()` keyword that no longer exists (as with `auto_from=`), +or import a deleted module, the import fails before generation runs. Fix +the generated files by hand first (e.g. `sed` out the stale keyword), or +temporarily stub the broken module with a first line of +`# autogenerated file, do not modify`, which `--existing-only` needs to +regenerate a file. Then run: +`pixi run -e dev python -W ignore -m flopy4.mf6.utils.codegen.dfn2py -d ../modflow6/doc/mf6io/mf6ivar/dfn --existing-only` + +## Goal + +Generated component classes should carry no hand-written structure: no field +overrides, no `__attrs_post_init__` logic, no special-cased base classes. Pure +behavior (factories, conversions) may stay on the classes as methods, but comes +from method-only mixins declared in codegen config, not from base classes that +codegen picks by name. + +## Current state + +- `flopy4/mf6/tdis_base.py` (`TdisBase`), chosen by codegen's `_base_class()` for `sim-tdis`. It: + 1. fills default `perioddata` (`nper` rows of `(1.0, 1, 1.0)`) + 2. derives `nper` from the row count, raising if an explicit `nper` other than 1 disagrees + 3. converts a `datetime` `start_date_time` to an ISO string + 4. exposes read-only `perlen` / `nstp` / `tsmult` column properties + 5. provides `get_dims`, `to_time`, `from_time`, `to_xarray` and `from_timestamps` +- `flopy4/mf6/utl/ncf_base.py` (`NcfBase`) overrides `wkt` / `crs_wkt` field + types. `utl-ncf` is still in the codegen skip list. The concrete + `flopy4/mf6/utl/ncf.py` is hand-written and adds `Ncf.from_grid()`. + +## Findings + +- **The NCF override is obsolete.** Codegen on the dev3 schema already emits + `wkt: Optional[str]` and `crs_wkt: Optional[str]`, verified by generating + `utl-ncf` into a scratch dir. Apart from `from_grid()`, the generated module + matches the hand-written one. It adds `longname`s and uses `FloatArrayLike` + instead of `NDArray` for latitude and longitude. +- **Nothing in the library needs the `perlen` / `nstp` / `tsmult` properties.** Their only readers are + `to_time()`, `to_xarray()` and tests (`test_mf6_component.py:511-513`, + `test_tdis_round_trip`, `test_tdis_period_columns_are_views`). Callers who + want arrays can use `to_time()`. Keeping the properties keeps the array API + on the class, which the refactor set out to remove. +- **Methods don't require base classes.** An instance method gets `self` and a + classmethod gets `cls`, so the module that implements them never imports the + generated class. There is no circular import. + +## Design: method-only mixins from codegen config + +Replace the name-based `_base_class()` special cases with a config table in +codegen: + +```python +# flopy4/mf6/utils/codegen/make.py (or a config module) +_MIXINS = { + "sim-tdis": ["flopy4.mf6.tdis_methods:TdisMethods"], + "utl-ncf": ["flopy4.mf6.utl.ncf_methods:NcfMethods"], +} +``` + +The template emits `class Tdis(TdisMethods, Package):` plus the imports. Rules for mixins: + +- methods only: no attrs fields, no `__attrs_post_init__`, no field overrides +- never redeclare fields. Annotate `self` / `cls` with the generated class, + imported under `TYPE_CHECKING`: + + ```python + if TYPE_CHECKING: + from flopy4.mf6.tdis import Tdis + + class TdisMethods: + def to_time(self: "Tdis") -> Time: ... + + @classmethod + def from_time(cls: type["Tdis"], time: Time) -> "Tdis": ... + ``` + + The DFN stays the only source of field names and types. (`TdisBase` + redeclared `time_units`, `start_date_time`, `nper` and `perioddata` under + `TYPE_CHECKING`, a second copy that could drift silently.) + + mypy flags the annotation itself; see departure 1. + +### Where DFN knowledge lives + +- **MF6 input semantics** (field types, defaults, list-block dimensions) come + from the DFNs only, through codegen features (see TDIS item 3). Mixins must + not restate any of it. +- **flopy API conveniences** (`from_grid`, `to_time`, ...) convert to and from + flopy types (`Grid`, `Time`) that MF6 knows nothing about, so they don't + belong in the DFNs. `_MIXINS` records only "this component also gets these + flopy methods". That knowledge belongs to flopy, and keeping it in one table + is better than the name checks scattered through `_base_class()` today. +- Conversion methods **use** DFN-derived field names; that coupling is the + point of a conversion. What's ruled out is **restating** those names or + types anywhere else. + +### Drift checks + +- Codegen fails if a `_MIXINS` key matches no loaded DFN. +- mypy runs over the mixins using the annotated `self`, so a DFN field rename + that breaks a mixin fails in CI. +- Round-trip tests (`Tdis` ↔ `Time`, `Ncf.from_grid`) cover runtime behavior. + +Alternatives considered: +- **Bind free functions in the generated class body** + (`from_grid = classmethod(ncf_from_grid)`). Works, but type checkers and IDEs + handle `classmethod()` poorly. +- **Attach methods after import** (`Ncf.from_grid = ...` in an `__init__`). + Hidden and invisible to type checkers. Rejected. +- **Find mixins by convention** (e.g. use `NcfMethods` automatically if + `utl/ncf_methods.py` defines it). Removes component names from codegen, but + a file's presence would silently change the generated class. Rejected in + favor of the explicit table plus drift checks. +- **Generic name-matching converter for TDIS** (`Time` and the DFN share + `nper`, `perlen`, `nstp`, `tsmult`, `time_units`, `start_date_time`). It + relies on a naming coincidence, only TDIS benefits, and NCF has no such + correspondence. Rejected. + +The `Solution` base class could move into the same mechanism later. It is a +real base class, though, with behavior of its own, so that can wait. + +## Work items + +### Codegen +1. Add the `_MIXINS` table and template support for extra bases and their + imports. Fail on keys that match no DFN. +2. Drop the `sim-tdis` case from `_base_class()`. +3. Test: the generated class has the mixin in its MRO, and an unknown + `_MIXINS` key raises. + +### NCF +1. Remove `utl-ncf` from `_SKIP` in `flopy4/mf6/utils/codegen/dfn2py.py`. +2. Move `from_grid()` into `NcfMethods`, a method-only mixin with + `cls: type["Ncf"]`, and delete `ncf_base.py`. +3. Regenerate `flopy4/mf6/utl/ncf.py`. +4. Check that the `FloatArrayLike` latitude/longitude typing still round-trips + through NetCDF output (`test_mf6_component.py` NCF tests, `netcdf.py`). + +### TDIS +1. Remove the `perlen` / `nstp` / `tsmult` properties. Make `to_time()` and + `to_xarray()` read the rows directly, and update the three tests to use + `to_time()` or `perioddata`. +2. Move the `Time` conversions into `flopy4/mf6/utils/time.py` + (`Time.from_tdis(tdis)`, `Time.to_tdis()`). `TdisMethods` keeps thin + `from_time` / `to_time` / `from_timestamps` / `to_xarray` wrappers so + `Tdis.from_time(...)` in `Simulation` and the docs examples keeps working. +3. Replace the remaining `TdisBase` setup logic with general codegen features: + - **default fill:** emit the DFN `default_value` for list blocks; it is + currently ignored. `((1.0, 1, 1.0),)` becomes + `attrs.Factory(lambda: [Tdis.Perioddata(1.0, 1, 1.0)])`. + - **`nper` from the row count:** `Package._init_item_lists` sets + `ns` by naming convention, which gives `nperioddatas` for TDIS. + Read which dimension a list block uses from the DFN instead, if dev3 + exposes it (check `modflow_devtools.dfns.schema`). The fallback is a + computed property like `maxbound`, but that breaks `Tdis(nper=5)`, which + `test_mf6_dimensions.py` uses. + - **datetime:** add a datetime field type to the DFN schema (suggested in + #378), or drop the conversion and accept only strings, with `from_time()` + converting. + - **`get_dims`:** give `Package` a default implementation that returns the + integer fields of its dimensions block. Only TDIS and the DIS packages + implement it today. +4. Delete `tdis_base.py`. Add a `_MIXINS` entry for `TdisMethods`, with + `self: "Tdis"` / `cls: type["Tdis"]` annotations and no redeclared fields. + Update `test_tdis_generates_on_tdis_base`. + +## Departures and follow-ups + +Where the implementation differs from the plan, and what is still open. + +### 1. mypy flags `self: "Tdis"` on a mixin + +The plan assumed mypy accepts a self type that is a subclass of the +enclosing class. It doesn't: it reports `The erased type of self "Tdis" is +not a supertype of its class "TdisMethods" [misc]` on each such signature. +It accepts only supertypes of the class, or a Protocol. + +What was done: one `# type: ignore[misc]` on each mixin method's `def` line +(5 in `TdisMethods`, 1 in `NcfMethods`). The ignore covers only that +diagnostic on the signature: + +- Method bodies are still checked against the generated class. Renaming + `self.nper` to `self.nperx` fails with `"Tdis" has no attribute "nperx"`. +- Callers are typed correctly. `Tdis.from_time(...)` is revealed as `Tdis`, + `Ncf.from_grid` has its full signature, and `Tdis.from_time(1)` is + rejected. +- `cls(...)` keyword arguments are checked against the attrs `__init__`. + +#### Mixin vs Protocol + +Tested with mypy (scratch module, attrs classes): + +| | mixin, `self: "Tdis"` (current) | mixin, Protocol `self` | +|---|---|---| +| diagnostic on the declaration | `[misc]` on each `def`, ignored | none | +| fields restated | no | yes: every field the mixins use, with types | +| drift in a mixin body (field renamed) | error in the mixin body | error only at call sites, as `Invalid self argument "Tdis" ...`; missed if no checked code calls the method | +| one mixin method calling another (`to_xarray` → `self.to_time()`) | works | error (`"_Fields" has no attribute "to_time"`) unless the method is also declared on the Protocol | +| factory `cls(...)` (`from_time`, `from_timestamps`, `from_grid`) | keyword arguments checked against the attrs `__init__` | error: a Protocol has no `__init__` (`Unexpected keyword argument "nper" for "_Fields"`). Needs an ignore, or `__init__` restated on the Protocol | +| caller-side types (`Tdis.from_time(...)` → `Tdis`) | correct | correct with `cls: type[T]`, `T` bound to the Protocol | +| IDE completion on `self.` in the mixin | from the real class | from the Protocol | + +Callers get correct type hints either way. The Protocol removes the six +`[misc]` ignores, but needs a hand-maintained field list, needs ignores on +the three factories anyway, and weakens drift detection. **Recommendation: +keep the mixin + `self: "Tdis"`.** + +If the ignores need to go, have **codegen emit the Protocol** (`TdisFields` +in `tdis.py`, from the same DFN fields) and import it into the mixin under +`TYPE_CHECKING`. That removes the hand-maintained field list, but the +factory and method-to-method problems above remain. + +Other alternatives, not taken: +- `class TdisMethods(Tdis if TYPE_CHECKING else object)`. This is circular, + because `tdis.py` imports the mixin. +- Turning off `misc` for the mixin modules in the mypy config. That is + broader than needed. + +Open: pyright/basedpyright (not installed in the dev environment) may treat +`self: "Tdis"` differently. If IDE users run Pylance, check it and add +pyright-specific ignores if needed. + +### 2. Row-count dimension: when explicit values are validated + +TDIS item 3 is implemented as a general `Package` feature rather than +TDIS-specific setup: + +- Codegen already worked out which DIMENSIONS field counts each list + block's rows (`_build_block_property_specs`; `perioddata` → `nper` + because it is the only field in the dimensions block). It used to emit + only `auto_from=`, a flag that nothing read. It now emits + `dim=""`. `Package._init_item_lists` also used to set + `ns`, a naming guess that matched no real field. That is removed. +- A dimension resolved for more than one block (MAW's `nmawwells` for + packagedata, connectiondata and angledata; SFR's `nreaches`) doesn't + count any one block's rows, so it gets no `dim`. +- 11 generated packages get a link now: TDIS, BUY, CSUB, LAK (×3), MVR, + VSC, PRT-OC, PRP (×2), ATS, LAKTAB and SFRTAB. For all of them except + TDIS this is new behavior: constructing with rows fills in the dimension + (e.g. `nlakes`). + +`Package._set_dim_from_rows(dim, nrows)`: + +| dimension value | result | +|------------------------------------|----------------------------| +| `None`, `0` or the field's default | set to `nrows` | +| explicit and `< nrows` | `ValueError` | +| explicit and `>= nrows` | kept, no error | + +`TdisBase` raised on any mismatch. Now `Tdis(nper=3, perioddata=[one +row])` is accepted silently and writes `NPER 3` with one PERIODDATA row, +which MF6 rejects at run time. Too few rows can't be an error in general: +some linked dimensions are upper bounds, where fewer rows is valid input +(ATS `maxats`: the Fortran read loop stops at END). The dev3 schema doesn't +say which kind a dimension is: `dfnspec.md` says a list's `shape` "names a +declared dimension that bounds the list", and neither `List` nor `InputDim` +records whether that bound is exact. The name doesn't say either: MVR +`maxpackages` is exact despite the `max` prefix (`gwf-mvr.f90` errors if +`npak /= maxpackages`). + +Other rough edges: +- The TDIS `nper` default is `1`, so an explicit `nper=1` can't be told + apart from the default. `Tdis(nper=1, perioddata=[3 rows])` becomes + `nper=3` without an error. `TdisBase` behaved the same way. +- The dimension is synced only at construction. Adding rows later doesn't + update it (same as before). + +Plan: **`../modflow-devtools/list-shape-bounds-plan.md`**. In short: +- The legacy `.dfn` format already marks upper bounds with a `<` prefix + (`shape ( 1`, as it already does for DIS +`botm`. The other `layered` hits in `netcdf.py` (around lines 425, 567 and +597) concern the model's mesh type, not field metadata. Test: +`test_layered_griddata_metadata`. Smoke passes (1407 passed). + +Original plan: + +The generated `Ncf.latitude`/`longitude` lost the hand-written +`layered=False`. The DFN and the devtools migration are correct: dev3 +reports `layered: False` for `utl-ncf` latitude and `layered: True` for +`gwf-npf` k. The gap is in flopy4: + +- Codegen emits `layered` only for period `readarray` fields + (`make.py`, `_ra_meta["layered"]`). Static griddata fields get none; + generated `npf.py`'s `k` has no `layered` either. +- `netcdf.py` (`_ArrayInfo`) reads `f.metadata.get("layered", True)`, so a + missing value means layered. That happens to be right for NPF `k` and + wrong for NCF `latitude`/`longitude`. For non-layered fields it controls + whether `ncpl` is normalized to `nodes`. +- The tests pass because NCF latitude/longitude have `netcdf: False` and + never reach that code path. + +No devtools change is needed. Fix, in flopy4: +1. Codegen (`make.py`): emit `layered=` in the field call + for every `Array` field, griddata included, not only period + `readarray` fields. +2. `netcdf.py` `_ArrayInfo`: change `f.metadata.get("layered", True)` to + `False`, or require the key. Check the other `"layered"` reads in + `netcdf.py` (around lines 425, 567 and 597), and any in `spec.py`, + adapters or the codec, for the same default. +3. Hand-written DIS/DISV fields already set `layered=` explicitly; no + change. +4. Regenerate (`dfn2py --existing-only`) and check the diff: `layered=True` + on NPF/STO/IC-style griddata, and `layered=False` on NCF + latitude/longitude and similar. +5. Test: a NetCDF export of a layered griddata field (NPF `k`) still gives + the `nodes` / per-layer layout, and `Ncf.latitude`'s `_ArrayInfo` keeps + its `ncpl` shape. + +### 9. Line count + +Component-specific hand-written code went from 206 lines to 123 (removed +`tdis_base.py`, `ncf_base.py` and the hand-written `ncf.py`; added +`tdis_methods.py` and `ncf_methods.py`). Shared codegen/`Package`/`spec` +code grew by about 45 lines. Generated `ncf.py` grew from 56 to 81 lines +because codegen writes one keyword per line. The net diff is positive; the +gain is that TDIS and NCF fields now come from the DFN only. diff --git a/docs/dev/plans/archive/codegen-comparison.md b/docs/dev/plans/archive/codegen-comparison.md new file mode 100644 index 00000000..6aae68ae --- /dev/null +++ b/docs/dev/plans/archive/codegen-comparison.md @@ -0,0 +1,427 @@ +# Code Generation: attrs vs pydantic + +## The Problem + +You're generating MF6 classes from DFN files. Requirements: +1. **Minimal generated code** - DFN → Python should be simple +2. **No duplicated validation logic** - `structure_array` is generic +3. **Parameterized by metadata** - only dimensions/dtype vary per field + +## Solution Comparison + +### Option 1: attrs with cattrs (Current Approach) + +#### Base Library (not generated) +```python +from cattrs import Converter +from attrs import define, field + +# Generic structuring function (500+ lines, exists once) +def structure_array(value, self_, field, *, dims=None): + # All your complex logic here + ... + +# Helper for generated code +def mf6_array(dims, dtype, block, **kwargs): + """Factory that attaches converter.""" + return field( + metadata={'dims': dims, 'block': block, **kwargs}, + converter=Converter( + structure_array, + takes_self=True, + takes_field=True + ) + ) +``` + +#### Generated Code +```python +# GENERATED from DIS.dfn +@define +class Dis(Package): + nlay: int = field(metadata={'block': 'dimensions'}) + nrow: int = field(metadata={'block': 'dimensions'}) + ncol: int = field(metadata={'block': 'dimensions'}) + + delr: NDArray[np.float64] = mf6_array( + dims=('ncol',), + dtype=np.float64, + block='griddata', + netcdf=True + ) + + delc: NDArray[np.float64] = mf6_array( + dims=('nrow',), + dtype=np.float64, + block='griddata', + netcdf=True + ) + + top: NDArray[np.float64] = mf6_array( + dims=('nrow', 'ncol'), + dtype=np.float64, + block='griddata', + netcdf=True + ) + + botm: NDArray[np.float64] = mf6_array( + dims=('nlay', 'nrow', 'ncol'), + dtype=np.float64, + block='griddata', + netcdf=True + ) +``` + +**Analysis:** +- ✅ Generic `structure_array` logic in one place +- ✅ Helper function (`mf6_array`) minimizes repetition +- ⚠️ Still need to call `mf6_array(...)` for every array field +- ⚠️ Converter attachment is in generated code (minor) +- Lines per class: ~60 (field declarations + converter attachments) + +--- + +### Option 2: pydantic with model_validator (Recommended) + +#### Base Library (not generated) +```python +from pydantic import BaseModel, Field, model_validator +from typing import Annotated + +# Generic structuring function (500+ lines, exists once) +def structure_array_generic(value, field_name, dims, dtype, context): + # All your complex logic here + ... + +# Field factory (just metadata) +def MF6Array(dims, dtype, block, **kwargs): + """Just stores metadata - NO validation logic.""" + return Field(json_schema_extra={ + 'mf6_type': 'array', + 'dims': dims, + 'dtype': dtype, + 'block': block, + **kwargs + }) + +# Base class with generic validator +class Package(BaseModel): + model_config = ConfigDict( + arbitrary_types_allowed=True, + validate_assignment=True + ) + + @model_validator(mode='before') + @classmethod + def structure_all_arrays(cls, data, info): + """ + Generic validator that runs for ALL Package subclasses. + Inspects model_fields, finds arrays, structures them. + """ + context = info.context or {} + + for field_name, field_info in cls.model_fields.items(): + if field_name not in data: + continue + + metadata = field_info.json_schema_extra or {} + + # Only process array fields + if metadata.get('mf6_type') == 'array': + data[field_name] = structure_array_generic( + value=data[field_name], + field_name=field_name, + dims=metadata['dims'], + dtype=metadata['dtype'], + context=context + ) + + return data +``` + +#### Generated Code +```python +# GENERATED from DIS.dfn +class Dis(Package): + nlay: Annotated[int, Field(json_schema_extra={'block': 'dimensions'})] + nrow: Annotated[int, Field(json_schema_extra={'block': 'dimensions'})] + ncol: Annotated[int, Field(json_schema_extra={'block': 'dimensions'})] + + delr: Annotated[ + NDArray[np.float64], + MF6Array(dims=('ncol',), dtype=np.float64, block='griddata', netcdf=True) + ] + + delc: Annotated[ + NDArray[np.float64], + MF6Array(dims=('nrow',), dtype=np.float64, block='griddata', netcdf=True) + ] + + top: Annotated[ + NDArray[np.float64], + MF6Array(dims=('nrow', 'ncol'), dtype=np.float64, block='griddata', netcdf=True) + ] + + botm: Annotated[ + NDArray[np.float64], + MF6Array(dims=('nlay', 'nrow', 'ncol'), dtype=np.float64, block='griddata', netcdf=True) + ] +``` + +**Analysis:** +- ✅ Generic `structure_array_generic` logic in one place +- ✅ **ZERO validation code in generated classes** +- ✅ Base class validator runs automatically for all subclasses +- ✅ Field factory is pure metadata (no converter attachment) +- ✅ Can make even more concise (see Option 3) +- Lines per class: ~50 (pure field declarations) + +--- + +### Option 3: pydantic with Custom Type (Advanced, Most Concise) + +#### Base Library +```python +from pydantic import GetCoreSchemaHandler +from pydantic_core import core_schema + +class MF6ArrayType: + """ + Custom type that handles validation automatically. + This is the most advanced pydantic pattern. + """ + def __init__(self, dims, dtype, block, **metadata): + self.dims = dims + self.dtype = dtype + self.block = block + self.metadata = metadata + + def __get_pydantic_core_schema__( + self, + source_type, + handler: GetCoreSchemaHandler + ): + """Define how to validate this type.""" + def validate(value, info): + return structure_array_generic( + value, + dims=self.dims, + dtype=self.dtype, + context=info.context + ) + + return core_schema.with_info_plain_validator_function( + validate, + serialization=core_schema.plain_serializer_function_ser_schema( + lambda x: x.tolist() if hasattr(x, 'tolist') else x + ) + ) +``` + +#### Generated Code +```python +# GENERATED from DIS.dfn - MOST CONCISE VERSION +class Dis(Package): + nlay: int + nrow: int + ncol: int + + delr: MF6ArrayType(dims=('ncol',), dtype=np.float64, block='griddata') + delc: MF6ArrayType(dims=('nrow',), dtype=np.float64, block='griddata') + top: MF6ArrayType(dims=('nrow', 'ncol'), dtype=np.float64, block='griddata') + botm: MF6ArrayType(dims=('nlay', 'nrow', 'ncol'), dtype=np.float64, block='griddata') +``` + +**Analysis:** +- ✅ **Most concise generated code possible** +- ✅ Validation logic in type definition +- ✅ Type hints are the field definition +- ⚠️ More advanced pydantic pattern (steeper learning curve) +- Lines per class: ~40 (absolute minimum) + +--- + +## Side-by-Side Comparison + +### Generated Class: NPF Package + +#### attrs + cattrs +```python +@define +class Npf(Package): + save_flows: bool = field( + default=False, + metadata={'block': 'options'} + ) + + icelltype: NDArray[np.int64] = mf6_array( + dims=('nodes',), + dtype=np.int64, + block='griddata', + netcdf=True, + longname='cell type' + ) + + k: NDArray[np.float64] = mf6_array( + dims=('nodes',), + dtype=np.float64, + block='griddata', + netcdf=True, + longname='hydraulic conductivity' + ) +``` + +#### pydantic (Option 2) +```python +class Npf(Package): + save_flows: Annotated[ + bool, + Field(default=False, json_schema_extra={'block': 'options'}) + ] + + icelltype: Annotated[ + NDArray[np.int64], + MF6Array( + dims=('nodes',), + dtype=np.int64, + block='griddata', + netcdf=True, + longname='cell type' + ) + ] + + k: Annotated[ + NDArray[np.float64], + MF6Array( + dims=('nodes',), + dtype=np.float64, + block='griddata', + netcdf=True, + longname='hydraulic conductivity' + ) + ] +``` + +#### pydantic (Option 3 - Custom Type) +```python +class Npf(Package): + save_flows: bool = False + + icelltype: MF6ArrayType( + dims=('nodes',), + dtype=np.int64, + block='griddata', + longname='cell type' + ) + + k: MF6ArrayType( + dims=('nodes',), + dtype=np.float64, + block='griddata', + longname='hydraulic conductivity' + ) +``` + +--- + +## Key Differences for Code Generation + +| Aspect | attrs + cattrs | pydantic (Option 2) | pydantic (Option 3) | +|--------|---------------|---------------------|---------------------| +| **Validation code in generated class** | Converter attachment | None | None | +| **Lines per class** | ~60 | ~50 | ~40 | +| **Validation logic location** | `mf6_array()` helper | Base class `model_validator` | Custom type `__get_pydantic_core_schema__` | +| **Readability** | Good | Good | Excellent | +| **Type safety** | Good | Excellent | Excellent | +| **Schema generation** | Custom code needed | Automatic (with caveats) | Automatic | +| **Learning curve** | Low | Medium | High | + +--- + +## Recommendation for Code Generation + +**Use pydantic Option 2** (model_validator on base class): + +### Why? + +1. **Zero validation code in generated classes** - they're pure declarations +2. **All complexity in base class** - easy to maintain and test +3. **Clear and explicit** - easier than Option 3 for team to understand +4. **Good balance** - not too magical, not too verbose + +### Code Generator Output + +Your DFN → Python generator would emit: + +```python +# Auto-generated from {package}.dfn +# DO NOT EDIT - regenerate from DFN file + +from typing import Annotated +from numpy.typing import NDArray +import numpy as np +from flopy4.base import Package, MF6Array, MF6Field + +class {ClassName}(Package): + {for field in fields} + {field.name}: Annotated[ + {field.python_type}, + {field.factory}({field.metadata}) + ] + {endfor} +``` + +**Simple template**, minimal logic, easy to maintain. + +### Example Generated Output + +```python +# Auto-generated from npf.dfn +# DO NOT EDIT + +from typing import Annotated +from numpy.typing import NDArray +import numpy as np +from flopy4.base import Package, MF6Array, MF6Field + +class Npf(Package): + save_flows: Annotated[bool, MF6Field(block='options', default=False)] + icelltype: Annotated[NDArray[np.int64], MF6Array(dims=('nodes',), dtype=np.int64, block='griddata')] + k: Annotated[NDArray[np.float64], MF6Array(dims=('nodes',), dtype=np.float64, block='griddata')] +``` + +That's it. **No validation code**, no converters, no complex logic. Just field declarations with metadata. + +--- + +## Migration Path + +If you choose pydantic: + +1. **Build base class** with `model_validator` (~100 lines) +2. **Port `structure_array`** to `structure_array_generic` (~500 lines, mostly copy-paste) +3. **Update code generator** to emit pydantic classes (simpler template than attrs!) +4. **Regenerate all packages** from DFN files +5. **Test** - validation should work identically + +Estimated effort: **1-2 weeks** for the base infrastructure, then regeneration is instant. + +--- + +## Bonus: Even More Concise with Defaults + +You can make common patterns even simpler: + +```python +# In base library +def GridDataArray(dims, dtype=np.float64, **kwargs): + """Shortcut for griddata block arrays.""" + return MF6Array(dims=dims, dtype=dtype, block='griddata', netcdf=True, **kwargs) + +# Generated code becomes: +class Npf(Package): + k: Annotated[NDArray[np.float64], GridDataArray(dims=('nodes',))] + icelltype: Annotated[NDArray[np.int64], GridDataArray(dims=('nodes',), dtype=np.int64)] +``` + +Even more concise! diff --git a/docs/dev/plans/archive/codegen-recommendation.md b/docs/dev/plans/archive/codegen-recommendation.md new file mode 100644 index 00000000..23c8466e --- /dev/null +++ b/docs/dev/plans/archive/codegen-recommendation.md @@ -0,0 +1,262 @@ +# Code Generation: Final Recommendation + +## Your Question + +> "We will eventually be generating the MF6 object model from DFN files. So ideally, we'd keep the generated files minimal, and move as much shared logic into base classes or shared functions. Does this impact your recommendations regarding validators? The array structuring logic is generic, the only thing that varies is the array's dimensions." + +## Direct Answer + +**This actually STRENGTHENS the case for pydantic.** Here's why: + +### With Pydantic: Zero Validation Code in Generated Classes + +Pydantic's `model_validator` decorator on the base class means: +- ✅ **All validation logic lives in the base class** +- ✅ **Generated classes are just field declarations** +- ✅ **No duplication of `structure_array` logic** +- ✅ **Validator runs automatically for all subclasses** + +### Example: Generated Code + +```python +# Auto-generated from npf.dfn - JUST FIELD DECLARATIONS! +class Npf(Package): + save_flows: Annotated[bool, MF6Field(block='options', default=False)] + + icelltype: Annotated[ + NDArray[np.int64], + MF6Array(dims=('nodes',), dtype=np.int64, block='griddata') + ] + + k: Annotated[ + NDArray[np.float64], + MF6Array(dims=('nodes',), dtype=np.float64, block='griddata') + ] +``` + +**That's it.** No validators, no converters, no logic. Just metadata. + +### How It Works + +In your **base library** (not generated): + +```python +class Package(BaseModel): + @model_validator(mode='before') + @classmethod + def structure_all_arrays(cls, data, info): + """ + Runs automatically for ALL Package subclasses. + Inspects model_fields, finds arrays, calls structure_array_generic. + """ + context = info.context or {} + + for field_name, field_info in cls.model_fields.items(): + metadata = field_info.json_schema_extra or {} + + if metadata.get('mf6_type') == 'array': + data[field_name] = structure_array_generic( + value=data[field_name], + dims=metadata['dims'], + dtype=metadata['dtype'], + context=context + ) + + return data +``` + +**One method, all packages get it for free.** + +--- + +## Comparison to attrs + +### attrs + cattrs Approach + +**Generated code:** +```python +@define +class Npf(Package): + save_flows: bool = field(default=False, metadata={'block': 'options'}) + + icelltype: NDArray[np.int64] = mf6_array( # ← Still need to call helper + dims=('nodes',), + dtype=np.int64, + block='griddata' + ) + + k: NDArray[np.float64] = mf6_array( # ← Repeated for every array + dims=('nodes',), + dtype=np.float64, + block='griddata' + ) +``` + +**Issues:** +- Still need `mf6_array(...)` call for every array field +- Converter attachment is in generated code +- Helper function reduces duplication but doesn't eliminate it + +### pydantic Approach + +**Generated code:** +```python +class Npf(Package): + save_flows: bool = Field(default=False) + + icelltype: Annotated[NDArray[np.int64], MF6Array(dims=('nodes',), ...)] + k: Annotated[NDArray[np.float64], MF6Array(dims=('nodes',), ...)] +``` + +**Benefits:** +- Just metadata, no function calls +- Base class handles everything +- Truly minimal + +--- + +## Code Generator Template + +With pydantic, your template is **simpler**: + +```jinja2 +# Auto-generated from {{ dfn_file }} +from typing import Annotated +from numpy.typing import NDArray +import numpy as np +from flopy4.base import Package, MF6Array, MF6Field + +class {{ class_name }}(Package): + {% for field in fields %} + {{ field.name }}: Annotated[ + {{ field.python_type }}, + {{ field.factory }}({{ field.metadata_args }}) + ] + {% endfor %} +``` + +**That's the entire template.** No logic for converter attachment, no special handling for arrays vs scalars. Just iterate fields and emit declarations. + +--- + +## Revised Recommendation + +Given **code generation is your goal**: + +### Choose: Pydantic with model_validator (Option 2) + +**Reasons:** +1. ✅ **Absolute minimum generated code** - just field declarations +2. ✅ **All validation in base class** - 1 method, ~50 lines +3. ✅ **Simpler code generator** - no complex logic in template +4. ✅ **Easy to maintain** - change validation logic once, all packages updated +5. ✅ **Schema-first design** - metadata is in `json_schema_extra` + +### Your Structure + +``` +flopy4/ +├── base/ +│ ├── __init__.py +│ ├── package.py # ← Package base class with model_validator +│ ├── fields.py # ← MF6Array, MF6Field factories +│ └── validation.py # ← structure_array_generic (500 lines) +│ +├── mf6/ +│ ├── gwf/ +│ │ ├── npf.py # ← GENERATED: ~40 lines, just fields +│ │ ├── dis.py # ← GENERATED: ~60 lines, just fields +│ │ └── ... # ← All generated, all minimal +│ └── ... +│ +└── codegen/ + └── dfn_to_python.py # ← Code generator (simple template) +``` + +### Base Class (~50 lines total validation logic) + +```python +# flopy4/base/package.py +class Package(BaseModel): + model_config = ConfigDict( + arbitrary_types_allowed=True, + validate_assignment=True + ) + + parent: Any = Field(default=None, exclude=True) + + @model_validator(mode='before') + @classmethod + def structure_all_arrays(cls, data, info): + # ~30 lines of introspection + calling structure_array_generic + ... + + def to_dataset(self) -> xr.Dataset: + # Generic implementation (~20 lines) + ... +``` + +### Generated Class (~40 lines total, zero logic) + +```python +# flopy4/mf6/gwf/npf.py - AUTO-GENERATED +class Npf(Package): + # ~10 field declarations + # That's it! +``` + +--- + +## Migration Path + +1. **Week 1:** Build base infrastructure + - `Package` base class with `model_validator` + - Port `structure_array` → `structure_array_generic` + - Create `MF6Array`, `MF6Field` factories + +2. **Week 2:** Build code generator + - Simple Jinja2 template (easier than current attrs version!) + - DFN parser (may already exist) + - Generate a few test packages + +3. **Week 3:** Regenerate all packages + - Run generator on all DFN files + - Test that validation works identically + - Update tests + +4. **Week 4:** Polish + - Documentation + - Edge cases + - Schema generation refinements + +**Total: ~4 weeks** for complete migration including code generator. + +--- + +## Bottom Line + +**For code generation, pydantic is objectively better:** + +| Metric | attrs + cattrs | pydantic | +|--------|---------------|----------| +| Lines per generated class | ~60 | ~40 | +| Validation code in generated class | Converter calls | 0 | +| Generator template complexity | Medium | Low | +| Duplication of structure logic | Via helper | None | +| Changes to validation logic | Regenerate all | Change base class only | + +The `model_validator` pattern is **exactly what you need** for minimal code generation. + +--- + +## Demo + +Run: `pixi run -e dev python codegen_architecture.py` + +This shows: +- DIS and NPF classes with **zero validation code** +- Base class validator that **runs automatically** +- Generic `structure_array_generic` function used for all arrays +- Confirmation that generated classes are minimal + +**Try it and see** - this is the architecture I recommend. diff --git a/docs/dev/plans/archive/codegen-role-removal-plan.md b/docs/dev/plans/archive/codegen-role-removal-plan.md new file mode 100644 index 00000000..a005bea8 --- /dev/null +++ b/docs/dev/plans/archive/codegen-role-removal-plan.md @@ -0,0 +1,71 @@ +# Remove the codegen "role" layer + +Status: proposal. Separate from the `count=` -> `shape=` refactor +(`count-to-shape-plan.md`), which only merges the three inline-array roles. +Re-check `origin/develop` before starting; codegen files move fast. + +## Background + +`make.py` (`_build_item_schema`, ~L180-300) classifies each column of an `Item` +row and stores the result as `entry["role"]`, a string. `filters.py` (~L718-860) +then branches on it in `_py_type`, `_is_optional`, `_field_meta` and +`_field_line`. A few other places in `make.py` read it too (~L905-918, ~L1599). + +Nothing outside `flopy4/mf6/utils/codegen/` reads `role` or the schema dict +(checked with `git grep` on `flopy4/`, excluding codegen). It is an +intermediate representation between two codegen stages, not runtime metadata. + +## Why remove it + +- Every role is derived from facts already on the column spec: `isinstance(f, + File)`, `col.is_cellid`, `col.is_index`, `col.is_row_keyword`, `shape`, + `col.name == "boundname"`. The string re-encodes them. +- Adding a column kind means editing the classifier in `make.py` and every + `role` branch in `filters.py`; the two must agree on role names. +- Roles bundle several output decisions (type, optionality, metadata, field + line). The same bundle is decided in four functions that must stay in sync. +- Several role checks are scattered as `col["role"] in (...)` tuples, which are + easy to leave out of date (`_is_optional` and `_py_type` list overlapping + but different sets). + +## Target + +Render from the column spec directly. One function per column kind returns the +whole rendering (annotation, optionality, metadata, field line) in one place, +so a kind's output decisions live together instead of across four functions. + +Suggested shape: a small dataclass (e.g. `RenderedColumn`: `annotation`, +`optional`, `meta`, `line`) built by a per-kind function chosen by a single +dispatcher. The dispatcher is the only place that inspects the column spec. + +## Steps + +1. Inventory every use of `role`: classifier (`make.py`), the four `filters.py` + functions, and the other `make.py` reads. List, per role, the output + decisions it implies. +2. Write a characterization test first: render every Item schema from the + current DFNs and snapshot the generated source. The cleanup must leave this + byte-identical. +3. Add the per-kind rendering functions and dispatcher; keep `role` populated + while the callers move over, one role at a time, running the snapshot after + each. +4. Remove the `role` key, the string tuples, and the classifier branches that + only existed to set it. +5. Regenerate and confirm no diff in `flopy4/mf6/` generated files; run the + full suite and the codegen tests. + +## Risks + +- Generated output drift. The snapshot test in step 2 is the guard; do not + skip it. +- `develop` conflicts: `filters.py`/`make.py` are edited by most codegen PRs. + Land this quickly once started, and keep it behavior-neutral. +- Non-Item code paths (e.g. keystring arms, `nested_union`) have their own + special cases that may not fit the dataclass cleanly. Decide per case whether + to fold them in or leave them for later. + +## Out of scope + +- Changing generated output. +- The `count=` -> `shape=` change (separate plan; it merges only the + `sized`/`counted`/`array` roles and should land first). diff --git a/docs/dev/plans/archive/dead-code-audit.md b/docs/dev/plans/archive/dead-code-audit.md new file mode 100644 index 00000000..0a32c5a5 --- /dev/null +++ b/docs/dev/plans/archive/dead-code-audit.md @@ -0,0 +1,147 @@ +# Dead code audit (develop @ 5ac2af7a) + +Audit of `flopy4/` on `develop` after #505. Generated component modules +(`gwf/`, `gwt/`, `gwe/`, `prt/`, `olf/`, `chf/`, `exg/`, `utl/`) were left out. +Line numbers are as of `5ac2af7a`; re-check before editing. + +Status re-checked at `febcfd51`: sections 1, 3 and 4 are done (section 1 +finished in #528). Section 2's deletions are merged (#530); the rest of it, +and the gaps found along the way, are in #531 (open). Section 5 is +untouched. + +Method: a reference scan (every function/class in `flopy4/` with no reference +elsewhere in `flopy4/`, counting `.lark` grammars and `.jinja` templates, +ignoring self-references, comments and `__all__` entries) plus ruff +`F401,F811,F841,ERA001,B018`. Every hit was checked by hand. + +Each item below is meant to be one session / one small PR. Per +`CLAUDE.local.md`, verify with `pixi run -e dev smoke` plus the test files +touched. + +## 1. Remove unused modules and functions + +- [x] **`flopy4/mf6/indexes.py`** (done in #510): delete the whole module (`alias`, + `MetaIndex`, `grid_index`, `time_index`). Nothing imports it, tests + included. Last touched in #291. +- [x] **`flopy4/dimensions.py:263-343`**: delete `validate_dimension_resolution`. + It only calls itself; no callers in source, tests, docs or benchmarks. + (Resolved differently in #528: fixed, not deleted. + It read `metadata["dims"]`, which nothing sets (arrays record `shape`), so + it always returned `[]`, and it recursed through `_parent`. It now checks + that each array field's shape resolves, and `Simulation.write` runs it + first. List shapes are left out: their counts are the package's own + dimension fields.) +- [x] **`flopy4/dimensions.py:69`**: delete the `DimensionResolver` protocol. + It's only mentioned in the docstring of `validate_dimension_resolution`. + (`DimensionProvider` and `DimensionResolverMixin` are used; keep them.) + (Resolved differently in #528: kept as the consumer-side contract next + to `DimensionProvider`. The mixin's `hasattr(parent, "resolve_dims")` + checks became `isinstance(parent, DimensionResolver)`.) +- [x] **`flopy4/adapters.py:268`**: delete `get_nn` and its `__all__` entry. + Nothing calls it. (Resolved differently in #510: moved to + `flopy4/mf6/utils/grid.py`, vectorized with numpy, and tested. Kept.) +- [x] **`flopy4/mf6/codec/reader/transformer/typed.py:227`** (done in #509): delete + `TypedTransformer.simple_string`. No grammar rule (hand-written or + generated) has that name, so Lark never calls it. + +## 2. flopy3 adapter cleanup (`flopy4/mf6/adapters.py`) + +- [x] **`:420-431`**: delete `Flopy3ListData`. (Resolved differently in + #531: implemented, not deleted. It was the + placeholder for a real gap: flopy3's plotting and export read boundary + conditions through `DataListInterface`, and the adapter never gave them + any. It now wraps stress period data, and `Flopy3Model.version` is + "mf6" so `plot_bc` works. The plan's 3.x item 2, a neutral protocol in + flopy3, may make it unnecessary after the merge.) +- [x] **`:77-91`** (`Flopy3Model.__init__`): unused `lenuni`/`xoff`/`yoff`/ + `angrot`. Deleted in #530: every `to_grid()` + already passes the origin, rotation, length units and crs. `_to_numpy`, + used only there, went too. +- [x] **`:337-345`**: the commented-out `DataType` enum. Deleted in #530. + +Keep the `Flopy3Model` properties that look unused (`model_ws`, `exe_name`, +`hdry`, `hnoflo`, `get_package`, ...). They exist for flopy3's +export and plot code, which duck-types these objects. + +Found along the way: +- [x] Fixed in #531: + - `crs=` was ignored. + - Only `dis` models got a grid. GWF's `dis` already covers Disv/Disu; + OLF/CHF (`dis2d`/`disv2d`/`disv1d`) got none, and a model with `dis` + unset failed. + - Models and packages had no `to_geodataframe`. + - `get_package` never matched. + - `name` was a string, not a list. + - An unset `boundname` column was exported as NaN. +- [ ] Lists besides stress period data (`packagedata`, `connectiondata`) + aren't wrapped, which flopy3's SFR/LAK/MAW/UZF `plot_bc` reads. Left for + later, or for the plan's neutral protocol (3.x item 2); #531's + description says why. +- [x] Array fields holding a scalar (a constant, e.g. `Ic(strt=5.0)`, or a + default like Npf's `k`) were skipped by `Flopy3Package`. Now filled to + their shape, also in #531. + +## 3. Unused constants and config, plus a test that never runs + +- [x] **`flopy4/mf6/constants.py`** (done in #509): delete `FILL_DEFAULT`, `FILL_INT32`, + `LENBOUNDNAME`. Never referenced. +- [x] **`flopy4/mf6/config.py`** (done in #509): `SPARSE_THRESHOLD` isn't read by any + library code. The only reader is + `TestSparseArrays.test_sparse_array_creation` + (`test/test_converter_structure.py:205`), whose body is gated on + `nper * nodes (1e6) > SPARSE_THRESHOLD (1e7)` and so never runs. The test + passes without testing anything. Delete the test class and `config.py` + (the module holds only this constant and a TODO). + +## 4. Small leftovers + +- [x] **`flopy4/mf6/codec/writer/__init__.py:22`** (done in #509): drop the + `_JINJA_ENV.filters["is_constant"]` registration. No template uses it. + Keep the function; `layer_is_constant` in `writer/filters.py` calls it. +- [x] **`flopy4/mf6/utils/grid.py:180`, `:933`** (done in #509): unused local `nlay` + (ruff F841) in the scalar-`botm` broadcasting branches. + +## 5. Decision needed: test-only attrs/xarray layer + +Only `test/test_attrs_xarray.py` uses these; nothing in `flopy4/mf6` does: + +- `flopy4/mixins.py` (`DatasetConvertibleMixin`, `DataTreeConvertibleMixin`) +- `flopy4/protocols.py` (`DatasetConvertible`, `DataTreeConvertible`) +- the dataset→object half of `flopy4/attrs_xarray.py`: `dataset_to_attrs` + (`:266`), `datatree_to_attrs` (`:370`), `_leaf_kwargs_from_dataset` + (`:251`), `_init_field_names` (`:244`), `_period_value` (`:181`) + +The object→dataset half (`attrs_to_dataset`, `attrs_to_datatree`, +`child_field_candidates`, `_child_field_spec`) **is** used by `mf6` and +codegen; keep it. + +Correction: `test/mf6/test_mf6_period_arrays.py` also round-trips real +`Welg`/`Rcha` through `dataset_to_attrs`. Comparison and design options are +in [`export.md`](../xarray/export.md); nothing gets deleted until those decisions are made. + +- [ ] Decide whether this is the planned generic conversion layer (keep it, + maybe wire `mf6` to it) or leftover from before xattree was dropped (#356) + (delete it along with the matching tests). + +## Not dead (for reference, so later sessions don't re-flag them) + +- **Lark callbacks:** `BasicTransformer.TOKEN`/`CNAME`/`INT`/`QUOTED_WORD`/ + `NUMBERS`/`RECORDS`, `TypedTransformer.block_kw_word`/`single_array`/ + `layered_array`, `_BufferLexerThread.from_custom_input` (called by + Lark's `LexerThread` machinery). +- **pydantic `field_validator`s** in `flopy4/mf6/netcdf.py`. +- **xarray hooks:** `CellIndex.roll`/`concat`/`isel` in `utils/grid.py`, and + the `grid` accessors in `accessor.py` (registered when `flopy4.mf6` + imports it). +- **Import hook:** `_StagedFinder.find_spec` in `_sync.py`. +- **Overrides and public API:** `StructuredGridWrapper.from_binary_grid_file` + (overrides flopy's), `IODescriptor.list_formats` (public, but no tests; + maybe add one). +- **Public, tested only by tests:** `component.get_fnames`, + `utils.grid.dims_from_grb`, `Package.from_xarray`, `NetCDFModel.from_model`. +- **Side effect on purpose:** `self._children` alone on a line at + `simulation_methods.py:56` names any unnamed models. +- **Generated aliases** like `TdisPerioddata` and `WelStressPeriodData`: public + API from codegen. +- **ruff ERA001 false positives** in `converter/egress/unstructure.py:305,316,329`: + comments describing types. diff --git a/docs/dev/plans/archive/dev3-migration-gaps.md b/docs/dev/plans/archive/dev3-migration-gaps.md new file mode 100644 index 00000000..3cc94f79 --- /dev/null +++ b/docs/dev/plans/archive/dev3-migration-gaps.md @@ -0,0 +1,203 @@ +# dev3 migration: gaps and overrides hit + +**Status (2026-09-17): fully resolved.** Gaps #2 and #3 below got fixed +upstream, and the flopy4-side stopgap layer this doc tracked +(`overrides.py`/`dfn_overrides.toml`) has since been deleted entirely +(`f3cd6f6`/#353, `e3075da`/#354 — "no more overrides"). Gap #1 +(`Array.repeat`) turned out not to be a devtools bug at all -- devtools +investigated it and found dev3's existing `shape=[]` self-sizing-array +semantics already cover what `repeat` would have expressed, and removed +the dead `repeat` mechanism rather than populate it (see +`~/dev/modflow-devtools/todo.md`, 2026-08-18 entry, checked off). The +remaining flopy4-side codegen/runtime work for self-sizing arrays -- +covering both real patterns hiding behind dev3's one `shape == []` +representation, see gap #1 below -- is now implemented and tested +(`filters.py`, `make.py`, `record.py`, `structure.py`, `unstructure.py`, +`spec.py`, `basic.lark`, `macros.jinja`; `test/mf6/test_mf6_codec.py`). +Kept for the design-decision rationale in the second section, which +explains several non-obvious pieces of the current DFN-interpretation code +(e.g. why `role="feature_id"` is Integer-only, why keystring period lists +have two record shapes) that aren't reconstructible from the code alone. + +**One new, unrelated bug surfaced along the way, since fixed** (found +2026-09-17, fixed 2026-09-17): `structure_component`'s Pass 1 +(`converter/ingress/structure.py`) matched a row's leading token against +top-level field names *before* checking the inner-record-keyword lookup +(`inner_class_fields`). For utl-tas this meant `time_series_namerecord`'s +keyword `NAME` collided with `Package.name` (silently overwriting the +package's own instance name with the record's data) and `sfacrecord`'s own +field name (`sfac`, after `_strip_record_words` drops "record") collided +with its own inner record's trigger keyword. Fixed with two changes: (1) +`all_fields` now only indexes fields carrying `block` metadata, so +identity/bookkeeping attributes with no block metadata (`Component.name`, +`filename`, ...) can never shadow a same-named block keyword; (2) whenever +a matched field's type resolves to an inner Record class +(`_inner_class_type`), it's always dispatched through `from_tokens()` +rather than falling into the generic scalar/list branch, so a field whose +own name coincides with its record's trigger keyword still works. Along +the way, `record.py`'s `from_tokens` also turned out to assume every token +was a string (`tokens[i].upper()`), which crashed the first time this path +was actually exercised with a numeric-valued record (`loads()` already +parses numeric tokens to `int`/`float`) — fixed with `str(tokens[i])`. +Verified against a real MF6-written `.tas` file and +`test_load_tas_inner_classes_name_collision` (no longer `xfail`). + +Working doc from `filters.py`/`make.py` rewrite. Two kinds of items: +**devtools-side gaps** (already logged in `devtools/todo.md`, need an +upstream fix) and **flopy4-side local overrides** (stopgaps in +`dfn_overrides.toml` that should come out once the upstream fix lands). + +## Devtools-side gaps + +1. ~~`Array.repeat` never populated by migration.~~ **Reclassified — not a + devtools bug, and turned out to be two distinct flopy4-side patterns + hiding behind one dev3 shape.** v1's `repeating: true` (affecting + `utl-tas.tas_array`, `prt-oc.times`, `prt-prp.times`) looked like it + needed a migration path into a `repeat` field, same bug class as + `layered` (which *was* fixed, in f3ae0f4). Devtools investigated instead + of migrating it: all 3 fields also carry v1 `shape (any1d)`, which the + mapper already collapses to dev3's `shape=[]` — matching `dfnspec.md`'s + self-sizing-array semantics, with no sibling count field for `repeat` to + reference. `Array.repeat` was removed from `schema.py`/`dfnspec.md`/ + `schema.json` rather than backfilled (see `~/dev/modflow-devtools/ + todo.md`, 2026-08-18 entry, checked off 2026-09-17). + + Implementing the flopy4 side revealed dev3's `shape == []` actually + covers **two unrelated wire-format patterns**, not one: + + - **Trailing-tokens record children** (`utl-tas.time_series_name`/ + `sfacval`, `prt-oc.track_timesrecord`/`prt-prp.release_timesrecord`'s + `times`): plain `NAME v1 v2 ...`/`SFAC v1 v2 ...` syntax, consumed + until the record runs out of tokens -- true "self-sizing" in the + "MF6 reads until it runs out" sense. Fixed generally: `filters.py`'s + `is_record_list_field`/`_record_child_supported`/ + `can_generate_record_class`, `make.py`'s `_build_record_class_specs` + (emits `list[str]`/`list[float]` children), and `record.py`'s + `to_tokens`/`from_tokens` (a `list[...]`-typed field consumes/emits + all remaining tokens instead of exactly one). Verified against the + whole dev3 corpus: flips exactly `time_series_namerecord`, `sfacrecord`, + `track_timesrecord`, `release_timesrecord`, plus utl-ts's 3 records + (not currently generated, but ready once utl-ts's other blockers + clear) — nothing else in ~250 Record fields. + - **`tas_array` itself**: not self-sizing in that sense at all -- a + griddata-style `Array` field (U2DREL CONSTANT/INTERNAL/OPEN-CLOSE + control-record syntax, sized by the grid) whose own block (`time`) + repeats per distinct header value (`BEGIN TIME ... END TIME`, + found via a real MF6-written `.tas` fixture -- analogous to + `BEGIN PERIOD ` but keyed by a float time, not an int period + index). Required four pieces: `filters.py`'s `is_readarray_field` + + `make.py`'s time-array-series field spec (`Optional[dict[float, + FloatArrayLike]]`, paralleling `stress_period_data`'s `dict[int, + ...]` -- `time_from_model_start` isn't a stored field, same as `kper` + isn't, it's carried by the dict's keys); `basic.lark`'s block-header + grammar (`block_name: CNAME [INT]` → `[NUMBER]`, a safe superset since + Lark's `NUMBER` already includes `INT`, verified against the full + suite); a new ingress pass in `structure.py` (parses repeating + ` ` blocks via the existing `_read_control_record`/ + `_griddata_flat_length` griddata primitives, needs `dims`); a new + egress path in `unstructure.py` plus a `macros.jinja` `array_bare` + macro (the block's content is the bare control record with no + field-name token first -- `just_data true` in the v1 DFN, unlike a + griddata field's `\n CONSTANT ...`). `spec.py`'s `to_field_type` + also needed a `dict[K, V]` unwrap case (resolves dtype from the dict's + value type) since existing period/griddata array fields never wrap in + a dict. + + Both verified via round-trip tests in `test/mf6/test_mf6_codec.py` + (`test_dumps_tas_inner_classes*`, `test_tas_array_roundtrip`, + `test_tas_array_constant_roundtrip`) and against a real MF6-written + `.tas` file. No upstream dependency for either. + +2. ~~`numeric_index` → `pk`/`fk` backfill (a838d84) covered 6 fields, but + 44 have the same gap.~~ **Resolved upstream, comprehensively — superset + fix, not a per-field audit.** A full recursive corpus scan (v1 + `numeric_index: true` vs dev3 leaf fields, including inside + `List.item`/`Record`/`Union` nesting — a flat scan finds 0) had turned up + 44 fields with no `pk`/`fk` in dev3; only `gwf-lak.iconn` was confirmed to + break a real test and got a local flopy4-side `pk = true` override. + Devtools resolved the whole class by decomposing `pk`/`fk`'s conflated + "needs 1-based/0-based conversion" + "is a relational key" concerns into a + dedicated `index` attribute (`index-node-attributes-plan.md`, Phase 1): + `Integer.index`/`Array.index` (the latter gated to `dtype == "integer"`), + set mechanically (`index = numeric_index`) for *every* `Integer`/ + `Array(dtype="integer")` field in `migrate_to_v2_0_0_dev2.py` — applies + unconditionally, not just where `pk`/`fk` are absent, so it's a superset + of the 44-field gap rather than an audit of it. Verified: `gwf-lak.iconn` + now comes through as `index=True` with no flopy4-side override needed + (the override is gone, per the top status line). + +3. ~~Bare `optional` attribute (no value) in v1 DFNs isn't migrated as + `true`.~~ **Resolved upstream, generically — not a per-field patch.** + v1 syntax allows a valueless `optional` line (vs. `optional true`/ + `optional false`) meaning "true"; the migration used to default it to + `false` instead (30 occurrences across 12 v1 DFN files, only 3 fields × + 6 packages confirmed breaking a real test at the time). Fixed at the + shared parsing utility (`try_parse_bool`: an empty string now parses as + `True`, not falling through to `default`), so it applies to every + `optional`-parsing call site at once, not just the confirmed-broken + printrecord fields. Verified live (2026-09-18): `gwf-oc`'s + `formatrecord.{columns,width,digits}` now migrate as `optional=True` + with no flopy4-side override present anywhere in the repo. + +## Flopy4-side local overrides (`flopy4/mf6/utils/codegen/dfn_overrides.toml`) + +**No longer exists** (`overrides.py`/`dfn_overrides.toml` deleted, +`f3cd6f6`/#353, `e3075da`/#354) — both stopgaps below were superseded by +the upstream fixes for gap #2 and #3 and removed along with the file. +Kept here for history only. + +- `["gwf-lak".iconn] pk = true` — gap #2, confirmed via a real MF6 write + failure (`iconn FOR LAKE 1 MUST BE > 1 and <= 9`) before the patch. +- `["".columns/width/digits] optional = true` for `chf-oc`, `gwe-oc`, + `gwf-oc`, `gwt-oc`, `olf-oc`, `gwt-ist` — gap #3, the printrecord family's + `formatrecord.{columns,width,digits}`. + +## Flopy4-side design decisions (not devtools bugs — schema is correct, +## flopy4 needed to interpret it more carefully than a first pass did) + +- **`role="feature_id"` (0-based↔1-based conversion) must be Integer-only.** + A `pk`/`fk` on a `String` field (e.g. MVR's `pname`, a package *name* + reference) isn't a numeric index — applying the conversion crashed on + `int("wel0")`. Fixed by gating `is_index` on `isinstance(col, Integer)`. +- **Keystring-shaped period lists have two real record shapes, not one.** + LAK: `item` is a single-field Record wrapping the Union (index embedded + per-arm, e.g. `lakeno`/`outletno`). LKE/LKT/SFR: `item` is a Record with + an index field *and* a sibling Union field. Both needed detecting via + `find_keystring_union`, not just the single-field-wrapper case. +- **Not every keystring union has an index column.** PRP's `releasesetting` + (ALL/FIRST/LAST/FREQUENCY/STEPS) has no per-row index anywhere (confirmed + via v1 DFN: bare `recarray releasesetting`, no feature-id field) — unlike + LAK/LKE/SFR. Emitting a fabricated "number" column there was wrong. + `_keystring_has_index` checks both the outer-sibling and per-arm-embedded + cases before deciding whether to include it. +- **A bare Union arm (no data payload, e.g. OC's `ocsetting` ALL/FIRST/LAST) + needs `Literal["ARM"]`, not generic `bool`** — a plain bool erases *which* + of several mutually-exclusive flags was set. +- **`STO.storage` (period) is a bare scalar directly in the period block**, + not a `List` — but still needs the same per-period `dict[int, ...]` + fill-forward treatment as any other period field. Only `List` fields were + getting that wrapping initially; bare period scalars needed the same path. +- **Nested records, one level deep, are a single recurring pattern, not + arbitrary depth.** The `head/temperature/concentration/qoutflow/cim + printrecord` family (6 fields) all wrap one `formatrecord: Record{columns, + width, digits, format}`. Implemented via one level of recursive flattening + in `_build_inner_class_spec`, replacing the old `extra_children` TOML + override mechanism (now empty/unused) which had gone stale (wrong + `optional` values baked in from an earlier, incorrect guess). +- **`is_file_record` (Record wrapping a File child, e.g. `ts_filerecord`) + and a bare `File` field with no wrapping Record (e.g. `prt-fmi.packagedata`'s + `gwfhead`/`gwfbudget`/`gwfgrid`) are different shapes** — both need + `path()` codegen, but only one has a `Record` to unwrap. + +## Explicitly out of scope for this pass (deferred, not forgotten) + +- **Union-item-class shape** (real per-arm typed classes instead of the + generic `(keyword, value)` approximation) — prototyped and validated + (`composite_v3.py`, a scratchpad script that didn't survive between + sessions). At the time this doc was written, `structure.py`/ + `unstructure.py` only understood the flat `Schema`/`Column`/`role=` + vocabulary, so this was deferred pending Phase 0.6's `Row` migration. + **Update (2026-08-20): Phase 0.6 has since landed** (`78c506b`) — `Row` + is now the sole list-block schema, so this blocker is gone. Still not + wired in; see Phase 0.7 in [`mf6-object-model-plan.md`](mf6-object-model-plan.md) for current status. +- ~~`utl-tas.TimeSeriesName`/`Sfac`~~ — **resolved**, see gap #1 above. diff --git a/docs/dev/plans/archive/develop-shared-fixes.md b/docs/dev/plans/archive/develop-shared-fixes.md new file mode 100644 index 00000000..c692c460 --- /dev/null +++ b/docs/dev/plans/archive/develop-shared-fixes.md @@ -0,0 +1,237 @@ +# Shared fixes: `pydantic-plan` → `develop` + +**Archived 2026-10-19.** Obsolete: the fixes are on develop or no longer apply. Equality is numpy-aware (`ARRAY_EQ`, `_types.py`), `Component.filename` is a `Path`, `sdd.md` no longer mentions cattrs, and `disbase.py` and the scalar TDIS fields are gone. pydantic-plan itself is stale (draft #367). + +Last checked: 2026-09-29, against `origin/develop` at `840b952` and +`origin/pydantic-plan` at `c4178c8` (CI green). + +## Background + +`develop` (attrs) and `pydantic-plan` (pydantic dataclasses) are kept +side by side as two versions of the MF6 object model, so the two approaches can be +compared. The goal is for them to differ **only** in the object-model +choice. Porting the pydantic migration turned up several bugs that have nothing to do +with pydantic. They also affect `develop`, and this file lists them. + +Branch off `develop` and port each fix in the attrs style. Use +`pydantic-plan` as the reference implementation, but don't copy +pydantic-specific code (`Field`, `field_validator`, `__pydantic_fields__`, +`field_meta`, ...). The attrs equivalents are `attrs.field(converter=...)`, +`attrs.fields()`, `f.metadata` and `f.type`. + +To read reference code without switching branches: +`git show origin/pydantic-plan:`, or `git show ` for the +commits named below. + +## Ground rules + +- **Environment:** run everything as `pixi run -e dev ...`. Never use + `uv run`: it re-syncs `.venv` and drops extras. +- **Checks:** verify with `pixi run -e dev smoke`, and run the full suite only + when asked. CI also runs `ruff check`, `ruff format --check`, + `mypy flopy4` and `codespell`, so run those too. +- **Worktrees:** put them under `~/dev/`, never `/tmp`. +- **GitHub:** don't open PRs, issues or comments unless explicitly asked. +- **Commits:** one commit per item is ideal. Every commit made by an + agent ends with + `Co-Authored-By: Claude Opus 5.5 `. +- **Re-check first:** `develop` moves fast (#371–#373 landed after this + list was first written and fixed part of item 1). Re-check each item + against current `develop` before starting it, then update this file: + remove an item once it's ported. + +## Suggested order + +| # | Item | Size | Why this position | +|---|------|------|-------------------| +| 1 | Griddata dtype in `_coerce_griddata` | tiny | one line plus a test; isolated | +| 2 | Tdis array defaults | small | isolated to `tdis.py` | +| 3 | `sdd.md` cattrs references | small | docs only | +| 4 | Remaining input-parsing gaps | small | two isolated spots | +| 5 | `filename` as `Path`, POSIX paths | medium | touches the writer, converter and tests; widest reach | +| 6 | Array-aware equality | medium | needs a design decision; do it the same way on both branches | + +--- + +## 1. `_coerce_griddata` casts all griddata to float64 + +- **Where:** `flopy4/mf6/gwf/disbase.py:45` + ```python + dtype = _PKG_DTYPE_MAP.get(f.metadata.get("dfn_type", "double"), np.float64) + ``` + Nothing sets a `dfn_type` key in field metadata, so this always + resolves to float64. +- **Symptom:** on `pydantic-plan`, a layered `idomain=1` was repeated per + layer and `.astype(float64)`'d. It was then written as + `CONSTANT 1.00000000e+00`, and the macOS mf6 build rejects that for an + integer array. This made `test_gwf_chd01` and + `test_gwf_oc_period_variations` fail on macOS. The Linux and Windows builds + accept it. +- **On develop:** latent, because scalars are expanded earlier so this + branch isn't hit in the same way. It's the same dead lookup, though. +- **Fix** (reference: `c4178c8`): resolve the dtype the way `Package` + does. + ```python + from flopy4.mf6.spec import to_field_type + dtype = _PKG_DTYPE_MAP.get(to_field_type(f.type), np.float64) + ``` +- **Test:** port `test_layered_int_griddata_keeps_int_dtype` from + `pydantic-plan:test/mf6/test_mf6_component.py`. It builds + `Dis(nlay=2, nrow=1, ncol=3, top=1.0, botm=[0.0, -1.0], idomain=1)` and + asserts `dis.idomain.dtype == np.int64`. + +## 2. `Tdis.perlen`, `nstp` and `tsmult` default to bare scalars + +- **Where:** `flopy4/mf6/tdis.py:32-34`. The fields are typed `NDArray[...]`, + but the defaults are `1.0`, `1` and `1.0`, so `__attrs_post_init__` has to + branch on scalar vs. array. +- **Reference:** `origin/pydantic-plan:flopy4/mf6/tdis.py`, lines 49–51 and + 84–98. + - The defaults are 0-d arrays created by a factory (e.g. + `np.array(1, dtype=np.int64)`). + - A converter runs `np.asarray` on any explicit value. + - Post-init checks `.size == 1` and broadcasts to `nper` with `np.full`. + Otherwise it casts to the declared dtype. +- **On attrs:** + `attrs.field(factory=lambda: np.array(1.0), converter=np.asarray)` or + similar. Keep the `perioddata` path working. +- **Tests:** existing Tdis and `Time` tests should keep passing. Add one + asserting that the default `Tdis()` fields are arrays of length `nper` + with the right dtypes. + +## 3. `docs/dev/sdd.md` still describes cattrs + +- **Where:** lines 146, 159, 161 and 170 say the conversion layer uses + `cattrs` converters with structuring and unstructuring hooks. Neither branch + uses cattrs. +- **Fix:** describe the actual converter. That's + `flopy4/mf6/converter/ingress/structure.py` (`structure_component`) and + `flopy4/mf6/converter/egress/unstructure.py`. + `origin/pydantic-plan:docs/dev/sdd.md` has a reworded version to start + from, but develop's wording should say attrs. + +## 4. Remaining input-parsing gaps + +pydantic's validation exposed these while loading the real MF6 input +corpus. attrs doesn't validate, so on `develop` they store the wrong type +without an error. + +**Already fixed on develop:** #373 (`840b952`) fixed keyword rows with a +trailing modifier (`NEWTON UNDER_RELAXATION`), a single `AUXILIARY` name, and +stale extra tokens on scalar options (IMS `OUTER_MAXIMUM 100 500`). It +picks the value from the field's type in `structure.py:~733`. The +`Npf.rewet` Record routing was already on develop too. + +Still to port: + +| Case | develop stores | Fix | Reference | +|---|---|---|---| +| `START_DATE_TIME 1970-01-01T00:00:00` | a token list, because the numeric grammar splits the leading digits off before the first `-`: `[1970, "-01-01T00:00:00"]` | the converter joins a list into one string | `pydantic-plan:flopy4/mf6/tdis.py:26-47` | +| `START_DATE_TIME 1997` (a bare year) | an `int` in an `Optional[str]` field | the same converter does `str()` on an int or float | same | +| Missing optional period column (e.g. `boundname`) | `NaN` (a float) in a str field, from pandas | drop `pd.isna(v)` values before `item_cls(**row)` so the default applies | `pydantic-plan:flopy4/mf6/package.py:383-388`; develop spot is `package.py:318` | + +Check first whether #373's grammar change for trailing text affects the +`START_DATE_TIME` tokenization. + +**Tests:** add unit tests that load a small input with each case and check +the stored type (e.g. `isinstance(tdis.start_date_time, str)`). + +## 5. `Component.filename` as a `Path`, with POSIX separators in written files + +- **Where:** `flopy4/mf6/component.py:153` has `filename: str | None`. develop's + own `docs/examples/frenchman-flat.py:706` already assigns a `Path` to + it. +- **Why:** + - `filename` is a path relative to the workspace, and namefile FNAMEs can + include subdirectories. + - The DFN file-record fields are already `Path`, so this makes them + consistent. + - Writing `as_posix()` makes input files portable across platforms, + since MF6 accepts `/` on Windows. +- **Reference:** `c900c60` (main change) and `c4178c8` (the Windows test + fix). + - **Type:** `filename: Optional[Path]`, with a converter from `str` to + `Path`, since attrs won't coerce it (pydantic did). + - **Internal assignments to wrap in `Path(...)`:** + - `simulation.py`: the `mfsim.nam` check and assignment + - `converter/ingress/structure.py`: `child.filename = Path(fname)` + - `converter/__init__.py` and `mf6/__init__.py`: `Path(path.name)` + - `gwf/disbase.py`: the `ncf.filename.name` path. develop now has + `Path(Path(ncf.filename).name)`, which simplifies once it's a `Path`. + - **Calls to `.as_posix()` wherever a path is written:** + - `converter/egress/binding.py`: `fname=` + - `converter/egress/unstructure.py`: `_path_to_tuple` + - `item.py`: the path branch of `to_tokens` + - `record.py`: `to_tokens`; add a `PurePath` branch + - `codec/writer/filters.py`: `quote_if_needed(value: Any)` returns + `value.as_posix()` for a `PurePath`. develop's `filters.py` changed + in #372, so merge carefully. + - `codec/writer/templates/macros.jinja`: the `OPEN/CLOSE` line becomes + `OPEN/CLOSE {{ value|quote_if_needed }}`. That branch can't be reached + today (`array_how()` never returns `"external"`). **Keep it** and + route it through the filter. +- **Watch out for:** + - Tests that compare `filename` to a str. On pydantic-plan, + `test_to_xarray_on_context` had to expect `Path("mfsim.nam")`. + - Tests that compare written output to `str(path)`. They fail on Windows. + Fixed in `test_quickstart_netcdf` and `test_quickstart_netcdf_mesh` by + comparing to `nc_fpth.as_posix()`. + - `attrs_xarray.py`: check that a `Path` filename serializes correctly. +- **Tests to port:** `test_filename_is_path` and + `test_external_array_path_is_posix`, both in + `pydantic-plan:test/mf6/test_mf6_component.py`. + +## 6. Equality breaks on components with array fields (not fixed on either branch) + +- **Problem:** + - attrs' generated `__eq__` compares field tuples, so ndarray fields hit + numpy's elementwise `==`. The result is an array or + `ValueError: truth value ... is ambiguous`. + - On pydantic-plan the generated `__eq__` raises the same way. + - Any `Dis(...) == Dis(...)` hits this. +- **Needed:** numpy-aware equality. + - Use `np.array_equal` for ndarrays. Decide about NaN + (`equal_nan=True`?) and dask arrays (don't compute them implicitly; + maybe identity or `NotImplemented`). + - Skip `parent`. On pydantic-plan, `_parent` has `compare=False` and + `dims` is an `InitVar`; see `test_eq_ignores_dims_and_parent`. +- **Options on attrs:** + - (a) `eq=False` on the class decorator and a hand-written `__eq__` on + `Component`. + - (b) A per-field `eq=attrs.cmp_using(eq=np.array_equal)` on array + fields. Fields are generated, so this would go in the codegen + templates (`flopy4/mf6/utils/codegen/`). +- **Gotcha:** a subclass decorated with `@attrs.define` / `eq=True` + **generates its own `__eq__` and overrides a base-class one**. That's + exactly how pydantic-plan's hand-written `Component.__eq__` was silently + dead. Make sure whichever approach you pick actually runs on generated + subclasses, and test it on a real subclass, not on `Component`. +- **Coordinate with the user before implementing.** This should be done + the same way on both branches, so it's a design decision, not a + mechanical port. +- **Tests:** two equal `Dis` instances compare equal; changing one element + of `idomain` makes them unequal; `parent` doesn't affect the result. + +--- + +## The other direction (develop → pydantic-plan), for context + +This isn't part of the task, but `pydantic-plan` will eventually need +develop's recent work merged in: + +- **#366:** TAS support, and DFN-driven rules instead of hardcoded ones. This is + partly done already: pydantic-plan's codegen matches develop apart + from the decorator. +- **#371:** creates a component's parent directory before writing it. +- **#372:** writes floats losslessly by default (`WriteContext`, + `filters.py`). +- **#373:** trailing-text support. Its type-driven option parsing is + cleaner than pydantic-plan's version (which matches `auxiliary` by name + and branches on `to_field_type`), so pydantic-plan should adopt + develop's version. + +## Not applicable (checked, nothing to port) + +- **Loop-variable shadowing in `structure.py`:** `for name, f in ...` + shadowed `structure_component()`'s `name` parameter. It exists only on + pydantic-plan, because develop iterates `attrs.fields()` and uses `f.name`. diff --git a/docs/dev/plans/archive/dimension-issues.md b/docs/dev/plans/archive/dimension-issues.md new file mode 100644 index 00000000..4a7c695a --- /dev/null +++ b/docs/dev/plans/archive/dimension-issues.md @@ -0,0 +1,155 @@ +# Dimension follow-ups (issue drafts) + +Two open dimension items left over from #167 (the data model refactor). +Each section below is written to be pasted as an issue. Checked against +develop `407102d` on 2026-10-08. + +**Status (2026-10-19): done except one item; archived.** + +- Issue 1 is done (#442, branch `dims-from-schema`): codegen gives + `DimsProviderMethods` to components with model- or simulation-scoped + dims (devtools #375), so there's no hand-kept provider list; `Ncf` and + package-local dims like `maxbound` aren't providers, so they don't + collide. +- Issue 2: assigning a dimension that disagrees with its rows raises + (checked on develop `e0d94b86`: `t.nper = 7` and `a.maxats = 1` with two + rows both raise `ValueError`). Still open: after an in-place append, + `t.nper` stays stale until `get_dims()`, and the docs don't say so. + That item moved to [`followups-after-row-layout.md`](followups-after-row-layout.md), item 5. + +--- + +## 1. Scope dimension resolution so any package can provide dimensions + +### Current state + +Dimension resolution (`DimensionResolverMixin` in `flopy4/dimensions.py`) +walks a component's direct children for `DimensionProvider`s, then its +parent, then its own `dims`. Only four classes are providers, each with a +hand-written `get_dims()`: `Dis`, `Disv`, `Disu` (`grid_dims_methods.py`) +and `Tdis` (`tdis_methods.py`). Codegen config lists which classes get +these methods. + +Other packages declare DIMENSIONS fields too, but they don't provide them. +Arrays elsewhere that use those dimensions can't resolve them through the +tree. + +### Problem + +A generic `Package.get_dims()` that returned every DIMENSIONS field would +make every package a provider. `_get_all_dimensions` raises when two +children of one component provide the same name, and several names repeat: + +| Dimension | Declared by | +| ---------- | --------------------------------------------- | +| `maxbound` | `Api`, `Chdg`, `Drng`, `Ghbg`, `Rivg`, `Welg` | +| `nexg` | `Gwfgwf`, `Gwegwe`, `Gwtgwt` | +| `ncpl` | `Disv`, `Ncf` | +| `nlay` | `Dis`, `Disv` | +| `nvert` | `Disv`, `Disu` | +| `nrow`/`ncol` | `Dis`, `Laktab`, `Sfrtab` | + +Some of these are the same quantity declared twice, like `Ncf`'s `ncpl`, +which repeats the grid's. Others share a name but mean different things, +and one model can hold several of them. A model with a `Chdg` and a `Welg` +has two different `maxbound`s, and a `Laktab`'s `nrow` is a table's row +count, not the grid's. + +`maxbound` is also a computed property on list packages, not a field, so +it isn't a dimension anyone else uses. + +### Options + +1. **Provide only dimensions other components use.** Mark a DIMENSIONS + field as shared (from the DFN, or derived from which other components' + arrays reference it). Generic `get_dims()` returns only shared + fields, and the hand-written providers go away. A package-local + dimension like `maxbound` stays local. +2. **Scope by name.** Qualify a dimension by its provider (`chdg.maxbound`) + when it isn't one of a known set of model or simulation dimensions. + This is more general but changes every lookup. +3. **Keep the provider list in codegen config.** This is the status quo. + It works, but each new provider needs a config entry and a methods + mixin. + +Option 1 seems smallest. The DFN may already say enough: a dimension +another component uses appears in that component's `shape`. + +### Done when + +- No hand-written provider list. Providers come from the DFNs. +- `Ncf`'s `ncpl` either agrees with the grid's or raises, instead of + colliding. +- Two same-named package-local dimensions on one model don't raise. + +--- + +## 2. Linked dimensions after construction + +### Current state + +A list field's row count sets a linked DIMENSIONS field (TDIS +`perioddata` → `nper`, DISV `vertices` → `nvert`, ATS `perioddata` → +`<=maxats`). This happens at construction, in `Package._init_item_lists`. +After that, `Package._sync_dims` updates the dimension, but only when +something reads dimensions (`get_dims()`, writing). Appending to the list +in place doesn't trigger `on_setattr`, so the sync can't happen on the +change itself. + +### Problems + +Checked on develop with `Tdis` and `Ats`: + +1. **The attribute is stale until something calls `get_dims()`.** + ```python + t = Tdis(perioddata=[r1, r2]) + t.perioddata.append(r3) + t.nper # 2, though there are three rows + t.get_dims() # {'nper': 3}, and t.nper is now 3 + ``` +2. **Setting an exact dimension is silently thrown away.** + ```python + t = Tdis(perioddata=[r1, r2]) + t.nper = 7 # accepted + t.get_dims() # {'nper': 2}; t.nper goes back to 2 + ``` + At construction, `Tdis(nper=7, perioddata=[r1, r2])` raises. + Assignment should behave the same way. +3. **Setting a bounded dimension too low is caught late.** + ```python + a = Ats(perioddata=[row, row]) + a.maxats = 1 # accepted + ... # raises later, wherever dimensions are next read + ``` + The error points to the dimension read, not to the assignment. + +### Options + +- **Check on assignment.** Add an `on_setattr` hook to linked dimension + fields that checks the value against the current row count, just as + construction does. Problems 2 and 3 then raise right away. Codegen + already knows which fields are linked (`dim` metadata on the list field). +- **Make exact dimensions read-only.** An exact linked dimension (like + `nper`) is always the row count, so it could be a property, not a field + the user sets. Problems 1 and 2 go away. This changes the constructor + (`nper=` would only be checked, never stored) and the generated class + shape. +- **Observable lists.** Item lists notify their package when they change. + This fixes problem 1 directly, but it's the most machinery, and it brings + back the proxying that #167 removed. + +Checking on assignment, plus reading through `_sync_dims` (or a property) +wherever the attribute is read, covers all three without new list types. + +### Done when + +- `t.nper = 7` with two rows raises at the assignment. +- `a.maxats = 1` with two rows raises at the assignment. +- `t.nper` matches `len(t.perioddata)` after an in-place append, or the + docs say it doesn't until the next dimension read. + +### Related + +- [`list-dim-default-problem.md`](list-dim-default-problem.md): the construction-time half, + where an explicit value and a default were indistinguishable. That's + resolved (`dim_default` metadata). diff --git a/docs/dev/plans/archive/dis-base-class-removal-plan.md b/docs/dev/plans/archive/dis-base-class-removal-plan.md new file mode 100644 index 00000000..8f167238 --- /dev/null +++ b/docs/dev/plans/archive/dis-base-class-removal-plan.md @@ -0,0 +1,248 @@ +# Removing DisBase: generating the discretization packages + +Follow-up to the TDIS/NCF work on `codegen-mixins` +([`codegen-base-class-removal-plan.md`](codegen-base-class-removal-plan.md)). Same goal: generated classes carry +no hand-written structure, and flopy API methods come from method-only +mixins listed in codegen's `MIXINS` table. + +**Status:** planned; nothing implemented. + +## Current state + +- `flopy4/mf6/gwf/disbase.py` (`DisBase(Package)`): + 1. declares derived dimensions `nlay`/`nrow`/`ncol`/`ncpl`/`nvert`/`nodes` + as `init=False` fields, which subclasses override or set in post-init + 2. `_coerce_griddata()`: list → ndarray, repeat a per-layer value + (size `nlay`) across `ncpl` for `layered` arrays, flatten anything with + more than one dimension, then `_broadcast_griddata()` + 3. `write()`: sets `ncf6_filerecord` from an attached `Ncf` child, then + writes the child at full precision + 4. `to_grid()` / `get_dims()` stubs +- Hand-written `Dis`/`Disv` in `gwf/`, identical copies in `gwt/` and `gwe/`, + and `prt/` copies without ncf and with `netcdf=False`. Each has its own + `get_dims`, `to_grid`, `from_grid`. Disv adds `iv`/`xv`/`yv`, + `cell2ddata`/`Cell2dRecord` and the `disv_to_grid_cell2d` / + `grid_to_disv_cell2d` helpers. +- No Disu class exists. +- The models annotate `dis: DisBase | None`. +- `dfn2py._SKIP` lists `gwf-dis`, `gwf-disv`, `gwt-dis`, `gwe-dis`, + `prt-dis`. The other hand-written disv modules survive regeneration only + because `--existing-only` skips files without the autogenerated header. + +## Findings + +### What codegen produces today + +Generated into a scratch directory from the dev3 DFNs with the skip list +emptied. The fields are close to the hand-written ones. The gaps: + +| | hand-written | generated | +|---|---|---| +| dis `top` shape | `("ncpl",)` | `("ncol", "nrow")` | +| dis `botm`/`idomain` shape | `("nodes",)` | `("ncol", "nrow", "nlay")` | +| disv `botm`/`idomain` shape | `("nodes",)` | `("ncpl", "nlay")` | +| derived dims | `init=False` fields + `get_dims` | none | +| ncf path | `ncf6_filerecord` | `ncf_file` | +| grb path | none | `grb_file` | +| ncf child | `ncf: Optional[Ncf]` | none | +| disv `cell2d` | raw tuples, `init=False` | `list[Cell2d]` with `icvert: int` (bug) | +| disv `vertices` | `list[Vertices]` built from `iv`/`xv`/`yv` | `list[Vertices]`, `dim="nvert"` | +| defaults | `xorigin=0.0`, `idomain=1` (dis) | `None`, `None` | + +### The DFN declares the derived dimensions + +`Component.dims` carries each dimension's expression: + +- gwf-dis: `ncpl = nrow * ncol`, `nodes = nlay * nrow * ncol`, `ncelldim = 3` +- gwf-disv: `nodes = nlay * ncpl`, `ncelldim = 2` +- gwf-disu: `ncelldim = 1`, and component-scoped `njas = (nja - nodes) / 2` +- olf-dis2d: `ncpl = nodes = nrow * ncol` + +So `get_dims` and the derived dimensions can be generated. Scope doesn't +pick out the grid packages: `maxbound` and friends are model-scoped too. +Departure 4 of the TDIS plan still holds (a default `get_dims` on every +package makes them all providers and names collide), so which components +provide dimensions stays explicit in codegen config. + +### Flat griddata + +Griddata is stored flat: `("nodes",)` or `("ncpl",)`. That's a flopy4 +convention, not a necessity. It matches MF6's own storage, and every other +package's DFN already gives griddata as `nodes` (NPF `k`, IC `strt`, ...). +Only the dis/disv DFNs spell out the structured shape, in Fortran order +(`ncol, nrow, nlay`). Egress reshapes a flat layered array to +`(nlay, ncpl)` to write `LAYERED`, and `netcdf.py` reshapes flat data +per-dimension. If dis stored `(nlay, nrow, ncol)` instead, `botm` and NPF +`k` would have different shapes on the same grid. Keep flat. + +### Disv follows the DFN + +The DFN has `vertices` (rows of `iv`, `xv`, `yv`) and `cell2d` (rows of +`icell2d`, `xc`, `yc`, `ncvert`, `icvert...`). Hand-written Disv has its own +inputs for the same data, which go away, as in OC and TDIS: + +- `iv`/`xv`/`yv` (the `vertices` columns as separate 0-based arrays) are + replaced by `vertices`. Columnar input still works through list + coercion: `Disv(vertices={"iv": iv, "xv": xv, "yv": yv})`. +- `cell2ddata`/`Cell2dRecord` is replaced by `cell2d`, a list of the + generated `Cell2d` Item. Hand-written Disv couldn't use a generated Item + for this only because of the `icvert` bug below. + +Callers (~70: tests, `utils/grid.py`, `docs/examples/circle.py`) move to +`vertices=` / `cell2d=`. No aliases. + +### Codegen bug: `icvert` generated as a scalar + +In the DFN, `icvert` is an integer `Array` with `shape=["ncvert"]`, the last +field of the `cell2d` record: an inline array sized by an earlier column of +the same row. Each row has as many trailing values as its `ncvert`: + +``` +CELL2D + 1 0.5 0.5 4 1 2 5 4 + 2 1.5 0.5 3 2 3 5 +``` + +Codegen turns every record field into a scalar Item attribute, so it emits +`icvert: int`, which can't hold the row's vertices. It should emit an +array-typed column. Fix: + +- Item: `icvert: tuple[int, ...]` with `shape` metadata naming the count + column (`ncvert`); index renumbering applied per element (as cellid + already does). +- Ingress: the row reads the count, then that many values (or the rest of + the line; a variable-length column must be last). +- Egress: flatten the tuple into the row. +- `ncvert` derived from `len(icvert)` when omitted. +- Codegen fails on a variable-length column that isn't last. + +Disu's `cell2d` has the same shape. Check the corpus for other inline +arrays in records; they get the same treatment. + +### The subpackage link isn't in the dev3 DFN + +The legacy `gwf-dis.dfn` starts with `# mf6 subpackage utl-ncf`. The dev3 +spec drops it: `ncf_filerecord` is a plain record (`ncf6` keyword + +`ncf6_filename` file), and `utl-ncf`'s parent is `sim-nam`. So nothing in +the dev3 DFN says this record names an NCF package. Options: + +- derive it: a `filein` record whose keyword is `6` names `utl-` + when that component exists (`ncf6` → `utl-ncf`, `ts6` → `utl-ts`, + `obs6` → `utl-obs`). No config, but it also turns TS/OBS file records + into child fields, which is a bigger change than this plan needs. +- a codegen table, like `MIXINS`: `SUBPACKAGES = {"ncf_filerecord": + "utl-ncf"}`. Smallest; NCF only. +- carry the annotation through the devtools migration (e.g. a `subpackage` + attribute on the record). Right long-term, but it's devtools work. + +Start with the codegen table; move to the devtools attribute later. The +devtools fix is written up in `../modflow-devtools/subpackage-links-plan.md` +(`File.component`, set from both legacy annotations). + +### Lists → arrays + +`Package._broadcast_griddata` handles scalars, ndarrays and empty dicts, and +leaves a list alone. `_coerce_griddata` converts lists so `Dis(delr=[...])` +works. Generated packages have the same gap today (`Npf(k=[...])` stays a +list), so the conversion belongs in `Package`. + +## Design + +### Codegen + +1. **Canonical griddata shapes.** For a component declaring derived dims, a + multi-name griddata shape whose names are exactly the factors of a + declared dimension's product expression becomes that dimension: + `(ncol, nrow)` → `ncpl`, `(ncol, nrow, nlay)` → `nodes`, + `(ncpl, nlay)` → `nodes` (via `nodes = nlay * ncpl`). Match on the + multiset of names, not order. Fail codegen if a griddata shape has more + than one name and matches nothing. (Alternatively devtools normalizes + these in the dev3 migration; decide with the devtools work.) +2. **Derived dimensions.** Emit a `ClassVar` of the component's + non-trivial dim expressions, e.g. + `_derived_dims = {"ncpl": "nrow * ncol", "nodes": "nlay * nrow * ncol", + "ncelldim": "3"}`, and `init=False` read-only properties for the ones + that aren't DIMENSIONS fields. Arithmetic only (`+ - * /`, names, + integer literals); evaluate with a small parser, not `eval`. Skip + lookups like `sum(packagedata.nlakeconn)`. +3. **Dimension providers.** A `GridDimsMethods` mixin supplies + `get_dims()` from the DIMENSIONS fields plus `_derived_dims`. Listed in + `MIXINS` for the grid packages only (dis, disv, disu, dis2d, disv1d, + disv2d). TDIS keeps its own. +4. **Subpackage fields.** For an options path record listed in a + codegen `SUBPACKAGES` table (NCF only, for now; see above), emit `ncf: Optional[Ncf]` next to the path + field. `Package.write` syncs the path from the child and writes the + child, generalizing `DisBase.write`. The full-precision rule for NCF + coordinates moves to `Ncf` (via `NcfMethods` or a field-level metadata + flag) rather than living in the parent. +5. **Array columns in Items** (the `icvert` bug above). +6. Remove the dis entries from `_SKIP`; generate gwf/gwt/gwe/prt dis and + disv, and disu. CHF/OLF grids come along if their model modules exist. + +### Runtime (`Package`) + +- In `__attrs_post_init__`, before broadcasting: convert list/tuple + griddata to arrays; repeat a size-`nlay` value per layer for `layered` + arrays; flatten arrays with more than one dimension. Use `self.get_dims()` + when the package is a `DimensionProvider`, else `dims`. This replaces + `_coerce_griddata` for every package. + +### Mixins + +- `DisMethods`: `to_grid`, `from_grid` (built with `cls(...)`, so one + mixin serves gwf/gwt/gwe/prt). +- `DisvMethods`: `to_grid`, `from_grid`, and the `Cell2d` ↔ grid + conversion helpers. +- `DisuMethods`: `to_grid`/`from_grid` if `UnstructuredGrid` supports it; + otherwise none yet. + +### Models + +- `dis: Dis | Disv | Disu | None` (per model module), and the `convert_grid` + converters keep dispatching on the grid type. +- `Gwf` file-extension logic (`"disv" if isinstance(..., Disv)`) keeps + working; could use `dfn_name` instead. + +## Work items + +1. **Shapes and derived dims** (codegen 1–3, runtime list/layer coercion). + Useful on its own; changes no generated class outside the grid packages + except that list griddata now becomes arrays. +2. **Generate dis** with `DisMethods` in all four models. Delete the + hand-written `dis.py` files. Decide on the field renames + (`ncf6_filerecord` → `ncf_file`) and defaults (`xorigin`, `idomain`); + the NCF child needs item 4, so either do item 4 first or keep a + hand-written `ncf` field temporarily (which breaks the goal; prefer + ordering item 4 first for gwf/gwt/gwe). +3. **Fix the `icvert` codegen bug**, then **generate disv** with + `DisvMethods`, and **disu**. +4. **Subpackage fields** (NCF child), generalizing `DisBase.write`. +5. Delete `disbase.py`; update the model annotations, the codegen test's + `DIS_TIER` comment, and the `_SKIP` comments in `dfn2py.py`. + +Suggested order: 1, 4, 2, 3, 5. + +## Decisions + +- **Shape mapping stays in flopy4 codegen** (design item 1). The DFN's + structured shape records how MF6 reads the array, and devtools shouldn't + drop that; storing griddata flat is flopy4's choice. +- **Accept the generated names and defaults.** `ncf_file` and `grb_file` + follow the `*_file` convention of every other generated path field + (`ts_file`, `density_file`, ...). `xorigin`/`yorigin`/`idomain` default to + `None`, meaning not written, which MF6 treats as 0.0 / all cells active, + so model behavior is unchanged. Overriding them would mean hand-written + fields again. Callers reading `dis.idomain` or `dis.xorigin` must handle + `None`; `to_grid` already does for `idomain`. +- **`nvert` longname:** a DFN fix upstream, described in + `../modflow6/dfn-nvert-longname.md`. + +## Verification + +- `pixi run -e dev smoke` after each item, and the full suite before + committing item 2 and item 3 (dis/disv round-trips are in the + integration tests). +- Codegen tests: add `DIS_TIER` cases asserting base `Package`, the + mixins, flat shapes, and `_derived_dims`. +- Round-trip a disv model with pentagonal cells (the `circle` example) to + exercise variable-length `icvert`. diff --git a/docs/dev/plans/archive/disv-plan.md b/docs/dev/plans/archive/disv-plan.md new file mode 100644 index 00000000..d62c588b --- /dev/null +++ b/docs/dev/plans/archive/disv-plan.md @@ -0,0 +1,65 @@ +# DISV: status after #391 + +Re-evaluated against `origin/develop` at `98f53b9` (2026-10-02). #391 ("generate +dis packages") did the work this plan proposed, so there is nothing left to +plan for the core fix. + +## What #391 delivered (verified) + +- `Disv` (gwf/gwt/gwe/prt) is generated from the DFNs; `DisBase` is gone. +- Stored data matches the DFN: `vertices: list[Vertices]`, `cell2d: + list[Cell2d]`. No duplicate `iv/xv/yv` arrays or `cell2ddata`. +- Indices are stored 0-based and renumbered at the I/O boundary: `Item` now + handles an index *array* (`icvert`, `array=True, index=True`) and a count + column (`ncvert`, `count="ncvert"`), which was the `Item` extension I had + proposed. +- Methods live in mixins (`DisvMethods`, `GridDimsMethods`), registered like + `TdisMethods`; derived dimensions use the new `DerivedDim` descriptor; the NCF + child is a `subpackage(file_field="ncf_file")`, which was the blocker I had + flagged for the mixin approach. +- The original bugs are fixed. On `develop`: write a `Disv`, `Disv.load` it, and + `to_grid()` works, re-writing keeps the `CELL2D` block, and `vertices`/`cell2d` + round-trip. + +## What is still open + +1. **Equality across a round trip.** A hand-built `Disv` and the same grid + loaded from its written file compare unequal. The only differing field is + `cell2d`: built `Cell2d(ncvert=None)` vs loaded `Cell2d(ncvert=3)`. + `ncvert` is a count column derived from `len(icvert)`, so it shouldn't be an + independent stored value. Options: normalize `ncvert` in post-init (or a + converter) to `len(icvert)`; or exclude it from comparison with `eq=False`. + I'd normalize it so the stored value is always consistent. +2. **Dropped API.** `iv=`, `xv=`, `yv=` and `cell2ddata=` constructor inputs, + the read-only `iv/xv/yv` views, and `Cell2dRecord` no longer exist. If you + want any of them back for convenience, it's a small addition to + `DisvMethods`. Otherwise this is a deliberate break and worth noting in + release notes. +3. **Array-equality branch conflicts.** `fix/array-equality` conflicts with + `develop` in `spec.py` and the four `disv.py` files (`merge-tree` check). The + `disv.py` edits (`eq=ARRAY_EQ` on `iv/xv/yv`) are obsolete (those fields are + gone) and should be dropped; `spec.py` needs a manual merge. After the rebase + I need to re-check that every array field still has `shape` metadata + (#391 flattened shapes and changed arrays to `IntArrayLike`/`FloatArrayLike`), + since the default comparer keys on it. +4. **#380 (draft)** still edits the four `disv.py` files and adds `Cell2d` / + derive-on-load code that #391 supersedes; it needs a substantial rebase or + a rewrite of its DISV parts. It also adds `grb_filerecord` to DIS/DISV; + #391 already renamed these to `*_file`. +5. **Structured `Dis`/`Disu`** were generated in the same PR, so the earlier + "do `Dis` as a follow-up" question is moot. + +## Status (2026-10-05) + +- Item 1 (`ncvert` equality): `fix/disv-ncvert` fills count columns from the + array they count, and keeps them in step on reassignment (`on_setattr` hook + from `spec.field(count=...)`). Counts that name a package dimension + (GNC, EVT) are skipped. Pushed; no PR yet. +- Item 2 (dropped API): `fix/disv-legacy-api` restores `iv`/`xv`/`yv`/ + `cell2ddata` inputs (via a metaclass on `DisvMethods`), read-only views and + `Cell2dRecord`. Pushed; no PR yet. The views are read-only. +- Item 3 (array equality): merged as #394; `fix/array-equality` is obsolete. +- Item 4 (#380): still needs a rebase or rewrite of its DISV parts. +- Not started: reconcile list-length dimensions (`nvert`, `ncpl`, `nper`) from + row counts at `get_dims()` and write time, since mutating a list can't + trigger an `on_setattr` hook. diff --git a/docs/dev/plans/archive/ecosystem-sync-plan.md b/docs/dev/plans/archive/ecosystem-sync-plan.md new file mode 100644 index 00000000..b7346432 --- /dev/null +++ b/docs/dev/plans/archive/ecosystem-sync-plan.md @@ -0,0 +1,527 @@ +# Ecosystem Sync Plan + +**Archived 2026-10-19.** Done: the stacked branches merged (#425, #427, #429, #434, #439, #440, #441, #445, #446, #448, among others), including lazy imports and the TOML spec. Only upstream parts remain (`mf6 --spec`), plus the idea in [`user-dir-classes.md`](../components/user-dir-classes.md). The status line below, "pending review", is out of date. + +Discussion: https://github.com/modflowpy/pyphoenix-project/discussions/81 + +**Status (2026-10-08):** the six steps below are done on stacked branches `sync-python-api` → `sync-safe-writes` → `sync-contract` → `sync-check-at-run` → `sync-install` → `sync-ci-matrix`, pending review. Before that (develop `2d3d4dc`) it was partly implemented, in a +different shape from the design below. The CLI, the contract file and the +compatibility check exist. The direction for the rest was revised on +2026-10-08; see [Revised direction](#revised-direction-2026-10-08) and +[Still to do](#still-to-do). The Design section is kept as the original +intent. + +## Motivation + +The version compatibility contract between flopy and MF6 is currently implicit. Users must manually coordinate `generate_classes` (syncs flopy to a specific MF6 DFN version) and `get-modflow` (installs a specific MF6 binary), with no enforcement or visibility into whether the two are aligned. Flopy and MF6 must be released in lockstep for users to get a coherent combination. + +This plan decouples flopy4 and MF6 releases. + +## Design + +### Namespace layout + +The Programs API is simulator-agnostic (manages mf6, mt3dms, gridgen, etc. uniformly). The DFNs and Models APIs are MF6-specific. This maps onto: + +```python +flopy4.programs # generic: re-export of modflow_devtools.programs +flopy4.mf6.dfns # mf6-specific: re-export of modflow_devtools.dfns +flopy4.mf6.models # mf6-specific: re-export of modflow_devtools.models +flopy4.mf6.sync() # install binary + regenerate classes for a given MF6 version +``` + +`flopy4.programs` sits at the top level so future simulator namespaces (e.g. `flopy4.mfnwt`, `flopy4.mt3d`) can share it without going through the MF6 namespace. + +### The version contract + +`MF6_CONTRACT_VERSION` is a static string written into the package by `sync()` and committed to the repo. It is read at import time and compared against the discovered binary. It lives in a generated sidecar so handwritten source stays clean: + +```python +# flopy4/mf6/_contract.py — written by sync(), committed, never edited by hand +MF6_CONTRACT_VERSION = "6.6.0" +MF6_DFN_SCHEMA_VERSION = "2" +``` + +```python +# flopy4/mf6/__init__.py +from flopy4.mf6._contract import MF6_CONTRACT_VERSION, MF6_DFN_SCHEMA_VERSION +``` + +### Writing to the install location + +`sync()` writes generated classes and `_contract.py` directly into the installed package directory in site-packages — the same approach used by current flopy's `generate_classes`. In practice this works because Python packages almost always live in a virtualenv, where site-packages is user-writable. + +If the install location is not writable (system Python, read-only container), `sync()` raises a clear error: + +``` +SyncError: flopy4 is installed to a read-only location ({path}). +Install into a virtual environment and try again, or use +`pip install --user flopy4` for a user-writable install. +``` + +No silent fallback to a user data directory — a partial sync (contract updated, classes not updated, or vice versa) would be worse than a clear failure. + +After a `pip install --upgrade flopy4`, the bundled classes and `_contract.py` are restored to the release version. The compatibility check will then prompt the user to re-sync if their binary is newer. + +### Runtime compatibility check + +On first use of `flopy4.mf6`, flopy4 resolves the MF6 executable (PATH lookup via `shutil.which`, matching existing flopy behavior) and queries its version with a subprocess call (`mf6 -v`). If the reported version doesn't match `MF6_CONTRACT_VERSION`, a `UserWarning` is emitted with a clear call to action. Silent if no binary is found. + +```python +def _check_mf6_compatibility(): + exe = shutil.which("mf6") + if exe is None: + return + binary_version = _query_mf6_version(exe) # subprocess: mf6 -v, parse stdout + if binary_version != MF6_CONTRACT_VERSION: + warnings.warn( + f"flopy4 is synced to MF6 {MF6_CONTRACT_VERSION} " + f"but the discovered binary reports {binary_version}. " + f"Run `flopy4 sync` to regenerate classes for MF6 {binary_version}.", + UserWarning, + stacklevel=2, + ) +``` + +### The sync operation + +`sync()` is user-facing. It installs the requested MF6 binary, fetches DFNs for that version, regenerates classes in-place, and updates `_contract.py`: + +```python +def sync(version: str | None = None, bindir: Path | None = None) -> SyncResult: + """Sync flopy4 to a specific MF6 version. + + Installs the MF6 binary, fetches matching DFNs, and regenerates + input/output classes in the installed package directory. + If version is omitted, syncs to the version of the discovered binary. + """ + if version is None: + version = _query_mf6_version(shutil.which("mf6")) + + _check_install_writable() # raises SyncError with helpful message if not + + # 1. install binary + exes = install_program("mf6", version=version, bindir=bindir) + + # 2. query installed binary — this is the source of truth for MF6_CONTRACT_VERSION + contract_version = _query_mf6_version(exes[0]) + + # 3. fetch DFNs for the same ref + registry = RemoteDfnRegistry(ref=version) + registry.sync() + + # 4. regenerate classes in-place + outdir = Path(__file__).parent # flopy4/mf6/ in site-packages + _generate_classes(registry, outdir=outdir) + + # 5. update contract + _write_contract(outdir, version=contract_version, dfn_schema=registry.schema_version) + + return SyncResult(version=contract_version, exes=exes) +``` + +`MF6_CONTRACT_VERSION` is always the version string reported by the installed binary +(`mf6 -v`), never the ref string passed by the user. This means `"latest"`, +`"develop"`, a tag, or a commit hash are all valid refs — the contract always +records the concrete version the binary reports, which is the single source of +truth tying the binary, the DFNs, and the generated classes together. + +### CLI surface + +``` +flopy4 sync # sync to discovered binary's version +flopy4 sync --version 6.6.0 # sync to a specific MF6 release +flopy4 sync --version develop # sync to MF6 develop (CI use case) +flopy4 status # show contract version vs discovered binary +``` + +## Prerequisites (modflow-devtools gaps) + +1. **`query_program_version(exe: Path) -> str`** — call `mf6 -v` and parse stdout, capturing the full version string including any `+shortsha` suffix. Needed both for the runtime compatibility check and for `sync()` when no version is specified. + +## Version alignment roadmap + +The current design works well for tagged releases: `MF6_CONTRACT_VERSION = "6.6.0"` is unambiguous. It breaks down for `develop` builds, because the DFNs and the binary are sourced independently — two users both synced to `develop` may have different DFN snapshots and different binaries, with no way to detect the mismatch. + +### Near-term: commit-grained version strings + +The most targeted fix requires coordinated changes in two places: + +**1. MF6 binary (upstream) — Meson `vcs_tag()` with a helper script** + +A `vcs_tag()` call in `meson.build` runs a small Python script at build time +that produces a `+shortsha` suffix for development builds and an empty string +for releases: + +```meson +# meson.build +version_f90 = vcs_tag( + command: ['python3', 'distribution/vcs_tag_suffix.py'], + input: 'src/Utilities/version.f90.in', + output: 'version.f90', + replace_string: '@VCS_TAG@', + fallback: '', +) +``` + +```python +# distribution/vcs_tag_suffix.py +# If HEAD is an exact tag match this is a release — no suffix. +# Otherwise it's a development build — emit '+shortsha'. +try: + subprocess.check_output(['git', 'describe', '--exact-match', '--tags', 'HEAD'], + stderr=subprocess.DEVNULL) + print('', end='') +except subprocess.CalledProcessError: + sha = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode().strip() + print(f'+{sha}', end='') +``` + +```fortran +! src/Utilities/version.f90.in +character(len=*), parameter :: VERSIONTAG = "@VCS_TAG@" +character(len=*), parameter :: VERSION = VERSIONNUMBER//VERSIONTAG +``` + +This produces PEP 440-style local version labels: + +| Build | `mf6 -v` output | +|-------|----------------| +| Release (exact tag) | `6.6.0` | +| Development | `6.8.0.dev0+abc1234` | + +`vcs_tag()` re-runs at every build (not just configure), so the embedded hash +always reflects the current HEAD. The `fallback: ''` ensures a clean release +string when building outside a git repository. + +**2. flopy4 — store and check the full version string** + +`_contract.py` records only the full version string from the binary and the +DFN schema version — no separate commit field is needed: + +```python +# flopy4/mf6/_contract.py — written by sync() +MF6_CONTRACT_VERSION = "6.8.0.dev0+abc1234" # full version including +shortsha suffix +MF6_DFN_SCHEMA_VERSION = "2" +``` + +The compatibility check is a direct string comparison of the binary's reported +version against `MF6_CONTRACT_VERSION`. For release builds the string is a +plain semver (`"6.6.0"`); for development builds it includes the `+shortsha` +suffix — both cases are handled identically with no special-casing. + +`_query_mf6_version()` must capture the full version string including the +`+shortsha` suffix (a regex matching only `\d+\.\d+\.\d+` would truncate it): + +```python +_MF6_VERSION_RE = re.compile(r"(\d+\.\d+\.\d+(?:\.\w+)?(?:\+[0-9a-f]+)?)") +``` + +A separate `MF6_DFN_COMMIT` field is not needed. For dev builds the shortsha +is already embedded in `MF6_CONTRACT_VERSION`; for tagged releases there is no +shortsha and `MF6_CONTRACT_VERSION` alone is sufficient. Storing the commit SHA +redundantly would require an extra GitHub API call per sync (two calls for +annotated tags) and would still be useless for the release case. + +This approach requires no changes to individual DFN files, no new fields on +`DfnRegistryMeta`, and no new CI infrastructure beyond Meson's built-in +`vcs_tag()` support. + +### Mid-term: self-describing MF6 (`mf6 --spec`) + +> **Status:** The `mf6 --spec` flag is implemented and working as of MF6 +> develop (branch `emit-spec`). See `../modflow6/utils/dfnembed/plan.md` for +> the full MF6-side plan and `../modflow-devtools/dfnspec-plan.md` for the +> devtools serialization work. + +The cleanest long-term solution is having the MF6 binary emit its own input +specification directly, making the spec and the binary a single artifact: + +```bash +mf6 --spec # emit full spec to stdout +mf6 --spec > spec.txt # capture to file +mf6 --spec | flopy4 sync # pipe directly to sync +``` + +Version alignment becomes structurally guaranteed — no registry fetch, no +cross-referencing commit hashes, no `develop` pointer ambiguity. Works offline +(valuable in HPC/air-gapped environments). + +**Current output format: DFN text with sentinels** + +The initial implementation embeds the raw DFN text verbatim and emits it as a +single stream, with `# --- ---` comment lines separating each component +file (these are legal DFN comments, ignored by all existing parsers): + +``` +# --- gwf-chd.dfn --- +# --------------------- gwf chd options --------------------- +... + +# --- gwf-dis.dfn --- +... +``` + +`modflow_devtools` can already parse this format. To feed it into the current +`sync()` pipeline (which expects a directory of DFN files), split on sentinels +and write to a temp directory: + +```python +def spec_to_dir(mf6_exe="mf6") -> pathlib.Path: + text = subprocess.check_output([mf6_exe, "--spec"], text=True) + tmp = pathlib.Path(tempfile.mkdtemp()) + current_name, current_lines = None, [] + for line in text.splitlines(): + if line.startswith("# --- ") and line.endswith(" ---"): + if current_name: + (tmp / current_name).write_text("\n".join(current_lines)) + current_name = line[6:-4] + current_lines = [] + else: + current_lines.append(line) + if current_name: + (tmp / current_name).write_text("\n".join(current_lines)) + return tmp +``` + +**Future output format: single TOML document** + +Once `DfnSpec.dump()` is added to `modflow_devtools`, `mf6 --spec` will switch +to emitting a single TOML document — no sentinels, no splitting, parseable +directly with `tomllib`: + +```bash +mf6 --spec | python -c "import sys, tomllib; spec = tomllib.load(sys.stdin.buffer)" +``` + +The schema falls directly out of `DfnSpec._flat`: each component's `asdict()` +dict nested under its name with a top-level `schema_version`: + +```toml +schema_version = "2" + +["gwf-chd"] +name = "gwf-chd" +advanced = false +multi = true + +["gwf-chd".options.auxiliary] +block = "options" +name = "auxiliary" +type = "string" +... +``` + +**How `sync()` evolves** + +Once `mf6 --spec` emits TOML, `sync()` can drop the registry fetch entirely +and derive everything from the installed binary: + +```python +def sync(version: str | None = None, bindir: Path | None = None) -> SyncResult: + exes = install_program("mf6", version=version, bindir=bindir) + contract_version = _query_mf6_version(exes[0]) + + # replaces: registry = RemoteDfnRegistry(ref=version); registry.sync() + spec_toml = subprocess.check_output([str(exes[0]), "--spec"]) + + _generate_classes_from_spec(spec_toml, outdir=_MF6_PACKAGE_DIR) + _write_contract(_MF6_PACKAGE_DIR, version=contract_version, dfn_schema="2") + return SyncResult(version=contract_version, exes=exes) +``` + +`_generate_classes_from_spec()` loads the TOML blob via `DfnSpec` and passes +it to the existing class-generation machinery. No GitHub API call, no cache, +no `develop` pointer ambiguity. + +This approach generalises: any simulator that can emit its own spec gains +flopy4 support without a separate DFN pipeline. + +## Phases + +### What shipped + +- **CLI** (`flopy4/cli.py`, entry point `flopy4`): `flopy4 mf6 sync + [release_id]` and `flopy4 mf6 status`, not the planned top-level + `flopy4 sync` / `flopy4 status`. + - `release_id` is `owner/repo@tag` or a local DFN directory. With none, + it uses the discovered binary's version, else latest. + - Fetches DFNs through `RemoteDfnRegistry` (falling back to + `fetch_dfns` from the repo tree when the release has no `dfns.zip`), + or reads them with `LocalDfnRegistry`. + - Regenerates with `dfn2py.make` into the package directory, existing + files only unless `--all-packages`. + - `--mf6-version` overrides the recorded version. +- **Contract** (`flopy4/mf6/_contract.py`, written by sync): `MF6_VERSION` + and `DFN_SCHEMA_VERSION`, not `MF6_CONTRACT_VERSION` / + `MF6_DFN_SCHEMA_VERSION`. Currently `"develop"` / `"2.0.0.dev3"`. +- **Compatibility check** (`flopy4/mf6/_compat.py`): + `check_mf6_compatibility()` runs `mf6 -v` and warns on a mismatch. + +### Where it differs from the design + +- **The check runs at import** (`flopy4/mf6/__init__.py`), not lazily on + first use as Phase 2 planned. +- **The `+shortsha` suffix is dropped.** `_query_mf6_version` strips + everything after `+`, the opposite of the near-term roadmap below. +- **A branch-name contract skips the check.** When `MF6_VERSION` isn't + semver (e.g. `"develop"`, today's value), nothing is compared. So the + develop case the roadmap targets is unchecked. +- **The version recorded is the DFN source's**, from the tag or + `--mf6-version`, not the installed binary's `mf6 -v` as the design says. +- **An unknown version warns on import**, rather than staying silent. + +### Revised direction (2026-10-08) + +These decisions replace parts of the Design section, which is kept as the +original intent. They came from reviewing the design against flopy #2385 +(closed: install hooks, multiple executables, tests against generated +code), flopy #1970, and discussions #35 and #81. + +- **Sync doesn't install binaries by default.** People manage several + executables, so installing one shouldn't be tied to regenerating + classes, in either direction. Sync regenerates classes from one of three + sources: a release tag, a local DFN directory, or an executable + (`mf6 --spec`, once upstream ships it). Installing becomes opt-in + (`--install`), replacing `--no-install`. The `--spec` source replaces the + design's reason for installing: the spec and the version come from the + same binary. +- **In-place writes are atomic and tracked.** Generate into a temporary + directory and swap it in, so a failed sync can't leave classes from mixed + versions. Record the generated files in `_contract.py`, because files + that `--all-packages` creates aren't in pip's RECORD and would outlive a + `pip install --upgrade`, generated from a different version from the + restored contract. Sync and status detect and remove such orphans. +- **The compatibility check moves off import.** It runs at + `Simulation.run` against the executable actually used, cached per path, + and in `flopy4 mf6 status`. Import no longer runs a subprocess or warns + about an unknown contract. +- **Develop contracts are meaningful.** For branch refs, sync records the + resolved commit SHA (one API call per sync is cheap), which also gives + #214 its reproducibility string. The `+shortsha` suffix is kept rather + than stripped: compare base versions until MF6's `vcs_tag()` lands, then + compare exactly. Release CI fails if the contract doesn't name a tag. +- **DFNs are stored with the classes.** The typed reader currently fetches + DFNs over the network at runtime using `MF6_VERSION` as the ref + (`flopy4/mf6/codec/reader/dfns.py`). With `"develop"` that's a moving + target that can drift from the generated classes, and it fails offline. + Sync writes the spec it generated from (as TOML) next to the classes, + and the reader uses that copy. This also prepares for `mf6 --spec` + emitting TOML, and covers #81's "emit definition files" idea. +- **No re-exports.** Re-exporting `modflow_devtools.programs`, `dfns` and + `models` would tie flopy4's public API to devtools APIs that are still + experimental. Users import devtools directly; the CLI wraps what's worth + wrapping. +- **A CI version matrix replaces `validate=`.** Test against the latest + MF6 release and develop on every PR, and document which MF6 versions are + supported (per #1970). That does more for the "older model from a + partner" case in #35 than a validation option. + +### Still to do + +In order: + +1. ~~**Python API.**~~ Done on branch `sync-python-api`: + `flopy4.mf6.sync()` returns a `SyncResult` (version, source, outdir, + generated files) and the CLI wraps it. Sources: release ID or bare ref, + local DFN directory; `exe=` raises `NotImplementedError` until + `mf6 --spec` ships. The `--force` help text is fixed here too. +2. ~~**Safe writes.**~~ Done on branch `sync-safe-writes` (stacked on + `sync-python-api`). Generation goes to a `.sync-*` staging directory + inside the output directory (so ruff uses the project config and the + swap is a same-filesystem rename), is checked with `compile()`, then + swapped in with rollback. `SyncError` for read-only installs. The + contract records `GENERATED_FILES`. Orphans are generated modules that + neither the new sync nor the previous contract lists, and are removed. + Listed modules with no DFN in the new source are kept with a warning, + because handwritten `__init__`s import generated modules and removing + one could break `import flopy4`. Not done: making sync runnable when + the generated classes fail to import. `flopy4/__init__.py` imports + `flopy4.mf6`, so that needs lazy imports there; it's out of scope here. +3. ~~**Contract contents.**~~ Done on branch `sync-contract` (stacked on + `sync-safe-writes`). The contract records `DFN_COMMIT`: the GitHub + ref's commit (API call, `GITHUB_TOKEN` if set) for remote sources, + the checkout's HEAD with `-dirty` for local ones. Sync copies the raw + DFNs into `flopy4/mf6/dfns` (2.4 MB, package data). The typed reader + reads them, with no network fetch. TOML waits on devtools being able + to write it. `_query_mf6_version` keeps `+`. The check compares + commits when the contract and binary both have one, else base + versions. The release check is a test that only runs with + `FLOPY4_RELEASE` set, since there's no release workflow yet. +4. ~~**Compatibility check.**~~ Done on branch `sync-check-at-run` + (stacked on `sync-contract`). Import no longer runs the check. + `Simulation.run` checks its `exe`, resolved under the workspace like + `run_cmd` resolves it, and binary versions are cached by path and + mtime. An unknown contract now warns at run time, not import. +5. ~~**Opt-in binary install.**~~ Done on branch `sync-install` (stacked + on `sync-check-at-run`). `--install`/`--bindir` (`install=`, + `bindir=`) install before any classes change: the `modflow6` release + for a tag or `latest`, the latest `modflow6-nightly-build` for + `develop`. Local DFNs and other branches are refused. `--no-install` is + gone. Tested against real installs of 6.6.3 and the nightly. The + nightly reports `6.9.0.dev0+14b4a67.dirty`, so MF6 already embeds the + commit (the `vcs_tag()` roadmap item looks done upstream). The commit + check now ignores `.dirty`; that fix is on `sync-check-at-run`. +6. ~~**CI version matrix.**~~ Done on branch `sync-ci-matrix` (stacked + on `sync-install`). A `test-mf6-release` job runs `flopy4 mf6 sync + latest --install` and the tests, non-blocking for now. Locally against + 6.8.1, the only real failures are test020's MAW inputs, which are + develop-only. The test models come from the develop models registry. + Making the job blocking needs release-matched models, or skipping + develop-only ones. Both test jobs print `flopy4 mf6 status`. The + support policy is in `docs/mf6-versions.md`. Also fixed: `sync latest` + 404'd, because `latest` was passed to the DFN fetch as a git ref. + +Follow-ups: +- ~~`flopy4/mf6/pts.py`~~ deleted on `sync-safe-writes`. +- Make sync runnable when generated classes fail to import (lazy + imports in `flopy4/__init__.py`). Then, on the same branch, delete + generated modules the new DFNs lack instead of keeping them. Today + deleting is unsafe: syncing to 6.5.0 would delete `utl/ncf.py`, which + `utl/__init__.py` imports, breaking `import flopy4`. Sync itself needs + that import, so a re-sync couldn't repair it. Once sync works without + the generated classes importing, a bad sync is recoverable. (6.6.3, + 6.8.1 and develop lack none of the imported modules.) + **Done** on `sync-lazy-imports`: `flopy4.mf6` loads generated + components on first use (`__getattr__`; `get_fnames`/`get_ftypes` + import them all first, so lookups stay complete). Codegen's + `_has_module` checks the file rather than `find_spec`, which imported + `utl/__init__`. Sync removes every generated module it didn't + regenerate; `SyncResult.kept` is gone. A handwritten `__init__` that + imports a removed module still breaks using the classes, but sync and + status still run, so a re-sync fixes it. +- Store the spec as TOML once devtools can write it, and sync from + `mf6 --spec`. **TOML done** on `sync-toml-spec` (devtools fbec2ab, #386, + fixed the round trip): sync migrates a `.dfn` source to per-component + TOML in `dfns/` (or copies a TOML source); tests check the stored spec + equals the source and re-syncing from it reproduces the classes. + devtools only loads a directory, so it's 129 files, not one. `mf6 --spec` + still waits on modflow6. History: checked 2026-10-08: devtools can write it + (`modflow_devtools.dfns.migrate(..., "2.0.0.dev3", fmt="toml")`, `.dfn` + input only) and `Dfns.load` reads `*.toml`, with the same load time. + But the round trip is lossy, so this is blocked on devtools: + - `_serialize_safe` drops `name` from every field dict, including a + list's `item`, which isn't keyed by name; on load it inherits the + parent's (`gwf-oc` `output_record` → `output`, `prt-prp` + `releasesetting` → `perioddata`; 6 OC components plus `prt-prp`). + - Tuple defaults load as lists (`sim-tdis` `perioddata`), so codegen + renders `([1.0, 1, 1.0],)`. Fix in devtools (normalize on load) or + in flopy4 codegen (`make.py` `_default = repr(tuple(...))`, make rows + tuples). + Otherwise classes generated from the TOML match those from the `.dfn` + files exactly (develop, 129 components). Once fixed, sync migrates a + `.dfn` source to TOML in `dfns/` (copying a TOML source as is). Test: + re-syncing from the stored `dfns/` reproduces the classes. `mf6 --spec` + isn't in modflow6 develop yet. + +- Maybe, later: generate classes into a per-user, per-version directory + instead of the installed package. See [`user-dir-classes.md`](../components/user-dir-classes.md). + +Upstream parts (MF6 `vcs_tag()`, `mf6 --spec` as TOML) are tracked in the +modflow6 and devtools plans. + +## Notes + +- **`develop` ref**: Both the Programs and DFNs APIs support `develop` as a ref. `flopy4 mf6 sync MODFLOW-ORG/modflow6@develop` is a first-class use case for CI pipelines testing against nightly MF6 builds. +- **DFN schema versioning**: `_contract.py` records both the MF6 version and the DFN schema version. A future flopy4 / MF6 v2 schema transition won't silently mix incompatible generations (see devtools issue #259). +- **Post-upgrade re-sync**: After `pip install --upgrade flopy4`, bundled classes are restored to the new release version. If the user's binary is newer than the new release, the compatibility check will prompt them to re-sync. Modules a sync created that the release doesn't ship survive the upgrade; the generated-file list in the contract lets sync and status find and remove them. diff --git a/docs/dev/plans/archive/field-functions-plan.md b/docs/dev/plans/archive/field-functions-plan.md new file mode 100644 index 00000000..0def0686 --- /dev/null +++ b/docs/dev/plans/archive/field-functions-plan.md @@ -0,0 +1,222 @@ +# flopy4: one field function per DFN field type + +**Archived 2026-10-19.** Done: merged as #463 ("Replace spec.field() with a function per field type"). Follow-up work is in [`record-unify.md`](../lists/record-unify.md). + +Repo: pyphoenix-project (flopy4). Base: `origin/develop` (833753b9 at time of writing). +Work in a new worktree: `.claude/worktrees/field-functions`, branch `field-functions`. +Leave `pyphoenix-project-item-decorator` (8f606e3d, unpushed) alone; this plan replaces it. + +## Decision + +Replace the catch-all `flopy4.mf6.spec.field()` (~28 kwargs) with one function +per **DFN v2 field type** (`modflow_devtools/dfns/schema.py`), each taking the +attributes that type has in the schema. No class decorator / `field_transformer`. +No package-vs-item split: the schema doesn't have one, and an item field isn't +necessarily a "column" (items can be irregular: unions, keystrings, varying +width). Where a field sits is just whether it has a `block`. + +Why: + +- `field()` runs before attrs attaches the annotation, so it can't see the + type. Today the generator restates the kind as flags (`array=True`, + `arms=True`) so `field()` can pick eq / converter / on_setattr. If the + function *is* the kind, it picks all of that itself at definition time. +- Rejected: inferring kind from the hint in a `field_transformer` (`@item`, + 8f606e3d): + - `field_transformer` isn't inherited, so every row class must remember the + decorator; forgetting it fails silently. + - Hints are ambiguous: `tuple[int, ...]` is an int array (OC `steps`) but a + cellid inside OBS's `Union[tuple[int, ...], int, str]`; a scalar cellid is + a bare `tuple`; "union with arm classes" == "annotation is a string". + - Behaviour split across field() + transformer + hint; needs + `@dataclass_transform` re-declared. + - Nothing it does needs the annotation at definition time: `to_union_arm` + already resolves arms lazily (`takes_field=True`, `_union_field_arms`). +- Mirroring the schema makes codegen a near 1:1 mapping (DFN type → function, + schema attributes → kwargs), and the vocabulary is the one the DFNs already + use. +- The hint stays for type checkers and users. Schema attributes that only + restate the Python type (e.g. `Array.dtype`) are left to the hint; everything + else (direction, cellid model, index, pk/fk, shape, layered, ...) is a kwarg. + +## One `shape`, as in the schema + +The schema has a single `shape: list[str]` on both `Array` and `List` +(`[]` = unsized). Do the same; no separate `count`: + +- grid array: `shape=("nlay", "nrow", "ncol")` +- inline array in an item: `shape=("auxiliary",)`, `("nseg-1",)`, + `("packagedata.ncon(ifno)",)`. item.py already tells sibling-field / + package-dim / lookup sizing apart from the expression + (`_sized_by_field/_package/_lookup`). +- unsized: `shape=()` (or omitted) — rest of the row in an item (OC/PRP + `steps`, PRP `fraction`). No sentinel needed because `array()` itself says + it's an array. +- list: `shape=("nodes",)` replaces today's `dim="nodes"` (disu `cell2d`; + the dim counting the rows; TDIS `nper` via `linked_dim`). +- cellid, as in the schema: shape starts with `ncelldim`. One cellid: + `array(cellid="1", index=True, shape=("ncelldim",))`; cellids: + `shape=("ncelldim", "numalphaj")`. So "one vs many" is `len(shape)`, not a + flag. flopy4 already knows `ncelldim` as a shared dim + (`gwe/dis.py shared_dims`, `LoadContext.ncelldim`, `item.infer_ncelldim`). + Check how item.py's cellid width inference keys off this before switching. + +## API (flopy4/mf6/spec.py) + +Common to all (the schema's `InputFieldBase` + attrs): `default`, `validator`, +`converter`, `repr`, `eq`, `init`, `metadata`, `on_setattr`, `alias`, +`longname`, `optional`, `block` (set ⇒ a block-level field; unset ⇒ a field +of an item/record). Keep private hooks (`_recount`, `_check_dim`) and +converters (`to_ts_value[s]`, `to_signed_indexes`, `to_union_arm`, +`coerce_child`). + +| function | schema type | kwargs | behaviour | +|---|---|---|---| +| `keyword()` | Keyword | — | bool flag; today's `tagged` metadata on items/records, plain bool option on packages | +| `string()` | String | `time_series`, `pk`, `fk` | `time_series` → `to_ts_value` | +| `integer()` | Integer | `index` (`True`/`"signed"`), `pk`, `fk`, `time_series` | `block == "dimensions"` → `_check_dim` hook (as today) | +| `double()` | Double | `time_series` | `time_series` (no block) → `to_ts_value` | +| `array()` | Array | `shape`, `layered`, `netcdf`, `index`, `fk`, `cellid`, `time_series`, `fill_forward` | see below | +| `list_()` (or `spec.list` via module namespace) | List | `shape`, `fill_forward` | list of Items (today `field(block=..., dim=..., eq=...)`) | +| `union()` | Union | `*kinds` | with kinds: token-level union (today `union=("cellid", "index", "string")`); without: arm-class union, converter `to_union_arm` (today `arms=True`) | +| `file()` | File | `direction`, `keyword` | today's `path()`, renamed to match the schema | +| `child()` | File → component | `direction`, `keyword` | as now | +| `record()`? | Record | — | nested record refs (`package.py.jinja` ~61–63 use plain `attrs.field`); only add if it carries something | + +`array()` behaviour: + +- Always `eq=ARRAY_EQ`, block-level or not. The tuple exception + (`eq = ARRAY_EQ if shape and not array`) came in with #406 (count= → + shape=), when "has a shape" stood in for "is numpy"; it isn't needed. + Checked `array_eq` on tuple values: empty, nested (cellids), mixed with + `TimeSeriesRef` objects and different lengths all behave like tuple `==`. + Differences: NaNs compare equal (better), and **int vs float is unequal** + (`(1, 2)` vs `(1.0, 2.0)`: tuple True, array_eq False, from its dtype + check). That matters for item arrays without a converter (PRP `fraction`, + GNC `alphasj`: user ints vs parsed floats). Fix in `array_eq`: compare + numeric values regardless of int/float dtype for non-ndarray inputs (or + generally, if no test relies on strict dtype), rather than special-casing + at declaration. Run the tests both ways. +- `time_series`: block-level = time-array series name, no converter; in an + item → `to_ts_values`. Still keyed on `block`, but that's what the name + refers to, not a representation difference. +- `index="signed"` → `to_signed_indexes`; a single-name shape that's a + sibling field → `_recount(name)` piped into on_setattr. +- Values stay numpy at block level and tuples in items; unifying the + representation is out of scope, but equality no longer depends on it. + +Open naming points (decide while implementing): + +- List function: `list_()`, or `spec.list(...)` through the module + namespace — either is fine (user). Pick whichever reads better in the + generated modules. +- Decided: `path()` → `file()`. +- `keyword` metadata key: keep `"tagged"` to limit churn, or rename. + +`fill_forward`, `write_if_empty`, `schema`, `repr=False`: grep generated code +for which types use them; put each only on those functions. + +## Metadata changes + +- Drop `"array"` (the function is `array()`; store `"type": "array"` or test + `"shape" in metadata` — pick one helper and route consumers through it). +- Item arrays keep `"shape"` (unchanged key; now also `()` for unsized). +- Cellids: `"shape"` gains the leading `"ncelldim"`; consumers that do + `metadata["shape"][0]` for the count must skip it. Helper e.g. + `_count_expr(f)` in item.py. +- Lists: `"dim"` → `"shape"`. Consumers in package.py, component.py, + structure.py — grep `"dim"`. +- `"signed"` → `"index": "signed"` (as in the schema); `to_tokens` etc. + check `index == "signed"`. +- Unchanged: `block`, `layered`, `netcdf`, `fill_forward`, `time_series`, + `index`, `pk`, `fk`, `cellid`, `union`, `optional`, `direction`, + `_keyword`, `child`. + +Consumers (metadata reads on develop, excluding codegen): item.py 31, +package.py 22, structure.py 13, spec.py 11, record.py 7, netcdf.py 5, +attrs_xarray.py 4, unstructure.py 2, component.py 1, load_context.py 1. +Row-array logic is mostly item.py: `construct_item`, `to_tokens`, +`from_tokens` (~line 697 `array_fields`), `_shape_dim`, +`_sized_by_field/_package/_lookup`, `_dim_counted`, `_count`, +`package_sized_fields`, `sized_by`, `_array_elem_type`. Don't disturb +package-array `shape` reads (package.py, netcdf.py, structure.py). + +## Codegen (flopy4/mf6/utils/codegen) + +Aim: dispatch on the DFN field type and pass schema attributes through. + +- filters.py item emitters → schema functions: + `attr_column` (generic, `rhs=f"field({margs})"`; take the function name), + `value_column` → `double/integer/string`, `feature_id_column` → + `integer(index=True, pk=/fk=)`, `union_column` → `union(...)`, + `array_column` → `array(...)` (cellid shape gains `ncelldim`; fix the bare + `tuple` annotation fallback), `class_union_column` → `union()`, scalar + cellid (`tuple = field(cellid="1")`) → `array(cellid=..., index=True, + shape=("ncelldim",))`. `file_column` → `file(...)`, `child_column` unchanged. + Consider renaming these `*_column` helpers while there, since "column" is + wrong for irregular items. +- filters.py package emitters: `field_metadata` + `field_call` (~547, 635) + pick the function from the DFN type; drop the explicit `eq=ARRAY_EQ` line; + lists: `dim=`/`linked_dim` → `shape=`. make.py:400 calls `field_call`. +- templates/package.py.jinja ~61–85: `attrs.field(metadata={"tagged": True})` + → `keyword()`. +- Generated imports: only the functions used (check how the template builds + the `from flopy4.mf6.spec import ...` line). + +## Steps + +1. Worktree + branch off `origin/develop`. +2. spec.py: add the schema functions alongside `field()`; keep `field()` + working meanwhile. +3. item.py and other consumers: helpers for array-ness / count / signed, + cellid `ncelldim`, list `shape`; switch reads. +4. Codegen: emit the new functions. +5. Regenerate classes against the same DFN version develop's classes were + made from (check `flopy4 mf6 status` / sync metadata first; + `pixi run generate-classes` targets modflow6@develop and may pull + unrelated drift). `git diff --stat`: expect only field-call lines and + imports. +6. Remove `field()` (or reduce it to nothing) once generated and hand-written + code no longer use it. Fix hand-written uses and tests + (`test/mf6/test_mf6_codegen.py` asserts on `array=True`/`arms=True`). + Same for `path()` → `file()`: grep hand-written modules (e.g. + `gwe/__init__.py` netcdf files) and codegen (`field_call`'s `fn = "path"`, + `file_column`, make.py ~471). +7. mypy: the attrs plugin doesn't see our wrappers as field specifiers (see + comment in `attr_column`); confirm no regression, and only if needed add + `@dataclass_transform(field_specifiers=...)` on base classes. +8. Test: `pixi run test`, mypy, ruff. Compare with develop for pre-existing + failures. +9. Local commits only (don't push): (a) spec functions + consumers, + (b) codegen + regenerated classes, (c) remove `field()` kwargs / + `field()`, (d) list `dim` → `shape` if it's big enough to separate. + Messages end with the Co-Authored-By line. + +## Not in scope + +- Inferring anything from type hints. +- Unifying numpy (package) vs tuple (item) array representations (equality + is unified; see `array()`). +- modflow-devtools DFN schema changes. + +## Outcome (implemented on `field-functions`, local commits only) + +728ba10c array_eq, fc526076 field functions + codegen + regenerated classes, +09a17e2b remove field()/path(). Deviations from the plan: + +- `array_eq`: ints vs floats compare by value only for tuples/lists; ndarrays + still need equal dtypes (`test_eq_array_dtype_and_shape_matter` relies on it). +- Lists: `list_(shape=...)` is the kwarg, but the metadata key stays `"dim"`. + Several package-level consumers treat any `"shape"` as an array + (`attrs_xarray._period_dim`, attrs_xarray:213, unstructure:295), so a list + carrying `"shape"` would be misread. +- Every function records `metadata["type"]` (DFN type name); item.py's + `_is_array` reads that (minus single cellids) instead of `"array"`. +- Record classes' inner fields (package.py.jinja) still use plain + `attrs.field(metadata={"tagged": True})`; they never went through `field()` + and a tagged record value isn't necessarily a Keyword. Left alone. +- File lists (`ts_file: list[Path]`) use `file()` (they need direction/keyword). +- `keyword()` always sets `tagged`, as the schema derives it; harmless at + package level (only read for dict-valued arrays there). +- Regenerated from local modflow6 at 524d1ace; LAK/MAW (and gwf-lak.lark) + drift under the newer devtools, so only the codegen delta was applied to them. diff --git a/docs/dev/plans/archive/flopy3-period-names-plan.md b/docs/dev/plans/archive/flopy3-period-names-plan.md new file mode 100644 index 00000000..a8e75a74 --- /dev/null +++ b/docs/dev/plans/archive/flopy3-period-names-plan.md @@ -0,0 +1,127 @@ +# Accept flopy3's period-data argument names + +Status: plan only, deferred (2026-10-22). This is the flopy4 side of +`modflow-devtools/period-list-name-plan.md`. Do it before 4.0. The devtools +rename can land any time after, because it doesn't change flopy4's API. + +## Why + +flopy4 calls every package's period list `stress_period_data`, which is +documented and canonical. flopy3 names each argument after its v1 DFN +field, so many flopy3 scripts pass a name flopy4 rejects. flopy4 should +accept those names in 4.x with a deprecation warning, so that an upgraded +script still runs and says what to change. The STO and OC arguments also +need converting, because their values are shaped differently. + +## The legacy names + +**Plain renames.** The value is the same shape, so forward it unchanged to +`stress_period_data`: + +| flopy3 argument | packages | +|---|---| +| `perioddata` | LAK, MAW, MVR, SFR, UZF, PRP, SPC, TVK, TVS | +| `lakeperioddata` | LKT, LKE | +| `reachperioddata` | SFT, SFE | +| `mwtperioddata`, `mweperioddata` | MWT, MWE | +| `uztperioddata`, `uzeperioddata` | UZT, UZE | + +Derive the per-package list from the stored DFNs. It's the DFN name of the +fill-forward list, wherever that name isn't already `stress_period_data`. +Hard-code it only after the devtools rename, once the DFNs no longer carry +the old name. + +Don't alias TDIS's `perioddata` or the grid packages' arrays. Those +already have the same names in flopy3 and flopy4. + +**STO: `steady_state`, `transient`.** flopy3 takes one of these forms: +- `{kper: True}`, the common one; +- a bare `True`, meaning period 0; +- a list of periods. + +Convert to `stress_period_data={kper: "steady-state" | "transient"}`. If +both are given, merge them. If they disagree on the same period, raise. + +**OC: `saverecord`, `printrecord`.** flopy3 takes: +- a list of `(rtype, ocsetting...)` tuples, which applies to period 0 and + fills forward. This is the common form, e.g. `saverecord=[("HEAD", + "ALL")]`. +- a `{kper: [tuples]}` dict. + +Convert each tuple to `("SAVE", rtype, *setting)` or +`("PRINT", rtype, *setting)`. Merge both arguments into one +`stress_period_data` list per period, saves first, which matches flopy3's +write order. + +flopy3 also differs elsewhere, e.g. OC's `budget_filerecord` and +`head_filerecord` versus flopy4's `budget_file` and `head_file`. Those +belong in a wider flopy3-compat inventory, not here. Keep the same +mechanism, so they can be added later without redesigning anything. + +## Mechanism + +- **Metadata.** Each generated class gets a ClassVar, the same way it gets + `headers`: + + ```python + legacy_kwargs: ClassVar[dict[str, Legacy]] = { + "perioddata": Legacy("stress_period_data"), + } + ``` + + Here `Legacy(name, convert=None)` is a NamedTuple in `spec.py`, beside + `Header`. `convert` names a function in a small hand-written module, + `flopy4/mf6/legacy.py`, for the STO and OC reshapes. Codegen adds the + entries: + - plain renames, read from the DFN as described above; + - STO and OC, from a short table in codegen. + +- **Where the translation happens.** attrs generates `__init__` after the + class body runs, so `__init_subclass__` can't wrap it, and attrs' + `alias=` allows only one name. Translate in a `__call__` on a metaclass + for `Component`, derived from `ABCMeta`, because `Component` is already + an ABC. Before calling `__init__`, for each key in `legacy_kwargs`: + - pop it; + - convert it; + - emit `DeprecationWarning(f"{cls.__name__}({old}=...) is deprecated; + use {new}=...")` with `stacklevel` pointing at the user's line; + - pass it on under the new name. + + If both the old and the new name are given, raise `TypeError`. + + If a metaclass looks too heavy, the fallback is to have the template emit + an explicit `__init__` wrapper for classes with legacy names, using + `attrs.define(init=False)` and calling `__attrs_init__`. That's more + generated code, but the mechanism is local to the class. + +- **Scope.** Only constructor keywords. Don't add attribute aliases such as + `lak.perioddata`: flopy3 code that reads these attributes back is rare, + and properties would clutter the classes. Mention this in the migration + guide. + +## Tests + +- Each plain alias builds an object equal to one built with the + `stress_period_data` form, and warns once. +- STO: `{0: True}`, `True` and the list form all convert. Conflicting + `steady_state` and `transient` raise. +- OC: the list form and the dict form convert, and save/print merge in + order. +- Passing both the old and the new name raises `TypeError`. +- A class without legacy names gets no metaclass overhead beyond an + empty-dict check. +- After the devtools rename, regenerate and check that `legacy_kwargs` + doesn't change. + +## Docs + +The migration guide gets a section with the old-name/new-name table, and +before/after examples for STO and OC. Keep the docs examples on the new +names. Use the aliases only in the guide. + +## Removal + +The plain renames are cheap, so keep them through 4.x. Consider dropping +the STO and OC converters at 5.0. Escalate the warning to +`FutureWarning`, which users see by default, one minor release before +removal. diff --git a/docs/dev/plans/archive/followups-after-row-layout.md b/docs/dev/plans/archive/followups-after-row-layout.md new file mode 100644 index 00000000..7d877765 --- /dev/null +++ b/docs/dev/plans/archive/followups-after-row-layout.md @@ -0,0 +1,154 @@ +# Plan: follow-ups after row layout, boundnames, conditional fields + +State as of 2026-10-15: flopy4 `origin/develop` at `995ae0c2`. Merged: +#467 (row layout), #469 (grid dims), #471/#472 (tests), #473 (conditional +fields, the DFN `when` attribute), #470 (boundname fixes: `""` -> `None`, +other empty strings rejected, writer raises on empty). + +Rules for every item: one branch and PR each, off a fresh `origin/develop`. +Run only targeted tests locally; CI runs the rest. Never push or touch GitHub +unless asked at the time. Local env: +`.pixi/envs/dev/bin/python`, with `PYTHONPATH` pointing at a devtools +checkout if the pinned devtools (`9a1662ef`) lacks something needed. + +Suggested order: 1, 3, 2, 4 (1 and 3 are small; 2 and 4 are large). + +**Status (2026-10-19, develop `e0d94b86`):** 1 merged as #478, 3 needed +nothing more (see below). Open: 2 and 4. Both touch the row codec that +#479-#484 since rewrote (`read_rows`, `ItemTable`, one number-conversion +point in #484), so the line numbers below are stale; re-find the spots +before starting. + +## 1. Boundnames on the `when` mechanism + +**Done:** merged as #478 ("use conditional metadata for boundnames"). + +#473 says "Boundnames move onto the same mechanism after #470". The DFNs +already say it: devtools `9e45471d` gives every boundname column +`when = "boundnames"` (still `optional`). flopy4 ignores that today: + +- Codegen: `make.py` `_is_label` branch (~286) emits + `string(default=None, empty_is_none=True, optional=True)` with no `when`. + Pass the DFN field's `when` through (as `filters.attr_column` does for + other columns via `meta["when"]`). Check the devtools pin has `when` on + boundnames; bump it if not. `pname` (name files) has no `when`, so + `_is_label` stays for `empty_is_none`. +- `item.py`: drop the boundname special case. + - `_has_boundname_field` (~126) and the `bn` loop in `solve` (~602-640): + the boundname becomes an ordinary conditional last field, present iff + `boundnames` is in `context.enabled`, absent rows allowed (optional). + - The Python-input fallback that infers a boundname when the option is + unset (~638) becomes the generic "conditional=False" retry #473 added. + - `fields = [... if f.name != "boundname"]` (~428) and `_columns`/ + `to_tokens` boundname-last handling (~759-829): check whether position + alone now covers it. +- `package.py` `_coerce_item_list` (~457): drop `on.add("boundnames")`; + `conditional_fields` covers it. +- `structure.py` / `RowContext`: the separate `boundnames` flag goes if + `enabled` carries it (~58-75, ~1067-1164). +- Regenerate; tests: the boundname round trips from #470 and #473's + conditional tests must pass unchanged. + +## 2. Keep number-like text as text (reader refactor) + +**Done locally (2026-11-02):** branch `keep-token-text` (commit +`938a67c6`, worktree `.claude/worktrees/keep-token-text`), not pushed. +Tokens stay text in both transformers and NumberRows; `utils.from_text` +converts by field type while structuring. + +Problem: the basic transformer +(`flopy4/mf6/codec/reader/transformer/basic.py`, `TOKEN`) turns tokens that +look like numbers into numbers before the schema is known, so an unquoted +obsname or boundname `1.50` reloads as `1.5`. All loading goes through the +basic transformer (`Package.load` passes no component). + +Experiment (local branch `token-text`, uncommitted, worktree +`/wt-token-text`, on old develop `e8f2b939`): +`TOKEN` returning `str(token)` gave ~848 failed / 653 passed / 29 errors: + +- dims and option scalars stay strings (`np.full` in + `package._broadcast_griddata`; count checks at `package.py` ~65; + `structure.py` ~345 `nodes // nlay`) +- grid data arrays come out as strings +- ~30 parser tests in `test_mf6_codec.py` assert numbers + +Fix: keep tokens as text and convert by field type while structuring. + +- One conversion point for scalars, keyed on the field's spec type + (integer, double, string, keyword, union arms by their own rules). +- Arrays: convert in bulk (`np.array(tokens, dtype=...)` / `np.fromstring` + style) rather than per token, for speed. +- Row items already convert per field in `item.py` `_convert`; make it take + text and convert there. +- Update the parser tests to expect strings (the parser no longer types). +- The old workaround (`_TextInt`/`_TextFloat`, `parse_number_text`, + `token_text`) is on local branch `boundname-text-full`, for reference + only; don't revive it. +- After this, `package.py` ~572's warning about numeric boundnames stays + (MF6 itself may read them as cells). + +Start a fresh branch off develop; the `token-text` experiment is a +one-liner and can be redone. + +## 3. Validate lookup-sized widths + +**Done.** `Package` checks lookup-sized widths too: `dim_value(..., +select=...)` in `package.py` (~338-360), column-wise for SFR's 10k rows, +reworked in #481. + +Already done: `Package._check_lookup_counts` (#392) checks SFR's ic +with `dim_value(..., select=...)` on construction and load +(`test_sfr_ncon_mismatch`). SFR's ic is the only row-level lookup in the +DFNs. Nothing to do. + +An inline array sized by a lookup into another block (LAK/MAW/SFR +`connectiondata` `ic` sized by `packagedata.ncon(ifno)`) isn't checked. +The package checks dim-sized widths already (`dim_input`/`dim_value`, +`package.py`). Add devtools' `dim_value(..., select=...)` there, so +`len(ic) == ncon` for that row's `ifno`. First confirm nothing checks it +today, and find every lookup-sized field in the DFNs. Add the lookup to +`RowContext` if the layout needs it (it shouldn't: these are always the +last field, which takes the rest of the row). + +## 4. Merge `Item` into `Record` + +**Open.** `class Item(Record)` (`item.py` ~931) still overrides the +row codec. Since #479 list rows can also live as columns in `ItemTable` +(`item_table.py`), so the merge has to cover that path too. + +Step 4 of [`record-unify.md`](../lists/record-unify.md), now unblocked (cellid and row layout are +in). One codec for row and line records: `Item`'s count-sized arrays, +1-based index/cellid, union columns, time series, trailing boundname, +`_columns` order; `Record`'s `convert` plain forms, nested records, tagged +values. `Item` becomes an alias or goes away. Do after 1 (fewer boundname +special cases to carry) and ideally after 2 (one conversion path). Own +branch, full CI. + +## 5. Dimensions after an in-place append + +From [`dimension-issues.md`](dimension-issues.md) (issue 2). Assigning a dimension that +disagrees with its list's rows raises, but appending to the list in place +doesn't update the dimension until something reads dims: + +```python +t = Tdis(perioddata=[(1.0, 1, 1.0), (2.0, 1, 1.0)]) +t.perioddata.append(t.perioddata[0]) +t.nper # 2, until t.get_dims(); then 3 +``` + +Either read linked dims through `_sync_dims` (a property, or on attribute +access), or document that they're current only after a dims read +(`get_dims`, `resolve_dims`, writing). Small; decide which first. + +## Local cleanup (ask before deleting) + +Still present as of 2026-10-19. Many other worktrees under +`.claude/worktrees/` and the session scratchpads are for branches since +merged (`git worktree list`); worth one pass to prune them. + +- `token-text` branch and its worktree: superseded by item 2's fresh branch. +- `boundname-text-full`: reference for item 2's old workaround. +- `row-layout` (worktree `.claude/worktrees/field-functions`, local commit + `dc64fa2d`): its docstring trim landed in #470. +- `boundname-text` worktree (`/wt-boundname`): + merged as #470. diff --git a/docs/dev/plans/archive/io-benchmark-results.md b/docs/dev/plans/archive/io-benchmark-results.md new file mode 100644 index 00000000..7e654bf3 --- /dev/null +++ b/docs/dev/plans/archive/io-benchmark-results.md @@ -0,0 +1,152 @@ +# flopy4 vs. flopy3 IO benchmark (2026-10-16) + +**Archived 2026-10-19.** Stale: measured at `6b233ddb`, before #482-#484. Regenerate from the suite with `pixi run -e dev bench-report`. + +Load and write times for flopy4 (`develop` @ `6b233ddb`, after #479 "more IO +optimizations") and flopy3 3.11.0, on 13 test models from 5 KB to 84 MB. + +## Summary + +- **Load:** flopy4 is 1.4–13× faster, mostly 2.5–5×. On biscqtg + (1.27M list rows) it drops from 10.8 s to 2.3 s. The 751x751 grid gains + least, at 1.4×. +- **Write:** flopy4 is 2.3–9.5× faster from about 1 MB up. On the tiniest + model it's slightly slower (11 ms vs. 9 ms). +- **Memory:** flopy4 uses more on every model. Its import alone costs + ~120 MB more (~295 vs. ~175 MB), and on biscqtg it peaks at 1.16 GB vs. + 0.60 GB. This is the main follow-up. +- **Startup:** importing flopy4 takes ~2.2 s vs. ~0.8 s for flopy3, and its + first load adds ~0.3 s to build the parser. Both are left out of the + table. +- **Output:** flopy4 writes ~half the bytes of flopy3 and the same number of + files. The difference is formatting: flopy3 pads every value to 8 decimals, + and flopy4 writes compact values. Reloading flopy4's output for secp, + 751x751 and Peterson gives float arrays that match the source exactly. + +## Results + +Times are median seconds. "flopy3 load" includes reading all data (see +Method). "Memory" is the process's peak RSS in MB, including import and +warm-up. + +| model | input MB | flopy3 load | flopy4 load | load × | flopy3 write | flopy4 write | write × | memory fp3 / fp4 | reps | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| test001a_Tharmonic | 0.005 | 0.010 | 0.003 | 2.9× | 0.009 | 0.011 | 0.8× | 175 / 298 | 5 | +| test006_gwf3_disv_ext | 0.015 | 0.035 | 0.005 | 6.6× | 0.019 | 0.015 | 1.3× | 175 / 295 | 5 | +| test028_sfr | 0.38 | 0.033 | 0.009 | 3.5× | 0.024 | 0.021 | 1.1× | 178 / 297 | 5 | +| test045_lake4ss_dev | 0.50 | 0.033 | 0.011 | 3.1× | 0.031 | 0.019 | 1.7× | 175 / 296 | 5 | +| test120_mv_dis-lgr | 1.6 | 0.070 | 0.017 | 4.0× | 0.092 | 0.033 | 2.8× | 178 / 300 | 3 | +| test016_Keating_dev | 3.3 | 0.355 | 0.028 | 12.8× | 0.101 | 0.044 | 2.3× | 185 / 311 | 3 | +| test033_wtdecay | 6.9 | 0.068 | 0.013 | 5.1× | 0.173 | 0.018 | 9.5× | 181 / 304 | 3 | +| test019_VilhelmsenGF | 10.8 | 0.368 | 0.129 | 2.9× | 1.84 | 0.26 | 7.0× | 300 / 476 | 2 | +| test1000_751x751 | 33 | 0.323 | 0.224 | 1.4× | 1.21 | 0.22 | 5.4× | 290 / 430 | 1 | +| test016_Keating_disu_mfusg | 55 | 2.16 | 0.86 | 2.5× | 1.93 | 0.81 | 2.4× | 320 / 479 | 1 | +| test1005_secp | 66 | 2.49 | 0.83 | 3.0× | 3.92 | 0.68 | 5.8× | 332 / 605 | 1 | +| test1002_biscqtg_disv_dev | 77 | 10.79 | 2.30 | 4.7× | 7.65 | 3.12 | 2.5× | 596 / 1161 | 1 | +| test1001_Peterson | 84 | 2.89 | 1.23 | 2.3× | 5.84 | 1.05 | 5.6× | 492 / 672 | 1 | + +flopy3's load time split into the load call itself and the data-reading pass: + +| model | `MFSimulation.load` | read all data | +|---|---:|---:| +| test033_wtdecay | 0.013 | 0.055 | +| test019_VilhelmsenGF | 0.054 | 0.314 | +| test1002_biscqtg_disv_dev | 8.837 | 1.956 | + +For the other models the data-reading pass takes under 0.1 s. + +Bytes written: + +| model | flopy3 MB | flopy4 MB | +|---|---:|---:| +| test1000_751x751 | 19.8 | 9.4 | +| test016_Keating_disu_mfusg | 17.0 | 7.4 | +| test1005_secp | 65.6 | 27.0 | +| test1002_biscqtg_disv_dev | 58.6 | 57.4 | +| test1001_Peterson | 80.4 | 36.1 | + +## Method + +- **Models:** from the modflow-devtools registry, all in `KNOWN_PASSING` + (`test/mf6/test_mf6_load_all_models.py`). They were picked to cover: + - the size range from 5 KB to 84 MB; + - array-heavy models (secp, Peterson, Keating_disu_mfusg, 751x751); + - list-heavy models (biscqtg); + - multiple models with exchanges (dis-lgr); + - package-specific cases (SFR, LAK); + - OPEN/CLOSE external files (disv_ext, VilhelmsenGF, wtdecay). + + Input MB is the total size of each model's copied files. +- **Isolation:** each (library, model) pair runs in its own Python process, + one after another. +- **Warm-up:** each process first loads `test001b_Tlogarithmic`, so import + and parser building are excluded from the timings. +- **Each repeat:** load from the source directory, then write to a fresh + empty directory. +- **flopy3:** + - Loading uses `MFSimulation.load(sim_ws=..., verbosity_level=0)` with + default options. + - flopy3 doesn't read OPEN/CLOSE files until data is accessed, so its load + time adds a pass that calls `get_data()` on every `data_list` item of + every simulation and model package. + - Writing uses `set_sim_path(out)` then `write_simulation(silent=True)`. +- **flopy4:** + - Loading uses `Simulation.load(ws / "mfsim.nam")`, which reads all data + into numpy arrays at load time. + - Writing sets `sim.workspace = out`, then calls `sim.write()`. +- **Environment:** Apple M2 Max (12 cores), 32 GB RAM, macOS 26.7.1, + Python 3.11.16, numpy 2.4.6, lark 1.3.1, pixi `dev` env. + +### Caveats + +- The five largest models ran once each, so treat their numbers as rough + (about ±10%). +- The data-reading pass doesn't cover flopy3's child packages (OBS, TS), so + flopy3's load time is slightly understated. +- flopy3 used its default load options, not the + `lazy_io=False, verify_data=False` that the `load-baseline` branch used, + so these numbers aren't directly comparable with that branch's. +- Write speed partly reflects how much is written: flopy3 writes about 2× + the bytes. +- The memory figures are each process's peak, not the model's own + footprint. They include the ~175 MB (flopy3) or ~295 MB (flopy4) baseline + from import and warm-up. + +## Reproducing + +The scripts live outside the repo, in the session scratchpad: + +- `size_corpus.py` copies the `KNOWN_PASSING` models and ranks them by size. +- `bench/worker.py` times one library on one model. +- `bench/run.sh` takes a list of `model:reps` arguments and runs both + libraries on each. +- `bench/summarize.py` builds the tables. + +If you check them in, use the benchmark infrastructure in the follow-up +below rather than copying them as-is. + +## Other benchmark tooling in the repo + +- **`docs/profile/` (on develop):** + - `run_all.py` runs the profile scripts, with `--runs`, `--memory`, + `--report`, `--flopy4-only` and `--include-slow` options. + - The scripts cover Frenchman Flat read and write, and write scenarios for + `test1000_751x751` and `test1005_secp`. + - Results are saved in `docs/profile/results/`. +- **`origin/load-baseline` branch (not merged, 2026-09-24):** + - An opt-in pytest-benchmark suite, `pixi run -e dev bench-load`, with + `--benchmark-compare`. + - It uses a fixed list of about 180 models + (`test/mf6/load_benchmark_models.txt`). + - Its tests: + - `test_load`: load only. + - `test_load_materialize`: load, then fetch and sum every array and + list. + - `test_load_materialize_largest`: the same, per model, for the largest + models. + - `test_cold_start`: a fresh interpreter per round. + - A basic vs. typed parser benchmark (`test_mf6_parse_benchmark.py`). + - `docs/dev/load-benchmarks.md` describes how each library reads data. + flopy3 doesn't read OPEN/CLOSE arrays at load time and rereads them on + every `get_data()`. + - It measures loading only, not writing. diff --git a/docs/dev/plans/archive/links-from-dfns-plan.md b/docs/dev/plans/archive/links-from-dfns-plan.md new file mode 100644 index 00000000..7ede496a --- /dev/null +++ b/docs/dev/plans/archive/links-from-dfns-plan.md @@ -0,0 +1,77 @@ +# Child links from the DFNs + +Replace `LINKS` in `flopy4/mf6/utils/codegen/make.py` with the component links devtools now migrates into the DFNs (`File.component`, `File.component_ftype`, modflow-devtools#379). + +## Where we are + +- `ncf-child` pins devtools at 42eea487 (#378 keyword aliases, #379 links). +- `field_link` reads `LINKS` only. It used to prefer a DFN link, but #379 links every file field, including to components flopy4 has no class for (codegen then failed on a missing `utl.tvk`). +- Every `LINKS` entry has an identical DFN link: + + | `LINKS` entry | DFN fields | + |---|---| + | `sim-nam` `tdis6` → `sim-tdis` | 1 | + | `sim-nam` `models.mfname` → `model` / `mtype` | 1 | + | `sim-nam` `exchanges.exgfile` → `exchange` / `exgtype` | 1 | + | `sim-nam` `solutiongroup.slnfname` → `solution` / `slntype` | 1 | + | `*` `ncf_filerecord.ncf6_filename` → `utl-ncf` | 8 | + | `*` `ts_filerecord.ts6_filename` → `utl-ts` | 37 | + | `*` `tas_filerecord.tas6_filename` → `utl-tas` | 3 | + + `LINKS` also makes `tdis6` required (`optional=False`); modflow6#3032 now does that in the DFN. + +- DFN links not in `LINKS`: + + | Target | Fields | flopy4 class | Where | + |---|---|---|---| + | `utl-obs` | 52 | no | `obs_filerecord` in most packages | + | `utl-tvk` | 1 | no | `gwf-npf` `tvk_filerecord` | + | `utl-tvs` | 1 | no | `gwf-sto` `tvs_filerecord` | + | `utl-spc` | 2 | no | `gw[te]-ssm` `fileinput` column (with `utl-spca`) | + | `package` | 6 | n/a | model namefiles' `packages.fname` (model classes are hand-written) | + | `gwf-mvr`, `gwf-gnc`, `gwe-mve`, `gwt-mvt` | 1 each | yes | exchange file records | + | `utl-ats` | 1 | yes | `sim-tdis` `ats_filerecord` | + | `utl-hpc` | 1 | yes | `sim-nam` `hpc_filerecord` | + | `utl-laktab` | 1 | yes | `gwf-lak` `tables` column | + | `utl-sfrtab` | 2 | yes | `gwf-sfr` `crosssections` column; period `cross_sectionrecord` | + | `utl-spca` | 2 | yes | `gw[te]-ssm` `fileinput` column (with `utl-spc`) | + +## Order + +Delete `LINKS` first. A link becomes a child only if flopy4 generates a class for its target, so the deletion changes only ATS and HPC (step 2), and each class added later picks up its links on the next regeneration. + +1. **Delete `LINKS`.** Done on `dfn-links`. + - `_find_link` reads `component` and `component_ftype` from the DFN; `Link`, `LINKS` and `field_link` are gone. + - A link is a child only if its target has a class (a generated module, or a hand-written base for an ftype-picked family) and its field is a file record, a list of them, or a namefile row. Files in rows with other data (LAKTAB, SFRTAB, SSM) stay paths until step 7. + - `tdis6` is required in the DFN (modflow6#3032); regenerating from modflow6 `develop` keeps `tdis: Tdis`. +2. **File-record children with classes.** ATS and HPC became children with step 1; a Context now writes keyword children as `KEYWORD FILEIN `, not namefile rows. MVR, GNC, MVE and MVT follow in step 4. No corpus model uses ATS6 or HPC6 (only `ex-gwt-adv-schemes` in the examples), so a synthetic test covers them. +3. **Generate TVK, TVS, SPC.** Done on stacked branches `tvk` → `tvs` → `spc`. NPF's TVK6 and STO's TVS6 are now children; rows keyed by a cellid before the arm keyword (`cellid K value`) needed an `item.py` fix. SPC is generated but SSM's column stays a path until step 7. No corpus model uses TVK/TVS; tests add them to `test001a_Tharmonic` / `test003_gwfs_tr`. +4. **Generate exchange classes.** Done on `exchanges`. `exg-gwfgwf`, `exg-gwtgwt` and `exg-gwegwe` are now generated, and their MVR, GNC, MVT and MVE file records are children. + - Generated exchanges subclass `Exchange`, which replaces the hand-written `GwfGwt`/`GwfGwe`. + - With MODELNAMES, MVR's optional `mname` columns come before its required ones. An `Item` can now declare `_columns` (DFN order), and those optional columns are present all together or not at all. + - **Open:** the cellids in an exchange's EXCHANGEDATA take their width from the row length. That assumes both models have the same kind of grid; a DIS–DISV exchange would split its cellids wrongly. A fix would pass each model's dims. No corpus model has a mixed-grid exchange. +5. **Generate OBS** (own branch). 52 packages link to it. + - **Repeating blocks.** CONTINUOUS repeats once per output file. Its header is the DFN's `Block.header` record `output` (`FILEOUT [BINARY]`). OBS is the only block whose header isn't a number; PERIOD/TIME headers are numbers and stay `dict[number, ...]`. + - Model the header after the devtools schema, with no fields the DFN doesn't have: generate a block class holding the header record's fields plus the block's list, giving `continuous: list[ContinuousBlock]` with `obs_output_file_name`, `binary` and the rows. + - Rejected: `dict[str, list]` plus a `binary` set, because the set isn't in the DFN. Also rejected: inferring BINARY from `.bsv`, because it's lossy. + - Needs work in four places. In the grammar, a string-record block header in place of `block_index`. In the transformer, the header record. In codegen, the header-plus-list block class. In egress, writing the header back. + - **`id`/`id2` typed by parent.** The DFN types them as `string` (v2 dropped `numeric_index`). Each is a 0-based cellid tuple when the parent's rows are keyed by cellid (WEL/CHD/..., or model obs), a 0-based int feature number for LAK/SFR/MAW/UZF and their transport counterparts, and a `str` boundname otherwise. + - Splitting the tokens needs the parent's `ncelldim` (from the dims passed to the child) and its kind. On a DISV grid `1 5` is ambiguous without the parent. + - **Status.** + - Header blocks are done on local branch `obs`, stacked on `exchanges`. Codegen generates `Obs.Output` (the header record, as an Item), `Obs.ContinuousBlock(output, continuous)` and `continuous: list[ContinuousBlock]`. The basic and typed readers keep each block's header; the writer writes `BEGIN CONTINUOUS FILEOUT [BINARY]` / `END CONTINUOUS`. + - All 115 corpus OBS files load and pass the typed grammar. Optional record keywords (`BINARY`, NPF's `RHS`/`DEWATERED`, the namefiles' `UNDER_RELAXATION`) are now named rules in typed grammars, so the typed reader keeps them. + - Done. devtools #380 (merged) makes `id`/`id2` an untagged union of a cellid, an index and a boundname. Codegen gives an untagged union one column (`union=` metadata, arm kinds `cellid`/`index`/`string`). `Item.from_tokens` reads it: a non-number is a boundname. Numbers take the reading that uses the row's tokens, preferring the parent's arm: index under an exchange or a package with a `pk` column (LAK, SFR, MAW, UZF, CSUB, the GWT/GWE advanced packages), cellid otherwise. `Package.load(parent=...)` carries the parent class. Tokens past the ids are ignored, as in MF6. + - Regenerated with `--existing-only`: 30 existing packages and exchanges now have `obs: Optional[Obs] = child(...)`. Packages not generated yet (MWE, SFE, UZE, MWT, SFT, UZT, ...) link when generated. + - Corpus sweep: every simulation loads; 75 OBS files, 860 rows, all tokens round-trip except test051's trailing notes (dropped, as MF6 ignores them). Smoke roundtrips pass. + - Known limits: on DISV a 2-token CSUB cellid reads as two indexes (and on DISU, a 1-token cellid as an index); either way it writes back the same tokens. UZF's water-content depth isn't in the union: a whole-number depth reads as an index, and other depths stay floats. The DFN also allows a boundname for id2, so a stray trailing note on a row that has no id2 is read as id2. + - **Model-level OBS** (in the name file's `packages`) becomes a child only with step 6 (model classes). +6. **Generate model classes** (own project). Independent of `LINKS`: the namefile `packages` links are already in the DFNs. Hand-written behavior (grid coercion, methods) moves to a `*_methods.py` mixin like `Simulation`'s; `packages` becomes a children dict keyed by `ftype`. +7. **List-column children.** Done on `list-children` (rebased onto develop with the OBS commits). A list's file column naming a component with a class holds the component: `Lak.Tables(ifno, laktab=Laktab(...))`, `Sfr.Crosssections(ifno, sfrtab=...)`, `Ssm.Fileinput(pname, spc=Spc | Spca, mixed)`. + - Every SSM row says `SPC6`; the file's `READASARRAYS` option picks SPCA (`_disambiguate_ga_variant`, as for RCH/RCHA). + - Row children are part of the parent's `_children` (named `tables0`, `fileinput0`, ...), so they get parents and are written with it. `__setitem__`/`__delitem__` and bottom-up `Laktab(parent=lak)` don't reach rows. + - SFR's period `CROSS_SECTION TAB6 FILEIN` setting stays a path. No corpus model uses SFR cross-sections or SSM SPC6 files; LAK tables are in test042/test045. + + +## Later, separately + +- Replace the cwd lookup for file children with a thread-local load context, like `WriteContext`. diff --git a/docs/dev/plans/archive/list-dim-default-problem.md b/docs/dev/plans/archive/list-dim-default-problem.md new file mode 100644 index 00000000..560827ff --- /dev/null +++ b/docs/dev/plans/archive/list-dim-default-problem.md @@ -0,0 +1,71 @@ +# Linked dimensions: explicit value vs. default + +Deferred from the list shape-bounds follow-up (see +[`codegen-base-class-removal-plan.md`](codegen-base-class-removal-plan.md), "Default rows" and the list-dimension +section; devtools `list-shape-bounds-plan.md`). + +## Problem + +`Package._set_dim_from_rows` (`flopy4/mf6/package.py`) decides whether the +user gave a linked dimension by comparing its value to `None`, `0` and the +field's attrs default: + +```python +declared = getattr(self, dim) +if declared in (None, 0, attrs.fields_dict(type(self))[dim].default): + object.__setattr__(self, dim, nrows) # treated as "not given" +``` + +When the DFN gives the dimension a default, an explicit value equal to that +default looks the same as no value at all. TDIS is the case that exists +today: `nper` defaults to `1` (from the DFN `default_value`), so + +```python +Tdis(nper=1, perioddata=[row, row, row]) +``` + +silently becomes `nper=3`. Once exact shapes are enforced, `nper=2` with three +rows raises but `nper=1` with three rows does not. The check is inconsistent +right at the most common value. + +The same applies to any linked dimension with a non-`None` default (a DFN +`default_value` on a DIMENSIONS field), and to an explicit `0`. ATS is the +other current case: `maxats` defaults to `1`, so `Ats(maxats=1, +perioddata=[2 rows])` silently becomes `maxats=2` instead of breaking the +`<=maxats` bound. + +## Options + +1. **Codegen emits `default=None` for linked dimensions** and moves the DFN + default into metadata (e.g. `dim_default=1`). `_init_item_lists` resolves + it: rows given → `nrows`, no rows → `dim_default`. `None` then really + means "not given". Recommended. It's the smallest change and fits how + `default_rows` already works. + - Cost: the attrs default is `None` instead of `1`, so anything that reads + the field's default from the class (docs, introspection) sees `None`. + `Optional[int]` is already the annotation. +2. **A private sentinel default** (`UNSET`) instead of `None`. Same logic, + but it keeps `None` free to mean something else. Nothing needs that today. +3. **Record which init args were given** (wrap `__init__` or use an attrs + hook). This is the most precise, but it's the most machinery and is harder + to keep working with `evolve`/copying. + +## Also to decide + +- Is an explicit `0` ever valid for a linked dimension? If not, it should + probably raise rather than be treated as "not given". +- The dimension is synced only at construction. Adding rows later doesn't + update it or check it. This is separate, but touches the same code. + +## Tests to add + +- `Tdis(nper=1, perioddata=[3 rows])` raises (exact shape). +- `Tdis(perioddata=[3 rows])` → `nper == 3`. +- `Tdis()` → `nper == 1`, one default row. +- `Tdis(nper=3)` → three default rows. + +## Resolved + +Option 1, on `codegen-mixins` (`aafd47b`). `nper`/`maxats` default to `None` +with `dim_default=1` metadata; `Package._dim_value` applies it only when no +rows are given. An explicit `0` now counts as given (checked like any value). diff --git a/docs/dev/plans/archive/load-baseline-plan.md b/docs/dev/plans/archive/load-baseline-plan.md new file mode 100644 index 00000000..a0dba54e --- /dev/null +++ b/docs/dev/plans/archive/load-baseline-plan.md @@ -0,0 +1,104 @@ +# Plan: baseline input-load benchmarks, flopy3 vs. flopy4 + +**Archived 2026-10-19.** Superseded: the `load-baseline` branch was folded into the benchmark suite merged as #482 (`benchmarks/`, `docs/dev/load-benchmarks.md`). + +Goal: a baseline for simulation *input* loading that later loader changes +can be measured against. Compare flopy3 (`flopy.mf6.MFSimulation.load`), +the flopy4 basic loader, and the flopy4 typed loader, as far as each can +honestly be compared today. + +## Context (read first) + +- Work in the existing worktree `~/dev/pyphoenix-typed-remarks`, on a new + branch off `typed-remarks` (e.g. `load-baseline`). Put any new worktree + under `~/dev/`, never `/tmp`. +- Read [`typed-grammar-remarks.md`](typed-grammar-remarks.md) on `typed-remarks`, especially + "Why the typed loader isn't faster (yet)". Then read + `test/mf6/test_mf6_reader_benchmark.py`, the reader benchmark to extend. + It's opt-in (`--benchmark-enable`), uses `pixi run -e dev bench`, and has + `--bench-manifest` for pinning a corpus. +- Run everything with `pixi run -e dev ...`, never `uv run`. flopy 3.11.0 + is already a core dependency (`flopy>=3.9.5,<4`). +- Verify with `pixi run -e dev smoke`, not the full suite. +- Sign commits (`Co-Authored-By:` trailer). Don't push, open PRs, or touch + GitHub unless the user asks. + +## What can be compared, and at which level + +| level | flopy3 | flopy4 basic | flopy4 typed | +|---|---|---|---| +| full simulation load | `MFSimulation.load` | `Simulation.load` | **not possible yet** | +| parse stage (`loads`) | n/a | yes (existing) | yes (existing) | + +Typed `loads` output isn't wired into `structure.py`, so there's no full +typed load. Don't fake one; for example, don't add the basic structure time +to typed `loads`. Report typed at the parse stage only, and state the gap +in the results. When typed is wired in, it gets a column in the first row. + +## Steps + +1. **Pick the model corpus.** Start from `KNOWN_PASSING` + (`test/mf6/test_mf6_load_all_models.py`). Keep only models that *both* + flopy3 and flopy4 load without error. Record the exclusions and why + (flopy3 errors vs. flopy4 errors) in the results doc. + - Pin the set with a model-level manifest, either by extending + `--bench-manifest` or adding a sibling option, so later runs time + identical input. + - First time one full round of each loader. If flopy3 over the whole + corpus is too slow for 3+ rounds, also define a smaller fixed subset + for routine runs: the ~10 largest models by input bytes plus a spread + of small ones. Keep the full corpus as an occasional run. +2. **Make the two loaders do comparable work.** This is the step most + likely to go wrong. + - flopy3: `MFSimulation.load(sim_ws=..., verbosity_level=0, + lazy_io=False, verify_data=False)`, with `use_pandas` at its default + (record it). Check whether flopy3 defers reading any data until access + (external `OPEN/CLOSE` arrays, `lazy_io`, pandas-backed lists). + - flopy4: `Simulation.load(path)`. It reads `OPEN/CLOSE` arrays eagerly + today; confirm that. + - If either defers work, benchmark two stages: `load`, and + `load + materialize`. The second touches every package's array and list + data, e.g. by summing every numeric array. Only the second is a fair + comparison when laziness differs. Write down exactly what + "materialize" touches for each loader. + - Silence output on both: flopy3 via `verbosity_level=0`; flopy4 by + disabling logging and warnings, as the corpus fixture already does. + Printing isn't what we're measuring. +3. **Separate warm from cold.** + - The warm benchmark (in-process, parsers/grammars already built, same + pattern as the existing corpus fixture) is the main number. + - Add a small cold-start measurement: a fresh `subprocess` per loader + that imports it and loads one mid-size model. It captures import time + plus flopy4's grammar build, which matters for CLI use. A handful of + repetitions is enough; this doesn't need pytest-benchmark precision. +4. **Implement** as a new module next to the existing one, e.g. + `test/mf6/test_mf6_load_benchmark.py`. Reuse the opt-in gate, the + `slow` mark, the `dfn_path` fixture, `copy_to`, and pytest-benchmark + groups (`sim-load`, `sim-load-materialize`). In `extra_info`, record + `n_models`, total input bytes, and the flopy version. pytest-benchmark + already saves the commit and machine info. Copy each model workspace + once per session, outside the timed region. Have the `bench` pixi task + run both modules, or add a `bench-load` task. +5. **Record the baseline** in a short results section: add it to + [`typed-grammar-remarks.md`](typed-grammar-remarks.md), or split out + `docs/dev/load-benchmarks.md` and link it from there. Include: + - the corpus and exclusions; + - flopy3 vs. flopy4-basic full-load times, with and without + materialize if both stages were needed; + - the existing basic vs. typed `loads` numbers alongside, labelled as + parse-stage only; + - cold-start numbers; + - the exact commands to reproduce. + Add a short per-model breakdown for the largest few models, where + differences show most. Give absolute times and ratios. + +## Done when + +- `pixi run -e dev bench` (or `bench-load`) produces saved, comparable + flopy3 and flopy4 full-load numbers on a pinned model set. +- The laziness question is answered in writing, and the benchmark times + equivalent work. +- The results doc states the baseline and what isn't measured yet (a full + typed load). +- `pixi run -e dev smoke` still passes, and the new benchmarks are skipped + in it. diff --git a/docs/dev/plans/archive/load-branch-todo.md b/docs/dev/plans/archive/load-branch-todo.md new file mode 100644 index 00000000..c921ca57 --- /dev/null +++ b/docs/dev/plans/archive/load-branch-todo.md @@ -0,0 +1,39 @@ +# TODO + +outstanding work items + +- [ ] migrate dimension resolution from xattree +- [ ] migrate parent/child binding from xattree + - [ ] create `ParentSettingDict` class in `flopy4/mf6/dimensions.py` + - [ ] auto-set parent references on dict assignment + - [ ] update `Gwf.__attrs_post_init__()` to wrap `packages` dict + - [ ] update `Simulation.__attrs_post_init__()` to wrap `models` dict +- [ ] migrate data tree conversion from xattree +- [ ] xattree refactor consolidating above? +- [ ] data tree caching +- [ ] attrs -> pydantic +- [ ] mf6 module generation +- [ ] structural validation framework (components) +- [ ] value validation (ranges, consistency, etc) +- [ ] documentation hosted online +- [ ] user guide (getting started) +- [ ] API reference documentation +- [ ] example notebook gallery +- [ ] create advanced tutorials +- [ ] create developer guide +- [ ] flopy3 -> flopy4 conversion guide/script? +- [ ] nice component string representations +- [ ] complete WriteContext with all options +- [ ] support per-component write configuration +- [ ] support external files +- [ ] CLI interface: validate, run, convert, inspect, etc +- [ ] structuring lazy Dask arrays (lazy loading for large models) +- [ ] support out-of-core computation +- [ ] support chunking configuration +- [ ] add big data/model examples +- [ ] NetCDF read/write working +- [ ] xugrid integration functional +- [ ] larger-than-memory models with Dask +- [ ] performance benchmarks against flopy3 +- [ ] self-reproducing objects +- [ ] input file linting/validation diff --git a/docs/dev/plans/archive/load-context-plan.md b/docs/dev/plans/archive/load-context-plan.md new file mode 100644 index 00000000..f8178eca --- /dev/null +++ b/docs/dev/plans/archive/load-context-plan.md @@ -0,0 +1,184 @@ +# Load context + +**Archived 2026-10-19.** Done: merged as #418 ("load context"), `flopy4/mf6/load_context.py`. + +Status: ready (2026-10-07), against develop `a6c52c8`. Replaces the +"thread-local load context" note in [`links-from-dfns-plan.md`](links-from-dfns-plan.md) +("Later") and the "Load context" item in [`open-followups.md`](open-followups.md). + +## Problem + +Loading passes what it knows down the tree one argument at a time. +`structure_component` takes `dims`, `workspace`, `name` and `parent`; +`Package.load`, `_load_file_child`, `_parse_rows` and `_resolve_bindings` +each pass some of them on. Every new fact the reader needs becomes another +argument on every layer. Three problems follow from this. + +1. **Paths resolve against the wrong directory.** MF6 resolves every + relative path in every input file against the simulation's working + directory, the one holding `mfsim.nam`. flopy4 resolves each path + against the directory of the file that names it (`workspace=path.parent` + in `_load_mf6` and `Package.load`). `_load_file_child` tries + `Path.cwd()` first, and its docstring says loading a simulation sets + the cwd to the workspace, but nothing does. So a model kept in a + subdirectory fails to load: + + ``` + mfsim.nam: GWF6 gwf/m.nam m + gwf/m.nam: DIS6 gwf/m.dis dis + ``` + + `Simulation.load("mfsim.nam")` raises `FileNotFoundError: .../gwf/gwf/m.dis`. + MF6 runs this layout. It loads only if the cwd happens to be the + simulation directory, and then only for file records (`_load_file_child`), + not namefile rows (`_resolve_bindings` uses `workspace` alone). +2. **Exchanges between grids can't be read.** An exchange's cellids each + take the width of their own model's grid: 3 for DIS, 2 for DISV, 1 for + DISU. The exchange file doesn't say which, and the row can't settle it + (5 tokens could be 3+2 or 2+3). The reader needs the two models' grids. + Today it guesses one width from the row length, so a DIS–DISV exchange + reads `cellidm1=(0, 0)` instead of `(0, 0, 1)`, and every column after + it is shifted. The same goes for the exchange's GNC. +3. **Each new fact means another argument everywhere.** The first + exchange fix (in the working tree, see below) added `ncelldims` to + `Package.load`, `structure_component`, `_parse_rows` and + `_load_file_child`. It works, but it's a public `load` argument only + exchanges use, and a second way to say grid widths next to `dims`. + +## Design + +One object, `LoadContext`, carries what a file's reader needs to know +about where the file sits in the simulation. It's passed down explicitly, +and each level derives its children's context from its own. + +```python +@attrs.frozen +class LoadContext: + workspace: Path # the simulation directory; relative paths resolve here + dims: Mapping[str, int] = {} # grid dims in scope (the model's DIS, or given) + parent: type | None = None # class of the component whose file named this one + models: Mapping[str, Model] = {} # models loaded so far, by name + exchange: tuple[Model, Model] | None = None # the models an exchange connects + + def resolve(self, path) -> Path: ... # workspace / path, absolute paths as is + def ncelldim(self, item_cls, rows) -> int | Mapping[str, int]: ... +``` + +Use `attrs.evolve` to derive a child's context (new `dims` after DIS loads, +`parent=cls` for a file child, `exchange=(a, b)` for an exchange). + +- **Explicit, not thread-local.** `WriteContext` is thread-local because + it holds user settings that apply across a whole write. A load context + changes at every level of the tree (dims, parent, exchange models), so a + thread-local stack would have to be pushed and popped around every child + load, and the dependency would be hidden. Passing it keeps the reader's + inputs visible in signatures, and loads in parallel threads stay + independent. +- **Public signatures shrink, not grow.** + - `Component.load(path, format, name)` is unchanged. + - `Package.load(path, dims=None, name=None)` keeps `dims` for standalone + loads (`Npf.load(path, dims={...})`, documented and tested). `parent` + and `ncelldims` go away and their information moves into the context. + - A keyword-only `context: LoadContext | None = None` on `load` lets a + caller load a standalone file with the simulation directory as its + base (`Gwf.load("sim/gwf/m.nam", context=LoadContext(workspace="sim"))`). + - `structure_component(raw, cls, *, context, name=None)` replaces + `dims`/`workspace`/`parent`. +- **Default workspace.** With no context, the loaded file's own directory, + as today. For `Simulation.load` that's the simulation directory, which is + the correct base for everything below it. +- **Paths.** Every relative path read during a load goes through + `context.resolve`: + - namefile rows (`_resolve_bindings`); + - file records (`_load_file_child`); + - `OPEN/CLOSE` (`_resolve_open_close_rows`, `_read_open_close_values`); + - row children (LAK/SFR `TAB6`). + + `Path.cwd()` goes from `_load_file_child`. (`Component.path` and + `Context`'s default `workspace` also use `Path.cwd()`, but for writing + and construction, not loading. They're out of scope.) +- **Loaded components keep their paths as written.** A model loaded from + `gwf/m.nam` gets `workspace` = the simulation directory and + `filename = gwf/m.nam`, so writing it back keeps the layout. Today + `_load_mf6` sets `workspace = path.parent` and `filename = path.name`, + and `_resolve_bindings` then overwrites `filename` with the row's path. + Check the write side with a round-trip test (below), and fix it if + needed. +- **Cellid widths.** `_parse_rows` asks `context.ncelldim(item_cls, rows)`: + - Under an exchange (`context.exchange` set), each cellid column takes the + width of its model's grid, picked by `Exchange.cellid_models` (0 for + `exgmnamea`, 1 for `exgmnameb`), as a mapping by column name. + - Otherwise as today: from `context.dims` if they say, else from the row + length (`item.infer_ncelldim`). + - If either exchange model has no discretization package, fall back to + the row length, as today. +- **Exchange loading.** `_resolve_bindings` on `Simulation` loads the + models block before the exchanges block, which is field order. When + loading an exchange row, `GWF6-GWF6 ab.exg a b`, it looks up `a` and + `b` in the models loaded so far (case-insensitively, as MF6 does) and + passes `exchange=(model_a, model_b)` in the exchange's context. The + exchange's GNC, MVR and OBS inherit that context, so GNC's columns + resolve too. + +## Starting point + +The working tree on `develop` has a first exchange fix (uncommitted). +Keep: + +- `Item.from_tokens(ncelldim: int | Mapping[str, int])` and its + per-column `cellid_width` (`flopy4/mf6/item.py`); +- `Exchange.cellid_models` (`flopy4/mf6/exchange.py`). The DFNs don't + say which model a column refers to; MF6 hard-codes it too + (`src/Utilities/Idm/mf6blockfile/StructArray.f90` ~L376–385, + `src/Exchange/GhostNode.f90` ~L773–792). A schema attribute is proposed + in `../modflow-devtools/cellid-model-plan.md`. Until it lands, the + table stays; +- `test_load_exchange_between_grids` (`test/mf6/test_mf6_namefile_load.py`). + +Replace with the context: + +- the `ncelldims` parameter on `Package.load`, `structure_component`, + `_parse_rows` and `_load_file_child`; +- `_exchange_ncelldims` in `structure.py`. Its lookup becomes + `LoadContext.ncelldim`. + +## Steps + +1. Add `LoadContext` (`flopy4/mf6/load_context.py`) with `resolve` and + `ncelldim`. Unit tests: absolute paths pass through; relative paths + join the workspace; widths from dims, from rows, and per column under + an exchange. +2. Make `structure_component`, `_parse_rows`, `_load_file_child`, + `_resolve_bindings`, `_load_mf6` and `Package.load` take or build + the context. Remove `dims=`/`workspace=`/`parent=` from + `structure_component`, and `parent=`/`ncelldims=` from `Package.load`. + Update callers and tests that pass them. +3. Route every relative path through `context.resolve`, and remove + `Path.cwd()` from `_load_file_child`. Fix its docstring. +4. Set `exchange=` when loading an exchange, and drop the `ncelldims` + plumbing. +5. Tests: + - the subdirectory layout above loads, and the model's DIS is found; + - writing that simulation back and reloading it keeps `gwf/m.nam` and + `gwf/m.dis`; + - `test_load_exchange_between_grids` still passes; + - loading with a cwd other than the simulation directory works (no + `Path.cwd()` dependence); + - `Package.load(path, dims=...)` standalone still works. +6. Run the touched test files, ruff, mypy, and the smoke tests + (`pixi run -e dev smoke`). Not the full suite. The smoke run's 17 + `test_mf6_typed_grammar_corpus` failures ("Model 'mf6/large/...' not + in registry") are about the environment and fail without these changes + too. +7. Update [`open-followups.md`](open-followups.md): remove "Load context", and note the + mixed-grid exchange fix as done. + +## Non-goals + +- `Path.cwd()` in `Component.path` and `Context`'s default `workspace` + (writing and construction, not loading). +- Dimension resolution after loading (`resolve_dims` walking parents) is + unchanged. The context only covers what the reader needs while the + tree is still being built. +- Hard-coding which model each column refers to stays until the + devtools schema carries it. diff --git a/docs/dev/plans/archive/load-corpus-gaps.md b/docs/dev/plans/archive/load-corpus-gaps.md new file mode 100644 index 00000000..68e32090 --- /dev/null +++ b/docs/dev/plans/archive/load-corpus-gaps.md @@ -0,0 +1,650 @@ +# Simulation.load() corpus gaps + +**Archived 2026-10-19.** Done: every cataloged gap is fixed or deliberately skipped (fixture problems). Corpus status now lives in `test/mf6/test_mf6_load_all_models.py` (`KNOWN_PASSING`, `KNOWN_TYPED_GAPS`). + +Working doc cataloging the 108 failures `test/mf6/test_mf6_load_all_models.py` +currently `xfail`s (2026-09-11 survey, full 242-model `modflow_devtools` +corpus, after the basic-grammar blank-line fix and the griddata +control-record fix landed — see `git log` on those). Ordered by model count, +each traced to an actual root cause via traceback + source-file inspection, +not guessed from the error string alone. Intent: work through these one at a +time, largest/highest-confidence first, each as its own fix + rerun of the +corpus test to move models from `xfail` into `KNOWN_PASSING`. + +## 1. Missing trailing newline at end of file — 34 models — FIXED 2026-09-11 + +`basic.lark`: made the closing `_NL+` optional (`[_NL+]`) and `start`'s +`(block [_NL*])+` a `*` so end-of-input and a zero-block file both parse. +22/34 cataloged models now pass; the other 12 were misattributed -- +their *next* real bug (categories 3, 4, and a new comma-separated-values +variant of 9, e.g. `test059_mvlake_lak_ss`'s `-250,-250,-250,...`) was +masked by this one firing first. Corpus-wide: 134/242 -> 156/242 (55% -> +65%), zero regressions (diffed against the pre-fix probe run). No +remaining work here. + +**Symptom**: `UnexpectedToken: Unexpected token Token('$END', '') at line N, +column M` (or similar). One variant is a genuinely empty file (0 blocks) — +`test201_gwtbuy-henryCHD`'s `henry.gwfgwt` is 0 bytes. + +**Root cause**: `basic.lark`'s `block` rule requires `_NL+` (at least one +newline) after the closing `"end"i block_name` — but plenty of real fixture +files (confirmed across `test021_twri`, `test006_gwf3`, `test019_VilhelmsenGC`, +several more) simply end right after `END PERIOD`/`END GRIDDATA` with no +trailing newline at all. `_NL+` can't be satisfied by true end-of-input, so +Lark's `$END` pseudo-token shows up where a newline was still expected. +Separately, `start: [_NL*] (block [_NL*])+` requires *at least one* block, +which a genuinely empty file has none of. + +**Fix shape**: make the closing `_NL+` optional (`[_NL+]`) so end-of-input +satisfies it same as a real newline would; change `start`'s `(block +[_NL*])+` to `*` to tolerate an empty file with zero blocks. Same family as +the blank-line fix already landed — worth doing together if revisiting +`basic.lark` again. Low risk: verify against the existing reader/namefile +test suites plus a synthetic no-trailing-newline / empty-file case. + +**Models**: test001e_UZF_3lay(+TS/_3layTS), test001e_noUZF_3lay, +test001h_evt_array4, test003_gwfs_disv(+_xt3d), test003_gwfs_reduced, +test006_gwf3, test006_gwf3_disv_trimesh(+_xt3d), test006a_gwf3_disv_thickstrt, +test017_Crinkle, test019_VilhelmsenGC, test019_VilhelmsenLGR(+_nr), +test021_twri, test025_ConstantCV(+_perched), test025_VariableCV, +test025_Vertically_Staggered, test041_flowdivert(+_mfusg/_nr/_nwt_dev), +test057_transientchd, test059_mvlake_lak_ss, test104_riv_mvr, test105_ghb_mvr, +test201-205_gwtbuy-henry* (the empty-file variant). + +## 2. Multi-line INTERNAL array data truncated to the first line — 29 models — FIXED 2026-09-11 + +Unified `_read_griddata_control_record` and `_parse_readarray_period_block`'s +duplicated CONSTANT/INTERNAL logic into one shared `_read_control_record` +that takes the array's target length and keeps consuming rows until that +many values are collected (instead of assuming one row), and always +broadcasts CONSTANT to that length itself rather than leaving it to the +caller. Also added the `OPEN/CLOSE` case to `_parse_readarray_period_block` +(it only existed on the griddata side before -- explains why +`test017_Crinkle`/`test019_VilhelmsenGC`(+`LGR`/`_nr`), cataloged under +gap #1's re-attribution as griddata OPEN/CLOSE, were actually hitting +*this* gap instead, in the READARRAY period reader). Verified the fix reads +real data correctly, not just avoids the crash: `test014_NWTP3High`'s 6400 +recharge values (spanning 80 lines) cross-checked exactly against the raw +file at both ends. Also spot-checked a previously-passing griddata model +(`test006_gwf3_disv`) to make sure the shared-function refactor didn't +regress the already-correct single-line case. + +28/29 originally cataloged models now pass; the 29th +(`test045_lake1tr_dev`) was masking gap #5 (LAK CONNECTIONDATA `HORIZONTAL` +token). Corpus-wide: 156/242 -> 188/242 (65% -> 78%), zero regressions. No +remaining work here. + +
Original writeup (superseded) + +Multi-line INTERNAL array data truncated to the first line — likely ~29 models + +**Symptom**: `ValueError: could not broadcast input array from shape (X,) +into shape (Y,)`, Y a multiple of X (e.g. `(80,)` into `(6400,)`). + +**Root cause**: confirmed via `test014_NWTP3High`'s `nwtp3_Rhigh.rch` — +its RECHARGE `INTERNAL` array is written across 80 physical lines (80 +values each, 6400 total: an 80×80 grid). `_parse_readarray_period_block` +(READARRAY period path, `structure.py`) reads exactly **one** row after an +`INTERNAL` control record (`data_row = rows[i]; i += 1`) — correct only +when the whole array fits on one physical line. A wrapped array silently +yields just the first line's worth of values, then crashes on the shape +mismatch (direct `accum[fname][kper] = arr` assignment against a +pre-allocated full-shape slot). `_read_griddata_control_record` (the +griddata path) already reads a full data row too, but per-field length +there is bounded by `_griddata_flat_length` + the LAYERED tiling added in +the griddata fix, so it's unclear whether an under-length INTERNAL row +can still silently slip through there — **worth a direct check** before +assuming this category is READARRAY-period-only. + +Also confirmed via `test017_Crinkle`/`test019_VilhelmsenGC`(+`LGR`/`_nr`): +`_parse_readarray_period_block` has no `OPEN/CLOSE` case at all (unlike +`_read_griddata_control_record`, which gained one in the griddata fix) — +`ValueError: could not convert string to float: 'OPEN/CLOSE'` at +`structure.py`'s `_parse_readarray_period_block`. Same missing-case bug, +not a length issue. + +**Fix shape**: `_parse_readarray_period_block` duplicates +`_read_griddata_control_record`'s CONSTANT/INTERNAL logic almost exactly +(the two were probably written independently) — worth actually unifying +them into one shared control-record reader used by both, picking up +OPEN/CLOSE and multi-line INTERNAL in one place instead of twice. That +reader needs to know the array's expected total length (already +available: `nodes`/`ncpl` for griddata, similarly derivable for READARRAY +period fields) and keep consuming subsequent rows until that many values +are collected, rather than assuming one row. `OPEN/CLOSE`'s file-based +read already isn't affected by the line-wrapping issue (reads the whole +referenced file's `.split()`, regardless of its own internal line +wrapping) — it just needs to exist in both places. + +**Models**: not enumerated individually here (29, dominated by +`test014_NWTP3*` and other larger-grid models) — rerun the corpus probe +after the fix and diff against `results_after_fix.jsonl` +(`/tmp/.../scratchpad/results_after_fix.jsonl` this session, won't survive +past it — regenerate via the probe script pattern in this session's +history if needed) to get the exact list. + +
+ +## 3. Trailing legacy MODFLOW-2005/NWT content after the last real block — FIXED 2026-09-11 + +Reconsidered per-user decision 2026-09-11 (originally skipped, revisited +same day): rather than accepting a hard data loss for ~25 models over +content that's genuinely outside the file's own real MF6 blocks, +`codec.reader.loads()` now falls back to trimming anything after the +last recognized `BEGIN...END` block when the file fails to parse as +written -- the trailing content itself is still never interpreted (it +isn't valid MF6 input and nothing tries to make sense of it), only +discarded. Fallback-only by design: a file that already parses cleanly +takes the same path as before, unchanged; the trim only ever engages +after a parse failure, so it can't silently change behavior for the 212 +models already passing. + +Implementation: `_BLOCK_RE`, a regex matching one well-formed +`^begin ... end $` span (case-insensitive, multiline, +dotall), applied with `re.finditer` to find every such block in the raw +text; truncate to just past the last one found. Deliberately done as +string-level pre-processing before the Lark parse, not as LALR grammar +leniency (recovering from "unexpected trailing input" isn't a natural +fit for LALR, and doing it at the string level keeps the change fully +isolated to this one fallback path). + +Verified against real data, not just crash-avoidance: spot-checked +several trimmed models' resulting components before running the full +suite. + +Corpus-wide (combined with gap #6's fix below, run together): 212/242 +-> 239/242 (88% -> 99%), zero regressions -- the single biggest jump of +any fix this session. 25 of the models originally cataloged under this +gap now pass; none of the trimming logic needed further tuning. No +remaining work here. + +## Per-line INTERNAL data annotations (not just the array's last line) — FOUND AND FIXED 2026-09-11, while verifying gap #3 + +**Symptom**: `ValueError: invalid literal for int() with base 10: 'row'` +-- found on `test004_bcfss`, surfaced only once gap #3's trim let this +model's `.npf` file reach griddata parsing at all. + +**Root cause**: `bcf2ss.npf`'s `icelltype`/`k` `INTERNAL` arrays are +written 10 lines (one per grid row), each with a trailing +` row N` remark: +``` + internal + 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 row 1 + 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 row 2 + ... +``` +Gap #10 (see above, in gap #2's writeup) already handled a *single* +trailing annotation on an array's *last* data line via the final +`values[:length]` truncation -- but here every one of the 10 lines +carries its own annotation, including "row" (non-numeric) *and* the row +number itself (numeric, e.g. "1", "2", ... indistinguishable from real +data by value alone). Collecting whole lines and truncating only once +at the very end left annotation tokens from early lines *inside* the +kept range and pushed real data from later lines *out* of it -- silently +wrong before the parser happened to hit the non-numeric word "row" and +raised instead. + +**Fix**: `_read_control_record`'s multi-line collection now takes each +line's tokens only up to (not including) its first non-numeric one, +via a new `_numeric_prefix()` helper, rather than the whole line +unconditionally. Correct because real INTERNAL/bare-data-row array +data is always purely numeric -- any non-numeric token marks the start +of an inline annotation for that line specifically, with no need to +know the line's true data width up front. Verified: +`test004_bcfss`'s `icelltype` (300 values, {0,1}) and `k` (300 values, +{5.,10.} -- CONSTANT 10 for layer 1, INTERNAL 5s for layer 2) both +match the file. This is a generalization of the gap-#10 fix, not a +separate one -- same underlying principle (don't trust a control +record's raw token count; trust "numeric" as the boundary of real +data), extended from "only the last line can have a trailing remark" +to "any line can." + +### Original writeup (context, superseded) + +19 models (+6 more, see below) + +Since gap #1's fix, also confirmed for `test003_gwfs_disv`(+`_xt3d`), +`test003_gwfs_reduced`, `test006_gwf3`, `test021_twri`, +`test057_transientchd` — these were previously masked by #1's +missing-trailing-newline symptom firing on some *other* file in the +model first. + +**Symptom**: `UnexpectedToken: Unexpected token Token('NUMBER', ...)`, +parser state expects `BEGIN`/`_NL`/`$END` (i.e. it's between blocks, +not mid-block). + +**Root cause**: confirmed across `test003_gwfs` (`.dis`/`.ims`) and +`test006_2models` (`model2.npf`) — the file's real MF6 content ends +cleanly at a legitimate `END GRIDDATA`/`END `, but legacy +MODFLOW-2005/NWT-style flat-format content is appended afterward (FORTRAN +format strings like `(10G13.0)`, comment tails like `HK() = Horizontal +hydraulic conductivity of layer 1`) — an apparent `mf5to15` conversion +leftover in these specific `modflow6-testmodels` fixtures, not valid MF6 +input. + +**Fix shape**: this is arguably not a flopy4 bug — real MF6 itself may or +may not tolerate this trailing content (unverified). Two options: (a) make +`start` lenient about trailing non-block content after the last real block +(stop parsing once no more `BEGIN` is found, ignore the rest) — a real +robustness win but changes what counts as a "valid" file; (b) treat as a +fixture-quality issue out of scope, possibly worth flagging upstream in +`modflow6-testmodels`. Recommend (a) only if it's low-risk to bolt on +without weakening error detection for genuinely malformed files elsewhere +(a trailing garbage block that happens to start with a real `BEGIN` token +would still be entered and fail normally, so this is fairly safe). + +**Models**: test003_gwfs(+_obs/_tr), test004_bcfss, test006_2models(+-XT3D/ +_gnc/_mvr_dev/_unconf/_unconf_nr/_unconf_nr_gnc/_unconf_tr_nr), +test006_gwf3_gnc(+_nr_mfusg), test006_gwf3_nr_mfusg, test006_gwf3_unconf, +test019_VilhelmsenGF, test106_tsnodata(+/alt_model). + +## 4. OPEN/CLOSE for list/period blocks (external file for row data) — 9 models — FIXED 2026-09-11 + +Added `_resolve_open_close_rows`: detects a block whose only content is a +single `OPEN/CLOSE ` row (period or Pass-2 Item-list blocks), +reads the referenced file, and re-tokenizes its content the same way +inline block content is -- by wrapping it in a synthetic `BEGIN DATA / ... +/ END DATA` and running it back through the existing `loads()` parser, +rather than writing a second, separate tokenizer. Verified against real +data: `test001a_Tharmonic_extlist`'s `chd.txt`-redirected CHD stress +period data (2 rows) loaded and matches the file exactly (cellid, +head, aux all correct, 1-based -> 0-based cellid conversion included). + +9 of the 9 confirmed models now pass (the originally-cataloged +`test026_WellReduction` turned out to belong to a different bucket +already fixed by then; `test050_circle_island` is still open, masking +some other bug -- not yet investigated). Corpus-wide: 188/242 -> 197/242 +(78% -> 81%), zero regressions. Not handled: any OPEN/CLOSE row option +beyond the bare filename (e.g. a scale factor) -- not seen in the corpus +yet, add if it comes up. + +
Original writeup (superseded) + +OPEN/CLOSE for list/period blocks (external file for row data) — 11 models + +**Symptom**: `ValueError: invalid literal for int() with base 10: +'open/close'` (9 models) or `could not convert string to float: 'open/close'` +/ `'OPEN/CLOSE'` (3 models — overlaps the float bucket below). + +**Root cause**: confirmed via `test001a_Tharmonic_extlist`'s +`flow15_constant.chd`: +``` +BEGIN Period 1 + open/close chd.txt +END Period +``` +A period/list block's row data can be sourced entirely from an external +file, the same `OPEN/CLOSE` keyword as griddata arrays but a *different* +mechanism (the referenced file contains list rows, not a flat numeric +array). `_parse_rows`/`Item.from_tokens` has no concept of this — it feeds +the literal tokens `["open/close", "chd.txt"]` straight into row parsing, +which tries to read `"open/close"` as a numeric column (cellid or similar) +and crashes. + +**Fix shape**: in `structure_component`'s Pass 2/3 (block Item-list / +period parsing), detect a block whose *only* content is a single +`open/close ` row, read the referenced file (relative to +`workspace`), and parse its content as this block's rows through the same +path a normal inline block would use. Separate, self-contained feature — +doesn't touch the griddata code from the current fix. + +**Models**: test001a_Tharmonic_extlist, test006_gwf3_disv_ext, +test020_mawconfined_openclose, test024_Reilly(+_ext), test026_WellReduction +(needs re-check — appears in both this bucket and elsewhere depending on +which file fails first), test042_lake0_embeddedh_openclose, +test046_periodic_bc_openclose, test050_circle_island, +test051_uzfp2_openclose (some of these overlap with bucket 1/3; re-verify +after fixing this one which ones remain). + +
+ +## 5. `infer_ncelldim` miscounts row width for LAK CONNECTIONDATA — 7 models — FIXED 2026-09-11 + +**Actual root cause** (the earlier `_n_fixed_tokens`-undercounts theory +was wrong -- `_n_fixed_tokens` was already computing the class's real 8 +fixed fields correctly): `test045_lake4ss`'s `lakeex4.lak` CONNECTIONDATA +rows carry a 12th trailing token (`... 202.8325 403.7100 2`) with no +counterpart in the *current* `gwf-lak` DFN's `connectiondata` block +(checked the real spec: `ifno iconn cellid claktype bedleak belev telev +connlen connwidth` -- exactly 8 fixed + 3-wide cellid = 11 tokens for a +DIS grid, not 12). This fixture predates a DFN revision that dropped a +field, and `infer_ncelldim`'s row-length heuristic (`total tokens - +fixed - aux - boundname`) has no way to tell "one extra genuinely +unexplained trailing token" apart from "the cellid is one element +wider than expected" -- it always assumes the latter. + +**Fix**: grid dimensions (`dims`) already unambiguously say which grid +type is in play (DIS has `nrow`+`ncol` -> ncelldim 3, DISV has `ncpl` -> 2, +DISU has `nodes` only -> 1) -- more reliable than counting tokens, and +already threaded through most of `structure.py`. Added `item.py`'s +`ncelldim_from_dims()`, preferred by `infer_ncelldim()` over the row-length +guess whenever `dims` is available; threaded `dims` down through +`_parse_rows` from `structure_component`'s Pass 2/3 (using the same +`dims or _self_dims_from_kwargs(kwargs)` "effective dims" pattern Pass 4 +uses). With the correct ncelldim now given directly rather than guessed, +`Item.from_tokens` consumes exactly the 11 tokens the current class +models and silently drops the unexplained 12th -- same as how MF6's own +list reader tolerates a trailing annotation. Verified against real data: +`test045_lake4ss`'s 110 CONNECTIONDATA rows all parse with correct cellid/ +claktype/bedleak/belev/telev/connlen/connwidth, cross-checked against the +raw file's first row by hand. + +This is also a strictly more correct general fix, not a LAK-specific +patch -- any package with a variable-width cellid now gets its ncelldim +from the grid instead of guessed row arithmetic whenever `dims` is +available, which should preempt the same failure mode for other +packages/fixtures not yet seen. + +7 of the (corrected) 7-model bucket now pass -- confirms the +`_n_fixed_tokens`-undercount theory was never the cause; the 8th model +originally lumped in here (`test016_Keating`, `'NLARRAY'`) was always +gap #6's separate issue. Corpus-wide: 197/242 -> 204/242 (81% -> 84%), +zero regressions. No remaining work here. + +## 6. `NLARRAY` field — FIXED 2026-09-11 (reconsidered alongside #3) + +`keating.rch` contains *two complete, alternative* RCH package +definitions concatenated -- a plain list-input `BEGIN PERIOD 1` block +(rows like `1 1 180 0.072`, real MF6 syntax) followed by a second, +entirely separate `BEGIN DIMENSIONS` / `BEGIN PERIOD 1` pair using the +READARRAY (`RCHA`-style) convention with a field named `NLARRAY` that +matches no real MF6 DFN field at all. Both halves parse as well-formed +blocks on their own (unlike #3's non-block trailing garbage), so +`BasicTransformer.start()` -- which folds all top-level blocks into one +`{name: rows}` dict -- was building that dict last-write-wins: the +second `DIMENSIONS`/`PERIOD 1` pair silently overwrote the first, so +the whole file effectively became just the broken, `NLARRAY`-based +half. + +**Fix**: `start()` now keeps the *first* occurrence of a repeated block +name, skipping later duplicates. Justified narrowly: a block name +repeating within one file is already an anomaly (real MF6 files have at +most one `OPTIONS`/`DIMENSIONS`/etc.; period/solutiongroup blocks are +told apart by their own number, part of the block name itself) -- and +in the one instance found this session, the first occurrence was +unambiguously the intended, real-MF6-syntax one. Verified against real +data: `test016_Keating`'s RCH stress period data (24 rows, cellid +`(0,0,168)`..`(0,0,191)`, `recharge=0.072`) now loads and matches the +file's first (list-format) block exactly. + +Corpus-wide (combined with gap #3's fix, run together): 212/242 -> +239/242 (88% -> 99%). No remaining work here. + +Models: test016_Keating, test016_Keating_dev. + +## 7+8. TIMEARRAYSERIES / per-period AUXILIARY-named array fields not recognized — FIXED 2026-09-11 + +Confirmed both together via `test027_TimeseriesTest`'s `timeseriestest.rch` +(and separately, `test001h_evt_array3`'s `flow15.evt`): +``` +begin options + ... + AUXILIARY Auxarray1 RchMult + AUXMULTNAME RchMult +end options + +begin period 1 + RECHARGE TimeArraySeries RchSeries + Auxarray1 + constant 2.3 + RchMult + open/close timeseriestest.mltarr FACTOR 1.0 IPRN 0 +end period +``` +Two distinct row shapes `_parse_readarray_period_block` didn't recognize: +a field sourced from a named time-array-series (`RECHARGE +TimeArraySeries RchSeries`, a control-record kind of its own, not +CONSTANT/INTERNAL/OPEN-CLOSE) and a per-period array keyed by an +`AUXILIARY`-declared name (`Auxarray1`, `RchMult` -- not a declared class +field at all, since the name is user-chosen). Both desynced the row +cursor the same way: falling through to "bare data row", consuming +whatever row came next (a different field's name) as if it were numeric +data. + +**Fix**: `_parse_readarray_period_block` now recognizes a +`TIMEARRAYSERIES`/`TAS6` token anywhere in a field's row and consumes +just that one row, contributing no array data (not resolved to real +values yet -- would need reading and time-interpolating the referenced +`.tas` file, a separate feature). For any row naming a field *not* in +`ra_fields` (declared class fields), it now still consumes that field's +whole control-record block (LAYERED-aware) so the cursor stays aligned +for whatever comes next -- just doesn't store the result, since it's not +resolved to a real array either. Both changes are conservative: neither +crashes, and neither fabricates data that isn't really being read. + +Corpus-wide (combined with the fix below): 204/242 -> 211/242, zero +regressions. Data completeness gap remains: TAS-driven and +AUXILIARY-named period arrays load without crashing but aren't +populated (still `FILL_DNODATA`/unset) -- a real, separate feature, not +attempted here. + +## delr/delc LAYERED per-record length — FIXED 2026-09-11 (found while investigating #7/#8) + +**Symptom**: `ValueError: could not convert string to float: 'delc'` +(or any field-name string) -- a griddata field's own control-record data +run overruns into the *next* field's name row. + +**Root cause**: confirmed via `test027_TimeseriesTest_idomain`'s +`timeseriestest.dis`, which (unusually) tags `delr`/`delc` -- true shape +`(ncol,)`/`(nrow,)`, single record, never really "per layer" -- as +`LAYERED` with one `INTERNAL` record each. `_parse_griddata_block`'s +LAYERED branch read each per-record control record using the grid's +per-layer cell count (`ncpl`) unconditionally, correct for genuinely +per-layer fields (`botm`/`idomain`, shape `(nodes,)`) but wrong for a +field like `delr` whose *true* declared length (51, `dims["ncol"]`) is +smaller than `ncpl` (2601 for this 51x51 grid) -- so it kept reading +physical lines (57x more than needed) until it collected 2601 values, +consuming `delc`'s own name row along the way. + +**Fix**: per-record read length is now `min(ncpl, target)`, where `target` +is the field's own declared length (`_griddata_flat_length`, already +computed for the post-loop tiling step, just moved earlier). Correct in +both directions: `ncpl` for a truly per-layer field (`target > ncpl`), +`target` itself for a field like `delr` tagged LAYERED despite not truly +being one (`target <= ncpl`). Verified against real data: +`test027_TimeseriesTest_idomain`'s `delr`/`delc` (51 values each, matching +the file), `top` (broadcast to 2601), `botm` (both layer values across +5202 cells) all correct; re-checked four previously-passing griddata +models (`test006_gwf3_disv`, `test001a_Tharmonic`, +`test020_NevilleTonkinTransient`, `test014_NWTP3High`) for no regression. + +Also, incidentally, this was gap #10's actual root cause too (see below) -- +retroactively fixed by this same change (confirmed, not guessed). + +## 9. Comma-separated values with no whitespace — FIXED (both cases) 2026-09-11 + +**Symptom**: `ValueError: could not convert string to float: +'-250,-250,-250,...'` (test059_mvlake_lak_ss) or `',1,1.00000000'` +(test001a_Tharmonic_tabs). + +**Part A -- OPEN/CLOSE-referenced array files**: `test059_mvlake_lak_ss`'s +griddata-referenced `bottom.ref` has comma-separated values with no +spaces at all (`-250,-250,-250,...`) -- `.split()` (whitespace-only) +treated each whole line as one token. `_read_open_close_values` now +replaces commas with spaces before splitting. + +**Part B -- inline grammar tokenization**: `test001a_Tharmonic_tabs`'s +`flow15.tdis` writes its `PERIODDATA` row as `1.00000000,1,1.00000000` +-- no OPEN/CLOSE involved, the main grammar's `word` rule tokenized the +whole comma-joined run as one token, since `,` was in its character +class. Excluding `,` alone surfaced a second, unrelated pre-existing +bug: `word`'s class was `[a-zA-Z0-9._'~,+-\(\)]` -- the `-` between `+` +and the escaped `\(` was read by the regex engine as a *range operator* +(`+` to `\`, i.e. 0x2B-0x5C), not a literal hyphen, so `word` actually +matched a much wider set of punctuation (`,-./:;<=>?@[\`) than its four +explicit characters (`.`, `_`, `'`, `~`) suggested -- confirmed by +listing every character the compiled pattern actually matched. Simply +dropping the `,` character (my first attempt) broke +`test011_mflgr_ex3` (a *regression*, caught by the full corpus probe): +its `.wel` file has an inline parenthetical remark -- +`(file for binary flow information; must be opened using DATA(BINARY))` +-- containing a `;` that was only ever matched via the accidental wide +range. Fixed properly: rewrote the character class to name the exact +same set explicitly (verified character-by-character against the +original compiled pattern), comma excluded, `-` moved to the end so it's +never a range operator: `[a-zA-Z0-9'().\/:;<=>?@_~+\[\\-]`. A `COMMA` +terminal is `%ignore`d, acting as a token separator like whitespace. + +Verified: comma-joined PERIODDATA now splits into 3 numbers matching the +file; the semicolon-containing parenthetical, `open/close` (still one +token, `/` retained), a Windows-style path, and an ISO datetime string +all still tokenize as before. Full reader/loader suite (50 tests) and +the corpus probe (zero regressions after the character-class fix, one +new pass) both confirm. + +Corpus-wide: 211/242 -> 212/242, zero regressions (after fixing the +regression the naive first attempt introduced). No remaining work here. + +## 10. `TOP LAYERED` with trailing garbage on the INTERNAL data line — CONFIRMED already fixed (retroactively, by the delr/delc fix above) + +Re-checked `test001e_UZF_spring` directly: loads correctly now, `top` +values `[10. 10. 10. 10. 6.1 10. 10. 10. 10. 10.]` match the file. The +original theory (genuinely corrupted fixture) was itself wrong -- the +trailing text after the real data (`CONSTANT 10.00000000 TOP array`) sits +on the *same physical line* as the 10 real values, so once the read +length was corrected to the field's true length (10, not `ncpl`), the +existing `values[:length]` truncation already dropped it harmlessly. +No separate fix needed. `test001e_UZF_spring` was already in +`KNOWN_PASSING` as of the delr/delc fix. + +## 11. OC `Format` record requires a field the DFN also marks required — investigated, SKIPPED, out of scope (reconsidered, same conclusion) + +**Symptom**: `TypeError: Oc.Format.__init__() missing 1 required +positional argument: 'format_'`. Model: test060_gms_ets. + +**Investigated**: checked the real `gwf-oc.dfn` directly -- +`formatrecord`'s `format` field is declared `tagged false` / `optional +false` (i.e. genuinely required, untagged), matching flopy4's +`Oc.Format.format_` (`attrs.field()`, no default). `ets3_dis.oc`'s +`HEAD PRINT_FORMAT COLUMNS 10 WIDTH 11 DIGITS 4` omits it entirely, so +this fixture doesn't conform to the current spec. + +**Re-checked against the precedent that fixed #3/#6** (revisiting a +previously-skipped gap turned out right twice already this session): +`flopy4/mf6/utils/codegen/dfn_overrides.toml` already has an entry +making this *same* `formatrecord`'s `columns`/`width`/`digits` +optional, for a documented reason -- the v1 DFN marks them with a +*bare* `optional` attribute (no value), and the v1->dev3 migration +doesn't handle that form, defaulting it to `false` instead of `true` +([`dev3-migration-gaps.md`](dev3-migration-gaps.md), `devtools/todo.md` 2026-08-18 entry, "30 +occurrences across 12 v1 DFN files", 18 of them this `columns`/`width`/ +`digits` override). That's a *migration bug* being corrected back to +the v1 DFN's real intent -- not a case of flopy4 overriding a +deliberately-required spec field. Checked whether `format` (the 4th +field in the same `formatrecord`) has the same bare-`optional` v1 +pattern: no -- the cached DFN shows `optional false` written out +explicitly, not bare. So this doesn't qualify for the same override +mechanism; `format` really is meant to be required per spec, unlike its +sibling fields. Distinct from #3's "predates a spec change, unverified +either way" framing -- this one's spec status is actually confirmed, +just not in the fixture's favor. + +Making `format_` optional anyway would mean flopy4 diverging from a +DFN field the migration got right, purely for one non-conformant +fixture's benefit, and (since `Oc.Format` is autogenerated) would need +a codegen change touching every OC-family package's `formatrecord` +across gwf/gwt/gwe/prt/chf/olf/swf -- a broad blast radius for one +model. Left `xfail`. + +## 12. LGR child model's external array file sized for the parent grid — CONFIRMED fixture mismatch, SKIPPED, out of scope + +**Symptom**: `ValueError: could not broadcast input array from shape +(26901,) into shape (7020,)`. Models: test019_VilhelmsenLGR, +test019_VilhelmsenLGR_nr. + +**Investigated**: this is a Local Grid Refinement (LGR) simulation with a +coarse parent model (`TM9_parent_GN`, 9x61x49 = 26901 nodes) and a fine +child model (`Child_GN`, 25x90x78, ncpl 7020). Confirmed the child's own +dims are threaded correctly (`{'nlay': 25, 'nrow': 90, 'ncol': 78, 'nodes': +175500, 'ncpl': 7020}` at the point `Child_GN.rch` loads -- no cross-model +dims leakage). The mismatch is in the fixture itself: +`Child_GN.rch.rech.ref` (the child's `OPEN/CLOSE`-referenced RECHARGE +array) contains exactly 26901 values -- the *parent* grid's total node +count, not the child's 7020-per-layer count. Read the referenced file +directly to confirm: it's really that many values, not a +parsing-side miscount. Same disposition as #3/#11 -- a mismatched/stale +fixture pairing (the ref file seemingly belongs to a different grid than +the model referencing it), not a flopy4 gap; there's no principled way to +reshape 26901 real values into a 7020-shaped slot. Left `xfail`. + +## 13. `OPEN/CLOSE ... (BINARY)` griddata arrays — never a corpus failure, IMPLEMENTED 2026-09-11 anyway + +Not one of the 108 originally-cataloged failures -- `_read_open_close_values` +had an explicit `raise ValueError(... "BINARY arrays are not supported +yet")` guard from the gap-#2 pass onward, and 0/242 corpus models ever +hit it (checked directly: searched every `.oc`/griddata-bearing file in +the corpus for `OPEN/CLOSE ... (BINARY)`, zero matches). Implemented +anyway on request, since real MF6 input can use it even though this +corpus never exercises it. + +**Format**: MF6's binary-array record is a 52-byte header (`KSTP`, +`KPER` int4; `PERTIM`, `TOTIM` float8; `TEXT` 16-byte label; `NCOL`, +`NROW`, `ILAY` int4) followed by `NROW*NCOL` values -- real arrays +double-precision, integer arrays 4-byte. This is the exact layout +flopy4's own binary head-output reader already decodes +(`utils/heads_reader.py::read_hds_timestep`, `f.seek(52, 1) # skip +kstp, kper, pertime`) -- MF6 has used one binary 2D-array record format +for both real and integer arrays since MODFLOW-2005 (`u2drel`/`u2dint`), +for output and OPEN/CLOSE (BINARY) input alike. + +**Verification, given no real fixture exists**: wrote a synthetic file +in this exact format and round-tripped it directly through +`_read_open_close_values` (real + integer + FACTOR-scaled), then +end-to-end through `Dis.load()` with a real `.dis` file referencing a +synthetic binary TOP array, confirming the values it produces match +what was written byte-for-byte. Added 3 permanent regression tests +(`test/test_package_load.py`) covering a plain real array, an integer +array with FACTOR, and a LAYERED field with one binary file per layer +(matching the existing per-layer text-CONSTANT/INTERNAL convention). +Corpus-wide: unchanged (239/242, expected -- nothing in the corpus +exercises this path either way), zero regressions. + +**Update -- verified against real MF6 fixtures**: `../modflow6/autotest` +(a sibling checkout of the actual MODFLOW 6 Fortran repo) has +`test_gwf_utl01_binaryinput.py`, MF6's own test for binary array input, +covering both shapes real MF6 supports -- one binary file per layer, and +one flat file spanning all layers. Ran its `build_models()` directly +(real `flopy` 3.10.0, `flopy.utils.BinaryHeader.create(bintype="HEAD", +...)` / `Util2d.write_bin`, the actual writer real MF6 reads output +from) to generate real fixtures, then loaded both through +`Simulation.load()`. Confirms the header format guess above exactly: +`bintype="HEAD"` is the same record layout. Values matched what +`build_models()` wrote in every field (top, botm, idomain, icelltype, +strt) for both fixtures. + +This also caught a real, separate bug the synthetic-only testing had +missed: flopy3's writer quotes OPEN/CLOSE filenames (`OPEN/CLOSE +'top.bin' ...`), and `word`'s tokenizer keeps the quote characters as +part of the token (`'` is a valid filename character in the class), so +`workspace / "'top.bin'"` (quotes and all) never resolved. Fixed with a +new `_strip_quotes()`, applied to both griddata (`_read_open_close_values`) +and list/period (`_resolve_open_close_rows`) filename resolution -- +this affected the plain-text OPEN/CLOSE path too, not just BINARY, +though 0/242 corpus fixtures happened to hit it on a *recognized* +field (35 corpus lines do quote an OPEN/CLOSE filename, but all 35 are +for a DISU `AREA` field flopy4 doesn't have a hand-written class for +yet, so the row is skipped before the filename is ever resolved -- +confirmed by checking directly, not assumed). Added a 4th regression +test (quoted single-quoted filename, both binary and text). Corpus-wide: +unchanged, 239/242, zero regressions -- consistent with that "never a +recognized field" finding. Full suite: 647 passed (+4 total from this +gap), same 7 pre-existing unrelated failures. + +No longer flagged as unverified -- confirmed against MF6's own binary +IO round-trip. + +## Noted in passing, not confirmed as any of the 108's root cause + +- `.obs` files (observation-package input) fail to parse at all: + `UnexpectedToken: Unexpected token Token('CNAME', 'FILEOUT') at line 4` + — seen on `test001e_UZF_3lay.obs`. Not necessarily on the load path + today (OBS attachment might not go through `_resolve_bindings` at all + yet), so didn't affect any of the 108 directly, but worth a look if/when + OBS loading is wired up. +- ~~Time-series (`.ts`) files use `!` as a comment/header marker (`! time + rate_1 rate_2`), which `SH_COMMENT` (`#`-only) doesn't recognize.~~ + **Resolved.** `basic.lark` and `typed.lark` ignore a `BANG_COMMENT` + (`/![^\n]*/`), whole-line or trailing (since #395), and TS6 files load as + `Ts` children. The TS models (test027_TimeseriesTest, test001e_UZF_TS, + test001e_UZF_3layTS, test020_NevilleTonkinTransientTS, test051_uzfp2TS) + are in `KNOWN_PASSING`. diff --git a/docs/dev/plans/archive/loader-roadmap.md b/docs/dev/plans/archive/loader-roadmap.md new file mode 100644 index 00000000..14eb5319 --- /dev/null +++ b/docs/dev/plans/archive/loader-roadmap.md @@ -0,0 +1,180 @@ +# Input loader roadmap + +**Archived 2026-10-19.** Superseded: stage 0 merged as #482 (benchmarks), stage 4 as #474-#484 (see [`typed-loader-perf.md`](typed-loader-perf.md)). Stages 2-3 (typed wiring) are in [`typed-loader-wiring.md`](typed-loader-wiring.md), not started. + +Status as of 2026-09-24: what's done, what we learned, and how the remaining +loader work is staged. This is the index. Details live in the docs and plans +linked below. + +## Where things are + +**Branches on the fork (`origin` = wpbonelli/pyphoenix-project), no PRs:** + +| branch | contents | status | +|---|---|---| +| `typed-remarks` (off `develop`) | parse-stage benchmark, basic vs. typed; [`typed-grammar-remarks.md`](typed-grammar-remarks.md) (remarks and performance analysis). Uses the old names (`bench`, `test_mf6_reader_benchmark.py`, `--bench-manifest`); `load-baseline` renames them. | superseded by `load-baseline` for review | +| `load-baseline` (on `typed-remarks`) | both benchmark suites under their final names: `bench-parse` (`test_mf6_parse_benchmark.py`, flopy4 basic vs. typed parsers) and `bench-load` (`test_mf6_load_benchmark.py`, flopy3 vs. flopy4 full load, 165 pinned models); `bench` runs both. `docs/dev/load-benchmarks.md` | ready for review; contains `typed-remarks` | +| `typed-no-remarks` | experiment: remark support removed from the typed grammar | reference only, never merge; lacks the later doc commits | + +**Worktrees:** `~/dev/pyphoenix-typed-remarks`, +`~/dev/pyphoenix-load-baseline`. + +**Plans (untracked, project root):** +- [`load-baseline-plan.md`](load-baseline-plan.md): done. +- [`typed-loader-wiring.md`](typed-loader-wiring.md): not started. Stages 2-3 below. +- [`typed-grammar-trailing-remarks.md`](typed-grammar-trailing-remarks.md): the original open question, since + answered. +- [`optimization-plan.md`](optimization-plan.md): basic-loader performance steps. Not started; + stage 4 below. + +## What we learned + +1. **Trailing remarks are load-bearing.** Removing them breaks 357 more + corpus files (47 → 404 failures), ~90% of them MF2005-style labels on + array control lines (`CONSTANT 100.0 DELR`). They cost nothing + measurable at parse time, only ~0.5 s of one-time grammar build. So + any case for dropping them rests on grammar complexity, not speed. +2. **The typed loader isn't faster, and moving logic into the grammar + won't make it so.** + - Typed parse is ~1.07x basic; parse + transform is ~1.25x. + - ~87% of a basic `Simulation.load` is Lark (lex, parse, tree); the + structuring the typed grammar was meant to absorb is ≤13%. + - Pure-Python grammar rules cost the same interpreter time as + post-processing, plus tree overhead. + - The headroom is bulk numeric data. Extracting it in C is 10-30x + faster than Lark's lexer alone. +3. **flopy4-basic vs. flopy3** (load + materialize, 165 models, equal work + verified per model): + - 14.44 s vs. 17.01 s overall, but one model (henrytidal) accounts for + ~10 s of flopy3's time. Without it, flopy4 is slower (9.99 s vs. + 6.24 s), though it wins on 127 of 165 models (less fixed cost). + - ~5x slower on large inline arrays (Lark cost per token). + - ~3x faster on `OPEN/CLOSE` arrays (read with numpy). This is + independent evidence for point 2's numpy idea. + - ~2.5x slower cold start, mostly imports: flopy3 for grids (~0.64 s) + and xugrid (~0.45 s). + - 74 models are excluded for unequal work (TS/OBS child files, DISU, + LAK tables). They need a new baseline when flopy4 supports them. +4. **The typed transformer has real bugs** that the parse-only corpus + test couldn't see, including silent data loss (TDIS + `START_DATE_TIME` → `'1997'`). The typed loader isn't wired into + `Simulation.load` / `Package.load` at all. + +## Staging + +**Stage 0: land the measurement baseline.** +- One PR from `load-baseline`, which includes `typed-remarks`. It's tests + and docs only, with no production code change. +- Everything after this is measured against it. +- `typed-no-remarks` stays on the fork as a reference. + +**Stage 1: small independent PRs off `develop`** (any order, in parallel +with other work): +- Typed transformer silent-loss fixes (wiring plan step 3a): the + `START_DATE_TIME` truncation, remark tokens in data rows, the + `layered`/`netcdf` slot mix-up, and integer arrays parsed as doubles. + Each gets a regression test. +- Cold-start imports: make the flopy3-for-grids and xugrid imports lazy + (optimization plan step 1). +- Grammar/DFN drift: regenerate against a pinned DFN version and remove + the 17 orphaned grammars, as its own labelled change. + +**Stage 2: typed wiring, the parts that don't touch the converter** +(wiring plan steps 1, 5, 2, 3b): +1. `ReadContext` switch (`basic` / `typed` / `typed-fallback`) plus the + parity test. At first the parity test reports how many models match; + that number is the progress metric. +2. Runtime schema from generated classes (option b): audit what the + transformer needs, then prototype b1 (read class metadata) against b2 + (codegen emits a per-component transformer) on 2-3 components. Review + the choice before committing to one. +3. Specify the typed output representation: numpy not xarray, no `Tree`s + left in the output, real list-block names. + +**Stage 3: typed structuring** (wiring plan step 4), **after the converter +simplification lands**. +- `structure.py` is also being reworked on `pyphoenix-simplify-converter`, + which is itself waiting on `roundtrip-fixes`. Build the shared + per-field-kind back end on the simplified converter, or fold it into + that work. +- Then one PR per block kind: options/dimensions → lists/periods → + griddata → name-file bindings. Parity should rise with each. +- Then add a `reader` parameter to `bench-load` for the first full + three-way comparison (flopy3 / basic / typed). + +**Stage 4: performance, in parallel with stages 2-3** (details in +[`optimization-plan.md`](optimization-plan.md)). These target the **basic** loader users run today, +so they don't wait on the typed wiring. Each is one PR, with `bench-load` +numbers before and after and a clean value-snapshot check: +0. Value-snapshot check (prerequisite). +1. Lazy imports (cold start). +2. Tolerant bulk external-array reader, with a count check. +3. Bulk numeric-run token in `basic.lark`, covering inline arrays and + all-numeric list rows (the big one; estimated ~2x faster than flopy3 + overall). +4. Lists with words (boundnames, keywords), if still worth it after step 3. +5. Lark inline transformer / `cache=True`, for whatever still goes + through Lark. + +The typed grammar gets the same bulk token when it's wired in. After +that, revisit remarks with the cheaper rest-of-line remark terminal; by then +remark use can be counted, so a deprecation argument has data. + +## Deferred: lazy external arrays (and an xarray backend) + +**The idea.** Read externally stored array data only when it's accessed, +not at load time. Today flopy4 reads every `OPEN/CLOSE` array eagerly. In +the load-baseline corpus that's 18 models and 12.6 MB of array files, +sometimes most of a model's bytes. Laziness would make those loads nearly +free, and users who touch only some arrays would never pay for the rest. + +**Evidence it matters:** +- flopy3 already defers `OPEN/CLOSE` array reads until `get_data()`. That + is one reason the benchmark has to materialize everything to compare + fairly. +- Unlike flopy3, which re-reads the file on *every* access with no + caching, flopy4 should read once and cache. + +**Shape of it:** +- Binary `OPEN/CLOSE ... (BINARY)` arrays: memory-map the file + (`np.memmap`, after the MF6 binary header) and wrap it as a lazily + indexed array. Slices never touch the rest of the file. +- Text `OPEN/CLOSE` arrays: can't be indexed without parsing, so defer the + whole read to first access. That's a `dask.delayed` read wrapped in a + `dask.array`, or a small lazy array type. Read once, then cache. +- The object model holds these lazy arrays in place of numpy arrays. Shape + and dtype are known at load from dims and the field type, so validation + that needs only shape still works without reading. +- Writing: an unmodified lazy array can be written back as the same + `OPEN/CLOSE` reference without being read (flopy3's `lazy_io` does the + same). That's a round-trip win too. +- Inline arrays (inside the package file) stay eager: they have to be + parsed to get past them anyway. + +**Things to settle first:** +- Which consumers assume a concrete `np.ndarray`: converters, `to_xarray`, + writers, validation. The converter simplification is the natural time to + find out. +- File-change semantics: what happens if the external file changes between + load and first access? Probably document "read at first access" and move + on. +- Benchmark impact: `bench-load`'s "load + materialize" stage stays the + fair comparison. A "load only" number will drop sharply and shouldn't be + quoted alone. + +**Xarray backend.** An `engine="mf6"` backend (`open_dataset` / +`open_datatree` → `load(...).to_xarray()`) is cheap given the existing +`to_xarray()` methods, but adds no speed and is a lossy view of input +files (options, records, bindings). If done, keep it a thin wrapper that +the reader never depends on. Lazy external arrays, built into the loader, +are the part of that idea worth pursuing, and a backend would get them for +free. Not on the critical path; revisit after stage 3. + +## Open decisions (yours) + +- Stage 0 as one PR or two (`typed-remarks`, then `load-baseline`). +- Stage 3: wait on the converter simplification, or merge the two. +- When the `roundtrip-fixes` PR goes up; the converter simplification is + blocked on it. +- flopy3 `lazy_io=True`: add it as an extra benchmark column? It would + change the henrytidal picture. diff --git a/docs/dev/plans/archive/mf6-object-model-plan.md b/docs/dev/plans/archive/mf6-object-model-plan.md new file mode 100644 index 00000000..a2abcfa9 --- /dev/null +++ b/docs/dev/plans/archive/mf6-object-model-plan.md @@ -0,0 +1,1036 @@ +# MF6 object model plan + +**Archived 2026-10-19.** Done: see its 2026-09-16 status update. Its remaining items are resolved too: the reader is off the legacy DFNs (#455), OC `ocsetting` is typed ([`nested-keystring-union-plan.md`](nested-keystring-union-plan.md)), and namefile grammars are generated; wiring them in is [`typed-loader-wiring.md`](typed-loader-wiring.md). + +## Background + +This plan grew out of a review of two things: + +1. PR #336 ("decouple xattree from codegen with api changes"), which rewrote + the codegen-v2 package pipeline (`Package` subclasses: Npf, Ic, Chd, ...). + It also added a `chunks=` parameter to `Package.load()` that wraps an + already-eagerly-loaded numpy array in dask post hoc — this provides no + real streaming/lazy read benefit (confirmed by the PR's own profiling + comments) and should be removed; the eager `Package.load(path, dims)` + path itself is worth keeping. +2. PR #284 ("input file loading", branch `load`, open, rough), which + prototypes recursive namefile loading: `Component.load()` no longer + manually loops over `self.children`; instead, a namefile's + `packages`/`models`/`exchanges`/`solutiongroup` blocks are parsed as + binding records, and a `Binding.to_component()` helper resolves each one + by looking up its type token in an `FTYPES` registry and recursively + calling `component_cls.load(path)`. This recursive-via-structuring + design is good and should be adopted. However PR #284's base predates + PR #336 by 27 commits — its `structure.py` is written against the old + xattree-spec system (`get_xatspec`, `xattree.has`, etc.) and can't be + merged as-is against current `develop`. + +## Key correction driving Phase 0 + +PR #336 introduced a new, ad hoc field-metadata convention for codegen-v2 +package fields (`metadata["dfn_block"]`, `metadata["schema"]`, +`metadata["oc_action"]`, detected via `has_dfn_metadata()`/`"dfn_block" in +f.metadata`), distinct from the metadata convention used everywhere else in +the codebase (`metadata["block"]`, set via `flopy4/mf6/spec.py`'s +`field()`/`array()`/`dim()`/`coord()` helpers, which wrap real `xattree` +field descriptors from `flopy4/spec.py`). `Model`, `Simulation`, `Gwf`, and +all other hand-written components still use the `block=` convention; only +the codegen-v2-generated leaf packages deviated. + +This spanned both directions of the codec: ingress (`structure.py`'s +`_structure_codegen_v2` reading `dfn_block`/`schema`/`oc_action`), egress +(`unstructure.py`'s `has_dfn_metadata()` branching on which convention a +class uses), and `Package.__attrs_post_init__` (`package.py`) branching on +`"dfn_block" in field.metadata` to decide whether to run v2-specific +post-init logic. + +Having two conventions is why a generalized structuring/binding-resolution +step (needed for namefile loading) would otherwise have to understand both. +Rather than build that dual-support permanently, fix the root cause first: +realign codegen-v2 field metadata onto the established `block=`/xattree +descriptor convention. This is a **prerequisite**, not an optional cleanup +— it removes the fork before more code gets built on top of the wrong side +of it. + +**Scope/risk note:** 76 generated package files carried `dfn_block` +metadata (all of `gwf/`, `gwt/`, `gwe/`, `prt/`, `utl/`, `exg/` except +`dis`/`disv`/`tdis`/`ncf`, which are hand-written). Fixing the generator and +regenerating touched all of them — needed careful, incremental validation +against `test_mf6_codec.py`, `test_mf6_codegen.py`, +`test_converter_structure.py`, `test_dataframe_api.py`, and +`test_row_api.py` — the test files PR #336 rewrote, since those encoded the +to-be-changed behavior. + +--- + +## Phase 0 — Realign codegen-v2 field metadata onto the `block=` convention + +**Status: done.** Landed in `3eda3f9` ("consistent field metadata approach", +#339). One field-metadata convention is now used by every component, +hand-written or generated: `flopy4/mf6/spec.py`'s `field()`/`path()`/ +`array()` carry `block=`/`schema=`/`oc_action=` as first-class kwargs. +`dfn_block`/`has_dfn_metadata()` no longer exist anywhere in the tree, and +the ingress/egress/`__attrs_post_init__` dual-path branches described above +are gone — one structuring path, not two. The `chunks=` removal (Background +item 1) landed in the same commit. + +--- + +## Phase 0.5 — Eliminate the `dfn_type` field-metadata key + +**Status: done except step 9 (`Column`) — folded into Phase 0.6.** Landed +in the same `3eda3f9` commit as Phase 0. `dfn_type` duplicated information +already recoverable from the Python type hint (scalar fields) or the +`inout` metadata key (file-record fields), so it's gone from `field()`/ +`path()` and from codegen. Griddata and READARRAY period fields — the only +two categories ever dask-backed — now type as `IntArrayLike`/ +`FloatArrayLike`, a dtype-parameterized structural `Protocol` in +`flopy4/mf6/_types.py` (matches numpy, dask, and any other duck-array type +with no per-library import, unlike the old explicit `Union[NDArray, +da.Array]`). The only remaining `dfn_type` occurrences live inside +`Column`/`Schema` (`flopy4/mf6/schema.py`), which Phase 0.6 removes +wholesale. + +--- + +## Phase 0.6 — Eliminate `Column`/`Schema`; make `Row` the sole row-item +schema, described with `field()` (no new builder) + +**Motivation.** Every list/recarray block (packagedata, connectiondata, +period, ...) is currently described **twice** from the same source data: +codegen emits both a `Column`-list `Schema` subclass (a hand-rolled +dataclass-based metadata system) and an `@attrs.define` `Row` nested class +with a hand-generated `__iter__`, from the identical schema dict. E.g. +`gwf/chd.py` carries both a `Row` class and a `_PeriodSchema(Schema)` class +describing the same 3 columns (`cellid`/`head`/`boundname`), with +`Row.__iter__` manually kept in column-order sync with `Schema.columns()`. +Same "two conventions describing one thing" problem Phase 0 fixed for +scalar package fields, just not yet fixed for list-row fields. +`flopy4/mf6/record.py`'s `Record` mixin (used for tagged inline records +like `Ims.Rclose`) already demonstrates the target pattern: one +`@attrs.define` class, metadata-tagged fields, generic +`attrs.fields(cls)`-driven (de)serialization — no parallel schema +description needed. + +**Goal:** `Row` (and `PackagedataRow`, `ConnectiondataRow`, etc.) becomes +the *only* description of a list block's columns. `Column`/`Schema` are +deleted. No new field-builder function — Row fields use the *same* +`field()` used everywhere else, extended with a couple of narrow kwargs. +Auditing what `Column`'s attributes (`role`, `dfn_type`, `optional`, +`shape`, `time_series`, `dtype`, `prefix`) actually do — checked against +real usage and occurrence counts, not assumed — shows most are dead, +redundant with the field's own name/type, or a bug: + +| `Column` attribute | Occurrences | Disposition | +|---|---|---| +| `role="value"` | 121/182 (66%) | Dropped — the fallback branch everywhere it's read; not a role, the *absence* of one. | +| `role="cellid"` | 18/182 | Dropped — 18/18 occurrences are on a field literally named `cellid` (MF6's own DFN convention reserves that name). Inferred from the field name. | +| `role="boundname"` | 17/182 | Dropped — 17/17 always named `boundname`. Inferred from the field name. | +| `shape` | dead | Dropped — grepped every read site; `col.shape` is never consulted at runtime. `cellid`'s width (`ncelldim`) is always computed dynamically from grid type / token count. | +| `dtype` | buggy | Dropped — confirmed actively wrong: string columns tagged `dtype="np.object_"` get incorrectly typed `Union[float, str]` instead of `str` (e.g. `gwf/buy.py`'s `modelname`). Storage dtype should come *from* the Row field's own annotation via `to_field_type()` (same principle as Phase 0.5), not be a second, independently-settable fact that can drift from it. | +| `prefix` | 5/182 | Dropped as a bespoke string blob. All 5 are file-reference columns where MF6 writes fixed tokens before a filename (LAK `TABLES`: `ifno TAB6 FILEIN `; SSM `FILEINPUT`: `pname SPC6 FILEIN `; FMI: `FILEIN `) — retype these `Path` and reuse the *existing* `path(inout="filein")` convention instead of inventing new row-level machinery. | +| `time_series` | genuine | Kept as-is — `field()` already has this kwarg. Confirmed necessary: drives try-float-then-string *parse order*, not inferable from the `Union[float, str]` annotation alone. | +| `role="feature_id"` | 17/182 | Replaced — see below. | +| `role in ("keystring", "keystring_value", "inline_keyword")` | 9/182 | Dropped entirely — see below. | +| (implicit: `aux: tuple` positional field) | n/a | Kept, but synthesized by codegen for period blocks, not per-DFN-column. Still needs special (name-based) handling in the generalized iteration logic. | + +**`feature_id` → adopt `pk`/`fk` from modflow-devtools' DFN spec.** +`dfnspec.md` ("Primary/foreign keys") already formalizes this with +`pk`/`fk`/`fk_ref` attributes on integer/string list-item columns, a +better fit than flopy4's ad hoc `role="feature_id"`: `fk: "node"` is a +literal match for `role="cellid"`; `pk: true` / `fk: "packagedata."` +matches the rest of today's `feature_id` columns and — unlike flopy4's +flag, which only says "shift by one" — records *what the column refers +to*. The spec's jagged/variadic-array case (`gwf-sfr.connectiondata.ic`'s +shape given by `packagedata.ncon(rno)`) is a real MF6 pattern flopy4 has +no equivalent for today, and dovetails with Phase 1.4's dimension work. + +`pk`/`fk` are only populated in the DFN corpus for a subset of packages +today (see Phase 0.6a's Findings for current coverage). **Resolved:** +rather than have flopy4 carry a `numeric_index`/`cellid` fallback (an +attribute `dfnspec.md` no longer documents at all), the DFN corpus is being +backfilled with `pk`/`fk`/`fk_ref` upstream in modflow-devtools (owner: +wpbonelli). `make.py` should read `pk`/`fk`/`fk_ref` directly, no fallback +path. + +`pk`/`fk` aren't purely an int concern — `dfnspec.md`'s own example table +includes *string* pk/fk too (`gwf-mvr.packages.pname`/`period.pname1`, +currently untagged upstream). The 1-based↔0-based conversion is meaningless +for a string reference, so the runtime rule should be "`pk`/`fk` set **and** +the field is int-typed → apply the shift," not "any tagged column → +shift." + +**Scope boundary:** `fk_ref` (runtime component resolution) and the +jagged/variadic-row-array case are **out of scope for Phase 0.6** — flopy4 +has no variadic-row-array support today, and building it is new capability, +not a schema-representation cleanup. Phase 0.6 only needs `pk`/`fk` far +enough to drive the existing 1-based↔0-based conversion; richer resolution +belongs with Phase 1's dimension work or a dedicated follow-up. + +**`keystring`/`keystring_value`/`inline_keyword` — resolved: drop all +three, no replacement metadata needed on `Row`.** + +- **`inline_keyword` dissolves into "just a `bool` field."** + `flopy4/mf6/record.py`'s `Record.to_tokens` already emits *any* + bool-typed field (tagged or not) as its own uppercased name when `True` + and nothing when `False` — exactly SSM's `mixed` column. A plain + `mixed: bool = False` field gets this behavior for free once row + iteration goes through the same generic logic. +- **`keystring`/`keystring_value` were never a real implementation of a + union — they're where "keystring" as a DFN concept was abandoned + upstream.** Traced in modflow-devtools' snapshot history: `keystring`/ + `choices` was renamed `union`/`arms` starting `dev1`, and every + occurrence of "keystring" is gone from the corpus and from `dfnspec.md` + since. `type: union` is now a proper first-class composite: a field + discriminated by a leading keyword token, each arm either a bare + scalar/keyword or (for `auxiliaryrecord`) a full nested `record`. + flopy4's `role="keystring"`/`"keystring_value"` predates this rename and + flattens the whole union down to a fixed two-column `{keyword: str, + value: object}` pair. + + This isn't just a naming issue — it's a **confirmed data-loss bug**. + Reproduced directly against `Lak._PeriodSchema`: parsing + `["1", "AUXILIARY", "concentration", "5.0"]` silently drops the `5.0` + (`auxval`) — there's no third column to catch it, because the flattened + schema has no concept of a multi-field arm. `structure.py`'s parse loop + and `unstructure.py`'s serializer both route `keystring`/ + `keystring_value` through the same single-token/generic-`value`-column + path — they carry **zero** distinguishing runtime behavior from an + ordinary `role="value"` column today. `gwf/sto.py`'s scalar + `role="keystring"` (`storagestate`) makes the same point from the other + direction: STO's `type: union` has two zero-payload keyword arms + (`transient`/`steady-state`), so it's processed identically to a plain + string column already — the tag adds nothing. + + Real union support — dispatch on the leading keyword token, per-arm + types, reusing `record.py`'s tagged-field machinery for record arms — is + genuine new capability, not a rename, and only touches 3-4 packages + (`gwf/lak.py`, `gwe/lke.py`, `gwt/lkt.py`, `gwf/sto.py`'s zero-payload + case). Out of scope for Phase 0.6, split out as **Phase 0.7**. Phase 0.6 + just deletes the roles and, for now, leaves LAK/LKT/STO period rows + exactly as buggy/simplified as they are today (not a regression) rather + than blocking `Column`/`Schema` removal on designing union support. + +**Steps. Status: all 12 done, landed 2026-08-19/20 in `6cdfb2a` (steps 9-10, +the pk/fk-tagged codegen rewrite) and `78c506b` (steps 3-8, 11-12, the +Row-mixin runtime rewrite).** `flopy4/mf6/schema.py` is deleted; +`flopy4/mf6/row.py` (a `Row` mixin, matching `record.py`'s `Record` +pattern) is now the sole row-item schema, introspected via +`attrs.fields()` by `package.py`, `structure.py`, and `unstructure.py` +directly — no separate `Schema`/`Column` lookup anywhere. All 76 (now 64, +after some consolidation in the dev3 regen) generated `Row` classes carry +real `field(pk=..., fk=..., time_series=...)` metadata sourced from the +dev3 DFN corpus. Two parsing bugs were found and fixed along the way (see +`78c506b`'s commit message): `row_class()` codegen was placing the +synthetic `aux` field ahead of a package's own optional non-aux columns +(wrong token order vs. the real DFN, e.g. EVT's `pxdp`/`petm`/`petm0`), and +`Row.from_row()` greedily consumed tokens for optional columns without +knowing whether they were present in a given row; it now infers the count +of trailing optional columns from the remaining token budget, same as +`naux`/`ncelldim` already did. Full suite green except two pre-existing, +unrelated integration-test failures (xattree mangling an explicit +`multi_package` `name=`, e.g. `"LAK-1"` → `"LAK-10"`). + +1. ~~Extend `field()` in `flopy4/mf6/spec.py` with `pk: bool = False`, + `fk: str | None = None`.~~ **Done** (`f1dd830` / #341). +2. ~~Retype the 5 `prefix=`-using file-reference row columns as `Path` + fields via `path()`.~~ **Done** (`f1dd830` / #341). +3. ~~Build the *name-based* half of the generalized row-iteration mixin + (`cellid`, `boundname`, `aux`).~~ **Done** — `flopy4/mf6/row.py`. +4. ~~Add the `pk`/`fk`-driven half of the mixin.~~ **Done** — + `flopy4/mf6/row.py`; sourced from real dev3 `pk`/`fk` tags via + `6cdfb2a`'s codegen rewrite. +5. ~~Update `package.py`/`structure.py`/`unstructure.py` to read + `attrs.fields(RowClass)` instead of `Schema.columns()`.~~ **Done.** +6. ~~Drop the `__{block}_schema__`/`__period_schema__` ClassVar + aliases.~~ **Done.** +7. ~~Drop `role in ("keystring", "keystring_value", "inline_keyword")` + entirely.~~ **Done** — superseded by real `Union.arms`-driven detection + from `6cdfb2a` (see Phase 0.6b/0.7 below); LAK/LKE/LKT/SFR keystring + period rows still use the flat `(index, keyword, value)` output shape + for now (typed union-item-class representation is a deferred follow-up, + see Phase 0.7 status below), but the detection mechanism itself is now + schema-driven, not a hardcoded `role=`. +8. ~~Delete `flopy4/mf6/schema.py`.~~ **Done.** +9. ~~Codegen (`make.py`, `filters.py`): read `pk`/`fk`/`fk_ref` directly + off the dev3 DFN attributes.~~ **Done** — landed as part of `6cdfb2a`'s + full pydantic-native rewrite (Phase 0.6a step 2), not a standalone + pk/fk-only pass as originally scoped; see that phase for detail. +10. ~~Regenerate all codegen-v2 packages; diff to confirm `Column`/`Schema` + gone.~~ **Done.** +11. ~~Rewrite tests: delete `test_mf6_schema.py`; fold assertions into + `test_mf6_row_api.py`.~~ **Done.** +12. ~~Re-run the full suite.~~ **Done**, green modulo the two pre-existing + unrelated xattree failures noted above. + +**Why before Phase 1, not after:** Phase 1's generalized structuring pass +wants exactly one field-introspection idiom to special-case (`field()`- +tagged attrs fields, resolved via `attrs.fields()`). Doing this first means +there's no second, `Schema`-shaped idiom left for that pass to also learn +about — and if `pk`/`fk` land now, Phase 1's binding resolution and +Phase 1.4's dimension propagation inherit real relational metadata instead +of reverse-engineering it later. + +--- + +## Phase 0.6a — Migrate flopy4 off `modflow_devtools.dfn` onto +`modflow_devtools.dfns` (dev3, pydantic-native) + +**Status: step 2 (the atomic pydantic-native rewrite + version flip) done, +landed 2026-08-19 in `6cdfb2a`. Step 3 (codec reader path) not started — +confirmed `flopy4/mf6/codec/reader/dfn2lark.py`, +`flopy4/mf6/codec/filters.py`, `flopy4/mf6/codec/reader/transformer/typed.py`, +and `flopy4/mf6/codec/reader/grammar/filters.py` still import legacy +`modflow_devtools.dfn` at `schema_version="2.0.0.dev1"`.** As noted below, +step 3 is decoupled from step 2 with no ordering dependency, so this is +expected, not a regression — it's simply still open. Step 1's devtools-side +`numeric_index` fix landed upstream and was verified: all 5 previously- +missing fields (`buy.irhospec`, `csub.icsubno`, `vsc.iviscspec`, +`prp.irptno`, `ats.iperats`) carry `pk=True`, and LAK's relational case +(`outlets.lakein` → `fk="packagedata.ifno"`) was unaffected. No temporary +override stopgap was needed. + +Prerequisite for Phase 0.6 step 9, discovered while attempting to start it: +`pk`/`fk` only exist in devtools' newer `modflow_devtools.dfns` module +(pydantic schema, discriminated `Scalar | Array | Record | Union | List` +field types, currently `CURRENT_SCHEMA_VERSION = "2.0.0.dev3"`). Every +flopy4 consumer of DFN metadata — `spec.py`, the codegen utils +(`make`/`filters`/`overrides`/`dfn2py`), and the codec reader chain +(`dfn2lark`, grammar filters, transformer) — still imports the older, +now-deprecated `modflow_devtools.dfn` module (flat TypedDict `Field`, no +`pk`/`fk` at all). Regenerating against real `pk`/`fk` metadata requires +this migration first. + +**Decisions (locked 2026-07-16):** +- Target schema version `"2.0.0.dev3"` (current), not an earlier pin. +- Adopt the pydantic `Field`/`Block`/`Component` types **natively** + throughout — no permanent translation shim back to the legacy TypedDict + shape (would recreate the "two conventions" problem Phase 0 eliminated). +- `overrides.py`'s dict-patching layer compensates for `dfn2toml`'s lossy + conversion, which is being fixed upstream directly (owner: wpbonelli, in + progress) — don't port its *internal patch logic* as-is, revisit/shrink + once the upstream fix lands. Its *call boundary* does move now, in + lockstep with `filters.py`. + +**Findings:** +- `LocalDfnRegistry.spec()`/`RemoteDfnRegistry.spec()` already do + fetch+migrate+validate in one call for the current schema version — no + separate manual `migrate()` step needed. But flopy4's sync command never + calls `.spec()` for field metadata today — it fetches raw `.dfn` files + and re-parses them itself via the legacy loader, discarding the registry + entirely for this purpose. A `dfns=` bridge parameter already exists in + the codegen entrypoint but was never wired to a real `dev3` call. +- pk/fk corpus coverage, checked at the field level: of the 17 + `role="feature_id"` columns across flopy4's 76 generated packages, only + the LAK family (12 of 17) has real `pk`/`fk` upstream. The remaining 5 + (`buy.irhospec`, `csub.icsubno`, `vsc.iviscspec`, `prp.irptno`, + `ats.iperats`) are self-referential primary keys on their own block, + needing only `pk: true`, no `fk`. + - **Root cause, not just missing backfill**: the v1→v2 migration + script's relation-resolver derives pk/fk purely from two structural + signals (v1 recarray shape lookups, same-field-name recurrence across + sibling blocks) and never consults the legacy `numeric_index` flag at + all — a field with `numeric_index: true` that doesn't also match one + of those two signals is silently dropped. Fix (owner: wpbonelli, + separate devtools-side track, in progress): add `numeric_index: true` + with no structural match → `pk: true` as a third signal. + - Not a blocker for the migration itself (the new schema drops + `numeric_index` entirely regardless) — only affects the *timing* of + step 9's final regeneration, which should run after the devtools fix + lands or those 5 fields silently lose their `pk`. + - Not relevant to current scope: MAW/SFR/UZF and gwt/gwe equivalents + already have upstream `pk`/`fk` but aren't generated in flopy4 yet; + MVR's string pk/fk are also still untagged upstream (fine to defer, no + behavioral effect until Phase 1's relational resolution). + +**Choreography.** `registry.spec(schema_version=X)` is strictly binary — +there's no intermediate shape between the legacy fallback and the pydantic +migrate/validate path, so there's no valid state where the `filters.py`/ +`make.py` rewrite has landed but the version string still says `dev1`. +That means the rewrite, the version-string flip, and `overrides.py`'s +call-boundary update have to land as **one atomic unit**, not staged +separately. `spec.py` is small enough to land with the same rewrite. The +codec/reader path (`dfn2lark.py` etc.) loads DFNs via its own, wholly +separate call and never goes through the codegen entrypoint's version +constant — genuinely decoupled, can migrate on its own schedule. + +**Steps:** + +1. ~~Wire the real entrypoint~~ — **done** (`registry.spec()` plumbed + through the codegen CLI, still requesting `dev1` at the time, pure + refactor). +2. ~~**Atomic pydantic-native rewrite + version flip.**~~ **Done**, + `6cdfb2a` (2026-08-19). Also merged in Phase 0.6b + 0.7's scope where the + runtime already supported it (see those sections) — broader than + originally scoped for this step alone, but landed atomically as planned: + - Rewrite `filters.py`, then `make.py`, to walk the pydantic + `Component`/`Block`/`Field` tree (`isinstance` dispatch over + `Scalar | Array | Record | Union | List`) instead of dict `.get()` + probing on the legacy TypedDict. `filters.py` first since `make.py`'s + templates call into its leaf-level field-metadata emission. Largest, + highest-risk part of the whole plan. Prefer devtools' own built-in + traversal (`ComponentBase.get_fields(recurse=True)`, `get_block()`) + over reimplementing a flopy4-side block-threading equivalent — + confirmed both already exist and cover what's needed. + - Scope explicitly **excludes** redesigning OC-record and + keystring-period handling to reflect DFN structure literally instead + of flattening (see Phase 0.6b) — carved out so this step's + regenerate-diff stays clean (identical except pk/fk). Port today's + flatten behavior as-is; 0.6b replaces it afterward. + - Update `overrides.py`'s call boundary to the new field type; leave + internal patch rules mostly as-is (expected to shrink once the + upstream TOML fix lands). + - Update `spec.py`'s field-type import source and construction to match + the concrete pydantic types now in play. + - Develop/test against dev3 fixtures loaded directly + (`LocalDfnRegistry(path=...).spec(schema_version="2.0.0.dev3")`), + independent of the production entrypoint's constants, so the rewrite + can be built and verified before touching it. + - Land the version-string flip as the final commit of this same PR, + once everything above passes against dev3 fixtures. + - Regenerate all 76, diff, review, commit. Expect the LAK family to gain + real `pk=`/`fk=` tags; everything else field-declaration-identical, + *provided* the devtools-side `numeric_index` fix has landed first — + otherwise the 5 affected fields silently lose their `pk`. If timing + doesn't line up, add a temporary 5-field `pk: true` patch in + `overrides.py` as a stopgap and drop it once the upstream fix ships. +3. **Codec reader path** (decoupled, independent schedule). **Status: not + started.** Same tree-walk conversion for the reader/grammar chain. + `field["block"]` isn't a per-field attribute in the new schema (block + membership is structural via `Block.fields`), but + `ComponentBase.get_block()` already does this reverse lookup. Own + independent `schema_version` literal, no ordering dependency on step 2 — + can land any time, including in parallel with Phase 1. + +**Ordering:** step 2 is the only place the devtools-side `numeric_index` +fix matters, and only for the regenerate-and-diff sub-step, not rewrite +correctness (mitigated by the temporary-override fallback if needed). Step +3 has no dependency on step 2. Step 2 blocks Phase 0.6 step 9 specifically +— nothing else. Phase 0.6b is carved out of step 2 and can land any time +after it. + +--- + +## Phase 0.6b — Reflect DFN structure literally instead of custom-flattening + +**Status: done (merged into `6cdfb2a`, 2026-08-19) for the scope that +landed — see "Revised plan" and the note at the end of this section for +exactly what shipped vs. what's still deferred.** Carved out +of Phase 0.6a step 2 mid-survey. No code changes yet; next step is the +concrete design pass for the `List[Union[Record]]`-shaped rendering (OC +records, keystring periods), then starting Phase 0.6a step 2's actual +`filters.py`/`make.py` rewrite. + +**Decision:** generated classes should mirror DFN block/field/record/ +union/list structure directly rather than being custom-flattened into +shapes invented during early codegen experimentation. Those flattening +transformations predate dev3's real discriminated `Union` type and were, +in every case found so far, workarounds for the old schema's inability to +represent the true structure — not worth preserving now that it can be +represented faithfully. Confirmed candidates: + +- **OC-family period records** (`gwf/gwt/gwe/prt-oc`): dev3 represents + `saverecord`/`printrecord` as a `List` field (`output`) whose `item` is a + `Union` with `saverecord`/`printrecord` `Record` arms — not two flat + sibling record fields, which is what the current v1-derived schema shows + and what today's codegen assumes. Current codegen further flattens this + into hardcoded per-rtype fields (`save_head`, `print_budget`, ...) via a + hand-maintained rtype table — redundant now that the valid rtype strings + are already in the schema. Replace with a representation that mirrors + the `List[Union[Record]]` shape directly; drop the hardcoded table. +- **Keystring periods** (LAK/SFR/MAW/UZF): currently consolidated into an + approximated `(number, keyword, value)` recarray — codegen's own comment + admits this is lossy ("a proper typed representation requires a sealed + class hierarchy or tagged variant type... deferred pending upstream + devtools schema work"). Dev3's real `Union` types resolve exactly the gap + that comment describes; replace with a faithful representation. +- **Standard stress-package period blocks** (CHD/WEL/DRN/...): current + v1-derived schema shows these as flat top-level `Array` fields per column + in the period block, with `cellid` hardcoded/synthesized (never a real + field). Confirmed dev3 represents these identically to any other list + block: a real `List[Record]` (`stress_period_data`) with `cellid` as a + genuine typed `Array` field (`shape=["ncelldim"]`), alongside + `head`/`aux`/`boundname` as normal Record fields — no synthesis needed. + The "period array fields vs. list block" special-case split, and the + synthetic-cellid injection, likely collapse into the same generic + list-block handling already used for `packagedata`/`connectiondata`/etc. +- **The `layered` flag** (griddata arrays like NPF's `k`/`k22`/`icelltype`, + STO's `ss`/`sy`): ~~dev3's `Array` schema has no `layered` field at all~~ + **correction (2026-08-18): this premise was wrong, see below.** Even in + the *current* generated code, `layered=True` sits alongside an unchanged + flat shape — it was never encoding a different in-memory shape, only a + text-serialization choice (whether an array may be split into per-layer + `CONSTANT`/`INTERNAL` blocks when written). The write path already + derives this dynamically and correctly, and the read grammar already + discriminates `single_array` vs `layered_array` from the literal token + present in the file, not from a prediction. The static-flag readers on + the ingress/netcdf side are instances of the same "static metadata + standing in for a per-instance, dynamically-derivable choice" pattern as + the OC rtype table. + + **Correction (2026-08-18):** "dev3 has no `layered` field" was an + artifact of a migration bug, not a schema design choice — `layered` was + being silently dropped by the v1→v2 migration script (never actually + removed from the schema) and has since been fixed upstream (devtools + `f3ae0f4`, "fix(dfns): migrate/render missing attributes" #345, landed + 2026-08-14): it's migrated onto `Array` correctly now and wired into + devtools' own `render()` so generated MF6IO-guide-style block templates + show `NAME [LAYERED]` correctly, matching the real reference guide. It + is real, intentional schema data, not vestigial — confirmed by walking + the commit history of `modflow_devtools/dfns/schema.py` back to its + original commit (`750f63b`, #311): `layered` was absent from `Array` + there too, so this genuinely was an unmigrated gap the whole time, not a + deliberate omission that got reversed. + + This does not change flopy4's read/write conclusion above — the + empirical derivation rule (`layered` iff `nlay` in shape) still holds, + and flopy4's writer/reader still don't need `layered` as an internal + behavior flag, since the writer already derives the split dynamically + and the reader already discriminates from the literal token in the + file. **Revised decision:** don't add `layered` back into flopy4's + generated field metadata (still unnecessary for read/write behavior), + but Phase 0.6a step 2's tree-walk should treat `Array.layered` as an + ordinary, always-present attribute (not something to special-case as + absent) when walking the pydantic schema. The ingress/netcdf fix + described above (stop assuming a field is statically layered-or-not) is + still worth doing — dynamic derivation is more correct than a static + default-`True` assumption — but it's now a clarity/correctness cleanup, + not a forced consequence of a schema gap that no longer exists. + +**Consequences:** changes the generated API shape for OC-family and +keystring-period packages (field names/types users see change, not just +internals) — touches downstream tests and any code constructing these +fields today. Also requires fixing the ingress `layered` use sites and +netcdf's default-`True` `layered` read to stop depending on a static flag, +deriving the same decisions dynamically instead (matching how the writer +already works) — `layered` is present in the schema, but flopy4 still +doesn't need it as a behavior switch. Deliberately **not** part of Phase +0.6a step 2's +landing so that step 2's regenerate-diff stays clean (identical except +pk/fk); lands as its own reviewable change, sequenced after or alongside +step 2. + +**Steps:** TBD, pending a concrete design for how a literal +`List[Union[Record]]`-shaped field (and similar) should render in generated +Python — this needs its own design pass before implementation starts. + +**Survey findings (structural survey, complete as of 2026-07-19).** +Loaded a real local DFN corpus (ahead of what flopy4 currently tracks — +fine for structural survey purposes) via +`LocalDfnRegistry(path=...).spec(schema_version="2.0.0.dev3")` and +introspected real component/block/field objects directly in a throwaway +script rather than reasoning from `schema.py` source alone — several +findings below only became clear by checking actual data. + +*Settled, no surprises — pure retype for Phase 0.6a step 2, `isinstance` +dispatch, same output:* `pk`/`fk`; file records (`Record` + one `File` +child, both `filein`/`fileout` directions); embedded compound records with +keyword trigger; aux variable lists (plain `Array(dtype="string", +shape=[])`); the `dimensions` block itself (a normal `Block` of `Integer` +fields); prefix/row-keyword columns in list blocks (the `is_prefix`/ +`is_row_keyword` distinction is carried by `Keyword` type + `optional` +exactly as today — `Record.fields` is an ordered dict in DFN declaration +order, so the accumulate-prefix algorithm ports directly); `cellid` in a +genuine static list block (checked MAW/SFR `connectiondata` — same real +`Array(dtype="integer", shape=["ncelldim"])` field already confirmed for +period blocks, no special casing beyond what's documented above); +collision-name resolution and dimension resolution for list blocks +(`List.shape` is a real `list[str]`, so string-parsing simplifies to a +direct list read; the alias/suffix-matching and `"maxbound"`-literal +fallback logic both carry over unchanged, verified against real examples; +`collision_names` itself is pure Python-side logic with zero dependency on +the source schema). + +*Needs new traversal, same output for now (step 2 scope):* OC records and +keystring periods need deeper `List`/`Union` tree-walking to locate, but +step 2 should port today's flattened *output* as-is — the redesign is +0.6b's job, not step 2's. + +*Delete/simplify rather than port (step 2 scope, net complexity +reduction):* the flat-fields/block-threading helper (devtools' own +recursive field/block lookups already cover this); the dual legacy-DFN/ +v2-DFN parameter threading used for numeric_index/block-schema v1-bridging +(vestigial now `pk`/`fk` are native everywhere — confirmed a real block +that today is only discoverable via the legacy-DFN fallback, e.g. SSM's +`fileinput`, is present natively as an ordinary `List` field under dev3, so +a single walk over the component's blocks replaces the current two-source +merge); the separate "readarray" detection flag for G-variant period +packages (confirmed G-variant period fields are plain `Array`s directly in +the period block, standard packages' period fields are `List[Record]` — +`isinstance` alone distinguishes them). + +*Phase 0.6b redesign candidates (API-shape change, own design pass, listed +above):* OC records; keystring periods; standard stress-package period +blocks; the `layered` flag. + +**Decision (2026-08-18): 0.6a step 2 and 0.6b are merged into one effort, +not staged sequentially.** Reasoning: building 0.6a step 2's tree-walk for +OC/keystring against the *old* flattened output requires the same +`List` → `Union.arms` → `Record.fields` descent that a faithful +representation needs — the "port as-is" step buys nothing but a second +rewrite of the same code shortly after, and produces two consecutive +breaking changes to the same ~10 packages instead of one. It also +surfaced a structural finding that makes deferring 0.6b actively wasteful: +OC's nested `ocsetting` union (`all|first|last|frequency|steps`) and +LAK's `laksetting` union are **the same mechanism** — a discriminated +union keyed by a leading keyword token, arms either bare scalars or +sub-records. Phase 0.7 (below) was scoped as a separate, later effort to +design exactly this mechanism for LAK/STO; designing it once now and +applying it to both OC and the keystring packages is more principled than +building a one-off OC flattener and a proper union type later. Standard +stress-package period blocks (CHD/WEL/DRN/...) have no such complication +— [`list-fields.md`](../lists/list-fields.md)'s `list[ItemType]` target already covers them exactly +as a plain `List[Record]`, so there's nothing "old-shaped" to carry +forward there either. + +**Union field Python shape (unblocks OC + keystring together):** +following [`list-fields.md`](../lists/list-fields.md)'s per-arm-class pattern, each union arm +becomes its own generated `attrs` item class; the field type is a Python +`Union` of those classes (e.g. `Optional[list[OcSaverecordItem | +OcPrintrecordItem]]` for OC's `output`, `Optional[dict[int, +list[LakStatusItem | LakStageItem | ... | LakAuxiliaryrecordItem]]]` for +LAK's `perioddata`). Parsing dispatches on the leading keyword token +against each arm's trigger keyword. Each item class carries its own real +`fk` where the schema has one (e.g. `LakRateItem.outletno: int = +field(fk="outlets.outletno")`) instead of collapsing to an untyped +`value` column. This generalizes the OC-specific per-rtype expansion the +current codegen already hardcodes (`_OC_RTYPES`) into one mechanism used +uniformly — no per-package rtype table needed, since arms are read +directly from the schema's `Union.arms`. + +**Course correction (2026-08-18, later same day): the union-item-class shape +above is validated but deferred, not wired in.** Built and verified the +tree-walker + union-item-class generator against real LAK/OC/SFR data +(scratchpad prototype, `composite_v3.py`) — it works and generalizes +correctly across three structurally different real cases. But before wiring +it into `make.py` for real, checked what actually *reads* the generated +output at runtime (`flopy4/mf6/converter/ingress/structure.py`, +`egress/unstructure.py`) and found they're driven generically by the +existing `Column`/`Schema`/`role=` mechanism (`flopy4/mf6/schema.py`) — +`role in ("cellid", "feature_id", "value", "boundname", "keystring", +"keystring_value", "inline_keyword")` — not per-package logic. That +mechanism has no way to represent "one of N distinct typed classes, chosen +by keyword" (what the union-item-class design produces); it can only +represent flat columns. Wiring the union-item-class shape into real +packages therefore needs new ingress/egress dispatch code, which hasn't +been scoped. Separately confirmed [`list-fields.md`](../lists/list-fields.md)'s `list[ItemType]` +target (bare attrs objects, no `Schema`/`Column` at all) **is not yet the +runtime representation for anything** — Phase 0.6 (which owns that +migration) is "steps 1-2 done, step 3+ next", still in progress, so +`Row`+`Schema`/`Column` is still what every currently-generated package +(including plain list blocks like LAK `packagedata`, not just the +keystring ones) actually round-trips through today. + +**Revised plan:** the 0.6a+0.6b codegen rewrite targets the *existing* +`Row`+`Schema`/`Column`/`role=` output shape, sourced from the dev3 +pydantic tree instead of the legacy schema — i.e., functionally reproduce +today's generated shape (including OC's per-rtype fields and the +`(index, keyword, value)` approximation for LAK/SFR/MAW/UZF period data) +from the new source, rather than emitting the union-item-class shape live. +Concretely this is *not* a return to "port the exact old flattening +code" — real improvements land immediately: `_OC_RTYPES`'s hardcoded table +is replaced by reading `rtype.valid` directly from the schema (confirmed +correctly populated, see `devtools/todo.md`), and the LAK/SFR/MAW/UZF +`extra_period_fields` TOML-override mechanism is no longer needed at all, +since the keystring arms are now discoverable directly from `Union.arms` +rather than requiring a hand-maintained override entry per package. Only +the *output shape* (flat `role="keystring"/"keystring_value"` columns, +not typed per-arm classes) stays as today, for runtime-compatibility +reasons. The union-item-class mechanism becomes a follow-up phase, to be +sequenced *after* Phase 0.6's `Row` migration actually lands (building +typed union arms on top of real `Row` objects, not on top of `Schema`/ +`Column`) rather than merged into this pass — the earlier merge decision +above (0.6a+0.6b+0.7 as one effort) undercounted this runtime dependency; +0.7's *design* (this section) still stands, only its landing sequencing +changes. `composite_v3.py`'s logic is worth keeping as a starting point +when that follow-up phase starts. + +**Landed 2026-08-19 in `6cdfb2a`** (the same commit as Phase 0.6a step 2 — +per the merge decision above, these were never separable in practice): the +`filters.py`/`make.py` tree-walk rewrite, OC's rtype table replaced by +reading `rtype.valid` directly from the schema, and LAK/LKE/LKT/SFR-style +keystring period settings now detected from the schema's real `Union.arms` +instead of a hand-maintained override list. **What did *not* land** (per +that commit's own message, and per the "Revised plan" above): the +union-item-class shape itself — real per-arm typed classes — is still not +wired into the generated output. Detection is schema-driven now; the +*generated Python shape* for keystring/union period rows is still the flat +`(index, keyword, value)`-approximation output, same as before, because +`structure.py`/`unstructure.py` only understood that flat vocabulary at +the time. **This blocker is now gone**: Phase 0.6's `Row` migration landed +the next day (`78c506b`, 2026-08-20), so the union-item-class mechanism +(prototyped and validated in a scratchpad script, `composite_v3.py` — not +committed, doesn't survive between sessions, would need re-deriving) is now +a well-scoped, unblocked follow-up phase rather than a hypothetical one. +Not required for Phase 1 (namefile loading doesn't touch period-row +parsing), so it's optional/parallelizable — see "Suggested order" below. + +--- + +## Phase 0.7 — Model DFN `union` fields properly (replaces `keystring`/ +`keystring_value`/`inline_keyword`) + +**Status (2026-08-20): partially landed, real scope remaining.** The +*detection* half (recognizing `type: union`/`arms` in the schema instead of +a hand-maintained override list) absorbed into and landed with the merged +Phase 0.6a+0.6b effort (`6cdfb2a`) as described there. The *representation* +half — this phase's actual goal, a real discriminated-union Python type +replacing the lossy flat `{keyword, value}` pair, fixing the reproduced +`AUXILIARY` data-loss bug below — has **not** landed; `6cdfb2a` explicitly +kept the flat output shape because `structure.py`/`unstructure.py` didn't +support anything richer at the time. That blocker is gone now that Phase +0.6's `Row` migration landed (`78c506b`, 2026-08-20) — this phase is +unblocked and its design steps (1-6 below) are still accurate as written. +Not required for Phase 1; scoped to 3-4 packages +(`gwf/lak.py`/`gwe/lke.py`/`gwt/lkt.py`/`gwf/sto.py`) and parallelizable +with the namefile-loading phases. Kept here for the scope list and the +`AUXILIARY` regression case, both still relevant and still unfixed. + +**Goal:** give list-item records a real discriminated-union field type, +matching modflow-devtools' `type: union`/`arms` (see `dfnspec.md`, +"Union"), and use it for LAK/LKT/STO period settings instead of the +flattened, data-lossy `{keyword, value}` pair Phase 0.6 deletes. This is +new capability (and a bugfix — see the reproduced `AUXILIARY` data-loss +case in Phase 0.6), not a rename, so it's split out from Phase 0.6's +`Column`/`Schema` removal. + +**Scope:** 3 packages today (`gwf/lak.py`, `gwe/lke.py`, `gwt/lkt.py` — +the `laksetting`-shaped period union) plus `gwf/sto.py`'s `storagestate` +as a degenerate zero-payload special case of the same mechanism. + +1. Design the Python shape of a union field: a small dispatcher (sibling + to `flopy4/mf6/record.py`'s `Record`, or an extension of it) that reads + the leading keyword token, looks it up against the arms, and either + parses a bare scalar (most arms: `stage`, `rainfall`, `status`, ..., + each with its own `time_series`/dtype per the DFN) or delegates to + `Record.from_tokens` for a multi-field arm (`auxiliaryrecord`). + Serialization is the mirror via `to_tokens`. +2. Decide the generated Python type: likely `Union[, ]` on the `Row` field, or a small wrapper + type — needs a concrete design pass, not assumed here. +3. Update codegen (`make.py`/`filters.py`/templates) to emit this shape + from DFN `type: union`/`arms` instead of the current `role="keystring"` + two-column flattening. +4. Update `structure.py`/`unstructure.py`'s row parse/serialize to + dispatch through the union instead of the generic `keystring_cols` + branch being deleted in Phase 0.6. +5. Regression test: the reproduced `AUXILIARY ` + round-trip case above must preserve both values. +6. Regenerate `lak.py`/`lke.py`/`lkt.py`/`sto.py`; re-run + `test_mf6_row_api.py`/`test_mf6_codec.py` for these packages + specifically. + +**Ordering:** independent of Phase 1/2 (parallelizable, like Phase 3) — +doesn't block namefile loading, since it only affects LAK/LKT/STO period +parsing correctness. Can land any time after Phase 0.6 removes the old +roles it replaces. + +--- + +## Phase 1 — Generalize structuring to resolve bindings + +**Status: done (2026-08-20).** `Simulation.load("mfsim.nam")`/`Gwf.load( +"model.nam")` now recursively resolve `packages`/`models`/`exchanges`/ +`solutiongroup` rows into real loaded, attached children, with dims +threaded from a `dis`-like sibling to the rest of its block. Landed +alongside two real, previously-latent bugs this work surfaced: + +- `Component.load()`/`Context.load()` were calling a dead code path + (`flopy4.mf6.converter.structure()`, a bare `cattrs.Converter.structure` + with no hook registered for the abstract `Component` base) — confirmed + by direct repro (`TypeError: node name must be a string or None`) and by + the fact that no test exercised it against real files, only a fully + mocked loader. Fixed by routing `_load_mf6` through the same + `structure_component()` pipeline `Package.load()` already used + successfully. +- **False start, corrected the same day**: initially "fixed" + `binding.py`'s `_get_binding_type` to treat G/A-variant package classes + (`Chdg`, `Drng`, `Evta`, `Ghbg`, `Rcha`, `Rivg`, `Welg`) as having + distinct namefile ftypes (`"CHDG6"`, not `"CHD6"`), based on legacy + flopy's `dfn_file_name`/`_package_type` attributes. This was wrong and + broke real `mf6` runs ("Model package type not supported [type=CHDG6]") + — confirmed against the actual MF6 Fortran source (`gwf.f90`'s + package-type `select case` has no `'CHDG6'`/`'RCHA6'`/etc. arm at all, + only the base tokens; `chd_create` handles both variants once + dispatched). Reverted to the original collapsing behavior. Real lesson: + a DFN's `dfn_file_name` describes the DFN/class identity, not + necessarily the namefile-level ftype token — verify against the actual + MF6 source dispatch table, not a generated client library's class + attributes. +- Because G/A variants genuinely share one namefile ftype, `Union[Chd, + Chdg]`-shaped fields can't be disambiguated by token alone — resolved by + peeking the referenced file's own `OPTIONS` block for the + `READASARRAYS`/`READARRAYGRID` marker keyword (the same signal real MF6 + itself uses) rather than the namefile row. + +Target: `component.py`, `converter/ingress/structure.py`, `binding.py`, +`flopy4/mf6/__init__.py`, `context.py`. + +1. ~~Add a component-type registry keyed by MF6 file-type token.~~ **Done** + as `FTYPES`/`get_ftypes()` in `component.py`, built lazily (not eagerly + in `__attrs_init_subclass__`, to dodge an import-reentrancy failure: + computing a token needs `binding.py`, whose own import chain can define + further concrete subclasses before finishing). Model-qualified keys + (e.g. `"gwf-dis6"`) disambiguate tokens that collide *across* model + types (every model has its own `Dis` via one shared abstract base); + G/A-variant collisions *within* one model are a different kind of + ambiguity, resolved in step 2 instead (a registry entry can only hold + one class per key). +2. ~~Write a `_resolve_binding` helper.~~ **Done** as `_resolve_bindings()` + in `structure.py`, using `xattree.get_xatspec(cls).children` to find + child-Component fields (mirroring `unstructure.py`'s + `_make_binding_blocks`, the egress side of this same job) rather than + hand-rolling type introspection from scratch. +3. ~~Route any `block=`-tagged binding-shaped field through + `_resolve_binding`.~~ **Done** — `structure_component()` gained a + `workspace` param and merges `_resolve_bindings()`'s result into its + kwargs before the single `cls(**kwargs)` call, same as every other + field. +4. ~~Dimension propagation.~~ **Done**, via a small load-order rule local + to `_resolve_bindings` (not a `dimensions.py` change): within one block, + `DimensionProvider`-typed rows (the `dis` field) are resolved first, the + resulting `dims` dict threaded into sibling `Package.load(..., + dims=dims)` calls. `dimensions.py`'s existing object-graph walk + (`resolve_dims()`) remains the right tool for *post*-attachment queries; + it just can't help *during* construction, before children exist to + walk. +5. ~~Simplify `Component.load()`/`Context.load()`.~~ **Done** — both drop + the manual child loop entirely (children now arrive already resolved + and attached from the constructor call). + +**pname preservation: fixed for `dict`-kind fields via `.name`, fixed for +`list`/`only`-kind fields via a new `Component.pname` field — landed +2026-08-22.** `Component.load()`/`Context.load()`/`Package.load()`/ +`structure_component()` all gained an optional `name=` param, threaded +from a binding row's pname through `_resolve_bindings`. This fully +round-trips for `Simulation.models`/`exchanges`/`solutions` (`dict`-kind +fields) — confirmed by test — since xattree reconciles a dict-kind +child's `.name` to the key it's attached under, so using pname as that +key (not the row's fname, previously used — fname has no round-trip +significance, but the dict key isn't purely cosmetic as assumed) makes it +stick. + +It does **not** help `list`-kind (`chd`, `wel`, ...) or `only`-kind +(`dis`, `ic`, `npf`, ...) package fields via `.name`: xattree reconciles +those to a field-derived name (`f"{field}{index}"` / the field name) +regardless of what's passed to the constructor — confirmed true even at +the original *write* time (`Chd(name="custom")` still becomes `"chd0"` +once attached), so changing that behavior itself is out of scope (would +mean changing xattree's own child-attachment convention, and the project +intends to eventually drop xattree entirely — not a thing to fold a +one-off feature's scope into). Resolved instead with a plain, +xattree-unmanaged `pname: str | None` field added to `Component` (same +category as the pre-existing `filename` field — an ordinary attrs +attribute xattree's `.name` reconciliation never touches). `_resolve_ +bindings` sets `child.pname = pname` after loading (alongside the +existing `child.filename = fname` line); `binding.py`'s +`Binding.from_component` emits `component.pname or component.name` for +the binding row's pname term instead of always `component.name`. This +closes the *load → write* round trip (a namefile with a hand-given CHD +pname like `"boundary_west"` now keeps that name through a load-then- +rewrite cycle, confirmed by `test_load_preserves_list_package_pname`) +without touching xattree at all — `.name` itself still reconciles to +`"chd0"` exactly as before, `pname` just carries the real name alongside +it. The one case this still doesn't cover, by design: a *freshly +hand-constructed* `Chd(name="custom")` that's never been round-tripped +through a load has no `pname` set either (defaults to `None`), so its +first-ever write still falls back to `.name` (`"chd0"`) — same +pre-existing xattree behavior, unchanged, still out of scope. A caller +who wants a custom name to survive from the very first write can now set +`pname=` explicitly at construction time as a workaround, though no +codegen/constructor convenience for that was added — out of scope for +this follow-up, small enough to revisit if it comes up in practice. + +`Solution.models`/`Exchange.exgmnamea`/`exgmnameb` are threaded through +from binding-row terms regardless (real semantic data, not display names +— see `_apply_binding_terms`), unaffected either way. + +Test coverage: `test/mf6/test_mf6_namefile_load.py` (new) covers recursive +`Simulation.load()`/`Gwf.load()`, dims propagation to griddata siblings, +list-package rows, and G/A-variant disambiguation, via a write-then-load +round trip built on the existing `docs/examples/quickstart.py` pattern. +`test/mf6/test_mf6_component.py` covers the `FTYPES` registry and the +G/A-variant ftype behavior directly. + +--- + +## Phase 2 — Wire it through Model/Simulation, prove it end-to-end + +Target: `context.py`, `component.py`, one real example model + +1. Decide the load entrypoint: give `Context` its own codec-based `load()` + (parallel to `Package.load`), or — since structuring is now fully + generic after Phase 1 — lift `Package.load()`'s body up to + `Component.load()` as a single implementation used by everything. +2. Run `Simulation.load(path/"mfsim.nam")` against an existing example + (`docs/examples/quickstart.py` or `twri.py`) and iterate until the full + tree loads: Tdis, each Model with its packages, Exchanges, Solutions. +3. Confirm DIS-before-NPF/IC ordering and dims propagation (Phase 1.4) + actually work end to end, not just in isolation. + +--- + +## Phase 3 — Namefile grammar refinement (parallelizable, not blocking) + +`basic.lark` already parses `packages`/`models`/`exchanges`/`solutiongroup` +blocks into generic `[type, fname, *names]` rows, which is sufficient input +for Phase 1-2. This phase is a validation/error-message improvement, not a +prerequisite, and can happen any time — including in parallel with Phases +1-2. + +Target: `codec/reader/dfn2lark.py`, `grammar/templates/component.lark.jinja`, +`grammar/templates/macros.jinja` + +1. Port PR #284's typed-record generation for `packages`/`models`/ + `exchanges`/`solutiongroup` blocks (generic `list` → typed + `packages_record: ftype fname pname NEWLINE` etc.). This is + generator/template-only and doesn't touch xattree or structuring, so it + should port over close to as-is. (Namefile-grammar "records" here are + an unrelated concept to `flopy4/mf6/record.py`'s `Record` class — same + word, different layer: Lark grammar rows vs. Python inline-record + objects. No overlap, just a naming coincidence worth not confusing.) +2. Switch the loader to `get_typed_parser()` for components with a + validated typed grammar; keep `get_basic_parser()` as fallback. + +--- + +## Phase 4 — Cleanup + +1. Close or narrow PR #284 to the salvageable pieces (the grammar template + diff from Phase 3, and the `FTYPES`/binding-resolution *design* — not + its `structure.py`, which is superseded by Phase 1). +2. Port `test_mf6_load_integration.py` / `test_mf6_load_all_models.py` from + PR #284 as acceptance tests for Phase 2, rewritten against the new + implementation. +3. Update `docs/dev/` (several files touched by PR #284: + [`on-representations.md`](on-representations.md), `sdd.md`, [`typed-grammar-issues.md`](typed-grammar-issues.md), etc.) to + describe the final single-convention structuring/binding design, since + PR #284's docs describe the old-xattree-spec version of this work. + +--- + +## Suggested order + +**Updated 2026-08-20.** ~~Phase 0~~ → ~~Phase 0.5~~ → ~~Phase 0.6~~ ⟷ +~~Phase 0.6a step 2 + Phase 0.6b (merged)~~ are **all done**. Everything +that was blocking namefile loading has landed: + +- Phase 0: one field-metadata convention (`3eda3f9`). +- Phase 0.5: `dfn_type` eliminated (`3eda3f9`). +- Phase 0.6: `Column`/`Schema` deleted, `Row` is the sole list-block schema + (`f1dd830`, `6cdfb2a`, `78c506b`). +- Phase 0.6a step 2: codegen migrated onto the pydantic-native dev3 DFN + schema (`6cdfb2a`). +- Phase 0.6b: DFN structure (OC rtypes, keystring/union detection) now read + from the schema instead of hardcoded tables (`6cdfb2a`). + +~~**Phase 1**~~ (generalize structuring to resolve namefile bindings) is +**also done** (2026-08-20) — `Simulation.load()`/`Gwf.load()` recursively +resolve real namefiles now, dims propagation included. See that section +for what landed and two bugs (one real, one a false-start that was found +and reverted the same day) it surfaced along the way. + +**Next up: Phase 2** (wire through Model/Simulation, prove end-to-end) — +Phase 1's own tests already exercise a full `Simulation.load()` round +trip on a small model, so Phase 2 is mostly "prove it against a larger/ +real-world example (`twri.py`?) and confirm DIS-before-NPF/IC ordering +holds beyond the two-package case," not new mechanism. + +**Still open, not blocking Phase 2, land any time / in parallel:** + +- Phase 0.6a step 3 (codec reader path off legacy `modflow_devtools.dfn`) + — not started. +- Phase 0.7 / the deferred union-item-class representation (real typed + union arms for LAK/LKE/LKT/SFR/STO period rows, fixing the `AUXILIARY` + data-loss bug) — unblocked now that Phase 0.6 landed, but no code written + yet; `composite_v3.py`'s prototype logic didn't survive between sessions + and would need re-deriving. +- Phase 3 (namefile grammar refinement) — was already flagged parallelizable. + +~~Phase 1's known limitation (binding-row `pname` not threaded to a loaded +child's `.name`)~~ is **done** (2026-08-22) — see the "pname preservation" +note in Phase 1 above for what landed (a new `Component.pname` field) and +the one narrow case that's still, by design, unchanged (a hand-constructed +component's very first write, never round-tripped through a load). + +**Also done in this same pass (2026-08-22):** bumped the `modflow-devtools` +pin (`29f7e0f` → `a9d66f5`, "multiple fixes/improvements" (#349) + +"fix olf idcxs fk backfill, trim migration comments" (#350)), which +splits the old overloaded `pk`/`fk` semantics into `index` (needs the +1-based/0-based conversion), `node` (grid-cell reference), and `pk`/`fk` +(pure relational identity). The bump surfaced a real upstream devtools +bug on the way — `_FK_BACKFILL`/`_ARRAY_FK_BACKFILL` in +`migrate_to_v2_0_0_dev2.py` pointed OLF's `idcxs` fk at a nonexistent +`olf-cxs` component (removed upstream, `c92f3e51`: "Overland flow has no +concept of cross section" — but OLF's Fortran source still shares CHF's +`SwfCxsType` cross-section object, so the real fk target is `chf-cxs`) +and still listed dead `swf-*` entries (SWF was split into CHF/OLF and +removed, `802d8e1d`) — fixed and landed upstream on +`MODFLOW-ORG/modflow-devtools` develop as #350 (owner: wpbonelli), pin +now points at that. + +Followed through on the semantic split itself: `flopy4/mf6/spec.py`'s +`field()` gained an `index=` kwarg (alongside `pk=`/`fk=`); codegen +(`filters.py`'s `list_columns()`/`row_class()`, `make.py`'s +`_schema_dict_from_columns()`/`_is_index()`) now reads dev3's real +`Integer.index` attribute directly instead of inferring "needs the +numeric shift" from `isinstance(f, Integer) and (pk or fk)`; `row.py`'s +`to_row()`/`from_row()` now check `metadata.get("index")` instead of +`metadata.get("pk") or metadata.get("fk")`. Regenerated all 63 component +modules against a live modflow6 develop checkout (`flopy4 mf6 sync + --no-install`) to pick up the new metadata — this also +surfaced one real, previously-silent bug: MVR's `id1`/`id2` period +columns (DFN-documented `numeric_index: true`, "the first well has an +identifier of one") had never carried `pk`/`fk` upstream at all before +`41dca93`, so flopy4 never applied the 0-based/1-based conversion for +them — callers had to pass an already-1-based value, unlike every other +index/fk column in the codebase. Now fixed; `test_gwf_mvr` updated to +pass a 0-based `id1`/`id2` like everywhere else. Full `test/` suite +(638 tests) green except the one pre-existing, unrelated +`test_quickstart_grid` failure. + +After Phase 2: Phase 4 (cleanup, close out PR #284, update `docs/dev/`). + +**Pre-existing, unrelated to Phase 0/1 work — noticed along the way, not +investigated:** the full suite has 50 pre-existing failures on `HEAD` +(confirmed via `git stash`, i.e. present before *and* after this session's +changes) — 47 in `test_mf6_row_api.py` (`TypeError`/`AttributeError` +patterns suggesting a `Row`-API round-trip regression from the Phase 0.6 +migration itself, not yet triaged) plus `test_quickstart_grid`, +`test_gwe_lke_flow_package_auxiliary_name`, and +`test_gwt_lkt_flow_package_auxiliary_name` in `test_mf6_integration.py` +(real-`mf6`-execution failures, the last two plausibly related to the +already-known LAK/LKE/LKT keystring data-loss bug Phase 0.7 is meant to +fix). Worth a dedicated triage pass — `78c506b`'s commit message claimed +"full suite is green except two ... failures," which no longer matches. + +--- + +## Status update (2026-09-16) + +Everything above predates six commits landed since (#350-#356); +[`plan.md`](plan.md) is the up-to-date source of truth going forward, but briefly, +since this doc's ending was written mid-flight ("Next up: Phase 2"): + +- **Phase 2 (prove end-to-end) is done** — `743ecf1`/#350, far exceeding + this doc's own "twri.py?" framing: a 242-model corpus survey against + the real `modflow_devtools.models` registry now passes 239/242 + (98.8%), documented in [`load-corpus-gaps.md`](load-corpus-gaps.md). DIS-before-NPF/IC + ordering and dims propagation are exercised across that whole corpus. +- **Phase 4 (close out PR #284) is done** — the PR was closed (not + merged) 2026-09-10; its useful docs/tests were salvaged into #350. +- **Phase 0.6a step 3 (codec reader off legacy schema), Phase 0.7's + still-open OC nested-`ocsetting` case (see + [`nested-keystring-union-plan.md`](nested-keystring-union-plan.md)), and Phase 3 (typed namefile + grammar) remain exactly as open as this doc left them** — verified + directly against current `develop`, zero progress on any of the three + since this doc was last edited. +- The pre-existing failures noted just above are stale. Verified + directly: `test_mf6_row_api.py` no longer exists at all (superseded by + the `Item`/`Record` refactor [`plan.md`](plan.md)'s status table credits to + #348/#349, not "fixed" so much as replaced); the three named + `test_mf6_integration.py` cases (`test_quickstart_grid`, + `test_gwe_lke_flow_package_auxiliary_name`, + `test_gwt_lkt_flow_package_auxiliary_name`) all pass now, with no + dedicated triage note anywhere crediting a specific fix — presumably + incidental to one of the intervening commits. diff --git a/docs/dev/plans/archive/model-codegen-plan.md b/docs/dev/plans/archive/model-codegen-plan.md new file mode 100644 index 00000000..1eaf9ba7 --- /dev/null +++ b/docs/dev/plans/archive/model-codegen-plan.md @@ -0,0 +1,111 @@ +# Generate the model classes + +Status: proposed (2026-10-06), against develop `82d2a99`. + +Implemented (2026-10-06) on branch `model-codegen` (worktree +`../pyphoenix-models`), commit `29aeffa` off develop `7e4fdc9`, not pushed. +Differences from the plan: +- `Gwt` already had `dependent_variable_scaling`. +- Breaking changes beyond `_list`: + - `Gwf.newton` is gone in favor of the DFN's `newtonoptions` record. + - `Gwf.csub` is a single package; the DFN doesn't mark it `multi`. + - `Gwf.api` is new. +- Package fields follow DFN (alphabetical) order, with `dis` first, and + name-file rows are written in that order. The grid union is + `Dis | Disu | Disv`. +- The netcdf options keep their names (`netcdf_mesh2d_file`, …) through + a `_PY_NAMES` table. The DFN names would have given `nc_*_file`. +- `Gwf.output` is a property, so it's no longer an attrs field. +- Binding code (`_resolve_bindings`, `_make_binding_blocks`) now looks + only at `child()` fields. Before, a record option on a model counted + as a child. +- The unmatched-row warning fires only in blocks holding nothing but + child fields. It doesn't check DFN ftypes. +- Model registration is unchanged. + +## Problem + +`Gwf`, `Gwt`, `Gwe` and `Prt` (`flopy4/mf6/{gwf,gwt,gwe,prt}/__init__.py`) +are hand-written and have drifted from the DFNs: + +- `Gwt` has no `fmi` or `ist` field, and `Gwe` has no `fmi`. Loading drops + their name-file rows without a warning. In `test205_gwtbuy-henrytidal`, + FMI's `FLOW_IMBALANCE_CORRECTION` is lost. The round-trip test still + passes because neither side has FMI. +- `Gwt.api` is `Api | None`, but the DFN says API can repeat. +- `Gwf` names its list option `_list`; the other models use `list_`. +- `Fmi` and `Ist` aren't exported from `flopy4.mf6.gwt` or `flopy4.mf6.gwe`. + +The simulation (`sim-nam`) is already generated, with its behavior in the +`SimulationMethods` mixin. The models should be generated the same way. + +## What the DFNs give + +- **Options:** the `*-nam` DFN's options block. +- **Packages:** every component whose `parent` is the model's name file. + gwt-nam has 19 children and gwe-nam has 17. +- **Repeating packages:** the `multi` flag. For GWT these are api, cnc, ist, + lkt, mwt, sft, src and uzt. + +## Rules + +1. **Grid packages share a slot.** DIS, DISV and DISU become one field, + `dis: Dis | Disv | Disu | None`. PRT has no DISU, so it gets + `Dis | Disv | None`. +2. **Packages sharing a name-file type share a field.** Variants such as + `chd`/`chdg` and `rch`/`rcha` become one field typed as their union, for + example `chd: list[Chd | Chdg]`. This is what `_disambiguate_ga_variant` + already handles at load time. +3. **One slot or a list.** A `multi` package becomes a list defaulting to + empty; any other becomes `X | None`. +4. **Packages with no class are skipped.** That's GWT's MWT/SFT/UZT and + GWE's MWE/SFE/UZE for now, using the same test as `_has_class`. They get + fields once they're generated. +5. **Order and naming:** fields follow the DFN order: options, then + packages. Each package field is named for its ftype (`npf`, `chd`), and + the list option is `list_`, which is a breaking change for + `Gwf(_list=...)`. + +## Behavior that stays hand-written + +These move into method-only mixins, as `DisMethods` did in #385: + +- `ModelMethods`, for all four models: the `grid` property, which all four + copies of today implement the same way. +- `GwfMethods`: `Output` (`head` and `budget` readers) and the `output` + attribute. + - `output` is an attrs field with a `takes_self` factory, so it may need + to become a cached property, since mixins declare no fields. + +`Gwf.NewtonOptions` should come from the DFN's `newtonoptions` record; +check that it does. + +## Steps + +1. **Codegen** (`utils/codegen/make.py`): build a model's package fields + from its DFN children using the rules above. Add a model template, or + extend the component template, and the `MIXINS` entries for the four + `*-nam` DFNs. +2. **Mixins:** add `model_methods.py` and `gwf_methods.py`. +3. **Regenerate:** write the four `__init__.py` model classes, keeping each + module's package imports and `__all__`, and add the missing exports. +4. **Diff `Gwf`:** compare it to today's version field by field. Only + `_list` should change. +5. **Warn on unmatched rows:** a name-file row that no field accepts warns, + as #380 proposed (`unsupported component`), checked against the DFN + registry's ftypes rather than `endswith("6")`. This catches packages + that rule 4 skips. +6. **Tests:** + - test205 loads with `gwt.fmi.flow_imbalance_correction`. + - A GWT FMI with PACKAGEDATA round-trips through the name file. + - Two IST packages round-trip. + - Two API packages are accepted. + - An unmatched row warns. + - Run the smoke tests and the docs examples. + +## Open + +- Whether to generate `Model` subclass registration (ftype lookup through + `get_ftype`) or keep it as is. Check what `Simulation` generation does. +- Name collisions between an option and a package. None exist today, but + codegen should raise if one appears. diff --git a/docs/dev/plans/archive/model-rebuild-explained.md b/docs/dev/plans/archive/model-rebuild-explained.md new file mode 100644 index 00000000..d1ee373f --- /dev/null +++ b/docs/dev/plans/archive/model-rebuild-explained.md @@ -0,0 +1,217 @@ +# Understanding `model_rebuild()` and Pydantic Naming + +## What is `model_rebuild()`? + +`model_rebuild()` is pydantic's mechanism for **resolving forward references** in type hints. It's needed when you have circular dependencies between classes. + +### The Problem: Circular Dependencies + +```python +class Package(BaseModel): + parent: Optional['MF6Model'] = None # ← Forward reference (string) + +class MF6Model(BaseModel): + npf: Optional['Npf'] = None # ← Forward reference + +class Npf(Package): + pass # Inherits parent: Optional['MF6Model'] +``` + +When Python parses `class Package`, `MF6Model` doesn't exist yet, so we use a **string literal** `'MF6Model'`. Pydantic needs to resolve these strings to actual classes. + +### The Solution: `model_rebuild()` + +```python +# After all classes are defined: +MF6Model.model_rebuild() +Package.model_rebuild() +Npf.model_rebuild() + +# Now pydantic knows: +# - Package.parent is MF6Model (not the string 'MF6Model') +# - MF6Model.npf is Npf +# - Type checking and validation work correctly +``` + +### When Do You Need It? + +**Only when you have circular type references:** + +```python +# ✓ No rebuild needed (no circular refs) +class Dis(BaseModel): + nlay: int + +# ✓ No rebuild needed (parent defined first) +class Package(BaseModel): + pass + +class Npf(Package): + k: float + +# ✗ Rebuild needed (circular: Package → Model → Package) +class Package(BaseModel): + parent: Optional['Model'] = None + +class Model(BaseModel): + packages: dict[str, Package] + +Model.model_rebuild() # ← Required +``` + +### Alternative: `from __future__ import annotations` + +With PEP 563 (postponed annotation evaluation), you can avoid manual rebuilding: + +```python +from __future__ import annotations # ← At top of file + +class Package(BaseModel): + parent: Optional[MF6Model] = None # ← No quotes needed! + +class MF6Model(BaseModel): + npf: Optional[Npf] = None + +# Automatically handled - no rebuild needed +``` + +**This is the cleaner approach** and what I used in the prototype. + +--- + +## The "Model" Naming Problem + +You're right - pydantic uses "model" everywhere, but in MODFLOW: +- **Model** = GWF, GWT, GWE (a simulation component) +- **pydantic "model"** = any class inheriting from BaseModel + +This creates confusion: `GwfModel.model_fields` - which "model"? + +### Solution 1: Alias the Base Class ✅ (Recommended) + +```python +from pydantic import BaseModel + +# Use a different name for the base class +MF6Base = BaseModel + +class GwfModel(MF6Base): # ← No "Model" in base class name + pass + +# Now "Model" only means MODFLOW model +``` + +### Solution 2: Alias the Methods ✅ (Also Good) + +```python +class MF6Base(BaseModel): + """Base class with MF6-friendly method names.""" + + @classmethod + def get_fields(cls): + """Alias for model_fields.""" + return cls.model_fields + + @classmethod + def get_schema(cls): + """Alias for model_json_schema.""" + return cls.model_json_schema() + + def to_dict(self): + """Alias for model_dump.""" + return self.model_dump() + + @classmethod + def from_dict(cls, data): + """Alias for model_validate.""" + return cls.model_validate(data) + +# Usage: +class GwfModel(MF6Base): + pass + +schema = GwfModel.get_schema() # ← Clear! +data = model.to_dict() # ← Clear! +``` + +This is what I did in the prototype (lines 38-50). + +### Solution 3: Accept the Collision + +```python +class GwfModel(BaseModel): + """ + MODFLOW 6 groundwater flow model. + + Note: This is an MF6 model (simulation component). + BaseModel is from pydantic (data validation framework). + """ + pass + +# Usage context makes it clear: +gwf = GwfModel(...) # ← Clearly MODFLOW +gwf.model_fields # ← Clearly pydantic API +``` + +**My recommendation**: Use **Solution 2** (alias methods). It gives you: +- Clear, MF6-specific API +- No confusion about what "model" means +- Still access to pydantic's raw API if needed +- Easy to document and teach + +--- + +## Complete Example with Best Practices + +```python +from __future__ import annotations # ← Avoids model_rebuild() + +from typing import Optional +from pydantic import BaseModel, Field + +# Alias base class and methods +class Component(BaseModel): + """Base for all MF6 components (packages, models, simulation).""" + + @classmethod + def get_fields(cls): + return cls.model_fields + + @classmethod + def get_schema(cls): + return cls.model_json_schema() + + def to_dict(self, **kwargs): + return self.model_dump(**kwargs) + + @classmethod + def from_dict(cls, data, **kwargs): + return cls.model_validate(data, **kwargs) + +# No "model" in sight! +class Package(Component): + parent: Optional[MF6Model] = Field(default=None, exclude=True) + +class MF6Model(Component): + """This is an MF6 model (GWF, GWT, etc.).""" + name: str + packages: dict[str, Package] = Field(default_factory=dict) + +class GwfModel(MF6Model): + """Groundwater flow model.""" + pass + +# Clean API, no confusion: +gwf = GwfModel.from_dict(data) # ← Not "model_validate" +schema = GwfModel.get_schema() # ← Not "model_json_schema" +fields = GwfModel.get_fields() # ← Not "model_fields" +d = gwf.to_dict() # ← Not "model_dump" +``` + +This approach: +- ✅ No `model_rebuild()` needed (using `from __future__ import annotations`) +- ✅ No naming confusion (aliased methods) +- ✅ Clean, MF6-specific API +- ✅ Still pydantic under the hood (can use raw API if needed) + +Would you like me to update the prototype to use this cleaner pattern throughout? diff --git a/docs/dev/plans/archive/model-relpath-plan.md b/docs/dev/plans/archive/model-relpath-plan.md new file mode 100644 index 00000000..2a73ff88 --- /dev/null +++ b/docs/dev/plans/archive/model-relpath-plan.md @@ -0,0 +1,174 @@ +# Model directories + +Status: implemented (2026-10-07) on branch `model-relpath`, on top of +`load-context` `54b8383`. Decisions taken where the draft left options: + +- Egress prefixes the token after every `FILEIN`/`FILEOUT` in a model's + unstructured blocks (rows and block headers), plus namefile rows via + `Binding`. All 94 path fields but the exchanges' `exgfile` have a + direction, and exchanges are simulation-level, so this covers them. +- Ingress walks a component's parsed kwargs (records, items, blocks; + not child components) and makes every `Path` model-relative. +- `Model.workspace` is a property over the inherited attrs field; + `update_child_attr` and `Context.write`'s `cd` are gone, and + `Component.path` uses the enclosing context's workspace. +- A loaded model's directory is its namefile's. `relpath` may leave the + simulation directory (`../x`); paths are normalized, not rejected. +- NetCDF: nothing is written implicitly (callers pass `to_netcdf` a + path), and `NC6 FILEIN` paths go through the same prefixing. + +## Problem + +A model is a `Context` and has a `workspace`, but under a simulation that +workspace is only a copy of the simulation's. `Context.workspace` has +`on_setattr=update_child_attr`, so setting the simulation's workspace +overwrites every model's, and `__attrs_post_init__` takes the parent's +when none is given. A model's workspace means something of its own only +when the model stands alone. The separate `Context` hardly earns its +keep. + +MF6 lets models live in subdirectories, and users think of them that way +("the GWF model is in `gwf/`"). Today flopy4 holds such a model with +filenames that carry the directory: `gwf.filename == "gwf/m.nam"`, +`gwf.dis.filename == "gwf/m.dis"`. That round-trips (the `load-context` +branch makes it load), but: + +- moving a model to another directory means rewriting every one of its + packages' filenames, and its output files'; +- a model built standalone and then attached to a simulation can't keep + its own directory; +- a user who sets `head_file="m.hds"` on a model "in `gwf/`" gets the + file in the simulation directory, because MF6 resolves paths there. + +## Design + +A model stores where its directory is relative to the simulation's, and +its files' paths are relative to its own directory. The translation +happens only at the file boundary, on load and on write. MF6's rule, +that every path in every file is relative to the simulation directory, +stays the one the files follow. `LoadContext.workspace` stays the +simulation directory. + +```python +@attrs.define(kw_only=True, slots=False) +class Model(Context, ABC): + relpath: Path = field(default=Path("."), converter=Path) + """The model's directory, relative to the simulation's.""" + + @property + def workspace(self) -> Path: # parent's workspace / relpath, or its own + ... +``` + +- **Workspace is derived, not copied.** A model's `workspace` is + `parent.workspace / relpath` under a simulation, and its own (settable, + as now) when standalone. `update_child_attr` stops copying the + simulation's workspace onto its models, so the two can't disagree. + Setting a model's `workspace` under a simulation sets `relpath` to its + path relative to the simulation's. +- **Model filenames are relative to the model directory.** `dis.filename + == "m.dis"`, `oc.head_file == "m.hds"`, `wel.obs.filename == "m.obs"`. + The model's own `filename` is relative to its directory too + (`"m.nam"`), and the simulation's `GWF6` row gets `relpath / filename`. +- **The simulation's own files** (TDIS, IMS, exchanges) keep paths + relative to the simulation directory. An exchange's paths aren't in + any one model's directory. + +### Load + +- `_resolve_bindings` on `Simulation`: for a `GWF6 gwf/m.nam m` row, + `relpath = Path("gwf/m.nam").parent`, and the model's `filename` is + `m.nam`. +- Inside the model's files, every path read (namefile rows, file + records, `FILEIN`/`FILEOUT` paths, row children) is stored relative to + the model's directory: `os.path.relpath(path, relpath)`, with POSIX + separators. A path outside the model's directory becomes `../x`. + Reading still resolves files with `context.resolve` against the + simulation directory. Only the stored path changes. +- `LoadContext` gains `relpath: Path = Path(".")` (the directory of the + model being loaded, relative to the workspace), set in the model's + child context and inherited by its packages and their children, with a + `relative(path) -> Path` method for the stored form. +- OPEN/CLOSE paths aren't stored (the arrays are read in), so they need + no change. +- A standalone `Gwf.load(path, context=LoadContext(workspace=sim))` gets + `relpath = path.parent` relative to the workspace, as in a simulation. + Without a context, `relpath = "."` and the workspace is the namefile's + directory, as now. + +### Write + +Every path a model's files write gets `relpath /` prefixed: + +- namefile rows (`binding.py`, `Binding.from_component`'s `fname`), both + the model's `PACKAGES` rows and the simulation's `MODELS` row; +- file records naming children (`unstructure.py`, `_child_path`) and + plain path fields (`_path_to_tuple`, the `FILEIN`/`FILEOUT` branches + around `unstructure.py` L276–280); +- external array files, if the writer produces any (`macros.jinja`'s + `external` branch writes `OPEN/CLOSE `; check where that value + comes from and whether it's a path flopy4 wrote); +- the files themselves: `Component.path` for a package is + `Path.cwd() / filename`, relying on `Context.write`'s `cd`. Under this + plan it should be `model.workspace / filename`, without the `cd`. + +The unstructure hooks don't know the component's model, so the prefix +has to reach them. Two ways: + +1. Each package finds its model (walk `_parent` to the nearest `Model`) + and prefixes its own paths, through one helper + (`_file_path(component, path) -> Path`) that every path site calls. +2. A write-side counterpart of `LoadContext` carries `relpath` down the + write, set by `Model.write`. `WriteContext` is thread-local and holds + user settings, so that isn't the place for it (same reasoning as + [`load-context-plan.md`](load-context-plan.md)). + +Prefer 1 unless the write path is reworked to pass an explicit context +anyway. + +### Round trip + +Loading a simulation and writing it back keeps every path in every file, +byte for byte, as long as the paths are inside the model's directory. +For a path outside it, the file's path and the written path name the +same file but may be spelled differently (`gwf/../shared/x` → +`shared/x`; normalize). + +## Open questions + +- Which directory a loaded model gets when its files are spread out + (namefile in `gwf/`, packages in `shared/`). Proposal: the namefile's + directory, always. Packages outside it get `../` paths. +- Whether `relpath` can leave the simulation directory (`../other`). + MF6 allows it. Proposal: allow, normalize, don't special-case. +- `NetCDFModel` and NetCDF export write their own files; check how they + name them. +- A user who sets `head_file="gwf/m.hds"` on a model with + `relpath="gwf"` today's way gets `gwf/gwf/m.hds`. That's a behavior + change for code written against the current flat convention; say so in + the changelog. + +## Steps + +1. `Model.relpath`, derived `workspace`, `update_child_attr` no longer + overwriting models. Unit tests: moving the simulation moves its + models; setting a model's workspace sets `relpath`. +2. Write side: the path helper, used by namefile rows, file records and + path fields; `Component.path` from the model workspace. Test: a model + with `relpath="gwf"` and plain filenames writes `gwf/m.nam`, + `gwf/m.dis`, `DIS6 gwf/m.dis` in its namefile, and `HEAD FILEOUT + gwf/m.hds` in OC. +3. Load side: `LoadContext.relpath` and `relative()`, set for a model's + children. Tests: the `load-context` subdirectory fixture loads with + `relpath == "gwf"` and `dis.filename == "m.dis"`; write-back keeps the + files byte for byte. +4. Run an MF6 simulation with a model in a subdirectory end to end + (write, run, check the head file lands in `gwf/`). +5. Touched test files, ruff, mypy, smoke. + +## Non-goals + +- Changing how MF6 paths resolve on read. `LoadContext.workspace` stays + the simulation directory. +- Simulation-level files (TDIS, IMS, exchanges) in subdirectories. Their + paths stay as written. diff --git a/docs/dev/plans/archive/mypy-converter-plan.md b/docs/dev/plans/archive/mypy-converter-plan.md new file mode 100644 index 00000000..5bb32dde --- /dev/null +++ b/docs/dev/plans/archive/mypy-converter-plan.md @@ -0,0 +1,99 @@ +# Let mypy see field converters + +Status: abandoned. The work was done on branch `mypy-converters` (2026-10-08) +but never merged, and #433 is closed. Record fields used a lambda converter +(`attrs.Converter` isn't readable by mypy), so they accepted any value +statically. By 2026-11-04 the plugin was a no-op on develop, because the field +functions (#463) replaced the `field()` and `path()` it hooks. Installed +packages also got nothing from it, because flopy4 has no `py.typed`. See +[`constructor-stubs.md`](../components/constructor-stubs.md) for the plan that +replaces it. + +## Problem + +Generated fields are declared with flopy4's `field()`/`path()` wrappers +(`flopy4/mf6/spec.py`). mypy's attrs plugin recognizes only `attrs.field`, +`attr.ib` and similar, so it ignores the wrappers' `converter=`. Every +`__init__` argument therefore takes only the annotated type, even where +the converter accepts more: + +- `Oc(budget_file="x.cbc")` is rejected (`Path | None` expected), though + `attrs.converters.optional(Path)` accepts it. +- `Gwf(newtonoptions=True)` is rejected, though `Record.from_flag` accepts + it. + +Widening the annotations would make reads like +`gwf.newtonoptions.under_relaxation` type errors, so that's out. + +## Approach: an opt-in mypy plugin + +Ship `flopy4/mypy_plugin.py`. It adds the wrappers to the attrs plugin's +field-function set: + +```python +import mypy.plugins.attrs as attrs_plugin +from mypy.plugin import Plugin + +attrs_plugin.attr_attrib_makers.update({"flopy4.mf6.spec.field", "flopy4.mf6.spec.path"}) + + +def plugin(version): + return Plugin +``` + +Tried on mypy 2.3.1 in a scratch config: + +- `Gwf(newtonoptions=True)`, `Npf(xt3doptions=True)` and + `Oc(budget_file="x.cbc")` pass. +- Attribute types are unchanged. + +## Steps + +1. **Add the plugin**, and enable it in `[tool.mypy]` + (`plugins = ["flopy4.mypy_plugin"]`). +2. **Make converters readable.** mypy only reads a converter that's a + named function, a type or a lambda. The plugin reports 32 that aren't: + - 27 are `attrs.converters.optional(to_array(np.str_))`. Generate a + named module-level converter in `_types.py` (e.g. + `to_optional_str_array`) and use it from `filters.field_converter`. + - 5 are `X.from_flag` on nested record classes. Options: + - a module-level `record_flag(value: Record | bool | None)` typed + for mypy, which looks the record class up from the field at + conversion time (an `attrs.Converter` with `takes_field=True`); + - a lambda, `lambda v: Gwf.Newtonoptions.from_flag(v)`; + - a generated `# type: ignore[misc]`. + + Prefer the first if `attrs.Converter` gives mypy a readable + argument type; check it. + - Then `mypy flopy4` is clean with the plugin on. +3. **Leave `child()` out.** Its default lives in the wrapper's signature, + and the plugin only reads arguments written in the call, so every + child would look required. Children don't need it. +4. **Tests:** a mypy test (`mypy.api.run` on a snippet, marked slow if + needed). It checks that: + - the bool, str and record forms are accepted; + - a wrong type is still rejected; + - reading the attribute keeps the record type. +5. **Docs:** a line in the user docs on adding the plugin to a mypy + config. + +## Caveats + +- `attr_attrib_makers` is a mypy internal (`Final`), not a public hook. A + mypy release could break it. Pin a tested range, or have the test catch + a break. +- Users only benefit if they enable the plugin. Without it, they get + today's behavior. +- Pyright/Pylance gets nothing. It has no plugins and doesn't read + converters. + +## Alternative, not planned + +Codegen could emit a typed `__init__` under `if TYPE_CHECKING:` in every +class. It has to list every constructor argument, including the hand-written +bases' (`name`, `workspace`, `parent`, `dims`, …), and keep them in sync. + +Correction (2026-11-04): this said the approach works for mypy and Pyright, +but it only works for Pyright. mypy's attrs plugin ignores that `__init__` and +uses the generated one. Generated `.pyi` stubs work for both: see +[`constructor-stubs.md`](../components/constructor-stubs.md). diff --git a/docs/dev/plans/archive/nested-keystring-union-plan.md b/docs/dev/plans/archive/nested-keystring-union-plan.md new file mode 100644 index 00000000..b6110748 --- /dev/null +++ b/docs/dev/plans/archive/nested-keystring-union-plan.md @@ -0,0 +1,74 @@ +# Recursively type nested keystring unions (OC's `ocsetting`) — shipped + +**Status: done.** OC's `Save`/`Print.ocsetting` is now a typed +`Oc.All | Oc.First | Oc.Last | Oc.Frequency | Oc.Steps` union (matching +PRP's `releasesetting` pattern), instead of an untyped +`tuple = field(default=(), array=True)`. Full suite green (910 passed) +including two new dedicated tests and a strengthened codegen test; the +existing `docs/examples/quickstart.py` end-to-end example (write → real +`mf6` run → read back) exercises the new serialization path too. + +Verified against current `develop` before implementing — two things in +the original design below turned out to need correction, not just the +`component_name` signature drift: + +- `ComponentSpec.period_arms` is one flat list that jinja unconditionally + folded into `_StressPeriodDataItem`. Nested arms (`All`/`First`/...) + needed to be excluded from that fold while still being emitted as flat + sibling classes — added `PeriodArmSpec.top_level: bool`. +- OC's `saverecord.ocsetting` and `printrecord.ocsetting` are two + independent `Union` objects with identical arms (DFN parsing doesn't + share field instances across sibling arms), not one shared union — + building them naively per top-level arm would have produced two + duplicate, differently-named class sets. Added a `nested_union_cache` + (keyed by the nested union's own arm-name set) local to one + `_build_period_arm_specs` call. + +Also, during review, "does MF6/the DFN schema actually forbid nesting +beyond 2 levels" was checked empirically rather than assumed (it doesn't +— `Union.arms` may hold a `Record`, whose fields may include another +`Union`; only `List` has an explicit no-recursion rule) — so the arm +builder (`_build_arm_specs_from_union` in `make.py`) was written as +genuinely self-recursive, not hardcoded to exactly one extra level. + +**Where it landed:** +- `flopy4/mf6/utils/codegen/make.py`: `PeriodArmSpec.top_level`, + `_build_arm_specs_from_union` (recursive, replaces the old + `_build_period_arm_specs` loop body), `_schema_dict_from_columns`'s + `nested_arm_classes` param → `role="nested_union"`. +- `flopy4/mf6/utils/codegen/filters.py`: `item_class(..., + package_class_name=...)`, `_field_line`'s `nested_union` branch. +- `flopy4/mf6/utils/codegen/templates/package.py.jinja`: threads + `package_class_name`; `_StressPeriodDataItem` fold filters to + `arm.top_level`. +- `flopy4/mf6/item.py`: `_nested_union_classes` (mirrors `record.py`'s + `_nested_class`, generalized to a tuple of sibling classes); + `Item.to_tokens`/`from_tokens`/`construct_item` each gained a + nested-union branch alongside the existing `array`/`tagged`/cellid + branches. +- `flopy4/mf6/record.py`: `_nested_class`'s sibling-resolution primitive + factored out as `_resolve_sibling_class`, shared with `item.py`. +- Regenerated `gwf/gwt/gwe/prt/oc.py` (`--existing-only` sync); `prt/prp.py` + unchanged, as expected (already top-level, untouched by the recursive + path). +- New tests: `test_oc_ocsetting_typed_dispatch`, + `test_oc_ocsetting_construct_item_positional` + (`test/mf6/test_mf6_codec.py`); `test_oc_tier_generates_importable_files` + strengthened to assert no arm duplication and correct `top_level` split + (`test/mf6/test_mf6_codegen.py`). + +**Known, disclosed behavior change:** `ocsetting` moved from a silently +optional/defaultable tuple to a required field — `Save(rtype=...)` without +`ocsetting` now fails at construction. Matches its true DFN status (never +optional). One downstream break found and fixed: +`docs/examples/quickstart.py` compared `ocsetting` to a raw tuple +(`== ("ALL",)`); updated to `isinstance(..., Oc.All)`. + +**Deliberately out of scope, noted during review, not fixed here:** the +codegen layer has two different representations for "one field's role/ +type/optionality" — a typed `FieldSpec` dataclass for top-level package +fields, and an untyped list-of-dicts (`role=...`) for per-row/arm fields, +which this change necessarily extended rather than reduced. Unifying them +would be a real, separate cleanup with a much larger blast radius (every +column-based renderer in `make.py`/`filters.py`) — worth a future design +note, not part of this A3 item. diff --git a/docs/dev/plans/archive/netcdf-spec-plan.md b/docs/dev/plans/archive/netcdf-spec-plan.md new file mode 100644 index 00000000..6aa1aa94 --- /dev/null +++ b/docs/dev/plans/archive/netcdf-spec-plan.md @@ -0,0 +1,169 @@ +# NetCDF spec: what belongs in devtools, what stays in flopy4 + +## Background + +flopy4 currently has two independent object models: + +1. **The MF6 input object model** — `Package`/`Component` and friends + (`flopy4/mf6/{component,package,item,record,row}.py`), attrs-based, + generated from the DFN corpus via `flopy4/mf6/utils/codegen/{make,filters}.py`. + This is what users construct/populate to build a simulation. +2. **The MF6 NetCDF I/O object model** — `NetCDFModel`/`NetCDFPackage`/ + `NetCDFParam` and friends (`flopy4/mf6/netcdf.py`, ~950 lines), pydantic + (`BaseModel`)-based, hand-written directly in flopy4. This describes the + structure of an MF6-conformant NetCDF file (dims, CF attribute + conventions, per-package/per-param layout) and converts a live + `Model`/`Package` into one. + +This plan is about #2. It came out of a discussion (see issue #282, +"Consider switching attrs -> pydantic") about whether flopy4's *object +model* needs JSON Schema, and whether that's actually the right layer for +it. Conclusion there: JSON Schema earns its keep on artifacts meant for +external/cross-tool interop (a file format, an API payload) — not +necessarily on an in-memory object model whose job is ergonomic +construction. The DFN spec is exactly the former, and devtools already +treats it that way. NetCDF should follow the same split, but doesn't yet. + +## The existing precedent: DFN spec in devtools + +- The DFN corpus (what packages/blocks/fields exist, their types, dims, + `pk`/`fk` relationships) lives in `modflow_devtools.dfns`, pydantic-native + as of schema `2.0.0.dev3`. +- flopy4 consumes it at codegen time (`flopy4/mf6/utils/codegen/{make,filters}.py`) + to generate the attrs-based `Package` subclasses under `flopy4/mf6/{gwf,gwt,gwe,prt,utl,exg}/`. + The generated classes don't carry pydantic anywhere themselves — they're + attrs, built from pydantic-described metadata. +- This migration ([`mf6-object-model-plan.md`](mf6-object-model-plan.md)'s Phase 0.6a) landed in + `6cdfb2a` (2026-08-19). It already gets JSON Schema generation for the + DFN spec for free (`model_json_schema()` on the devtools side) — this is + what the issue's "free JSON schema" motivation was actually about, + independent of whatever flopy4's own `Package`/`Component` classes are + built on. +- Net effect: **spec lives in devtools (shared, versioned, machine-readable, + schema-native); object model lives in flopy4 (whatever's ergonomic for + construction)**. Two different concerns, two different owners. + +## Current state of the NetCDF side + +`flopy4/mf6/netcdf.py` does *not* follow this split — it's simultaneously +the spec and the consumer, and it's the *only* machine-readable encoding of +the MF6 NetCDF format that exists anywhere: + +- The actual spec is prose: the [MODFLOW 6 NetCDF format wiki + page](https://github.com/MODFLOW-ORG/modflow6/wiki/MODFLOW-NetCDF-Format), + owned by the modflow6 core team, referenced from `docs/dev/sdd.md` + ("NetCDF" section). It is not structured or versioned anywhere. +- flopy4 hand-encodes that format as pydantic classes directly in + `netcdf.py`: `NetCDFModel`, `NetCDFPackage`, `NetCDFParam`, + `NetCDFModelAttrs`, `NetCDFParamAttrs`, `NetCDFParamEncodings`, all + implementing an `NetCDFInput` ABC that requires `.jsonschema()` (→ + `self.model_json_schema()`) as a first-class method. The team already + treats this as a spec-shaped artifact worth generating JSON Schema for — + it just isn't anywhere but flopy4. +- Helper logic (`_cf_var_attrs`, `dimmap`) encodes MF6's own NetCDF + conventions: CF `grid_mapping`/`coordinates`/`mesh` attrs, dimension name + mapping (`layer`/`y`/`x`/`nmesh_face`) per structured vs. vertex grid. + This is pure "what does a conformant file look like" — spec content, no + flopy4-specific state involved. +- This is exercised, not aspirational: `docs/examples/{frenchman-flat,twri,circle}/netcdf_*` + are real generated outputs. + +One nuance vs. the DFN case: *which package fields are NetCDF-eligible* is +already spec-driven from devtools — dev3 DFN field objects carry a real +`netcdf` attribute (`flopy4/mf6/utils/codegen/filters.py:442-443`, +`getattr(f, "netcdf", False)`), which codegen threads into the generated +`Package` field's attrs metadata (`flopy4/mf6/spec.py:65-66`). `netcdf.py`'s +`_PackageSpec` (`netcdf.py:80-126`) then reads that metadata back off +`attrs.fields(cls)` to decide which fields become NetCDF variables and +what shape/dims they get. **So per-field eligibility is already correctly +sourced from devtools.** What's missing from devtools is the *container* +format — how a `NetCDFModel`/`NetCDFPackage`/`NetCDFParam` document is +structured, and the CF/dimension conventions that govern it. + +## Where NetCDF differs from DFN (and why that matters) + +DFN's devtools schema formalizes something MF6 already emits as structured +text — the `.dfn` files themselves — and went through `dev1`→`dev2`→`dev3` +revisions with devtools as the convergence point for multiple consumers. +The NetCDF format has no structured source anywhere yet; formalizing it in +devtools would be the *first* machine-readable encoding of a spec the +modflow6 core team owns via a wiki page. That raises the bar: this isn't +"port flopy4's existing hand-rolled classes into devtools unilaterally," it's +"get the modflow6 core team (the issue thread already names @mjreno as the +collaborator on this) to agree the schema is right, ideally formalizing the +wiki page's content in the process." Moving prematurely risks devtools +carrying a schema that's really just flopy4's private opinion, dressed up +as shared infrastructure. + +## Recommended phased plan + +**Phase A — now: stay in flopy4, treat `netcdf.py` as the working +prototype.** No structural change yet. This mirrors how DFN's own schema +lived and iterated inside consumer-adjacent code before devtools' dev3 +stabilized it. Keep building out NetCDF support here while the mjreno +collaboration and the format itself are still evolving. + +**Phase B — trigger: format stabilizes + core-team alignment.** Split +`netcdf.py` into: + +- A pure structural/spec module → new devtools module (e.g. + `modflow_devtools.netcdf`), pydantic, versioned the way `dfns` is + (`dev1`/`dev2`/... or plain semver — team call). Candidates for the move: + `NetCDFModelAttrs`, `NetCDFParamAttrs`, `NetCDFParamEncodings`, the shape + of `NetCDFModel`/`NetCDFPackage`/`NetCDFParam` themselves, `_cf_var_attrs`, + `dimmap`. This is the part that answers "what does a valid MF6 NetCDF + file look like" — no flopy4 object touched anywhere in it. +- flopy4-side binding/consumption glue that **stays**, regardless of this + split: `NetCDFModel.from_model()` (walks a live `Model`'s attrs `Package` + fields via `_attrs.fields(type(package))`, reads `block`/`netcdf`/`shape` + metadata, broadcasts scalars, builds the params list — this is + application logic connecting flopy4's object model to the spec, not spec + content itself), `_PackageSpec`/`get_spec`/`_pkgclass` (same reason — + they read *flopy4's* `Package` classes), `to_dataset()`/`to_netcdf()` + and the rest of flopy4's own xarray/dask I/O mechanics. + +**Phase C — flopy4 imports the devtools spec classes** the same way +`netcdf.py`/`codegen/{make,filters}.py` already import +`modflow_devtools.dfns` today, and its own binding layer subclasses/wraps +them rather than re-declaring the structure. + +## What stays in flopy4 either way + +Regardless of timing, the following are flopy4-owned application concerns, +not spec, and don't move: + +- Binding a live simulation's field values into NetCDF variables + (`from_model()`, `_PackageSpec`). +- I/O mechanics: `to_xarray()`, `to_netcdf()`, dask/xarray laziness choices. +- Anything specific to *flopy4's* representation of a package/model (its + attrs field metadata, its dimension-resolution machinery in + `flopy4/dimensions.py`) — devtools has no reason to know flopy4 uses + attrs (or, per issue #282, might later use pydantic) for its own object + model. + +## Open questions for the team + +1. Who drives the devtools-side module — is this a joint effort with + mjreno/the modflow6 core team, given they own the wiki-page spec it + would formalize? Does formalizing it in devtools also mean updating or + replacing the wiki page as the source of truth? +2. Versioning scheme: mirror `dfns`' `devN` convention, or start at a + stable version since this is greenfield (no legacy consumers to keep + compatible)? +3. Sequencing against other in-flight work: [`mf6-object-model-plan.md`](mf6-object-model-plan.md) + Phase 0.6a step 3 (codec reader path still on legacy + `modflow_devtools.dfn`) is still open, and any attrs→pydantic move for + flopy4's own object model (issue #282) is a separate, orthogonal + decision. Recommend not running all three architecture changes + concurrently — DFN reader-path cleanup and this NetCDF split don't + depend on each other, but reviewer/contributor bandwidth does. + +## Non-goals + +This plan doesn't take a position on attrs vs. pydantic for flopy4's own +`Package`/`Component` object model (issue #282) — that's independent. If +that migration happens, consuming devtools' (pydantic) NetCDF spec classes +from a pydantic-based flopy4 object model would be slightly more uniform, +but it isn't required: flopy4 already consumes pydantic-based DFN spec +classes into attrs objects today with no friction, and would do the same +here. diff --git a/docs/dev/plans/archive/oc-format-load-bug.md b/docs/dev/plans/archive/oc-format-load-bug.md new file mode 100644 index 00000000..c63b80ac --- /dev/null +++ b/docs/dev/plans/archive/oc-format-load-bug.md @@ -0,0 +1,49 @@ +# `Simulation.load` fails on an OC print format with no format word + +Found by the DDL slice's corpus run, 2026-11-05, at flopy4 `8d25e4b7`. Not fixed. + +`mf6/test/test060_gms_ets` is the one model in `mf6/test` that doesn't load, and +MF6 runs it to normal termination (`bin/mf6`, `1469b358b9`). + +``` +TypeError: Oc.Format.__init__() missing 1 required positional argument: 'format_' +``` + +The line it fails on, in `ets3_dis.oc`'s OPTIONS block: + +``` +HEAD PRINT_FORMAT COLUMNS 10 WIDTH 11 DIGITS 4 +``` + +## Cause + +`format_` is required (`flopy4/mf6/gwf/oc.py:27`), but MF6 doesn't require the word. +flopy4 is following the spec here, and the spec is stricter than MF6: + +- The DFN says `optional false` for `format`, in the OC `formatrecord`. +- MF6's `print_format` (`src/Utilities/InputOutput.f90`, the loop after line 1475) + walks whatever words remain after `DIGITS`, and exits cleanly when there are none. + The caller's edit descriptor stands. + +So this is a gap between the DFN and MF6 rather than a parsing mistake. A fix in +flopy4 alone would diverge from the DFN. + +## To decide when fixing + +- Make `format_` optional in `Oc.Format`, and check that it writes back without the + word rather than as `None`. +- Whether the DFN should also say `optional true`. That's a change in MF6, and then + a new devtools snapshot. +- `grep -rn 'format_: str' flopy4/mf6` for the other components with the same record. +- `columns` and `width` are the reverse: optional in `Oc.Format`, but MF6 requires + them (the `nwords >= 4` check in the same routine). + +## After fixing + +The model isn't in `KNOWN_PASSING` (`test/mf6/test_mf6_load_all_models.py`), so it's +an xfail today, and should be added. Reproduce with: + +```python +workspace = copy_to(tmp_path, "mf6/test/test060_gms_ets", verbose=False) +Simulation.load(workspace / "mfsim.nam") +``` diff --git a/docs/dev/plans/archive/on-representations.md b/docs/dev/plans/archive/on-representations.md new file mode 100644 index 00000000..83f526f8 --- /dev/null +++ b/docs/dev/plans/archive/on-representations.md @@ -0,0 +1,309 @@ +# On data representations + +The topic of this document is data schemas, systems that work with them, and difficulties arising from data representations. + +## Overview + +The flopy4 MF6 parser handles representational mismatches between MODFLOW 6 DFN (Definition) files and our generated Python classes. This document describes the systematic transformation patterns implemented in `TypedTransformer` to bridge these gaps. + +## The Dual Mismatch + +### Primary: DFN ↔ Generated Classes + +**The Core Issue**: Current DFN files conflate two separate concerns: + +1. **Structure** (logical model): What variables exist, which components contain them, their types and dimensions +2. **Format** (serialization): How they appear in MF6 input files (tabular layout, record suffixes, keyword patterns) + +**DFN Representation** (structure + format): +- Tabular "list input" (recarrays) with separate column definitions +- File record suffixes (e.g., `budget_filerecord`) +- Generic record types with keyword discriminators (e.g., `saverecord` + 'HEAD') + +**Generated Python Classes** (structure only): +- Separate DataArrays for each conceptual field +- Clean field names without format artifacts +- Natural xarray idioms with dimensional indexing + +### Derived: Grammar ↔ Spec + +Because grammars are generated to **faithfully represent DFN structure** (including format), a secondary mismatch emerges: + +- **Grammar**: Reflects DFN (structure + format) +- **Python Classes** (the "spec"): Reflect clean structure only + +This grammar↔spec mismatch exists because of the primary DFN mismatch. + +### Future Direction + +We hope to eventually disentangle structure from format in DFN files, making format/serialization a separate concern. This would simplify or potentially eliminate much of the transformer's bridging work. + +## Data Flow + +``` +DFN Files (MODFLOW 6 format definition) + ↓ +Grammar Generation (dfn2lark) ──┐ + ↓ │ Both generated from DFN +Python Class Generation ────┘ but with different goals + ↓ +Lark Parser (parse tree from grammar) + ↓ +TypedTransformer (bridge patterns) ← THIS DOCUMENT + ↓ Bridges DFN→Class mismatch +Converter (structure into classes) + ↓ +Python Class Instances (xarray-based) +``` + +**Key insights**: +- **DFN files** currently conflate structure (logical model) with format (serialization details) +- **Grammar** is generated to faithfully represent DFN, including both structure and format +- **Python classes** are designed for clean structure only, using natural xarray idioms +- **TypedTransformer** bridges both mismatches: DFN↔Classes (primary) and Grammar↔Spec (derived) +- These patterns handle format-specific artifacts from DFN that don't belong in the Python class structure + +## Transformation Patterns + +### 1. List Input / Recarray Splitting + +**Problem**: DFN defines tabular data columns as separate field definitions (format concern), but they represent a single logical recarray. We want separate arrays per field (clean structure). + +**Example - WEL Package Period Block**: + +DFN has separate field definitions for each column: +``` +block period +name q +type double precision +shape (maxbound) + +block period +name aux +type double precision +shape (maxbound, naux) + +block period +name boundname +type string +shape (maxbound) +``` + +**Transformation pipeline**: + +1. **Grammar generation**: Groups period fields into single `stress_period_data` recarray rule (see `group_period_fields()` in `grammar/filters.py:54`) +2. **Transformer**: Collects parsed records into `stress_period_data` key (`transformer.py:493`) +3. **Converter**: `structure_array()` unpacks `stress_period_data` into separate per-field arrays (`converter/ingress/structure.py:675`) + +**Final Python class representation** (WEL): +```python +q: Optional[NDArray[np.float64]] = array(dims=("nper", "nodes"), ...) +aux: Optional[NDArray[np.float64]] = array(dims=("nper", "nodes"), ...) +boundname: Optional[NDArray[np.str_]] = array(dims=("nper", "nodes"), ...) +``` + +**Why**: DFN format concern (tabular columns) is bridged to clean structure (separate conceptual arrays). + +### 2. Record Pattern: `{prefix}record` + Keyword + +**Problem**: DFN defines generic record types with keyword discriminators (e.g., `saverecord`), but our Python classes expect specific field names for each variant. + +**Example - OC Package**: + +Grammar rule: +```lark +saverecord: "save"i word ocsetting +printrecord: "print"i word ocsetting +``` + +Input: `SAVE HEAD ALL` + +Parse tree: `saverecord['HEAD', ocsetting_all]` + +Python class expects: `save_head` field + +**Transformation** (`transformer.py:582-600`): +```python +if data.endswith("record") and data not in self._flat_fields: + if children and isinstance(children[0], str): + prefix = data[:-6] # Remove "record" + keyword = children[0].lower().replace("-", "_") + field_name = f"{prefix}_{keyword}" + + if field_name in self._flat_fields: + return (field_name, value) +``` + +Result: `('save_head', 'ALL')` + +**Why**: DFN uses generic format pattern with keyword discriminator; Python classes need explicit structure with named fields. + +**Scope**: Affects all OC packages (CHF, GWE, GWF, GWT, OLF, PRT, SWF) plus others with similar patterns. + +### 3. Grammar-Only Wrapper Unwrapping + +**Problem**: Grammar generation introduces intermediate wrapper rules for organization (to reflect DFN structure), but they don't correspond to Python class fields. + +**Example**: + +Grammar: +```lark +ocsetting: ocsetting_all | ocsetting_first | ... +ocsetting_all: "all"i +``` + +Parse tree: `ocsetting[ocsetting_all]` + +**Transformation** (`transformer.py:575-580`): +```python +if self._flat_fields and data not in self._flat_fields: + if len(children) == 1: + return children[0] # Passthrough unwrap +``` + +Result: Returns `ocsetting_all` directly, bypassing the wrapper. + +**Why**: Grammar wraps alternatives to reflect DFN hierarchical format; Python classes only need the terminal value (structure). + +### 4. Record Suffix Stripping + +**Problem**: DFN uses `*_filerecord` naming, but our Python classes use cleaner `*_file` names. + +**Example**: `budget_filerecord` → `budget_file` + +**Transformation** (`transformer.py:530-538`): +```python +if field is None and data.endswith("record"): + stripped = data[:-6] + field = self._flat_fields.get(stripped) + if field is None and stripped.endswith("file"): + field = self._flat_fields.get(stripped[:-4]) + if field is not None: + data = stripped +``` + +**Why**: DFN `*record` suffix is a format artifact that doesn't belong in clean structural field names. + +**Note**: This is a workaround pending DFN updates. + +### 5. Version Suffix Stripping + +**Problem**: Grammar rules include version suffixes (e.g., `tdis6`, `gwf6`) but our Python classes don't. + +**Example**: `tdis6` → `tdis` + +**Transformation** (`transformer.py:523-527`): +```python +if field is None and data.endswith("6"): + stripped = data[:-1] + field = self._flat_fields.get(stripped) + if field is not None: + data = stripped +``` + +### 6. Hyphen-Underscore Normalization + +**Problem**: DFN uses hyphens, Python uses underscores. + +**Example**: `field-name` → `field_name` + +**Transformation** (`transformer.py:521`): +```python +if field is None and "-" in data: + field = self._flat_fields.get(data.replace("_", "-")) +``` + +### 7. Union Alternative Handling + +**Problem**: Union types in Python classes have alternatives that grammar represents with `_` separators. + +**Example**: `ocsetting_all` where `ocsetting` is a union type in the Python class + +**Transformation** (`transformer.py:501-516`): +```python +if "_" in data: + parts = data.rsplit("_", 1) + field_name, alternative_name = parts + parent_field = self._flat_fields.get(field_name) + if get_field_type(parent_field) == "union": + if alternative_name in field_children: + if get_field_type(alt_field) == "keyword": + return alternative_name + return children[0] if len(children) == 1 else children +``` + +### 8. Record Type Flattening + +**Problem**: DFN record types have nested children that need to be collected into dicts. + +**Transformation** (`transformer.py:546-564`): +```python +if field_type == "record": + field_children = get_fields_dict(field.type) + record_dict = {} + for i, (child_name, _) in enumerate(non_keyword_children): + if i < len(children): + record_dict[child_name] = children[i] + return data, record_dict +``` + +### 9. Keyword Field Handling + +**Problem**: Keyword fields in DFN should become boolean flags. + +**Transformation** (`transformer.py:543-544`): +```python +if field_type == "keyword": + return data, True +``` + +## Implementation Location + +All transformation patterns are implemented in: +- **File**: `flopy4/mf6/codec/reader/transformer.py` +- **Class**: `TypedTransformer` +- **Method**: `__default__(data, children, meta)` +- **Lines**: ~370-602 + +Grammar generation utilities: +- **File**: `flopy4/mf6/codec/reader/grammar/filters.py` +- **Functions**: `group_period_fields()`, `get_recarray_name()`, `is_period_list_field()` + +## Design Principles + +1. **Generic over specific**: Patterns use structural matching rather than hardcoded names +2. **Fail-safe**: Only activate when mismatch detected (when `data not in self._flat_fields`) +3. **Layered**: Patterns tried in order from specific to general +4. **Transparent**: Transformations preserve semantic meaning (structure) while removing format artifacts +5. **Separation of concerns**: Patterns bridge the conflation of structure+format in DFN toward clean structure in Python classes +6. **Maintainable**: Adding new packages shouldn't require transformer changes + +## When to Add New Patterns + +Add a new transformation pattern when: +1. A systematic format artifact in DFN needs to be removed from Python class structure +2. DFN conflates structure+format in a way that affects multiple packages +3. Grammar faithfully represents DFN (including format), but Python classes need clean structure only +4. Pattern can be expressed generically using structural matching +5. Converter alone can't bridge the gap (needs parse-stage transformation) + +**Note**: As DFNs evolve to separate structure from format concerns, some patterns may become obsolete. + +## Related Documents + +- `transformer-dim-awareness.md`: How transformer handles xarray dimension names +- `array-converter-design.md`: How converter structures arrays from transformer output +- `grammar/README.md`: Grammar generation from DFN files + +## Summary + +The TypedTransformer's patterns address a dual mismatch: + +1. **Primary**: DFN files conflate structure (logical model) with format (serialization) +2. **Derived**: Grammar faithfully represents DFN → Python classes represent clean structure → mismatch + +These patterns systematically strip format artifacts from parsed data, preserving semantic structure while adapting representation. As DFN files evolve to separate concerns, this bridging layer may simplify or become unnecessary. + +## History + +**March 2026**: Added generic handlers for grammar-only wrappers and record patterns to fix OC period data loss bug, establishing systematic approach to handling structure/format conflation in DFN files. diff --git a/docs/dev/plans/archive/open-followups.md b/docs/dev/plans/archive/open-followups.md new file mode 100644 index 00000000..a0e25592 --- /dev/null +++ b/docs/dev/plans/archive/open-followups.md @@ -0,0 +1,82 @@ +# Open follow-ups + +Leftover items from finished plans, now in [`archive/`](.). Checked against +develop `a6c52c8` (#415) on 2026-10-07, and again against `e0d94b86` +(#489) on 2026-10-19. Each item names the plan it came from. Newer +follow-ups are in [`followups-after-row-layout.md`](followups-after-row-layout.md). + +## Open + +None. + +## DFN questions + +None open. + +## Done + +- **mypy and field converters** ([`mypy-converter-plan.md`](mypy-converter-plan.md)). + Not doing it: #433 never merged, and is closed. Typed constructors are + planned through generated stubs instead + ([`constructor-stubs.md`](../components/constructor-stubs.md)). +- **One-row list defaults.** The devtools spec now says a list's default + holds a single row, repeated to fill the list's shape (branch + `list-default` on the fork, unmerged). flopy4 already does this. +- **Datetime DFN type.** Not adding one: MF6 treats `start_date_time` as an + opaque label (list file, NetCDF time units) and never parses it. Typing + it is flopy4's job; the field takes `str`, `from_time()` converts. +- **Load context** ([`load-context-plan.md`](load-context-plan.md), branch `load-context`). + `LoadContext` carries the workspace, dims, parent and exchange models + down the load. Paths resolve against the simulation directory, so + models in subdirectories load, and the load no longer depends on the + cwd. +- **Exchanges between grids.** An exchange's cellids, and its GNC's, + each take their own model's grid width (DIS–DISV etc.), from the + models the exchange connects. +- **Which model a cellid column refers to.** The DFNs now say it + (devtools `cellid: true | "1" | "2"`). Codegen emits it as the + column's `cellid` metadata, `LoadContext.ncelldim` reads it, and + `Exchange.cellid_models` is gone. (branch `followups`) +- **Time-series names.** Writing a package checks that every name in a + time-series column (aux and keystring rows included) is defined by + one of its `Ts` children, as `_check_tas_refs` does for TAS. + (branch `followups`) +- **Pyright on mixin `self` types.** Pyright 1.1.414 accepts the mixins' + `self: "X"` annotations, and call sites resolve (`Dis().to_grid()` is a + `StructuredGrid`). Pyright reports at the `cls` line, not the `def` + line, so `Tdis.from_timestamps`'s wrapped signature needed a + `# pyright: ignore` there too. (branch `followups`) +- **SFR period `CROSS_SECTION TAB6 FILEIN`.** The table is an `Sfrtab` + child, as in `CROSSSECTIONS`. Codegen follows file links into keystring + arms; period rows load their children once per file, and the children + walk includes them. It's the only DFN with a file in a period row. + (branch `sfr-cross-section`) +- **Time series from data.** Decided against accepting a series in a + time-series column (it widens column types, and built and loaded + models would differ). Instead, `Ts.from_series` builds a TS6 file from + a DataFrame, a named Series or a mapping, and `Ts.to_dataframe` gives + the series back. (branch `ts-from-series`) +- **TS6/TAS6 attributes from plain values.** `Ts(time_series_name=["a"])` + kept the list, so the writer dropped `ATTRIBUTES` and the name checks + raised. Single-value records now convert (`Record.from_value`). The TS + and TAS checks share their name lookup. (branch `followups`) +- **Record fields from plain values** ([`record-values-plan.md`](record-values-plan.md)). Every + record field converts through `Record.convert` (a bool, the file's + text, a tuple, a mapping, or a lone value), replacing `from_flag` and + `from_value`; anything else raises at construction or assignment, and + the writer raises on a record field holding a non-record. + (branch `record-convert`) +- **Orphaned `pts.py`.** MF6 removed `sln-pts.dfn` (modflow6 #2912); + the module and its codegen test entry are gone. It was the only + generated file with no DFN. (branch `remove-pts`) +- **Dimension providers from DFN scopes** (branch `dims-from-schema`). + devtools #375 gives each dim a scope. A component with model- or + simulation-scoped dims gets `DimsProviderMethods` and `shared_dims` + from codegen, not a hand-kept list; `get_dims` gives only those (DISU's + `njas` stays its own). `DerivedDim` uses devtools' `dim_value` (#383) + and flopy4's evaluator is gone. +- **Dimension sync after construction** (branch `dims-from-schema`). + #407 synced dims on `get_dims`/`resolve_dims`/`write`; assigning a + dimension that disagrees with its list's rows now raises too. Reading + the attribute right after an in-place append is still stale until a + sync. diff --git a/docs/dev/plans/archive/optimization-plan.md b/docs/dev/plans/archive/optimization-plan.md new file mode 100644 index 00000000..fba413bd --- /dev/null +++ b/docs/dev/plans/archive/optimization-plan.md @@ -0,0 +1,457 @@ +# Plan: input-load optimizations (basic loader first) + +**Archived 2026-10-19.** Superseded by [`typed-loader-perf.md`](typed-loader-perf.md) and merged as #474, #479, #481, #483 and #484. Lazy imports, bulk numeric runs and bulk external arrays are all in. + +Goal: make flopy4 simulation loading faster than flopy3 across the board, by +taking per-token Python work out of the hot paths. Each step is measured +against the committed baseline and checked for unchanged results. + +Written 2026-09-29. It applies to the **basic** loader, which users run +today. None of it waits on the typed-loader wiring +([`typed-loader-wiring.md`](typed-loader-wiring.md)); the typed grammar gets the same bulk-token +idea when it's wired in. Index of related work: [`loader-roadmap.md`](loader-roadmap.md). + +## Background (read first) + +- `docs/dev/load-benchmarks.md` (branch `load-baseline`): the flopy3 vs. + flopy4 baseline, and "Why array reading differs: Python work per token, + per line, or per file". +- [`typed-grammar-remarks.md`](typed-grammar-remarks.md), "Why the typed loader isn't faster + (yet)": where parse time goes, and the original data-block idea. + +**`develop` has moved since the baseline.** The numbers below were measured +at `fe5423c`. Since then: +- #370 ("fix number parsing") changed both grammars. `basic.lark` now lexes + every whitespace-separated token as one generic `TOKEN`, and + `BasicTransformer` decides per token whether it's a number. `typed.lark`'s + number terminals now only match whole tokens. +- #373 ("fix support for trailing text") changed `structure.py`. +- #369 added a round-trip test (`test/mf6/test_mf6_io_roundtrip.py`). + +So **re-run `bench-load` and `bench-parse` on current `develop` first**, +and use that as the "before" for step 1. The per-token cost and the +flopy3/flopy4 ratios may have shifted. + +The core finding: speed tracks how much Python work runs per value. +Per-token work (Lark: lex, parse, a tree node, a callback per number) is +slowest, per-line work (flopy3's loops) is in between, and a whole-file or +whole-block pass in C is fastest. flopy4 is per-token for all inline data +today, and per-file only for `OPEN/CLOSE` arrays, which is the one place it +clearly beats flopy3. + +## Where time goes now (measured) + +| | flopy3 | flopy4 basic | notes | +|---|---|---|---| +| corpus, load + read all data (165 models) | 17.0 s | 14.4 s | one outlier is ~10 s of flopy3's time | +| same, without henrytidal | 6.2 s | 10.0 s | | +| cold start (fresh interpreter) | 0.94 s | 2.41 s | flopy4: 1.52 s imports, of which flopy3 ~0.64 s, xugrid ~0.45 s | + +- **Corpus inline tokens:** 1.44 M in total, split 59.5% array data, 38.6% + list rows, 1.9% other. External (`OPEN/CLOSE`) data is excluded. +- **Lark costs ~7-10 µs per token**, about 11.5 s of the 14.4 s. A bulk + numpy read costs ~0.25-0.4 µs per value, including tolerance for + comments and delimiters. +- **Lists: parsing dominates, not structuring.** On mv_disv_xt3d's DISV the + split is 1.61 s parse vs. 0.08 s structure; on henrytidal it's GHB 1.64 s + vs. 0.19 s and DRN 0.57 s vs. 0.08 s. + - So building one `Item` per row is *not* the next bottleneck after a + bulk list parse, at least until parsing is 5-10x faster. (This corrects + an earlier assumption that bulk lists needed a column-oriented object + model first.) + +## Ground rules + +- Work in a new worktree under `~/dev/` (never `/tmp`), off `develop` once + `load-baseline` has merged, or off `load-baseline` (rebased onto current + `develop`) until then. +- Run everything with `pixi run -e dev ...`. Verify with + `pixi run -e dev smoke`, not the full suite, unless asked. Sign commits + with the session's trailers. No pushes, PRs or GitHub actions unless the + user asks. +- **One PR per step.** Each PR description gives: + - `pixi run -e dev bench-load --benchmark-compare` before and after + (`bench-parse` too for grammar changes); + - the result of the snapshot check (step 0). +- **Coordinate on `structure.py`.** Steps 3 and 4 touch its array and list + consumers (`_read_control_record`, `_numeric_prefix`, `_parse_rows`). The + converter simplification (`~/dev/pyphoenix-simplify-converter`) is + reworking the same file. Check its state first and keep these changes + small and local. + +## Steps + +### 0. Snapshot check (prerequisite) + +A benchmark proves speed, not correctness. Before any optimization, add a +way to show that loaded values are unchanged: + +- A script (or `slow`-marked test) that loads every `KNOWN_PASSING` model + and writes a canonical snapshot per model: every scalar, array + (dtype + values) and list field, walked the same way as + `materialize_flopy4` in `test/mf6/test_mf6_load_benchmark.py`. Save it to + an `.npz`/JSON pair per model under a scratch directory. +- A compare mode that diffs two snapshot directories exactly: dtype + included, arrays compared with `np.array_equal`. +- Take one snapshot on the base branch and one on each step's branch. Any + diff must be explained in the PR (e.g. a value that was previously a + string and is now numeric). +- The round-trip test from #369 (`test/mf6/test_mf6_io_roundtrip.py`) has + since landed on `develop`, and there's the planned typed-vs-basic parity + test ([`typed-loader-wiring.md`](typed-loader-wiring.md), step 5). Check whether the round-trip + test's comparison can be reused for the snapshot compare before writing a + new one. Note that a round trip alone doesn't prove values are unchanged: + a value misread the same way on both loads still round-trips. + +### 1. Lazy imports (cold start) + +- Profile with `pixi run -e dev python -X importtime -c "import + flopy4.mf6.simulation" 2> imports.txt` and sort by cumulative time. +- Known module-level offenders (found by grep; confirm with the profile): + - `flopy4/adapters.py`: `import xugrid as xu`, + `from flopy.discretization.structuredgrid import StructuredGrid`; + - `flopy4/mf6/adapters.py`: several `flopy.*` imports (grid, modeltime, + export, plot); + - `flopy4/mf6/gwt/__init__.py`: + `from flopy.discretization.structuredgrid import StructuredGrid`. +- Move these into the functions that use them, or behind a lazy accessor. + Keep `TYPE_CHECKING` imports for annotations. Check that loading a + simulation never needs them. +- Target: flopy4's import time drops by roughly the ~1.1 s of flopy3 + + xugrid. Measure with `test_cold_start`, or directly with + `python test/mf6/test_mf6_load_benchmark.py cold flopy4-basic WS`. +- Low risk and independent of everything else, so this can go first. + +### 2. Harden the external-array reader + +`_read_open_close_values` (`structure.py`) is fast but fails on any `#` +comment or stray text in an external file. Make it tolerant without losing +the bulk path: + +```python +COMMENT = re.compile(r"[#!][^\n]*") +TABLE = str.maketrans({",": " ", ";": " ", "\t": " ", "'": None, '"': None, "d": "e", "D": "e"}) +values = np.array(COMMENT.sub("", text).translate(TABLE).split(), dtype=dtype) +``` + +- Measured on VilhelmsenGF's external files: 0.062 s vs. 0.042 s today, + still ~2x faster than flopy3's loop (0.134 s). +- **Check what MF6 accepts first.** The tolerated set above was a + synthetic stress test, not MF6's spec. Only accept what MF6's own reader + accepts; e.g. confirm whether `!` comments, `;` and quoted numbers are + valid in external arrays, and drop what isn't. +- **Count check.** Take the expected length from dims. Too few values is + an error ("expected N values for , got M"); extra trailing values + are ignored, matching MF6. +- **The same helper serves step 3,** so write it as a reusable function: + text in, flat ndarray out, with an optional expected count. +- **Tests:** unit tests with comments, commas, `D` exponents, and too-few + values, plus the snapshot check. + +### 3. Bulk numeric runs in the basic grammar (the big one) + +**Idea.** Lex a run of consecutive numeric-only lines as a single token and +convert it with step 2's helper, instead of one Lark token per number. + +**Grammar** (`flopy4/mf6/codec/reader/grammar/basic.lark`): +- Since #370 the grammar has a single generic `TOKEN` terminal, and + `BasicTransformer` decides per token whether it's a number. Add a + terminal matching one or more lines that contain only numbers and + separators, roughly `/(?:[ \t]*[-+.\d][-+.\deEdD, \t]*(?:\r?\n|$))+/`. + Refine it so it can't swallow a line that starts with a keyword. +- Lines with any word stop the run and go through the existing per-token + path. That covers control records, list rows with boundnames, and data + rows with trailing remarks (`0 0 0 ... row 1`). +- **Don't rely on longest match.** Per #370's own finding, Lark's lexer + takes the first matching terminal, not the longest; that's what split + ISO datetimes. So give the run terminal an explicit priority over + `TOKEN` (e.g. `DATA_RUN.2`), and test the edge cases directly: + - a lone number on a line of its own; + - a numeric line followed by a keyword line; + - `CONSTANT` values and dimension scalars (these sit on lines with + keywords, so they should be unaffected); + - ISO datetimes, `D` exponents, signed numbers: the #370 regression + tests in `test_mf6_codec.py` / `test_mf6_reader.py` must still pass. + +**The design issue:** the basic grammar has no types, so the terminal +captures **both** inline array data and rectangular list rows (e.g. +all-numeric stress-period rows). The token value must keep row boundaries: +- In the transformer, produce `(values: flat ndarray, row_lengths: int + array)`. Row lengths come from the run's lines, e.g. `[len(l.split()) for + l in lines]` (C-level splits, one Python iteration per line), or + vectorized from newline positions. +- If all row lengths are equal, the consumer can reshape to 2-D directly. +- Parse to float64 and cast in structuring, where the field's dtype is + known. That covers integer arrays, and cellid/integer list columns. + Check that every numeric form `flopy4.utils.parse_number` accepts today + still parses (e.g. `D` exponents). + +**Consumers** (`structure.py`): +- **Arrays:** `_read_control_record` and `_numeric_prefix` currently + extend a Python list row by row. Accept a bulk token by concatenating + arrays and trimming to `length`, keeping the "values can wrap over many + lines" behavior. +- **Lists:** `_parse_rows` builds one `Item` per row from token lists. + Feed it rows split from the bulk token (`np.split(values, + cumsum(row_lengths))`, or 2-D slices when rectangular). Per-row `Item` + construction stays, since it's ~5-15% of list cost (measured). +- **Anything else** that iterates block rows and could now see a bulk + token must be checked, e.g. `_resolve_open_close_rows` and the Pass 1 + scalar loop. Keep the token confined to data blocks (griddata, period, + packagedata-style), or make the transformer fall back to per-row lists + outside them. + +**Expected effect.** These are estimates from token shares × measured +per-token costs, not measurements. The step's PR must replace them with +`bench-load` numbers. + +| | flopy3 | flopy4 now | flopy4 after arrays | after arrays + numeric lists | +|---|---|---|---|---| +| corpus | 17.0 s | 14.4 s | ~8 s (6.5-9.5) | ~4-5 s | +| corpus without henrytidal | 6.2 s | 10.0 s | ~3.5 s | ~2.5 s | +| Keating (98% array) | 0.38 s | 0.67 s | ~0.1 s | ~0.1 s | +| mv_disv_xt3d (54% array, 46% list) | 0.56 s | 2.74 s | ~1.3 s | ~0.3 s, if DISV rows are numeric-only | + +- Small models gain less: they're dominated by fixed per-file overhead. +- DISV `cell2d` rows are variable-length, not rectangular, and + `vertices` rows are rectangular. Both are numeric, so both should take + the bulk path with row lengths. + +**Validation:** the snapshot check (no diffs), smoke, the load-all-models +test, and `bench-load` / `bench-parse` compares. Also compare the +`KNOWN_PASSING` count before and after: the grammar change mustn't make +any model stop loading. + +### 4. Lists that aren't all-numeric + +After step 3, the remaining per-token lists are rows with words: boundnames, +keywords, `OC` records, name-file bindings. **Measure first.** After step +3, check with a per-package split (parse vs. structure; see +`load-benchmarks.md` for how it was measured) whether these still matter. + +If they do: +- **Per-block regex, built from the list's column dtypes**, run in C over + the block text. It needs the component's field types, which the basic + path gets in structuring, not parsing. So this likely belongs in + `structure.py`, fed the block's raw text rather than tokens, or waits + for the typed grammar, where types are available at parse time. +- **Or `pandas.read_csv(sep=r"\s+", comment="#")`** on the block text, like + flopy3's `MFPandasList`. flopy3 pays a fixed cost per block with it + (henrytidal: ~1,900 blocks), so batch the blocks, or use it only for + large ones. + +### 5. Cheaper Lark for whatever still goes through it + +After steps 3-4, Lark handles mostly structure: keywords, options, control +lines. Measure whether these are still worth it: +- **Inline transformer.** Pass `transformer=BasicTransformer()` to the LALR + `Lark(...)` in `parser.py`, so callbacks run during the parse and no tree + is built. Check that terminal callbacks (`TOKEN` since #370, and `CNAME`) + still apply in that mode. +- **`cache=True`**, to save grammar analysis between processes. It matters + mostly for the typed loader's ~1.7 s of grammar construction, and for + cold start. +- **`debug=True`** is set on both parsers. Check whether it costs anything + outside development. + +### 6. Deferred: lazy external arrays + +See [`loader-roadmap.md`](loader-roadmap.md), "Deferred: lazy external arrays". This changes +what the object model holds, not how fast parsing is, so it's out of scope +here. Revisit after the converter simplification. + +## Benchmark infrastructure: one approach, and a weekly CI job + +This isn't a speed-up itself, but it's what makes the steps above +trustworthy over time. It can proceed in parallel with steps 1-3, since +each step only needs the local `bench-load` compare. + +### Today: two systems + +| | `docs/profile/` (on `develop`, #317) | `test/mf6/test_mf6_{parse,load}_benchmark.py` (`load-baseline`) | +|---|---|---| +| covers | writing input files (frenchman-flat, test1000, test1005, chunked dask streaming, list scaling); reading output (HDS/CBC, lazy vs. eager, chunk sweeps) | parsing and loading input files (flopy4 basic vs. typed; flopy3 vs. flopy4) | +| harness | argparse scripts with their own timer (`_timer.py`: min/mean over N runs, skip slow variants after one run) | pytest-benchmark, opt-in via `--benchmark-enable` | +| run | `python docs/profile/run_all.py [--runs N] [--memory] [--models-root DIR]` | `pixi run -e dev bench` / `bench-parse` / `bench-load` | +| results | custom JSON plus a Markdown report, **committed** in `docs/profile/results/` | pytest-benchmark JSON in `.benchmarks/` (git-ignored), compared with `--benchmark-compare` | +| extras | tracemalloc peak memory (`--memory`), cProfile (`--profile`) | equal-work check, pinned corpora, cold start in a fresh process | +| large models | `modflow6-largetestmodels` via `--models-root` / `MODFLOW6_LARGE_MODELS` | not used | +| its own tests | `docs/profile/test_profile_scripts.py` (outside `testpaths`, so not in normal runs) | skipped unless benchmarking is enabled | + +Both compare flopy3 and flopy4 variants, and both use min-of-N timing. They +differ in harness, result format, where results live, and how they're run. + +### Target: one consistent approach + +**One harness: pytest-benchmark.** +- It's already a dependency. It gives saved JSON, `--benchmark-compare`, + the `pytest-benchmark compare` CLI for reports, machine and commit info + in every save, and `--benchmark-cprofile`, which replaces the old + `--profile` flag. +- Memory has no built-in equivalent. Add a small fixture that runs the + target once under `tracemalloc` (as `_timer.profile_memory` does) and + records peak MiB in `extra_info`, enabled by an option such as + `--bench-memory`. + +**One layout.** Move all benchmarks to `test/bench/`, one module per +operation: + +| module | operation | parametrized by | +|---|---|---| +| `test_bench_parse.py` | input text → Python (flopy4 parsers only) | `basic`, `typed` | +| `test_bench_load.py` | input files → objects, full simulations | `flopy3`, `flopy4-basic`, later `flopy4-typed` | +| `test_bench_write.py` | objects → input files | `flopy3`, `flopy4`, and flopy4 output variants (ASCII list, grid, NetCDF) | +| `test_bench_output.py` | MF6 output (HDS/CBC) → arrays | `flopy3`, `flopy4-lazy`, `flopy4-eager`, chunk sizes | + +- A shared `test/bench/conftest.py` holds the opt-in gate, `quiet`, + corpus fixtures and manifests, the large-model fixture (skipped unless + `MODFLOW6_LARGE_MODELS` is set), and the memory fixture. +- A module that also needs a fresh-process script (like the load + benchmark's file check and cold start) keeps the three-part layout: + stdlib-only helpers, then the script entry point, then the benchmarks. + +**One naming scheme.** +- Test functions are named for the operation and stage: `test_load`, + `test_load_materialize`, `test_write_list`, `test_read_hds_eager`, and + so on. +- The first parameter is always the implementation ID: `flopy3`, + `flopy4-basic`, `flopy4-typed`, `flopy4-netcdf`, ... +- `benchmark.group` is the stage (e.g. `load-materialize`), so each group + table compares implementations side by side. +- `extra_info` always records the workload: bytes, files or models, + elements, and flopy's version. + +**One way to run.** +- `pixi run -e dev bench` runs everything available locally. Per-area + tasks are `bench-parse`, `bench-load`, `bench-write` and `bench-output`. +- Rounds come from one option, e.g. `--bench-rounds N` (default 3), so CI + can raise it without code changes. +- Keep the old slow-variant guard: a variant whose first run exceeds a + threshold runs once, unless `--bench-include-slow` is passed. + +**Results aren't committed.** +- Delete `docs/profile/results/` once the migration is done. History lives + in CI (below). +- A dated summary with numbers belongs in the relevant doc + (`docs/dev/load-benchmarks.md`, and a write/output counterpart ported + from the old `profile-write-performance.md`, removed from `docs/dev` in + #322 but still in history at `5119a70`). + +**Migration.** +1. Move the two existing modules to `test/bench/` under the new names, and + factor their shared pieces into `conftest.py`. +2. Port the `docs/profile` scripts one at a time: `ff_write`, + `test1000_write`, `test1005_write`, `chunked_profile`, + `diag_list_scaling` → `test_bench_write.py`; `ff_read` → + `test_bench_output.py`. Run old and new side by side on one machine and + check the numbers agree (within noise) before deleting each script. +3. Remove `docs/profile/` (scripts, `_timer.py`, `run_all.py`, + `test_profile_scripts.py`, `results/`) and point anything that + referenced it at the new tasks. + +### Weekly CI job + +**Workflow:** `.github/workflows/benchmark.yml`. +- Triggers: + - `schedule` (weekly, e.g. Sunday night UTC); + - `workflow_dispatch`, with an optional `ref` input to benchmark a + branch on demand; + - optionally, a PR label such as `benchmark`, for changes to the reader + or converter. +- It should never run on every push. The full set takes ~10 minutes + locally and more on CI runners. +- One runner type (`ubuntu-latest`) and one Python version. Cross-OS + numbers aren't comparable, and the question is the trend on one + platform. + +**Steps:** +1. Checkout; setup pixi (`dev` environment, same versions as `ci.yml`). +2. `pixi run mf sync`, as in `ci.yml`, for the models registry. +3. Cache the downloaded models and DFNs across runs with `actions/cache`, + so setup time doesn't vary. +4. Install MF6 (`modflowpy/install-modflow-action`) if the output + benchmarks need to run a simulation first. +5. Large models: clone or download `modflow6-largetestmodels` (cached) + and set `MODFLOW6_LARGE_MODELS`. This is the full set, as wanted, but + it's the slowest step. Check its size and cache it. +6. Run the full set: + `pixi run bench --bench-rounds 5 --benchmark-json bench.json`. +7. Upload `bench.json` as an artifact (90-day retention). +8. Append it to a long-term history (below). + +**Noise.** GitHub-hosted runners are shared VMs, and the CPU model can +differ from week to week. Two kinds of noise need different fixes: +- **Jitter within a run:** handled by rounds. Use 5 rounds plus one warm-up + (vs. 3 locally), and compare the **min**, which is the most robust + statistic for "how fast can this code go". Keep the mean and stddev in + the JSON for diagnosing. +- **Differences between runs** (different CPUs or neighbors each week): + rounds don't fix this. Two mitigations: + 1. **Ratios within a run.** flopy3 runs in the same job on the same + machine, so `flopy4 / flopy3` per group is largely + machine-independent. Make that the headline trend metric, rather + than absolute seconds. + 2. **A reference ref in the same job,** when a precise comparison is + needed: benchmark both the previous week's `develop` commit (or the + last release tag) and `HEAD` in one job, and report `HEAD / ref`. + This doubles the time, so use it for `workflow_dispatch` comparisons, + not every week. +- Record the CPU model (`lscpu` / `/proc/cpuinfo`) in the saved JSON, so + a jump that coincides with a hardware change can be told apart from a + regression. +- If noise stays a problem, a self-hosted runner on fixed hardware is the + real fix. That's a project decision. + +**History and alerts.** +- [`benchmark-action/github-action-benchmark`](https://github.com/benchmark-action/github-action-benchmark) + reads pytest-benchmark JSON (`tool: 'pytest'`; verify against its + current docs). It stores a history, renders trend charts, and can post + an alert when a benchmark crosses a threshold vs. the previous run. +- Docs deploy via `actions/deploy-pages`, not a `gh-pages` branch. So + point the action at a dedicated data branch (e.g. `bench-data`) rather + than `gh-pages`, and either publish the chart page from the docs build + or just link the data. +- Alert threshold: generous (e.g. 30% slower), since weekly runs land on + different hardware. Treat alerts as "look at this", not failures. Alert + on the flopy4/flopy3 ratio as well as absolute time, if the action + allows; otherwise compute it in a small post-processing step. +- The job never fails the build for a slowdown. It fails only if a + benchmark itself errors, which is how rot gets caught, e.g. a pinned + model that no longer loads, or an equal-work check that no longer holds. + +**Keeping benchmarks from rotting between weekly runs.** Normal CI skips +the benchmarks entirely. Add a cheap check to `ci.yml`'s test job that +runs the benchmark modules once with `--benchmark-disable` on a tiny +subset (e.g. `-k "largest1 or cold_start"` plus a 5-model manifest). That +proves the code still runs, without timing anything. + +### Order + +1. Merge `load-baseline` (stage 0 of the roadmap). +2. The weekly CI job for the existing parse/load benchmarks. It's useful + immediately, and it records a "before" history for the optimization + steps. +3. Unify the layout (`test/bench/`, shared conftest, `--bench-rounds`). +4. Port the `docs/profile` scripts, then delete them. + +## Not in this plan + +- **The typed grammar.** It gets the same bulk terminal when it's wired in + ([`typed-grammar-remarks.md`](typed-grammar-remarks.md), "Ideas", item 2). With types available, + the terminal can target arrays exactly and carry each list's column + dtypes. +- **Speeding up flopy3.** Worth noting in discussion #55 that a whole-file + fast path plus read caching would fix its external-array cost. That's not + our work. +- **Adding a flopy3 `lazy_io=True` column** to the benchmark. That's an + open decision in the roadmap. + +## Done when + +- Steps 1-3 are merged, each with before/after `bench-load` numbers and a + clean snapshot check in its PR. +- `load-benchmarks.md` has a new results section with measured numbers + replacing this plan's estimates, and flopy4 vs. flopy3 per regime (inline + arrays, external arrays, simple lists, complex lists, cold start). +- Steps 4-5 are either done or recorded as measured and not worth it, with + the numbers. diff --git a/docs/dev/plans/archive/package-binding-plan.md b/docs/dev/plans/archive/package-binding-plan.md new file mode 100644 index 00000000..027616ec --- /dev/null +++ b/docs/dev/plans/archive/package-binding-plan.md @@ -0,0 +1,169 @@ +# Package-level binding + +A package can name another component's input file in a file record, e.g. +`TS6 FILEIN rates.ts` in WEL's options. flopy4 should hold that file as an +attached `Ts`, load it with the package, and write the record from it, the +same way a model holds its packages. Most of the machinery exists. This is +what's missing. + +## What exists + +- **Component tree** (`component.py`). Any attrs field typed as a component + (single, list or dict) is a child (`child_field_candidates`). It gets + `_parent`, top-down and bottom-up attach, naming and collision checks, + `_children`, and a recursive `write()`. This already works for packages + under packages: `Dis.ncf` is written this way. +- **Bindings** (`converter/binding.py`, `_resolve_bindings`, + `_make_binding_blocks`). This is for contexts only: a row ` + [terms]` in a binding block (`packages`, `models`, `exchanges`, + `solutiongroup`). + - On load, ingress matches each row's token to a child field's class, + loads the file and attaches it. Dimension providers load first, and + their dims are passed to the rest. + - On write, the rows are generated from the attached children. + - `_make_binding_blocks` returns `{}` for anything that isn't a `Context`. +- **`subpackage(file_field=...)`** (`spec.py`, `6fecd21`). This is the + package-level half. Codegen emits `ncf: Optional[Ncf]` next to `ncf_file` + for each record in `SUBPACKAGES` (only `ncf_filerecord` today). On write, + `Package._sync_subpackage_files` names the child's file in the path field. + Nothing loads it. +- **Path fields** (`spec.path`). Each file record is a `Path` (or + `list[Path]` when the record repeats) with a `_keyword` (`ts6`). Ingress + fills it from ` FILEIN ` rows (`file_fields` in + `structure_component`). + +## What's missing + +### 1. Loading + +On load, `Package.load` / `structure_component` should, for each path field +that has a child field: + +- read the path(s); +- load each file as the child's class; +- attach the result. + +Reuse `_resolve_bindings`'s pieces rather than writing a second loader: + +- target lookup by ftype, including the G/A-variant disambiguation (TAS has + none, but SPC/SPCA would); +- the `Package.load(..., dims=..., name=...)` call; +- `child.filename = Path(fname)`. + +TAS needs the parent's grid dims passed to it (`tas_array` is `nodes`- or +`ncpl`-shaped); TS, OBS, ATS and NCF don't need any. + +Paths in a file record are relative to the simulation's working directory, +not the parent file's directory. `Package.load` passes `workspace=path.parent`, +which is only right when every file sits in the simulation directory. For a +package loaded through a model, the workspace should come from the +simulation. + +### 2. List children + +`subpackage()` and `_sync_subpackage_files` assume a single child: + +- TS and TAS records repeat (`ts_file: list[Path]`), so their child is + `ts: Optional[list[Ts]]`. Syncing has to give each child its own file + name. +- `Package.default_filename` stems from the parent's name, so two `Ts` + children of one WEL would both default to `wel.ts`. It should use the + child's own resolved name, which the tree already makes unique. + +### 3. Path field vs. child field + +Contexts have no separate path field: the binding row is generated from the +child. Packages keep both (`ncf_file` and `ncf`) and sync them one way, child +to path. That leaves: + +- **What if both are set and disagree?** Today the user's path wins silently. +- **A path with no child** is a reference to a file flopy4 didn't load or + doesn't manage. That should stay possible, e.g. for an OBS file before + `utl-obs` is generated, or a hand-written TS file the user only wants to + point at. + +Options: + +- **(a) Keep both, sync both ways.** On load, set both. On write, derive the + path from the child when there is a child. Smallest change; it's what + NCF does now plus loading. +- **(b) Derive the path field from the children.** It becomes a read-only + property when a child field exists, the same as a context's binding rows. + Cleaner, but it loses "point at a file without loading it". +- **(c) Union-typed field** (`ts: list[Ts | Path]`). One field, but every + consumer has to handle both. + +I'd start with (a), and raise on write when the path and the child's +filename are both set and differ. + +### 4. Which records link to which component + +- **Now:** add to `SUBPACKAGES` in `make.py`. `_subpackage_field_spec` picks + `Optional[X]` or `Optional[list[X]]` from the path field's type. +- **Later:** replace the table with `File.component` once the dev3 + migration carries it (`../modflow-devtools/subpackage-links-plan.md`). + +From the legacy DFNs' `# flopy subpackage` / `# mf6 subpackage` lines: + +| record | component | used by | flopy4 class exists? | +|---|---|---|---| +| `ncf_filerecord` | utl-ncf | `*-dis`, `*-disv` | yes (wired, write only) | +| `ts_filerecord` | utl-ts | ~25 stress/advanced packages | yes | +| `tas_filerecord` | utl-tas | gwf-rcha, gwf-evta, utl-spca | yes | +| `obs_filerecord` | utl-obs | most stress packages, exchanges | no (utl-obs not generated) | +| `ats_filerecord` | utl-ats | sim-tdis (hand-written `Tdis`) | yes | +| `tvk_filerecord` / `tvs_filerecord` | utl-tvk / utl-tvs | gwf-npf / gwf-sto | no | +| `hpc_filerecord` | utl-hpc | sim-nam | yes | +| `mvr`/`mvt`/`mve_filerecord`, `gnc_filerecord` | gwf-mvr / gwt-mvt / gwe-mve, gwf-gnc | exchanges | mvr, gnc yes | +| SSM/SSE `fileinput` rows (`pname spc6 filein `) | utl-spc | gwt-ssm, gwe-sse | no | + +Notes: + +- `Tdis` is hand-written, so ATS is wired by hand rather than by codegen. +- `hpc_filerecord` is on the simulation name file, which is a context. HPC + could go through either path. +- SSM/SSE `fileinput` isn't an options record: the file is a column in a + list row, keyed by `pname`. It needs a different shape, so leave it until + the rest works. +- MVR and GNC appear twice: under a model (a model binding row in `gwf-nam`'s + packages block) and under an exchange (a file record). The exchange case + waits on exchange support (#3 in the remaining issues). + +### 5. Write + +Writing mostly works already, since children are written recursively. What's +left: `_sync_subpackage_files` for lists, the conflict check from §3, and +making sure every linked child is written once. A child reachable both as a +model package and as a file record shouldn't be. + +### 6. Time-series names (separate, later) + +Once `Ts` objects are attached, a package can check that every +time-series name used in a `time_series=True` column is defined in one of +its `Ts` children. Optionally, it could also accept a series directly and +generate the `Ts`. This is an API decision, not binding work. + +## Order + +1. List children in `subpackage()` / `_sync_subpackage_files` / codegen, plus + per-child default file names. +2. Load path: file record → child, reusing `_resolve_bindings`'s lookup and + load. Test with NCF (already generated) and TS. +3. Add `ts_filerecord` and `tas_filerecord` to `SUBPACKAGES`. Regenerate. +4. Path/child conflict check on write. +5. ATS on `Tdis` by hand; OBS once `utl-obs` is generated; TVK/TVS once + generated. +6. Switch `SUBPACKAGES` to `File.component` when devtools lands. +7. SSM/SSE `fileinput`, exchange MVR/GNC: separately. + +## Tests + +- Load a WEL with two `TS6 FILEIN` lines: two `Ts` children, names and rows + intact, `ts_file` matching. +- Round trip: write the package, check the records and the TS files, load + again, compare. +- RCHA with a `TAS6` file: the child gets the grid dims. +- DIS with `NCF6 FILEIN`: `dis.ncf` is loaded (currently `None`). +- A path field with no child file present: load doesn't fail, the path is + kept, `ts` is `None`. +- Conflict: `ts_file` and the child's filename differ, so write raises. diff --git a/docs/dev/plans/archive/package-naming.md b/docs/dev/plans/archive/package-naming.md new file mode 100644 index 00000000..0feeb30d --- /dev/null +++ b/docs/dev/plans/archive/package-naming.md @@ -0,0 +1,552 @@ +# Package naming plan + +One rule for default package names and one for default filenames, plus a +write-time check that no two components share a file. Agreed with the user +on 2026-10-19, after the period-data guide turned up two WEL packages both +written to `model.wel`. + +Priorities: a simple, predictable user experience, and maintainability. + +## Background + +- **The bug.** Default filenames were `{model}.{class}`, so two packages of + one type wrote the same file, and the second silently overwrote the + first. MF6 still ran, reading one file twice. +- **Stopgap on `period-data-guide`.** Commit `79e36834` names a package's + file `{model}.{name}` only when a same-class sibling exists. This plan + replaces it. +- **PR #491 (mjreno, merged).** It made explicit names survive a rewrite + (an explicit `name="wel"` was being renamed `wel0`, which broke GWT SSM's + reference to it). It did this with a `_NAME_UNSET` sentinel default and + a `_name_given` flag set in `__attrs_post_init__`. It also numbers list + children by concrete class (`rcha0`, not `rch0`). Verified by applying + its diff to `e0d94b86`, it has two regressions, because names assigned + automatically are never marked as given, and `_set_child_parents` + re-resolves every child's name on each `_children` access: + - **Renumbering.** Delete `wel0`, and `wel1` becomes `wel0` on the next + access. Anything referring to `wel1` by name (SSM, MVR) breaks. + - **Lost renames.** `w.name = "north"` after attaching reverts to `wel0` + on the next access. +- **Old attempt.** The `package-names` branch/worktree (Oct 9, unmerged) + named the first child without an index (`wel`, `wel1`). The user chose + numbering from 0 instead, so that branch is superseded. Don't delete it + without asking. +- **flopy3 comparison.** In `names-and-paths.txt`. flopy3 numbers + multi-packages `chd_0`, `chd_1`, and checks for filename clashes via + `package_filename_dict`. We keep the clash check, and not its lenient + lookup (by type or prefix) or its warn-and-replace behaviour. + +## The rules + +### Names + +- **A given name is kept as is**, and raises a `ValueError` if a sibling + already has it. This covers `name=`, a name-file pname on load, a + `parent[key] = child` key, and assigning `.name` later. +- **An unnamed child in a single-child field** takes its class name: + `dis` for `Dis`, `disv` for `Disv` (even though the field is `dis`), + `npf`, `ic`. +- **An unnamed child in a list field** takes its class name plus the first + free index from 0: `wel0`, `wel1`, `chd0`, `chdg0`, `rcha0`, `ts0`. + Simulation's abstract-base lists follow the same rule: `gwf0`, `ims0`, + `gwfgwf0`. Indexes count per class, since the name includes the class. +- **Once assigned, a name stays.** An automatic name is stored, and is then + kept like a given one. Deleting a sibling never renumbers anything. +- **A detached component** (not attached to a parent) has `name=None` + until it's attached. +- **`parent[key] = child` needs the key and the name to agree.** If the + child's name is `None`, it takes the key. If it has a different name, + raise a `ValueError`: `gwf["north"] = Wel(name="south")` is an error, not + a silent rename. (Today `__setitem__` overwrites the name.) This also + applies to a component moved from another parent, which keeps the name + it had there; the error says to rename it or use that name as the key. +- **Renaming is assigning `.name`.** A clash with a sibling raises on the + next access to the parent's children, not at the assignment, since + `name` has no hook. Renaming doesn't update references to the old name + elsewhere: `Ims(models=[...])`, GWT SSM sources, MVR package names, + exchange model names. Those are the user's to update (see Out of scope). + Automatic names never change on their own, so references to them stay + valid. + +Field names are no longer used for naming anywhere. Lookups already go by +`.name` (`__getitem__`, `__setitem__`, `__delitem__`), so `gwf.dis` still +holds a DISV named `disv`. + +### Filenames + +The default for a package is `{stem}.{name}.{ext}`: + +- **`ext`** is the name-file ftype, lowercase, without the `6`: what a + user writes in a name file's packages block. Use `component_ftype(cls)` + (`converter/binding.py:12`), e.g. `CHD6` → `chd`. + - Most packages: `wel`, `chd`, `disv`, `npf`, `ts`, `obs`, `tas`, `ncf`. + - The grid and array variants (`Chdg`, `Drng`, `Evta`, `Ghbg`, `Rcha`, + `Rivg`, `Spca`, `Welg`) take their base ftype: a CHDG file is `.chd`, + an RCHA file `.rch`. The names still differ (`chd0` vs `chdg0`), so a + CHD and a CHDG in one model don't collide: `model.chd0.chd`, + `model.chdg0.chd`. + - Solutions: `IMS6` → `ims`. + - Exchanges: the ftype is `GWF6-GWF6`. Drop the `6`s and the hyphen: + `gwfgwf`. This matches flopy3, whose exchange `package_type` is + `gwfgwf` (default file `{sim}.gwfgwf`, `flopy/mf6/mfpackage.py:1870`), + and the DFN name `exg-gwfgwf`. Settled (2026-10-19). +- **`stem`** depends on the parent: + - under a model, the model's name; + - under the simulation, the simulation's name; + - under a package (a subpackage named by a file record: TS, TAS, OBS, + NCF…), the parent's filename without its suffix. So `model.wel0.wel` + has the child `model.wel0.ts0.ts`, and a parent given + `filename="north.wel"` has `north.ts0.ts`. + - with no parent, the component's own name, or its class name if it has + none. +- **Not packages:** + - a model is `{name}.nam`, unchanged; + - the simulation is `mfsim.nam`, unchanged. +- **One `filename` property.** It returns the filename that was set, or + else the default, worked out fresh on each read. There's no separate + "resolved" accessor. Setting it to `None` goes back to the default. +- **A set filename doesn't follow a rename.** This covers an explicit + `filename=` and a loaded package, whose filename comes from the name + file. Renaming a loaded package keeps its original file, which matches + what's on disk. Only a default filename follows the name: after + `w.name = "north"`, the next write gives `model.north.wel`, and its + subpackages follow (`model.north.ts0.ts`, with the WEL's `ts6` record + naming that file, since the record is built at write time). The old file + stays in the workspace; that's harmless, since MF6 reads the name file. + +### Known limitation: attaching by plain assignment is lazy + +`parent=`, `Wel(..., parent=gwf)` and `gwf["north"] = w` attach a child +at once. A plain assignment (`gwf.wel = [w]`) or a list mutation +(`gwf.wel.append(w)`) doesn't: the child gets its `_parent` and name only +when the parent's `_children` is next read (`component.py:351`). In +between, `w.name` is `None` and `w.filename` is `wel.wel.wel`, not +`model.wel0.wel`. Writing is unaffected, since a write goes through the +parent, which reads `_children` first. This PR doesn't close the gap; see +"Follow-up: attach on assignment". + +Examples: + +| Component | Name | File | +|---|---|---| +| One WEL | `wel0` | `model.wel0.wel` | +| A second WEL | `wel1` | `model.wel1.wel` | +| A CHD and a CHDG | `chd0`, `chdg0` | `model.chd0.chd`, `model.chdg0.chd` | +| An RCHA | `rcha0` | `model.rcha0.rch` | +| A DISV | `disv` | `model.disv.disv` | +| An IC | `ic` | `model.ic.ic` | +| A TS file under the first WEL | `ts0` | `model.wel0.ts0.ts` | +| A WEL named `north` | `north` | `model.north.wel` | +| TDIS | `tdis` | `.tdis.tdis` | +| An IMS solution | `ims0` | `.ims0.ims` | +| A GWF-GWF exchange | `gwfgwf0` | `.gwfgwf0.gwfgwf` | +| An unnamed model | `gwf0` | `gwf0.nam` | + +Default filenames can't collide: names are unique among siblings, and the +stem is unique per parent. Explicit filenames still can, hence the check. + +### Collision check + +Before writing, a simulation (or a model or package written on its own) +walks its tree and reads each component's `filename`. It raises a `ValueError` naming both components if two resolve to +the same path relative to the workspace. Compare the paths case-insensitively +(`casefold()` on the POSIX form), since macOS and Windows filesystems are +case-insensitive. + +## Steps + +### 0. Set up + +1. Fetch the latest `develop`. The user said to do this once the plan is + approved, but confirm in the moment, per CLAUDE.md. It should include + #491. +2. Create a branch `package-naming` from the latest `develop`. Don't reuse + the old `package-names` branch. +3. Read the merged #491 code in `flopy4/mf6/component.py`: + - `_NAME_UNSET` + - `_name_given` + - `__attrs_post_init__` + - `_is_default_child_name` + - `_resolve_child_name` + - `to_dict` + + Read `_RESERVED_FIELD_NAMES` in `flopy4/attrs_xarray.py` too. Line + numbers below are from before #491 and will have shifted. +4. Check whether `period-data-guide` (with `79e36834`) has merged. If not, + this branch still replaces `default_filename` wholesale. Expect a + trivial conflict in `package.py`, and resolve it in favour of this + plan. + +### 1. Names: `Optional[str]`, not a sentinel and a flag + +In `flopy4/mf6/component.py`: + +- `name: Optional[str] = attrs.field(default=None)`. Update the docstring: + `None` means not given yet, and attaching resolves it. +- Remove `_NAME_UNSET` and the `_name_given` field, along with their + handling in `__attrs_post_init__`. +- Remove `"_name_given"` from: + - the `to_dict` filter; + - `_RESERVED_FIELD_NAMES` in `flopy4/attrs_xarray.py`. +- Replace `_is_default_child_name(child)` with `child.name is None`. + Inline it if it's then trivial. +- Rewrite `_resolve_child_name(used, kind, child)`. Drop the `field_name` + and `by_class` parameters. + - **Given (`child.name is not None`):** if the name is in `used`, raise + the existing collision error; otherwise return it. + - **`kind == "only"`:** return `type(child).__name__.lower()`. + - **`kind == "list"`:** return `f"{cls}{i}"` for the first `i` from 0 + not in `used`. + - Update its docstring to state the rules above. +- Delete `_names_by_class` and every `by_class=` argument: + `_set_child_parents` (≈ line 435–443) and `_attach_to_parent_field` + (≈ line 529). +- `_set_child_parents` also names period-row children (≈ line 421, which + passes `f.alias or f.name`). Use the list rule, by class. +- **The collision check in `_set_child_parents`.** It builds `used` + incrementally. A given name must still raise on a clash, but a child's + own stored name mustn't count as a clash with itself on re-resolution. + The current code adds each resolved name to `used` after resolving it, + so re-running on stored names works, provided every stored name is + treated as given. Check this with the deletion and rename tests below. +- **`__setitem__`** sets `value.name = key`. Change it to: if + `value.name` is `None`, set it to `key`; if it equals `key`, fine; + otherwise raise `ValueError` (see the rules). Confirm it still raises on a + clash with a different child already under `key`, except when replacing + that child in place. + +**Places that read a name before attaching** may now see `None`. Grep +`\.name\b` in `flopy4/mf6/` and check each read that can happen on a +detached component: +- `__repr__`/`repr`: show the class name, or `None`. +- `default_filename` with no parent: falls back to the class name (see + step 2). +- `Model.default_filename`: `{name or cls}.nam`. +- Standalone `write()` of a model or package. +- `NetCDF` variable naming (`netcdf.py` ≈ 760, `f"{package_name}_{param}"` + for multi-packages): runs at write time, after resolution, so names are + set. Confirm. +- `Binding.from_component` (`converter/binding.py:39`): runs at write + time. Confirm. +- Loading (`structure_component`, `converter/ingress/structure.py`): passes + the pname as `name=`, so it's given. A name-file row without a pname + leaves `name=None`, so it gets the automatic name. + +### 2. Filenames + +- Add a helper on `Component`, `_file_ext()`, returning + `component_ftype(type(self)).lower().replace("6", "").replace("-", "")`. + Check there's no import cycle with `converter/binding.py`. If there is, + move `component_ftype` somewhere both can import, or import it inside + the method. Check that every class with a file gets an ftype from it + that MF6 recognises, especially subpackages (`Ts` → `TS6`, `Obs` → + `OBS6`, `Tas` → `TAS6`, `Ncf` → `NCF6`) and period-row table files + (`laktab`, `sfrtab`…), whose names may not follow the pattern. +- `Package.default_filename` (`package.py:778`), replacing both the + `79e36834` sibling check and the subpackage branch: + + ```python + def default_filename(self) -> str: + parent = self._parent + if isinstance(parent, Package): # named by a file record + stem = Path(parent.filename or parent.default_filename()).stem + elif parent is not None: + stem = parent.name + else: + stem = self.name or type(self).__name__.lower() + name = self.name or type(self).__name__.lower() + return f"{stem}.{name}.{self._file_ext()}" + ``` + + `Path(...).stem` drops only the last suffix (`model.wel0.wel` → + `model.wel0`), which is what we want. +- **Remove these overrides** so they inherit the rule: + - `Solution.default_filename` (`solution.py:14`, fixed + `solution.`, which has the same collision bug with two + solutions); + - `Exchange.default_filename` (`exchange.py:17`, `{name}.exg`). + + Check the base `Component.default_filename` (`component.py:470`) is + still right for anything left using it. Exchanges and solutions are + `Package`s? Confirm they inherit `Package.default_filename`. If not, + give them the same rule. +- **Keep these overrides:** + - `Model.default_filename` (`model.py:36`, `{name}.nam`, with the + `None` fallback above); + - `Simulation` (`simulation_methods.py:17`, `mfsim.nam`). +- **Check the simulation's name.** Confirm `Simulation.name` is always set. + If it can be `None` or a default, decide what the stem is for TDIS, IMS + and exchanges (probably the default name, e.g. `sim`). + +### 3. Stop saving default filenames + +Agreed to do this (2026-10-19), since it's small. Sized beforehand: +- 10 source sites read `.filename`, and most already use + `filename or default_filename()`: + - `package.py:781` + - `__init__.py:169` + - `egress/unstructure.py:45` + - `binding.py:51` +- About 6 tests read `.filename` after a write without setting it: + - `test_mf6_component.py:1396`, `:1397` and `:2239` + - `test_mf6_integration.py:2909` and `:2913` + - `test_mf6_sync_dims.py:41` + +Loaded components get `filename` from the name file, so they're +unaffected. After a rename, the old file is left in the workspace, which is +harmless, since MF6 reads the name file. + +Today the first `write()` or `.path` saves the default into `filename`: +- `component.py:451` (`path`); +- `component.py:659` (`write`, with a TODO calling it a temp hack); +- `context.py:24`; +- `package.py:729` (`_name_file_children`). + +That makes the filename depend on history: rename a package after its +first write, and its file keeps the old name. With deterministic defaults +the saving isn't needed. + +- Make `filename` a property over a private attrs field, so there's one + accessor (the user's preference, 2026-10-19): + ```python + _filename: Optional[Path] = attrs.field( + default=None, alias="filename", converter=attrs.converters.optional(Path) + ) + + @property + def filename(self) -> Path: + return self._filename or Path(self.default_filename()) + + @filename.setter + def filename(self, value) -> None: + self._filename = None if value is None else Path(value) + ``` + `Component(filename=...)` still works through the alias. The class is + `slots=False` (`component.py:245`), so the property is fine there. +- Remove the four saving sites above. At `package.py:781`, + `__init__.py:169`, `egress/unstructure.py:45` and `binding.py:51`, replace + `x.filename or x.default_filename()` with `x.filename`. +- Only code that needs to know whether a filename was *set* reads + `_filename`. Expected: serialization, and the simulation's `mfsim.nam` + check (`simulation_methods.py:23`), which should check `sim._filename` + is `None` or `mfsim.nam`. Anything else reading `_filename` is a smell. +- Code that iterates attrs fields and names `filename` now sees + `_filename`: `package.py:812` (`_exclude`), `component.py:711` + (`data.pop("filename")`), `attrs_xarray.py` field filtering, `to_dict`, + unstructure and any `attrs.evolve`. Check each. +- `root` (`component.py:466`) reads `filename` to check for an absolute + path. With the property it computes the default, which doesn't call + `root`, so no recursion; confirm. +- Tests: `test_mf6_component.py:1391` asserts `ncf.filename is None` + before a write; change it to assert the default. The tests listed above + that read `.filename` after a write should pass unchanged. +- If this turns out to touch much more than these sites and their tests + (for example, something that needs `filename` set for reasons other than + writing), stop. Keep the saving, and note it as a follow-up. It's + separate from the naming bug. + +### 4. Collision check + +- Add a function, e.g. `_check_filenames(root)` in `component.py`, that + walks the tree from the component being written. Use `_children` + recursively, including package children like TS and OBS. It maps + `filename.as_posix().casefold()` to the component, and raises + on a repeat: + `ValueError("model.wel0.wel is the file for both and ; give one a different filename")`. +- Call it once at the top-level `write()`: the component with no parent, + or the one `write()` was called on. Not in every recursive child write. + `Component.write` (`component.py:641`) recurses into children, so either + check `self._parent is None` there, or add it to `Simulation.write` and + standalone writes. Pick whichever keeps it in one place. +- Paths are relative to the workspace. Treat an absolute `filename` as is. +- Out of scope: external array files (OPEN/CLOSE) written by + `external.py`. Mention it in the PR as a possible follow-up. + +### 5. Tests + +New tests, probably in `test/mf6/test_mf6_component.py`, near +`test_several_packages_of_a_type_default_filenames` (from `79e36834`, +which these replace). + +**Names:** +- a single child gets its class name: a `Dis` → `dis`, and a `Disv` in the + `dis` field → `disv`; +- list children: two WELs → `wel0`, `wel1`; a CHD and a CHDG → `chd0`, + `chdg0`; +- an unnamed `Gwf` in a simulation → `gwf0`, and an `Ims` → `ims0`; +- an explicit `Wel(name="wel")` stays `wel`, and an unnamed sibling WEL + becomes `wel0`; +- an explicit name that clashes with a sibling raises; +- **deleting:** two WELs, delete `wel0`, access `gwf` again, and the other + is still `wel1`. A new WEL added then gets `wel0`. (Regression for #491.) +- **renaming after attaching:** `w.name = "north"`, access `gwf` again, + and it's still `north`. (Regression for #491.) +- `gwf.wel = [Wel(...), Wel(...)]` (plain assignment): names are resolved + on the next access, and after reading `gwf`, `gwf.wel[0].filename` is + `model.wel0.wel` (pins down the known limitation); +- a detached `Wel(...).name is None`; +- `gwf["north"] = Wel(...)` names it `north`. +- `gwf["north"] = Wel(name="north")` is fine, and + `gwf["north"] = Wel(name="south")` raises. + +**Loading:** +- a name file with a pname keeps it; +- a row without a pname gets the automatic name. + +Check that existing load tests cover these, and add them if not. + +**Filenames:** +- model-level: `model.wel0.wel`, `model.disv.disv`, `model.chdg0.chd`, + `model.rcha0.rch`, and a CHD and a CHDG in one model writing separate + files; +- simulation-level: `.tdis.tdis`, `.ims0.ims`, and an exchange; +- a subpackage: `model.wel0.ts0.ts`, and under a parent with + `filename="north.wel"`, `north.ts0.ts`; +- an explicit `filename=` is used; +- standalone, with no parent; +- step 3, if done: rename after a write, and the next write uses the new + name. +- `filename` returns the default before any write, and `w.filename = None` + goes back to it; +- a loaded package renamed keeps its loaded filename. + +**Collision check:** +- two packages with the same explicit `filename=` raise at `sim.write()`; +- names differing only in case raise; +- a model with several of each type writes without raising. + +**End to end:** the period-data guide (two WELs, a TS, an EVT) writes and +runs. `test/test_examples.py -k period-data` covers it once `mf6` is on +the `PATH`. + +### 6. Update existing tests, snapshots, examples and docs + +- **Literal auto-names** (`"wel0"`, `"chd0"`, `"ts0"`, …) appear in: + - `test_mf6_period_lists.py` (8) + - `test_mf6_codec.py` (5) + - `test_mf6_netcdf.py` (5) + - `test_mf6_integration.py` (5) + - `test_mf6_io_roundtrip.py` (5) + - `test_mf6_adapters.py` (2) + - `test_mf6_component.py` (2) + - `test_mf6_grammar_gen.py` (2) + - `test_mf6_codegen.py` (1) + - `test_mf6_netcdf_cf_conventions.py` (1) + + Most list names are unchanged (`wel0`). Look for single children whose + name changes (`dis` → `disv`), models (`gwf` → `gwf0`), and tests + asserting filenames (`.wel`, `.chd`, `solution.ims`, `.exg`). Some + matches may be grammar tokens or MF6 text, not names. Check each. +- **Snapshots:** `test/mf6/__snapshots__/test_mf6_load_all_models/`. Loaded + packages keep their pnames and filenames from the name file, so changes + should be limited to single children without a pname (e.g. a DISV: + `dis` → `disv`). Regenerate with `pixi run -e dev pytest + test/mf6/test_mf6_load_all_models.py --snapshot-update` (the snapshot + plugin is from `modflow_devtools.snapshots`). Review the diff rather + than accepting it wholesale. +- **Examples:** grep `docs/examples/*.py` for package names and filenames: + `gwf["…"]`, `.name ==`, `solution.ims`, `mymodel.chd`. Regenerate the + notebooks that change. `twri.ipynb` was already out of date with its + script, so don't re-execute it blindly. + - Regenerate with `PATH="$PWD/.pixi/envs/dev/bin:$PATH" pixi run -e docs + python scripts/update_ghpages.py --example `. The docs env has no + `mf6`. + - Never `git checkout` a notebook before syncing. `jupytext --sync` will + copy the newer notebook over the `.py`. If a notebook needs + resetting, touch the `.py` afterwards. + - Strip stderr outputs afterwards, since they contain local paths. +- **Docstrings:** `Component.name`, `_resolve_child_name`, `_set_child_parents`, + `_children` (its docstring mentions the class-name default), and + `Package.default_filename`. +- **`docs/dev`:** grep for naming and filename descriptions. Don't edit the + plan files in the project root or `archive/`. + +### 7. Verify + +- `pixi run -e dev ruff check` and `ruff format --check` on the touched + files, plus `codespell`. +- `pixi run -e dev mypy` on the touched modules. `name` becoming + `Optional[str]` will show up at the places that read it. +- Run the touched test files, and `test/mf6/test_mf6_load_all_models.py`. +- Run `pixi run -e dev smoke`, which CLAUDE.local.md asks for after + library changes. Not the full suite. +- Write and run the period-data guide, and check the output folder: + `model.wel0.wel`, `model.wel1.wel`, `model.wel0.ts0.ts`, and normal + termination in `mfsim.lst`. + +### 8. Commit and PR + +- Suggested commits: + 1. names: `Optional[str]`, class names, no renumbering; + 2. default filenames `{stem}.{name}.{ext}`, and drop the `Solution` and + `Exchange` overrides; + 3. stop saving default filenames (if done); + 4. filename collision check; + 5. tests, snapshots, examples and docs, either with each change or + separately. +- Don't push or open a PR unless the user asks at the time. +- **The PR description must call out the visible changes:** + - default filenames are now `{stem}.{name}.{ext}`, with `ext` the + name-file ftype (`model.wel` → `model.wel0.wel`, `model.npf` → + `model.npf.npf`, `model.rcha` → `model.rcha0.rch`, `solution.ims` → + `.ims0.ims`, `.exg` → `.gwfgwf0.gwfgwf`); + - a default filename no longer changes depending on whether the package + has been written (if step 3 is done); + - single children are named by class (`dis` → `disv` for a DISV); + - unnamed models are numbered (`gwf0`); + - automatic names no longer renumber when a sibling is deleted, and a + rename after attaching sticks (fixes the regressions from #491); + - writing raises if two components resolve to the same file; + - `parent[key] = child` raises if the child already has a different + name; + - `filename` always returns a path (the default if none was set), and a + default filename follows a rename; a set or loaded filename doesn't. + +## Follow-up: attach on assignment + +A separate PR after this one, to close the known limitation. Sketch: + +- **A hook for whole-field assignment.** The class is `slots=False`, so + either attrs `on_setattr` on child fields or a `Component.__setattr__` + that, for a child field, attaches the new value(s) (stamp `_parent`, + resolve the name) and detaches the old ones (`_parent = None`, the name + kept). Pick whichever also covers fields added by subclasses without + per-field wiring; `__setattr__` probably. +- **A list proxy for list fields**, so `append`, `extend`, `insert`, + `__setitem__`, `+=` attach and `pop`, `remove`, `del`, `clear` detach. + Like xattree's `DataTreeList` (`../xattree/xattree/__init__.py:47`), + with these differences: + - **Don't key children by list index.** `DataTreeList` names children + `f"{prefix}{index}"`, so deleting one renumbers the rest (the same + bug as #491), and its `insert` overwrites the item at that index + instead of inserting. Names come from `_resolve_child_name` and are + stored, as in this PR. + - **Subclass `list` rather than `MutableSequence`**, so the 23 + `isinstance(..., list)` checks in `flopy4/mf6`, equality, cattrs and + repr keep working. That means overriding every mutator, since `list` + doesn't route them through one method; test each. + - Wrap the value in the proxy in the hook (and on init), so assigning a + plain list still gives a proxy. +- **Keep the access-time pass** (`_set_child_parents` on `_children`) as + a safety net, at least at first. Not every attach goes through a hook: + children inside period-data rows (`_period_row_children`) are reached + through row data, not a child field, and a list built elsewhere and + mutated before assignment is only seen when assigned. With the hooks in + place, the pass should find nothing to do; once that's confirmed, it + could run only before a write. +- **Tests:** after `gwf.wel = [w]`, and after `gwf.wel.append(w)`, + `w.name == "wel0"` and `w.filename` is `model.wel0.wel` without reading + `gwf` first; removing a child clears its `_parent` and keeps its name; + each list mutator attaches or detaches. + +## Out of scope + +- Model subdirectories, moving external files with the workspace, and + quoting filenames with spaces (from `names-and-paths.txt`). +- Collision checks for external array files. +- Lenient lookup by type or prefix (flopy3's `get_package`). +- Attaching on plain assignment or list mutation (see "Follow-up: attach + on assignment"). +- Updating references on rename. Renaming a component doesn't update other + places that name it (`Ims(models=[...])`, SSM sources, MVR packages, + exchange model names). That would be its own feature. diff --git a/docs/dev/plans/archive/period-lists.md b/docs/dev/plans/archive/period-lists.md new file mode 100644 index 00000000..f71b0077 --- /dev/null +++ b/docs/dev/plans/archive/period-lists.md @@ -0,0 +1,173 @@ +# Period lists in xarray + +Plan for giving list packages' stress period data an xarray form, generically, +with no per-package code. Starts from commit `e87c53e` ("put list packages' +stress period data on the grid"), which exists only on `ts-ref` (local and +`origin/ts-ref`). + +## Starting point: e87c53e + +- New `flopy4/mf6/period_lists.py` (276 lines), `Package.to_xarray()` branch, + new `Package.from_xarray()`, `test/mf6/test_mf6_period_lists.py` (12 tests). +- Puts each value column on the dense grid, `(per, ...grid)`, a variable per + aux name, `FILL_DNODATA` where a period has no boundary, carrying periods + forward as MF6 does. `from_xarray` writes a period block only where a period + differs from the one before. +- Already structural, not name-based: the cell is found from `cellid` + metadata, or through an `fk` to a table with one cellid (UZF's `ifno` -> + packagedata). +- Raises (pointing to `to_dataframe()`) for: two rows in one cell, time-series + values, keystring rows (SFR, LAK, MAW, TVK), two cellids (HFB), no cellid + (MVR, STO). Drops boundnames with a warning. +- Builds dense `(per, grid)` arrays eagerly via `period_arrays.dense`. +- Array columns (EVT's `pxdp`, `petm`, length `nseg - 1`) get an extra dim, + named with a placeholder `_dim0`. +- Branch state: `ts-ref`'s other commits (`f9caa23`, `9428067`, `f3ad6f3`) + landed in develop via squash merges (#450, #456). `e87c53e` merges cleanly + onto develop `22e1fd9`; its dependencies (`period_arrays.dense`, + `FILL_DNODATA`, `TimeSeriesRef`, `_row_dict`) are all on develop. Not yet + tested against #454/#455. + +## Design + +### Index dim from the row's key + +| Row key | Index dim | Examples | +|---|---|---| +| one `cellid` | grid: `(layer, row, col)` / `(layer, cell)` / `(node,)` | CHD, WEL, DRN, TVK | +| `fk` to another table's `pk` | that table's sizing dim, e.g. `(per, nreaches)` | UZF, SFR, LAK, MAW, LKT | +| anything else (two cellids, no key) | `entry` dim, period and key columns as coordinates | HFB, MVR | + +All from metadata: `cellid`, `fk`, `pk`, `union`, `array`, `time_series`, +`dim`, `shape`. + +### Dim naming + +Name dims after the DFN sizing dimension, which the user sees, not a field +name, which they don't. + +- fk-keyed rows: `fk="packagedata.ifno"` names the `packagedata` field, whose + `dim=` metadata gives the dim (`nreaches` for SFR, `nuzfcells` for UZF, + `nlakes`, `nmawwells`, ...). +- Array columns: name the extra dim from the column's `shape` metadata, not + `_dim0`. +- Grid dims (decided 2026-10-14): the exception, named for the grid's axes, + `(layer, row, col)`, `(layer, cell)`, `(node,)`, since they're typed in + selections constantly and xarray has no dim aliases. DFN sizing names only + in `constants.GRID_DIMS`, for size lookup. No index coordinates (positions + are the indices; `k` clashes with NPF's/TVK's `k`). Sparse cellid columns + `cellid_layer`, ...; sparse dim `entry` (not `row`). `cell`, not `node`, + for DISV: UGRID's node is a vertex. Branches: `grid-dims` (#469: + `to_dataarray`; DIS output readers to `row`/`col` with indexed x/y and no + 1-based layer coord; DISV output keeps xugrid's face dim), `period-lists`, + `grid-coords` (`utils/grid.py`, `grid.dims`). `disv-j` abandoned. + +### Dense by default, sparse opt-in + +- `to_xarray()` is dense along the key dim: the grid for cellid-keyed rows, + the feature dim for fk-keyed rows (one slot per feature, missing where a + period doesn't set it). Grid projection of fk-keyed rows (through + packagedata's cellid) is a separate opt-in view. +- Rows with no dense form (HFB, MVR) fall back to the row form, the same + output as sparse. No raising. +- `to_xarray(sparse=True)`: lossless `(per, row)` form for any package, cellid + (or other key) as a coordinate. +- Two rows in one cell: `to_xarray(duplicates="raise" | "sum" | "first")`. + Default `"raise"` (lossless), and the error offers `sparse=True` or + `duplicates="sum"`. `"sum"`/`"first"` are documented as lossy (merged + rows lose boundnames; `from_xarray` won't round-trip). No per-column + "additive" knowledge: the reduction is the user's choice. +- Array columns stay a dim (ordered, like quantities); aux stays a variable + per name (named, unlike quantities). Flattening to `pxdp_1..n` is the + flopy3 adapter's job. + +### Keystring (union) rows + +Pivot `(key, setting, value)` rows into one variable per setting: +`status(per, nreaches)`, `inflow(per, nreaches)`, ... (members from `union` +metadata). Text settings become string variables. Missing means not set this +period. Settings persist per setting in MF6 (not replaced as a block), so +carry forward per setting; the union-ness of the row says which semantics +applies. + +### Lossless extras + +- Boundnames: a string coordinate on the key dim (dense) or row dim (sparse). +- Time-series values: numeric variable with NaN where a series applies, plus a + `_ts` string coordinate/variable holding series names, mirroring + period arrays' `_tas`. Evaluation (see tas-interpolation-plan.md) can + later fill values as a separate opt-in step. + +### Memory + +Don't build `(nper, grid)` per column eagerly. Store only periods with blocks +and fill forward, as period arrays do, or build lazily (dask). + +## Steps + +1. Rebase `e87c53e` onto develop in a new worktree, branch `period-lists`: + `git rebase --onto origin/develop f3ad6f3 ` (don't touch `ts-ref` + or the main checkout, which is on `ts-ref`). +2. Rename DISV dim `face` -> `j` in `Package.to_dataarray`, its dask test, and + `_GRIDS`. Possibly a small separate PR first. +3. Rework `period_lists`: + - key dim from `cellid` / `fk`->`dim` / row fallback; + - pivot keystring rows, per-setting carry-forward; + - boundnames and TS names as coordinates; + - sizing-dim names (fk features, array columns); + - periods-with-blocks storage or lazy build; + - `sparse=True`; + - `duplicates=` reduction; + - `from_xarray` inverse for each form. +4. Tests: rework existing; add SFR/LAK/MAW (keystring, feature dims), UZF + (feature dim + grid view), HFB/MVR (row fallback), boundnames, TS values, + two-rows-in-a-cell raise, `duplicates="sum"`/`"first"`, sparse round + trips, multi-value aux with boundnames. +5. Verify: `pixi run -e dev pytest test/mf6/test_mf6_period_lists.py` plus + other touched files, ruff, mypy. Push and open a PR only when asked. +6. After merge: delete `ts-ref` locally and on the fork. + +## Follow-ups + +- Fill in `Flopy3ListData` (empty stub in `flopy4/mf6/adapters.py`) over the + new Dataset: `to_array(kper, mask)` -> `{column: grid}` with + `duplicates="sum"`, extra dims flattened to `_1..n`; + `masked_4D_arrays_itr`; wire into `Flopy3Package`. See + `.claude/worktrees/list-to-xarray/209-ideas.txt`. +- `list-to-xarray` branch: `0348be2` is superseded by this work (drop it); + `99077f9` landed separately as branch `trailing-none-boundname`. + +- `attrs_to_dataset` still puts rows as objects into the dataset's attrs. +- Once TS evaluation exists, optionally fill evaluated values instead of NaN. + +## Status (2026-10-19) + +- **Merged as #476** ("support xarray conversion for list-based stress + packages"), from branch `period-lists`. #489 (`series-nan`) then made + time-series placeholders `NaN` in the xarray form, with the series name + in a `_ts` / `_tas` coordinate (`None` where a period has no + series), and fixed the netCDF round trip of those names. +- DISV's within-layer dim stays `cell`; the `disv-j` rename (#461) was + closed 2026-10-18. +- Still open, from the follow-ups above: + - `Flopy3ListData` in `flopy4/mf6/adapters.py` is still an empty stub. + - `attrs_to_dataset` still puts rows as objects into the dataset's + attrs. + - Lazy/periods-only storage (the dense form is eager). + - `auxiliary` as an int count when rows give aux values without names. + - Filling evaluated series values instead of NaN, once TS/TAS + evaluation exists ([`tas-interpolation.md`](../xarray/tas-interpolation.md)). +- Local cleanup (ask first): branches/worktrees `period-lists`, + `list-to-xarray`, `disv-j` are merged or closed. + +## Status (2026-10-13) + +- Branch `period-lists` (worktree `.claude/worktrees/period-lists`), local + only: `08aff59` (rebased e87c53e), `da83387` (DISV `j`), `b0dd085` + (rework). `disv-j` holds the rename alone, off develop. +- Done: steps 1-4 except lazy/periods-only storage (dense form is eager; + documented). Grid dim names in `constants.GRID_DIMS`. +- Fallback dim names where the DFN has no sizing dim: the key field + (LKT/LKE `ifno`/`lakeno`, SPC `bndno`) and setting indexes (`idv`, `icon`). +- Open: `auxiliary` becomes an + int count when rows give aux values without names. diff --git a/docs/dev/plans/archive/period-storage-plan.md b/docs/dev/plans/archive/period-storage-plan.md new file mode 100644 index 00000000..4b5dc308 --- /dev/null +++ b/docs/dev/plans/archive/period-storage-plan.md @@ -0,0 +1,223 @@ +# Per-period storage for period arrays + +Status: agreed (2026-10-05), against develop `82d2a99` (#406). Decisions are +recorded under "Decisions" below. + +Implemented (2026-10-06) on branch `period-storage`, commit `3e8d527`, steps +1–4 together (not pushed). It differs from the plan in a few ways: +- Conversion happens in `Package._init_period_arrays` (replacing + `_init_dynamically_named_arrays`), not as a per-field converter in codegen. + The aux checks need `self`. +- Each period's array is flattened to the field's DFN shape, as griddata is. + Before, a file loaded as `(nlay, ncpl)` while input stayed `(nodes,)`. +- Egress reshapes each period's array to the grid's layer shape. MF6's + period-block reader failed on a 225-value layer written on one line. +- `attrs_to_dataset` stacks the given periods along `_period`, so + cleared periods survive the xarray round trip. +- Side effect: `test019_VilhelmsenLGR{,_nr}` now load (they xpass). + +## Problem + +Period array fields (WELG `q`, RCHA `recharge`/`irch`, …) hold one dense +array with an implicit leading `nper` axis. Periods and cells with no value +are filled with `FILL_DNODATA` (3e30). That one value currently means three +things: + +1. **No stress in this cell.** This is MF6's own meaning: a grid package + activates the cells that aren't `DNODATA`. +2. **This period wasn't in the file** and carries forward. Ingress pre-fills + it (`structure.py:955`), and egress skips any period that's entirely + `FILL_DNODATA` (`unstructure.py:306`). +3. **Fill value for integer arrays.** `irch`/`ievt` are stored as floats so + they can hold 3e30, and the index conversion skips it (`unstructure.py:149`, + `netcdf.py:273`). + +Meanings 1 and 2 collide in grid packages: a period block that clears every +boundary can't be told apart from a period that isn't in the file. PR #380 +works around this with an undeclared `reset_periods` attribute. Meaning 3 is +the `int-fill` problem. + +## MF6 semantics (checked in `modflow6/src/Utilities/Idm/mf6blockfile`) + +- **Grid packages** (CHDG, DRNG, GHBG, RIVG, WELG; `Mf6FileGridArray.f90`): + - Every period block present in the file resets the boundary set + (`reset` sets `nbound = 0`). + - The block's first array decides which cells are active (the ones that + aren't `DNODATA`). Later arrays, including aux, are read at those cells. + - So a period is one unit: an empty block clears everything, and a + missing block keeps the previous period. +- **Layer-array packages** (RCHA, EVTA, SPCA; `Mf6FileLayerArray.f90`): + - A block resets only which arrays were read this period. Each array keeps + its previous values until it's given again. + - So carrying forward happens per array, and there's no "clear". + +## Representation + +Store each period array field as a dict keyed by 0-based stress period. The +dict holds only the periods the file gives: + +```python +q: Optional[dict[int, FloatArrayLike]] # {0: arr(nodes), 3: arr(nodes)} +irch: Optional[dict[int, IntArrayLike]] # stays int, no sentinel +aux: Optional[dict[int, dict[str, FloatArrayLike]]] # aux[kper][name] +``` + +- **Period is always the outer key**, for every period field including aux, + as in list packages' `stress_period_data[kper]`. +- **Missing period:** no key. The previous period's values carry forward. +- **Cleared period (grid packages):** a key whose stress arrays are entirely + `DNODATA`. Here 3e30 is MF6's value for "no boundary in this cell", not + something flopy4 invented. +- **Field `shape` metadata doesn't change.** It already leaves out `nper`; + only the declared type wraps it. +- **Precedents in develop:** + - List packages store `dict[int, list[StressPeriodData]]` (`wel.py:111`). + - TAS arrays are already a dict per header (Pass 3c, `structure.py:~985`). + - `array_eq` already compares dicts key by key. + - This brings array packages in line with all of these. + +Affected fields, 24 in all, all in `gwf/` and `utl/spca` (gwt/gwe grid +packages aren't generated yet): + +- `Optional[FloatArrayLike]` (15): `rivg.{stage,cond,rbot}`, + `rcha.recharge`, `evta.{surface,rate,depth}`, `welg.q`, + `drng.{elev,cond}`, `ghbg.{bhead,cond}`, `chdg.head`, + `spca.{concentration,temperature}` +- `Optional[IntArrayLike]` with `index` (2): `rcha.irch`, `evta.ievt` +- `Optional[dict[str, FloatArrayLike]]` aux (7, becoming + `dict[int, dict[str, FloatArrayLike]]`): `rivg`, `rcha`, `evta`, + `welg`, `drng`, `ghbg`, `chdg` + +## Input conversion + +The examples pass dense `(nper, …)` arrays (twri `chd_head`, frenchman-flat +`q_crt`, circle `head`) and rely on "a period that's entirely DNODATA means +nothing new". For example, twri sets `chd_head` only in period 0 and expects +it to carry forward. + +A converter on each field accepts: + +- `dict[int, array]`: kept as given. This is the canonical form and the only + way to write a clear explicitly. +- A dense `(nper, …)` array, split into periods. A period that's entirely + `DNODATA` is dropped, which keeps today's meaning so the examples behave + the same. Integer dense input keeps every period. Optionally, a period + identical to the one before can be dropped; that's equivalent under + carry-forward. + +This mirrors how #398 kept DISV's array-style inputs working. + +## Dense view + +Plotting, xarray, adapters and NetCDF want `(nper, …)`. Add one method on +`Package`, e.g. `period_array(name, nper=None, carry_forward=True)`: + +- **`carry_forward=True`:** each period holds the values in effect. Use this + for plots and analysis. Grid packages need care: a period's arrays replace + the previous period's as a whole, which a plain forward-fill per array does. +- **`carry_forward=False`:** missing periods get a fill value suited to the + dtype (`FILL_DNODATA` for floats, `FILL_INT64` for ints). This is what + NetCDF export needs; `netcdf.py:877` already uses `FILL_INT64` for + integer encodings. + +`nper` comes from the dims, the same as today's `_ArrayInfo`. + +## Changes by file + +**Codegen** (`utils/codegen/make.py` ~1314, ~1525) +- Wrap generated types for period arrays in `dict[int, …]`, and aux in + `dict[int, dict[str, …]]`. +- Attach the converter. +- Regenerate the 9 packages. + +**Package init** (`package.py`) +- Convert period arrays the same way `_init_dynamically_named_arrays` + handles aux. +- Check the aux names as before, now per period. +- Grid packages: check that aux's periods are a subset of the stress + fields' periods. + +**Ingress, Pass 3b** (`structure.py:921-971`) +- Drop `_empty` and the pre-fill. +- Each parsed period sets `field[kper] = arr` in its native dtype. +- Integer fields parse as `int64`. +- An empty grid-package block sets the key to an all-`DNODATA` array for + each stress field. +- This is also where PR #380's "unstored rows" (TAS references) go. See the + open questions. + +**Egress** (`unstructure.py:297-308`) +- Write one period block per key, across the union of keys from all the + package's period fields. +- Remove the skip for periods that are entirely `FILL_DNODATA`. +- A grid-package period whose stress arrays are entirely `DNODATA` is + written as an empty block. + +**Index conversion** (`unstructure.py:149`, `_from_file_index`, `netcdf.py:273`) +- Plain ±1 on integer arrays, with no sentinel check. + +**Writer filters** (`codec/writer/filters.py:197-205`) +- Recheck the hardcoded `FILL_DNODATA` mask. It should stay, but only for + meaning 1. + +**NetCDF** (`netcdf.py`) +- `_ArrayInfo` still declares a leading `nper`. +- Export builds data with `period_array(..., carry_forward=False)`. +- Import splits by the periods the ASCII file lists. + +**Adapters, xarray** (`adapters.py:296`, `attrs_xarray.py:102`, `package.py:500`) +- Use the dense view. + +**Constants** +- `FILL_DNODATA` is no longer needed to mark missing periods. It keeps only + MF6's meaning. + +**Tests** +- Update `test_rcha_irch_round_trip`: `irch` becomes an int dict with no + 3e30. +- Update the period-array tests in `test_mf6_codec.py`, `test_mf6_netcdf.py` + and `test_mf6_integration.py`. +- New tests: + - WELG round-trip with a cleared period + - RCHA with `irch` given only in period 0 + - dense input converted with all-DNODATA periods dropped + - the `period_array` views +- Run the docs examples, which use dense input. + +## Effect on PR #380 + +- `reset_periods` goes away. A cleared period is a key whose stress arrays + are entirely `DNODATA`. +- Storing integers as floats goes away, along with its cast. +- The rest of #380's period work has to be rebuilt on this model rather than + merged. + +## Decisions + +1. **Dense input rule: accepted.** In dense `(nper, …)` input, a period + that's entirely `DNODATA` means the period isn't given, so it carries + forward. Document this on the converter. Clearing a period needs the dict + form. +2. **Aux nesting: `aux[kper][name]`.** The period is the outer key for every + period field, so aux fits the same pattern, and a grid period reads as + one unit, as in MF6. +3. **Clear on write: empty block.** A grid-package period whose stress + arrays are entirely `DNODATA` is written as `BEGIN PERIOD k` / `END + PERIOD`. +4. **Key consistency: validate.** In grid packages, aux's periods must be a + subset of the stress fields' periods. The check raises at init. + +## Open, out of scope + +- **TAS references** (`RECHARGE TIMEARRAYSERIES RchSeries`; the + `test_tas_period_array_kept` xfail). A per-period value could later be an + array or a TAS name. This representation can hold that; a dense array + couldn't. + +## Order + +1. Converter and dense view on `Package`, with tests (no codegen change yet). +2. Codegen types and regeneration; Pass 3b; egress; index path. +3. NetCDF and adapters. +4. Examples and docs check; drop the `int-fill` item. +5. Then PR #380 rebases on top. diff --git a/docs/dev/plans/archive/plan.md b/docs/dev/plans/archive/plan.md new file mode 100644 index 00000000..7699b6d3 --- /dev/null +++ b/docs/dev/plans/archive/plan.md @@ -0,0 +1,181 @@ +# flopy4 development plan — Track A (input file loading) + +**Archived 2026-10-19.** Done: Track A and Track B are both complete. The two open rows are resolved: the codec reader is off the legacy DFNs (#455), and namefile grammars are generated (`sim-nam.lark`, `*-nam.lark`). Using them is part of [`typed-loader-wiring.md`](typed-loader-wiring.md). + +Track B (remove `xattree`, issue #167) is done — see "Track B: closed +out" near the bottom. This file now tracks Track A only. + +## Status snapshot (updated 2026-09-16) + +**Track A was "largely done" as of the last snapshot (2026-09-10); the +two items that were still blocking are now both done too.** Six commits +landed since then: `743ecf1`/#350 ("make Simulation.load() work, +mostly"), `9192136`/#351, `034afa1`/#352, `f3cd6f6`/#353, +`e3075da`/#354 (the last two delete `overrides.py`/ +`dfn_overrides.toml` entirely — the flopy4-side stopgap layer +[`dev3-migration-gaps.md`](dev3-migration-gaps.md) tracked is now gone, its upstream gaps fixed), +`5f1f4e2`/#355, and `67d0922`/#356 (Track B's xattree removal, which also +touched `structure.py`/`unstructure.py`). Of these, #350 is the one that +matters for Track A: it's what closed both previously-open items below. + +| Piece | State | +|---|---| +| One field-metadata convention (`block=`), `dfn_type` gone | ✅ #339 | +| `Column`/`Schema` deleted; `Row` → then `Item`/`Record` refactor; `row.py` removed | ✅ #341, #348, #349 | +| Codegen on pydantic-native `modflow_devtools.dfns` dev3 schema | ✅ #348 | +| OC rtypes / union arms read from schema, not hardcoded tables | ✅ #348 | +| Generic structuring + binding resolution; `Simulation.load()`/`Gwf.load()` recurse, dims threaded from `dis` sibling | ✅ #348 | +| Typed union-item classes, top-level period unions (`item.py`, `dispatch_union_item`) | ✅ #349 | +| Flopy4-side DFN-override stopgap layer (`overrides.py`/`dfn_overrides.toml`) | ✅ deleted, #353/#354 — upstream devtools/mf6 fixes made it unnecessary | +| **Prove `Simulation.load()` end-to-end on real input** | ✅ #350 — see below, far exceeded original scope | +| **Close out PR #284** | ✅ closed (not merged) 2026-09-10; useful pieces salvaged into #350 | +| Codec reader path off legacy `modflow_devtools.dfn` | ❌ not started | +| Nested `ocsetting` union (OC) | ✅ done — [`nested-keystring-union-plan.md`](nested-keystring-union-plan.md) | +| Typed namefile grammar (Phase 3) | ❌ not started | + +**Test state (reverified after the `ocsetting` change, same session):** +pixi `dev` env is green — `910 passed, 11 skipped, 4 xfailed` (908 base + +2 new tests). Three of the four xfails are the corpus-survey +fixture-quality gaps below (`test019_VilhelmsenLGR`, `_nr`, +`test060_gms_ets`); the fourth (`test_mf6_adapters.py::test_fails`) is an +intentional demonstration test, not a gap. + +## Prove `Simulation.load()` end-to-end — done, exceeded scope + +The original plan (2026-09-10 snapshot) called for running +`Simulation.load()` against `docs/examples/twri.py` and +`frenchman-flat.py`. What actually landed in #350 is stronger: a +**242-model corpus survey** against the real `modflow_devtools.models` +registry (`test/mf6/test_mf6_load_all_models.py`), not flopy4's own +hand-written examples. A first pass (2026-09-11) found ~9% loading +cleanly; a same-day debugging session (documented blow-by-blow in +[`load-corpus-gaps.md`](load-corpus-gaps.md), 13 numbered root causes, 11 fixed) took the pass +rate to **239/242 (98.8%)**, including the registry's own `test021_twri` +model — a real MF6 reference input, a stronger proof than the +hand-written `docs/examples/twri.py` would have been (which itself +remains an unexercised example script, not wired into any test). The +`_param()` helper in that test file marks every model not in +`KNOWN_PASSING` `xfail(strict=False)`, so a regression is a hard +failure and a newly-fixed model shows up as XPASS — the signal to +promote it. + +The remaining 3 xfails are confirmed, documented, out-of-scope fixture +issues, not flopy4 bugs ([`load-corpus-gaps.md`](load-corpus-gaps.md) gaps #11-#12): +`test060_gms_ets`'s DFN fixture omits a field the current spec genuinely +requires; `test019_VilhelmsenLGR`(`_nr`)'s child model references an +external array file sized for the *parent* grid (26901 values into a +7020-shaped slot) — a stale/mismatched fixture pairing, confirmed by +reading the referenced file directly. + +DIS-before-NPF/IC ordering and dims propagation (the two things the old +plan wanted separately confirmed) are implicitly exercised across this +entire corpus, not just a two-package case — no separate verification +needed. `test/test_package_load.py` (410 lines, also from #350) covers +package-level loading in more depth. + +## Close out PR #284 — done + +PR #284 ("input file loading") was closed, not merged, 2026-09-10 — the +right outcome per the original plan (its `structure.py` was superseded; +only the docs and the `FTYPES`/binding-resolution design were worth +keeping). #350's own description confirms the salvage: "this PR +salvages some docs and tests loading the examples and test models." +Concretely, `docs/dev/sdd.md`, `map.md`, and `test.md` were updated in +#350. [`on-representations.md`](on-representations.md) and [`typed-grammar-issues.md`](typed-grammar-issues.md) — the +other two files the original plan named — never existed outside the +closed PR #284 branch and weren't carried forward; nothing currently on +`develop` references them. No further action needed here. + +## A3 — remaining, independent, non-blocking items + +All three verified unstarted directly against current `develop` (no +progress since the 2026-09-10 snapshot, despite six intervening +commits — none of #351-#356 touched this scope): + +- **Codec reader path off legacy `modflow_devtools.dfn`.** Confirmed via + grep: `flopy4/mf6/spec.py`, `flopy4/mf6/codec/filters.py`, + `flopy4/mf6/codec/reader/dfn2lark.py`, + `flopy4/mf6/codec/reader/grammar/__init__.py`, + `flopy4/mf6/codec/reader/grammar/filters.py`, and + `flopy4/mf6/codec/reader/transformer/typed.py` all still import + `modflow_devtools.dfn`/`modflow_devtools.dfn.schema` (the legacy flat + schema), not `.dfns`. (`flopy4/cli.py` and two test files, + `test_mf6_grammar_gen.py`/`test_mf6_reader.py`, also still use the + legacy module — same migration would need to reach them too.) Same + tree-walk conversion already done for codegen (`6cdfb2a`); + `ComponentBase.get_block()` covers the block reverse-lookup. Own + `schema_version` literal, no ordering dependency on anything else here. +- **Nested `ocsetting` union — done.** `OC`'s `Save`/`Print.ocsetting` + (all 4 model families: gwf/gwt/gwe/prt) is now + `Oc.All | Oc.First | Oc.Last | Oc.Frequency | Oc.Steps`, matching + PRP's already-typed `releasesetting`. Full detail, including two design + corrections found during review (a `top_level` flag so nested arms + don't leak into the outer dispatch union; a dedup cache since + `Save.ocsetting`/`Print.ocsetting` are independent-but-identical + `Union` objects) in [`nested-keystring-union-plan.md`](nested-keystring-union-plan.md). Disclosed + behavior change landed as planned: `ocsetting` is now required, no + default; one downstream break found and fixed + (`docs/examples/quickstart.py`'s `== ("ALL",)` comparison). +- **Typed namefile grammar** ([`mf6-object-model-plan.md`](mf6-object-model-plan.md) Phase 3). + `basic.lark` still parses `packages`/`models`/`exchanges`/ + `solutiongroup` into generic rows — confirmed no `packages_record`- + style typed rule exists anywhere in the grammar. Note: + `get_typed_parser()`/`typed.lark` *do* exist and are exercised in + `test_mf6_reader.py`, but for per-package block grammars (`gwf-ic`, + `gwf-wel`, `gwf-oc`, `gwf-dis`, `gwf-npf`, `gwf-sto`) — an unrelated, + already-landed mechanism, not the namefile-row typing this item is + about. Porting PR #284's typed per-row grammar here is a + validation/error-message improvement only, not a prerequisite for + anything currently working. + +No choreography needed between these three or against anything else — +all independent, land in any order, any time. + +## Track B: closed out + +`xattree` removal (issue #167's runtime-object-model half) merged into +`develop` via `67d0922` ("drop xattree (#356)", 2026-09-16). Verified +directly: zero `xattree` imports/decorator usage anywhere in source, no +`xattree` in `pyproject.toml`/`pixi.lock`, full suite green (908 passed). + +What shipped: `flopy4/protocols.py` (`DatasetConvertible`/ +`DataTreeConvertible`), `flopy4/mixins.py`, `flopy4/attrs_xarray.py` (the +generic attrs↔xarray conversion layer), explicit parent/child +attach/naming built directly into `flopy4/mf6/component.py` +(`_children`/`_parent`/`_resolve_child_name`/`_attach_to_parent_field`/ +`_set_child_parents`), `@xattree` → `@attrs.define(kw_only=True, +slots=False)` on all 13 decorated classes, and `Component.pname`/ +`Component.strict` deleted (both were workarounds the decorator's +removal made obsolete). + +One deviation worth remembering: [`xattree-child-naming-plan.md`](xattree-child-naming-plan.md) had +scoped the child-naming consolidation as **upstream** work in the +`xattree` repo itself, gating flopy4's side on that fix landing and +being pulled in. That never happened — the equivalent logic was instead +built directly in `flopy4/mf6/component.py` as part of the final +cutover, which is why that plan is now superseded rather than completed +as written. + +A follow-up pass (2026-09-16, same day) removed the loose ends the +xattree-era code/docs/tests still carried after the decorator itself was +gone: two dead functions in `flopy4/mf6/spec.py` +(`get_field_type`/`to_field`) that read a `"xattree"` metadata key +nothing sets anymore, and every remaining comment/docstring across +source, tests, and `docs/` that named "xattree" for historical +explanation rather than describing current behavior. Full suite +reverified green after that pass too. + +Full design history: [`xattree-removal-plan.md`](xattree-removal-plan.md) (now a short "Final +status" note), [`protocols-plan.md`](protocols-plan.md) (dimension-resolution design, Phases +1-2 of which shipped as designed), [`xattree-child-naming-plan.md`](xattree-child-naming-plan.md) (now +superseded, per above). + +## Housekeeping / loose ends + +- The repo-root `*.md` plan files are untracked. Not versioning them per + your call — but they're the design record this plan defers to, so keep + them somewhere durable. +- [`nested-keystring-union-plan.md`](nested-keystring-union-plan.md) references `namefile-load-plan.md` and + [`mf6-object-model-plan.md`](mf6-object-model-plan.md) interchangeably — same doc, two names in + different comments (`filters.py:11` says `namefile-load-plan.md`). Pick + one name if you touch it. diff --git a/docs/dev/plans/archive/polymorphism-hypotheses.md b/docs/dev/plans/archive/polymorphism-hypotheses.md new file mode 100644 index 00000000..3623b98f --- /dev/null +++ b/docs/dev/plans/archive/polymorphism-hypotheses.md @@ -0,0 +1,355 @@ +# #287: where the contours point + +Working notes, 2026-11-02. Not for posting as is. Builds on +`discussion-287-comment.md` (draft), `discussion-relational-spec-draft.md`, +[`schema-polymorphism.md`](../dfn-spec/schema-polymorphism.md), [`reference-fields-notes.md`](../archive/reference-fields-notes.md) and devtools +`relational-model-plan.md`. + +The draft sorts what we've found into three kinds of polymorphism: sum types +(variants), interfaces (roles) and maybe parametric (grids), plus rules for +name resolution. Taking each a step further, three hypotheses come out. They +reinforce each other, and each changes something in the tentative designs. + +1. **Almost every sum type in MF6 input is a layout choice, not a logical + one.** The logical layer may need no sum types beyond enums and the grid + type. +2. **List and array variants differ in whether bounds have identity** + (entities vs. fields). That difference, not the variant itself, is what + leaks into keys. Key *types* on roles handle it without declaring + variants. +3. **A component is parameterized by its binding context.** Rules are + mostly formal parameters that the binding row fills in. This collapses + most rules, CellID `model = "1"/"2"`, OBS, SPC, GNC and grid dependence + into one construct that many components would use. + +--- + +## 1. Most sum types are layout + +### Tags sit at every level of nesting + +Every variant we've found is chosen by a tag. Where the tag sits: + +| Level | Tag | Chooses | Logical or layout? | +|---|---|---|---| +| binding row | ftype (`CHD6` vs `CHDG6`) | component variant | layout | +| file options | `READASARRAYS` | component variant (RCH vs RCHA) | layout | +| field | `CONSTANT` / `INTERNAL` / `OPEN/CLOSE` / `NETCDF`, `LAYERED` | array encoding / data source | layout | +| row | union keyword (`STATUS`, `RATE`, `DIVERSION`, ...) | which attribute a row sets | layout (see below) | +| model | discretization type (DIS / DISV / DISU) | the grid's domain | **logical** | + +The array control records are a sum type flopy4 already handles without +ever calling it polymorphism. `NETCDF` belongs in the same row. It's another +answer to "where are this array's values?", not a schema condition. That +corrects the draft, which puts `NETCDF` with `when` and MOVER as +option-conditional schema. + +### Union records are attribute updates + +SFR's period block is a log of updates to reach attributes. `ifno STATUS +INACTIVE` and `ifno RATE 0.1` set two columns of one logical row, +`(reach, kper) → status, rate, stage, ...`. The keyword names the column. +That's entity-attribute-value encoding of a sparse wide table. The arms with +their own key (`DIVERSION idv divflow`) set columns of a child table keyed +by `(reach, idv)`. + +So the relational plan's "union record convention" question (variant tables, +sparse columns or EAV) has an answer. Logically the columns are sparse. In +layout the file uses EAV, and the keyword is the attribute name. Union +records then aren't a logical sum type at all, and "databases don't have sum +types" stops being a gap. + +This needs checking per case. OC's `SAVE HEAD ALL` / `PRINT BUDGET STEPS 1 3` +reads more like `(action, rtype) → step selection`, which still fits a +table. MAW `connstatus` and the LAK/UZF period settings look like SFR. A +counterexample would be an arm whose fields don't belong to one table. + +### Periods are the same pattern + +MF6 carries a period block forward until the next one. That's a delta +encoding of a step function over periods. The relational plan's "sparse +plus fill rule vs. dense" question is the same as for union records: the +logical layer holds the state (a step function), and the layout writes +changes. Union records and period blocks are both **deltas in layout over +state in logic**. One convention can cover both. + +### Logical sum types left over + +- Enums (`status`, `connection type`, ...). These are plain column domains. +- The grid type, which changes the domain every array is indexed by. + Section 3 treats that as a parameter, not a variant. + +If the DDL slice confirms this, the logical layer needs no sum types, and +the list of candidate languages gets simpler. SQL DDL, Frictionless and our +own TOML all become viable. + +### What "functionally equivalent" means + +#287 calls RCH and RCHA functionally equivalent. That's only partly true, +and saying how makes it useful: + +- RCHA → RCH always works. Each column's value becomes a bound at the + `irch` layer. +- RCH → RCHA works only when each column has at most one entry, all on one + layer per column, with no boundnames, no per-bound aux the array can't + hold, and nothing referring to bound numbers. MF6 lists allow several + entries in one cell. +- EVT → EVTA also needs `nseg = 1`, since EVTA has no segments (check). + +Equivalence holds for the hydrologic effect, the source term per cell per +period. It doesn't hold for *identity*: bound numbers, boundnames, +observations, movers and SPC all refer to bounds. In lens terms, each layout +is a partial lens onto one logical relation. Conversion goes through the +relation and is defined where both lenses are. flopy4 can implement it once, +not once per variant pair, and its error messages fall out of the +conditions above. + +--- + +## 2. Entities vs. fields + +The list/array split is really this: + +- **List variant: a collection of entities.** Each bound has identity (its + number, optionally a boundname). It can be named, observed, moved, or + given concentrations by SPC. +- **Array variant: a field over the grid.** A value per column (or per + cell). Its only identity is the cell. + +Roles attach to identity, so RCH has `bound` and RCHA doesn't. But SPCA +still refers to RCHA, through the grid domain: an SPCA array is keyed by +column, the same as RCHA's. So **both SPC and SPCA are "keyed by the +target's key"**. They differ only in what that key is: + +| Referrer | Target | Shared key | +|---|---|---| +| SPC `bndno` | RCH, WEL, SFR, LAK, ... | the target's bound number (`<=maxbound`, `ifno`, `<=nlakeconn`, ...) | +| SPCA arrays | RCHA, EVTA | the target's column (`ncpl`) | + +In PL terms, a role is a trait, and its key is an **associated type**. Each +`[keys]` entry is a trait impl that states the key path. If the role also +states what kind of key it is (bound number vs. grid column), then +"SPCA binds only to array variants" is just a key-type mismatch, caught by +the ordinary check. **No `variant_of` is needed for checking.** + +`variant_of` (or `layout_of`) could still help docs group the alternatives, +but it no longer carries validation. + +This also says where roles should be *defined*. Today a role exists only +through its impls. Target-side declaration is what keeps the set open +(Haskell instances and Rust impls work the same way), but the trait itself +needs one home: its key kind, who may refer to it, and its prose. That +answers the relational plan's open question about first-class roles: yes, a +central `[roles.*]` declaration, with per-component `[keys]` as impls. + +--- + +## 3. Components are parameterized by their binding + +### Rules as parameters + +The draft calls rules "closer to dynamic scope". A better name: **a file is +a parameterized component, and the row that binds it supplies the +arguments.** That's an ML functor or a constructor's dependency injection. +Once a parameter is declared, a reference to it is lexical again. + +| Component | Parameter | Supplied by | Today | +|---|---|---|---| +| `utl-spc` / `utl-spca` | `target`: a package playing `bound` | SSM `fileinput` row (`pname`) | rule `ssm_package` | +| `gwf-mvr` | `models`: one or two models | model namefile (this model) or exchange (`exgmnamea`, `exgmnameb`) | rule `exchange_models` | +| `gwf-gnc` | `models`, as for MVR | model or exchange | CellID `model = "1"/"2"` | +| exchange cellids | `models` | the exchange's own name columns | CellID `model = "1"/"2"` | +| `utl-obs` | `target`: its parent package | the package's `OBS6 FILEIN` | none (see below) | +| `utl-ts` / `utl-tas` | `target`: its parent package | the package's `TS6` / `TAS6` | none | +| every package | `grid`: its model's discretization | the model | implicit in cellid encoding | + +Every child file is in this table. That clears the bar #405 set for +syntax: a construct used by enough fields that a second consumer could +implement it from the spec alone. Rules failed that test one at a time. +Together, as parameters, they pass it. + +It also matches the relational plan's observation #1, that "a rule is a +parent key or join the format leaves out". The parameter is that omitted +parent key column. The PL framing and the relational framing reach the same +place from opposite sides, which is some evidence the concept is real. + +### OBS: all three at once + +OBS is the best test case, and it isn't on our list yet: + +- **Parameter.** An OBS file belongs to the package that binds it. +- **Associated items.** The valid `obstype`s depend on the target type, + which is the `observations[obstype]` pattern: another trait with + per-package impls. +- **Sum type.** `id` is a union (`cellid | index | boundname`). Which arm + applies depends on the `obstype` in the same row, and the `boundname` arm + on the *target's* `BOUNDNAMES` option, which is a cross-file `when`. + +If one design handles OBS cleanly, it'll handle MVR and SPC. + +### What stays a rule + +- **`flow_model`** (SSM `pname`). The GWT model's flow model isn't supplied + by any binding row of SSM. It comes from a GWF-GWT exchange that names + the model, or from FMI's `GWFBUDGET` file when there's no flow model in + the simulation. In that case SSM's `pname` should match a package name in + the budget file (to check against MF6). That's a genuine view, and it has + its own sum type (exchange or file). It stays a named rule. +- **`mover_enabled`** becomes `when` on the key set (#407). It's an + option-based refinement of the role, not a context. + +So the rule registry would shrink from four entries to one. + +### Grid as parameter: re-encoding vs. regridding + +Treating the grid as a parameter sorts #287's mesh example: + +- **Re-encoding** keeps the domain and changes how cells are named. DIS + cellid `(k, i, j)` ↔ node, or a DIS model's NetCDF written as structured + vs. UGRID layered mesh. That's a bijection, it's lossless, and it's + layout. It's mechanical, so flopy4 can do it in general. +- **Regridding** changes the domain (DIS → an arbitrary DISV). That's + interpolation, not schema, and it's out of scope for the spec. + +#287's "converting a structured model to a UGRID mesh representation" +is mostly the first kind. Under this framing it's cheap. + +--- + +## How the tentative designs change + +| Draft says | Revised | +|---|---| +| Component variants are sum types; flopy4 uses discriminated unions | They're layouts of one relation, related by partial lenses. Keep separate classes (users write `Rcha`), discriminate at load (ftype + `READASARRAYS`), and add conversion through the logical relation. Discriminated unions are for typed slots and the loader, not the object model at large. | +| Declare `variant_of` to check SPCA → RCHA | Key kinds on roles check it. `variant_of` / `layout_of` is optional and only for docs. | +| Rules are "dynamic scope" | They're parameters filled by the binding. Lexical, given the declaration. | +| `when`, MOVER, `READASARRAYS` and `NETCDF` should share one vocabulary | `when` and MOVER yes. `READASARRAYS` and `NETCDF` are layout tags, in the same family as `CONSTANT` / `INTERNAL`. Different construct. | +| Union records have no relational equivalent | Sparse columns with EAV layout. Same pattern as period blocks: deltas over state. | +| Grid: per-component variant or model parameter? | Model parameter. Conversions are re-encodings (layout) or regridding (out of scope). | + +A side question this raises: options themselves split into **layout +flags** (`READASARRAYS`, the `LAYERED` and `NETCDF` tags) and **logical +settings** (`MOVER`, `SURF_RATE_SPECIFIED`, `FIXED_CELL`). This is too +coarse. `PRINT_INPUT` is an output directive, and `BOUNDNAMES` declares +identity (schema). See [`option-classification.md`](../dfn-spec/option-classification.md) for the full +categories. `when` can test either kind, but the spec should say which +kind each option is. Field `when` on a layout flag means a column that's +always there logically but only written sometimes. On a logical setting, it +means a column that only exists sometimes. That settles the relational +plan's "is `when` logical or layout?": it depends on the option. + +--- + +## Plan + +In order. Each step answers a question and is cheap to abandon. + +1. **Classify options (desk exercise, devtools).** Script out every option + in the DFNs, then label each as layout, logical or both by hand. + Detailed in [`option-classification.md`](../dfn-spec/option-classification.md) (categories, decision test, + failure threshold, memory catalog coordination). + *Answers:* whether the layout/logical split is clean at the option level. + Wherever "both" shows up often, the two-layer idea is in trouble. + **Done (2026-11-03), see [`option-classification.md`](../dfn-spec/option-classification.md): the split + holds.** Of 224 names, 20 are layout, 121 logical, 82 output, and 1 mixed + (CSUB `HEAD_BASED`). Layout lenses come in four kinds: syntactic, + cross-component, computed and partial. The RCHA partial lens above is one + of them. + **Derivation axis.** The `AUXMULTNAME` borderline case showed that the + re-encoding test was underspecified. It can mean the same values now, or + the same behaviour under change. Options that declare a rule ("this + quantity is computed from those") are now marked `none`, `static` or + `dynamic`: + - No layout option is dynamic. + - 14 of 20 layout options are static derivations. + - `AUXMULTNAME` is dynamic. MF6 applies the multiplier at each use, + against an aux value that can be a TS. That makes it logical, alongside + `AUXDEPTHNAME`, TS/TAS and `LOCAL_Z`. Criterion 2 holds. + + Consequence: the logical layer needs **derived attributes** (generated + columns), and flopy4 should keep dynamic derivations unmaterialised. +1b. **Constraint audit (desk exercise).** **Done (2026-11-03), see + [`constraint-audit.md`](../dfn-spec/constraint-audit.md).** + - The layering holds for constraints. The only layout-to-logical crossings + are lens-domain conditions (new kind L), which MF6 enforces and which + are step 5's partiality conditions. + - `when` and binding parameters fit one form: a predicate resolved in a + scope chain (own options → model → binding). + - `MODELNAMES` is a binding parameter written as an option. + - The DFN expresses 46 of 166 sampled constraints at least partly. It + misses numeric domains, option coherence, co-occurrence and the grid + joins. + + The original plan for this step: + Make a taxonomy of + the constraints a valid simulation must satisfy (K1 format through K10 + structural/physical). For each kind, record its level, its DFN coverage + and the DDL construct it would need. Sources: DFN keys, MF6 + `store_error` sites (sampled) and mf6io prose. Detailed in + [`constraint-audit.md`](../dfn-spec/constraint-audit.md). + *Answers:* whether constraints respect the layering the way options do; + whether `when` and binding parameters are one construct (feeds step 3); + and what step 2's DDL has to express. + It also adds a second axis that crosses layout/logical: + **configuration vs. data**. + - Options and dimensions are configuration, which maps to DDL schema. + - Arrays and rows are data, which maps to table contents. + - The weak spots sit where the two meet: `MAXBOUND`/`NODES` (configuration + derived from data) and `IDOMAIN` (data that acts as structure). + `AUXMULTNAME` was on this list, but it's a derivation, not a mix of + configuration and data (see step 1). +2. **DDL slice (already planned, sharper verdict).** Add OBS to the slice, + for one package. Test the hypotheses directly: + - H1: no logical sum types except enums and the grid. + - H2: union records and period blocks are both sparse tables plus deltas. + - H3: every rule except `flow_model` is a parent key column, i.e. a + parameter. + - H4: SPC/SPCA pairing is a key-kind match. + - H5: dynamic derivations are generated columns. Model one boundary + package with `AUXMULTNAME`. +3. **Parameter syntax sketch (devtools, paper only).** Write `[params]` + (name open) for SPC, MVR, GNC, OBS and TS, and write how binding rows + supply them. Then rewrite `ssm_package`, `exchange_models` and CellID + `model` in it. Keep it if the result is smaller than the rules plus + CellID `model`. Drop it otherwise. + From 1b, also: + - make parameters names that `when` can test; + - replace MVR `MODELNAMES` with the `models` parameter; + - check that the K8a co-occurrence rules (MVR ⇒ MVT, LKT ⇒ LAK) can be + written in the same form. +4. **Binding-graph navigation (flopy4).** This is the "when package + references come" revisit in [`reference-fields-notes.md`](../archive/reference-fields-notes.md). Add explicit + navigation (`Spc.target`, `Mvr.models`, `Obs.target`, `Model.flow_model`), + implemented once over the binding graph. Then have `keys.py` and + reference fields both use it. It's worth doing whatever happens in the + spec, because both already need it. +5. **One conversion end to end (flopy4).** RCHA ↔ RCH through an explicit + intermediate (per-cell rate per period). Report partiality with the + conditions from section 1. If that's clean, EVT ↔ EVTA and CHD ↔ CHDG + follow. If not, we learn where "one logical relation" breaks. +6. **Then post.** Revise the #287 comment with whatever survives steps 1 to + 3. Probably shorter: the taxonomy, the three hypotheses, and the plan. + The relational draft stays a separate discussion. + +## What would change our minds + +- An option that's irreducibly both layout and logical, used widely + (step 1). +- A union record whose arms don't fit one table (step 2). +- A child file whose context isn't supplied by its binding row, beyond + `flow_model` (step 3). Each one is another rule, and if there are enough + of them, the parameter construct isn't paying for itself. +- Conversion needing per-pair special cases beyond the stated conditions + (step 5). +- Many constraints of a logical kind triggered by layout options + (step 1b). These would be the constraint-level counterpart of mixed + options. + +## To check against MF6 + +- Whether list RCH/EVT allow several entries in one cell (assumed yes, as + for WEL). +- Whether EVTA lacks segments. +- What SSM `pname` refers to when FMI reads flow from a `GWFBUDGET` file. +- Whether OBS `id`'s `boundname` arm depends only on the target's + `BOUNDNAMES`. diff --git a/docs/dev/plans/archive/protocols-plan.md b/docs/dev/plans/archive/protocols-plan.md new file mode 100644 index 00000000..0a43cf80 --- /dev/null +++ b/docs/dev/plans/archive/protocols-plan.md @@ -0,0 +1,44 @@ +# Dimension Resolution Refactor Plan + +**Status: Phases 1-2 (dimension resolution protocols/mixin) shipped +largely as designed below. Everything past that — removing `@xattree` +from `Component`, replacing its parent management, xarray conversion — +was superseded by [`xattree-removal-plan.md`](xattree-removal-plan.md), which sequenced it into +concrete sub-phases and landed all of them (merged `develop`@`67d0922`, +2026-09-16). See [`plan.md`](plan.md)'s "Track B: closed out" for the current +summary.** + +## What shipped from this design + +`flopy4/mf6/dimensions.py`: `DimensionProvider`/`DimensionResolver` +protocols (renamed from this doc's original `DimensionRegistry`), +`DimensionResolverMixin` (renamed from `DimensionRegistryMixin`), lazy +runtime resolution by walking the object graph on-demand (no +pre-registration, no global state), caching after first resolution, +construction-order-agnostic (supports both top-down and bottom-up +construction). `resolve_dims()` unifies what were originally two +methods (`resolve_dimension()`/`get_all_dimensions()`) into one: +`resolve_dims()` for everything available, `resolve_dims('nlay')` for +one. `Dis`/`Disv`/`Tdis` implement `DimensionProvider` for both explicit +and derived dimensions; `Component` implements `DimensionResolver`, so +every component can register/resolve dimensions uniformly. +`validate_dimension_resolution()` exists as an optional test/CI +validation tool. + +This design's own migration section anticipated the parent-management +handoff happening incrementally, with `Component` keeping `@xattree` +"at first." In practice the actual cutover happened all at once in +[`xattree-removal-plan.md`](xattree-removal-plan.md)'s Phase 3d/3e, not incrementally as sketched +here — by the time `dimensions.py`'s `.parent`-walk needed to be +xattree-independent, the whole decorator was being removed in the same +pass rather than swapped underneath it first. + +## Original design notes (kept for context) + +Lazy resolution order: check cache → walk children for +`DimensionProvider`s → delegate to parent if not found → cache result. +Framework-agnostic by construction (works with attrs, pydantic, +dataclasses) since it depends only on `.metadata`/`.parent`, not a +specific class-decorator mechanism. Conflict detection: eager, raises on +multiple providers declaring the same dimension at one level; children +may still override a parent's dimension of the same name. diff --git a/docs/dev/plans/archive/pydantic-prototype-summary.md b/docs/dev/plans/archive/pydantic-prototype-summary.md new file mode 100644 index 00000000..2b0bb220 --- /dev/null +++ b/docs/dev/plans/archive/pydantic-prototype-summary.md @@ -0,0 +1,340 @@ +# Pydantic Prototype Summary + +## Answers to Your Specific Questions + +### 1. Can we type hint array dtypes with `NDArray[...]`? + +**YES!** The prototype demonstrates this works perfectly: + +```python +class Npf(MF6Base): + icelltype: Annotated[ + NDArray[np.int64], # ← Precise dtype in type hint! + MF6Array(dims=("nodes",), block="griddata", netcdf=True) + ] + + k: Annotated[ + NDArray[np.float64], # ← Works great! + MF6Array(dims=("nodes",), block="griddata", netcdf=True) + ] +``` + +**Output from prototype:** +``` +NPF.k shape: (300,) +NPF.k dtype: float64 # ← Correctly typed as float64! +``` + +Type checkers (mypy, pyright) will understand these hints and validate usage. + +### 2. What does `model_rebuild()` do? + +See [`model-rebuild-explained.md`](model-rebuild-explained.md) for the full explanation, but in summary: + +- **Purpose:** Resolves forward references in circular type dependencies +- **When needed:** Only when classes reference each other before they're defined +- **Solution:** Use `from __future__ import annotations` to avoid it entirely (recommended) + +**Example:** +```python +from __future__ import annotations # ← Avoids model_rebuild() + +class Package(BaseModel): + parent: Optional[MF6Model] = None # ← No quotes needed! + +class MF6Model(BaseModel): + npf: Optional[Npf] = None + +# No rebuild needed - Python handles it automatically +``` + +### 3. Can we alias pydantic's "model" terminology? + +**YES!** The prototype includes a clean aliasing pattern: + +```python +class MF6Base(BaseModel): + """Base class with MF6-friendly method names.""" + + @classmethod + def get_fields(cls): + """Alias for model_fields.""" + return cls.model_fields + + @classmethod + def get_schema(cls): + """Alias for model_json_schema.""" + return cls.model_json_schema() + + def to_dict(self): + """Alias for model_dump.""" + return self.model_dump() + + @classmethod + def from_dict(cls, data: dict, **kwargs): + """Alias for model_validate.""" + return cls.model_validate(data, **kwargs) +``` + +**Usage:** +```python +class GwfModel(MF6Base): # ← "Model" means MF6 model + pass + +# Clean API - no confusion! +schema = GwfModel.get_schema() # Not model_json_schema() +data = gwf.to_dict() # Not model_dump() +``` + +## What the Prototype Demonstrates + +### ✅ Fully Compatible with #167 Refactor Design + +1. **Protocols work perfectly** - DimensionProvider, DatasetConvertible +2. **Parent-child relationships** - Set via `model_post_init()` +3. **Dimension resolution** - Via parent chain walking +4. **Explicit conversion methods** - `.to_dataset()`, `.to_datatree()` +5. **No inheritance conflicts** - Protocols are for type checking only + +### ✅ Implicit Validation on Assignment (Your Requirement!) + +```python +# Validation runs automatically on assignment +dis.nlay = -1 # ❌ Raises ValidationError +npf.k = "invalid" # ❌ Raises ValidationError +``` + +**Output from prototype:** +``` +2. Testing implicit validation on assignment... + OK Validation caught error: ValidationError +``` + +This is enabled by `validate_assignment=True` in ConfigDict. + +### ✅ Array Fields Can Be xarray.DataArray + +```python +class Dis(MF6Base): + delr: Annotated[ + xr.DataArray, # ← Fields are DataArrays! + MF6Array(dims=("ncol",), block="griddata") + ] +``` + +**Output:** +``` +delr type: , shape: (10,) +``` + +Arrays are validated and structured via `@field_validator`. + +### ✅ Complex Array Structuring Works + +The prototype includes a simplified version of your `structure_array` logic: + +```python +@field_validator("icelltype", "k", mode="before") +@classmethod +def structure_arrays(cls, v: Any, info: ValidationInfo) -> NDArray: + """Structure arrays from various input formats.""" + # Access metadata + field_info = cls.model_fields[info.field_name] + metadata = field_info.json_schema_extra + + # Get dimensions from context + dim_values = info.context.get("dims", {}) + + # Handle different input types + if isinstance(v, np.ndarray): ... + if isinstance(v, (int, float)): ... # Broadcast scalar + if isinstance(v, dict): ... # Cellid format + + return structured_array +``` + +Your full 500-line converter logic can be ported to validators. + +### ✅ Schema-First Design + +Metadata stored in `json_schema_extra` is **schema-native**: + +```python +k: Annotated[ + NDArray[np.float64], + Field(json_schema_extra={ + 'dims': ['nodes'], + 'block': 'griddata', + 'netcdf': True, + 'longname': 'hydraulic conductivity' + }) +] +``` + +This makes JSON Schema generation automatic (with some caveats for numpy types). + +### ✅ Dimension Resolution + +Works exactly like the #167 design: + +```python +class DimensionRegistry: + def resolve(self, name: str, node: Any) -> int: + """Walk up parent chain to resolve dimension.""" + current = node + while current is not None: + if isinstance(current, DimensionProvider): + dims = current.get_dimensions() + if name in dims: + return dims[name] + current = getattr(current, "parent", None) + raise ValueError(f"Dimension '{name}' not found") +``` + +## Prototype Output + +``` +====================================================================== +Pydantic-based MF6 Data Model Prototype +====================================================================== + +1. Creating DIS package... + DIS dimensions: {'nlay': 3, 'nrow': 10, 'ncol': 10, 'nodes': 300} + delr type: , shape: (10,) + OK Arrays are xr.DataArray + +2. Testing implicit validation on assignment... + OK Validation caught error: ValidationError + +3. Creating NPF package with dimension resolution... + NPF.k shape: (300,) + NPF.k dtype: float64 + OK Dimension context resolved from parent + +4. Creating GWF model... + Model dimensions: {'nlay': 3, 'nrow': 10, 'ncol': 10, 'nodes': 300} + NPF parent: mymodel + OK Parent references set automatically + +5. Generating JSON Schema... + OK Schema generation path demonstrated + +6. Serialization... + OK Round-trip serialization works + +7. Converting to DataTree... + OK Hierarchical xarray structure + +====================================================================== +Prototype demonstrates: + OK NDArray[dtype] type hints work + OK Implicit validation on assignment + OK Dimension resolution via parent chain + OK Complex array structuring in validators + OK JSON Schema generation (schema-first) + OK Compatible with #167 refactor design +====================================================================== +``` + +## Key Architectural Points + +### 1. **Validators Replace Converters** + +**Current (attrs + cattrs):** +```python +# Converter registered globally +converter.register_structure_hook( + Package, + lambda data, cls: structure_package(data) +) +``` + +**Pydantic:** +```python +# Validator is a class method +class Package(MF6Base): + @field_validator("k", mode="before") + @classmethod + def structure_k(cls, v, info): + return structure_array(v, info.context) +``` + +**Better encapsulation** - logic lives with the class. + +### 2. **Metadata is Schema-Native** + +**Current (attrs):** +```python +k: NDArray = array( + dims=("nodes",), + metadata={"block": "griddata", "netcdf": True} +) +``` + +**Pydantic:** +```python +k: Annotated[NDArray, Field( + json_schema_extra={ + "dims": ["nodes"], + "block": "griddata", + "netcdf": True + } +)] +``` + +Metadata is **in the schema representation**, not separate. + +### 3. **No Technical Blockers** + +Every requirement from #167 works with pydantic: +- ✅ Protocols +- ✅ Parent-child refs +- ✅ Dimension walking +- ✅ Explicit conversions +- ✅ XArray fields +- ✅ Complex validation + +## Migration Considerations + +### Pros of Switching During Refactor + +1. **Already doing a major rewrite** - adding pydantic is marginal cost +2. **Implicit validation** - core requirement, free with pydantic +3. **Schema-first** - core goal, native to pydantic +4. **Type safety** - better than attrs for modern Python +5. **Future-proof** - ecosystem momentum, commercial backing + +### Cons + +1. **Slightly more verbose** field definitions +2. **Learning curve** for pydantic patterns +3. **JSON Schema for numpy** requires custom serializers (minor) + +### Realistic Effort + +- **2-3 weeks** to port ~30 component classes +- **1 week** to port converter logic to validators +- **1 week** for testing and edge cases + +**Total: ~4-5 weeks**, but you're already planning a multi-month refactor. + +## Recommendation + +Given your stated requirements: +- ✅ Schema-first design +- ✅ Implicit validation +- ✅ Experimental project (high risk tolerance) +- ✅ xarray DataArray fields preferred + +**Pydantic is architecturally compatible and philosophically aligned.** + +The refactor is the **ideal time** to make this change. Post-refactor migration would be much harder. + +## Next Steps + +1. **Review the prototype** (`pydantic_prototype.py`) +2. **Try porting one package** (e.g., DIS) to pydantic yourself +3. **Evaluate developer experience** - does it feel right? +4. **Make the call** - commit to pydantic or stay with attrs + +The prototype proves it's **technically viable**. The question is whether the **developer experience** matches your team's preferences. diff --git a/docs/dev/plans/archive/python-models.md b/docs/dev/plans/archive/python-models.md new file mode 100644 index 00000000..075121b1 --- /dev/null +++ b/docs/dev/plans/archive/python-models.md @@ -0,0 +1,66 @@ +# Python-based models + +A speculative possibility, not a goal for 4.x: extending flopy4 beyond MF6 +run as a subprocess, to MF6 in process (through modflowapi or XMI) and to +modelling codes written in Python, such as pywatershed and timflow. From +Feb 2026, when flopy4 still used xattree; the full draft, with sketch code, +is on the `dev-plans-full` branch. + +## The idea + +flopy4 has a generic layer (`flopy4/spec.py`: field factories, dimensions, +the attrs ↔ xarray conversion) under an MF6 overlay (`flopy4/mf6`: blocks, +DFN metadata, the component registry, file IO). Other codes could reuse the +generic layer with overlays of their own: + +- **MF6 through modflowapi/XMI:** the same components and input files, run + in process with runtime access. No new components, the same engine (so + results are easy to check), and immediately useful for monitoring, + optimisation and coupling. The obvious first case; see + [`modflowapi.md`](../integration/modflowapi.md). +- **pywatershed (PRMS):** its own components (parameters, processes), its + own metadata (`prms_name`, units) and dimensions (`nhru`, `nsegment`), + run by Python calls, with file IO optional. +- **timflow:** the same pattern, lower priority. + +## Design + +- **Executors.** A small `Executor` interface (subprocess, modflowapi, a + Python callable), registered per component type, with + `Simulation.run(executor=...)` defaulting to the subprocess. Only worth + adding with a second executor; as a keyword argument with a default, it + can be added then without breaking anything. +- **Overlays.** Each code adds its own field metadata and base classes on + top of the generic layer, as MF6 adds `block`. Each owns its dimension + names; converters map between them. +- **IO is optional.** Python codes are typically used in memory. Files + (native formats, or JSON/TOML) are for reproducibility. +- **Packages.** `flopy4//`, in this repo at first, behind extras + (`flopy4[pywatershed]`). + +Alternatives: + +- **Separate packages** (`flopy4-pywatershed`): looser coupling and + independent releases, but harder to share machinery. Possible later. +- **Converters only**, without component classes: less code, but no + shared specification, xarray views or IO. +- **Entry-point plugins:** lets others add codes, but is harder to keep + consistent. Possible later. + +## If pursued + +1. pywatershed: package, metadata, a first set of components, converters to + and from pywatershed objects, a Python executor, and the executor + interface with it. +2. A guide to adding a code. + +MF6 through XMI is a separate possibility; see [`modflowapi.md`](../integration/modflowapi.md). + +## Open questions + +- Results as code-specific objects, xarray, or the code's native objects + (probably all three, through converters). +- Testing without the external packages: mocks in CI, optional integration + tests. +- Pinning the underlying packages' versions through the extras. +- Executors vs. the explicit XMI classes in [`modflowapi.md`](../integration/modflowapi.md). diff --git a/docs/dev/plans/archive/record-values-plan.md b/docs/dev/plans/archive/record-values-plan.md new file mode 100644 index 00000000..cc421b75 --- /dev/null +++ b/docs/dev/plans/archive/record-values-plan.md @@ -0,0 +1,80 @@ +# Accept plain values for every record field + +Status: done (2026-10-08), branch `record-convert`. PTS's `no_ptc` was +left out: `sln-pts.dfn` is gone upstream, so `pts.py` is an orphan. The +narrow fix for TS6/TAS6 landed earlier on branch `followups` (`036ccbf`). + +## Problem + +A record field (a keyword and its values, e.g. TS6's `NAMES a b`) is +typed as its generated `Record` class. Passing anything else used to be +accepted and stored as given, and then the writer silently dropped it: +`Ts(time_series_name=["a"])` wrote no `ATTRIBUTES` block, and the name +checks raised `AttributeError`. + +Two converters cover some record fields today, chosen by codegen +(`make.py`, where record fields are emitted): + +- `Record.from_flag`: a keyword and options, all optional (NPF's + `XT3D [RHS]`): `True` is the bare record and `False` leaves it out. + 5 fields. +- `Record.from_value`: a keyword and one required value (TS6's `NAMES`, + `METHOD(S)`, `SFAC(S)`; TAS6's `NAME`, `METHOD`, `SFAC`). The value is + the record's value, and a list value can be given as one element. 8 + fields, all in `utl-ts`/`utl-tas`. Added by the narrow fix. + +## What's left + +Seven record fields still take only the record. Anything else is stored +as given and written wrong, or not written: + +| Field | Record's fields | Natural plain form | +|---|---|---| +| `gwf-oc.headprint`, `gwt-oc.concentrationprint`, `gwe-oc.temperatureprint`, `gwt-ist.cimprint` | `formatrecord` (nested) | the format string, `"COLUMNS 10 WIDTH 12 DIGITS 6 GENERAL"`, which `Record.from_tokens` already parses | +| `gwf-npf.rewet` | `wetfct, iwetit, ihdwet` | a tuple `(1.0, 1, 0)` | +| `sln-ims.rclose` | `inner_rclose, rclose_option` | a float, or `(value, "strict")` | +| `sln-pts.no_ptc` | `no_ptc_option` | the option, like IMS's `no_ptc` (which has `from_flag`; check why PTS's doesn't) | + +To recount, walk every generated component's fields whose type is a +`Record` but not an `Item`, and group them by converter. + +## Plan + +1. **One converter for every record field**, `Record.convert`, replacing + the per-shape choice in codegen. It takes: + - the record, or `None`: kept; + - a bool: `from_flag`'s rule, only where all fields are optional; + otherwise an error; + - a `str`: `from_tokens`, as for the OC print formats; + - a tuple: the record's fields in order; + - a mapping: the record's fields by name; + - any other value, for a record with one field: `from_value`'s rule. + + Anything else raises `TypeError` and names the field. Never store an + unconverted value. If codegen can't tell how to convert a record, + it should still emit `convert`, so a bad value fails at construction + rather than being dropped on write. +2. **Codegen** emits `converter=X.convert` for every record field. Keep + `from_flag`/`from_value` as the helpers `convert` uses, or fold them in. + Regenerate. Expect every record field's line to change. +3. **Write-time guard:** where the writer reads a record field, raise if + the value isn't a `Record`, instead of skipping it. It's a backstop for + values set after construction (`pkg.field = [...]` bypasses the + converter unless `on_setattr` runs it; check what `Component` sets). +4. **Typing.** mypy ignores `field()`'s converters + ([`mypy-converter-plan.md`](mypy-converter-plan.md)), so plain forms still fail type checks in + user code until that plan lands. Do this plan first: it fixes runtime + behavior, and the converter plan then makes the same forms + type-check. Library code should keep building records explicitly + (e.g. `Ts.from_series` builds `Ts.TimeSeriesName`) so it passes mypy + either way. + +## Tests + +- Each plain form above round-trips: construct, write, load, compare to + the record-built component. +- A wrong-shape value (a list for `rewet`, a bool for `rclose`) raises + at construction, naming the field. +- Setting a record field after construction converts, or the write + guard raises. +- Existing `from_flag`/`from_value` tests keep passing. diff --git a/docs/dev/plans/archive/reference-fields-notes.md b/docs/dev/plans/archive/reference-fields-notes.md new file mode 100644 index 00000000..cbd0987e --- /dev/null +++ b/docs/dev/plans/archive/reference-fields-notes.md @@ -0,0 +1,79 @@ +# Reference fields + +**Status: implemented** (#496, #497 and later refactors on develop). +This records the design, what was tried and dropped, and what's deferred. + +## What they are + +A **reference field** holds components a component refers to but doesn't +own: a solution's `models`, an exchange's `exgmnamea`/`exgmnameb`. Values +are model objects or names. Names are looked up late, at write. A component +created without a parent joins the simulation its references belong to. + +## Design + +- **Generated from the DFNs.** A column of the component's row in its + parent's name file with an `fk` to the parent's models is a reference + field, declared with `flopy4.spec.reference(many, required, alias)`. +- **Resolved and checked once, generically.** `flopy4.spec.resolve`, + `Component._resolved`, `_resolve_references` and `_reference_errors` do + this for every reference field. References resolve among the parent's + (the simulation's) models. +- **Each component says what it accepts.** `_reference_problem(field, + model)` returns a specific message, or `None`. An exchange checks the + model type on each side ("needs a GWT model b"), taking the types from + its ftype (`GWF6-GWT6` → `("gwf", "gwt")`). A solution checks that it + can solve the model (`_cant_solve`), and the same check covers a lone + solution's implied models, so that message exists once. + +### Copying, pickling and joining + +- **Deepcopy keeps reference fields as they are** (the original model + objects). +- **Joining a simulation retargets them.** When a component joins a + simulation (construction with `parent=`, `.parent = sim`, `sim[key] = x`, + a child field's assignment or list append, or a copied simulation + adopting its copied children), each reference to a model object that + isn't among that simulation's models is retargeted to the simulation's + model of the same name (`flopy4.spec.retarget`, + `Component._retarget_references`). Unmatched references are kept and + error on write. +- References to the simulation's own models, and references held as names, + are left alone. Nothing retargets at write, so replacing a model under a + solution that refers to the old one is still a dangling-reference error. +- **Pickling** reduces references to names. Unpickling a simulation + resolves them. + +So `deepcopy(sim)` refers to the copied models, in any field order; +`deepcopy([ims, gwf])` keeps the originals; a lone copy follows renames; +and a lone copy added to another simulation refers to that simulation's +models. + +## Tried and dropped + +- **A reverse lookup (`referrers`).** `Model.solution` and + `Model.exchanges` are read-only properties, and `solution` has + solution-only semantics (a lone solution listing nothing solves every + model), so a generic reverse lookup didn't fit. +- **`among`**, a parameter saying where to resolve. Every use resolved + against the referrer's parent, and the likely next references (below) + wouldn't be served by it. +- **Copying references by name**, as pickling does. A lone copied solution + or exchange then held names and stopped following model renames. The + earlier approach, deferring the mapping with a deepcopy memo key, kept + objects but needed order-sensitive bookkeeping. Retarget-on-join keeps + objects without the bookkeeping. + +## Deferred: package references + +The likely next references are package names: `flow_package_name` +(LKT/SFT/MWT/UZT and the GWE equivalents), SSM `pname`, MVR +`mname`/`pname`. Most point across models (a GWT/GWE package to a GWF +package), and several live inside rows or records, not top-level fields. +The lookup would need to reach another model's packages through the +exchange connecting them, and references inside records need their own +design. + +Don't generalize `reference()` before then. Binding-graph navigation +([../dfn-spec/overview.md](../dfn-spec/overview.md#step-5-binding-graph-navigation)) +is the likely shared mechanism for both. diff --git a/docs/dev/plans/archive/reference-fields-plan.md b/docs/dev/plans/archive/reference-fields-plan.md new file mode 100644 index 00000000..02d1352f --- /dev/null +++ b/docs/dev/plans/archive/reference-fields-plan.md @@ -0,0 +1,193 @@ +# Reference fields: generic resolve and check + +Status: done on `reference-fields` (`57056f57`..`7b230198`), to be merged +as is. Copy semantics changed after this plan: see [`reference-fields-notes.md`](reference-fields-notes.md) +("retarget on join"); the "deepcopy like pickle" section below is superseded. +Next: "References from DFN keys", at the end. + +Stacked on #497 (`exchange-models`). Branch `reference-fields`, worktree +`.claude/worktrees/reference-fields`. + +## Problem + +#496/#497 made the *declaration* of a reference field generic +(`reference()`), but everything that *uses* one is still written per field: + +| Place | Per-field code | +|---|---| +| `Simulation.__setstate__` | lists `sln.models`, `exgmnamea`, `exgmnameb` | +| `_apply_binding_terms` (ingress) | `isinstance` Exchange/Solution, assigns fields by hand | +| `_solution_models` / `_exchange_models` | each repeats find + "isn't in the simulation" + type check | +| `Model.exchanges` | goes through `_exchange_models()` | +| `Binding.from_component` | `isinstance` Exchange/Solution | +| `Exchange.model_types` / `_model_type` | slice `dfn_name` 3 chars at a time | + +A new reference field means touching each of these. The field doesn't say +what collection it resolves against or what it accepts, so callers hard-code +both. + +## Design + +Put the target and constraint in the field's metadata: + +```python +reference(many=True, among="models", accepts=None, required=False) +``` + +- `among`: the field of the owner's parent the references resolve against + (a solution's or exchange's parent is the simulation; `among="models"`). +- `accepts`: optional `(owner, target) -> bool`; a resolved target it + rejects is an error on write. +- `required`: (single references only) a missing value is an error on write. + +Value helpers in `flopy4/spec.py` alongside `reference_names` / +`referenced` (and `reference()` itself moves there from `flopy4/mf6/spec.py`); +the ones that need the tree are `Component` methods: + +- `resolve(value, candidates)`: names -> the candidate of that name + (case-insensitively, as MF6 does), objects kept, field shape kept. + Unresolved names stay. Never uses truthiness (an empty component is falsy). +- `Component._resolved(name)`: `resolve` against `getattr(owner._parent, among)`. +- `Component._resolve_references()`: set each reference field of each of the + component's children to its resolved value. Used by unpickling (replaces + the hand-written list in `Simulation.__setstate__`). +- `Component._reference_errors()`: missing required, not in `among`, rejected + by `accepts`, for each child. `Simulation.write` raises on any. + +Call sites then: + +- `Simulation.__setstate__` -> `resolve_references(self)`. +- `Simulation.write` -> `reference_errors` + `_solution_models(check=True)`; + `_solution_models` keeps only solution rules (empty list = all; each model + solved exactly once); `_exchange_models` and `_find_model` are deleted. +- `Model.exchanges` -> filter `sim.exchanges` on resolved references. + No generic reverse lookup: `Model.solution` / `Model.exchanges` are + read-only properties, and `solution` has solution-only semantics. +- `_apply_binding_terms` -> generic: reference fields in order, a single + one takes one term, a list takes the rest; resolved with `resolve` + against the models loaded so far. +- `Binding.from_component` -> Solution keeps `_solved_models()`; anything + else with reference fields writes their names, in field order. +- Exchange model types: from the registry (`FNAMES[f"{t}-nam"]`), splitting + the exchange's pair where both halves are model types instead of slicing + 3 characters. Generated packages load lazily, so each candidate model + package is imported first (`Exchange.model_classes`, cached). Expressed as `accepts` on `exgmnamea` / `exgmnameb`. + (Codegen is external, so the base `Exchange` declares this, not the + generated classes.) +- `Solution.models` -> `accepts=lambda sln, m: sln.solves(m)`; the + "can't solve" check moves to the generic one. + +Also: a validator on single references (str, component or None). + +## Optional: deepcopy like pickle + +Copy references as names; the outermost copy calls `resolve_references` +on itself. Drops `_PENDING_REFERENCES` and `_copied`. Behavior change: a +lone copied solution/exchange holds names, not the original models (same +as unpickling). Separate commit so it can be dropped. + +## Error messages + +Generic, naming the owner, the field and the target: + +- `exg0 has no exgmnameb` +- `exg0 refers to 'nope' in exgmnameb, which isn't in the simulation` +- `exg0 (Gwfgwt) can't refer to 'b' (Gwf) in exgmnameb` + +Tests matching the old messages are updated. + +## Testing + +`pixi run -e dev smoke` plus `test/mf6/test_mf6_component.py` and any +test file touched. + +## Next: references from DFN keys + +### Why + +Which way a reference points (solution -> models, exchange -> models) is +hand-coded: `reference()` declarations on the `Exchange` / `Solution` base +classes, and explicit Exchange / Solution branches in `Binding` and ingress. +It isn't arbitrary: in `mfsim.nam`, `models.mname` is a key, and the +`exchanges` / `solutiongroup` rows' model-name columns refer to it. Each row +belongs to its exchange / solution, so the reference sits on that component. +But nothing in the DFNs says so. The devtools v2 schema already has the +vocabulary (`pk`, `fk = "[component.]block.field"`, used for MVR's `pname1` / +`pname2`). sim-nam just isn't annotated. + +### devtools side (another agent) + +Plan: `~/dev/modflow-devtools/sim-nam-model-keys-plan.md`. In short: + +- `models.mname`: `pk = true` (MF6 rejects duplicate names; checked in + `SimulationCreate.f90`) +- `exgmnamea`, `exgmnameb`, `slnmnames`: `fk = "models.mname"` +- `utl-hpc.partitions.mname`: `fk = "sim-nam.models.mname"` +- schema: allow `fk` on string-dtype Arrays (for `slnmnames`); today Array + `fk` must be integer (a per-cell zone index) + +### flopy4 side (here) + +Codegen is in `flopy4/mf6/utils/codegen` (`make.py`; base classes chosen by +`_base_class`, ~line 1350). + +1. **Find the binding row.** For a component whose parent is sim-nam, find + the sim-nam list row that binds it (the row with its type and file: + `exchanges` for `exg-*`, `solutiongroup` for `sln-*`, `models` for + models). Its columns with `fk` to `models.mname` are the component's + reference fields, in row order. Single String -> `reference(many=False)`, + string Array -> `reference()`. `required` follows the column's + optionality. +2. **Generate the fields** on the generated classes (`Gwfgwf.exgmnamea`, ..., + `Ims.models`) instead of declaring them on the bases. Name: keep the DFN + column name (`exgmnamea`); for the solution, `slnmnames` vs our `models` + needs a decision: rename in codegen (as `sim-nam` -> `simulation`) or + accept `slnmnames`. Lean: keep `models` via a codegen rename table, since + it's public API. +3. **Checks stay in code.** `check` (`_connects`, `_cant_solve`) can't be + attached to a generated field from the base class. Options: a base-class + hook `_reference_problem(field, model) -> str | None` that + `_reference_errors` calls, replacing the field's `check` metadata. Lean: + the hook, and drop `check` from `reference()`. +4. **Generic Binding / ingress again.** `Binding.from_component` writes the + reference fields' names in field order; `_apply_binding_terms` assigns + trailing terms in that order (a single field takes one, a list the rest). + We dropped this before because order was our declaration order; now it's + the DFN row order, which is the file's. The Solution special case + (`_solved_models()`: the only solution listing none solves all) stays. +5. **Fallback.** Until devtools ships the annotations, the DFNs we sync from + won't have them. Either wait to merge until a devtools release has them, + or keep the hand declarations as a fallback when no `fk` is found. Lean: + wait. A fallback would mean two sources of truth. +6. **Package references later.** MVR's `pname1` / `pname2` are already + `fk = "packages.pname"`; the same codegen path could cover them when we + get there (inside records, though: see the notes doc). + +### Testing + +Develop against a local devtools branch (install it editable in the pixi dev +env, or point sync at locally migrated DFNs), regenerate, then run +`test_mf6_component.py`, `test_mf6_load_all_models.py`, codegen tests, and +smoke. Generated-file diffs should only add the reference fields to exchange +and solution classes. + +### Status (flopy4 side) + +Branch `dfn-references` (off develop `46ef2fab`), commit `dc86e843`, local +only. Steps 1-4 done: `_binding_row` / `_reference_specs` in codegen, the +`slnmnames -> models` rename table, `_reference_problem` hook (`check` +dropped from `reference()`; `many` added to its metadata), generic +`Binding` terms and `_apply_binding_terms`. Generated exchange/solution +modules were produced from DFNs annotated in-process (the cached develop +DFNs aren't at the contract commit, so only the annotation's diff was +applied). `test_reference_fields_from_keys` annotates the fixture DFNs +itself until devtools ships the keys. Remaining: once devtools releases, +re-sync and drop the test's annotation helper; don't `sync` before then +(it would regenerate exchanges/solutions without reference fields). + +Update: devtools #396 (b7c43566) merged with the sim-nam keys (`mname` pk; +`fk = "models.mname"` on `exgmnamea`, `exgmnameb`, `slnmnames`; utl-hpc +`partitions.mname` fk `sim-nam.models.mname`). `dfn-references` now pins it +(commit after dc86e843); sync at the contract commit reproduces the committed +modules exactly, `_with_model_keys` is gone, and the mixin gap check follows +base classes into leaf subclasses (it flagged `Solution.models`). diff --git a/docs/dev/plans/archive/repeating-block-header-generalization.md b/docs/dev/plans/archive/repeating-block-header-generalization.md new file mode 100644 index 00000000..ddd26bcf --- /dev/null +++ b/docs/dev/plans/archive/repeating-block-header-generalization.md @@ -0,0 +1,131 @@ +# Generalizing repeating-block handling via `Block.header` + +**Archived 2026-10-19.** Partly adopted: codegen now reads `Block.header` (`make.py` ~1374, `filters.py` ~139) for repeating blocks and fill-forward. The full generic consumer sketched here wasn't pursued, and the doc's own conclusion was "not now". + +Found while investigating whether flopy4's `utl-tas` support (self-sizing +arrays, [`dev3-migration-gaps.md`](dev3-migration-gaps.md)) could be simplified. It can't be +simplified in isolation, but the investigation surfaced something flopy4 +doesn't currently use: dev3's schema already has a generic "this block +repeats, keyed by this header field" concept +(`modflow_devtools.dfns.schema.Block.header` / `.repeats`), richer than +legacy's bare `repeating: bool`. flopy4 doesn't consume it anywhere — not +even for PERIOD blocks, which hardcode the literal string `"period"` at +every call site instead. This doc sketches what a generic consumer would +look like. **Not a plan to implement now** — see "Is this worth doing" at +the end. + +## What the schema already gives us + +```python +class Block(BaseModel): + name: str + fields: dict[str, InputField] + header: "InputField | None" = None + + @property + def repeats(self) -> bool: + return self.header is not None +``` + +Verified in the current dev3 snapshot (`~/dev/modflow-devtools/autotest/dfns/__snapshots__/v2.0.0.dev3/`): + +- `gwf-rch.toml`: `[blocks.period.header]` → `name = "iper"`, `type = "integer"` +- `utl-tas.toml`: `[blocks.time.header]` → `name = "time_from_model_start"`, `type = "double"` + +Both repeating blocks in the whole corpus (PERIOD and TIME — TAS is the +only non-PERIOD repeating block) already carry this. `header` gives the +dict key's *name*, *type*, and *description* — enough to drive parsing +generically instead of hardcoding per block. + +## Current state: every hardcoded `"period"` (and TAS-equivalent) site + +None of these consult `block.header`/`block.repeats`. All hardcode the +literal block name: + +**`flopy4/mf6/utils/codegen/filters.py`** +- `has_period_block`: `"period" in component.blocks` +- `is_period_array`: `block_name == "period"` + +**`flopy4/mf6/utils/codegen/make.py`** +- `_period_keystring_names`: `component.blocks.get("period")` +- `_build_block_property_specs`: skips `block_name == "period"` explicitly +- `build_component_spec`'s main field loop: four separate + `block_name == "period"` branches (list field / keystring union / bare + scalar / READARRAY array), plus the TAS-equivalent branch gated by + `filters.is_readarray_field(f)` (name-matched, not block-matched) +- period dict-field assembly (`_stress_period_data`, `_readarray_period_fields`): + hardcodes `"block": "period"` metadata and the `kper = iper - 1` 1-based→0-based + convention + +**`flopy4/mf6/converter/ingress/structure.py`** +- Pass 1's skip condition: `block_name.startswith("period")`, plus a + separate `tar_block_prefixes` set built from `reader == "readarray"` + metadata (TAS's parallel mechanism, keyed off field metadata rather than + block name) +- Pass 3 (period): `block_name.startswith("period")`, splits on whitespace, + parses `int(parts[1]) - 1` +- Pass 3b (READARRAY period): `f.metadata.get("block") == "period"` +- Pass 3c (TAS): separate pass entirely, parses `float(parts[1])`, no `- 1` + +**`flopy4/mf6/converter/egress/unstructure.py`** +- `block_name == "period"` branch (two sub-cases: READARRAY ndarray, and + Item-list dict), producing `f"period {kper + 1}"` keys +- Separate branch for `reader == "readarray" and isinstance(field_value, dict)` + (TAS), producing `f"{block_name} {tval}"` keys — no `+ 1`, float not int + +The PERIOD and TAS code paths are structurally the same shape (a block +that repeats, keyed by a header value, containing either an Item-list, a +keystring union, or a bare array) but are implemented as two unrelated, +independently-hardcoded passes. + +## What a generic consumer would look like + +**Codegen (`make.py`):** iterate `component.blocks.items()` and check +`block.repeats` instead of comparing `block_name == "period"` inline in +the field loop. For any repeating block, dispatch by its field shape — +same three cases already handled for PERIOD, generalized: + +- Item-list field (with or without a keystring union) → `dict[, list[ItemClass]]` + (generalizes `_stress_period_data`/`period_arms`) +- Bare Array field → `dict[, ArrayLike]` (generalizes + `_time_array_series_fields` — this is what TAS already does) +- `` comes from `block.header.type` (`int` for PERIOD's `iper`, + `float` for TAS's `time_from_model_start`) instead of being implied by + which hardcoded branch matched + +**Ingress (`structure.py`):** one generic pass over `component.blocks` +where `.repeats`, replacing Pass 3/3b/3c: split `raw_lower` keys on each +repeating block's name prefix, parse the header token using the declared +header type, dispatch to item-list or bare-array parsing by field shape — +same dispatch already written, just triggered by `block.repeats` instead +of string-matching `"period"` or a `reader == "readarray"` metadata scan. + +**Egress (`unstructure.py`):** the PERIOD branch and the TAS branch merge +into one "repeating block" branch, formatting `f"{block_name} {header_value}"`. + +**The one genuine PERIOD-specific wrinkle:** MF6's PERIOD block header +(`iper`) is 1-based in the file, and flopy4 stores it 0-based (`kper`) for +Python-list-index convenience. That offset is an MF6/flopy4 convention, +not something `Block.header` encodes (TAS's `time_from_model_start` has no +such offset — it's used as-is). A generic consumer still needs a per-block +hook or metadata flag for this, not something inferrable from the schema +alone. + +## What does *not* get simplified by this + +`is_readarray_field`'s shape check (`Array.shape == []`) and the +`array_bare` writer path are about *shape ambiguity* — dev3 can't tell "a +plain self-sizing trailing-tokens field" from "a griddata-style field +whose own block repeats" by shape alone (see `is_array`'s docstring). That +is a completely separate problem from block-repetition and doesn't go +away if `Block.header` is consumed generically. See the companion +walkthrough on `just_data` for the related, but distinct, "no leading +field-name token" question. + +## Is this worth doing + +Not now. The entire MF6 corpus has exactly two repeating-block shapes +(PERIOD, TIME) — generalizing a mechanism that has exactly one non-trivial +consumer beyond the original is speculative. Worth revisiting if/when a +third repeating-block package shows up, or if the PERIOD-side hardcoding +itself becomes a maintenance burden independent of TAS. diff --git a/docs/dev/plans/archive/solution-binding.md b/docs/dev/plans/archive/solution-binding.md new file mode 100644 index 00000000..54189a6c --- /dev/null +++ b/docs/dev/plans/archive/solution-binding.md @@ -0,0 +1,134 @@ +# Solution–model binding + +How solutions (IMS, EMS) and exchanges are associated with the models they +solve or connect. Fixes #285 (and covers flopy#2355). + +## Status + +- **PR #496, `solution-models`** (on `develop` 3813b016): solutions hold + their models. Open. +- **`exchange-models`** (cb41db42, based on `solution-models`): exchanges + hold their models the same way. Pushed; open its PR against + `solution-models`, retarget to `develop` once #496 merges. + +## Background + +- **#285**: support `Ims(models=[gwf])` alongside `Ims(models=["gwf"])`, and + let the solver take over the models' association with the simulation. +- **flopy#2354 / flopy#2355** (mbakker7): flopy3's + `sim.register_ims_package(ims, model_list=[gwf.name])` is hard to teach. + He suggests `ModflowIms(sim, model=gwf)`, or `ModflowIms(model=gwf)` with + the sim inferred. One solution solving many models is real (LGR, + parallel, flow + transport), so the relation is many-to-one. +- **The bug**: `Ims(models=[gwf])` was accepted, but `sim.write()` put the + model's repr in `mfsim.nam`, because `Binding.from_component` wrote + `tuple(component.models)` unconverted. + +## Design, as built + +### Reference fields + +`Solution.models` (`list[Model | str]`) and `Exchange.exgmnamea` / +`exgmnameb` (`Model | str | None`) are declared with `reference()` +(`metadata={"reference": True}`). A reference field: + +- isn't a child field: walkers (`_children`, writing, `to_dict`, parent + assignment) skip it, so a solution never takes over its models' parent or + name; +- is stored as model names in xarray, and its repr and eq go by name; +- holds objects or names. Names are resolved at write time, + case-insensitively as MF6 does, so default names (given on attach) and + renames are followed for objects. A name doesn't follow a rename. +- A single model is wrapped in a list (`Ims(models=gwf)`). + +The simulation owns its models; solutions and exchanges only refer to them, +as `mfsim.nam` does (models declared in MODELS, named again in +SOLUTIONGROUP / EXCHANGES). + +### Ways to associate (user's view) + +1. **Default**: one solution listing no models solves every model in the + sim, including ones added later. `Simulation(models=[gwf], + solutiongroup=[Ims()])`, `Ims(parent=sim)`. +2. **By object**: `Ims(models=[gwf, gwt])`, bottom-up + (`Simulation(solutiongroup=[ims])`) or top-down (`Ims(parent=sim, ...)`). +3. **By name**: `Ims(models=["flow"])`; the model needn't exist yet. Mixed + lists work. +4. **Join through models**: `Ims(models=[gwf])` with no parent, where `gwf` + is already in a sim, joins that sim's `solutiongroup` (at construction). + Error if the models are in different sims. Exchanges do the same + (`Component._references_parent`). +5. **Later changes**: `ims.models.append(gwt)`, `ims.models = [...]`, + `sim.solutiongroup.append(ims)`, `sim["ims0"] = ims`. +6. **Loading**: solutions and exchanges hold the loaded model objects. +7. **Lookup**: `gwf.solution` (read-only, `None` if unsolved or no sim). No + stored back-pointer and no `Gwf(solution=...)`. + +### Write-time checks (`SimulationMethods._solution_models`) + +Errors start with "Invalid solution group: ...". + +- every model a solution lists is in the sim (catches stale names and + removed objects); +- every model is solved by exactly one solution; with two or more + solutions the default doesn't apply, so each must list its models; +- EMS solves PRT only, IMS everything else; +- exchanges: both models present, and of the exchange's model types. + +Loading more than one SOLUTIONGROUP is an error (multiple groups aren't +supported; `mxiter` and the hardcoded `SOLUTIONGROUP 1` are unchanged). + +### Copying and pickling + +- `deepcopy` copies what a component owns and shares what it refers to: + copying a sim gives a sim whose solutions refer to the copied models; + copying a lone `Ims` keeps referring to the original models. References + are resolved after the outermost copy, so order doesn't matter + (`deepcopy([ims, gwf])` works). +- `pickle` stores references as names; `Simulation.__setstate__` resolves + them, as loading does. +- `copy.copy` stays shallow. + +### Filenames + +Solutions use #495's uniform default, `{sim}.{name}.{ext}` +(`sim.ims0.ims`), like any list-field child (`model.wel0.wel`). Decided: +no `{model}.ims` default and no dropping the `0` for a lone solution. + +- flopy#2355 doesn't ask for `{model}.ims`. Its `gwf.name + '.ims'` is + flopy3 code, setting a filename by hand so two solutions' files don't + collide. What it asks is that flopy take care of the filename, which the + uniform rule does: names are unique among siblings. +- `{model}.ims` would be the only default based on a reference (which can + change) rather than the owner, with fallbacks for zero or several models. +- Dropping the `0` would break "once assigned, a name stays": adding a + second solution would rename the first, or name them inconsistently. + 2cbc0da7 drops the name only when it's the extension (single-child + fields: `model.dis`), never a list index. + +## Done since the first push + +- **CI failures on #496** (`test_mf6_sync`, `test_mixin_fields`): + `SimulationMethods.__setstate__: no Model.lower`. The mixin gap check + (`flopy4/mf6/_mixin_fields.py`) had two bugs, now fixed (add09054): + - comprehension variables were bound once per method, so two + comprehensions using `m` over different fields were conflated. They're + now scoped to their comprehension (`_scoped`); + - a field holding a plain type besides components (`list[Model | str]`) + was checked as if every item were a component. Past such a field + (`_open`), an attribute only some types have isn't reported. +- **Examples** (0a6d5a23): `period-data` and `selection` pass `models=[gwf]` + instead of a name, like the other examples. +- **Stale example outputs** (8eb40524): `docs/examples/period-data/` is + gitignored like the other examples' output dirs, but its outputs had been + committed and went stale with #495's naming. Untracked them. +- `exchange-models` rebased onto the above (452b9e51) and pushed. + +## Open + +- **Other model-name fields**, as a separate issue: MVR `mname1` / `mname2` + (`flopy4/mf6/gwf/mvr.py`) and HPC `partitions` (`flopy4/mf6/utl/hpc.py`). + The same reference mechanism would apply, but they're inside record/list + fields, so it's more involved. +- **Comment on #285 / flopy#2355** with the design, to close them (only + when asked). diff --git a/docs/dev/plans/archive/spatial-index-design.md b/docs/dev/plans/archive/spatial-index-design.md new file mode 100644 index 00000000..9bea9588 --- /dev/null +++ b/docs/dev/plans/archive/spatial-index-design.md @@ -0,0 +1,302 @@ +# Spatial Index Design + +## Background + +Currently, the `StructuredGrid` class wraps `flopy.discretization.StructuredGrid` and provides xarray coordinate-based indexing using: + +- `x` dimension coordinate: 1D array `(ncol,)`, unique x values, one per column +- `y` dimension coordinate: 1D array `(nrow,)`, unique y values, one per row +- `z` non-dimension coordinate: 3D array `(nlay, nrow, ncol)`, cell-specific z values (varies with topography) + +The `x`/`y` coordinates use `PandasIndex` and enable e.g. `sel(x=..., y=...)`. The `z` coordinate has no index and acts as an auxiliary coordinate only. + +This works well for horizontal spatial queries but doesn't support more advanced queries like vertical (elevation-based) or full 3D spatial queries. + +## Dimension and Coordinate Naming Strategy + +MF6 uses topology names (`nlay`, `nrow`, `ncol`, `ncpl`). CF conventions expect spatial names (`x`, `y`, `z`). Support both. + +Structured grids (DIS) have regular spatial dimensions. Unstructured grids (DISV/DISU) don't. + +### Structured Grids (DIS) + +Spatial coordinates as primary dimensions, MF6 names as auxiliary: + +```python + +Dimensions: (x: 100, y: 100, z: 3, time: 10) +Coordinates: + * x (x) float64 # spatial coordinate (dimension coord) + * y (y) float64 # spatial coordinate (dimension coord) + * z (z) float64 # layer elevations (dimension coord) + * time (time) float64 # simulation time + col (x) int64 # MF6 column index (1-based) + row (y) int64 # MF6 row index (1-based) + layer (z) int64 # MF6 layer index (1-based) +Data variables: + head (time, z, y, x) float64 + ... +``` + +Why: +- Leverages native xarray `.sel(x=..., y=...)` without custom indexes +- Follows CF conventions +- MF6 indices still accessible as auxiliary coords + +Note: Make `z` a 1D dimension coordinate (layer top/midpoint elevations). Store full 3D elevations as auxiliary coords if needed. + +### Unstructured Grids (DISV/DISU) + +Topology dimensions as primary, spatial coords as auxiliary: + +```python + +Dimensions: (nlay: 3, ncpl: 1000, time: 10) # DISV example +Coordinates: + * nlay (nlay) int64 # layer index (dimension coord) + * ncpl (ncpl) int64 # cells per layer (dimension coord) + * time (time) float64 # simulation time + cell_x (ncpl) float64 # cell center x coordinate (auxiliary) + cell_y (ncpl) float64 # cell center y coordinate (auxiliary) + cell_z (nlay, ncpl) float64 # cell center z coordinate (auxiliary) +Data variables: + head (time, nlay, ncpl) float64 + ... +``` + +Why: +- No regular spatial dimensions +- Follows UGRID conventions (xugrid/uxarray pattern) +- Spatial coords require custom indexing + +### Unified `.grid` Accessor + +Consistent API across both grid types: + +```python +# Works for both structured and unstructured grids: +ds.grid.sel(x=1000, y=2000) # spatial selection +ds.grid.sel(layer=1, row=5, col=10) # MF6 topology selection (DIS) +ds.grid.sel(layer=1, ncpl=42) # MF6 topology selection (DISV) +ds.grid.sel(x=1000, y=2000, z=50) # 3D spatial query +ds.grid.isel(...) # always 0-based indexing +ds.grid.plot(...) # grid-aware plotting +ds.grid.neighbors(layer=1, row=5, col=10) # cell connectivity +``` + +Implementation: +- **DIS**: Spatial selection → native `.sel()`, topology selection → map MF6 coords to spatial +- **DISV/DISU**: Topology selection → native `.sel()`, spatial selection → custom `GeospatialIndex` +- Detect grid type via dimension inspection, dispatch accordingly + +### Relationship to Standards + +**xugrid/uxarray**: Topology dimensions primary for unstructured grids, spatial coords auxiliary. Moving to custom xarray indexes ([xugrid #35](https://github.com/Deltares/xugrid/issues/35), [PR #373](https://github.com/Deltares/xugrid/pull/373)) instead of wrapper classes. + +**CF conventions**: Structured grids use matching dimension/coord names (`x`, `y`, `z`). Unstructured use auxiliary spatial coords. Our approach aligns with both. + +**Key insight**: Asymmetry is correct. Structured/unstructured grids have different natural dimension systems. `.grid` accessor unifies the interface. + +### Implementation Considerations + +#### Z-coordinate for structured grids + +Two options: + +**1. Layer-representative 1D** (recommended) +- `z = [top1, top2, top3]` or layer midpoints +- Enables native `.sel(z=...)` +- Store full 3D elevations as auxiliary coords + +**2. 3D auxiliary** (current) +- `z = (nlay, nrow, ncol)` array +- No native `.sel(z=...)` support +- Requires custom index + +#### Grid accessor dispatch + +Detect grid type via dimensions: + +```python +if {'x', 'y'}.issubset(self.dims): + return StructuredGridAccessor(self) +elif 'ncpl' in self.dims or 'nlay' in self.dims: + return UnstructuredGridAccessor(self) +``` + +Each accessor implements: `.sel()`, `.isel()`, `.plot()`, `.neighbors()`, `.query_ball()`, `.contains()` + +#### Coordinate transformation + +**Topology → spatial (structured)**: +```python +# ds.grid.sel(layer=2, row=10, col=5) +# → find indices where layer==2, row==10, col==5 +# → ds.isel(z=..., y=..., x=...) +``` + +**Spatial → topology (unstructured)**: +```python +# ds.grid.sel(x=1000, y=2000) +# → spatial_index.query_point(x=1000, y=2000) +# → ds.isel(ncpl=...) +``` + +## Proposal + +Extend `GeospatialIndex` from [flopy3 PR 2654](https://github.com/modflowpy/flopy/pull/2654) to implement `xarray.core.indexes.Index`. Required methods: `sel()`, `isel()`, `equals()`, `union()`, `intersection()`. Start with `.sel()`. + +**Primary need**: Unstructured grids (DISV/DISU) where spatial selection can't use native dimension coords. Structured grids (DIS) get basic spatial selection natively, but advanced queries benefit from custom index too. + +Integrate with: `StructuredGrid`, `VertexGrid`, `UnstructuredGrid`. + +Advanced spatial queries to support: + +```python +# Point queries with different semantics +head = ds.grid.sel(x=250, y=650, z=50, method='contains') # head in cell containing point +head = ds.grid.sel(x=250, y=650, z=50, method='nearest_center') # head in cell with center nearest to point + +# Spatial range queries +nearby = ds.grid.query_ball(point=(1500, 2300, 45), radius=100) # 3D ball query + +# Elevation-based slices +cells = ds.grid.sel(z=slice(40, 60)) # find all cells between elevations 40-60 +``` + +### Point Queries + +Use xarray `.sel()` syntax. + +#### Open Questions + +**1. Z-only queries** + +What does `sel(z=50)` mean when z varies in (x, y)? Multiple cells may match. + +Options: +- Require x, y when selecting by z +- Return all cells matching z +- Support explicit mode: `sel(z=50, mode='all' | 'first' | 'nearest_to_origin')` + +**2. Method parameter** + +Two semantics: **containment** vs **nearest center**. + +**Containment** (default): Cell that contains the point. +- Use cases: locating wells, boundaries, observation points +- Points outside grid: return `KeyError` or `NaN` +- On boundaries: tie-break to lowest-numbered cell (current flopy 3.x behavior) + +**Nearest center**: Cell whose center is closest to point. +- Use cases: out-of-bounds queries, cross-grid comparisons +- Always returns a result, even outside grid bounds +- May return cell that doesn't contain the point + +``` +┌──────────────────┬──────────────────┐ +│ │ │ +│ │ ○ ← nearest center +│ │ ╱ │ +│ │ ╱ │ +│ ★ ─────┼──╱ │ +│ (query) │ │ +│ │ │ +├──────────────────┼──────────────────┤ +│ │ │ +│ │ │ +└──────────────────┴──────────────────┘ + +Query point (★): +- contains → left cell +- nearest center → right cell +``` + +```python +# Default: containment +ds.grid.sel(x=250, y=650, z=50) +ds.grid.sel(x=250, y=650, z=50, method='contains') + +# Nearest center +ds.grid.sel(x=250, y=650, z=50, method='nearest_center') +``` + +Should `method="nearest"` alias one of these, or disallow it to avoid ambiguity? + +### Spatial Range Queries + +Ball queries can't use `sel()` (only accepts coord names, not arbitrary params). + +Use cases: zone of influence, observation network radius, local refinement region. + +#### 2D (x, y) + +Use geopandas: + +```python +cells_gdf = gpd.GeoDataFrame(geometry=[Point(x, y) for x, y in zip(...)]) +mask = cells_gdf.sindex.query(Point(1500, 2300).buffer(100), predicate='intersects') +``` + +Expose via accessor to avoid manual `GeoDataFrame` creation. + +#### 3D (x, y, z) + +Use scipy.spatial.cKDTree (from flopy 3.x `GeospatialIndex`): + +```python +tree = cKDTree(np.column_stack([x, y, z])) +indices = tree.query_ball_point((1500, 2300, 45), radius=100) +mask = np.zeros(grid.shape, dtype=bool) +mask.ravel()[indices] = True +``` + +Expose via accessor: + +```python +ds.grid.query_ball(point=(x, y, z), radius=100) +``` + +### Elevation-Based Slicing + +```python +# Extract vertical profile at x=250, y=650. +# For structured grids, this works natively +profile = ds.sel(x=250, y=650, method='nearest') # returns all z values at (x, y) + +# For unstructured grids, use .grid accessor +profile = ds.grid.sel(x=250, y=650) # spatial query returns vertical profile + +# With z-slicing: get only specific elevation range +profile_shallow = ds.grid.sel(x=250, y=650, z=slice(80, 100)) + +# Select all cells in water table zone (elevation 40-60) +wt_zone = ds.grid.sel(z=slice(40, 60)) +``` + +## References + +Several external libraries already have some support for advanced spatial queries: + +- **GeoPandas**: 2D spatial queries, already a dependency - use for ball queries, spatial joins +- **scipy.spatial**: 3D KD-tree, likely already in dependency tree - use for 3D ball queries +- **xvec**: xarray vector data extension, wraps GeoPandas - unnecessary if already using GeoPandas +- **rioxarray**: Raster operations only, not applicable for cell center queries +- **rtree/pyvista**: Heavier dependencies, not needed given GeoPandas/scipy coverage + +Some specific resources: + +- [xarray Custom Index Guide](https://docs.xarray.dev/en/stable/internals/how-to-create-custom-index.html) +- [scipy.spatial.cKDTree](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.cKDTree.html) - 3D ball queries +- [GeoPandas Spatial Index](https://geopandas.org/en/stable/docs/reference/api/geopandas.sindex.SpatialIndex.html) - 2D spatial queries +- [GeoPandas sjoin_nearest](https://geopandas.org/en/stable/docs/reference/api/geopandas.sjoin_nearest.html) - Nearest neighbor with max distance +- [xvec](https://xvec.readthedocs.io/) - Vector data cubes for xarray (wraps GeoPandas) + +## Future + +Some things to consider after an initial implementation: + +- Time-varying coordinates: support transient z coordinates (subsidence) +- Multi-grid queries: queries across nested grids +- Spatial aggregations: average within spatial regions +- Coordinate transformations: support for different CRS/projections diff --git a/docs/dev/plans/archive/tas-shorthand-typing-plan.md b/docs/dev/plans/archive/tas-shorthand-typing-plan.md new file mode 100644 index 00000000..17b488b3 --- /dev/null +++ b/docs/dev/plans/archive/tas-shorthand-typing-plan.md @@ -0,0 +1,53 @@ +# Type the time-array series name shorthand + +Status: proposed (2026-10-06); not started. Small. Rechecked 2026-10-07 +against develop `a6c52c8`, where the TAS period references landed (#413). +The `period_arrays` module docstring still shows `{0: "rchseries"}` with no +mypy caveat. + +## Problem + +Period arrays from a time-array series are typed +`dict[int, FloatArrayLike | TimeArraySeriesRef]`. At runtime a plain name +also works, `recharge={0: "rchseries"}`, because +`Package._init_period_arrays` converts it. mypy rejects it, because the +conversion happens in `__attrs_post_init__` rather than a field +converter. So [`mypy-converter-plan.md`](mypy-converter-plan.md)'s plugin won't cover it either: +there's no `converter=` for it to read. + +The reference class is the typed form, and user code that checks +`isinstance(v, TimeArraySeriesRef)` already passes mypy. This plan is about +the shorthand alone. + +## Options + +1. **Document it, no code.** Say in the `TimeArraySeriesRef` and + period-array docs that the typed form is `TimeArraySeriesRef("name")`, + and that a plain name works at runtime but not under mypy. + - Recommended. +2. **Widen the annotation** to `... | TimeArraySeriesRef | str`. The + shorthand then type-checks, but reading a period types it as possibly + `str`, which it never is after init. Every reader would have to + narrow it away. Rejected for the same reason as widening the record + flags. +3. **Move the conversion into a field converter**, so the mypy plugin + could read it. + - That needs `_init_period_arrays`'s work per field: dense input + splitting, flattening, the int cast, and the time-series check. + - The aux checks need `self` (the auxiliary names, the grid + package's periods), which is why it's in post-init today (see + [`period-storage-plan.md`](period-storage-plan.md)). + - A converter with `takes_self=True` could do the per-field part. + The cross-field checks would stay in post-init. + - Worth it only if the plugin lands and more period-array input forms + need typing, not for this shorthand alone. + +## Steps (option 1) + +1. In `TimeArraySeriesRef`'s docstring, state that a plain name is + accepted at init but isn't the typed form. +2. Make the same note in `period_arrays`' module docstring, where the + shorthand is shown. +3. Show the typed form first in any user docs or examples that use time + series. +4. Revisit option 3 if [`mypy-converter-plan.md`](mypy-converter-plan.md) is done. diff --git a/docs/dev/plans/archive/trailing-text-load-fix.md b/docs/dev/plans/archive/trailing-text-load-fix.md new file mode 100644 index 00000000..90afad6a --- /dev/null +++ b/docs/dev/plans/archive/trailing-text-load-fix.md @@ -0,0 +1,132 @@ +# Fixing the silent loss of period data on trailing text + +`Simulation.load` silently drops every period block of a file whose blocks end `END PERIOD 1`, +if text follows the last one. This is a one-line fix to a regex. Tested at `8d25e4b7`, in a +throwaway worktree. Nothing here has been applied. + +Found by the DDL slice's corpus run, 2026-11-05. + +## What to do + +1. **Apply the regex and the test below, as one small PR.** It's silent data loss with a one-line + fix, and on the corpus it changes only `test021_twri`. CI runs the full suite. +2. **Add a warning when the trim removes text,** naming the file and the lines. I'd do this in the + same PR or straight after. It's what would have made this bug visible, and it isn't tested yet. +3. **Leave text between blocks for later.** With the fix it raises where it used to lose data + silently, which is already better. Accepting it, as MF6 does, is a separate decision (see + "Related"). +4. **Afterwards, rerun `test021_twri` through the DDL slice,** since its recharge is missing from + the slice's results. + +Don't widen this into the OC load bug (`oc-format-load-bug.md`), which is unrelated. + +## The problem + +`mf6/test/test021_twri` loads without an error, with `recharge=None` on its RCHA. MF6 applies +that recharge: the nightly `db59037` runs the model, with 13,608,000 of `RCHA` inflow in the +budget. `twri.rch` has MODFLOW-2005 lines left after its last block: + +``` +BEGIN PERIOD 1 + RECHARGE + CONSTANT 3.E-8 +END PERIOD 1 + + 1 0 NRCHOP,IRCHBD +``` + +When a file fails to parse, `loads_typed` and `_parse_basic` cut it back to the last block +`_BLOCK_RE` recognizes (`flopy4/mf6/codec/reader/__init__.py:21`), and parse again. That +regex allows one word after `END`. So `END PERIOD 1` doesn't end a block, the last match is +`END OPTIONS`, the trim removes every period block, and the second parse succeeds. A bare +`END PERIOD` is fine, which makes this look intermittent. + +It isn't specific to RCHA. A WEL file loses its 40 rows the same way. + +## The fix + +Allow anything after `END`, as MF6 does: + +```diff +- r"^[ \t]*begin\s+\S+.*?^[ \t]*end\s+\S+[ \t]*$", ++ r"^[ \t]*begin\s+\S+.*?^[ \t]*end(?:[ \t][^\n]*)?$", +``` + +The comment above `_BLOCK_RE` can stay as it is. + +## Test + +`test/mf6/test_mf6_reader.py`: + +```diff +--- a/test/mf6/test_mf6_reader.py ++++ b/test/mf6/test_mf6_reader.py +@@ -1074,3 +1074,22 @@ def test_transform_record_header(): + keys = ["continuous FILEOUT Heads.csv", "continuous FILEOUT flows.bsv BINARY"] + assert list(loads_typed(text, "utl-obs")) == keys + assert [k.lower() for k in loads(text)] == [k.lower() for k in keys] ++ ++ ++TRAILING = "\n 1 0 NRCHOP,IRCHBD\n 1 INRECH\n" ++ ++ ++@pytest.mark.parametrize("end", ["END PERIOD", "END PERIOD 1", "end period 1", "END PERIOD 1 "]) ++def test_text_after_the_last_block_keeps_the_blocks(end): ++ """mf5to6 can leave MODFLOW-2005 lines after the last block, which MF6 ignores. ++ Trimming them mustn't take a block whose END has a period number with them.""" ++ from flopy4.mf6.codec.reader import loads_typed ++ ++ text = ( ++ "BEGIN OPTIONS\nEND OPTIONS\n" ++ "BEGIN DIMENSIONS\n MAXBOUND 1\nEND DIMENSIONS\n" ++ f"BEGIN PERIOD 1\n 1 1 1 -1.0\n{end}\n" ++ ) ++ clean = loads_typed(text, "gwf-wel") ++ assert "period 1" in clean ++ assert list(loads_typed(text + TRAILING, "gwf-wel")) == list(clean) +``` + +It fails without the fix for the three `END`s with a number, and passes for the bare one. + +`test_mf6_load_all_models.py` can't catch this, since the model loads. Comparing `twri`'s +recharge there would pin it on a real file, if that's worth a special case. + +## How it was tested + +| Check | Result | +|---|---| +| `test/mf6/test_mf6_reader.py`, with the fix | 49 passed | +| The new test, without the fix | 3 failed, 1 passed | +| What each package loads, before and after, on 253 corpus models (all but the Biscayne 8) | Only `test021_twri` changes, gaining `recharge` | +| A scan of the corpus's 36 input files with text after their last `END` (under 3 MB, without Biscayne) | Only `twri.rch` loses a block to today's trim | + +The full test suite wasn't run, and is left to CI. + +## Related, and not fixed by this + +**Text between blocks.** MF6 skips it, and runs a copy of `test120_mv_dis-lgr` with +`some stray words here` before `BEGIN period 1`. Today flopy4 silently loses the period data +there too, by the same route. With this fix it raises `UnexpectedToken`. Stricter than MF6, +but no longer silent. + +To accept what MF6 accepts, the trim could keep all the blocks, not just everything up to the +last one: + +```python +def _strip_text_outside_blocks(data: str) -> str: + matches = list(_BLOCK_RE.finditer(data)) + return "\n".join(m.group(0) for m in matches) + "\n" if matches else data +``` + +I tried this on that one copy only. It loads the recharge, and a bad array in the period still +raises. Line numbers in a later parse error would shift, and it hasn't seen the test suite. I'd +take the one-line fix first. + +**The retry is silent by design.** A trim that removes text says nothing to the user, even +when it's right. A warning naming the file and the lines dropped would have made this bug +visible long ago, and it would help with mf5to6's leftovers generally. + +**Real parse errors aren't swallowed.** A bad row in a period block, or a bad option, behaves +the same in both `END` styles: a bad row raises and an unknown option loads. Checked on WEL, +RCHA and MAW. The errors raised are `ValueError`s rather than Lark's, so these don't look like +they reach the retry, though I didn't trace it. diff --git a/docs/dev/plans/archive/typed-grammar-issues.md b/docs/dev/plans/archive/typed-grammar-issues.md new file mode 100644 index 00000000..01d8083b --- /dev/null +++ b/docs/dev/plans/archive/typed-grammar-issues.md @@ -0,0 +1,265 @@ +# Typed Grammar Issues + +This document catalogs issues with generated typed grammars that cause fallback to the basic parser. + +## Overview + +The typed parser uses auto-generated Lark grammars for each MF6 component type (e.g., `gwf-nam.lark`, `gwf-chd.lark`, `gwf-disv.lark`). When these grammars have errors or are incomplete, parsing falls back to the basic (untyped) parser. + +The basic parser returns blocks as flat lists of lines, requiring the mapper layer to use heuristic workarounds. The typed parser with `__default__()` returns properly structured dicts, eliminating the need for workarounds. + +## Issue 1: Rule Name Conflicts + +**Component**: GWF (model-level name file) +**File**: `flopy4/mf6/codec/reader/grammar/generated/gwf-nam.lark` +**Error**: `Rule 'list' defined more than once` + +### Root Cause + +The grammar imports a `list` rule from the typed module (for structured list/recarray data), then tries to define its own `list` rule for the LIST filename option: + +```lark +%import typed.list -> list # Line 10: imports 'list' rule +... +list: "list"i string # Line 29: field rule for LIST option +``` + +Lark doesn't allow duplicate rule names, so the grammar fails to compile. + +### Why This Happens + +The grammar generator creates a rule for each field using the field's name. When a field name conflicts with an imported rule name (`list`, `record`, `array`, etc.), we get a collision. + +### Fix Options + +**Option A**: Grammar generator should detect conflicts and rename field rules +```lark +list_option: "list"i string # Renamed to avoid conflict +``` + +**Option B**: Grammar generator should use a consistent prefix for field rules +```lark +field_list: "list"i string +field_newton: "newton"i "under_relaxation"i +``` + +**Option C**: Don't import unused rules (if `list` structured data isn't used in this file) + +### Impact + +- GWF model-level files always fall back to basic parser +- Affects every simulation (all have GWF name files) +- Lower priority since name files are simple (just file listings) + +### Workaround + +The basic parser handles name files fine, just less cleanly. Not urgent. + +--- + +## Issue 2: Missing Keywords in Grammar + +**Component**: NPF (Node Property Flow package) +**File**: `flopy4/mf6/codec/reader/grammar/generated/gwf-npf.lark` +**Error**: `Unexpected token Token('__ANON_2', 'wetfct') at line 3, column 10` + +### Root Cause + +The NPF file contains a `wetfct` keyword that's not defined in the generated grammar. This suggests: +1. The grammar generator missed a field from the definition file +2. The definition file is incomplete +3. The keyword is deprecated or from a different MF6 version + +### Example File Content + +``` +BEGIN options + SAVE_FLOWS + wetfct 0.1 + ... +END options +``` + +The grammar likely has: +```lark +options_fields: (save_flows | ... )* +# Missing: wetfct rule +``` + +### Fix + +Check the NPF definition file (DFN) and ensure all valid options are included in the grammar generation process. Add: +```lark +wetfct: "wetfct"i double +``` + +### Impact + +- NPF files with `wetfct` option fall back to basic parser +- Affects models using wetting/drying functionality +- Medium priority (common in certain model types) + +--- + +## Issue 3: Strict Whitespace Handling + +**Component**: DISV (Vertex discretization) +**File**: `flopy4/mf6/codec/reader/grammar/generated/gwf-disv.lark` +**Error**: `Unexpected token Token('NEWLINE', '\n') at line 8, column 10. Expected one of: END, NVERT, NCPL, NLAY` + +### Root Cause + +The grammar expects dimension fields to appear consecutively without extra whitespace: + +```lark +dimensions_fields: (nlay | ncpl | nvert)* +``` + +But MF6 files often have blank lines or extra spacing: +``` +BEGIN dimensions + NLAY 1 + NCPL 121 + # <-- This blank line causes the error + NVERT 148 +END dimensions +``` + +### Why This Happens + +The grammar correctly ignores whitespace within lines (`%ignore WS`) and comments (`%ignore SH_COMMENT`), but doesn't handle optional newlines between field definitions gracefully. + +The `*` operator means "zero or more fields", but the parser gets confused when it sees a NEWLINE after fields but before END. + +### Fix Options + +**Option A**: Make the grammar more permissive about newlines +```lark +dimensions_fields: (NEWLINE* (nlay | ncpl | nvert) NEWLINE*)* +``` + +**Option B**: Add explicit newline handling in the grammar +```lark +dimensions_fields: (dimension_field)* +dimension_field: (nlay | ncpl | nvert) NEWLINE+ +``` + +**Option C**: Preprocess files to remove blank lines (not recommended) + +### Impact + +- DISV models always fall back to basic parser +- Affects all vertex discretization models +- **High priority** - common model type + +### Workaround + +The basic parser handles DISV files, but loses the clean dict structure benefits. The mapper workarounds compensate. + +--- + +## Issue 4: OC Print Format Keywords + +**Component**: OC (Output Control) +**File**: `flopy4/mf6/codec/reader/grammar/generated/gwf-oc.lark` +**Error**: `Unexpected token Token('__ANON_2', 'COLUMNS') at line 5, column 23` + +### Root Cause + +The OC file has print format keywords that aren't in the grammar: + +``` +BEGIN options + HEAD PRINT_FORMAT COLUMNS 10 WIDTH 15 DIGITS 6 GENERAL + ... +END options +``` + +The grammar likely only has: +```lark +head: "head"i "fileout"i word +``` + +But not: +```lark +head_format: "head"i "print_format"i "columns"i integer "width"i integer ... +``` + +### Fix + +The grammar needs to handle both HEAD FILEOUT and HEAD PRINT_FORMAT variants: +```lark +head: head_fileout | head_format +head_fileout: "head"i "fileout"i word +head_format: "head"i "print_format"i format_spec +format_spec: ("columns"i integer)? ("width"i integer)? ... +``` + +### Impact + +- OC files with print format options fall back to basic parser +- Common in models with detailed output control +- Medium priority + +--- + +## Statistics + +### Fallback Frequency + +To measure fallback frequency, search for warnings in test output: +```python +grep "Typed parsing failed for" test_output.txt | sort | uniq -c +``` + +Expected common failures: +- **Gwf**: Every simulation (name file has `list` conflict) +- **Disv**: Most vertex grid models (whitespace issue) +- **Npf**: Some models (missing keywords) +- **Oc**: Some models (format keywords) + +### Success Rate + +Components with **working** typed grammars: +- CHD (Constant Head) +- DRN (Drain) +- WEL (Well) +- RIV (River) +- GHB (General Head Boundary) +- RCH (Recharge) +- Most other boundary packages + +These benefit from the `__default__()` structured output immediately. + +--- + +## TODO + +1. Fix high-priority grammar issues: + - DISV whitespace handling (affects many models) + - Rule name conflicts in GWF/GWT/GWE name files +2. Validate generated grammars: add CI check that all grammars compile +3. Document known limitations: which keywords/options aren't supported +4. Consider whether basic parser can structure blocks as dicts too? + +## Testing Strategy + +### Verify Typed Parsing Works + +```python +from flopy4.mf6.codec.reader import loads +from flopy4.mf6.gwf.chd import Chd + +# Should NOT see fallback warning +data = loads(chd_file_content, component_type=Chd) +assert isinstance(data['dimensions'], dict) # Clean structure +``` + +### Verify Fallback Still Works + +```python +# Even if typed parsing fails, basic parser should work +data = loads(disv_file_content, component_type=Disv) +# Will see fallback warning, but still get data +assert 'dimensions' in data +``` diff --git a/docs/dev/plans/archive/typed-grammar-remarks.md b/docs/dev/plans/archive/typed-grammar-remarks.md new file mode 100644 index 00000000..6fdd5497 --- /dev/null +++ b/docs/dev/plans/archive/typed-grammar-remarks.md @@ -0,0 +1,308 @@ +# Typed grammar: trailing remarks and performance + +Findings from 2026-09-23. There were two questions: + +- Does the typed grammar (`flopy4/mf6/codec/reader/grammar/typed.lark` plus + the generated per-component grammars) need to keep supporting trailing + remarks, and what does supporting them cost? +- Why isn't the typed loader faster than the basic one, and what would make + it faster? This is covered in + [Why the typed loader isn't faster (yet)](#why-the-typed-loader-isnt-faster-yet). + +## Background + +MF6 input lets most lines end with free text after their real tokens, and it +isn't `#`-prefixed: `CONSTANT 100.0 DELR`. `basic.lark` never models this. Every +line is `item*`, and the structuring code downstream ignores the extra tokens. +`typed.lark` is field-shaped, so it threads an optional `[_remark]` through 7 +rule sites (array control lines, data rows, scalar/record/union/file fields, +block headers). That forces: + +- significant newlines, to bound the remark; +- a `COMMA` terminal that remarks can't contain, so a remark never swallows a + comma-joined data run; +- a low-priority `_REMARK` terminal. + +It also causes one known corpus gap: 2 `gwf-npf` files where a remark contains +a reserved word (e.g. "constant") that lexes as a keyword. + +## Branches + +- `typed-remarks` (off `develop`): opt-in reader benchmarks and this document. +- `typed-no-remarks` (on top of it): an experiment that removes remark support. + It deletes every `[_remark]`, the `_REMARK`/`_remark` definitions and the + template import, and makes the same textual edit to the committed generated + grammars. That edit was checked against a scratch regeneration: only + pre-existing DFN drift differs. **Not for merge.** It exists to measure + things against. + +## Remarks are load-bearing + +The typed grammar was run over every package file of the 239 `KNOWN_PASSING` +corpus models (2,228 files): + +| | failures | +|---|---| +| with remarks (`develop`) | 47 (exactly the `KNOWN_TYPED_GAPS` components) | +| without remarks | 404 | + +357 files newly fail and none newly pass. The 2 `gwf-npf` gap files still +fail, because their remark has nowhere to go. The rejected remarks are: + +- **~90%: labels trailing an array control line**, an MF2005-era convention + that mf5to6-converted inputs carry over. Examples: + - `CONSTANT 100.0 DELR` (146 `gwf-dis` files alone) + - `INTERNAL FACTOR 1.0 IPRN -1 Initial head for layer 1` + - `OPEN/CLOSE f.top FACTOR 1.0 IPRN 5 top layer 1` +- **Option lines**: `PRINT_INPUT (echo input to listing file)`, + `SAVE_FLOWS flow15_flow.cbc`. +- **A few data rows**: `0 0 0 ... row 1`. + +## Remarks cost (almost) nothing at runtime + +Both variants were benchmarked on the same pinned 1,800 files (18.0 MB, 30 +components). These are the files the no-remark grammar still parses. Figures +are the minimum of 3 rounds: + +| stage | with remarks | without | +|---|---|---| +| typed parse | 11.15 s | 11.63 s | +| typed parse + transform | 15.24 s | 15.27 s | +| typed grammar build (30 components, once per process) | 1.71 s | 1.25 s | + +The parse-stage differences are noise. `basic` parse, whose grammar is +unchanged, moved 3-5% between the same two runs. The one real cost is about +0.5 s of grammar construction per process. The transformer never sees +remarks, because Lark drops `_`-prefixed rules and terminals from the tree. + +So the case for dropping remarks rests on grammar complexity and the 2-file +`gwf-npf` gap, not on speed. + +Also: the claim that both lexer-priority tricks exist because of remarks +is only half right. `WORD_STR.-1`'s comment traces it to LALR state merging +between `integer+` rules and a trailing `word`. Only `_REMARK.-1` is +remark-specific. + +## Options for a cheaper remark + +1. **One rest-of-line terminal that can't start numeric**, roughly + `/(?![-+.\d,])[^\n]+/`, instead of a run of word-like tokens. + - The whole remark is one token, so a keyword inside it can't win. This + should close the `gwf-npf` gap. + - A remark can never start at a comma or number, which removes the + remark-related reason for the `COMMA` carve-out. + - Catch: after `CONSTANT 1.0`, a rest-of-line match beats the optional + `IPRN 5` by maximal munch. It needs a lookahead excluding `iprn`, + `factor` and `(binary)`, so some complexity moves rather than + disappears. +2. **Allow remarks only where the corpus has them**: array control lines, + option lines and data rows. Record/union/file fields and block headers may + not need them. Verify each site by re-adding it and diffing the corpus + failure set. +3. **Strip remarks before parsing**, with a per-line pass that knows each + control line's arity. That's cheap for the ~90% on array control lines, + but it's a second, non-grammar parser to keep in sync. +4. **Deprecate.** Keep today's grammar, count and warn when a remark is + consumed (make `_remark` a named rule to see it in the transformer), and + drop remarks in a later input-format version. + +## Reproducing + +```shell +# benchmarks (opt-in; ~4 min) +pixi run -e dev bench +pixi run -e dev bench --benchmark-compare # vs. the last saved run + +# A/B two grammar variants on identical input: write the manifest on the +# stricter variant first, then reuse it on the other +git switch typed-no-remarks +pixi run -e dev bench --bench-manifest /path/to/manifest.txt +git switch typed-remarks +pixi run -e dev bench --bench-manifest /path/to/manifest.txt --benchmark-compare +``` + +The corpus failure-set comparison used a scratch script that runs +`loads_typed` over `_collect_package_files` for every `KNOWN_PASSING` model, +with the `KNOWN_TYPED_GAPS` skip removed. That skip is the only difference +from `test_mf6_typed_grammar_corpus.py`. + +## Why the typed loader isn't faster (yet) + +The typed grammar had two main motivations: + +- better error messages; +- speed, on the tentative reasoning that knowing field types lets logic + move out of post-parse transformation (`structure.py`) and into the parser. + +The first holds up. The second hasn't so far. The measurements below say +why, and point to a version of the idea that could work. + +### Measurements + +**End-to-end basic load (cProfile, whole corpus).** This is +`Simulation.load` over all 239 `KNOWN_PASSING` models: + +| | cumulative time | +|---|---| +| total | 51.8 s | +| `loads_basic` (Lark lex + parse + tree + generic transform) | 45.2 s (~87%) | +| everything else in `structure.py` (`_parse_rows` 3.1 s, griddata 0.65 s, ...) | ~6 s at most | + +cProfile inflates Python-heavy code, but the split is unambiguous. Almost +all load time goes to getting text through Lark. The structuring step the +typed transformer was meant to absorb is at most ~13%. + +**Per-stage split on the five largest corpus files.** Best of 3; "floor" is +one compiled regex over the whole file plus `np.array(..., dtype=float)`: + +| file | MB | basic lex / parse / transform | typed lex / parse / transform | tokens (both) | floor | +|---|---|---|---|---|---| +| `gwf_henry.ghb` | 2.29 | 0.61 / 0.82 / 0.22 | 0.47 / 1.57 / 0.79 | 239,751 | 0.061 | +| `model.disv` | 2.09 | 0.37 / 1.15 / 0.22 | 0.38 / 1.23 / 0.47 | 195,255 | 0.048 | +| `model.npf` | 1.63 | 0.34 / 0.42 / 0.22 | 0.52 / 0.57 / 0.17 | 99,450 | 0.056 | +| `keating.npf` | 0.97 | 0.24 / 0.25 / 0.07 | 0.16 / 0.19 / 0.08 | 65,385 | 0.021 | +| `keating.npf` (2nd model) | 0.97 | 0.13 / 0.20 / 0.07 | 0.15 / 0.41 / 0.07 | 65,384 | 0.023 | + +Times are in seconds. Parse excludes lex, i.e. `parse() - lex()`. + +**Whole-corpus baseline on `develop`.** `pixi run -e dev bench`, 2,149 files +(21.6 MB) both grammars parse, min of 3 rounds: + +| stage | basic | typed | ratio | +|---|---|---|---| +| parse | 12.53 s | 13.45 s | 1.07x | +| parse + transform | 14.64 s | 18.53 s | 1.27x | + +### Diagnosis + +1. **The optimization targeted the wrong ~13%.** Moving structuring into + the grammar could save at most the structuring time. Meanwhile ~87% goes + to lexing and parsing in pure Python, which the typed grammar doesn't + reduce. +2. **The typed grammar doesn't reduce per-token work; it adds to it.** + - Both grammars produce identical token counts, and lexing costs about + the same, since Lark's lexer is Python either way. + - The typed grammar builds more tree per value: `double: SIGNED_NUMBER | + NUMBER` creates a `Tree` node and a transformer callback for every + number. That's why typed parse is ~2x basic on `gwf_henry.ghb`. + - The typed transform builds an `np.array` from a Python list of floats + and wraps each array in an `xr.DataArray`. It also routes most nodes + through `__default__`, which does a `str.rsplit` and dict lookups per + node. +3. **Grammar rules aren't cheaper than Python code.** In a pure-Python + LALR parser, a rule is dispatched by the same interpreter that would run + the equivalent post-processing, plus tree-node overhead. Moving logic + into the grammar can clarify it, but only speeds it up if the move lets + work be skipped entirely. +4. **The headroom is bulk numeric data, and neither loader uses it.** + Almost all bytes in large files are array and list data. Getting the + same numbers out in C (the "floor" column) is 10-30x faster than Lark's + lexer alone, and ~50x faster than the full typed path. Any Python work + per number, in the lexer, parser or transformer, is the ceiling. + +### Is the principle valid? + +- **Better errors: yes, and delivered.** A typed failure names the line and + the tokens that were expected there. The cost is strictness: 357 corpus + files fail once remarks are removed. MF6's own reader is lenient, + line-oriented and keyword-driven, so a grammar stricter than the program + it models will keep chasing fixtures. That's a design tension to manage, + not a bug. +- **"Faster by moving logic into the parser": wrong as stated,** at least + for a pure-Python parser (see diagnosis 3). +- **"Faster because type knowledge lets the parser avoid work": plausible, + and untested.** The typed grammar knows: + - where an array's data starts (after `INTERNAL`); + - how many values it holds (from dims); + - the dtype of each list column. + + That's enough to hand a whole data section to numpy as one token instead + of lexing, parsing and transforming each number in Python. + - Caveat: grabbing runs of numeric-only lines is mostly lexical, so the + basic grammar could do much of it too. + - Where typing uniquely helps: mixed-dtype list rows (int cellids, float + values, a trailing boundname word), expected-size checks, and knowing + what a data run means without a second pass. + +Also not yet measured: a full typed load against a full basic load. Typed +`loads` output isn't wired into `structure.py`, so the benchmark stops at +`loads` for both. + +### Ideas, cheapest first + +**1. Lark's inline transformer, and grammar caching.** +- Passing `transformer=` to an LALR `Lark(...)` runs callbacks as each rule + reduces, and never builds the tree. That targets exactly the tree cost + separating typed parse (1.57 s) from basic (0.82 s) on `gwf_henry.ghb`. +- Things to check: + - Inline mode dispatches by rule-name callbacks. Verify whether + `__default__`, which the typed transformer relies on heavily, and + token callbacks (`BasicTransformer.NUMBER`) are honored there. If not, + the typed transformer needs explicit per-rule methods, or generated + ones: the grammar is generated per component, so the callbacks could + be too. + - The transformer becomes bound to the parser. `get_typed_parser(name)` + and `get_typed_transformer(name, dfn_path)` are cached separately + today, so the parser cache would also need keying on `dfn_path`. + - `__getattr__`'s `typed__` prefix delegation should still work, since + inline lookup goes through `getattr`, but verify. +- `Lark(..., cache=True)` saves the analyzed grammar to disk, cutting the + ~1.3-1.8 s per-process construction of the 30 corpus grammars to loading + a cache file. It matters for CLI/short-lived use, not throughput. Also + check whether `debug=True`, set on both loaders, costs anything outside + development. +- Measure with `pixi run -e dev bench --benchmark-compare`; no manifest + needed, since the grammar doesn't change. + +**2. One token per numeric data block.** +- Add a terminal that matches a whole run of numeric lines, reachable only + where the grammar expects array data (after an `INTERNAL` control line), + and later list rows. For example, roughly: + `DATA_BLOCK: /(?:[ \t]*[-+.\d][-+.\deEdD, \t]*(?:\n|$))+/` +- LALR's contextual lexer only offers terminals valid in the current + state, so it shouldn't collide with `NUMBER` elsewhere. Confirm, since + the grammar already hit a state-merging surprise (see `WORD_STR.-1`). +- In the transformer, parse it with `np.fromstring(block, sep=" ")`, or + `np.array(block.split(), dtype=float)`, after normalizing commas and + Fortran `D` exponents (`1.0D+00`). `flopy4.utils.parse_number` handles + those per value today. +- Check the value count against the expected size from dims. That gives a + better error ("expected 1000 values for STRT, got 999") than any + token-level parse failure, which serves the error motivation too. +- Remarks: a trailing label on a data row ("`0 0 0 ... row 1`") ends the + numeric run. Either let the block regex allow and strip an alphabetic + tail per line, or end the block there and let the existing + `_data_line ... [_remark] _NL` rule handle that line. The block regex is + also a natural place to make remarks cheap, since it runs in C. +- List (period) blocks: rows mix int cellids, floats, and optional + boundname/aux words. Options, roughly in order of effort: + - a per-row regex built from the DFN's column dtypes; + - `np.genfromtxt` or `pandas.read_csv(sep=r"\s+")` on the block text, + both C parsers; + - leave lists to Lark and do arrays only first, since arrays are most of + the bytes in the largest files. +- The basic grammar could adopt the numeric-run terminal for arrays too. + If it does, the comparison between loaders shifts from speed back to + errors and structure. +- Validate on the corpus: `test_mf6_typed_grammar_corpus.py` for parse + parity, the pinned benchmark for speed, and a value-level comparison + against the current transformer output on a sample of files. + +**3. Then decide what structuring belongs in the grammar.** With bulk data +out of Lark, grammar vs. transformer vs. `structure.py` is a question of +clarity and error quality, not speed. Wire typed `loads` output into +structure (or replace it), and add an end-to-end `Simulation.load` stage +to the benchmark, so both loaders are compared on the full path. + +### How these numbers were produced + +- The per-stage split used `parser.lex(text)` (materialized to a list) for + lex, `parser.parse(text)` minus lex for parse, and `transformer.transform + (tree)` for transform. The floor is `np.array(NUM.findall(text), + dtype=float)` with `NUM = r"[-+]?(?:\d+\.?\d*|\.\d+)(?:[eE][-+]?\d+)?"`. + That's a lower bound, not a correct parser: it ignores structure and + would also pick up digits inside words. +- The end-to-end profile was cProfile around `Simulation.load` for every + `KNOWN_PASSING` model, after one warm-up load. +- Both were scratch scripts, not committed. They're easy to reconstruct + from the above. diff --git a/docs/dev/plans/archive/typed-grammar-trailing-remarks.md b/docs/dev/plans/archive/typed-grammar-trailing-remarks.md new file mode 100644 index 00000000..31f1f072 --- /dev/null +++ b/docs/dev/plans/archive/typed-grammar-trailing-remarks.md @@ -0,0 +1,139 @@ +# Should the typed grammar support trailing remarks at all? + +**Archived 2026-10-19.** Answered: keep remarks. Removing them broke 357 more corpus files at negligible parse cost (see [`loader-roadmap.md`](loader-roadmap.md), "What we learned"). + +Raised 2026-09-18 during typed-reader-corpus review; not investigated further, +parking here for an issue. + +## What a "remark" is + +MF6 input lets almost any line end with free trailing text after its "real" +tokens -- not `#`-prefixed (that's `SH_COMMENT`, handled separately). E.g. a +CONSTANT line can read `CONSTANT 1.0 some free-text note`, or a record can +carry extra descriptive words after its real fields. + +## How the two grammars handle it, and why that's asymmetric + +`basic.lark` never models this explicitly: it's fully generic +(`line: item* _NL+`, `item: word | NUMBER`), so any trailing tokens are just +more items -- `structure.py` downstream knows how many tokens matter for a +given line and ignores the rest. No grammar-level concept of "remark" +needed. + +`typed.lark` is field-shaped: every field/record/array/block rule has a fixed +arity, so an explicit `[_remark] _NL` had to be threaded onto nearly every +terminal rule to tolerate trailing free text without rejecting otherwise-valid +files. Call sites: `typed.lark:14,21-24,49` (layered/constant/internal/ +external/timearrayseries/data/open_close_redirect), plus `macros.jinja`'s +`record_field`, `union_field`, `simple_field` (x2), `file_field`, `block_def` +-- 7 separate places. + +## Why that single feature is a real complexity driver + +Supporting a trailing remark forced: + +- Newlines becoming significant (can't blanket `%ignore` them -- a remark + needs a bound to stop at; see `typed.lark`'s header comment). +- A dedicated `COMMA` terminal excluded from `word`'s/`_REMARK`'s character + class and `%ignore`'d separately, so a remark can't out-length a + COMMA-separated data run with no surrounding spaces (confirmed against a + real fixture: TDIS's `"1.00000000,1,1.00000000"` -- maximal munch always + prefers the longer match regardless of priority). +- `WORD_STR`/`_REMARK` lexer-priority tricks (`typed.lark:34-41,44-52`) to + keep numbers/keywords resolving correctly in states reachable from both a + remark and a real field. + +That single feature is also the direct cause of the one still-open real gap +in the typed grammar's corpus results: 2 `gwf-npf` files where a remark's +free text happens to contain a reserved word (e.g. a parenthetical comment +mentioning "constant"), and it loses to that word's keyword literal under +Lark's single-lexer priority rules. Fixing that specific case would need +per-context lexer modes, which Lark's LALR contextual lexer doesn't support. + +## Open question for the issue + +Is trailing-remark support actually load-bearing for the typed path -- i.e. +are there real corpus fixtures that fail to parse *without* it, or does every +real remark-bearing fixture happen to also be tolerated some cheaper way +(e.g. `basic.lark`'s free-form-item approach, or just not exercising the +field it would attach to)? If it's not load-bearing, dropping remark support +from the typed grammar would remove significant lexer complexity (7 grammar +call sites, the COMMA carve-out, and the two priority tricks) and the gwf-npf +gap along with it. If it is load-bearing, is there a lower-complexity way to +bound it than a dedicated lexer terminal threaded through every rule? + +## Findings (2026-09-23) + +Worktree `~/dev/pyphoenix-typed-remarks`. Branch `typed-remarks` (off +develop) has the reader benchmarks. `typed-no-remarks` stacks on top of it +and removes remark support: every `[_remark]`, `_REMARK`/`_remark`, and the +template import. The committed generated grammars get the same edit. + +### Load-bearing: yes + +Across all 2,228 corpus files, typed-parse failures go from 47 to 404: 357 +newly fail and none newly pass. The 2 gwf-npf gap files still fail, since +their remark now has nowhere to go. Where the rejected remarks sit: + +- ~90%: MF2005-style labels trailing an array control line -- + `CONSTANT 100.0 DELR` (146 dis files alone), + `INTERNAL FACTOR 1.0 IPRN -1 Initial head for layer 1`, + `OPEN/CLOSE f.top FACTOR 1.0 IPRN 5 top layer 1`. +- Option lines: `PRINT_INPUT (echo input to listing file)`, + `SAVE_FLOWS flow15_flow.cbc`, `DEV_OMEGA 1D-5`. +- A few data rows: `0 0 0 ... row 1`. + +### Performance: no parse-time case for dropping it + +`pixi run -e dev bench --bench-manifest ` pins both variants to the +same 1,800 files, the ones the no-remark grammar still parses. Min of 3 +rounds: + +| stage | with remarks | without | +|---|---|---| +| parse-typed | 11.15 s | 11.63 s | +| loads-typed (parse + transform) | 15.24 s | 15.27 s | +| typed grammar build (30 components, one-time) | 1.71 s | 1.25 s | + +The parse differences are within run-to-run noise: parse-basic moved 3-5% +between the runs on an unchanged grammar. The only measurable cost is +37% +grammar build, ~0.5 s once per process. The transformer never sees +remarks: Lark filters `_`-prefixed rules and terminals out of the tree, so +none of `transformer/typed.py`'s complexity or cost comes from them. The +typed loader's ~27% overhead vs. basic is in the transform, not the parse. + +So the argument for dropping remarks is grammar/lexer complexity (plus the +2-file gwf-npf gap), not speed. + +## Ideas for a cheaper remark + +1. **One rest-of-line terminal, can't start numeric.** Replace the + `_REMARK+` word run with a single token, roughly + `_REMARK.-1: /(?![-+.\d,])[^\n]+/`. + - Taking the whole remark as one token means no keyword inside it can + win the lexer, so the gwf-npf reserved-word gap should close. + - A remark can't start with a digit, sign, dot, or comma, so it can + never swallow a comma-joined data run. The COMMA carve-out's + remark-related reason goes away (COMMA may still be needed as a data + separator). + - Catch: after `CONSTANT 1.0`, a rest-of-line match beats `IPRN 5` by + maximal munch. It needs a lookahead excluding the optional trailing + keywords (`iprn`, `factor`, `(binary)`). That moves some complexity + rather than removing it. +2. **Only allow remarks where the corpus has them.** Array control lines, + scalar/keyword option lines, and data rows cover everything above. + Record/union/file fields and block headers may not need them. Verify + per site: re-add them one at a time and diff the failure set. +3. **Strip remarks before parsing.** A per-line preprocessing pass that + knows each control line's arity (CONSTANT/INTERNAL/OPEN/CLOSE and their + modifiers) and cuts the rest. That's cheap for array control lines, + which are ~90% of the cases, but it's a second, non-grammar parser to + keep in sync. +4. **Deprecate.** Keep today's grammar, warn when a remark is consumed + (it's easy to count in a transformer if `_remark` becomes a named + rule), and drop remarks in a later format version. Most real + occurrences come from mf5to6-converted legacy fixtures. + +The doc's claim that both priority tricks exist because of remarks is only +half right: `WORD_STR.-1`'s own comment traces it to LALR state merging +between `integer+` rules and a trailing `word`, not to remarks. diff --git a/docs/dev/plans/archive/typed-loader-perf.md b/docs/dev/plans/archive/typed-loader-perf.md new file mode 100644 index 00000000..31e8abc9 --- /dev/null +++ b/docs/dev/plans/archive/typed-loader-perf.md @@ -0,0 +1,1489 @@ +# Loader performance + +**Status (2026-10-19): all of the work below is merged, and optimization +is paused.** The branches named in the sections below landed as #474 +(`numeric-runs`, approach 1), #479 (`item-table`, approaches 5 and 6, +with `list-ingress` folded in), #480 (`load-perf`), #481 +(`load-snapshots`, step 0), #483 (`load-memory`, M1-M4 and the mapped +read) and #484 (`load-simplify`, which merged the readers' number +conversion and `read_table`/`read_ragged` into `read_rows`). Benchmarks +are in `benchmarks/` (#482). What's left is under "Paused: what remains". +Branch names, commits and "no PR yet" notes further down are history. + +Both readers (`loads_basic`, which real loads use, and `loads_typed`, the +per-component generated grammars) make one lexer token per value, and the +typed reader also builds two subtrees per value. On the corpus's big +files this dominates CI. Each `test1002_biscqtg_disv*` case in +`test_load_simulation` takes ~5.3 min on an ubuntu runner (it runs the +typed parse too for known-passing models, see `_check_typed_grammar`). +Those cases set the CI floor even with `--dist worksteal`. + +## Measurements + +Local (macOS), parser already built. + +`biscayne.riv` from `mf6/large/test1002_biscqtg_disv_dev`: 57 MB, 1.27M +list rows (`1 42622 0.245364 76175.2 -4.13766`), ~6.3M values. + +| | lex | parse (lex + tree) | tree nodes | transform | +|-------|--------|--------------------|------------|-----------| +| basic | 10.4 s | 32.8 s | 1.27M | 4.0 s | +| typed | 12.3 s | 78.3 s | 13.9M | 33.6 s | + +The same rows read without Lark: `[l.split() for l in text.splitlines()]` +takes 1.3 s. Converting the five columns with numpy (`astype(int)` / +`astype(float)` per column) takes another 0.4 s. Per-value +`_NUMBER.fullmatch` + `parse_number`, as basic's `TOKEN` callback does, +takes 3.1 s. + +`secp_mf6.npf` from `mf6/large/test1005_secp`: 28.9 MB, 48 INTERNAL +arrays, 1.7M values. + +| reading the arrays with | time | +|--------------------------------------------|---------------| +| basic parse + transform | 6.5 s + 0.8 s | +| typed parse + transform | 9.8 s + 1.1 s | +| `np.array(s.split(), dtype=float)` | 0.10 s | +| `np.fromstring(s, sep=" ")` | 0.14 s | +| one `re.finditer` for the numeric-line runs | 0.31 s | + +`biscayne.disv` (9.4 MB) is **not** an array file: its griddata are all +OPEN/CLOSE, which ingress already reads with numpy +(`converter/ingress/structure.py:153`). Its size is the VERTICES and +CELL2D lists. Basic takes 6.0 s parse + 0.8 s transform, and typed takes +15.5 s parse + 7.1 s transform. + +A `cProfile` of the typed transform on a 100k-row slice of the riv file +(~500k values) shows ~1.1M user callbacks, ~1M +`TypedTransformer.__getattr__` calls, and ~500k `__default__` + +`parse_rule_name` calls, all for the per-value nodes. + +### Where the values are + +The largest corpus files are list rows. The biscqtg `.riv` files are +57 MB each, and there are 8 of them. Files with the most INTERNAL array +values: + +| INTERNAL values | file | +|-----------------|-------------------------------------------------------| +| 1,716,000 | `mf6/large/test1005_secp/secp_mf6.npf` (28.9 MB) | +| 1,347,525 | `mf6/large/test1001_Peterson/v6.evt` | +| 1,179,750 | `mf6/large/test1005_secp/secp_mf6.dis` | +| 1,017,600 | `mf6/test/test016_Keating_disu_mfusg/keating.disu` | +| 898,350 | `mf6/large/test1001_Peterson/v6.dis`, `v6.npf` | +| 795,324 | `mf6/test/test016_Keating_disu_mfusg/kzusg.disu` | +| 582,025 | `mf6/large/test1000_751x751*/mfusg_C.npf`, `.ic` | + +In CI (PR #472, ubuntu 3.11), the array-heavy models cost ~4–5 min of +worker time: Peterson 84 s, secp 66 s, and Keating ~84 s over the load and +round-trip tests. The list-heavy biscqtg cases cost ~32 min (six disv +variants at ~5.3 min each, and two dis variants at ~2.5 min each). + +## Root causes + +1. **One token per value (both readers).** The lexer makes a token for + every number, and each token costs a regex match, a `Token` object and + (in basic) a `TOKEN` callback. That is ~10 s of lex alone on + `biscayne.riv`, before any tree. Building the tree on top of that is + most of basic's ~33 s parse. + +2. **Per-value subtrees (typed only).** In `grammar/typed.lark`: + + ``` + record: token (token | block_kw_word)* _NL + token: number | word + number: SIGNED_NUMBER | NUMBER + word: WORD_STR | QUOTED + ``` + + Every list value becomes `token -> number|word -> TERMINAL`, two + subtrees per value (~11 nodes per 5-value row vs basic's 1 `line` + node). Arrays have a smaller version of this: `_data_line: double+` + is one `double` subtree per value. + +3. **Slow per-node dispatch (typed only).** Rules imported from + `typed.lark` are namespaced in the generated grammars (`typed__token`, + `typed__number`, `typed__double`, ...). Lark looks up a callback by + that name. The lookup misses and hits `TypedTransformer.__getattr__` + (`transformer/typed.py:49`), which strips the prefix and retries. + `token` has no callback at all, so that lookup raises + `AttributeError` and Lark falls back to `__default__` + (`typed.py:302`). That runs `parse_rule_name` on the rule name, finds + no tags, and builds a `Tree` anyway. `record` (`typed.py:220`) then + unpicks those `token` trees by hand. All of this happens once per + value. + +4. **Building the tree at all.** Both readers build a full parse tree and + then transform it. + +## Approaches + +In order. Approach 1 removes cause 1 for the big files and most of 2–4 +along with it, because the per-value tokens and nodes are gone. Approaches +2–4 still help the typed reader and smaller files, but they matter much +less once 1 lands. Measure again after 1 before you start them. +Approach 5 (list rows by column, in ingress) came out of profiling after +1. It's what's left of biscqtg's load time, so it comes next. Approach 6 +(store tabular lists as columns) came out of benchmarking after 5. It's a +change to how packages hold list data, not just to loading. + +### 1. Lex runs of values as one terminal, convert with numpy (both readers) + +**Status: step 1 done for the basic grammar** (branch `numeric-runs`, +commits f4b44eeb + 7fe4b112 (rename to `NUMBER_LINES`), pushed to the fork, no PR yet; worktree `.claude/worktrees/numeric-runs`). The `NUMBER_LINES` +terminal is in `basic.lark`, and its callback in `transformer/basic.py` +gives the same rows as `line`. It also takes the blank and comment-only +lines after a numeric line, as `line`'s `_NL+` does. Without that, a +comment after a run gave an extra empty row (seen in +`test001a_Tharmonic_tabs/flow15_constant.chd`). Output matched the old +grammar exactly, by `repr`, on 143 files from 10 sample models and on 8 +edge-case strings. The basic parse on that sample went from 100 s to +7.6 s. Full `Simulation.load`, old → new: + +| model | total | basic parse | rest | +|---|---|---|---| +| test1005_secp | 19.8 → 2.4 s | 18.6 → 1.1 s | 1.2 s | +| test1001_Peterson | 23.5 → 5.5 s | 20.0 → 1.9 s | 3.5 s | +| test1002_biscqtg_disv_dev | 73.0 → 30.8 s | 44.6 → 2.9 s | 28 s | + +Parsing is no longer the cost. Step 2 below (numpy in ingress) would +save little, since converting values isn't what's slow. biscqtg's 28 s +is building one object per list row, so approach 5 is next. The typed +grammar's version of this is still to do. + +Lex a whole run of number-only lines as **one** token, then convert it in +bulk. numpy can parse the 1.7M secp values in 0.1 s, and Python can split +the 1.27M riv rows in 1.3 s. Lark spends 7–11 s and ~37 s on those files. + +**Basic grammar first.** It is the real load path (Simulation.load → +`Package.load` → `_codec_load` with no component), so it helps users and +every load test. + +- Add a terminal for one or more consecutive lines made of numbers only, + e.g. (sketch, not checked against every edge): + + ``` + NUMBER_LINES.2: /([ \t]*[+-]?(\d+\.?\d*|\.\d+)([eEdD][+-]?\d+)?([ \t,]+[+-]?(\d+\.?\d*|\.\d+)([eEdD][+-]?\d+)?)*[ \t,]*\r?\n)+/ + ``` + + Then allow it in a block body: `_list: (line | numeric_run)*`. The + value pattern must accept exactly what `_NUMBER` in + `transformer/basic.py` accepts (including Fortran `D` exponents), so no + value changes type. +- This one terminal covers INTERNAL array data (and keyword-less data + rows) **and** all-numeric list rows like `biscayne.riv` and DISV + VERTICES/CELL2D. Any line it can't take falls back to `line` as it + does today, so correctness doesn't depend on catching every case. Lines + that fall back include rows with a boundname, a quoted value, a + comment, a remark, or a keyword. +- **Lexer behaviour to check first.** Lark's basic/contextual lexer + joins the allowed terminals into one regex alternation sorted by + priority, then by pattern width. It is not true longest-match, so the + run terminal needs a priority above `TOKEN` (0) and `QUOTED` (1) to be + tried first. With the contextual lexer it only competes in states where + a body line can start. The regex must end at a newline, so it can never + take part of a line and leave the rest. Check `BEGIN`/`END` (an `END` + line isn't numeric, so it ends the run), `\r\n` endings, blank lines + inside a run (common's `NEWLINE` swallows several newlines in one + token), and leading whitespace. Also check the tokens before the + run: the block name line's trailing `_NL` must not eat the run's first + line. +- **Representation, in two steps:** + 1. The callback (`def NUMBER_LINES(self, tok)` with + `visit_tokens`, or a `numeric_run` rule callback) expands the run + into the same `list[list[int | float]]` rows `line` gives today. This + needs no ingress change and removes the parse cost, which is the big + part (~33 s → a few seconds on riv). Convert per row with + `str.split` + a fast int/float conversion. Keep `parse_number`'s + semantics: an integer-looking token stays `int`, `1.0` stays + `float`, and `D` exponents work. + 2. Hand ingress the raw run instead (e.g. a small `NumericRun(text)` + wrapper in the block's row list), and convert it where the dtype is + known. `_read_control_record` (`structure.py:271`) can then do + `np.array(text.replace(",", " ").split(), dtype=dtype)`, or the same + with `D`→`E` when the text has a `D`. That is what + `_read_open_close_values` already does at `:153`. It must still read + exactly `length` values and continue after them. One run can hold + the end of one array's data, but not the next control record, which + isn't numeric. List-row ingress can convert by column + (`np.array(rows)` → `astype` per column) instead of by value. This + step touches every consumer of block rows, so do it only where the + profile says the per-row conversion still matters. +- **Memory:** a run token holds the whole run's text, which can be tens + of MB. That's fine, because it is smaller than millions of `Token` + objects. + +**Typed grammar after.** Replace `data: _data_line+` with a run terminal +and convert it in `data()` with one `np.array(..., dtype=float)`. This +covers `_data_line`'s `[_remark]` by fallback, or keep `_data_line+` as +an alternative for remarked lines. Typed list rows (`record`) could take +the same run terminal, giving rows of numbers, but this interacts with +the generated grammars' per-field rules. Look at how `stress_period_data` +and the `*__fields` rules use `record` before you choose. + +**Edge cases the current grammars handle that the run must too (or fall +back on):** comma separators (`1.0,1,1.0`), trailing `#`/`!` comments +and bare remarks on a data line, `D` exponents, a value like `1.` or +`.5`, signed values, and the whole-token rule (a token like +`1997-07-16T19:20:30` is a word, not `1997` and so on). The run regex's +per-value lookahead is what enforces that rule, as `(?![^\s,#!])` does +for the existing terminals. + +### 2. Inline the per-value rules (typed only) + +Make the per-value rules inline so their one child goes straight into the +parent: `?token: number | word`, and the same for `number`, `double` and +`integer` where possible (`?rule` inlines when it has one child, which +these always do). Then convert terminals where they land instead of in +per-rule callbacks: + +- `record` gets raw `Token`s (`NUMBER`, `SIGNED_INT`, `WORD_STR`, + `QUOTED`, namespaced as `typed__NUMBER` etc.) and converts each by type + (`parse_number` for numeric terminals, `unquote` for words). +- Or use terminal callbacks (`visit_tokens=True` with + `def NUMBER(self, tok)`, plus the `typed__` names). That is cheaper + than a subtree but still one call per value. + +Watch out: + +- Inlined rules never call their own callback, so the conversion in + `number()`/`double()`/`integer()`/`word()` (`typed.py:184-200`) must + happen somewhere else for every place these rules appear. In the + generated grammars, `word` is used 561 times, `integer` 434, `double` + 374, `number` 258 and `string` 357 (`token` appears only via + `record`). Options fields like `maxbound__val: "maxbound"i integer` + rely on `integer` returning an `int`. +- `double`, `number` and `integer` mean different conversions (always + float, int-or-float, and int). An inlined `double` loses that unless + the parent knows its field type. It may be simplest to inline only + `token` and `number` inside `record`. Approach 1 already covers + `_data_line`'s `double+`. +- The transformer types each value in a record by whether it is a + `word` or a `number`. Keep the conversion behaviour identical: ints + stay ints, `1.0` stays float, and quoted strings are unquoted. + +Expected: a typed tree size near basic's, and most of the 33.6 s +transform gone, for rows approach 1 doesn't take. + +### 3. Make transformer dispatch cheap (typed only, small) + +Every remaining namespaced node pays the `__getattr__` miss. Instead: + +- Define the `typed__*` callbacks explicitly (e.g. bind + `typed__record = record` etc. on the class, or generate them in a loop + at class-creation time) so Lark's lookup hits directly. +- Cache `parse_rule_name` results (`grammar/names.py:40`) by rule name, + e.g. with `functools.lru_cache`. There are only a few hundred distinct + rule names. +- Give `token` a callback if it stays a subtree, so it never goes through + `AttributeError` + `__default__`. + +Cheap to do and safe. It can land with any of the others. + +### 4. Transform while parsing (both readers) + +With LALR, `Lark(grammar, parser="lalr", transformer=t)` calls the +callbacks as rules reduce and never builds the tree. After approach 1 +the trees are much smaller, so this is worth less. Profile first. + +Caveats to check first: + +- Lark builds the in-parse callbacks with `getattr(transformer, + rule_name)` and builds a plain `Tree` when that's missing. As far as I + know it does **not** route missing names to `__default__`. If so, the + typed transformer's `__default__`-driven generated rules (`*__block`, + `*__fields`, `*__val`, ...) would come back as Trees, and + `__getattr__` would need to return a `__default__`-backed callback for + every name. Verify this against the installed Lark version + (`lark/parse_tree_builder.py`). +- In-parse callbacks get only `children` (no `meta`) unless wrapped with + `v_args`. Check that `__default__`'s use of `meta` still works. +- The parsers are cached and shared (`get_typed_parser` is + `lru_cache`d, and `BASIC_PARSER` is module-level), so the transformer + must be stateless. `_TYPED_TRANSFORMER` looks stateless now. Keep it + that way. +- A parser built with a transformer can't also return a plain tree. + Check that nothing calls `.parse()` and expects a `Tree`, including + tests and the trailing-garbage retry in `loads_basic`/`loads_typed`. +- Error behaviour: an exception inside a callback now fires mid-parse, + not after. The `LarkError` catch + `_trim_trailing_garbage` retry in + `codec/reader/__init__.py` must still behave. + +### 5. Build list data by column, not one Item per row (ingress, both readers) + +After approach 1, biscqtg_disv's load is 2.9 s of parse and 28 s of +everything else. The time goes to `biscayne.riv`'s ~1.27M period rows +(1.51M list rows in all). From a cProfile of `Simulation.load` (97 s +under the profiler): + +- `_parse_rows` (`converter/ingress/structure.py:51`) calls + `Item.from_tokens` once per row: 77 s cumulative. + - `_layout` (`item.py:460`), 53 s. This includes `solve`/`first` + (`:602`/`:620`), 24 s, and `options` (`:507`), 14.6M calls. + - The attrs `__init__` of `Riv.StressPeriodData`, 19.5 s. This + includes `__attrs_post_init__` → `_counted_fields` (`:87`), + `_sized_by_field` and `attrs.fields_dict`, 13M calls. + - `_convert` (`:663`), 16 s, and `_has_boundname_field` (`:126`), + 6 s. +- `Package.__attrs_post_init__` (`package.py:75`), 15.5 s. This includes + `_set_child_parents` → `_period_row_children` → `_row_children` + (`component.py:140-161`), 9.7 s, and `_init_item_lists` + (`package.py:127`), 5.8 s. + +Almost all of that work is the same for every row of a **tabular list**. +That is a list block whose rows all have one record type: no unions or +keystrings, no child columns, the same width in every row (riv's period +data, packagedata, DISV's VERTICES/CELL2D). Its rows share a layout: the +same columns, the same cellid width, and whether there's a boundname. +Their values don't change it. + +- **Scope: list packages only.** Grid- and layer-array packages (RCHA, + EVTA, the grid-based CHDG/WELG and so on) already hold arrays and read + them by column through array ingress (`_read_control_record`). They + never build per-row Items, so this doesn't touch them. +- **What it builds:** the same `list[Item]` the generated class holds + (`Riv._stress_period_data: dict[int, list[StressPeriodData]]`), just + built faster. Going straight to columns would mean changing that + representation, which is out of scope here. The xarray form in + [`period-lists.md`](period-lists.md) is a view derived from the list (`to_xarray`), + not storage, so it doesn't change this. +- **Fast path:** lay out the first row with `_layout` as now and check + that the block is a tabular list. Every row must have the same token + count, and every value column must be numbers only. A time-series name + is a word in a `Union[float, TimeSeriesRef]` column, so a block with one + takes the slow path, and a fixed-position boundname word is fine. If the + block qualifies, convert by column once (`np.array(rows)` or per-column + slices → int cellids minus 1, floats). Then make each Item with a cheap + constructor that skips `_layout` and the per-row `__attrs_post_init__` + checks: one `cls(**kwargs)` from precomputed column slices, or + `object.__new__` plus field sets if attrs validators/converters cost + too much. Anything that doesn't qualify falls back to `from_tokens` per + row, as now. Profile the cheap constructor first. If attrs `__init__` + alone is most of the 19.5 s, the Item-per-row representation is the + floor, and changing it is a separate decision. +- **Per-row checks that move to per-block:** `_counted_fields`, + `_has_boundname_field` and the `fields_dict` lookups depend only on + the class, so cache them per class (`functools.cache` on the class) + whatever else happens. That alone is a cheap first step. Measure it + before the column path. +- **`_set_child_parents`:** `_row_children` walks every row's fields to + find child components. A class with no child fields (directly or in a + nested record) can't have any, so check that once per class and skip + the walk. +- **Validating:** the equivalence check compares the loaded package's + rows, or its period data, old vs new, on riv/disv (biscqtg), wel/chd + with boundnames, a list with unions (SFR/MAW/LAK), and a list with + child columns (LAK's TAB6). Then run the load tests for those models. + +**Status: done, differently from the fast path sketched above.** It's on +branch `list-ingress`, commit `77b67374`, in the +`.claude/worktrees/numeric-runs` worktree, rebased onto develop `6f456c65` +(after #474, approach 1) and pushed to `origin/list-ingress` (no PR yet). +The rebase kept #473's conditional-options check in `_coerce_item_list`, +for loaded Items too, checking only options not set yet. It merges cleanly +with `period-lists` (#476). + +- **Not by column.** Profiling after the caching step showed attrs + `__init__` is cheap: 2.9 s profiled for the 1.27M riv rows, so it isn't + the floor. The cost was `_layout`'s search plus `_convert`'s generality. + So rows keep being read one at a time, but by a plan: + - `_layout` is split into `_solve` (the search) and `_convert`. + - `Item.from_rows` lays out the first row of each length with `_solve`. + `_plan` turns that layout into column positions and a reader for each + field. + - Each later row of that length is read by the plan, then built with + `cls(**kwargs)`. +- **Which classes:** `_plannable` admits a class only if its layout depends + on the number of tokens alone: no keyword, flag, count column (cell2d's + ncvert), untagged unions, child column, or union-arm `rest`. A boundname + is fine, and so are time-series names: the reader converts them as + `_coerce` does, so they don't force the slow path. +- **Fallback:** a row the plan's readers can't convert (a NONE cellid as + wide as a DISU node) falls back to `from_tokens`. +- **Also in the commit:** + - `_counted_fields` and `_has_boundname_field` are cached per class. + - `_row_children` skips classes that `_may_hold_children` says can't + hold a child. + - `_coerce_item_list` skips `_row_context` when rows are already Items. + That was a 5 s profiled hidden cost: `resolve_dims` was walking all + 1.28M rows per period, with a runtime-Protocol `isinstance` on each. +- **Equivalence:** I loaded each sim with the plan off and on and compared + every field. All 16 copied models matched: the 10 earlier sample models, + plus lake1ss_table (TAB6), sfr_mvr (wel boundnames, SFR, MVR), + mawconfined, NWTP3Low_MAW, lake2tr, and mvlake_laksfr_tr. A deliberate + perturbation of the fast path's readers showed up as diffs, so the check + bites. +- **Tests:** `test_rows_as_each_row_reads` was added. The codec, + component, roundtrip, reader, dimensions, load-context and namefile + tests pass. + +| Model | Approach 1 | + approach 5 | +|---|---|---| +| `mf6/large/test1002_biscqtg_disv_dev` | 30.8s | 8.0s | +| `mf6/large/test1001_Peterson` | 5.5s | 4.4s | +| `mf6/large/test1005_secp` | 2.4s | 1.7s | + +What's left in biscqtg's 8 s: +- the basic parse, 2.7 s, of which `NUMBER_LINES`' callback is about 1 s; +- 106k rows of DISV's CELL2D, which still take `from_tokens` because it + has a count column (ncvert), about 1.8 s; +- `_file_cellid` and the riv attrs `__init__`, about 1 s each. + +A count column could be planned when the count is the only thing that +varies by value, but that's not done. + +### 6. Store tabular lists as columns, not one Item per row + +Approach 5 still converts each value in Python (the `NUMBER_LINES` +callback, then the plan's readers) and still makes one Item per row. +Converting by column only pays off fully if the columns are what's +stored. + +**Benchmark** (`biscayne.riv`'s 100 period blocks, 1,267,300 rows; each +path's output checked equal to A's; lexing not included, it's the same for +all three; script `scratchpad/runs/colbench.py`): + +| Path | Steps | Time | +|---|---|---| +| A. now (approach 5) | `NUMBER_LINES` callback 1.29 s + `from_rows` 2.40 s | 3.69 s | +| B. columns → Items | `np.fromstring` + reshape 0.46 s, `astype` per column ~0 s, `tolist` + `cls(**kw)` 1.86 s | 2.32 s | +| C. columns stored | `np.fromstring` + `astype`, no Items | 0.47 s | + +B saves ~1.4 s, and its floor is making the Items (1.86 s). C saves ~3.2 s +on riv alone, before the per-row costs after ingress +(`__attrs_post_init__`, item-list init, writing) that it also removes. + +**Why not just use the grid packages.** CHDG/DRNG/GHBG/RIVG/WELG +(`readarraygrid`) and RCHA/EVTA (layer arrays) already store +`dict[kper, array]`, but dense over the grid. A RIV file is list input and +loads as RIV, and the dense form can't hold two rows in one cell, or +per-row boundnames or time-series names. Columns are the sparse +counterpart. They still follow the DFN: a list block is a record type with +a column per field, and Item-per-row and array-per-field are two layouts +of that one schema (flopy3's recarrays were the column layout). + +**Container.** `ItemTable[X]`, one per period: a numpy array per field of +`X`, keyed by field name. +- `cellid`: `(n, ncelldim)` int, 0-based; NONE as a sentinel + (`FILL_INT64`). +- Value columns: float64. `aux`: `(n, naux)`. +- `boundname`: a str array, or None if no row has one. +- Time-series columns: floats with NaN where a series applies, plus a + parallel array of series names, as [`period-lists.md`](period-lists.md) lays out for + xarray. + +That's close to period lists' `sparse=True` form, so `to_xarray` becomes +almost a copy. + +**Scope.** Only classes `_plannable` admits. Union rows (SFR/MAW/LAK +periods, OC), child columns (TAB6) and count columns (CELL2D's ragged rows) +stay `list[Item]`. Two representations is the main design cost, but it's +the split approach 5 already makes. + +**Row access, in-place edits included.** `ItemTable` acts as a +`Sequence[Item]`: `len` (so `maxbound` works), iteration, `==` against a +list of Items, and `t[i]` returning a **row view** that writes back. A +prototype (`scratchpad/runs/rowview.py`) shows this works with attrs: +- The view class is a subclass of `X`, made once per class, with + `__slots__ = ("_table", "_i")` and a property per column field. It's + built with `object.__new__`, skipping `__init__`. +- `isinstance(view, X)` holds, so code that checks the Item class keeps + working. +- `view.stage = 3.5` and `view.cellid = (1, 9)` go through attrs' + generated `__setattr__` (its `on_setattr` converters and validators), + then the property writes into the column. +- Making 1M views takes 0.23 s, so building one per access is cheap. +- **Equality fails.** attrs' `__eq__` requires `other.__class__ is + self.__class__`, so a view never equals a plain `X`. The view class needs + its own `__eq__` comparing field values against `X` or another view. +- **Still to decide:** + - assigning a `TimeSeriesRef` into a float column (it also has to set + the names array); + - a value for a column that's absent (`boundname` when None: allocate + it); + - a cellid of another width; + - `append`/`insert` (concatenate; O(n), fine for occasional edits); + - `t[i] is t[i]` is False. + +**Load path.** +- `NUMBER_LINES` also keeps the token's text. +- For a tabular class with a fixed row width, ingress runs `np.fromstring` + and slices columns at the `(a, b)` positions `_plan` already computes for + each field. +- Blocks with words in them (boundnames, time-series names) go through a + str array and convert by column: still a table, just slower. +- The typed transformer (`codec/reader/transformer/typed.py`) needs the + same. + +**Uses of item lists to switch over:** +- Writing: `converter/egress/unstructure.py` calls `to_tokens` per row; + write columns in bulk instead. +- User input: `Package._coerce_item_list`/`_items_from` (Items, tuples, + dicts, dict-of-lists, DataFrames) all produce a table. Dict-of-lists is + columns already. +- `to_dataframe`/`from_dataframe` become nearly trivial. +- `period_lists.py` (PR #476) reads rows with `getattr` per row: read + columns instead. +- Boundnames and conditional options: one test per column. +- `_row_children` skips tables. The dimension walk treats a table as one + element. +- `adapters.py`: flopy3 wants recarrays, so it gets simpler. +- Codegen: the 95 `list_(` fields' types go from `list[X]` to + `ItemTable[X]` (or `Sequence[X]`). + +**Validating.** The plan-off/plan-on equivalence script (`eqfast.py`), +comparing through `ItemTable == list[Item]`, plus the codec, roundtrip and +period-lists tests. + +**Order.** +1. `ItemTable` plus the load path, turned back into Items at the end so + nothing downstream changes. That's B's ~1.4 s. +2. Store tables, with row views, and switch the uses above over one at a + time: writing, DataFrames, `to_xarray`. +3. Codegen types last. + +**Progress** (branch `item-table`, on `list-ingress`, on develop +`49f9daf3`; both pushed to the fork, no PRs). +- Step 1, `e9d68f35`: `NUMBER_LINES` keeps its text (`NumberRows`, + converted only when read); a block of only such lines reaches + `_parse_rows` as is, and `read_table` reads it by column. +- Step 2, `bf5fbbf6`: tables are stored, loaded or given (any class + `_plannable` admits). `ItemTable` is a `MutableSequence`. + - A row is a view: a subclass of the Item class, of the same name, with + a property per field. It sets through attrs' converters and + validators, compares equal to the Item, and copies or pickles as one. + - A value a column can't hold (a time-series name, a boundname) makes + that column objects. Signed-index fields (SFR's `ic`) aren't read by + column. + - Switched over: writing (`ItemTable.tokens`), `to_dataframe`, setting + options and dimensions from rows, the time-series and boundname + checks, the dimension walk. + - Fixed on the way: SFR's `_check_lookup_counts` scanned packagedata per + row (quadratic; with views, Peterson's load took 36.6 s). It now + indexes the rows once. + - biscqtg: load 8.7 s → 4.0 s, write 10.4 s → 3.1 s. Loads equal and + written files byte-identical on all 16 corpus models (scripts + `eqtable.py`, `eqwrite.py`, `eqframe.py`). + - Behavior changes: a user's Items are copied into the columns, so + editing one afterwards doesn't change the package (edit + `pkg.stress_period_data[k][i]`); `t[i] is t[i]` is False; a + non-period table (packagedata) is a leaf in `attrs_to_datatree`, so a + dataset attribute, not a node per row. + +**Next: `period_lists` (#476) by column.** It loops over rows +(`to_dataset`'s events, `_to_sparse`'s `put`), which views make slower. +On biscqtg's RIV (1,267,300 rows, `duplicates="last"`; script +`xrtime.py`): + +| `to_xarray` | Item lists | tables | +|---|---|---| +| dense | 3.9 s | 6.8 s | +| sparse | 8.1 s | 13.4 s | + +Give it a column path for `ItemTable`s, keeping the row loop for lists of +unions and plain Item lists: +1. Sparse. Concatenate each column across periods, `per` from the period + lengths. Split cellid `(n, ncelldim)` into the cell coordinates and aux + `(n, naux)` into one variable per name. Object columns become NaN plus + `_ts`. +2. Dense. Positions for a whole column at once from the cellid columns. + Fill each period's state with fancy indexing. Duplicates by `np.unique` + on positions: `first`/`last` by the index kept, `sum` with `np.add.at`, + `raise` on any repeat. +3. `from_dataset` builds tables directly. + +Validate: `Dataset.identical` between tables and the same package +`to_items()`'d, dense and sparse, every `duplicates` mode, on the corpus; +plus `test_mf6_period_lists.py`. + +**Still open after that:** the typed transformer (`typed.py`) doesn't +keep block text; `adapters.py` could build recarrays from the columns; +step 3 (codegen types). + +### Test-side (not an optimization) + +The nine `test1002_biscqtg*` variants are the same big grid. The typed +parity check could run on one of them only, while the full load still +runs on all of them. That is independent of the above, and it's the +cheapest way to cut CI time if the loader work stalls. + +## Status and next steps (2026-10-15) + +Approaches 1, 5 and 6 are on `item-table` (`fa5103ba`), approach 1 in +both grammars. Measured locally on the 16 scratch corpus models, parser +built and imports warm. + +Parse only, `loads_basic` vs `loads_typed`, 174 package files: basic +parses all of them, and typed fails 6 (`sln-ims` ×5 and `gwf-lak` in +mvlake, the known gaps). On the 168 both parse, basic takes 3.85 s in +total and typed 6.17 s. Typed is within 0.8–1.3× basic on the big array +files (e.g. `biscayne.riv` 0.82 s vs 0.85 s, down from 32.8 s / 78 s). +Most of the remaining difference is per-component parser builds +(0.02–0.15 s each) plus one file: + +- `v6.sfr` (Peterson, 3 MB): basic 0.70 s, typed 2.07 s. Its + packagedata/connectiondata mix words and ragged rows, so they miss + `NUMBER_LINES` and go through `record` per value. + +Full `Simulation.load` vs flopy3 3.11 `MFSimulation.load`. flopy3 reads +OPEN/CLOSE arrays on access, so the second flopy3 time also reads every +data item: + +| model | flopy4 (parse) | flopy3 | flopy3 + read all | +|----------------|----------------|--------|-------------------| +| biscqtg_disv | 3.96 s (1.00) | 8.84 s | 10.76 s | +| secp | 1.41 s (1.02) | 2.38 s | 2.46 s | +| Peterson | 3.21 s (1.71) | 2.90 s | 3.02 s | +| 13 small ones | 0.01–0.03 s | 0.01–0.07 s | | + +Next: + +1. Profile Peterson's load. It's the one model where flopy4 isn't + faster: half of its 3.1 s is parsing (`v6.sfr` 0.7 s of that), plus + 1.4 s after the parse. Find out what the 1.4 s is (SFR ingress? + structuring?). +2. Typed `record` path for mixed/ragged list rows (the SFR gap). Either + a run terminal that allows words, or splitting such rows as text in + the transformer the way `NumberRows` does. +3. Re-check the CI time of the `test1002_biscqtg*` cases once the + branch's CI has run, and decide whether the test-side trim below is + still worth it. +4. Later, optional: deferring OPEN/CLOSE reads until access, as flopy3 + does. It doesn't help Peterson (only its DIS is external). +5. Deferred, as before: `adapters.py` recarrays, and step 3 (codegen + types). + +Scripts: `scratchpad/runs/tvb.py` (parse, per file), `vs3.py` and +`vs3full.py` (vs flopy3), all in the session scratchpad. + +### Done (2026-10-15, later) + +On `item-table`, local commits `b6889388`..`f7f111de`. Loaded values are +identical to `fa5103ba`'s on all 16 corpus sims (2719 fields digested), and +typed output is identical on all 174 files. + +- **Step 1, profiled Peterson.** Two things after the parse: + - Griddata number lines were flattened into Python lists and walked + token by token (`_numeric_prefix`). Array blocks now keep each run + as a `NumberRows` part (`BlockRows`) and read its text with numpy. + - SFR's 41,844 period rows (`1 inflow 0.0`) each had their layout + solved alone. `from_rows` now plans rows ending in a keyword union by + length, plus each arm per length. +- **The same profile on biscayne** found its DISV CELL2D rows (106k, + icvert counted by ncvert) also solved one by one. They're now planned + by length, checking each row's count against the plan. +- **Step 2, typed rows with words.** Tried both: + - A, a run terminal `RECORD_LINES`: v6.sfr typed 2.03 s -> 0.15 s + (basic 0.80 s), corpus typed total 6.17 s -> 4.24 s. + - B, one token per line: 0.38 s, 4.65 s. + Kept A. A list sharing its block with other fields (sim-nam's + solutiongroup, with MXITER) uses `token_record`, lexed by value. + Grammars regenerated from the cached DFNs at modflow6 `524d1ace`, which + reproduce the committed ones exactly. +- **Step 4, deferring OPEN/CLOSE reads:** not done. They cost flopy4 + 0.08 s of biscayne's load and 0.02 s of Peterson's (numpy reads), so + deferral isn't worth a lazy-field mechanism. + +| model | flopy4 before | flopy4 now (parse) | flopy3 | +|--------------|---------------|--------------------|--------| +| biscqtg_disv | 3.96 s | 2.77 s (1.01) | 9.02 s | +| secp | 1.41 s | 1.12 s (0.63) | 2.44 s | +| Peterson | 3.21 s | 2.16 s (1.28) | 2.97 s | + +Noticed: the typed sim-nam transform keeps only `mxiter` from a +solutiongroup block and drops its rows (at `fa5103ba` too). Fixed below. + +### Done (2026-10-15, evening) + +Local commits `366fa2f5`, `b875d8e8` on `item-table` (not pushed). Loaded +values are still identical to `fa5103ba`'s on all 16 corpus sims. + +- **Fix, typed solutiongroup rows** (`366fa2f5`). `_fields` skipped a + list's rows (not a `(name, value)`) among named fields, so a + SOLUTIONGROUP with MXITER lost its solutions. Rows are now kept under + `stress_period_data` whether they come before or after MXITER. + `test_typed_loads_record_lines` covers both orders. +- **`WORD_LINES` in basic.lark** (`b875d8e8`). Real loads parse with basic, + where lines with a word on them (SFR's `1 inflow 0.0`) still took a token + per value; #474's `NUMBER_LINES` only takes number-only lines. + `WORD_LINES.1` takes a run of lines that each have a non-number token, + beside `NUMBER_LINES`. Kept as two terminals, not one, so the lexer + (C regex) does the number/word split that downstream needs anyway (array + data read from `NumberRows` text, headers as word rows), and #474's path + is untouched. Lines it can't take whole fall back to `line`, as before: + - first token starting with `begin`/`end` (a prefix, not only the whole + word, so they lex exactly as before); + - a quote anywhere (quoting forces a string); + - a trailing comment. + + `BasicTransformer.WORD_LINES` splits each line and converts each value + with the same `_value` as `TOKEN` (`D` exponents included); `block` + splices the rows in. Test: `test_loads_word_lines` in + `test_mf6_codec.py`. + +| model | flopy4 before (parse) | flopy4 now (parse) | flopy3 | +|--------------|-----------------------|--------------------|--------| +| Peterson | 1.95 s (1.29) | 1.41 s (0.70) | 2.89 s | +| biscqtg_disv | 2.62 s (1.03) | 2.62 s (1.01) | 8.99 s | +| secp | 1.00 s (0.63) | 0.98 s (0.61) | 2.43 s | + +`v6.sfr`'s basic parse alone: 0.68 s -> 0.09 s. "Before" is today's re-run +of the previous commit (this machine varies; it was 2.16 s for Peterson +earlier). flopy4 is now faster than flopy3 on all three. + +Next: + +1. Profile what's left after the parse (Peterson 0.7 s, biscayne 1.6 s): + ingress conversion and structuring. +2. Optional: let `WORD_LINES` take lines with quotes or trailing comments. + No large corpus file seems to need it. +3. Remaining parse time is mostly array runs plus Lark's per-line + overhead; look at it only if the profile points there. +4. Unchanged: re-check CI time of `test1002_biscqtg*`; `adapters.py` + recarrays and codegen types stay deferred. + +Gotcha: in zsh, `$D` with several sim dirs isn't word-split, so `vs3.py $D` +passes one path (only the last name prints). List the dirs out. + +### Done (2026-10-15, night) -- pushed to the fork at 10a8d23e + +Profiled after the parse (Next #1 above). Commits (all pushed: +fa5103ba..10a8d23e on wpbonelli/pyphoenix-project item-table): + +- **2acf0b6b `perf(reader): possessive NUMBER_LINES`**. The lexer is + still most of the parse, and nearly all of it is `NUMBER_LINES` scanning + array data (~7.5 ms/MB; Peterson 78 MB, biscayne 67 MB). Every quantifier + in the number pattern is now possessive. Equivalent because a number is + always followed by a separator or newline, which no number character + is, so backtracking can't find another match. A linear scan over every + Peterson/biscayne file found the same 224 runs at the same spans; tricky + single lines (`1.`, `.5`, `.`, `1e`, `1e5x`, `1.2.3`, `-1.d-3`, ...) + agree. Regex time 1.45 s -> 0.98 s. Also tried: a lookahead form without + the alternation (1.19 s, slower). A character-class-only scan is 5x + faster but takes bad numbers (`1.2.3`), so it's rejected. `WORD_LINES`' + number lookahead is left as is (few lines). +- **10a8d23e `perf(item-table): share a missing column's plain default`**. + `ItemTable.values` built a column the rows didn't set with a `_default` + call per row: biscayne.riv's `aux`, 1.27M rows, via `package._lengths` + in `_set_dims_from_counts`, ~0.3 s profiled. Now, unless the default is + an `attrs.Factory` (a row's own object), it's `[default] * n`. +- Checked and not worth it: array reads (`run_values`). `np.fromstring(sep=" + ")` is no faster than `np.array(text.split(), dtype=float)` (both ~0.03 + s per 500k values); it's float parsing. + +Verification: basic digests 0 differences vs `dig_head.pkl`; typed 0 +differences vs `td_head.pkl`, errors 6/6; codec, reader, grammar_gen and +period_lists tests pass. + +| model | before (parse) | now (parse) | flopy3 | +|--------------|----------------|---------------|--------| +| Peterson | 1.41 s (0.70) | 1.25 s (0.57) | 2.92 s | +| biscqtg_disv | 2.62 s (~1.0) | 2.23 s (0.76) | 9.05 s | +| secp | 0.98 s (0.61) | ~0.95 s (0.44)| 2.5 s | + +(Whichever sim is first in a `vs3.py` run also pays the component +imports, ~0.2 s warm.) + +Next (deferred 2026-10-15: optimization paused here; next step is a PR description for everything on item-table): + +1. **Per-row Items in `Item.from_rows`**, the biggest cost after the parse: + - DISV `CELL2D`, 106k rows: ~1.0 s profiled (`read_row` 0.56, `Cell2d` + init 0.35, `_file_array` 0.30, `NumberRows.rows` 0.26); + - Peterson SFR period data, 52k rows (41.8k period + 10.5k + connectiondata): ~0.47 s profiled. + + These miss `read_table` (column-wise). Probably `CELL2D`'s + variable-length `icvert` and SFR's keyword/union rows, but **unconfirmed**: + check `_parse_rows`'s routing first. Then either teach `read_table` + those shapes or make the per-row path cheaper. +2. Still: optional `WORD_LINES` quotes and comments; CI recheck; + `adapters.py`; codegen types. + +Scratch: `runs/lexfile.py ` gives per-file parse time and the +size of each terminal's tokens (it skips files that don't parse, like +external arrays). `runs/rebench2.py ` gives a linear-scan timing +of the regex variants plus the equivalence cases. Profiles: +`runs/peterson4.prof` and `runs/bisc6.prof`. Digests: `dig_poss.pkl`, +`dig_def.pkl`, `td_poss.pkl`. Note that `digest.py` takes +` `. + +### Profiled after the parse (2026-10-16) + +On `item-table` at `4f0c2929` (local commit, not pushed; the previous +"Next" list is superseded by this one). Wall time per phase, from timers +wrapped around the hot functions rather than cProfile, which roughly +doubles per-row code (`scratchpad/runs/phases.py [runs]`). Most +of what's left after the parse is turning text into numbers. The +structuring logic itself costs less than expected. + +biscqtg_disv, 2.25 s: + +| phase | time | +|----------------------------------------------------------|--------| +| Lark parse | 0.76 s | +| RIV `read_table`, 100 periods, 1.27M rows, 57 MB | 0.69 s (`np.fromstring` 0.48, `_widths` 0.08) | +| DISV `cell2d`, 106k rows 9–13 values wide | 0.53 s | +| `_set_child_parents`, mostly walking `cell2d` rows | 0.09 s | +| everything else (vertices, WEL, ...) | ~0.1 s | + +`cell2d` alone (`runs/c2d.py `): `NumberRows.rows` 0.16 s, +`_plan` + `read_row` 0.22 s, `Cell2d(**kwargs)` 0.17 s. Its widths: +83,648 rows of 9, 16,315 of 10, 6,042 of 11, 240 of 12, 12 of 13. It's +ragged, so it misses `read_table`, and `_plannable` is False (an +`ncvert` count column), so it's a list of Items, not an `ItemTable`. + +Peterson, 1.2 s: + +| phase | time | +|-------------------------------------------------------|---------| +| Lark parse | 0.53 s | +| internal arrays (`row_values`, 10 runs, 76 MB) | 0.30 s | +| component imports (once per process) | 0.15 s | +| SFR period rows (per-row union reads, `_check_keyword`) | 0.14 s | +| `_check_lookup_counts` (SFR connectiondata `ic`) | 0.055 s | +| `_set_child_parents` (`_row_children` on SFR rows) | 0.045 s | +| SFR connectiondata + packagedata rows | 0.04 s | + +Number readers, benchmarked on the same text (`runs/numread.py`, +`numread2.py`, `arrread.py`): + +| reader | RIV, 57 MB | Peterson arrays, 76 MB | +|------------------------------------------------|------------|------------------------| +| `_widths` + `np.fromstring(sep=" ")` (current for tables) | 0.59 s | | +| `np.fromstring(sep=" ")` alone | 0.48 s | 0.38 s | +| `np.array(text.split(), dtype=float)` (current for arrays) | | 0.29 s | +| `np.loadtxt(StringIO(text), ndmin=2)` | 0.29 s | n/a (ragged lines) | +| `np.loadtxt`, `comments=("#", "!")` | 0.48 s | | +| `pd.read_csv(sep=r"\s+", engine="c")` | 0.23 s | | +| `pd.read_csv(..., float_precision="round_trip")` | 0.67 s | | + +`loadtxt` gave the same values as `fromstring` bit for bit. Pandas +matched on this file too, but its default parser can round off by a +unit in the last place, and round-trip parsing is slower than we have +now, so it's rejected. Array reads (`row_values`) are already at the +float-parsing limit, as found before. + +Plan, by size: + +1. **`read_table` reads with `np.loadtxt`** instead of `_widths` + + `np.fromstring`. RIV 0.59 s -> 0.29 s (about 0.3 s off biscqtg). + `loadtxt` raises ValueError on a row of another width or a value + that isn't a number, which replaces the `_widths` check: catch it + and return None. Check the result has `len(first)` columns. Keep + `without_comments` (cheap when there are no comments) rather than + `comments=`, which is slower. Commas and `D` exponents are already + fixed by `NumberRows.text`. +2. **Ragged numeric blocks by column, one width group at a time** + (`cell2d`). Prototype `runs/ragged.py `: 0.53 s -> 0.33 s, + giving equal Items. For each width: lay out its first row with + `_spans`, build the columns as `read_table` does (`_column`), check + that each count column (`ncvert`) equals its array span's width + (`_sized_by_field`), else fall back. Then `ItemTable(...).to_items()` + per group, placed back in row order. Parse the text once: the + prototype splits it twice. Then about 0.18 s is left, nearly all the + 106k `Cell2d` inits (an `on_setattr` hook for `icvert` and the + count check in `Item.__attrs_post_init__`). Two ways past it: + - build them like `item_table._plain` (`object.__new__` + + `object.__setattr__`), skipping converters and validators, which is + safe only for values read by column, already typed; + - store `cell2d` as an `ItemTable` with `icvert` as an object column + of tuples (`_fit` already falls back to objects), which means + relaxing `_plannable` for count columns. No Items at all, but a + semantic change: rows become views. +3. **Skip the per-row child walk for rows that can't hold children.** + `_set_child_parents` and the children property (component.py ~365) + call `_row_children` on every row, about 227k times for biscqtg. + `_may_hold_children` is cached per class, so check it once per field + (the list's item type) before iterating. About 0.1 s off biscqtg, + 0.05 s off Peterson. For SFR, whose union arm may hold a TAB6 child, + also precompute which fields to look in, rather than `getattr` and + `attrs.has` on every field of every row. +4. **Smaller:** + - `_check_lookup_counts` builds lambdas and calls `dim_value` for + every SFR connectiondata row: compute the expected counts by + column once (about 0.05 s); + - SFR period rows check their arm's keyword per row, which is mostly + unavoidable; + - component imports, 0.15 s once per process, are out of scope. + +5. **Slots on Items (memory, not speed).** The generated item classes + are `@attrs.define` (slotted), but their plain bases `Record` + (record.py:179) and `Item` (item.py:926) have no `__slots__`, so + every instance still has a `__dict__`, and the slots save nothing. + Tried `__slots__ = ()` on both: a `Cell2d` init stays at 1.22 µs, while + memory for 100k of them drops from 12.0 MB to 8.8 MB + (`runs/slotbench.py`). biscqtg and Peterson load in the same time, and + `test_mf6_period_lists.py` passes (45). Do it, first of all the steps: + it's two lines and a quarter less memory for every row of every list + that isn't an `ItemTable` (more as the rows get smaller). First grep for anything + setting a non-field attribute on an Item or Record, or reading its + `__dict__` (none in item.py, record.py or item_table.py). `ItemTable`'s + views (`_view_class`) declare `__slots__ = ("_table", "_i")` but still + get a `__dict__` from the same bases; this fixes that too. + + Components (`Package`, `Model`, ..., `slots=False`) don't need slots: + a simulation has tens of them. They read and write their own + `__dict__` in about 30 places (package.py, component.py, model.py, + period_lists.py), mostly without needing to: + - `c.__dict__["_parent"] = self` (component.py 394–559): `_parent` is + a plain attrs field with no `on_setattr`, so `c._parent = self` does + the same; + - writes of a field's own value (package.py 396, 573, 605–613): + these skip the field's `on_setattr` hooks (convert, validate, + `_coerce_rows`, `_check_dim`), which is deliberate, but + `object.__setattr__` is the usual way to do that, and + `_init_item_lists` already uses it; + - reads, `self.__dict__.get(f.name)` (package.py ~10 places, + period_lists.py 326): every field is set by the time these run, so + `getattr(self, f.name)` should give the same value. Check each one, + though: a read during init, or a property shadowing a field, would + behave differently; + - `Model.workspace`: a property overriding `Context`'s `workspace` + field, its value kept in a non-field attribute `_workspace`. This one + really needs `__dict__` (or its own attribute). + + Clean these up along with the structuring simplification. They aren't + a speed or memory issue. + +1–3 bring biscqtg to about 1.6 s. After that, the parse is the larger +half: `NUMBERS`' regex alone is 0.67 s of it. + +On simplifying: rows reach Items three ways: `read_table` (by column), +`_plan`/`read_row` (by row, per width) and `from_tokens` (`_solve`, +the fallback). With step 2 generalized, every block of only numbers +would be read by column, ragged or not, and by-row reading would be +left for rows with words (SFR's). That's the natural place to simplify +`_parse_rows` / `from_rows` while doing the speedups. + +Verify each step with the load snapshots (step 0), then +`test_mf6_period_lists.py`, the item-table and reader tests, and the +large loads of biscqtg and Peterson by test ID (targeted tests only, no +full smoke run). Re-time with `runs/phases.py`. + +### Step 0: load snapshots, before any of the above + +Work is on branch `load-perf`, cut from `origin/develop` at `6b233ddb` +(#479, squash-merged from `2c79a17f`). The unmerged +`4f0c2929` (reassigned list rows) is cherry-picked onto it as +`00e1aef0`. Its tree is identical to `4f0c2929`'s. + +`runs/digest.py` already pins load results: it hashes every package +field of every corpus sim, and `runs/digcmp.py` diffs two digests. Make +it a test with devtools' `snapshot` fixture (syrupy; +https://modflow-devtools.readthedocs.io/en/latest/md/snapshots.html). +That's better than a CI job that compares the PR against its base +branch: + +- intended changes show in the PR diff as changed snapshot entries, + accepted with `pytest --snapshot-update`; +- it runs locally in the normal suite, with no second checkout; +- it catches drift from dependency upgrades (numpy, xarray, attrs), + which a base-branch comparison misses since both sides share the + environment. devtools also warns on a numpy major change. + +Shape: + +- In `test_mf6_load_all_models.py`, for each known-passing model, + `assert load_digest(sim) == snapshot`. Use syrupy's default (amber, + readable) format, not devtools' array extensions: those store whole + arrays (Peterson has a 76 MB one; ~242 models would bloat the repo). + An entry is a nested dict, package → field → `(dtype, shape, sha1)` + or a repr, so a failure names the field, like `digcmp.py`. +- Harden the digest into a test helper first: + - cast ints to int64 and floats to float64 before hashing (the + default int dtype is platform-dependent under numpy 1.x); + - include xarray dims and coords, not just values; + - no reprs with memory addresses (`object at 0x…`): fail loudly on + one, rather than snapshot it; + - decide whether public attrs fields per package are enough, or + children's state too. +- Add `syrupy` to the test/dev pixi features. devtools' snapshots + module raises ImportError without it; it isn't installed yet. +- Caveats: float bits should match across OSes (correctly rounded + parsing), but the first matrix run confirms it. A models-registry + update changes the snapshots of the fixtures it touches; that's a + true failure but noise in an unrelated PR, so consider pinning the + registry ref. + +As built (branch `load-snapshots`, from `load-perf`, which is pushed +for its own PR): + +- One JSON file per model (syrupy's `JSONSnapshotExtension`), not one + amber file: no xdist write contention on `--snapshot-update`, and a + diff stays local to its model. ~5-15 KB each. +- Every component's fields, sim and models too, private ones as well: + period data is in `_stress_period_data`, which the scratch digest + skipped. Only `_parent` and `write_context` are left out, and fields + that are None. +- Canonical, so representation changes don't churn: an ItemTable and a + list of one attrs class' instances digest the same (by field, from + row values); numeric lists and arrays as `` (200 chars for + plain values, 4000 with nested tables/records). +- No xarray values turned up among loaded fields, so nothing for dims. +- syrupy warns "Can not relate snapshot location" on writes, since the + param IDs have slashes; harmless (subset updates don't delete other + snapshots). + +Status (2026-10-16, branch `load-snapshots`, local only): done, one +commit each. Step 0 `cb9716f4`, 5 `2e034411` (slots only; the component +`__dict__` cleanup is left for the simplification), 1 `8b8d6110` +(read_table 0.72 -> 0.36 s), 3 `7bcdbcb3`, 2 `54f0ec86` (`read_ragged`; +cell2d 0.64 -> 0.40 s, still via `to_items`, neither of the two ways +past the inits taken), 4 `ff6765a1` (`_check_lookup_counts` only), mypy +`6434653d`. biscqtg_disv 2.34 -> ~1.7 s; Peterson ~1.2 s, about the +same. Snapshots unchanged throughout. `load-perf` got `8988eaae` (mypy) +after this branch was cut; rebase onto develop once it merges. + +Order: harden the helper, generate snapshots on `load-perf` before any +perf change, commit, then use the snapshot test as each step's +`differences: 0` check (instead of the scratch pickles). Then step 5 +(slots), 1, 3, 2, 4. Commit each separately. + +## Memory (profiled 2026-10-16) + +Work is on branch `load-memory`, worktree +`.claude/worktrees/load-memory`, cut from `load-snapshots` at +`86da2ba3`. Nothing is pushed. + +[`io-benchmark-results.md`](io-benchmark-results.md) reports peak RSS. On biscqtg that's 1161 MB +for flopy4 vs 596 MB for flopy3. **On macOS, RSS overstates the gap.** +Pages that malloc has freed but the OS hasn't reclaimed still count +toward RSS (and `ru_maxrss`). fp4 frees several big transient buffers +during a load, so its RSS stays high: after `del sim`, RSS is 915 MB and +the footprint is 268 MB. Measure `phys_footprint` instead, read through +`proc_pid_rusage` (what Activity Monitor shows). Linux glibc returns +large freed blocks to the OS, so RSS there should track the footprint +more closely. Not checked. + +None of the plan's earlier items explain the gap. All of `cell2d`'s +Items, the lists of tuples included, come to about 44 MB. + +### Footprint per phase (MB) + +Each library ran in its own process, with a warm-up load of +`test001b_Tlogarithmic` first. Each "+" is the footprint change for +that phase. + +| phase | biscqtg fp3 | biscqtg fp4 | Peterson fp3 | Peterson fp4 | secp fp3 | secp fp4 | +|---|---:|---:|---:|---:|---:|---:| +| import + warm-up | 128 | 218 | 125 | 222 | 125 | 222 | +| load | +223 | +283 | +113 | +29 | +69 | +63 | +| fp3 data access (`get_data`, kept) | +172 | n/a | +41 | n/a | +34 | n/a | +| peak footprint (whole run) | 552 | 775 | 310 | 312 | 280 | 345 | +| peak RSS (for comparison) | 721 | 1215 | 530 | 697 | 380 | 613 | + +- **Load:** fp4's model is no larger than fp3's once fp3 has read its + data. On biscqtg, fp4's load has no transient above what it keeps + (tracemalloc: 300 MB current, 300.8 MB peak). +- **Write:** on biscqtg, fp4 peaks 257–283 MB above the loaded state; + fp3 peaks about 30 MB above it. +- **Peterson:** fp4 keeps only +29 MB, which seems small for 76 MB of + INTERNAL array text. Not investigated. + +### Causes, and fixes in order + +**M1. Write builds every list row as Python objects at once.** About ++270 MB of peak on biscqtg. `_rows_to_tuples` +(`converter/egress/unstructure.py:138`) calls `ItemTable.tokens()` +(`item_table.py:305`), which turns all 100 RIV periods into tuples of +Python numbers (1.27M tuples, about 6.3M boxed values) in +`_unstructure_package` (`unstructure.py:269`, `spd_period`) before any +of it is written. The peak is in `filters.py:343` `data2lines`, called +from `macros.jinja:186`. Fix: stream instead. Hand the template one +period's rows at a time (a generator, or keep the `ItemTable` and +convert in the filter), or format a table's columns to text with numpy +and skip the tuples altogether. Same output; check it with the write +and roundtrip tests. This is also likely the bulk of biscqtg's 3.1 s +write. + +**M2. Eager imports of interop code.** +90 MB and about 1.3 s, on every +model. flopy4's own modules add about 4 MB. The rest is: +- flopy3: +36 MB, 0.47 s; +- xugrid: +14 MB, 0.8 s cumulative, pulling in numba (+21 MB) and dask + (+17 MB). + +They're imported at the top of `flopy4/adapters.py` (xugrid, flopy +StructuredGrid), `flopy4/mf6/adapters.py` (flopy datbase, Grid, +ModelTime), `mf6/gwf_methods.py` (xugrid) and +`mf6/{dis,disv,disu,model}_methods.py` (flopy discretization). Fix: +import them inside the functions that use them, or behind a lazy module +attribute. Check that nothing at module level (base classes, type +annotations evaluated at runtime) needs them. Target: +`import flopy4.mf6.simulation` without `flopy`, `xugrid`, `numba` or +`dask` in `sys.modules`. A test can assert that. + +**M3. CONSTANT period arrays expanded.** 171 MB kept on biscqtg. +`_read_control_record` (`structure.py:290`) returns +`np.full(length, v)` for CONSTANT, and READARRAY period blocks +(`_parse_readarray_period_block`, `structure.py:551`) call it once per +period. biscqtg's RCH (100 periods) and EVT (101) are all +`CONSTANT x`, giving 201 × 106,257 float64s from about 2.4 KB of input. +flopy3 keeps constants as constants. Options: +- store the scalar. The writer must then emit CONSTANT, and every + reader of the field (`to_xarray`, period lists, adapters) must + broadcast it; +- store `np.broadcast_to(np.float64(v), (length,))`. No copy, and + everything that reads it still works, but it's read-only, so in-place + edits (`pkg.recharge[1][5] = ...`) raise. That needs copy-on-write or + a documented limit. + +Decide which before starting. This changes how the values are stored, +so check that load snapshots stay equal (they digest values, so they +should). The writer could also stop writing a full array as INTERNAL +when it's really a constant (`external.py` already does +`nanmin`/`nanmax`/`array_equal` on each array). + +**M4. `_disambiguate_ga_variant` reads whole files.** +`structure.py:577` does `path.read_text().upper()` on every file whose +ftype has a G/A variant, just to find `READASARRAYS` or +`READARRAYGRID`. For biscqtg's RIV that's about 114 MB of temporary +strings, read before the real parse reads the file again. The memory is +transient, so it mostly shows in macOS RSS, but it's also a wasted +57 MB read. Fix: read lines only until `END OPTIONS`. + +Estimate: M1–M3 bring biscqtg's peak footprint from 775 MB to about +250 MB (flopy3: 552). Rough, not measured. + +Order: M1, M2 (no behavior change), then M4 (small), then M3 once the +storage question is settled. Commit each separately. Check each against +the load snapshots and the write/roundtrip tests, and re-measure with +`phases.py`. + +### Done (2026-10-16) -- branch `load-memory` + +All four are in, a commit each, on top of `load-snapshots` (`86da2ba3`): + +- **M1** `4f68fe76`: `unstructure` gives a block's list rows as a lazy + `Rows` sequence (`converter/egress/unstructure.py`), whose tuples are + made per block as the template iterates. JSON/TOML writes `materialize` + them first. `field_type` and `data2list` accept any `Sequence`. The write + peak above the loaded model fell from about +270 MB to 0–45 MB. Write + time didn't change (about 3.0 s on biscqtg), so the tuples weren't the + write's cost. That's a separate item if write speed matters. +- **M2** `c9bb7ac2`: + - `flopy4.mf6.utils` loads its modules on first use (module + `__getattr__`). + - The dis/disv/disu/tdis/model/GWF/simulation methods and `NetCDFModel` + import flopy and xugrid inside the methods that use them, with + annotations under `TYPE_CHECKING`. + - The `coerce` hooks and `_wrap_array` check against flopy's and dask's + classes with `flopy4._imports.isinstance_of`, which doesn't import + the module. A value can't be an instance of a class from a module + that was never imported. + - Import time fell from about 2.1 s to 0.8 s. `test/test_imports.py` + checks that build, write and load import none of flopy, xugrid, + numba, dask, matplotlib, shapely or pyproj. + - Still loaded at import: pandas and scipy, from xarray itself, and + pooch, from `modflow_devtools.dfns`. Those come from dependencies, + not from this repo. +- **M4** `e312c66a`: `_disambiguate_ga_variant` reads lines only up to the + end of OPTIONS (or the first other block). On biscqtg's RIV file that's + 0.2 ms and no allocation, down from 49 ms and 114 MB. +- **M3** `2862cf91`: we chose the 0-d array. + - `_parse_readarray_period_block` keeps a non-layered CONSTANT as + `np.asarray(v)`, as construction already stores a scalar. Layered + CONSTANTs and griddata are still expanded. + - `dense()` takes a `shape`, which `period_shape(component, f)` in + `package.py` resolves from the DFN shape. `period_array` and the + NetCDF export pass it. + - `attrs_xarray._period_dataarray` broadcasts a 0-d period to the shape + of the periods given in full. That also fixes the existing failure + when a constructed package mixed a scalar period with a full one. + - `array_eq` treats a 0-d array as equal to an array holding its value + everywhere. + - Load snapshots were updated for 73 models. Only the shapes of the + CONSTANT arrays changed (`float64[N]` to `float64[1]`), checked in the + diff. The dense `period_array` values matched the base on all 207 + fields across the corpus. + +Validation: +- Write output is byte-identical to `86da2ba3` on all 258 corpus models, + checked after M1 and again after M3. The script is + `scratchpad/mem/writeall.py`, with the base in a detached worktree at + `scratchpad/base`. +- Smoke passed after M2. After M3, only the period-array tests, mypy and + ruff were run locally; the full suite is left to CI. +- The worktree's own pixi env has no `mf6` binary, so locally the + integration and output-reader tests need + `PATH=
/.pixi/envs/dev/bin:$PATH`. + +biscqtg footprint now (MB; single runs vary about ±50): + +| | before | after | flopy3 | +|---|---:|---:|---:| +| import + warm-up | 218 | ~122 | 128 | +| kept after load | ~283 | ~125 | +223 (+395 with data read) | +| write peak above loaded | +257–283 | 0–45 | +30 | +| peak, whole run | 775 | 386–425 | 552 | + +Left: +- The peak is now during the load, about 130 MB above what it keeps. + Profiled below, under "Load peak". +- Peterson's small "kept" figure is still unexplained. +- Layered and griddata CONSTANTs are still expanded. + +### Load peak (profiled 2026-10-16) + +`scratchpad/mem/stages.py` patches `reader.load` and `structure_component` +and takes a tracemalloc timeline (with `reset_peak`) per file and stage. +The peak was in parsing, not in structuring. biscqtg's 57 MB +`biscayne.riv` was held four times over during the parse: + +| Stage, biscayne.riv | Traced | +|---|---| +| `fp.read()` | +57 MB, briefly +114 MB (bytes and str together) | +| lark builds the tree | +114 MB: a `Token` is a `str` subclass that also keeps `.value`, so its text twice | +| transform the tree | +57 MB: `NumberRows(str(token))` copied again | +| structure Riv | the 71 MB it kept, plus the raw text still held | + +Done, on `load-memory` (local, not pushed): +- `482f3600` perf(reader): the basic parser is built with + `Lark(transformer=BasicTransformer())`, so callbacks run as it shifts + and reduces, and no tree is held. `NUMBERS`/`RECORDS` use `token.value`. + `_list` was removed: an inlined rule can't have a callback in this mode + (lark fails in `parse_tree_builder`). The codec test that inspects + tokens builds its own tree parser with `get_basic_parser()`. The parse + of biscayne.riv went from 228 to 115 MB. +- `a7d5784d` perf(item_table): `read_table`'s columns were views into + `loadtxt`'s whole matrix, so Riv also kept a float copy of its cellids. + A column that shares memory with the matrix is copied. Riv keeps 51 MB, + down from 71. + +biscqtg traced: load peak 285 → 171 MB, kept 130 → 108 MB. Footprint +peak about 316–320 MB (was 386–425). Codec tests, 258 load snapshots +(unchanged), smoke, mypy and ruff pass. Load time unchanged (1.6–1.75 s). + +What's held now, above the ~56 MB loaded before the `.riv`: + +| Step | Held at once | Extra | +|---|---|---| +| `fp.read()` | the file's bytes and the decoded str | ~114 MB, briefly | +| `parse(data)` | the input str, plus the `NumberRows` text (each `token.value` is a slice, a copy) | ~114 MB | +| `structure_component` | the `NumberRows` text in `raw`, plus Riv's arrays (51 MB) | ~108 MB + temporaries | + +The three are about the same height, so lowering one alone doesn't lower +the peak. Options: + +1. **Read without the bytes.** `mmap` the file and decode from the mapping + (`str(memoryview(mm), "utf-8")`, no bytes copy). Mapped pages are file + cache, not heap. Read step ~114 → ~57. +2. **Free each period's text once structured.** Pass 3's loops keep every + period's rows until the load returns, through `_raw` in `Package.load`, + `raw` in `_load_mf6`, `raw_lower`, `kper_rows` and `rows`. The callers + would hand `raw` over, and the loop would `kper_rows.pop(kper)` and + drop the block from `raw_lower`. Structuring ~108 → ~60. Makes + `structure_component` consume its input (opt-in if tests reuse `raw`). +3. Parsing still holds input and output, ~114: + - **3a.** `NumberRows` holds (start, end) into the one decoded str. + Parse ~57, but the str lives until structuring ends, so 2 frees + nothing until then. Peak ≈ text + arrays, ~108. Small gain. + - **3b.** Parse the mmap'd bytes with lark's `use_bytes=True`. + `NumberRows` holds (start, end) into the mapping and decodes one + period's slice when structuring asks. The file text never lives on + the heap, so the peak is about what the load keeps plus one period. + 2 is then unnecessary for the number text. Unverified: whether lark's + lexer takes an mmap; else read into `bytes` (57 MB, not doubled). + +Chosen: 1 + 3b, prototyped on biscqtg before anything else. + +Done (local, not pushed): +- `c0fee105` perf(reader): `reader.load` maps the file. The basic parser + is built with `use_bytes=True`, and lexes the mapping through a + `LexerThread` plugin (`from_custom_input`), since lark's `TextSlice` + takes only `str`/`bytes`. `NumberRows` holds a `Span` (buffer, start, + end, encoding), decoded when structured. The transformer gets the + buffer from the `SOURCE` context var. `loads` parses its text's bytes. + Files with `\r`, non-ASCII-compatible encodings, empty files and + streams are read as text. The mapping is released with the parsed + values (vmmap shows none after a load). +- `347406fe`: lark makes a bytes token's `str` its repr, which cost + 0.14 s on the 57 MB RIV (+10% load). A `ContextualLexer` plugin, whose + `BasicLexer.match` returns NUMBERS as a `bytes` subclass with an empty + `str`, removes it. A mapping costs a fixed ~4 ms per file on macOS (as + much as reading 18 MB of text), so only files of 16 MB or more are + mapped (`_MAP_SIZE`); 9 corpus files are. + +biscqtg: traced load peak 171 → 116 MB (108 kept); footprint load peak +~320 → ~200–220 MB, and the run's peak is now the write's (232–258). +Load time unchanged (~1.55 s, alternating runs against `a7d5784d`). +Depends on two lark internals (`LexerThread.from_custom_input`, +`ContextualLexer.BasicLexer`); lark is pinned `<2`. + +Also kept by the load, not yet addressed: +- Disv `cell2d` as 106k Items, about 44 MB of Python objects + (`item_table.py` `read_ragged` → `to_items()`). Columns would be far + smaller. +- Disv structuring holds about 28 MB transient above what it keeps. + +### Tools + +In `/private/tmp/claude-502/-Users-wbonelli-dev-pyphoenix-project/534761f2-da2c-4b52-aad7-86653208ceb8/scratchpad/mem/`: +- `footprint.py`: `footprint()` returns (current, lifetime-max, + resident), in MB, via `proc_pid_rusage`. +- `phases.py {fp3,fp4} [warm_dir]`: RSS and footprint after + each phase. +- `trace.py `: tracemalloc around one fp4 load; + prints what's kept, by line and by traceback. +- `sample_write.py `: samples the footprint against the + main-thread stack during a write; prints the peak and its frames. +- `imports.py ...`: footprint added by each import, in order. + +Run them from the worktree with the main checkout's dev env, so the +worktree's `flopy4` comes first on the path: +`PYTHONPATH=$PWD /Users/wbonelli/dev/pyphoenix-project/.pixi/envs/dev/bin/python