Skip to content

perf(ims): shorten the dependency chain in the ILU0 backward solve - #3160

Open
jdhughes-dev wants to merge 5 commits into
MODFLOW-ORG:developfrom
jdhughes-dev:ilu0-solve
Open

jdhughes-dev wants to merge 5 commits into
MODFLOW-ORG:developfrom
jdhughes-dev:ilu0-solve

Conversation

@jdhughes-dev

@jdhughes-dev jdhughes-dev commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

The ILU0 and MILU0 backward solve in ims_base_ilu0a subtracted the term from the row solved just before first and divided by the diagonal last, so each row waited on the previous row before it could start. The upper factor is now scaled by the inverse of the diagonal in ims_base_pcilu0, and the backward solve subtracts the nearest column last, which makes the linear solve 10 to 14 percent faster per inner iteration with COMPLEXITY SIMPLE and MODERATE. Results change at the level of roundoff.

Builds on #3146, whose commits are included until it merges.

Checklist of items for pull request

  • Replaced section above with description of pull request
  • Closed issue Faster application of the ILU preconditioners #3145
  • Added new test or modified an existing test
  • Ran ruff on new and modified python scripts in .doc, autotests, doc, distribution, pymake, and utils subdirectories.
  • Formatted new and modified Fortran source files with fprettify
  • Added a release note to doc/ReleaseNotes/items/ with a plain-language description of the bug fix, change, or feature; required for changes that may affect users
  • Removed checklist items not relevant to this pull request

For additional information see instructions for contributing and instructions for developing.

@jdhughes-dev

Copy link
Copy Markdown
Contributor Author

Reviewer Summary

  • Intent: shorten the dependency chain in the ILU0 and MILU0 backward solve to reduce run times; this is the change made to the MODFLOW-USG PCGU solver in perf(pcgu): shorten the dependency chain in the ILU0 backward solve mfusg#92.
  • Change: after a complete factorization, ims_base_pcilu0 multiplies the upper entries of each row by the stored inverse of the diagonal, and ims_base_ilu0a starts each backward row from D(n) * APC(n) and subtracts the upper terms from the farthest column to the nearest. The factorization itself is unchanged, since it reads the unscaled upper entries before they are scaled.
  • Bad pivots: the scaling is in the same block as the IPCFLAG reset from fix(ims): retry the ILU0 factorization after a bad pivot #3144, so it runs only when MAIN completes. After EXIT MAIN the caller always refactors, so a partial factorization is never scaled or used.
  • Consumers: ims_base_ilu0a is the only routine that reads the ILU0 and MILU0 factors, in the serial CG and BICGSTAB solvers and in the PETSc shell preconditioner, so parallel runs get the same speedup.
  • Timing (from Faster application of the ILU preconditioners #3145): linear solve time per inner iteration, equal inner iteration counts, BICGSTAB, macOS arm64, gfortran 13: SIMPLE (ILU0) 3.32 to 2.85 ms for 120,000 cells and 14.1 to 12.3 ms for 480,000 cells; MODERATE (MILU0) 3.31 to 2.84 ms and 13.9 to 12.0 ms; SIMPLE with RCM reordering 3.71 to 3.20 ms and 15.2 to 13.8 ms.
  • Unit tests: ilu0_apply_exact factors a nonsymmetric tridiagonal matrix exactly with ILU0 and MILU0, twice to check the scaling is not applied twice, and checks that ims_base_ilu0a returns the solution. ilu0_apply_inverse factors a nonsymmetric 4 by 4 grid, where fill is dropped, and checks that multiplying the result of ims_base_ilu0a by the stored factors returns the input vector. Leaving out the scaling fails both tests.
  • mf2005/testevt: excluded in test_external_models.py. The model is steady state with no constant heads or storage, so once its EVT cells fall below the extinction depth after the first outer iteration the matrix is singular and the last ILU0 pivot is roundoff. With this change the solution diverges; develop converges by chance, and reversing the order of the forward solve, which only changes roundoff, also makes it fail to converge.
  • prt_newton_disv1: the ternary method track for prtdisv1 and prtdisv1trts has 191 and 352 events instead of the 195 and 355 from MODPATH 7, while the prtdisv1 heads differ from develop by at most 8e-16. The test already allows a difference of fewer than five events for ifort on Windows; that allowance now applies on any platform when the counts differ, and the values are compared exactly as before when they match.
  • prtdry_stay snapshot: updated. The snapshot rounds times to two places, and two cell exit times at about 339.975 change from 339.97 to 339.98; positions and the number of events are unchanged.
  • Local testing: unit tests pass, and the 125 PRT and IMS autotests pass with the changes above.

lusol, which applies the ILUT and MILUT preconditioners, accumulated
each row of the forward and backward solves in x(i). Because x(i) can
alias x(jlu(k)), the compiler stored and reloaded x(i) for every term.
Each row is now summed in a local variable and stored once. The order of
operations is unchanged, so results are bit-identical, and the routine
can still be called in place.

Adds unit tests that lusol solves a tridiagonal system that ILUT and
MILUT factor exactly, and that it matches the original algorithm to the
bit for an inexact factorization, including in place.

Refs MODFLOW-ORG#3145
A bare underscore in PRECONDITIONER_LEVELS stops the LaTeX release
notes build.
The ILU0 and MILU0 backward solve subtracted the term from the row solved
just before first and divided by the diagonal last, so each row waited on
the previous row before it could start. The upper factor is now scaled by
the inverse of the diagonal after the factorization, and the backward solve
subtracts the nearest column last, which makes the linear solve about 14
percent faster per inner iteration with COMPLEXITY SIMPLE and MODERATE.
Results change at the level of roundoff.

Add unit tests that apply an exact and an inexact factorization. Exclude
mf2005/testevt, which is singular once its EVT cells fall below the
extinction depth, allow the ternary track in prt_newton_disv1 the few-event
difference already allowed on Windows with ifort, and update the
prtdry_stay snapshot, in which two rounded event times change by 0.01.

Closes MODFLOW-ORG#3145
Starting at the top of the convertible cell, with no specific storage, gave the first Newton matrix no storage term, so it was singular and the solution depended on the bad pivot replacement and roundoff. The ILU0 backward solve change made it fail to converge. Start 0.01 below the top so that specific yield storage is active from the first iteration; the automatic flow reduction checks are unchanged.
With a relaxation factor of 1.0, the MILU0 factorization flipped the sign of the pivots of dry cells about 300 times, so the solution depended on the replaced pivots and roundoff, and the ILU0 backward solve change overflowed in BiCGSTAB on Linux. ILU0 replaces no pivots, takes fewer outer iterations, and gives the same outflow (4.8598) with and without that change, within the test tolerance of the 4.8595 answer.
wpbonelli pushed a commit that referenced this pull request Nov 16, 2026
The comparison helpers get_dvclose and get_rclose read only IMS files ending in .ims, so a model whose IMS file ends in .ims6 kept the default head and budget tolerances instead of five times its OUTER_DVCLOSE and INNER_RCLOSE. Of the external test models, only test028_sfr_rewet is affected: its head tolerance becomes 0.005 for an OUTER_DVCLOSE of 0.001. That model failed the head comparison in #3160 by 0.00127 m on Linux and Windows, within its solver closure, because its rewetting is sensitive to roundoff.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant