Reuse linear rotated free-slip solver workspace#418
Conversation
Fix repeated transient Stokes solves rebuilding the rotated PtAP operator, fieldsplit Schur preconditioner, and GAMG hierarchy on every timestep. Cache the linearity classification, rotation matrices, transformed operator, pressure mass, nullspaces, KSP/PC, and deterministic operator probes. Skip Jacobian assembly for RHS-only updates, refresh safely when constitutive coefficient MeshVariable state or explicit physical time changes, and destroy all owned PETSc state during solver rebuilds. Store the Cartesian reaction directly for rotated topography instead of retaining native operator/RHS copies. Expose the existing physical time argument through the high-level Stokes.solve wrapper. Add regression coverage for RHS reuse, viscosity-driven invalidation, explicit-time refresh, and inverse-viscosity scaling. Document lifecycle, invalidation, MPI launcher requirements, and validation for issue underworldcode#417.
Bounded production A1 restart: timing and memoryI reran the actual Zhong A1 production path after this fix, rather than only the synthetic repeated-solve probe. Configuration:
The run completed normally in 87.79 seconds and neither guard was reached. The external RSS value is the first sample after each Stokes completion.
Timing summary
Interpretation
Step-510 diagnostics continue the checkpoint trajectory: paper time 0.242451, RMS velocity 57.8450, surface/CMB Nu 2.69543/3.20704, and mean temperature 0.317257. This validates the fix over a bounded ten-step production continuation. It does not yet prove long-run whole-process stability; a guarded run of at least 100 steps on a larger-memory host remains appropriate. Full benchmark report: results_bench_100_zhong2008_thermal_convection_A1.py.md |
Guarded production follow-up: 100 additional A1 stepsI extended the real Zhong A1
Each process includes one extra restart-state Stokes solve, excluded from the advancing-step timing. In the first process, aggregate RSS reached 3220.98 MiB after step 559 and then released about 644 MiB at step 560. The second process released about 141 MiB at step 570; its external RSS slope over steps 590-609 was 0.74 MiB/step aggregate. Warm Stokes-local deltas were mostly zero or sub-MiB and included periodic net-negative releases. This is unlike the former per-step The physical trajectory remained continuous. At step 610 (
This is deliberately a segmented 100-step validation, not a single uninterrupted 100-step memory profile. It supports the cache/reuse fix and validates a guarded checkpoint workflow; it does not justify an unmonitored full run on this machine. Full logs, RSS interpretation, and checkpoint metrics are documented in benchmark commit |
Extended production validation: checkpoint 810The real Zhong A1
The four processes completed in 1172.39 s (19.54 min). Advancing solve intervals averaged 5.37 s, with a 5.21 s median. The largest process peak was 3.74 GiB, less than half the configured limit. Warm Stokes-local RSS repeatedly flattened to zero/sub-MiB changes and included multiple collective releases rather than cumulative workspace growth. Physical checkpoint results remained continuous:
Guarded post-fix production evidence now covers 310 evolved steps after checkpoint 500. Full details are in benchmark commit |
1 participant
Summary
Fixes the unbounded resident-memory growth observed when a linear rotated free-slip Stokes system is solved repeatedly, as in transient mantle-convection models.
The change caches the rotated linear solver workspace and reuses it while the operator is unchanged. It also adds explicit invalidation for coefficient, time, boundary-condition, constitutive-model, solver, and DM changes.
Tracks #417.
Root cause
Repeated solves rebuilt two expensive classes of PETSc state:
_residual_is_nonlinear()assembled and compared two Jacobians on every solve.Stage-isolation measurements showed approximately 76.77 MiB growth across 12 repeated nonlinearity probes and 117.23 MiB across 12 direct rotated solves. Explicitly destroying only the previous result vectors did not address the retained PETSc solver state.
Implementation
Q,Q^T, transformed operators, pressure mass, nullspaces, KSP/PC, coefficient state, and deterministic operator probes.MeshVariable._statevalues are unchanged.time=is supplied, coveringmesh.t-dependent symbolic operators.time=separately fromtimestep=inStokes.solve().The design and invalidation contract are documented in
docs/developer/design/ROTATED_FREESLIP_LINEAR_REUSE.md.Validation
The isolated fix was cherry-picked onto the current
developmenttip. The branch is exactly one commit ahead with no merge conflicts;git diff --checkand Python syntax checks pass.Focused serial and MPI validation covered repeated RHS solves, viscosity invalidation, explicit-time refresh, spherical velocity and leakage, Zhong 3-D reaction topography, boundary traction, dynamic topography, and partition independence.
Before this change, the serial spherical probe grew by 102.03 MiB over 12 solves (about 8.54 MiB/solve). The production 8-rank Zhong A1 run had reached 54.29-59.18 GiB aggregate RSS on a 16 GiB machine and triggered macOS watchdog restarts.
MPI checks used the Pixi environment's Open MPI launcher. The Homebrew Open MPI launcher is ABI/version-mismatched with the environment and can hang during startup; no
FI_PROVIDER=tcpworkaround was used.Scope
This PR changes only the UW3 rotated free-slip solve lifecycle, focused tests, and developer documentation. Benchmark-repository scripts and result documents are not included.