Repository navigation
Conversation
… builds build_functions.sh had grown to ~600 lines. It is now a small loader that sources four files, so the existing unix_*_install.sh entry points work unchanged: - build_helpers.sh: run_with_log, clone_if_missing, sync_submodules, prepare_build_dir - build_functions_common.sh: BLT, CAMP, RAJA, Umpire, CHAI - build_functions_mfem.sh: Hypre, METIS, MFEM - build_functions_exaconstit.sh: SNLS, ExaCMech, HDF5, Conduit, Axom, ExaConstit and build_all_dependencies Every RADIUSS-stack package now shares a single BLT checkout (v0.7.2) through BLT_SOURCE_DIR. SNLS (v0.4.4) is built on its own with the batch solvers enabled, and ExaCMech is built against it. The scripts also build the I/O stack the upcoming checkpoint/restart support uses: HDF5 1.14.6 (serial, C only; the 2.x series has caused CMake problems downstream), Conduit v0.9.7 with MPI and HDF5, and Axom v0.15.0 with Sidre. All versions live in dependency_versions.sh, and preflight_checks.sh prints them in the build summary. The CPU configs (cpu_mac_config.sh, cpu_intel_config.sh) now build CHAI with CHAI_ENABLE_MANAGED_PTR=ON. SNLS and ExaCMech use chai::managed_ptr whenever SNLS is built with the RAJA Portability Suite, so ExaCMech failed to build against a CPU CHAI built without it. On the CMake side, Axom is an optional TPL: passing AXOM_DIR finds Axom's package config and sets ENABLE_AXOM. A stub `lumberjack` target works around a dangling reference in Axom's exported axom::slic target. Nothing links against Axom yet. doc/install.md describes the new script layout, versions, build order and directories, and the manual build steps. developers_guide.md lists Axom as an optional dependency. The full dependency stack was built with these scripts on macOS (CPU). On that machine SNLS installs its CMake package to share/snls/cmake and Umpire to lib rather than lib64, so the SNLS_DIR and UMPIRE_DIR paths the scripts pass to ExaConstit had to be adjusted by hand there.
…safe A few SimulationState accessors used std::map::operator[], which inserts an entry when the key is missing. On a rank that owns no elements of a region, that left nullptr entries behind in the quadrature function and quadrature space registries for any code that later loops over them: - GetQuadratureFunction() now uses find() and returns nullptr without inserting anything. - AddQuadratureFunction() no longer creates a function on a null quadrature space when the region has no elements on this rank; it returns false instead, as it already did for a duplicate name. - UpdateModel() and SetupModelVariables() use at(), since every registered begin/end pair must exist. PartialQuadratureFunction gets an explicit copy assignment that copies the data only, plus a defaulted copy constructor. The implicit copy assignment it relied on is deprecated (-Wdeprecated-copy), because mfem::QuadratureFunction has a user-provided copy constructor, and it also re-pointed the function's space members at the source's. SetupModelVariables() and ExaCMechModel::InitStateVars() only ever copy between functions on the same space, which the new assignment asserts. SimulationState::UpdateDeltaTime() named its second parameter `failure` with a default of false, but passed it on as TimeManagement's `success`, so a one-argument call would have been treated as a failed step. It is now `success` with no default. Every existing caller passes both arguments, so behavior is unchanged.
Adds RestartOptions, parsed from a new [Restart] table: - write, interval, write_final: write a checkpoint every `interval` completed steps and/or when the run finishes - keep_last: how many complete checkpoints to keep (0 keeps all) - directory: where checkpoints are written, one cycle_<N>/ directory per checkpoint - read, read_cycle, read_directory: restart from the newest (or a given) checkpoint, optionally from another run's directory Validation rejects negative intervals and keep_last values, empty directories, and a restart whose read directory doesn't exist, and warns about settings that are ignored or would never write a checkpoint. Writing or reading checkpoints requires ExaConstit built with Axom; until the checkpoint code lands in a later commit, any [Restart] table that turns on write or read is rejected with that message. print_options() prints the restart settings when they're used. On a restart (read = true), the mesh, grain, region-mapping, orientation and state-variable files are replaced by the checkpoint and may have been deleted, so validation no longer checks them in that case. MeshOptions, GrainInfo, StateVariables and MaterialOptions::validate() gain a check_input_files flag (default true) for this. The region-mapping check for multiple materials was duplicated in ExaOptions::validate(); one copy is removed. The old [Time] restart, restart_time and restart_cycle keys are removed. They were parsed and printed, but nothing ever used them. src/options.toml documents the new table, and drops the old commented-out keys (also from the calibration workflow example) and a line claiming restart files go to the output directory.
Adds ExaSidreParMesh (sim_state/exa_sidre_parmesh.hpp/cpp), which writes one rank's piece of an mfem::ParMesh into an Axom Sidre group and rebuilds the identical ParMesh from it in a later run with the same number of ranks. Identical means the same local element and vertex numbering, element vertex order, attributes, communication groups, shared vertices, edges and faces, and shared-entity orientation, so every finite element space built on the rebuilt mesh numbers its DOFs exactly as the original did. That is what lets a checkpoint store DOF vectors and quadrature data as plain arrays. axom::sidre::MFEMSidreDataCollection is not used. It records shared edges and faces in each rank's local orientation, which matches shared DOFs incorrectly once a shared face has more than one interior DOF (H1 order 3 and up on hexes), and it references the mesh's memory rather than copying it. The layout follows Conduit's mesh Blueprint (coordsets, topologies and adjacency sets). Each group's local index is stored explicitly, so the rebuilt group order doesn't depend on the order Sidre or HDF5 iterate over groups. The rebuild checks the stored counts and group indices and reports problems on the ranks that find them before every rank aborts together. Ranks that own no elements write and read empty arrays and take part in every collective step. Conforming meshes with one element geometry and one boundary geometry are supported (segments, triangles, quadrilaterals, tetrahedra, hexahedra). restart_sidre_utils.hpp holds small helpers shared with the checkpoint code in the next commit. ExaConstit now links Axom and defines EXACONSTIT_USE_AXOM when configured with AXOM_DIR; the unused Axom placeholder member in SimulationState is removed, since it would no longer compile with that define. test_restart_mesh_roundtrip writes and rebuilds hex and tet meshes, with and without parallel refinement, plus a mesh where half of the ranks own no elements. It checks the topology, groups, shared-entity order and orientation, and that H1 spaces of order 1 to 4 have identical DOF layouts. It runs on 1 to 4 ranks, but never on more ranks than EXACONSTIT_TEST_MAX_MPI_TASKS (default: the logical core count detected at configure time), so it doesn't oversubscribe the CI runners.
Adds exaconstit::RestartCheckpoint (sim_state/restart_checkpoint.hpp/cpp),
driven by the [Restart] options. A run can write binary checkpoints, one
Sidre HDF5 file per rank, and a later run with the same number of ranks
restarts from one and continues bitwise identically to a run that never
stopped. A restart needs only the checkpoint and the options file; the
mesh, grain, region-mapping, orientation and state-variable files are
never read again.
A checkpoint holds the mesh (through ExaSidreParMesh), the grain IDs, the
reference and current coordinates, the velocity, every registered
quadrature function except the tangent stiffness (recomputed before use),
the time state and the material list. An end-of-step function identical
to its beginning-of-step partner is stored as a reference to it. Rank 0
writes commit.json only once every rank has saved its file, so a
checkpoint interrupted mid-write is never read. A failed write is
reported as a warning and the run continues. keep_last prunes old
checkpoints, and the first write after a restart supersedes checkpoints
from the abandoned timeline.
Restarting has two phases, "build first, overlay last":
1. In the SimulationState constructor, the mesh, reference coordinates and
grain IDs come from the checkpoint instead of the input files, and the
state-variable and orientation initialization is skipped. The region
IDs come from the checkpointed mesh, checked against this run's
material list.
2. Once SystemDriver and PostProcessingDriver exist, the driver calls
RestoreFromCheckpoint(), which overwrites the evolving state, resumes
the time state and recomputes the derived fields. It then replays the
boundary-condition update steps already completed, so the active BC
segment matches.
Before anything is overwritten, the restart checks the format version,
rank count, mesh, DOF layout, material identities and every quadrature
space and function's shape. A problem on any rank is reported by that
rank and aborts every rank together. Material properties may change (with
a warning), as may the final time, time stepping, solver and output
settings. TimeManagement gains CheckpointState and ResumeFromCheckpoint(),
which use this run's time-step settings so a continuation can extend
t_final. The previous and original time steps are stored too, so the
restarted run logs exactly what the uninterrupted run would have.
TimeManagement::RestartTimeState(), which nothing used, is removed.
The AddQuadratureFunction() docs, the RestartCheckpoint class docs and a
new developers_guide.md section ("Keeping New State Restartable") spell
out what is checkpointed automatically and what new features must add:
registered quadrature data is saved automatically, but new grid functions
and meshes built on top of the volume mesh are not. doc/restart.md
describes usage, what may change between runs, the design and the
limitations.
test_restart_e2e builds a real SimulationState with two UMAT regions and
orientations, gives it a synthetic evolved state under AUTO time
stepping, writes a checkpoint, deletes the input files, restarts, and
compares everything bitwise, including the logged time-step line. It
runs on 2 and 3 ranks within EXACONSTIT_TEST_MAX_MPI_TASKS. Two
diagnostic tools are built with the tests: inspect_restart_rank_file and
compare_restart_checkpoints (bitwise by default, --rtol optional).
test/test_restart_runs.py runs real simulations (Voce FCC crystal plasticity through ExaCMech on a 4096-element mesh) and checks that restarted runs match an uninterrupted reference bitwise: - cyclic: four ±1% strain cycles, chained every cycle and every two cycles, and restarted from each past checkpoint of the reference - monotonic: 10% tension in z, chained halfway and restarted from the reference's halfway checkpoint - empty_ranks: a 2-element mesh, so all but two ranks own no elements Every checkpoint a restarted run writes is compared with the reference checkpoint of the same step, and its volume-average output row by row. Each restarted run's log must show that it restarted from the expected checkpoint and ran only the remaining steps. A negative control restarts from a checkpoint with the state variables perturbed by 1e-12 and requires the results to differ. The suite takes 10 to 15 minutes on 8 ranks, so it is not part of the default (CI) test suite. It is registered only when configured with -DEXACONSTIT_ENABLE_LONG_TESTS=ON, runs with `ctest -L long`, and uses EXACONSTIT_RESTART_RUN_TEST_NP ranks (default 8, capped at EXACONSTIT_TEST_MAX_MPI_TASKS). Developers changing the restart code must run it locally; developers_guide.md and doc/restart.md say so.
The install scripts conflicted because this branch split build_functions.sh into build_helpers.sh and three build_functions_*.sh files. build_functions.sh stays the loader, and exaconstit-dev's changes to it are moved into the split files: - build_umpire finds Umpire's CMake config directory (UMPIRE_CMAKE_DIR) instead of assuming lib64; SNLS, ExaCMech and ExaConstit use it - MFEM's batched gemv tuning (mfem_gemv_tuning_file, mfem_gemv_tuning_header, tune_mfem_gemv) goes into build_functions_mfem.sh, and GPU builds of MFEM use it - the nvcc rules override is passed to the CUDA builds of MFEM and ExaConstit doc/install.md keeps this branch's dependency list with exaconstit-dev's MFEM commit (74ded9019).
Three problems stopped CUDA builds of the restart I/O stack and of ExaConstit against it: - Axom's AXOM_ENABLE_CUDA and AXOM_ENABLE_HIP are cmake_dependent_options on BLT's ENABLE_CUDA and ENABLE_HIP, like AXOM_ENABLE_MPI on ENABLE_MPI. The install scripts only passed AXOM_ENABLE_<backend>, so Axom defined AXOM_USE_CUDA (found through RAJA and Umpire) but compiled its sources with the host compiler, and the build stopped in axom::Array's check that device-aware code is compiled with a CUDA/HIP compiler. GPU builds now also pass ENABLE_<backend>=ON. - With both the shared and the static HDF5 library installed, Conduit links hdf5-static and Axom hdf5-shared, so ExaConstit got two copies of HDF5 on one link line, and nvcc (the linker in CUDA builds) rejects the versioned libhdf5.so.<version> that hdf5-shared names. HDF5 is now built as a static library only, so Conduit and Axom both link libhdf5.a. - inspect_restart_rank_file and compare_restart_checkpoints only depended on axom, so BLT compiled them as host C++, which a GPU-enabled Axom rejects for the same axom::Array reason. They now also depend on blt::cuda / blt::hip in GPU builds; CPU builds are unchanged. Tested with a CUDA build (GTX 1070): HDF5, Conduit, Axom and ExaConstit build from clean and the full test suite passes. The HIP paths use the same options for the other backend and have not been built.
A rank without elements writes its mesh_current and primal_field views with no data pointer, and Sidre stores such a view as EMPTY rather than EXTERNAL. On restart, BindFieldsAndQuadratureFunctions() required both views to be external before looking at their size, so every restart with an empty rank aborted with "fields/mesh_current/values has 0 entries or the wrong type, but this run needs 0 doubles". Empty nodal fields are now handled first, as empty quadrature functions already were: only the stored size is checked (it must be 0), and nothing is bound or loaded. Tested with the empty_ranks case of test_restart_runs.py (2 elements on 4 and 6 ranks), which failed before, on the CPU and on the GPU with PA and EA: every restart matches the uninterrupted run bitwise.
MFEM and SNLS each created their own host and device Umpire pools, and in GPU builds SNLS starts out with a GPU execution strategy. ExaCMech allocates its model parameters through SNLS before it sets the strategy of the model, so runs with rtmodel = "CPU" also allocated GPU memory. - GPU builds of MFEM in the install scripts now enable MFEM_USE_UMPIRE with the Umpire the scripts build. - The driver creates one host and one device Umpire pool and hands them to MFEM (by allocator name, which MFEM reuses instead of creating its own) and to SNLS (setHostAllocator/setDeviceAllocator), which ExaCMech allocates through. - The driver sets SNLS's execution strategy from rtmodel before any material model is created. CPU builds (MFEM without Umpire) only get the execution strategy setting. Tested with a CUDA build: MFEM reports host-umpire,cuda-umpire memory and the full test suite passes.
…physical cores test_restart_runs.py wrote rtmodel = "CPU" and assembly = "FULL" into every run. It now takes --rtmodel and --assembly (defaults unchanged). GPU builds register test_restart_runs_gpu (labels restart, long, gpu), which runs the same cases with rtmodel = "GPU" and EXACONSTIT_RESTART_RUN_TEST_GPU_ASSEMBLY (default EA) in its own work directory. The two long tests share a RESOURCE_LOCK so they never run at the same time. EXACONSTIT_TEST_MAX_MPI_TASKS defaulted to the number of logical cores, but Open MPI counts physical cores as slots by default, so on a machine with hardware threads the cap allowed rank counts that mpirun refuses to start (test_restart_runs at 8 ranks on 6 cores with 12 threads). The default is now the number of physical cores CMake detects, falling back to logical cores when that is unknown. restart.md and the developers guide document both; restart.md replaces "not yet tested on GPUs" with what has been tested: CUDA with PA and EA, not yet HIP. Tested with a CUDA build (GTX 1070): ctest -L long runs both tests and they pass. The GPU cases also passed with PA (run before the Umpire pool change, with the same harness settings).
Checkpoints went to the [Restart] directory relative to the working directory, while every other output of a run goes to <output_directory>/<basename>. A relative directory is now taken inside that output folder (by default ./results/<basename>/restart), so a run's checkpoints sit next to its logs, volume averages and visualizations, and a restart with the same output_directory and basename finds them there. read_directory names checkpoints written elsewhere, often by another run, so a relative read_directory stays relative to the working directory. The check that a restart's checkpoint directory exists ran in RestartOptions::validate(), before the directory could be resolved; it now runs in ExaOptions::parse_options() right after. The long restart tests look for each run's checkpoints in its output folder and accept the full path the restart log now prints.
A restarted run saved cycle 0 again at t = 0 when its data collections were created, rewriting the first run's step-0 files, and the ParaView collection file (.pvd) was started anew, so ParaView no longer showed the steps before the restart. A restart now skips the cycle-0 save, and ParaView collections use MFEM's restart mode, which keeps the .pvd entries before the current time. VisIt writes one root file per cycle, so its earlier cycles stay listed. MFEM's restart mode compares the current time with the times printed in the .pvd file, so when a run restarts from an earlier checkpoint, the entry of the step it computes again can look earlier than the current time, and MFEM stops the run rather than overwrite that cycle. Before the first save of a restart, rank 0 therefore removes the .pvd entries from that cycle on, by cycle number. test_restart_outputs (a short "outputs" case of test_restart_runs.py, in the default suite) checks that checkpoints are written inside the output folder and that, after a restart and after a restart from an earlier checkpoint, the first run's step-0 files are unchanged, every .pvd lists each step once, and the visualization files are byte for byte those of an uninterrupted run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Once again largely a claude coded PR... My goal when architecting this work was to heavily lean on Axom's
SidreDataCollectionand its related objects as it already natively supports most of what we need from the native MFEM side of things. The restarts just require us storing a few things: the mesh, our various primal related variables, our state variables (meta data related to the material and its region ID is also saved), stress state, material properties, the ExaConstit version, and various time-related variables. Everything else can be reconstructed from these quantities so they currently aren't saved off.Other changes: Axom and its related libraries will be required if user wants restart capabilities. Next, the whole install script aspect has been revamped to make adding new things easier as well as to pull out common libraries and share them between various libraries. Examples of this are BLT and SNLS. We also now require the RAJA Portability Suite for both ExaCMech, SNLS, and ExaConstit.
New tests have been added to make sure this works across various MPI ranks as well.
This will finally close #7 ...