[hzhou-muse]

Pull Request Description

#5725 added the run_mpitests framework, which runs many tests inside one MPI_Init/MPI_Finalize, but main has only the attr tests and some of the coll tests converted. This PR converts the rest of the C tests in attr, coll, comm, datatype, group, info, io, part, pt2pt, rma and topo, plus the errhan, init and mpi_t tests that don't depend on init/finalize. Converting coll, pt2pt and rma alone cut the test phase of a CI run by about 8%.

• Converts the tests to test functions linked into run_mpitests, replacing the *_shm, *_allocation, part -DTEST_* and similar build variants with test arguments (e.g. arg=-use-win-allocate). Tests that need large memory, GPUs, or to test init/finalize stay standalone.
• Converts the io tests in a separate commit for review; they take the file name as an argument, and make clean in test/mpi removes the files they leave behind.
• Frees the MTest helper objects (randomized communicators, window keyval) after each test.
• runtests now runs tests of any process count in run_mpitests (it ran only 1 to 10 and silently skipped the rest), and runs individually the testlist lines that run_mpitests cannot handle (mpiexecarg=, resultTest=).
• Fixes testlist entries that passed flags without arg=, which runtests ignores.
• Renames the wiki page Writing_New_Tests.md to Writing_Tests.md and documents writing and maintaining both test functions and standalone tests.

Author Checklist

• Provide Description
• Commits Follow Good Practice
• Passes All Tests
• Contribution Agreement

🤖 Generated with Claude Code

[hzhou-muse]

test:mpich/ch3/most
test:mpich/ch4/most
✔️

[hzhou-muse]

CI timing comparison

All Jenkins jobs pass on 733b464. I compared each test cell against the median of every other successful build Jenkins still keeps for that job (10–17 builds per job, Oct 7–20). Test counts are the same as before (e.g. 3216 per ch4 cell).

Overall (21 cells): the test phase dropped from 742 to 681 min (−8.2%), and cell wall time dropped from 1104 to 1021 min (−7.5%).

Job	PR wall time	Past median
ch4-ofi	1h07m	1h17m
ch4-ucx	1h00m	1h07m
ch3-sock	51m	54m
ch3-tcp	44m	45m

Test phase per cell (JUnit total, minutes; in parentheses, the range of past runs):

Job	Cell	PR	Past median (range)	Change
ch4-ofi	asan	50.1	54.7 (52.4–133)	−8.4%
	debug	33.1	37.9 (36.8–41.5)	−12.5%
	default	32.8	36.6 (36.0–37.5)	−10.2%
	shmem	35.1	38.3 (37.4–41.2)	−8.3%
	vci	34.5	38.8 (38.3–41.2)	−11.1%
ch4-ucx	am-only	27.7	32.0 (30.8–32.5)	−13.6%
	asan	44.8	49.5 (47.6–54.6)	−9.5%
	debug	32.4	35.6 (33.5–38.6)	−9.0%
	default	31.5	34.0 (32.7–35.4)	−7.3%
	external	30.9	35.6 (33.8–36.7)	−13.2%
	no-inline	31.7	34.2 (32.8–36.6)	−7.4%
	strict	31.6	34.2 (32.8–36.3)	−7.7%
	ubsan	35.5	38.3 (35.7–42.1)	−7.2%
	vci	35.8	40.0 (39.0–42.0)	−10.4%
ch3-tcp	freebsd64	33.8	33.1 (28.8–34.8)	+2.1%
	linux_debug	26.9	28.5 (27.3–31.6)	−5.9%
	osx	17.7	20.3 (17.0–26.6)	−12.9%
	ubuntu22.04	28.0	29.8 (28.5–32.5)	−5.8%
ch3-sock	freebsd64	35.2	35.9 (35.1–36.9)	−1.8%
	osx	17.5	19.1 (18.0–26.5)	−8.3%
	ubuntu22.04	34.8	36.2 (35.4–36.7)	−3.9%

In the ch4-ofi default cell, the savings come from the converted directories: rma 9.0 → 6.8 min, pt2pt 6.7 → 5.9, and coll 6.2 → 5.5. All other directories take the same time as before.

On ch4 the improvement is clear: in 12 of 14 cells, the PR's test time is below the fastest past run. The ch3 gains are small and within the normal run-to-run spread. This is a single PR build. JUnit totals may slightly undercount the startup cost of the batched run_mpitests runs.

🤖 Generated with Claude Code

[hzhou-muse]

test:mpich/ch3/most
test:mpich/ch4/most

[hzhou-muse]

CI timing comparison (update)

All Jenkins jobs pass on 40dc64d. I compared each cell's test phase (JUnit total) with the earlier run on 733b464 and with the past median from the previous comment. Every one of the 21 cells is faster than on 733b464.

Overall (21 cells, minutes):

	Test phase	vs past median
Past median	742
733b464	681	−8.2%
40dc64d	654	−11.8%

Job	Past median	733b464	40dc64d
ch4-ofi (5 cells)	206	185.6	179.7
ch4-ucx (9 cells)	334	301.9	284.4
ch3-tcp (4 cells)	112	106.4	104.0
ch3-sock (3 cells)	91	87.5	86.2

Test phase per cell (minutes):

Job	Cell	Past median	733b464	40dc64d
ch4-ofi	asan	54.7	50.1	49.1
	debug	37.9	33.1	32.9
	default	36.6	32.8	31.9
	shmem	38.3	35.1	33.0
	vci	38.8	34.5	32.8
ch4-ucx	am-only	32.1	27.7	25.3
	asan	49.6	44.8	43.4
	debug	35.4	32.4	31.8
	default	34.1	31.5	29.2
	external	35.5	30.9	29.9
	no-inline	34.2	31.7	29.2
	strict	34.3	31.6	29.0
	ubsan	38.3	35.5	33.4
	vci	40.3	35.8	33.2
ch3-tcp	freebsd64	33.1	33.8	33.2
	linux_debug	28.5	26.9	26.4
	osx	20.3	17.7	17.2
	ubuntu22.04	29.8	28.0	27.2
ch3-sock	freebsd64	35.9	35.2	34.6
	osx	19.1	17.5	17.2
	ubuntu22.04	36.2	34.8	34.4

ch4-ucx gained the most this round, 1–2.6 min per cell. ch4-ofi gained 0.2–2.1 min per cell. The ch3 changes (0.3–0.8 min per cell) are within the normal run-to-run spread. This is still a single build per job.

🤖 Generated with Claude Code

[hzhou-muse]

TODO: tests that still run individually

After this PR (40dc64d), 869 of the 3202 tests in a ch4-ofi default cell still launch individually instead of running in run_mpitests. Below is what they are and how feasible each group is to convert, as follow-up work.

Method

• Measured from CI, ch4-ofi default cell (mpich-review-ch4-ofi#5659 on
  40dc64d, compared with hydra: fix slurm nodelist parsing #5657 on 733b464). runtests runs all
  run_mpitests batches first, then the individual tests, so the JUnit
  cases after the sum of the run_mpitests np=N, M tests... log lines are
  the individual ones.
• runtests' rule: a line runs in run_mpitests when the test is in
  maint/all_mpitests.txt and the line has no init-time env=, no
  mpiexecarg= and no resultTest=.
• An individual launch costs about 0.25-0.3 s (shortest test runs).
  Conversion only removes that cost; it doesn't make slow tests faster.
• cxx, ft, perf, ckpoint and errors/faults don't run in the review cells.

Status

	733b464	40dc64d
Tests run (not skipped)	3202	3202
In run_mpitests	2019	2333
Individual	1183	869
Individual JUnit time	16.9 min	15.2 min

Between 733b464 and 40dc64d, these moved into run_mpitests: coll (77 lines), datatype (65), part (46, all but pingping), comm (45), io (29), topo (18), info (10), group (5), pt2pt (5), rma (4), and a few each in errhan, init, mpi_t and attr.

Remaining groups (lines in the ch4-ofi default cell)

Group	Lines	JUnit time	Feasibility	Saving per cell
Stragglers: impls/mpich (comm_info_hint x6, comm_dup, misc/type_iov x8, type_float16, gpu_query, memory_alloc_kinds, reduce_equal, allreduce_equal, recvq_events, collparmt), mpi_t (mpit_isendirecv x4, mpit_vars), datatype (*_oldapi x4, typename, struct_no_real_types, longdouble), errhan (predef_eh x2, commcall_oldapi), io (tst_fileview, zero_count)	~39	0.3 min	probably easy; check why each was left out of PR 8042	~12 s
part/pingping	24	3.7 min	easy to convert, but the time is the tests themselves	~7 s; shortening them saves minutes
Init-time CVAR lines: coll (96 CIRC_GRAPH_CHUNK_SIZE, 48 CH4_OFI_ENABLE_TRIGGERED, 12 COORDINATES_FILE, 2 SOCKET_BUFFER_SIZE), pt2pt OFI eager/rndv/multi-NIC/async variants (~24), impls stream (RESERV_VCIS, 4), inactive_reqs, rqfree, qmpi_test	~190	1.2 min	only if the CVAR becomes runtime-settable; CIRC_GRAPH_CHUNK_SIZE sizes a pool created at init (cga_init.c)	~55 s, half of it circ_graph
errors (excluding errors/spawn)	~79	0.5 min	mostly feasible; risk of state left by erroneous calls polluting later tests	~25 s
threads (+ impls/mpich/threads, excluding threads/spawn)	85	2.0 min	needs a second run_mpitests initialized with MPI_THREAD_MULTIPLE; mt_* are build variants	~30 s
Fortran: f77 106, f90 121, f08 136	363	2.6 min	largest gain; needs a Fortran-aware driver and Fortran MTest changes; f90 is generated from f77 by maint/f77tof90	~2 min
Slow or large-memory C tests: large_type_sendrec x3, large_vec, io bigtype/i_bigtype, pt2pt_large x3, gather_big, coll_large, init/timeout	~15	2.0 min	keep standalone	-
Not feasible: spawn (31), errors/spawn (7), threads/spawn (3), session (19), init/finalize tests (init/*, attrend2, info_no_init, infocreateenv, errfatal, session_strict_finalize), f2c	~75	3 min	-	-

The remaining launch overhead is about 869 x 0.3 s = 4.3 min per cell. Conversion
alone can reach about 3.2 min of it (Fortran ~2 min), and making the
init-time CVARs runtime-settable adds about 1 min.

Suggested order

1. Stragglers (~39 lines, low risk, small gain).
2. errors separately (state pollution risk).
3. threads after a small runtests change for a THREAD_MULTIPLE driver.
4. Fortran if the remaining ~2 min per cell matters.
5. Init-time CVARs: make CIRC_GRAPH_CHUNK_SIZE runtime (lazy pool) to move
   96 coll lines into run_mpitests.

Separately, part/pingping (3.7 min) and the threads tests (2.0 min) are
slow by themselves; shortening them saves more than converting them.

🤖 Generated with Claude Code