Repository navigation
Conversation
hzhou
changed the title
hzhou
force-pushed
the
2111_mpitests
branch
2 times, most recently
from
December 17, 2021 15:07
85cb76f to
c959178
Compare
hzhou
force-pushed
the
2111_mpitests
branch
4 times, most recently
from
December 22, 2021 03:30
b763246 to
349003a
Compare
Collaborator
Author
|
test:mpich/ch3/tcp ✔️ |
hzhou
force-pushed
the
2111_mpitests
branch
3 times, most recently
from
December 23, 2021 05:52
6b24080 to
d730f7a
Compare
Collaborator
Author
|
test:mpich/ch3/tcp Comparing the timing:
This means we shortened the testing time of ( ch4:ofi-centos64 total testing time shortens from On Reference for solaris slow init: #5645 (comment) |
This was referenced Dec 23, 2021
Collaborator
Author
|
Added timeout. Experiment with setting default timeout to The dtp tests didn't timeout because most of them have per-test timeout. |
4 tasks done
hzhou
force-pushed
the
2111_mpitests
branch
2 times, most recently
from
December 28, 2021 01:45
d072a18 to
98d744c
Compare
Collaborator
Author
|
test:mpich/ch3/tcp ❌ With ch3-tcp: When freebsd frees a window and recreates it, it appears that the interprocess mutex no longer work. Dramatic speedup on Solaris. From 97 minutes down to 50 minutes. It's 23 minutes on centos |
Contributor
|
Does this change anything for the xfail system? |
Collaborator
Author
|
No.
|
hzhou
force-pushed
the
2111_mpitests
branch
3 times, most recently
from
January 10, 2022 21:42
beae7c8 to
c4aaca5
Compare
Collaborator
Author
|
test:mpich/ch3/tcp |
hzhou
marked this pull request as ready for review
January 10, 2022 21:43
Make -arg equivalent to -arg=1.
The tests with mpicolltest.h can be compiled with -DUSE_MTEST_NBC to become a non-blocking test. Compile the source multiple times with macro is inflexible to move into multi-tests framework -- run multiple tests inside a single MPI_Init/MPI_Finalize window. Convert it to use explicit option instead.
The new framework will allow running tests inside a single MPI_Init/MPI_Finalize window by making each test a uniform function interface. Each test file defines a run function that will run the test. The test file is linked with a stup util/run_mpitests.c to create individual test program that should work exactly as before. In addition, all the test files will be linked together in the binary run_mpitests, that can be used to run multiple tests within a single MPI_Init/Finalize window. All such functional tests are listed in test/mpi/maint/all_mpitests.txt. During autogen, gen_all_mpitests.py will load this file and generate all the Makefile targets. In this commit we didn't modify runtests. All tests should still work by running them individually. We'll add the ability to run multiple tests in runtests in the next commits.
Filter the test list and find all tests that can be run using run_mpitests and run them first. Tests are grouped by testlist and number of processes. The running of tests are controlled by runtests using the input/output pipes, thus we still have the granular control of individual tests. When run_mpitests abort due to e.g. segfault, restart it with next test in the testlist. Track the number of such restart and abort in case something systematic causing it to fail repeatedly, for example, error in run_mpitests itself.
Instead of `echo something > .stopfile`, make `touch .stopfile` to work as well. Refactor the code so that check stopfile once aborts all tests.
It is cleaner to split the utilities for multi-tests into its own source file. Since it will only be used in run_mpitests.c, it is simpler to just include the file as static code.
Use alarm() to enforce timeouts.
If any of the environment variables affects init, we need run that test individually so its settings does not affect the rest of the tests.
Only test in attr folder that is not converted is attrend2 since that test tests MPI_Finalize behaviors.
Convert all tests used in testlist.cvar to mpitests framework.
Remaining not converted:
gather_big
bcast
bcast_comm_world
uoplong
uoplong_large
The remaining unconverted tests are
sendrecv2
probe_unexp
probenull
mprobe
huge_underflow
precv_anysrc
All these tests are tested with CVARs. They remain as individual tests
for now to avoid complications.
These tests were compiled multiple times with different macros defined
in the make rules. Convert them into functions that accept argument
instead.
Tests with type parameters -
atomic_get
fetch_and_op
get_accumulate
Tests with -op=xxx -
wrma_flush_get
win_shared_rma_flush_load
overlap_wins_rma
win_dynamic_rma_flush_get
Tests with -collattach -
win_dynamic_acc
window_attach
aint
The tests uses -use-win-allocate option.
Convert rma/mutex_bench to test function. Merge the source mcs_mutex.h and mcs_mutex.c into mutex_bench for easiler manipulation. Convert macros into command line options.
3 of 4 tasks
4 tasks done
Collaborator
Author
|
Superseded by #8042 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Description
The current testsuite consists of thousands of individual mpi test programs. Running the entire testsuite involves invoking process manager to spawn mpi processes and each process goes through
MPI_INITagain and again. Both the process spawning and MPI initialization are very slow compared to the tested MPI operation itself. The current testsuite runs for a couple of hours and we run hundreds of them every day.This PR attempts to convert individual tests into functions, so multiple tests can be tested within a single
MPI_Init/Finalizewindow. I believe this can significantly reduce the CI testing time.Design goals
The current testsuite workflow should still work.
maketarget for each individual test is still maintained by linkingutil/run_mpitests.o, which supplies themainfunction. Individual test targets should work exactly as before.An additional
maketarget --run_mpitests-- is provided by linkingutil/run_mpitests.o,all_mpitests.o, and all the individual test objects.run_mpitestscan be invoked to run multiple tests in a singlempiruninvocation. We'll invoke it separately with a different number of processes for test coverages.To keep the current workflow as much as possible,
run_mpitestswill be driven by the Perl scriptruntests, thus with the sametestlistmanagement.runtestsstill runs the legacy tests by invokingmpirunindividually; but for new tests, it will aggregate and invokerun_mpitests. The test and its argument is piped in usingstdinand result piped out usingstdoutChallenges
alarm()run_mpitestsat the next test indexrun_mpiteststo accept commands to set and reset CVARStatus
attr/attrend, but workflow can be tested now.Fixes #2142
Author Checklist
Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
Commits are self-contained and do not do two things at once.
Commit message is of the form:
module: short descriptionCommit message explains what's in the commit.
Whitespace checker. Warnings test. Additional tests via comments.
For non-Argonne authors, check contribution agreement.
If necessary, request an explicit comment from your companies PR approval manager.