Repository navigation
Control threading behavior #36
Description
Activity
With multi-threading enabled, it is expected that the results can be non-deterministic because of rounding differences when adding numbers in different orders. We only utilize TBB in the toolkit (excluding experimental CUDA CCD), so to make the results deterministic you can limit TBB's max number of threads to one. The following code shows how to do this:
#include <tbb/global_control.h> // ... tbb::global_control thread_limiter(tbb::global_control::max_allowed_parallelism, 1);
As long as this
thread_limiterobject stays alive, the number of threads will be limited to 1. This example shows the thread limiter stack allocated, so when the object goes out-of-scope, the limit will be released. If you want this limit for the entire length of your program it is best to define it inmainor you can define it statically using astd::shared_ptr<tbb::global_control>. For example,#include <tbb/info.h> #include <tbb/global_control.h> static std::shared_ptr<tbb::global_control> thread_limiter; void set_num_threads(int nthreads) { if (nthreads <= 0) { nthreads = tbb::info::default_concurrency(); } else if (nthreads > tbb::info::default_concurrency()) { logger().warn( "Attempting to use more threads than available ({:d} > {:d})!", nthreads, tbb::info::default_concurrency()); nthreads = tbb::info::default_concurrency(); } thread_limiter = std::make_shared<tbb::global_control>( tbb::global_control::max_allowed_parallelism, nthreads); }
How about controlling this via an environment variable? We have a situation where TBB is consuming more CPUs than we want, but using
ipctk.set_num_threadsin a dynamic way is not appropriate for us as we want to be able to control this from the global, production environment in a predictable way, and not in any one particular Python file.Amazing, thank you very much!
Any chance this is worth an official patch release?
Yes, I'll include it in the
v1.6.0release, which I am putting together now.Thank you!
I am experiencing some non-determinism problems. This appears to be due to threading behavior. When running without contacts, it looks as if I am able to get deterministic (enough) output by disabling threading on my end. However, when running simulations with contacts with
ipc-toolkit, I'm getting very non-deterministic behavior (seen, for example, through very varying number of solver iterations across runs with identical parameters).I'd like to disable threading in
ipc-toolkitto see if I get deterministic results this way. However, I am not so familiar with TBB and I'm not even sure if TBB is the only thing that's being used inipc-toolkit, or if there are other sources of parallelism. Is there a way to disable parallelism or otherwise control the number of threads inipc-toolkit?Note: The problems I'm looking at are extremely sensitive (presumably due to near-singularity of matrices), I don't believe there's anything wrong with
ipc-toolkit.