binding may be floating to all places

[naughtont3]

Adding this as issue to track status while testing. Using gv_pmix branch at da5ea2c (after PR #4 was merged).

I enabled debug prints/instrumentation to __kmp_affinity_set_init_mask() to show the binding of OMP threads to places. I see that some threads show all places instead of a specific place X.

Here's the command I used on machine with 12 cores (libomp/moc test show NCPUs=12).

NOTE: I lowered the KA_TRACE() lines to debug level=5 in code, otherwise use KMP_A_DEBUG=101.

  beaker:$ env KMP_A_DEBUG=6 mpirun \
                   -x OMP_PLACES=threads \
                  -x OMP_CPU_AFFINITY=TRUE \
                  -x OMP_PROC_BIND=spread \ 
                  --map-by core -np 2 \
                 ./mpi_omp_hello.dbg

Here's the relevant output sorted for easier reading and with the PID info:

  __kmp_affinity_set_init_mask: binding T#0 (PID:13615) to place 0
  __kmp_affinity_set_init_mask: binding T#1 (PID:13615) to all places
  __kmp_affinity_set_init_mask: binding T#2 (PID:13615) to place 1
  __kmp_affinity_set_init_mask: binding T#3 (PID:13615) to place 1
  __kmp_affinity_set_init_mask: binding T#4 (PID:13615) to place 1

  __kmp_affinity_set_init_mask: binding T#0 (PID:13614) to place 0
  __kmp_affinity_set_init_mask: binding T#1 (PID:13614) to all places
  __kmp_affinity_set_init_mask: binding T#2 (PID:13614) to all places
  __kmp_affinity_set_init_mask: binding T#3 (PID:13614) to place 0
  __kmp_affinity_set_init_mask: binding T#4 (PID:13614) to place 0

[gvallee]

Test on the Thomas' desktop: 6 cores, 12 HT. I will include the output of the test on my desktop asap, I believe I do not have that problem but there is clearly a bug somewhere.

[gvallee]

The test is running only 2 MPI ranks but the work I did was assuming 1 MPI rank per core and then spread OpenMP threads on the HT of that core. It is still a use case we want to support but it is not yet working.

[naughtont3]

Quick follow-up test on BEAKER desktop (lstopo shows: 6 cores, each with 2 hwthreads).

beaker:$ env KMP_A_DEBUG=6 mpirun  -x OMP_PLACES=threads -x OMP_CPU_AFFINITY=TRUE -x OMP_PROC_BIND=spread  --map-by core \
             -np 2     ./mpi_omp_hello.dbg >& LOG.np2
beaker:$ grep binding LOG.np2 | sort -k4
__kmp_affinity_set_init_mask: binding T#1 (PID:56312) to all places
__kmp_affinity_set_init_mask: binding T#3 (PID:56312) to all places
__kmp_affinity_set_init_mask: binding T#0 (PID:56312) to place 0
__kmp_affinity_set_init_mask: binding T#2 (PID:56312) to place 0
__kmp_affinity_set_init_mask: binding T#4 (PID:56312) to place 0
__kmp_affinity_set_init_mask: binding T#1 (PID:56313) to all places
__kmp_affinity_set_init_mask: binding T#0 (PID:56313) to place 0
__kmp_affinity_set_init_mask: binding T#2 (PID:56313) to place 1
__kmp_affinity_set_init_mask: binding T#3 (PID:56313) to place 1
__kmp_affinity_set_init_mask: binding T#4 (PID:56313) to place 1
beaker:$ env KMP_A_DEBUG=6 mpirun  -x OMP_PLACES=threads -x OMP_CPU_AFFINITY=TRUE -x OMP_PROC_BIND=spread  --map-by core \
               -np 2     ./mpi_omp_hello.dbg >& LOG.np2.2
beaker:$ grep binding LOG.np2.2 | sort -k4
__kmp_affinity_set_init_mask: binding T#1 (PID:56336) to all places
__kmp_affinity_set_init_mask: binding T#2 (PID:56336) to all places
__kmp_affinity_set_init_mask: binding T#0 (PID:56336) to place 0
__kmp_affinity_set_init_mask: binding T#3 (PID:56336) to place 0
__kmp_affinity_set_init_mask: binding T#4 (PID:56336) to place 0
__kmp_affinity_set_init_mask: binding T#1 (PID:56337) to all places
__kmp_affinity_set_init_mask: binding T#3 (PID:56337) to all places
__kmp_affinity_set_init_mask: binding T#0 (PID:56337) to place 0
__kmp_affinity_set_init_mask: binding T#2 (PID:56337) to place 1
__kmp_affinity_set_init_mask: binding T#4 (PID:56337) to place 1
beaker:$ env KMP_A_DEBUG=6 mpirun  -x OMP_PLACES=threads -x OMP_CPU_AFFINITY=TRUE -x OMP_PROC_BIND=spread  --map-by core \
            -np 6     ./mpi_omp_hello.dbg >& LOG.np6
beaker:$ grep binding LOG.np6 | sort -k4
__kmp_affinity_set_init_mask: binding T#0 (PID:56360) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56361) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56362) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56363) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56364) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56365) to place 0
beaker:$ env KMP_A_DEBUG=6 mpirun  -x OMP_PLACES=threads -x OMP_CPU_AFFINITY=TRUE -x OMP_PROC_BIND=spread  --map-by core \
               -np 6     ./mpi_omp_hello.dbg >& LOG.np6.2
beaker:$ grep binding LOG.np6.2 | sort -k4
__kmp_affinity_set_init_mask: binding T#0 (PID:56387) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56388) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56389) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56390) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56391) to place 0
__kmp_affinity_set_init_mask: binding T#0 (PID:56392) to place 0
beaker:$

[naughtont3]

The --np 6 case creates the 6 places, one for each MPI rank per core, as expected.

[naughtont3]

lstopo-beaker.pdf