Ookami: MPI error opal_libevent2022_evthread_use_pthreads

[PetrKryslUCSD]

This is what I did on Ookami after I logged on. (Some more details on https://iacs-group.slack.com/archives/C016DRQ321M.)

My setup of the environment looks like this:

export JULIA_DEPOT_PATH="~/a64fx/depot"
module load julia
module load gcc/13.1.0
module load slurm
# module load mvapich2/arm22/2.3.7
module load openmpi/gcc8/4.1.2

1. I cloned the repository FinEtoolsDDParallel.jl from GitHub.
2. I activated the environment and instantiated the packages.
    julia -e 'using Pkg; Pkg.activate("."); Pkg.instantiate()'
3. I ran julia interactively to install mpiexecjl.
    julia --project=.
   I also set up the system binary: MPIPreferences.use_system_binary().
4. I ran srun to get an interactive session.
   srun -p short -n 10 --ntasks-per-node=1 --pty bash
5. I ran the example.
    cd FinEtoolsDDParallel.jl/examples/
   ~/a64fx/depot/bin/mpiexecjl -n 4 julia --project=. heat/Poisson2D_cg_mpi_driver.jl

After several minutes, the job was terminated. The error message is below.
Note well: On my laptop this example runs to completion in ~70 seconds.

julia: symbol lookup error: /lustre/home/pkrysl/a64fx/depot/artifacts/58dcf187642cdfbafb3581993ca3d8de565acc78/lib/openmpi/mca_pmix_pmix3x.so: undefined symbol: opal_libevent2022_evthread_use_pthreads
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[1972752] signal (15): Terminated
in expression starting at /lustre/home/pkrysl/a64fx/FinEtoolsDDParallel.jl/examples/heat/Poisson2D_cg_mpi_driver.jl:160
_ZN12_GLOBAL__N_117InterleavedAccess13runOnFunctionERN4llvm8FunctionE at /lustre/software/julia/julia-1.10.3/lib/julia/libLLVM-15jl.so (unknown line)
_ZN12_GLOBAL__N_117InterleavedAccess13runOnFunctionERN4llvm8FunctionE at /lustre/software/julia/julia-1.10.3/lib/julia/libLLVM-15jl.so (unknown line)
unknown function (ip: (nil))
Allocations: 14621931 (Pool: 14605380; Big: 16551); GC: 19
julia: symbol lookup error: /lustre/home/pkrysl/a64fx/depot/artifacts/58dcf187642cdfbafb3581993ca3d8de565acc78/lib/openmpi/mca_pmix_pmix3x.so: undefined symbol: opal_libevent2022_evthread_use_pthreads
julia: symbol lookup error: /lustre/home/pkrysl/a64fx/depot/artifacts/58dcf187642cdfbafb3581993ca3d8de565acc78/lib/openmpi/mca_pmix_pmix3x.so: undefined symbol: opal_libevent2022_evthread_use_pthreads
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
  Process name: [[7833,1],1]
  Exit code:    127

[PetrKryslUCSD]

There is something wrong with the system mpi binary. Doing julia --project -e 'using MPIPreferences; MPIPreferences.use_system_binary()' leads to the above error; doing julia --project -e 'using MPIPreferences; MPIPreferences.use_jll_binary("OpenMPI_jll")' does not.

[luraess]

Seems something may go indeed wrong then when trying to hook into system libs. Maybe look into https://github.com/giordano/julia-on-ookami as IIRC @giordano did quite some extensive testing and use of that machine.

[giordano]

I'm a bit late to the party here, but I think I have a solution for your problem. In short, this isn't an issue in MPI.jl per se (which if configured correctly to use system MPI would do the right thing), but this happens if using other packages which would load JLLs which link to MPI, and when they see the system MPI has OpenMPI ABI they go and load the OpenMPI_jll library, because we have no notion of "system ABI" in the JLL multiplexing mechanism. A common culprit here is HDF5_jll, when I looked into this issue last year I traced the issue to that package.

Now, how to address it.

$ module load openmpi/gcc8/4.1.2
Loading openmpi/gcc8/4.1.2
  Loading requirement: ucx/1.11.2
$ module load julia
$ julia --project -q
(openmpi) pkg> add MPIPreferences, MPI
   Resolving package versions...
    Updating `~/tmp/openmpi/Project.toml`
  [da04e1cc] + MPI v0.20.22
  [3da0fdf6] + MPIPreferences v0.1.11
  [...]

julia> using MPIPreferences

julia> MPIPreferences.use_system_binary()
┌ Info: MPI implementation identified
│   libmpi = "libmpi"
│   version_string = "Open MPI v4.1.2rc4, package: Open MPI decarlson@fj100 Distribution, ident: 4.1.2rc4, repo rev: 2022-04-05, Unreleased developer copy\0"
│   impl = "OpenMPI"
│   version = v"4.1.2-rc4"
└   abi = "OpenMPI"
┌ Info: MPIPreferences changed
│   binary = "system"
│   libmpi = "libmpi"
│   abi = "OpenMPI"
│   mpiexec = "mpiexec"
│   preloads = Any[]
└   preloads_env_switch = nothing

julia> exit()

$ julia --project -q
julia> using MPI, Libdl

julia> filter(contains("libmpi"), dllist())
1-element Vector{String}:
 "/lustre/software/openmpi/gcc8/4.1.2-rocky/lib/libmpi.so"

We followed the documentation and MPI.jl loads only system libmpi, so far so good, no surprises. As I anticipated above, the problem comes when you load another package which links to libmpi:

(openmpi) pkg> add HDF5_jll
   [...]

julia> using MPI, HDF5_jll, Libdl

julia> filter(contains("libmpi"), dllist())
2-element Vector{String}:
 "/lustre/software/openmpi/gcc8/4.1.2-rocky/lib/libmpi.so"
 "/lustre/home/mosgiordano/.julia" ⋯ 34 bytes ⋯ "43e725d0e5f9dfa0a/lib/libmpi.so"

Now we have a problem: when loading both MPI and HDF5_jll two different libmpis are loaded, which is the cause of the segfault you were observing: MPI.jl initialises system libmpi, but if you use some hdf5 functionalities they end up calling libmpi from the JLL, which wasn't initialised, and then we go boom.

Luckily, for the case where system OpenMPI has the same ABI as the libmpi in OpenMPI_jll (i.e. also same soname, libmpi.so.40) we have an option: point libmpi in OpenMPI_jll to system libmpi. In the LocalPreferences.toml file created by MPIPreferences.toml in your environment directory add the following section, below the [MPIPreferences] one:

[OpenMPI_jll]
libmpi_path = "/lustre/software/openmpi/gcc8/4.1.2-rocky/lib/libmpi.so"
mpiexec_path = "/lustre/software/openmpi/gcc8/4.1.2-rocky/bin/mpiexec" # probably not necessary, but we set this for good measure

The paths above are the paths of libmpi and mpiexec for the openmpi module you want to use (here I used the paths for the openmpi/gcc8/4.1.2 module on Ookami, other people reading this would need to adapt to their case). Then, also add the OpenMPI_jll package to the environment (]add OpenMPI_jll) and restart the Julia session. If all goes well, at the restart you should see:

julia> using MPI, HDF5_jll, Libdl

julia> filter(contains("libmpi"), dllist())
1-element Vector{String}:
 "/lustre/software/openmpi/gcc8/4.1.2-rocky/lib/libmpi.so"

Now when loading both MPI and HDF5_jll we're back to the ideal situation of having a single libmpi loaded at all times.

I won't say this is a great solution, because it requires duplicating some information MPIPreferences already tries to put there, plus some manual fiddling with the LocalPreferences.toml file. Also, I considered automating this in MPIPreferences, but this won't work in general if system libmpi has OpenMPI ABI but not the soname libmpi.so.40. For that case the only option I see is to use mpitrampoline.

Hopefully, life will be better in a couple of years when the new standard MPI ABI will be more widespread, because facilities will quickly adopt it, right?

Update: JuliaPackaging/Yggdrasil#10884 added 3 more products to OpenMPI_jll@v5. If you use that version of OpenMPI_jll you will have to do something like

[OpenMPI_jll]
libmpi_path = "/lustre/software/openmpi/gcc8/4.1.2-rocky/lib/libmpi.so"
libmpi_mpifh_path = "..."
libmpi_usempi_ignore_tkr_path = "..."
libmpi_usempif08_path = "..."
mpiexec_path = "/lustre/software/openmpi/gcc8/4.1.2-rocky/bin/mpiexec" # probably not necessary, but we set this for good measure

[johnomotani]

@giordano if I follow your instructions to point OpenMPI_jll at my system MPI (which has libmpi.so.40), I see the same as you in dllist() output, but get a crash if I try to MPI.Init() (see below). Does this work for you? I get the same error if I replace OpenMPI_jll with HDF5_jll, but success if I don't import a jll (just do using MPI, Libdl).

The output below is using Julia-1.10.7. I get the same results with Julia-1.11.3 for all the cases I mentioned.

$ bin/julia --project -O3
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.7 (2024-11-26)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using MPI, OpenMPI_jll, Libdl

julia> filter(contains("libmpi"), dllist())
1-element Vector{String}:
 "/lib/x86_64-linux-gnu/libmpi.so"

julia> MPI.Init()
--------------------------------------------------------------------------
It looks like opal_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_shmem_base_select failed
  --> Returned value -1 instead of OPAL_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    orte_init:startup:internal-failure
But I couldn't open the help file:
    /.../.julia/artifacts/9c87bfed7ec16598705b8ebf99a65c5b53bbfcd8/share/openmpi/help-orte-runtime: No such file or directory.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "Error" (-1) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[P1091:1620203] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

[giordano]

    /.../.julia/artifacts/9c87bfed7ec16598705b8ebf99a65c5b53bbfcd8/share/openmpi/help-orte-runtime: No such file or directory.  Sorry!

Somehow MPI from the artifact is still pulled in. You'll have to investigate why/how. I think the more reliable solution is mpitrampoline, as it has a single entry point. #838