Don't download any artifacts if JULIA_MPI_BINARY=system

[carstenbauer]

MPI.jl seems to always download the MPICH and OpenMPI artifacts despite having set JULIA_MPI_BINARY=system. It is using the correct system MPI eventually but ideally it shouldn’t download the unused artifacts in the first place.

I guess / fear that this is a limitation of Pkg and the lack of optional dependencies. But I wanted to voice and archive this point somewhere and thought here would be the right place.

[simonbyrne]

I guess we could make them lazy, and download the appropriate ones at package build time?

[simonbyrne]

There are some new features in 1.6 for handling lazy artifacts, i haven't looked into it though. If someone wants to try it, i'd certainly be interested.

[carstenbauer]

MKL_jll seems to be lazy, see discussion over at JuliaLang/Pkg.jl#2664. However, don't know what that laziness means exactly. Does it also allow optional download without an Override.toml?

[simonbyrne]

We don't support Overrrides.toml, as it isn't flexible enough for our needs (e.g. Overrides.toml requires the library version number to be the same, which means we can't support different MPI library versions, let alone different implementations).

We should also look into using Preferences.jl instead of our current homebrew approach. I don't have time to tackle this at the moment, but am happy to review PRs.

[sloede]

However, don't know what that laziness means exactly. Does it also allow optional download without an Override.toml?

lazy means that the artifact is downloaded at first use. That is, the actual artifact is not downloaded until you use something like artifact"XYZ" to get the path, at which point all lazy artifacts will be downloaded.

From https://pkgdocs.julialang.org/v1/artifacts/#Using-Artifacts

For the specific use case of using artifacts that were previously bound, we have the shorthand notation artifact"name" which will automatically search for the Artifacts.toml file contained within the current package, look up the given artifact by name, install it if it is not yet installed, then return the path to that given artifact.

If you want to make use of laziness to prevent downloading artifacts greedily, e.g., for the MPICH and OpenMPI packages, you have to go to the package that defines the artifacts for them, e.g., MPICH_jll, and check their Artifacts.toml (spoiler: the artifacts are not marked as lazy there). Since the Artifacts.toml of JLL packages is auto-generated by BinaryBuilder, you have to go look there if you want to mark their artifacts as lazy. And indeed (from the docs for BinaryBuilder.build_tarballs):

[...] The kwargs are passed on to autobuild, see there for a list of supported ones. A few additional keyword arguments are accept: [...]

• lazy_artifacts sets whether the artifacts should be lazy.

Thus going back to the OP,

But I wanted to voice and archive this point somewhere and thought here would be the right place

I think this here is the right place to discuss whether it's useful and necessary to make the MPI artifacts all lazy, but the correct place to actually effect these changes are the build_tarballs(...) calls in the build_tarballs.jl for the respective package on Yggdrasil, e.g., for MPICH:
https://github.com/JuliaPackaging/Yggdrasil/blob/572f34dad652e4fb49ad030bee8250ba2c81ed70/M/MPICH/build_tarballs.jl#L85

[vchuravy]

I am sympathetic with not wanting to download unnecessary binaries, but making them lazy has its own problems, I shuder thinking of a thousand processes hitting the network and racing to download an artifact.

[carstenbauer]

I am sympathetic with not wanting to download unnecessary binaries, but making them lazy has its own problems, I shuder thinking of a thousand processes hitting the network and racing to download an artifact.

We should still be able to avoid this by forcing the eager download of the artifact at build time if necessary, right? We don't really want to use the laziness "dynamically" (i.e. at runtime after MPI.jl is built). Instead, can't we just explicitly make sure at build time (in e.g. build.jl) that if JULIA_MPI_BINARY!=system the artifact is downloaded (by calling artifact"XYZ" or whatever necessary) and if JULIA_MPI_BINARY=system it is not? By a quick look, it already seems like there are no "runtime references" to the jll packages / the artifacts anywhere outside build.jl (and Project.toml).

[sloede]

Yes, I think that should work (from my limited experience, though).

[carstenbauer]

BTW, if no one disagrees with the described approach, I'm willing to test-implement this in a PR (and a corresponding PR to Yggdrasil). Actually, I might try the same strategy over at MKL.jl first. There the artifact is much bigger (~1.5 GB) and, fortunately, is already marked as lazy. (see JuliaLinearAlgebra/MKL.jl#82)

[simonbyrne]

Certainly, please feel free.

We could try to trigger the downloading at Pkg.build time (though i guess upgrade may not trigger that).

[carstenbauer]

Works nicely for MKL.jl, see my PR here. Though, MKL.jl is simpler and I could avoid build.jl altogether.

[simonbyrne]

Nice. Once you're happy with that, any chance you can work your magic here as well?

[carstenbauer]

Yes, I will give it a shot next week.

[giordano]

I agree with Valentin that making these artifacts lazy isn't a good idea. Also because loading time of JLL packages with lazy artifacts is larger than for non-lazy packages.

I think the solution should be resolving JuliaLang/Pkg.jl#2664, coming up with custom solutions for each package is not useful, nor scalable. BTW, my experience with Pkg is that if you really want something to be implemented, you should do it yourself.