Can mill native launcher be statically linked?

[jackkoenig]

In the hardware industry, it's common to have an older Linux distro, e.g. RHEL8 is really common which will gives the following linkage error when trying out Mill 1.0.0.

/scratch/koenig/.cache/mill/download/1.0.0-native-linux-amd64: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /scratch/koenig/.cache/mill/download/1.0.0-native-linux-amd64)
/scratch/koenig/.cache/mill/download/1.0.0-native-linux-amd64: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /scratch/koenig/.cache/mill/download/1.0.0-native-linux-amd64)

This can be worked around by using the JVM version of the client, but it would be nice if the client could be statically linked.

[lihaoyi]

Mill delegates the native image building to Graal. Perhaps you could try following https://www.graalvm.org/latest/reference-manual/native-image/guides/build-static-executables/ and adding --static-nolibc in dist/package.mill's nativeImageOptions, running ./mill show dist.native.executable to generate a statically linked executable? If it works, we could make it the default, or available with a version suffix like -static

[lihaoyi]

Building a native executable with --static-nolibc seems to work

$ git diff
diff --git a/dist/package.mill b/dist/package.mill
index fc069e6357c..5012e87e684 100644
--- a/dist/package.mill
+++ b/dist/package.mill
@@ -306,7 +306,8 @@ object `package` extends MillJavaModule with DistModule {
     def nativeImageOptions = Seq(
       "--no-fallback",
       "--enable-url-protocols=https",
-      "-Os"
+      "-Os",
+      "--static-nolibc"
       // Enable JVisualVM support
       // https://www.graalvm.org/latest/tools/visualvm/#using-visualvm-with-graalvm-native-executables
       // "--enable-monitoring=jvmstat,heapdump"

@jackkoenig probably need you to try it out and see if it fixes the problem in your environment, if it does then we can just turn it on by default

[jackkoenig]

Hey @lihaoyi, I did some digging and here is what I figured out and think.

I think --static-nolibc is a good idea regardless. It looks like the only non-system library is zlib (at least on arm64 linux) so that's the only thing that will be statically linked when you add that option. It's still a good idea to not require Mill users to have zlib installed just in case they don't.

That doesn't actually solve my original problem though, as it specifically is the system libraries like libc that cause issues on older Linux distributions. For circt/firtool we fully statically link using musl libc¹. I am not certain if a fully statically linked binary should be the default as there is disagreement about that, but it's worth considering.

Another option that might be simpler would be to have the Mill boostrap script check the system's glibc version (at least on Linux). If it's new enough then it can fetch the native binary, if it isn't, then it can fall back to the JVM client.

My main concern is Chisel users (who as I mentioned are often on older Linux distros) following default Mill instructions and being confused. I could add -jvm to the Mill version in all Chisel documentation and our template which may be good enough, but I also do like using the native client on my Macbook 🙂.

Footnotes

1. by building in an Alpine Linux Docker container which is much easier than trying to use other distros. Be warned if you try this that the default thread stack size is 128KiB, we increase it to 8MiB for firtool. ↩

[lihaoyi]

@jackkoenig sounds like Graal's --static should do what you want right? That includes libc, so the binary should work regardless of what libc version the user has installed

[alexarchambault]

I ran into the same issue as @jackkoenig on a Ubuntu 20.04 installation. Getting the following error when trying to run Mill 1.0.5:

Downloading mill 1.0.5 from https://repo1.maven.org/maven2/com/lihaoyi/mill-dist-native-linux-amd64/1.0.5/mill-dist-native-linux-amd64-1.0.5.exe ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54.7M  100 54.7M    0     0   331M      0 --:--:-- --:--:-- --:--:--  331M
~/.cache/mill/download/1.0.5-native-linux-amd64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ~/.cache/mill/download/1.0.5-native-linux-amd64)
~/.cache/mill/download/1.0.5-native-linux-amd64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ~/.cache/mill/download/1.0.5-native-linux-amd64)

[alexarchambault]

In Scala CLI and coursier, the native Linux launchers are built from docker. This allows to build them from Ubuntu 20.04 for example, which generates launchers that work all the way back to Ubuntu 20.04. This is setup in the build like this, using my mill-native-image plugin.

[alexarchambault]

I'm kind of surprised (and suspicious) that --static or --static-nolibc work. In coursier and Scala CLI, static Linux launchers are also generated (alongside the standard ones that I mentioned above), but image generation needs to happen in an environment that has the right musl tooling, which we do like this with the help of mill-native-image too.

[lihaoyi]

@alexarchambault why shouldnt --static work? I thought the point of generating a statically linked image is that it bundles everything and doesnt dynamically depend on the libraries available on the host system, and so should work regardless of what kind of mibc or musl is available at runtime

[alexarchambault]

Usually it's not recommended to build static binaries embedding glibc. See this discussion for example. While it's a common use case for musl.

[jackkoenig]

Apologies, I always wanted to follow up on this but always struggling to find the time. A couple of observations from doing this with llvm/circt for similar reasons:

• Alpine Linux ships with muslc instead of glibc which makes it better (or at least easier) for creating portable statically linked binaries.
• Muslc has a small (128 KiB) default stack size for threads which can cause issues in highly parallel code ([ci] Build static binaries w/ 8MiB thread stack llvm/circt#8473).
• Muslc's default allocator considered harmful for performance so it's generally recommended to use microsoft/mimalloc ([ci] Build static binaries with mimalloc llvm/circt#8858).

Those latter two points may be of less concern to the Mill client, but I think are worth getting right. It is my understanding that the Rust community follows this advice (it uses muslc, sets default stack size to 2 MiB, and uses mimalloc).

[jackkoenig]

I already made some of these points above, although note I wasn't aware of the need for mimalloc at that point, nor that this is what Rust folks do.