Add binaries built with an older glibc or musl

[alcroito]

Hi,

Trying to run latest bob 4.1.4 on debian 12 arm64 (which ships glibc 2.36) shows

$ ./bob
bob: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by bob)

The same with the AppImage that was added for #259

$ ./bob-linux-arm-appimage/bob-arm.AppImage
./bob-linux-arm-appimage/bob-arm.AppImage: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by ./bob-linux-arm-appimage/bob-arm.AppImage)

The app image doesn't help because it still ships the binary built on the ubuntu-latest (ubuntu 24) github runner image.

As the docs for AppImage say https://docs.appimage.org/reference/best-practices.html#binaries-compiled-on-old-enough-base-system
an older distro has to be used to build binary.

There are a few approaches how to handle this.

One simple approach would be to build against musl, rather than glibc, as described at https://kobzol.github.io/rust/ci/2021/05/07/building-rust-binaries-in-ci-that-work-with-older-glibc.html#removing-dependency-on-glibc

Another is to cross-build against a sysroot with an older glibc with something like https://github.com/cross-rs/cross

Would be great if bob could run on older distros version like Debian 11 / RHEL 8

[MrDwarf7]

This looks somewhat similar to #293 . Though from what I understand in your request it's more about the bob release itself right? Please confirm I'm understanding this correctly.

[alcroito]

This looks somewhat similar to #293 . Though from what I understand in your request it's more about the bob release itself right? Please confirm I'm understanding this correctly.

Indeed on both accounts. I'd like to have a bob release that can run on old glibc systems.

[MrDwarf7]

I had a quick look into building with musl and seems like it would be relatively straightforward. This is assuming that you're needing the bob binary to build against it only - the problem with glibc as a dependency would then be neovim itself as I imagine it requires it, or at least will need arguments passed through when we call to build it.

I did a quick rustup target list | rg -i musl and it gave me back this list:

aarch64-unknown-linux-musl
arm-unknown-linux-musleabi
arm-unknown-linux-musleabihf
armv5te-unknown-linux-musleabi
armv7-unknown-linux-musleabi
armv7-unknown-linux-musleabihf
i586-unknown-linux-musl
i686-unknown-linux-musl
loongarch64-unknown-linux-musl
powerpc64le-unknown-linux-musl
riscv64gc-unknown-linux-musl
x86_64-unknown-linux-musl

Are you able to test locally to see if it'll build locally & with relative ease?
Process would simply be (Am assuming that aarch64-unknown-linux-musl is what we're after here - though feel free to play around with it).

1. Cloning the repo locally and (assuming you've got rustup installed)
2. Running rustup target install aarch64-unknown-linux-musl
3. Calling cargo build --target aarch64-unknown-linux-musl --profile optimized
4. Run ./target/optimized/bob and see if it works

Removing targets can be done via rustup target remove <name> - mentioning it as they can be quite chonky if you're low on space.

Reading around it seems there's somewhat mixed approaches depending on the system/architecture. Some people mentioning they need to add

[target.x86_64-unknown-linux-musl]
linker = "x86_64-linux-musl-gcc" 
# or, some recommendations are to use: 
# linker = "rust-lld"

under $project/.cargo/config.toml and others mentioning needing to hand through flags like RUSTFLAGS="-Clink-arg=--dynamic-linker=/lib/ld-linux-aarch64.so.1" (or whichever your local/native one is).

If it simply compiles and runs no problems, then yes adding this via a PR for release should be easily done, similarly if it's compile options for a specific target.
The complexity will come if we need to tweak the calls to neovim's actual build via env. vars. or similar.

If you're able to do testing like the above it would be hugely appreciated.

[alcroito]

I did a test build on a debian 12 machine x86_64 machine, which succeeded. Didn't have to tweak linkers options or extra flags.

From a stock debian 12 image, i had to do:

sudo apt install build-essential musl-tools git

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

rustup target install x86_64-unknown-linux-musl

git clone git@github.com:MordechaiHadad/bob.git

cd bob
cargo build --target x86_64-unknown-linux-musl --profile optimized

./target/x86_64-unknown-linux-musl/optimized/bob --version

bob-nvim 4.1.4

For supporting building neovim from source targeting a lower glibc, I think the tool could be modified to do what the neovim unofficial releases CI does, which is use zig to provide the older gllbc.

See https://github.com/neovim/neovim-releases/blob/master/.github/workflows/release.yml#L61 which creates a zigcc wrapper which is then assigned to CC env var, so that the neovim cmake build uses that for the build.

Alternatively, for unofficial prebuilt neovim binaries, bob could be taught to download releases from https://github.com/neovim/neovim-releases/releases

But I think both these approaches are out of scope for this particular issue, which is only about making bob work on machines with an older glibc.

EDIT:
Did a few more tests.

The above instructions worked for both Debian 11 and 12 x86_64 arch.

For Debian 12 arm64, the instructions obviously are slightly different due to the arch:

sudo apt install build-essential musl-tools git

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

rustup target install aarch64-unknown-linux-musl

git clone git@github.com:MordechaiHadad/bob.git

cd bob
cargo build --target aarch-unknown-linux-musl --profile optimized

But it builds and runs bob successfully.

For Debian 11 arm64, that is insufficient. It shows this error:

warning: ring@0.17.14: Compiler family detection failed due to error: ToolNotFound: failed to find tool "aarch64-linux-musl-gcc": No such file or directory (os error 2)
warning: ring@0.17.14: Compiler family detection failed due to error: ToolNotFound: failed to find tool "aarch64-linux-musl-gcc": No such file or directory (os error 2)
error: failed to run custom build command for `ring v0.17.14`

Seems like debian 11 musl-tools doesn't create the aarch64-linux-musl-gcc shell wrapper in /usr/bin. Ring crate had an upstream issue, but unclear why it's marked as solved: briansmith/ring#1414

The x86_64-linux-musl-gcc wrapper is also missing on debian 11 x86_64, but that somehow doesn't stop the build from succeeding.

[MrDwarf7]

Alright so a recap, for referencing:

Debian 11, 12 on <x86_64> with musl

• - Working

Requires:

• sudo apt install build-essential musl-tools git
• rustup target x86_64-unknown-linux-musl
• cargo build --target x86_64-unknown-linux-musl --profile optimized

Debian 12 on <arm64> with musl

• - Working

Requires:

• sudo apt install build-essential musl-tools git
• rustup target install aarch64-unknown-linux-musl
• cargo build --target aarch-unknown-linux-musl --profile optimized

Debian 11 on <arm64 / aarch(?64)> with musl

• - Not Working

Tried:

• sudo apt install build-essential musl-tools git
• rustup target install aarch64-unknown-linux-musl
• cargo build --target aarch-unknown-linux-musl --profile optimized

Failing with:

warning: ring@0.17.14: Compiler family detection failed due to error: ToolNotFound: failed to find tool "aarch64-linux-musl-gcc": No such file or directory (os error 2)
warning: ring@0.17.14: Compiler family detection failed due to error: ToolNotFound: failed to find tool "aarch64-linux-musl-gcc": No such file or directory (os error 2)
error: failed to run custom build command for `ring v0.17.14`

Potential cause:

Debian 11 musl-tools doesn't create the aarch64-linux-musl-gcc shell wrapper in /usr/bin

[MrDwarf7]

The proposal for using zig/zigcc is something I'm very much for personally.
Zig is a great tool for things like this and in this case, is also proven to an extent by already being used in the archive releases repo.

The second option you've mentioned is something that seemed to be briefly discussed in the 293 issue I linked, though would require some fair code changes to how bob calls out and makes requests to GitHub.

• I'll note; I have been heavily considering some refactors/reworks on the code. Specifically to this part re reaching out to GH, among other things, make it more extensible/easier to modify for reasons like this.

Regarding the issue closure I found this which may be the solution on that front. May be a sort-of hack-around for CI but I'd prefer (personally) a more robust code solution so this is handled somewhere more robust than having to generate a handful of env. vars. lol