.buildkit_qemu_emulator: operation not supported

[anthr76]

Discussed in #23890

^{Originally posted by anthr76 September 6, 2024}
Hello! I'm trying to use the Docker CLI with the DOCKER_HOST set to the Podman socket on MacOS.

I have a project I'm trying to build (https://github.com/goauthentik/authentik) that uses buildx.

I'm using a arm64 Mac and the command I'm trying to invoke is

DOCKER_BUILDKIT=1 docker build . --tag authentik:test1 --platform=linux/amd64 -f Dockerfile --load

This fails with a error

/dev/.buildkit_qemu_emulator ... : did not complete successfully: failed to copy xattrs: failed to set xattr "security.selinux" on /tmp/buildkit-qemu-emulator2534885165/dev/.buildkit_qemu_emulator: operation not supported

This does run okay on the native architecture.

Has anyone ever seen this before and know if it's a bug or a workaround?

I found this related issue dagger/dagger#8054

I know disabling SELinux on CoreOS is usually not a route you can really take so not sure how to approach this.

Does anyone know which codepath this lies in?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Sayed-Imran]

@anthr76 did you install the qemu binaries ?

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

Try this once

[rgov]

Moving a comment over from #24646:

A partial explanation of what is happening, in my understanding:

1. Podman's machine VM, which runs Fedora CoreOS, ships with SELinux enabled.

2. When using docker buildx build, a BuildKit container is started on the machine VM to build the desired container image. This container provides the emulator buildkit-qemu-x86_64 (say), which is used to execute foreign-architecture binaries during the build process.

3. The buildkit-qemu-x86_64 binary has a security.selinux extended attribute:

    $ docker exec -it buildx_buildkit_default /bin/sh -c 'apk add attr && getfattr -d -m "" -- $(command -v buildkit-qemu-x86_64)'     
    ...
    security.selinux="unconfined_u:object_r:container_ro_file_t:s0"
   

4. When BuildKit prepares the build, it copies buildkit-qemu-x86_64 to a temporary location which is then mounted into the building container (source).

   This copy operation attempts to duplicate all extended attributes. However, SELinux denies this in order to protect the security.selinux access control policy. BuildKit treats the failure to copy the xattr as fatal.

   BuildKit could ignore this error, which was previously done elsewhere in the Docker ecosystem. Copy of buildkit-qemu-emulator should ignore xattr failures moby/buildkit#5544

As a workaround, you can disable SELinux on the machine VM.

I solemnly swear I will not do this in production.

1. Enter the machine VM with podman machine ssh and edit /etc/sysconfig/selinux to set SELINUX=disabled.

2. Restart the machine VM with podman machine stop && podman machine start.

3. Ensure there are no lingering BuildKit containers -- existing ones won't work after SELinux policy has changed. Clear them out with docker ps -a and docker rm -f <id>.

4. Clear the Docker build cache for good measure: docker buildx prune

[kaovilai]

@Sayed-Imran the multiarch/qemu-user-static is only published for linux/amd64 and would not work on arm64 podman host.

[rgov]

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

I do not recommend this command. This is one of the not-solutions that crops up in many discussions of the issue but I think it's distracting from the root cause. Please, let's consider further discussion of this command as off-topic for this issue and continue elsewhere.

Why not to use `multiarch/qemu-user-static`

---

Firstly, this is an outdated container image that that was last pushed 2 years ago and contains old copies of the qemu-user-static emulators.

The BuildKit container provides its own, newer qemu-user-static. And, so does the Podman machine VM.

% podman machine ssh -- find /usr/bin -iname '*qemu*'
/usr/bin/qemu-i386-static
/usr/bin/qemu-x86_64-static
/usr/bin/qemu-aarch64-static

As @kaovilai points out, this old multiarch/qemu-user-static is not really multi-arch, because it's only published for linux/amd64, and it can't be run natively on other architectures (e.g.., Apple Silicon).

The container's entry point writes to /proc/sys/fs/binfmt_misc/register to register these emulators with the binfmt_misc kernel feature. This affects the container host, which is why --privileged is required, and persists until reboot. This will cause unexpected behavior if the container host already had a different version of QEMU installed, which then gets overridden by this container.

Although it's a neat trick that you can do this, it's a much better idea to just configure your container host properly so that the change persists across reboots. Use your distro's (probably newer) packaged QEMU. Don't let weird container images reconfigure your kernel.

Again, if you are running on macOS or Windows, the Podman machine VM is already properly configured.

---

As I wrote in my earlier comment, the issue has to do with SELinux denying an attempt to set an extended attribute on a file, which BuildKit treats as a fatal error but shouldn't. It has basically nothing to do with the particular binary in question and any discussion of emulation, binfmt_misc, cross-architecture anything is basically a red herring.

[yxtay]

I experienced a similar error message with building amd64 container using docker buildx with podman on M3 MacBook Pro. The following command to install qemu manually worked for me.

podman run -it --rm --privileged tonistiigi/binfmt --install all

Installing qemu manually is mentioned in the docker docs.
https://docs.docker.com/build/building/multi-platform/#install-qemu-manually

On the other hand, while looking thru github, there are warnings not to disable selinux.
#14376 (comment)

[yxtay]

I experienced a similar error message with building amd64 container using docker buildx with podman on M3 MacBook Pro. The following command to install qemu manually worked for me.

podman run -it --rm --privileged tonistiigi/binfmt --install all

Installing qemu manually is mentioned in the docker docs. https://docs.docker.com/build/building/multi-platform/#install-qemu-manually

On the other hand, while looking thru github, there are warnings not to disable selinux.
#14376 (comment)

This solution I tried ended up giving me issues using docker compose to run any containers. Every command lead to segmentation fault without any stdout. I suspected it was due to rosetta misconfiguration.
#22471

https://podman-desktop.io/docs/podman/rosetta
I ended up disabling rosetta and recreated podman machine. This allowed me to run and build amd64 and arm64 containers without any issue.

[kaovilai]

FYI posted pr moby/buildkit#6015

[anthr76]

Thanks!