storage-provisioner keeps crashing on Linux From Scratch

[Oberonc]

What Happened?

Latest minikube as of today (1.37.0)

the storage-provisioner keeps restarting and dashboard doesnt work.

minikube start:

😄  minikube v1.37.0 on Lfs 
✨  Automatically selected the docker driver. Other choices: qemu2, podman, ssh
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.48 ...
💾  Downloading Kubernetes v1.34.0 preload ...
    > gcr.io/k8s-minikube/kicbase...:  488.52 MiB / 488.52 MiB  100.00% 7.56 Mi
    > preloaded-images-k8s-v18-v1...:  337.07 MiB / 337.07 MiB  100.00% 4.08 Mi
🔥  Creating docker container (CPUs=2, Memory=7900MB) ...
🐳  Preparing Kubernetes v1.34.0 on Docker 28.4.0 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

kubectl get pods -A:

NAMESPACE              NAME                                         READY   STATUS    RESTARTS      AGE
kube-system            coredns-66bc5c9577-zrnn7                     0/1     Running   0             2m21s
kube-system            etcd-minikube                                1/1     Running   0             2m27s
kube-system            kube-apiserver-minikube                      1/1     Running   0             2m27s
kube-system            kube-controller-manager-minikube             1/1     Running   0             2m27s
kube-system            kube-proxy-phk5s                             1/1     Running   0             2m21s
kube-system            kube-scheduler-minikube                      1/1     Running   0             2m27s
kube-system            storage-provisioner                          1/1     Running   3 (35s ago)   2m25s
kubernetes-dashboard   dashboard-metrics-scraper-77bf4d6c4c-d22n2   1/1     Running   0             70s
kubernetes-dashboard   kubernetes-dashboard-855c9754f9-2rrqz        1/1     Running   1 (24s ago)   70s

(as can be seen "storage-provisioner" and "kubernetes-dashboard" keep restarting)

minikube logs:

.
.
.
==> storage-provisioner [a44340da6c24] <==
I1130 23:04:50.286427       1 storage_provisioner.go:116] Initializing the minikube storage provisioner...
F1130 23:05:20.288836       1 main.go:39] error getting server version: Get "https://10.96.0.1:443/version?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout

I tried running minikube with podman (rootless and non rootless) and docker (rootless and non rootless) and had the same problem with them all.

Attach the log file

log.txt

Operating System

Other

Driver

Docker

[nirs]

@Oberonc What is "Lfs"?

Did you try a VM driver? (e.g. kvm if you run on Linux).

Is this a new issue with minikube 1.37.0? older version worked (e.g. 1.36, 1.35)?

[Oberonc]

@nirs
LFS = Linux From Scratch (https://www.linuxfromscratch.org)
I tried to use 1.36.0 (previous version) had same problem.

I run on Linux (was kernel 6.15.1, updated kernel to 6.17.10 today .. problem persists)

Tried following drivers:
docker (rootless + regular)
podman (rootless + regular)
qemu (no support for services = deal breaker)

I have no idea how to debug the "dial tcp 10.96.0.1:443: i/o timeout" error in the logs

[nirs]

qemu (no support for services = deal breaker)

What is the missing "services" thing? qemu runs a vm using minikube standard iso. This is the best way to get exactly the same cluster you get on any other OS.

If you are on Linux you can may be able to install libvirt, which will give you better networking support - the vm will have an ip address in the linux default network (e.g. 192.168.122/24).

[Oberonc]

when using the QEMU driver on Linux you can only use the builtin network.
.. and when you do that you get no service or tunnel support (as seen here)

[nirs]

when using the QEMU driver on Linux you can only use the builtin network. .. and when you do that you get no service or tunnel support (as seen here)

Yes, you get only user network. If you want real network you need to use the kvm driver. Building libvirt is not hard so you can try it if nobody can help with the docker/podman issue.

Alternatively you can use a more common distro. We use minikube on latest Fedora without issues, and minikube CI is using Ubuntu LTS. It will not give you the joy of building Linux from scratch, but you will not have to fix all the bugs from scratch.

[nirs]

/retitle storage-provisioner keeps crashing on Linux From Scratch

[nirs]

/cc @afbjorklund

[afbjorklund]

Possibly something missing from the kernel (check the logs and dmesg), but this is not a supported configuration.

	exit status 1: Warning: Extension nfacct revision 0 not supported, missing kernel module?
	iptables-restore: line 16 failed

Your OS release info looks odd: https://www.linuxfromscratch.org/lfs/view/stable-systemd/chapter11/theend.html

[Oberonc]

You are right .. the missing module was the problem (nfacct)
Thanks for your help

Everything seems to be working fine now.

[nirs]

@Oberonc please consider contributing to minikube or lfs documentation.

[nirs]

/kind documentation