cacpck-system/cacpck-controller-manager is stuck deleting a CP that does not exist #191
Description
Sometimes, the cacpck-system/cacpck-controller-manager is stuck trying to remove a CP that does not exist anymore in the microcluster: failed to remove machine from microcluster: failed to remove management-cluster-control-plane-hwls6 from cluster: HTTP request failed with status code: 520 (node \"management-cluster-cp-40540ce1fe-7jhzq.novalocal\" is not part of the cluster: nodes \"management-cluster-cp-40540ce1fe- 7jhzq.novalocal\" not found)"} .
This error makes the rolling upgrade scenario stuck, as there is no timeout for this action.
Full error message:
│ 2025-11-20T11:07:02Z INFO controllers.CK8sControlPlane Rolling out Control Plane machines {"namespace": "sylva-system", "CK8sControlPlane": "management-cluster-c │
│ ontrol-plane", "cluster": "management-cluster", "needRollout": ["management-cluster-control-plane-p9l4k", "management-cluster-control-plane-hwls6", "management-cluster-contr │
│ l-plane-jkv2w"[]} │
│ 2025-11-20T11:07:02Z ERROR failed to remove machine from microcluster {"controller": "ck8scontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "contro │
│ llerKind": "CK8sControlPlane", "CK8sControlPlane": {"name":"management-cluster-control-plane","namespace":"sylva-system"}, "namespace": "sylva-system", "name": "management-c │
│ luster-control-plane", "reconcileID": "123a3978-b9dc-474b-ac22-137527a6763b", "error": "failed to remove management-cluster-control-plane-hwls6 from cluster: HTTP request fa │
│ iled with status code: 520 (node \"management-cluster-cp-40540ce1fe-7jhzq.novalocal\" is not part of the cluster: nodes \"management-cluster-cp-40540ce1fe-7jhzq.novalocal\" │
│ not found)"} │
│ github.com/canonical/cluster-api-k8s/controlplane/controllers.(*CK8sControlPlaneReconciler).scaleDownControlPlane │
│ github.com/canonical/cluster-api-k8s/controlplane/controllers/scale.go:134 │
│ github.com/canonical/cluster-api-k8s/controlplane/controllers.(*CK8sControlPlaneReconciler).upgradeControlPlane │
│ github.com/canonical/cluster-api-k8s/controlplane/controllers/ck8scontrolplane_controller.go:765 │
│ github.com/canonical/cluster-api-k8s/controlplane/controllers.(*CK8sControlPlaneReconciler).reconcile │
│ github.com/canonical/cluster-api-k8s/controlplane/controllers/ck8scontrolplane_controller.go:530 │
│ github.com/canonical/cluster-api-k8s/controlplane/controllers.(*CK8sControlPlaneReconciler).Reconcile │
│ github.com/canonical/cluster-api-k8s/controlplane/controllers/ck8scontrolplane_controller.go:141 │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile │
│ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116 │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler │
│ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303 │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem │
│ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263 │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 │
│ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224 │
│ 2025-11-20T11:07:02Z INFO controllers.CK8sControlPlane ClusterStatus {"namespace": "sylva-system", "CK8sControlPlane": "management-cluster-control-plane", "clust │
│ er": "management-cluster", "workload": {"Nodes":2,"ReadyNodes":2,"HasK8sdConfigMap":true}} │
│ 2025-11-20T11:07:02Z ERROR Reconciler error {"controller": "ck8scontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "CK8sControlPla │
│ ne", "CK8sControlPlane": {"name":"management-cluster-control-plane","namespace":"sylva-system"}, "namespace": "sylva-system", "name": "management-cluster-control-plane", "re │
│ concileID": "123a3978-b9dc-474b-ac22-137527a6763b", "error": "failed to remove machine from microcluster: failed to remove management-cluster-control-plane-hwls6 from cluste │
│ r: HTTP request failed with status code: 520 (node \"management-cluster-cp-40540ce1fe-7jhzq.novalocal\" is not part of the cluster: nodes \"management-cluster-cp-40540ce1fe- │
│ 7jhzq.novalocal\" not found)"}
Thanks.