Skip to content

feat(cluster-addons): Use clouds.yaml instead of cloud.conf #225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

feat(cluster-addons): Use clouds.yaml instead of cloud.conf #225

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a3225cd
feat(cluster-addons): Use clouds.yaml instead of cloud.conf
jschoone May 22, 2025
15af899
Merge branch 'main' into feat/use-clouds
jschoone May 27, 2025
f24afde
Merge branch 'main' into feat/use-clouds
jschoone May 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions providers/openstack/scs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,12 @@ kubectl create namespace cluster
```

```
# Add secret using csp-helper chart
helm upgrade -i openstack-secrets -n cluster --create-namespace https://github.com/SovereignCloudStack/openstack-csp-helper/releases/latest/download/openstack-csp-helper.tgz -f <PATH TO CLOUDS YAML>
# Prepare the Secret as it will be deployed in the Workload Cluster
kubectl create secret -n kube-system generic clouds-yaml --from-file=<PATH TO clouds.yaml> --dry-run=client -oyaml > clouds-yaml-secret
Copy link
Contributor Author

@jschoone jschoone May 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@garloff to add the cacert it's probably enough to add a second file called cacert (containing the cacert obviously) and point to that in the clouds.yaml with clouds.openstack.cacert=/etc/openstack/cacert since the clouds-yaml secret will be mounted to /etc/openstack for CCM and CSI

Copy link
Member

@garloff garloff May 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions:

  • The idea is to save those two files and then inject them into the workload cluster manually later?
  • How does capo then get its secrets?
  • Any chance I can still tell the CCM to default to octavia-ovn rather than to octavia-amphora?

Copy link
Contributor Author

@jschoone jschoone May 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions:

  • The idea is to save those two files and then inject them into the workload cluster manually later?

No they will be applied by CAPI as CRS

  • How does capo then get its secrets?

The dry-run can be removed than use that secret.

  • Any chance I can still tell the CCM to default to octavia-ovn rather than to octavia-amphora?

Of course, can just be configured in the values.yaml
All these options are available then: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md#load-balancer

Copy link
Member

@garloff garloff May 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I can generate a secret that capo can work with:

apiVersion: v1
data:
  clouds.yaml: $CL_YAML_B64
  cacert: $CACERT_B64
kind: Secret
metadata:
  name: openstack
  namespace: $CS_NAMESPACE
  labels:
    clusterctl.cluster.x-k8s.io/move: "true"
type: Opaque

Note that the CL_YAML_B64 contains cacert: /etc/certs/cacert which appears to be the place that capo maps it to.

I do not see the secrets propagated into the workload cluster yet, so OCCM and friends have no way to access the OpenStack API yet.

Copy link
Member

@garloff garloff May 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, the helper chart (which we no longer want to use) created a clusterresourceset,, will have to do something similar.

Copy link
Contributor Author

@jschoone jschoone May 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is created as connection between the Cluster and the ClusterResourceSet as soon as a Cluster is created with matching labels
It has the name of the cluster:

kubectl get clusterresourcesetbindings.addons.cluster.x-k8s.io my-cluster1 -oyaml
apiVersion: addons.cluster.x-k8s.io/v1beta1
kind: ClusterResourceSetBinding
metadata:
  name: my-cluster1
  namespace: default
spec:
  bindings:
  - clusterResourceSetName: testclouds-v1-clouds-yaml
  clusterName: my-cluster1

Copy link
Member

@garloff garloff May 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, the clusterSelector.matchLabels entry is wrong.
I apply the label managed-secret: cloud-config to the cluster, thus matchLabels.manged-secret: clouds-yaml can't work. Mystery solved, thanks @jschoone

Copy link
Member

@garloff garloff May 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I have it working.
I now look for clusterstackrelease status.resources openstack-scs-$VERSION-clouds-yaml and if it exists, we add a label managed-secret: clouds-yaml instead of cloud-config to the cluster, so the CRS can do its matching and works.
I renamed the openstack-workload-cluster-secret and the opestack-crs-secret into ...-newsecret for the new way in my scripts, so I can handle clusters of both kinds coexisting in the same namespace on the mgmt cluster.

Cinder-CSI works out of the box with this, which is nice. (I had to do some patching before ...)

Questions:

  • I understand that with the usage of clouds.yaml, we have more freedom to configure clusterstackreleases with different settings. This is good.
    I would like to take the decision which loadbalancer provider I use per cluster, for example.
    Is there a way to do this?
  • I guess there is some complexity with upgrading clusters with cloud.config configuration to clusters with clouds.yaml configuration, with coexistence of both (in the same mgmt cluster namespace) and with autosubscribe ... I fear we need to create some documentation and/or automation for the upgrade case. Maybe only do this for 1.33.x and leave <= 1.32 alone?

Copy link
Contributor Author

@jschoone jschoone May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I understand that with the usage of clouds.yaml, we have more freedom to configure clusterstackreleases with different settings. This is good.
    I would like to take the decision which loadbalancer provider I use per cluster, for example.
    Is there a way to do this?

Yes, using the overwrite.yaml, options/variables from the Cluster resource can be passed. We had
We can just set the default to ovn.
We had an approach before with the workload_loadbalancer variable and adjusting the values.yaml accordingly, see here. But since we did create=false this was a dummy. Now we are able to use that.

  • I guess there is some complexity with upgrading clusters with cloud.config configuration to clusters with clouds.yaml configuration, with coexistence of both (in the same mgmt cluster namespace) and with autosubscribe ... I fear we need to create some documentation and/or automation for the upgrade case. Maybe only do this for 1.33.x and leave <= 1.32 alone?

Can also be done using the overwrite.yaml. e.g. checking which "managed secret" is used and turning on or off the create option depending on that. And yes I would use clouds-yaml in newer versions only.

Copy link
Contributor Author

@jschoone jschoone May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better would be do create new Cluster Stacks. That would follow the idea of Cluster Stacks and no breaking changes exist which needed to be workarounded.
The whole SCS Cluster Stack can then be deprecated and until it reaches EOL the autosubscribe continues working.
Then it's also more clear there are differences which need to be handled which can easily be documented, such as ORC instead of CSPO or the change to use-clouds in the CPO Helm values.

So personally I'd prefer to have new Cluster Stacks.
Especially the use-clouds thing will be very tricky and breaking.
The ORC thing is already implemented, but luckily ORC must be installed anyway since CAPO 0.12 so there nothing breaks.


# Add the Secret to the ClusterResourceSet Secret in the Management Cluster
kubectl create -n cluster secret generic clouds-yaml --from-file=clouds-yaml-secret --type=addons.cluster.x-k8s.io/resource-set

```

```sh
Expand Down
18 changes: 13 additions & 5 deletions providers/openstack/scs/cluster-addon/ccm/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,21 @@
openstack-cloud-controller-manager:
secret:
enabled: true
name: cloud-config
create: false
name: ccm-cloud-config
create: true
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
- key: node.cloudprovider.kubernetes.io/uninitialized
value: "true"
effect: NoSchedule
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
extraVolumes:
- name: clouds-yaml
secret:
secretName: clouds-yaml
extraVolumeMounts:
- name: clouds-yaml
readOnly: true
mountPath: /etc/openstack
cloudConfig:
global:
use-clouds: true
32 changes: 27 additions & 5 deletions providers/openstack/scs/cluster-addon/csi/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,33 @@
openstack-cinder-csi:
secret:
enabled: true
name: cloud-config
create: false
name: csi-cloud-config
create: true
filename: cloud.conf
data:
cloud.conf: |-
[Global]
use-clouds = "true"
clouds-file = /etc/openstack/clouds.yaml
storageClass:
delete:
isDefault: true
csi:
plugin:
volumes:
- name: clouds-yaml
secret:
secretName: clouds-yaml
- name: cloud-conf
secret:
secretName: csi-cloud-config
volumeMounts:
- name: clouds-yaml
readOnly: true
mountPath: /etc/openstack
- name: cloud-conf
readOnly: true
mountPath: /etc/kubernetes
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
Expand All @@ -11,6 +36,3 @@ openstack-cinder-csi:
effect: NoSchedule
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
storageClass:
delete:
isDefault: true
12 changes: 12 additions & 0 deletions providers/openstack/scs/cluster-class/templates/cluster-resource-set.yaml
View file
Open in desktop
Copy link
Member

@garloff garloff Jun 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to have this as addon?
I see:

  - error: 'failed to apply object: clusterresourcesets.addons.cluster.x-k8s.io "openstack-scs-1-32-v0-git.0e3e6e8-clouds-yaml"
      is forbidden: User "system:serviceaccount:cso-system:cso-controller-manager"
      cannot patch resource "clusterresourcesets" in API group "addons.cluster.x-k8s.io"
      in the namespace "ciabns2"'
    group: addons.cluster.x-k8s.io
    kind: ClusterResourceSet
    name: openstack-scs-1-32-v0-git.0e3e6e8-clouds-yaml
    status: not-synced
    version: v1beta1

when testing openstack-scs-1-32-v0-git.0e3e6e8. Not sure why it happens, though.
I do create the CRS from a script https://github.com/SovereignCloudStack/scs-training-kaas-scripts/blob/main/07-cluster-secret.sh when creating the secret ...

Isn't the

Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
apiVersion: addons.cluster.x-k8s.io/v1beta1
kind: ClusterResourceSet
metadata:
name: {{ .Release.Name }}-{{ .Chart.Version }}-clouds-yaml
spec:
strategy: "Reconcile"
clusterSelector:
matchLabels:
managed-secret: clouds-yaml
resources:
- name: clouds-yaml
kind: Secret