OCPEDGE-1502: feat: podman-etcd: add support for cert rotation

[eggfoobar]

added a cert check function to the monitor call to force a restart of etcd when the certs have been changed

[knet-jenkins]

Can one of the admins check and authorise this run please: https://ci.kronosnet.org/job/resource-agents/job/resource-agents-pipeline/job/PR-2085/1/input

[clobrano]

Overall, it looks quite good, and I've added a few comments. My main concern now is that the nodes can act on certificate change at the same time, which results in the etcd-cluster being disrupted until both recover.
First of all, is it expected that both nodes can update certificate at the same time? If so, we might want to synchronize them, e.g. with CIB attributes. There are a few examples in the code, look for attribute_learner_node function for example.

[knet-jenkins]

Can one of the admins check and authorise this run please: https://ci.kronosnet.org/job/resource-agents/job/resource-agents-pipeline/job/PR-2085/2/input

[eggfoobar]

@clobrano I don't know about the CIB mechanism, looking into it now so I can port over the attribute_learner_node functionality for syncing this restart. I'll update the code soon, but the other suggestions should be up now

[knet-jenkins]

Can one of the admins check and authorise this run please: https://ci.kronosnet.org/job/resource-agents/job/resource-agents-pipeline/job/PR-2085/3/input

[eggfoobar]

@clobrano I don't think we'll need the CIB, was able to validate this and the cert rotation is a rolling change, so the first node updates, then the second. I think we'll be adding extra complexity to have CIB in the mix. I haven't had a chance to look at the etcd-operator code yet, but I'll find this out so we can be sure that is the behavior. In my experimentation it was a rolling process from node to node.

[clobrano]

@clobrano I don't think we'll need the CIB, was able to validate this and the cert rotation is a rolling change, so the first node updates, then the second. I think we'll be adding extra complexity to have CIB in the mix. I haven't had a chance to look at the etcd-operator code yet, but I'll find this out so we can be sure that is the behavior. In my experimentation it was a rolling process from node to node.

If that so, I agree, there is no need to add complexity.

the cert rotation is a rolling change

does it mean the node also reboots, or that CEO just waits for one to be completed before moving to the next?

If the first, we need no logic at all, but I remember we reproduced a scenario when the node does not reboot after a certificate change.

If the latter, we should ensure the process is long enough for the agent to see (and signal) the file change, be stopped (which might require up to 80 seconds) and restart (which is usually much quicker), and rejoin the cluster, before the peer node is affected by the certificate change.