Skip to content

Make sure rollout restarts are performed for stateful sets #281

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Make sure rollout restarts are performed for stateful sets #281

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e13b375
Make sure rollout restarts are performed for stateful sets when updat…
Mar 2, 2021
47a548b
Use time now from k8s api.
Mar 3, 2021
1042a2e
Use time now and MarshalQueryParameter from k8s api.
Mar 3, 2021
10021af
Merge branch 'master' into linasn/sts-rolling-restart
Apr 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions pkg/controller/controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1291,6 +1291,7 @@ func updatedStatefulSet(
// annotation. This ensures that users can always set an update annotation and then
// wait for it to be removed to know if the operator has processed a StatefulSet.
if !update {
updateSpecTemplate(actual)
delete(actual.Annotations, annotations.Update)
delete(actual.Annotations, annotations.ParallelUpdate)
return actual, true, nil
Expand All @@ -1304,9 +1305,20 @@ func updatedStatefulSet(
expected.Status = actual.DeepCopy().Status

copyAnnotations(expected, actual)
updateSpecTemplate(expected)
return expected, true, nil
}

// updates spec template so that rollout update is always performed.
func updateSpecTemplate(sts *appsv1.StatefulSet) {
if sts.Spec.Template.Annotations == nil {
sts.Spec.Template.Annotations = map[string]string{}
}

now, _ := metav1.Now().MarshalQueryParameter()
sts.Spec.Template.Annotations[annotations.Rollout] = now
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should only do this if the user opts in to this behavior by setting a corresponding field in the cluster spec? I just worry it might come as a surprise to someone that was expecting the old behavior of the operator not performing an update if the statefulsets hadn't changed.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You still need to update sts with update annotation for rollout to happen, I am assuming that if someone already added this annotation they probably expect that sts should be updated. Otherwise what's the point for adding an update annotation in the first place?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main time I can think of that having the operator skip statefulsets that don't need to be updated has come in handy is when an update to a cluster has been interrupted partway through. For example, we would set the annotation on all statefulsets, but then if we ran into an issue we could remove the annotation from all statefulsets to ensure the operator doesn't perform any more updates while we debug any issue that we encountered. Once we've decided to continue with the update we could once again set the annotation on all statefulsets and leave it up to the operator to determine which statefulsets still need to be updated and which don't, skipping any nodes which have already been updated. It's admittedly not a very common use case but I worry that if anyone has become used to it then they may be surprised if the behavior changes here. But I'm also not convinced that we necessarily need to keep supporting this behavior. @schallert what do you think?

}

func copyAnnotations(expected, actual *appsv1.StatefulSet) {
// It's okay for users to add annotations to a StatefulSet after it has been created so
// we'll want to copy over any that exist on the actual StatefulSet but not the expected
Expand Down
3 changes: 3 additions & 0 deletions pkg/k8sops/annotations/annotations.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ const (
// ParallelUpdateInProgress is the annotation used by the operator indicate a parallel update
// is underway. This annotation should only be used by the operator.
ParallelUpdateInProgress = "operator.m3db.io/parallel-update-in-progress"
// Rollout is the annotation used by the operator to force rollout update of a StatefulSet.
Rollout = "operator.m3db.io/rollout"

// EnabledVal is the value that indicates an annotation is enabled.
EnabledVal = "enabled"
)
Expand Down