VPA for asymmetric pods in a deployment

[msudheendra-cflt]

We have several operator deployments of 2 pods with one leader pod and one non-leader pod in our environment. The operator runs multiple controllers and therefore have high resource requests. Currently both the leader and non-leader pods have the same resource allocation since leader election is dynamic, the non-leader's resource allocation is wasted capacity since it is just waiting for leadership.

If VPA is configured for this deployment it will recommend resource usage based on both the leader and non-leader pods which may not be accurate. Is this assumption correct? If so, is there a way to use VPA for such a use case? With VPA support for in-place scaling expected to be stable soon this will be a good tool to make the resource allocation more efficient.

The operator pods need to be in the same deployment since there is a spread constraint defined on it to keep the pods in two different AZs. One workaround I thought of is to configure separate deployments for each pod and use pod antiaffinity to ensure they both don't land in the same AZ. But I wanted to ask this question here to see if there were better ways to support this. Thank you!

[adrianmoisey]

/area vertical-pod-autoscaler

[iamzili]

The latest VPA does not account for the individual needs of single pods - over time, all pods within the same workload (e.g. Kubernetes Deployment) will receive the same recommendation. This can lead to significant resource waste if some pods require more or fewer resources than others within the same workload. This risk can be reduced by evenly distributing the load across the pods, but that's not applicable in your case.

I'm not sure whether any discussions are happening in this area to improve this, and I can't even decide whether this needs to be implemented. I don't have a strong opinion on it.

btw, the workaround you proposed (i.e. using podAntiAffinity) with one pod per deployment seems good to me. However, you could also use topologySpreadConstraint on each Pod with a common label.

[adrianmoisey]

I think this sort of feature has been asked for a few times before.

Some questions I have:

1. In the scenario that OP has described, what happens if the smaller pod becomes the primary, and due to low limits, it gets OOM killed and CPU throttled often?
2. What resources should a brand new pod be receiving?

[msudheendra-cflt]

Thanks for your responses Erik and Adrian. Thinking more about this, I realized that even if VPA supports this kind of feature natively it can't be for workload resources like deployments or statefulsets because they would expect one single pod spec and apply the same requests and limits to all the pods that belong to the workload.

In the scenario that OP has described, what happens if the smaller pod becomes the primary, and due to low limits, it gets OOM killed and CPU throttled often?

Ideally, resources would be expanded fast and contracted slowly. That way with in-place resize the new leader is able to allocate more resources before the current limit is hit in most cases. Perhaps we can also hint that the user should size the minAllowed field correctly that the controller should survive until the first resize interval can run and make the pod bigger?

If no limit is specified then we could rely on the headroom being available on the node for it to not be OOM killed but even if it gets killed, the controller should be able to handle it as long as it is not in a loop.

What resources should a brand new pod be receiving?

The VPA spec should have a default request and limit fields included. This should be set based on the leader's expected resource consumption. So we'd have all the pods starting at a leader's config and the non-leader slowly contracts.

[adrianmoisey]

Thanks for your responses Erik and Adrian. Thinking more about this, I realized that even if VPA supports this kind of feature natively it can't be for workload resources like deployments or statefulsets because they would expect one single pod spec and apply the same requests and limits to all the pods that belong to the workload.

In theory it can work with in-place resizes.
I imagine we can do this one day in the future, once we figure out the details.

Ideally, resources would be expanded fast and contracted slowly.

The way this works currently, I don't how know possible this is, since the VPA look at the last x days of history (where x is configurable).

That way with in-place resize the new leader is able to allocate more resources before the current limit is hit in most cases. Perhaps we can also hint that the user should size the minAllowed field correctly that the controller should survive until the first resize interval can run and make the pod bigger?

It would be nice to figure out a way to solve this, without the user being required to set something.
But, whatever that solution is, it shouldn't be too magical.

The VPA spec should have a default request and limit fields included. This should be set based on the leader's expected resource consumption. So we'd have all the pods starting at a leader's config and the non-leader slowly contracts.

Interesting, my thinking was that the VPA should have an average value that it uses as a suggestion. For example, what if the user didn't specify a request/limit?

Also, the solution should apply to other workload types. I don't think it makes sense for the VPA to have a mechanism to find the leader.

Overall, I think this would be a good feature to have. We just need to figure out how to handle it.