[FEATURE] sharding to distribute load #1740
Description
Is there an existing issue for this?
- I have searched the existing issues
Feature description
Currently yace doesn't have sharding to distribute scraping load. So when you have huge number of resources in AWS, the metric that it exposes per instance is too many. Currently our metric collector have max-number-of-series-per-target limit and sometimes yace exceeds this limit.
So it'd be great to have sharding between yace cluster so that automatically detects members and distribute load quite evenly in best effort. consistent hashing could be used to distribute load without resharding too many resources. It will make yace horizontally scalable.
What might the configuration look like?
No response
Anything else?
kube-state-metrics: https://github.com/kubernetes/kube-state-metrics?tab=readme-ov-file#horizontal-sharding