Skip to content

Conversation

SoumyaRaikwar
Copy link

@SoumyaRaikwar SoumyaRaikwar commented Aug 12, 2025

What this PR does / why we need it

Adds explicit rule-based pod affinity and anti-affinity metrics for deployments to provide granular visibility into Kubernetes scheduling constraints, addressing issue #2701.

Refactored from count-based to explicit rule-based approach following maintainer feedback for enhanced operational value.

Which issue(s) this PR fixes

Fixes #2701

Metrics Added

  • kube_deployment_spec_affinity - Pod affinity and anti-affinity rules with granular labels

Labels provided:

  • affinity - podaffinity | podantiaffinity
  • type - requiredDuringSchedulingIgnoredDuringExecution | preferredDuringSchedulingIgnoredDuringExecution
  • topology_key - The topology key for the rule
  • label_selector - The formatted label selector string

- Add kube_deployment_spec_pod_affinity_required_rules metric
- Add kube_deployment_spec_pod_affinity_preferred_rules metric
- Add kube_deployment_spec_pod_anti_affinity_required_rules metric
- Add kube_deployment_spec_pod_anti_affinity_preferred_rules metric
- Update deployment metrics documentation
- Add comprehensive test coverage for all scenarios
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 12, 2025
@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 12, 2025
@mrueg
Copy link
Member

mrueg commented Aug 13, 2025

How would you use this metric for alerting and/or showing information about the deployment?

@SoumyaRaikwar
Copy link
Author

SoumyaRaikwar commented Aug 13, 2025

How would you use this metric for alerting and/or showing information about the deployment?

These metrics enable critical alerting on scheduling constraint violations. For example: (kube_deployment_spec_pod_anti_affinity_preferred_rules > 0) and (kube_deployment_spec_pod_anti_affinity_required_rules == 0) alerts when deployments rely only on soft anti-affinity rules that can be ignored during node pressure, creating single points of failure.

They also help monitor missing protection: (kube_deployment_spec_pod_anti_affinity_required_rules == 0) and (kube_deployment_spec_pod_anti_affinity_preferred_rules == 0) identifies deployments without any anti-affinity rules.

For dashboards, you can visualize cluster-wide scheduling health with count(kube_deployment_spec_pod_anti_affinity_required_rules > 0) to show how many deployments have proper distribution protection.

During incidents, these metrics help correlate why workloads ended up co-located or why pods failed to schedule due to overly complex constraints.

This addresses #2701's core need: visibility into "preferred vs required" scheduling logic to maintain reliable workload distribution during cluster events. Thanks @mrueg for the question - these use cases demonstrate the operational value of these scheduling constraint metrics!

@CatherineF-dev
Copy link
Contributor

/triage accepted
/assign @mrueg

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 13, 2025
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 14, 2025
@mrueg
Copy link
Member

mrueg commented Aug 14, 2025

i think the metric should be explicit, something like:

kube_deployment_affinity{affinity="podaffinity", type="requiredDuringSchedulingIgnoredDuringExecution",topologyKey="foo",labelSelector="matchExpression foo in bar,baz"}  1

then you can count over these and get the desired result, as well as gather exactly that information about the specific affinity setting.

I'm not sure about the labelSelector at this point, if this should be split into subtypes as well or just calling https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/helpers.go#L171 is enough.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: SoumyaRaikwar
Once this PR has been reviewed and has the lgtm label, please ask for approval from mrueg. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 14, 2025
@SoumyaRaikwar
Copy link
Author

Thanks @mrueg for the feedback! I understand you're looking for more explicit metrics that expose individual affinity rule details rather than just counts.

You're absolutely right that explicit metrics would provide much more granular visibility. Instead of simple count metrics

…etrics

- Replace 4 count-based metrics with single kube_deployment_spec_affinity metric
- Add granular labels: affinity, type, topology_key, label_selector
- Enable individual rule visibility and flexible querying
- Update tests and documentation for new metric structure
@SoumyaRaikwar SoumyaRaikwar force-pushed the add-deployment-pod-affinity-metrics branch from b99f07e to d1612ee Compare August 14, 2025 20:32
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 14, 2025
@SoumyaRaikwar
Copy link
Author

Hi @mrueg,

I've successfully refactored the implementation to use explicit rule-based metrics as you requested.

Key Changes:

  • Replaced 4 count-based metrics with single kube_deployment_spec_affinity metric
  • Added granular labels for individual rule visibility and flexible querying
  • Used metav1.FormatLabelSelector() for consistent labelSelector formatting
  • Updated comprehensive tests and documentation

The new approach provides exponentially more operational value while maintaining low cardinality and following the individual object-level data principle from the best practices document.

@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 14, 2025
@k8s-ci-robot k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Aug 25, 2025
@SoumyaRaikwar SoumyaRaikwar force-pushed the add-deployment-pod-affinity-metrics branch from d826011 to a7bb15f Compare August 26, 2025 11:28
Copy link

linux-foundation-easycla bot commented Aug 26, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 26, 2025
timonegk and others added 6 commits September 13, 2025 23:47
This pull requests fixes a logic error in metrics_writer.go where metrics
headers are replaced when a protobuf format is requested. However, the
existing logic is never used because the content type negotiation is already
done in a previous step (in metrics_handler.go). There, the content type for
proto-based formats is changed to text/plain before passing the argument
to SanitizeHeaders.

The pull request changes the condition in SanitizeHeaders to check for
the plain-text format instead of protobuf. I changed the signature of
SanitizeHeaders to accept expfmt.Format instead of string. This makes checking
the content type a bit cleaner. If this is considered a breaking change, we
can also change it to a string prefix comparison.

I encountered the error when I tried to use native histogram parsing in
prometheus and found errors while parsing kube-state-metrics' metrics.
The issue is already described in kubernetes#2587.

Signed-off-by: Timon Engelke <[email protected]>
…etrics

- Replace 4 count-based metrics with single kube_deployment_spec_affinity metric
- Add granular labels: affinity, type, topology_key, label_selector
- Enable individual rule visibility and flexible querying
- Update tests and documentation for new metric structure
@SoumyaRaikwar SoumyaRaikwar force-pushed the add-deployment-pod-affinity-metrics branch from 6f5dd85 to 994a3d0 Compare September 19, 2025 11:06
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 19, 2025
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 19, 2025
@SoumyaRaikwar
Copy link
Author

Hi @mrueg,
When you have a moment, could you please review my PR?

@SoumyaRaikwar
Copy link
Author

Hi @CatherineF-dev, @logicalhan, and @rexagod — could you please review this PR when you have a chance?

… stray comments; keep header spacing; docs+tests updated
@SoumyaRaikwar SoumyaRaikwar requested a review from mrueg September 20, 2025 12:01
@SoumyaRaikwar
Copy link
Author

Hi @mrueg, can you review it please sir.

@SoumyaRaikwar SoumyaRaikwar requested a review from mrueg September 21, 2025 22:46
@SoumyaRaikwar
Copy link
Author

Hi @mrueg — I’ve restored the deleted kustomization.yaml files in examples/autosharding and examples/standard.
Reverted the whitespace change in internal/store/deployment_test.go; could you please take another look?

@SoumyaRaikwar
Copy link
Author

@mrueg @CatherineF-dev , could you please review my pr.

@SoumyaRaikwar
Copy link
Author

@rexagod could you please review my pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

Add schedule spec and status for workload
6 participants