Skip to content

PB-14482: Helm Chart Updates for Cluster Count Limit Alerting#875

Open
t-nidhin wants to merge 24 commits into3.0.0-devfrom
PB-14482
Open

PB-14482: Helm Chart Updates for Cluster Count Limit Alerting#875
t-nidhin wants to merge 24 commits into3.0.0-devfrom
PB-14482

Conversation

@t-nidhin
Copy link
Copy Markdown
Collaborator

This PR adds the Prometheus alerting rules and email templates required for the global cluster count limit feature in PX-Backup. When the cluster count limit is reached, alerts are sent via AlertManager to notify administrators.

Related PR

  • px-backup PR: PB-14482 (implements the backend logic and metrics emission)

New Prometheus Metric (from px-backup)

The alert is triggered by the pxbackup_cluster_count_limit_status metric emitted by px-backup:

pxbackup_cluster_count_limit_status{
  user_id="<user-uuid>",
  current_count="2",
  max_count="2",
  error_reason="Cluster count limit reached: 2/2 clusters",
  backfill=""
} 100

Email Alert Preview

When the cluster count limit is reached, administrators receive an email with:

Screenshot 2026-03-16 at 2 36 24 PM

Testing

Manual Testing

  1. Applied PrometheusRule directly:
kubectl apply -f test-prometheusrule.yaml
  1. Applied AlertManager secret directly:
kubectl apply -f test-alertmanager-secret.yaml
kubectl rollout restart statefulset -n px-backup -l app.kubernetes.io/name=alertmanager
  1. Verified alert fires when metric is present in Prometheus
  2. Verified email is received with correct formatting

Prometheus Alert Verification

kubectl port-forward svc/px-backup-dashboard-prometheus 9090:9090 -n px-backup
# Navigate to http://localhost:9090/alerts
# Verify ClusterCountLimitAlert is listed and firing

Backward Compatibility

  • No breaking changes: Existing deployments will continue to work
  • Alert only fires when metric exists: The alert only triggers when px-backup emits the pxbackup_cluster_count_limit_status metric (i.e., when the limit is reached)

px-kesavan and others added 24 commits December 23, 2025 11:43
Signed-off-by: Kesavan Thiruvenkadasamy <kthiruvenkadasamy@purestorage.com>
Signed-off-by: Kesavan Thiruvenkadasamy <kthiruvenkadasamy@purestorage.com>
PB-13325 | upgrading to official postgres-18

---------

Co-authored-by: Ayush Sharma <aysharma@purestorage.com>
Signed-off-by: Kesavan Thiruvenkadasamy <kthiruvenkadasamy@purestorage.com>
…resources (#843)

* PB-12294 | Tagging resources with label app.kubernetes.io/part-of
updated the tar ball and repo index
…b, alertmanager and prometheus packages to fix high CVEs
* PB-13866 | bumping mysql version


---------

Co-authored-by: Ayush Sharma <aysharma@purestorage.com>
- Add app.kubernetes.io/part-of label only when pxbackup.enabled=true
- Remove labels from StatefulSet volumeClaimTemplates

Co-authored-by: Bhanuchandra K <bk+pure@purestorage.com>
Signed-off-by: Kesavan Thiruvenkadasamy <kthiruvenkadasamy@purestorage.com>
pxb prometheus will not be consuming these metrics, so they are ignored using service monitor
- Add pxbackup.federated flag in values.yaml (default: false)
- Add PX_BACKUP_FEDERATED environment variable to px-backup deployment when federated mode is enabled
- Update README.md to document the new federated parameter

When pxbackup.federated is set to true, the PX_BACKUP_FEDERATED=true
environment variable will be injected into the px-backup container.
Add federated mode flag for PX-Backup
* PB-14502 | inject configmap as env in middleware

* PB-14502 | Flag added for federated PXB

---------

Co-authored-by: Ayush Sharma <aysharma@purestorage.com>
Signed-off-by: kgarg-px <kgarg@purestorage.com>
Co-authored-by: Ayush Sharma <aysharma@purestorage.com>
Copy link
Copy Markdown
Collaborator

@dbinnal-px dbinnal-px left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to update the index.yaml before merging to point pxb-federated branch

@yassharma-px yassharma-px force-pushed the 3.0.0-dev branch 4 times, most recently from 785b72a to 43e326b Compare March 24, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants