[P2] Jobs are Queued - autoscaled-machines #360
Description
Jobs are queueing, please investigate. Max queue time: [no value] mins, Max queue size: [no value] runners. Please visit http://hud.pytorch.org/metrics to see which runners are queueing.
Alert Details
- Occurred At: Oct 10, 1:35pm PDT
- State: FIRING
- Team: pytorch-dev-infra
- Priority: P2
- Description: Alerts when any of the regular runner types is queuing for a long time or when many of them are queuing.
- Runbook: https://hud.pytorch.org/metrics
- View Alert: https://pytorchci.grafana.net/alerting/grafana/dez2aomgvru2oe/view?orgId=1
- Silence Alert: https://pytorchci.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Ddez2aomgvru2oe&matcher=rulename%3DJobs+are+Queued+-+autoscaled-machines&matcher=type%3Dalerting-infra&orgId=1
- Source: grafana
- Fingerprint:
8df406a1ae5c4cc7f49088ab7a287e854ae48489938d6e08db51022d10b3d697