[P2] Jobs are Queued - autoscaled-machines #311
Description
Jobs are queueing, please investigate. Max queue time: 62 mins, Max queue size: 7 runners. Please visit http://hud.pytorch.org/metrics to see which runners are queueing.
Alert Details
- Occurred At: Oct 9, 7:48am PDT
- State: FIRING
- Team: pytorch-dev-infra
- Priority: P2
- Description: Alerts when any of the regular runner types is queuing for a long time or when many of them are queuing.
- Reason: max_queue_size=7, max_queue_time_mins=62, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
- Runbook: https://hud.pytorch.org/metrics
- View Alert: https://pytorchci.grafana.net/alerting/grafana/dez2aomgvru2oe/view?orgId=1
- Silence Alert: https://pytorchci.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Ddez2aomgvru2oe&matcher=type%3Dalerting-infra&orgId=1
- Source: grafana
- Fingerprint:
6cb879982663494a82bd6a1e362f44e5a8b053fa901388436b27da8f793bbf58