[P2] Jobs are Queued - Nvidia B200s

[pytorch-alerts]

Nvidia B200 jobs are queueing, please investigate. Max queue time: 241 mins, Max queue size: 8 runners. Please visit http://hud.pytorch.org/metrics to see which runners are queueing.

Alert Details

• Occurred At: Dec 2, 3:41pm PST
• State: FIRING
• Teams: pytorch-dev-infra, nvidia-infra
• Priority: P2
• Description: Alerts when the B200 runners are queuing for a long time or when many of them are queuing.
• Reason: max_queue_size=8, max_queue_time_mins=241, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
• Runbook: https://hud.pytorch.org/metrics
• View Alert: https://pytorchci.grafana.net/alerting/grafana/eez5ua39adslce/view?orgId=1
• Silence Alert: https://pytorchci.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Deez5ua39adslce&matcher=type%3Dalerting-infra&orgId=1
• Source: grafana
• Fingerprint: c13639384b5c8c24bb89711a42bbdc76c3eee865c371a943b41e875a13103436

[pytorch-alerts]

Alert Update

• State: FIRING
• Occurred At: Dec 2, 3:41pm PST
• Reason: max_queue_size=11, max_queue_time_mins=481, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
  The alert condition is still active.
  Fingerprint: c13639384b5c8c24bb89711a42bbdc76c3eee865c371a943b41e875a13103436

[pytorch-alerts]

Alert Update

• State: FIRING
• Occurred At: Dec 2, 3:41pm PST
• Reason: max_queue_size=20, max_queue_time_mins=725, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
  The alert condition is still active.
  Fingerprint: c13639384b5c8c24bb89711a42bbdc76c3eee865c371a943b41e875a13103436

[pytorch-alerts]

Alert Update

• State: FIRING
• Occurred At: Dec 2, 3:41pm PST
• Reason: max_queue_size=20, max_queue_time_mins=726, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
  The alert condition is still active.
  Fingerprint: c13639384b5c8c24bb89711a42bbdc76c3eee865c371a943b41e875a13103436

[pytorch-alerts]

Alert Update

• State: FIRING
• Occurred At: Dec 2, 3:41pm PST
• Reason: max_queue_size=27, max_queue_time_mins=966, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
  The alert condition is still active.
  Fingerprint: c13639384b5c8c24bb89711a42bbdc76c3eee865c371a943b41e875a13103436

[pytorch-alerts]

Alert Update

• State: FIRING
• Occurred At: Dec 2, 3:41pm PST
• Reason: max_queue_size=31, max_queue_time_mins=1207, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
  The alert condition is still active.
  Fingerprint: c13639384b5c8c24bb89711a42bbdc76c3eee865c371a943b41e875a13103436

[pytorch-alerts]

Alert Update

• State: FIRING
• Occurred At: Dec 2, 3:41pm PST
• Reason: max_queue_size=39, max_queue_time_mins=1424, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
  The alert condition is still active.
  Fingerprint: c13639384b5c8c24bb89711a42bbdc76c3eee865c371a943b41e875a13103436

[pytorch-alerts]

Alert Update

• State: FIRING
• Occurred At: Dec 2, 3:41pm PST
• Reason: max_queue_size=38, max_queue_time_mins=1432, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
  The alert condition is still active.
  Fingerprint: c13639384b5c8c24bb89711a42bbdc76c3eee865c371a943b41e875a13103436

[pytorch-alerts]

Alert Update

• State: FIRING
• Occurred At: Dec 2, 3:41pm PST
• Reason: max_queue_size=43, max_queue_time_mins=1387, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
  The alert condition is still active.
  Fingerprint: c13639384b5c8c24bb89711a42bbdc76c3eee865c371a943b41e875a13103436

[pytorch-alerts]

Alert Update

• State: FIRING
• Occurred At: Dec 2, 3:41pm PST
• Reason: max_queue_size=36, max_queue_time_mins=1413, queue_size_threshold=0, queue_time_threshold=1, threshold_breached=1
  The alert condition is still active.
  Fingerprint: c13639384b5c8c24bb89711a42bbdc76c3eee865c371a943b41e875a13103436