True Elastic DDP on spot instances with K8s StatefulSet workers

[akhileshh]

NOTE: rbac has been updated, will need to be applied in each cluster.

Summary

• Add elastic distributed training support via torch elastic launch: master Job on standard node + worker StatefulSet on spot nodes, with headless Service for stable DNS-based rendezvous
• Add DCP (distributed checkpoint) support with async saves, automatic format detection on load, and SIGTERM-triggered graceful exit for spot preemption resilience
• Add K8s pod monitoring infrastructure: log capture to GCS, pod disruption/event watchers, rank0 log streaming, and compute cost tracking
• Reduce log noise from job polling, secret operations, and K8s event streams (batched per watch window)

Key components

• K8s StatefulSet + headless Service: Workers get stable DNS names that survive spot rescheduling; master hosts c10d rendezvous store
• ZettaDefaultTrainer: Extended with DCP checkpointing (distributed) and AsyncCheckpointIO (single-node), periodic time-based checkpoints, configurable DDP timeout for fast failure detection
• SIGTERMCheckpointCallback: Graceful exit on preemption — sets flag, finishes current step, then exits (no DCP save in handler to avoid distributed deadlock)
• lightning_train: New params for elastic training (min_nodes, max_restarts, provisioning_model, rdzv_configs), automatic StatefulSet/Service lifecycle management
• Pod monitoring: capture_pod_logs, watch_for_pod_disruptions, watch_for_run_events, follow_rank0_logs — all run as background threads during training

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.98%. Comparing base (2b01257) to head (3e8332e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #1186    +/-   ##
========================================
  Coverage   99.98%   99.98%            
========================================
  Files         192      196     +4     
  Lines       10199    10312   +113     
========================================
+ Hits        10197    10310   +113     
  Misses          2        2            

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.