[Feature&Fix] 1. Add ETA in log 2. Fix error e2e_tgs when resuming

[HAOCHENYE]

No description provided.

[Copilot]

Pull Request Overview

This PR adds ETA (Estimated Time of Arrival) logging functionality and fixes an error when resuming training related to the e2e_tgs calculation. The changes track training time more accurately and provide users with estimated completion time information.

• Add ETA calculation and logging to show estimated training completion time
• Fix e2e_tgs calculation error when resuming training by properly tracking training time offset
• Persist training time offset in checkpoint state for accurate resume functionality

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

This will cause a KeyError when resuming from checkpoints created before this change, since 'train_time_offset' won't exist in older checkpoint files. Add a default value using train_state.get('train_time_offset', 0) to handle backward compatibility.

Suggested change

	self._train_time_offset = train_state["train_time_offset"]
	self._train_time_offset = train_state.get("train_time_offset", 0)

[Copilot]

This will cause a ZeroDivisionError if tgs is zero or very close to zero, which can happen during the first few training steps. Add a guard condition to check if tgs > 0 before performing the division.

Suggested change

	eta_seconds = remaining_tokens / tgs
	eta_hms = str(timedelta(seconds=int(eta_seconds)))
	if tgs > 1e-8:
	eta_seconds = remaining_tokens / tgs
	eta_hms = str(timedelta(seconds=int(eta_seconds)))
	else:
	eta_seconds = float('inf')
	eta_hms = "inf"