Unstable loss during Kontext lora training

[Twinkle-ce]

During Kontext LoRA training (learning rate = 1e-5, rank = 64), the training loss shows large fluctuations across epochs, sometimes dropping to ~0.02 and then surging above 0.7 within the same epoch. Is this degree of instability normal for LoRA training？

[Artiprocher]

@Twinkle-ce The loss value depends on the timestep; during each training step, a timestep is randomly sampled and the loss is computed accordingly. Oscillations in the loss are therefore normal. Please use a moving average to obtain a smoothed loss curve.