Multi-GPU Results in Static Noise After Initial Frames

[pftq]

I have no problem generating a video with single GPU, but the same settings on 8-GPU results in heavy static noise across the video after the initial frame. Hunyuan I2V had this issue too as a result of the seed not being in sync across GPUs, but it seems like Skyreels V2 randomizes the seed outside the inference code, so that shouldn't be an issue.

My commandline prompt:

model_id=Skywork/SkyReels-V2-DF-14B-540P
torchrun --nproc_per_node=8 generate_video_df.py \
  --model_id ${model_id} \
  --resolution 540P \
  --ar_step 0 \
  --base_num_frames 97 \
  --num_frames 257 \
  --overlap_history 17 \
  --inference_steps 100 \
  --guidance_scale 8 \
  --image "image.jpg" \
  --prompt "" \
  --addnoise_condition 20 \
  --use_usp \
  --offload

[Howe2018]

Does the command specify a seed parameter? When performing multi-GPU inference, a fixed seed is required; otherwise, latents may vary across different GPU.

SkyReels-V2/generate_video.py

Line 56 in 0790f89

assert (args.use_usp and args.seed is not None) or (not args.use_usp), "usp mode need seed"

SkyReels-V2/generate_video_df.py

Line 47 in 0790f89

assert (args.use_usp and args.seed is not None) or (not args.use_usp), "usp mode need seed"

[Howe2018]

sorry, since the seed's default value is -1, the previous assertion check failed to catch it. We've now fixed this issue.#25

[pftq]

Ah - good to know thanks. Why is this affected if the seed randomization is happening before the inference? What would be the right way to handle this if we wanted to loop the inference call for multiple videos?

[pftq]

I made a more robust fix to this issue in this PR by broadcasting the randomly generated seed to all GPUs, along with other enhancements like batch mode (now possible with the random seed fix). Let me know if you guys are open to the requested changes:
#31