Different generation with `Diffusers` in I2V tasks for LTX-video

[Kaihui-Cheng]

Describe the bug

Hello, I encountered an issue with the generation when attempting the I2V task using Diffusers. Is there any difference between the diffusers implementation and the LTX-video-inference scripts in the I2V task?

• The above is the result from the inference.py, and the following is the result generated with diffuser.
• Prompts: a person

img_to_vid_0_a-person_42_512x512x161_0.mp4

diffusers_512x512_a_person.mp4

diffusers_without_negative_prompt_512x512_a_person.mp4

• test img
  

Besides, it seems that the text prompt has a significant impact on the I2V generation with 'diffusers'. Could I be missing any important arguments?
https://huggingface.co/docs/diffusers/api/pipelines/ltx_video

• results

demo-A-young-girl-stands-calmly-in-the-foreground.mp4

demo-A-young-girl-stands-calmly.mp4

demo-A-young-girl-stands.mp4

demo-A-young-girl.mp4

Reproduction

• for LTX-video generation
  https://github.com/Lightricks/LTX-Video/blob/main/inference.py

python inference.py \
    --ckpt_path ./pretrained_models/LTX-Video \
    --output_path './samples' \
    --prompt "A person." \
    --input_image_path ./samples/test_cases.png \
    --height 512 \
    --width 512 \
    --num_frames 49 \
    --seed 42 

• for diffuser generation: it seems that the negative prompts are causing the issues. However, even when I remove them, the results are still not satisfactory.

import argparse
import torch
from diffusers import LTXVideoTransformer3DModel
from diffusers import LTXImageToVideoPipeline
from diffusers import FlowMatchEulerDiscreteScheduler, AutoencoderKLLTXVideo
from diffusers.utils import export_to_video, load_image, load_video


from moviepy import VideoFileClip, AudioFileClip
import numpy as np
from pathlib import Path
import os
import imageio
from einops import rearrange
from PIL import Image
import random

def seed_everething(seed: int):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed(seed)

def generate_video(args):

    pipe = LTXImageToVideoPipeline.from_pretrained(args.ltx_model_path, torch_dtype=torch.bfloat16)
    pipe.to("cuda")

    negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

    image = load_image(args.validation_image)
    prompt = "A person."
    negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"
    generator = torch.Generator(
        device="cuda" if torch.cuda.is_available() else "cpu"
    ).manual_seed(42)

    video = pipe(
        image=image,
        prompt=prompt,
        guidance_scale=3,
        # stg_scale=1,
        generator=generator,
        callback_on_step_end=None,
        negative_prompt=negative_prompt,
        width=512,
        height=512,
        num_frames=49,
        num_inference_steps=50,
        decode_timestep=0.05,
        decode_noise_scale=0.025,

    ).frames[0]
    export_to_video(video, args.output_file, fps=24)

• for demo images with difference text prompts
  https://huggingface.co/docs/diffusers/api/pipelines/ltx_video

import torch
from diffusers import LTXImageToVideoPipeline
from diffusers.utils import export_to_video, load_image

pipe = LTXImageToVideoPipeline.from_pretrained("./pretrained_models/LTX-Video", torch_dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image("samples/image.png")
prompt = "A young girl stands."
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

video = pipe(
    image=image,
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=704,
    height=480,
    num_frames=161,
    num_inference_steps=50,
).frames[0]
modified_prompt = "-".join(prompt.split()[:14])
export_to_video(video, f"samples/test_out/demo-{modified_prompt}.mp4", fps=24)

Logs

System Info

torch 2.4.1
torchao 0.7.0
torchvision 0.19.1
diffusers 0.32.1
python 3.10

Who can help?

No response

[bghira]

i can confirm this is how it's working. it seems a bug in Diffusers.

[DN6]

cc: @a-r-r-o-w

[a-r-r-o-w]

Thanks for reporting and the detailed thread - I'll give this a deeper look soon. There are few things to consider here:

• Short prompts don't really work with LTX Video. To make a fair comparison, I would try using the exact same latents and prompt embeddings in both implementations. On our end, during the integration, we matched the transformer and pipeline implementations 1:1 numerically so there should be no difference if all starting conditions are the same.
• Making sure that nothing like STG or any other additions made to the original repo pipeline are being used.
• After the diffusers integration, they added some noise perturbation to first latent of image-to-video generation to improve generation quality. Diffusers pipeline does not have this yet

[tin2tin]

@a-r-r-o-w There is some talk about the vae here: https://huggingface.co/calcuis/ltxv-gguf/discussions/1

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[kxhit]

@a-r-r-o-w @Kaihui-Cheng
Hi all, I tested the diffusers and the og LTX img2video, the diffusers output is worse than the og version. Is the problem solved or is any more update? Thanks!

[a-r-r-o-w]

@kxhit, the original implementation added a few changes to the image-to-video generation that improves quality. We are yet to do the same & port the necessary changes. Some of the changes that we would need are:

• per-token timesteps (I've implemented this elsewhere for finetuning is a simple change we can introduce without problems)
• spatio-temporal guidance (We're ideating what's the best way to restructure/design our pipelines to work with different guidance methods)
• initial noise addition upto a threshold to the first-frame latent for inference time adaptation to training setting

I'd be happy to PR when I have the bandwidth but otherwise contributions are welcome for (1) and (3). These changes did not exist at the time we integrated LTX-Video first, so there's now a significant difference in generation quality.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[abcdvzz]

@a-r-r-o-w Hi can you share some update about this? how about the first per-token timestep, where did you implement it? Can you share it with us?

[a-r-r-o-w]

Hi @abcdvzz

per-token timesteps is implemented here:

diffusers/src/diffusers/schedulers/scheduling_flow_match_euler_discrete.py

Line 436 in 5dc503a

if per_token_timesteps is not None:

Spatio-temporal guidance has been implemented with #11311. We will soon add LTXVideo support to modular diffusers for this to be readily usable

Initial noise addition has not yet been supported