wan 2.2 cause nan in latent in i2v

[chaowenguo0]

Describe the bug

import asyncio, aiohttp.web, math, pathlib, torch, builtins, transformers, diffusers, PIL, av, io, numpy, uvloop, concurrent, multiprocessing, os, zipfile, sys
from modelscope.hub.snapshot_download import snapshot_download
from modelscope.hub.file_download import model_file_download
#import sd_embed.embedding_funcs

model_dir = snapshot_download('Wan-AI/Wan2.2-I2V-A14B-Diffusers')
onload_device=torch.device('cuda')
pipe = diffusers.WanImageToVideoPipeline.from_pretrained(model_dir, vae=diffusers.AutoencoderKLWan.from_pretrained(model_dir, subfolder='vae', torch_dtype=torch.float32).to(onload_device),
low_cpu_mem_usage=True,
torch_dtype=torch.bfloat16,
quantization_config=diffusers.quantizers.PipelineQuantizationConfig(quant_backend='bitsandbytes_4bit', quant_kwargs={'load_in_4bit':True, 'bnb_4bit_quant_type':'nf4', 'bnb_4bit_compute_dtype':torch.bfloat16}, components_to_quantize=['transformer', 'transformer_2']))
diffusers.hooks.apply_group_offloading(pipe.text_encoder, onload_device=onload_device, offload_type='leaf_level', use_stream=True, record_stream=True, non_blocking=True, low_cpu_mem_usage=True)
pipe.transformer.enable_group_offload(onload_device=onload_device, offload_type='leaf_level', use_stream=True, record_stream=True, non_blocking=True, low_cpu_mem_usage=True)
pipe.transformer.enable_layerwise_casting(storage_dtype=torch.float8_e4m3fn, compute_dtype=torch.bfloat16)
pipe.transformer_2.enable_group_offload(onload_device=onload_device, offload_type='leaf_level', use_stream=True, record_stream=True, non_blocking=True, low_cpu_mem_usage=True)
pipe.transformer_2.enable_layerwise_casting(storage_dtype=torch.float8_e4m3fn, compute_dtype=torch.bfloat16)
pipe.vae.enable_tiling()
pipe.vae.enable_slicing()

async def config(request):
return aiohttp.web.json_response({1:2})

def decode_tensors(pipe, step, timestep, callback_kwargs):
latents = callback_kwargs["latents"]
print(step, latents)
return callback_kwargs

def generate(prompt, negative, image):
with zipfile.ZipFile('output.zip', 'w') as output:
buffer = io.BytesIO()
with av.open(buffer, mode='w', format='mp4') as writer:
stream = writer.add_stream('h264', rate=8)
stream.width = 1280
stream.height = 720
stream.pix_fmt = 'yuv420p'
#prompt_embeds, prompt_neg_embeds = sd_embed.embedding_funcs.get_weighted_text_embeddings_sd15(pipe, prompt=prompt, neg_prompt=negative)
for frame in pipe(image=PIL.Image.open(io.BytesIO(image)).convert('RGB').resize((stream.width, stream.height)), prompt=prompt, negative_prompt=negative, width=stream.width, height=stream.height, num_frames=81, num_inference_steps=30, callback_on_step_end=decode_tensors).frames[0]: writer.mux(stream.encode(av.VideoFrame.from_ndarray((frame * 255).astype(numpy.uint8))))
writer.mux(stream.encode())
output.writestr('output.mp4', buffer.getvalue())

Reproduction

see above

Logs

System Info

ubuntu22.04-cuda12.4.0-py311-torch2.8.0 diffusers 0.35.2

Who can help?

@yiyixuxu @DN6

[asomoza]

Hi @chaowenguo0 we need a smaller and simple code snippet with just diffusers, your code is not formatted but I can see that you have a quantization_config but never use it, have commented parts and external libraries.

Can you please remove all the functions, all the comments and any external libraries or custom processing you are using.

As an example, a working WAN 2.2 T2V example with formatted code and just diffusers would be like this:

import torch

from diffusers import AutoencoderKLWan, WanPipeline
from diffusers.utils.export_utils import export_to_video


dtype = torch.bfloat16
device = "cuda"

model_id = "Wan-AI/Wan2.2-T2V-A14B-Diffusers"
vae = AutoencoderKLWan.from_pretrained(model_id, subfolder="vae", torch_dtype=torch.float32)
pipe = WanPipeline.from_pretrained(model_id, vae=vae, torch_dtype=dtype)

pipe.enable_model_cpu_offload()

height = 720
width = 1280
num_frames = 81
num_inference_steps = 40
guidance_scale = 4.0
guidance_scale_2 = 3.0
generator = torch.Generator(device=device).manual_seed(42)


prompt = "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
negative_prompt = "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"

output = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=height,
    width=width,
    num_frames=num_frames,
    guidance_scale=guidance_scale,
    guidance_scale_2=guidance_scale_2,
    num_inference_steps=num_inference_steps,
    generator=generator,
).frames[0]

export_to_video(output, "wan22_t2v.mp4", fps=16)

[chaowenguo0]

import torch, diffusers, PIL, av, numpy
from modelscope.hub.snapshot_download import snapshot_download

model_dir = snapshot_download('Wan-AI/Wan2.2-T2V-A14B-Diffusers')
onload_device=torch.device('cuda')
pipe = diffusers.WanPipeline.from_pretrained(model_dir, vae=diffusers.AutoencoderKLWan.from_pretrained(model_dir, subfolder='vae',      torch_dtype=torch.float32).to(onload_device),
    low_cpu_mem_usage=True,
    torch_dtype=torch.bfloat16,
    quantization_config=diffusers.PipelineQuantizationConfig(quant_backend='bitsandbytes_4bit', quant_kwargs={'load_in_4bit':True, 'bnb_4bit_quant_type':'nf4', 'bnb_4bit_compute_dtype':torch.bfloat16}, components_to_quantize=['transformer', 'transformer_2']))
diffusers.hooks.apply_group_offloading(pipe.text_encoder, onload_device=onload_device, offload_type='leaf_level', use_stream=True, record_stream=True, non_blocking=True, low_cpu_mem_usage=True)
pipe.transformer.enable_group_offload(onload_device=onload_device, offload_type='leaf_level', use_stream=True, record_stream=True, non_blocking=True, low_cpu_mem_usage=True)
pipe.transformer_2.enable_group_offload(onload_device=onload_device, offload_type='leaf_level', use_stream=True, record_stream=True, non_blocking=True, low_cpu_mem_usage=True)
pipe.vae.enable_tiling()
pipe.vae.enable_slicing()

def decode_tensors(pipe, step, timestep, callback_kwargs):
    latents = callback_kwargs["latents"]
    print(step, latents)
    return callback_kwargs

prompt = "On October 1, 2025, the U.S. federal government officially entered a shutdown as Congress failed to pass the new budget, leaving millions of public services and federal employees in limbo."
negative = 'Bright tones, overexposed, static, blurry details, subtitles, style, work, painting, picture, still, overall grayish, worst quality, low quality, JPEG compression residue, ugly, incomplete, redundant fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, deformed limbs, finger fusion, still picture A messy background, three legs, a lot of people in the background, walking backwards'

diffusers.export_to_video(pipe(prompt=prompt, negative_prompt=negative, width=720, height=1280, num_frames=81, num_inference_steps=30, callback_on_step_end=decode_tensors).frames[0], "output.mp4", fps=8)

quantization_config is a must, I can not use it in my computer, without it will cause oom kill
from modelscope.hub.snapshot_download import snapshot_download
from modelscope.hub.file_download import model_file_download
is a must, I can not download from huggingface, although I use huggingface mirror, it is very slow and takes 30 mins not finish download
i do not know layerwise casting can not be working with quantization. I just use all the methods to reduce memory usage to avoid oom kill.
also note that similar issue happen to WanImageToVideoPipeline Wan-AI/Wan2.2-I2V-A14B-Diffusers
@asomoza I already format my code to the most simple one. Please take a look to it.

the av is pyav, diffusers depend on imageio, imagio depend on pyav. but the recently update of pyav cause bug and incompatabile with imagio, i have to use av directly.

[asomoza]

Still not good, you have unused imports, incorrect indentation and please you can just use ```python to start a python code block. You're also using a decode_tensors custom function and some custom image conversion.

Just a copy & paste into my IDE gives me a lot of warnings and errors, and I don't have that much free time to start correcting them to see what's wrong with your code.

Also why are you using layerwise casting with quantization? That really doesn't make any sense to me.

[asomoza]

quantization_config is a must, I can not use it in my computer, without it will cause oom kill

that perfectly fine, I've tested WAN with quantization and it works without any problem

from modelscope.hub.snapshot_download import snapshot_download
from modelscope.hub.file_download import model_file_download
is a must, I can not download from huggingface, although I use huggingface mirror, it is very slow and takes 30 mins not finish download

this is not an issue either, I can use the repo_id in the pipeline, however, if it works for me and not for you, I can't test it with modelscope because for me is the opposite, really slow.

i do not know layerwise casting can not be working with quantization. I just use all the methods to reduce memory usage to avoid oom kill.

you can't just use everything and expect them to work, you will need a basic understanding of what you're doing. The quantization you're using is converting the weights to 4-bit, layerwise casting converts the weights to 8-bits and upscale them to half precision on inference making the quantization before useless and honestly I don't know what happens here because using them together doesn't make any sense. Please remove either quantization or layerwise casting and test your code.

also note that similar issue happen to WanImageToVideoPipeline Wan-AI/Wan2.2-I2V-A14B-Diffusers
@asomoza I already format my code to the most simple one. Please take a look to it.

I suspect that the reason it also happens with the other wan models is because of your code and not diffusers, we have tests to ensure that they work, and also I tested and seen other people using it without problems, so my initial guess is that it is something in your code and not diffusers.

Finally, just a copy & paste of your code shows me errors, you have a double ( in your function (are you testing your code?) and I don't have av installed, if the issue is a nan latent, you don't really need to use any video encoding or just the default which doesn't need the use of a stream or a yuv420p conversion.

[chaowenguo0]

the most headache part is i can not use the original code as shown in huggingface. that cause oom kill or network issue. but the doc in huggingface reduce memory and quatization just list a Iot of methods, but never tell they can not be used together.

[asomoza]

maybe the it would be better if you give me your specs (RAM and VRAM) and I can help you with the code

[chaowenguo0]

I already remove the layerwise casting part, still get NaN in all the lattens. my ram 64g, 48g ram on gpu on ada gpu machine.

[asomoza]

mmhm with those specs it doesn't make sense that you get OOM without any optimization, the code I posted here runs on a 48GB GPU (L40S) without any issue.

You don't need quantization or vae tiling with those specs, just using enable_model_cpu_offload() should be sufficient.

P.S.: take into account that the official examples use the vae with float32

[chaowenguo0]

enable_model_cpu_offload() is extremely slower than group offloading. i just can use my machine in a sesseion about 10 hours. then my machine will auto restart. enable_model_cpu_offload() can not produce the result.

…

On Sat, Nov 8, 2025, 9:25 AM Álvaro Somoza ***@***.***> wrote: *asomoza* left a comment (huggingface/diffusers#12613) <#12613 (comment)> mmhm with those specs it doesn't make sense that you get OOM without any optimization, the code I posted here <#12613 (comment)> runs on a 48GB GPU (L40S) without any issue. You don't need quantization or vae tiling with those specs, just using enable_model_cpu_offload() should be sufficient. — Reply to this email directly, view it on GitHub <#12613 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BWUG5QD3CVPAJVPJRQMOSA333YRPZAVCNFSM6AAAAACLQWTVLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMBWG4ZTIOJXGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[asomoza]

enable_model_cpu_offload() is extremely slower than group offloading

You're confusing things here, did you test what you are saying?

What you did when you just put every optimization technique was to lower the VRAM consumption which is completely different than speeding up inference.

group_offloading is for when you are VRAM constrained, and what it does is (with leaf and cuda streams) is to move the current layer to the GPU while preloading the next one, model cpu offload moves the complete model to the GPU, if you have the VRAM, offloading the whole model and group offloading with leaf and cuda streams will be almost the same, if you don't use leaf, cuda streams and enable low_cpu_mem_usage it will be slower, and this is also very dependent on your cpu and ram speed, and since you don't have enough RAM you can't use it with the fast configuration.

If you want faster inference, you will need something like torch.compile and the lighting loras with 4-steps, also you will probably have a speed increase if you use fp8 that ADA GPUs have native support, but not sure how much more. You can also use flash attention and If you don't care about quality, you can use a different attention backend or caching.

the av is pyav, diffusers depend on imageio, imagio depend on pyav. but the recently update of pyav cause bug and incompatabile with imagio, i have to use av directly.

we use the ffmpeg plugin imageio-ffmpeg which doesn't depend on pyav, but as I said, if your issue is with the latents that are NaN the video encoder is not relevant and your code reproduction can be a lot cleaner if you remove that part.

[asomoza]

BTW this takes a lot of time to test, but I'll post a script that makes WAN 2.2 run fast, but if your issue is also the loading time and you don't care about lowering the quality, I can use fp8, so let me know if you want quantization.

[LukeLIN-web]

also has this bug

    )
lib/python3.13/site-packages/diffusers/pipelines/wan/pipeline_wan_i2v.py", line 211, in _get_t5_prompt_embeds
    prompt = [prompt_clean(u) for u in prompt]
              ~~~~~~~~~~~~^^^
env/lib/python3.13/site-packages/diffusers/pipelines/wan/pipeline_wan_i2v.py", line 109, in prompt_clean
    text = whitespace_clean(basic_clean(text))
                            ~~~~~~~~~~~^^^^^^
env/lib/python3.13/site-packages/diffusers/pipelines/wan/pipeline_wan_i2v.py", line 97, in basic_clean
    text = ftfy.fix_text(text)
           ^^^^
NameError: name 'ftfy' is not defined