Issue with FluxKontextPipeline

[nitinmukesh]

Describe the bug

If we pass width/height in inference no changes in output image.
If we do not pass width/height, it defaults to 1024 x 1024 and crop the image.

Source image used (832 x 1216)

Reproduction

Working code but wrong image size as defaults to 1024 x 1024

import torch
from diffusers import FluxKontextPipeline
from diffusers.utils import load_image

pipeline = FluxKontextPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-Kontext-dev", 
    torch_dtype=torch.bfloat16
)
pipeline.enable_sequential_cpu_offload()
image = load_image(
    "https://huggingface.co/datasets/newgenai79/testing/resolve/main/women_with_hat.png"
).convert("RGB")

prompt = "Remove the hat"
image = pipeline(
    image=image, 
    prompt=prompt, 
    guidance_scale=2.5
).images[0]
image.save("flux-kontext-dev_working.png")

Output image (1024 x 1024)

---

Not working

import torch
from diffusers import FluxKontextPipeline
from diffusers.utils import load_image

pipeline = FluxKontextPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-Kontext-dev", 
    torch_dtype=torch.bfloat16
)
pipeline.enable_sequential_cpu_offload()
image = load_image(
    "https://huggingface.co/datasets/newgenai79/testing/resolve/main/women_with_hat.png"
).convert("RGB")

prompt = "Remove the hat"
image = pipeline(
    image=image, 
    prompt=prompt, 
    width=832,
    height=1216,
    guidance_scale=2.5
).images[0]
image.save("flux-kontext-dev2.png")

Output image (832 x 1232)

Generation height and width have been adjusted to 1232 and 832 to fit the model requirements.

Logs

Not required

System Info

latest from source

Who can help?

@a-r-r-o-w

[a-r-r-o-w]

@nitinmukesh There is a private marked parameter that you can pass to disable the automatic resizing:

diffusers/src/diffusers/pipelines/flux/pipeline_flux_kontext.py

Line 759 in 0454fbb

_auto_resize: bool = True,

We do the resizing by default to keep the implementation aligned with original implementation, and because there are only certain specific resolutions that Kontext was trained with. Other resolutions may work but how it affects the quality is unknown.

[nitinmukesh]

Sorry I should have mentioned the issue is not related to cropping. Instead this is not working, output is same as input.

import torch
from diffusers import FluxKontextPipeline
from diffusers.utils import load_image

pipeline = FluxKontextPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-Kontext-dev", 
    torch_dtype=torch.bfloat16
)
pipeline.enable_sequential_cpu_offload()
image = load_image(
    "https://huggingface.co/datasets/newgenai79/testing/resolve/main/women_with_hat.png"
).convert("RGB")

prompt = "Remove the hat"
image = pipeline(
    image=image, 
    prompt=prompt, 
    width=832,
    height=1216,
    guidance_scale=2.5
).images[0]
image.save("flux-kontext-dev2.png")

[a-r-r-o-w]

This does not seem like a diffusers-side issue to me and could probably be related to the model and how it was trained. Maybe a more detailed instruction or playing with guidance parameters may help. The following works for me:

import torch
from diffusers import FluxKontextPipeline
from diffusers.utils import load_image

pipeline = FluxKontextPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-Kontext-dev", 
    torch_dtype=torch.bfloat16
)
pipeline.to("cuda")
image = load_image(
    "https://huggingface.co/datasets/newgenai79/testing/resolve/main/women_with_hat.png"
).convert("RGB")

prompt = "In this picture a woman is wearing a hat. Remove the hat and make her wear a tiara"
image = pipeline(
    image=image, 
    prompt=prompt, 
    width=832,
    height=1216,
    guidance_scale=4.5
).images[0]
image.save("output.png")

[nitinmukesh]

Thank you, Aryan.