Kontext pipeline resizing image even if it is within PREFERRED_KONTEXT_RESOLUTIONS

[nitinmukesh]

Describe the bug

My assumption is, if resolution of image matches one of PREFERRED_KONTEXT_RESOLUTIONS, pipeline should not resize the input image.

Even though I use the image with one of the preferred resolution, the pipeline is resizing the image.

PREFERRED_KONTEXT_RESOLUTIONS = [
(672, 1568),
(688, 1504),
(720, 1456),
(752, 1392),
(800, 1328),
(832, 1248),
(880, 1184),
(944, 1104),
(1024, 1024),
(1104, 944),
(1184, 880),
(1248, 832),
(1328, 800),
(1392, 752),
(1456, 720),
(1504, 688),
(1568, 672),
]

Reproduction

I chose 672x1568 image for inference

Logs

Generation `height` and `width` have been adjusted to 1552 and 656 to fit the model requirements.

System Info

Latest diffusers

Who can help?

@a-r-r-o-w

Just want to quickly confirm if it's an issue or this is how it's supposed to work.

[DN6]

cc @a-r-r-o-w Could you take a look please.

[a-r-r-o-w]

I believe the behavior is the same as original repository. See here.

The following is the code path taken for width/height calculation:

PREFERRED_KONTEXT_RESOLUTIONS = [
    (672, 1568),
    (688, 1504),
    (720, 1456),
    (752, 1392),
    (800, 1328),
    (832, 1248),
    (880, 1184),
    (944, 1104),
    (1024, 1024),
    (1104, 944),
    (1184, 880),
    (1248, 832),
    (1328, 800),
    (1392, 752),
    (1456, 720),
    (1504, 688),
    (1568, 672),
]

height = 1568
width = 672
max_area = 1024**2
vae_scale_factor = 8

original_height, original_width = height, width
aspect_ratio = width / height
width = round((max_area * aspect_ratio) ** 0.5)
height = round((max_area / aspect_ratio) ** 0.5)

multiple_of = vae_scale_factor * 2
width = width // multiple_of * multiple_of
height = height // multiple_of * multiple_of

if height != original_height or width != original_width:
    print(
        f"Generation `height` and `width` have been adjusted to {height} and {width} to fit the model requirements."
    )

image_width, image_height = width, height
aspect_ratio = image_width / image_height
_, image_width, image_height = min(
    (abs(aspect_ratio - w / h), w, h) for w, h in PREFERRED_KONTEXT_RESOLUTIONS
)
image_width = image_width // multiple_of * multiple_of
image_height = image_height // multiple_of * multiple_of

print(width, height)
print(image_width, image_height)

We can see that 1568x672 image will have an area greater than the max area. This triggers the first resize, which results in the warning message (656x1552). But, in the second resizing for preferred kontext resolutions, we get back 1568x672. This ofcourse loses some information due to bigger image -> smaller image -> bigger image.

I can add a simple check to see if specified resolution is already in preferred resolutions, so that the first resize based on max area can be skipped

[a-r-r-o-w]

Oh actually I had a misunderstanding re-reading the code: the height/width warning you're getting is because the actual height and width parameters you pass corresponds to the resolution you want to generate at. This does not have anything to do with preferred kontext resolutions (which is just the recommended resolutions for resizing the context image).

You can rid of the warning by setting max_area to a higher value (as long as the resolution you want to generate is a multiple of 16). We defaulted to the one in original repo, but it can definitely be increased.

The diffusers implementation does not require updates.

[a-r-r-o-w]

I believe this can be closed. TLDR; increasing the max_area should fix the issue described here. Please LMK and feel free to re-open if this seems incorrect