-
Notifications
You must be signed in to change notification settings - Fork 6.4k
Modify apply_overlay for inpainting with padding_mask_crop (Inpainting area: "Only Masked") #8793
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Modify apply_overlay for inpainting with padding_mask_crop (Inpainting area: "Only Masked") #8793
Conversation
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
sorry for the late answer, I'll try to test this today or as soon as possible. |
can you run |
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
@asomoza
|
thanks, it seems the script is not working for you, it should fix the errors for your files only and not give a log for all the other files. Probably a wrong version or configuration of ruff. If you want I can do it for you. |
this change looks good to me and makes sense. IMO the user that utilizes this option should ensure that the mask has the correct size instead of us enforcing it. WDYT @yiyixuxu? |
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
…g area: "Only Masked") (#8793) * Modify apply_overlay for inpainting * style --------- Co-authored-by: root <root@debian> Co-authored-by: Álvaro Somoza <[email protected]> Co-authored-by: yiyixuxu <[email protected]>
What does this PR do?
First of all, thanks for your great work. Here is my personal understanding. If there are any mistakes, feel free to correct me!
Regardless of whether it is the official documentation's description of the usage of the
padding_mask_crop
parameter or the actual effect when the inpainting area is set to "Only masked" when using the AUTOMATIC1111 WebUI, the original input image size should be maintained, thereby eliminating the need for additional super-resolution operations.The description of the docs about padding_mask_crop :
However, in practice, when this feature is enabled in diffusers, the method
apply_overlay
will resize the init_image to the size of actually inpainted part(512 * 512 in default) at first. And if the input image to the pipeline is not resized before generation, the overlaid result will be incorrect. On the other hand, resizing the original image at the input stage fails to preserve the original image size. It will significantly degrades the image quality and necessitating super-resolution to restore it.I don't think this logic aligns with the original intent of this feature and differs from the implementation in automatic1111. Therefore, I have modified the apply_overlay function accordingly to ensure that the output image retains the same size as the original image.
Here is the comparison.
Original Image and mask


If I do not resize the original image size before pipeline(the existing version code): the overlay result is incorrect and the image is also resized at the same time.

If I resize the original image at first: the overlay result is correct. But it degrades the image quality since it has resized the init image size to 512.

Modified Version: The output image is of the same size as the original input and the overlaid result is correct.

Using AUTOMATIC111 WebUI: Just select the checkbox shown below. The output image is definitely the same size as the original image without resizing.


Test Code
Modified Code: I modify the code as below.
original code:
apply_overlay
in src/diffusers/image_processor Line 651https://github.com/huggingface/diffusers/blob/main/src/diffusers/image_processor.py#L651
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.