Create modular pipeline from existing pipeline

[vladmandic]

new concept of modular pipelines added via #9672 is very flexible way of creating custom pipelines
and one of the best early use-cases is new concept of modular guiders added via #11311

however, this would require a complete rewrite of the existing user apps/codebases to use new concepts
and would likely significantly slow down adoption (if not even block adoption for a long time)

ask here is to provide a way to use an existing pipeline to instantiate a modular pipeline,
very similar to how different standard diffuser pipelines can be instantiated
from a single pipeline class using from_pipe method

example of desired workflow:

import torch
import diffusers

# load pipeline using any normal method  
# such as DiffusionPipeline, AutoPipelineForText2Image, StableDiffusionPipeline, etc.  
pipe = diffusers.DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.bfloat16,
)

# create modular pipeline from loaded pipeline
modular = diffusers.ModularPipeline.from_pipe(pipe)

# create guider and activate it
cfg = diffusers.ClassifierFreeGuidance(guidance_scale=5.0, guidance_rescale=0.0, start=0.0, stop=1.0)
modular.update_states(guider=cfg)

output = modular(
    prompt='astronaut in a diner',
    height=1024, width=1024)

cc: @yiyixuxu @a-r-r-o-w @sayakpaul

[yiyixuxu]

thanks @vladmandic
We have a section in the doc here about using guider (the API changed a bit since that PR, update_states is update_components now https://huggingface.co/docs/diffusers/main/en/modular_diffusers/modular_pipeline#customizing-guidance-techniques)

I think it would be easy to add a mapping between existing pipelines to modular to support from_pipe API. But I'm not sure if it's neede, we have a set of "auto" blocks you can use to replace all pipelines we support at diffuesrs for that pipeline group( just SDXL for now but wan/flux/ltx coming up soon). For apps it should be the easiest way to use modular

from diffusers import StableDiffusionXLAutoBlocks
sdxl_modular = StableDiffusionXLAutoBlocks().init_pipeline(...)

we could make modular to work directly with standard repos like "stabilityai/stable-diffusion-xl-base-1.0"

[yiyixuxu]

doc here for auto
https://huggingface.co/docs/diffusers/main/en/modular_diffusers/auto_pipeline_blocks

[vladmandic]

thanks @yiyixuxu

anything that replaces current loaders would require massive code changes and would delay adoption on my side.
(plus we have more than few custom pipelines that do not exist in diffusers main, but that's another story)
in order to use modular pipelines, we need to find a way to create them dynamically from existing loaded standard pipeline.

if that cannot be done on high-level such as modular = ModularPipeline.from_pipe(pipe)
and preferred method would be something like modular = StableDiffusionXLAutoBlocks.from_pipe(pipe), that would still be fine.

in that case, we'd need to maintain a mapping of orig-pipeline to modular-pipeline to know
a) if modular equivalent exists, b) when and which to use, but that is not a problem.

btw, priority-wise for autoblocks, i'd still say that sd15 (amazing how popular it still is) and flux (by far most popular newer model) should follow closely after sdxl. then comes everything else.

[yiyixuxu]

@vladmandic

would this not work? (currently not, but we canmake this work, bit more general than from_pipe)

modular = StableDiffusionXLAutoBlocks.init_pipeline()
modular.update_components(**pipe.components, **pipe.config)

[vladmandic]

@yiyixuxu
yes, that would work nicely if existing pipes have components attribute and it can be directly used in such manner.
then we could automate it with something like this:

modular_map= {
    'StableDiffusionXLPipeline': 'StableDiffusionXLAutoBlocks`,
}

standard_pipe = diffusers.DiffusionPipeline.from_pretrained(...)
modular_cls = modular_map.get(standard_pipe.__class__.__name__)
modular_cls = getattr(diffusers, modular_cls)
modular_pipe = modular_cls.init_pipeline()
modular_pipe.update_components(**standard_pipe, **standard_pipe.config)

[yiyixuxu]

thanks @vladmandic
sounds good, will add that support shortly