[Modular] support standard repo

[yiyixuxu]

fix #11915

this PR support using standard repo in Modular system

option1

from diffusers import ModularPipeline
import torch

pipe = ModularPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16)
print(pipe)

option2

from diffusers import StableDiffusionXLAutoBlocks
modular_pipe = StableDiffusionXLAutoBlocks().init_pipeline("stabilityai/stable-diffusion-xl-base-1.0")
modular_pipe.load_default_components(torch_dtype=torch.float16)

option3

from diffusers import DiffusionPipeline, StableDiffusionXLAutoBlocks
import diffusers

modular_map= {
    'StableDiffusionXLPipeline': 'StableDiffusionXLAutoBlocks',
}

standard_pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16)
modular_cls = modular_map.get(standard_pipe.__class__.__name__)
modular_cls = getattr(diffusers, modular_cls)
blocks = modular_cls()
modular_pipe = blocks.init_pipeline()

components_to_update = {k: v for k, v in standard_pipe.components.items() if v is not None}
modular_pipe.update_components(**components_to_update, **standard_pipe.parameters)

[yiyixuxu]

cc @vladmandic let me know if this works for you

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[vladmandic]

thanks @yiyixuxu, this is pretty much what i asked for!
ran some basic tests and converting pipeline works fast (about ~0.02sec) and without apparent leaks
(see my example code below)

few comments:

• warnings
  there are several warnings posted during the conversion and later during generate request
  see below for details
• outputs
  standard pipeline returns StableDiffusionXLPipelineOutput with .images property
  modular pipeline returns PipelineState which does not have images property and instead
  images is only found .intermediates['images']
  in general, i really like PipelineState, but it should have images property directly available
• differences
  resulting image from standard pipeline and modular pipeline is NOT the same
  my guess defaults for things like guidance are not the same, but didnt dig too deep
  and users really do care and need reproducibility
  see below for both images

warnings

warnings during convert:

ModularPipeline.update_components: text_encoder has no valid _diffusers_load_id. Updating the component but skipping spec update, use ComponentSpec.load() for proper specs
ModularPipeline.update_components: text_encoder_2 has no valid _diffusers_load_id. Updating the component but skipping spec update, use ComponentSpec.load() for proper specs
ModularPipeline.update_components: tokenizer has no valid _diffusers_load_id. Updating the component but skipping spec update, use ComponentSpec.load() for proper specs
ModularPipeline.update_components: tokenizer_2 has no valid _diffusers_load_id. Updating the component but skipping spec update, use ComponentSpec.load() for proper specs
ModularPipeline.update_components: unet has no valid _diffusers_load_id. Updating the component but skipping spec update, use ComponentSpec.load() for proper specs
ModularPipeline.update_components: vae has no valid _diffusers_load_id. Updating the component but skipping spec update, use ComponentSpec.load() for proper specs
ModularPipeline.update_components: scheduler has no valid _diffusers_load_id. Updating the component but skipping spec update, use ComponentSpec.load() for proper specs

warnings during generate:

diffusers/modular_pipelines/modular_pipeline.py:445: UserWarning: Multiple different default values found for input 'strength': 0.3 (from block 'set_timesteps') and 0.9999 (from block 'prepare_latents'). Using 0.3.
skipping auto block: StableDiffusionXLAutoIPAdapterStep
skipping auto block: StableDiffusionXLAutoVaeEncoderStep
skipping auto block: StableDiffusionXLAutoControlNetInputStep

images

standard->modular->standard

code

# pip install git+https://github.com/huggingface/diffusers@modular-standard-repo

import torch
import diffusers


model = '/mnt/models/stable-diffusion/mine/tempest-by-vlad-0.1.safetensors' # https://civitai.com/models/1157409/tempest-by-vlad 
cache_dir = '/mnt/models/huggingface'
modular_map= {
    'StableDiffusionXLPipeline': 'StableDiffusionXLAutoBlocks',
}


def get_args():
    return {
        'prompt': 'a photo of an astronaut in a diner',
        'num_inference_steps': 20,
        'generator': torch.Generator(device='cuda').manual_seed(42),
    }


def convert_to_modular(standard_pipe):
    try:
        modular_cls = modular_map.get(standard_pipe.__class__.__name__, None)
        if modular_cls is None:
            raise ValueError(f'unknown: cls={standard_pipe.__class__.__name__}')
        modular_cls = getattr(diffusers, modular_cls, None)
        if modular_cls is None:
            raise ValueError(f'invalid: cls={standard_pipe.__class__.__name__}')
        modular_blocks = modular_cls()
        modular_pipe = modular_blocks.init_pipeline()
        components_dct = {k: v for k, v in standard_pipe.components.items() if v is not None}
        modular_pipe.update_components(**components_dct, **standard_pipe.parameters)
        modular_pipe.original_pipe = standard_pipe
        print(f'convert: from={standard_pipe.__class__.__name__} to={modular_pipe.__class__.__name__}')
        return modular_pipe
    except Exception as e:
        print(f'convert: error={e}')
        raise e


def restore_standard(modular_pipe):
    if hasattr(modular_pipe, 'original_pipe'):
        print(f'convert: from={modular_pipe.__class__.__name__} to={modular_pipe.original_pipe.__class__.__name__}')
        return modular_pipe.original_pipe


def gc():
    import gc
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()
    print(f'memory: free={torch.cuda.mem_get_info()[0]} peak={torch.cuda.memory_stats()["allocated_bytes.all.peak"]}')
    torch.cuda.reset_peak_memory_stats()


if __name__ == '__main__':
    print('loading')
    pipe = diffusers.StableDiffusionXLPipeline.from_single_file(model, torch_dtype=torch.bfloat16, cache_dir=cache_dir).to('cuda')
    gc()

    print('generate: standard')
    output = pipe(**get_args())
    print(f'output: {output}')
    gc()
    image = output.images[0]
    image.save('standard-default1.png')

    print('generate: modular')
    pipe = convert_to_modular(pipe)
    output = pipe(**get_args())
    gc()
    print(f'output: {output}')
    image = output.intermediates['images'][0]
    image.save('standard-modular1.png')

    print('generate: standard')
    pipe = restore_standard(pipe)
    output = pipe(**get_args())
    gc()
    image = output.images[0]
    image.save('standard-default2.png')

    pipe = None
    gc()

[yiyixuxu]

@vladmandic

warnings

I removed the warnings during the generation (changed to logger.info), but kept the ones you get during the conversion for now. This is because currently the expected behavior for update_components is to update the underlying loading spec as you update the component itself, so that we should warn users if not the case

However, we are still iterating everything, so that may change in future too

image difference

The results from modular and standard pipelines are not going to be the same: Other than the guider, we are also taking this opportunity to refactor all our pipeline methods to make them more modular and easy to customize upon - this is an ongoing effort, we are likely to change more as we integrate more pipelines into modular diffusers

However, once modular is out of experimental feature, moving forward, we will make sure new pipelines will have same results in both systems

for now, I think if reproducibility is important for SD.NEXT users, I think we can only convert if they need to use a new guider class that's not supported in standard pipeline

output

I updated PipelineState so now you can access images or anything else else in intermediate states just by using that attribute; now below 3 methods to get the images outputs are all supported

image = t2i_pipe(prompt=prompt, num_inference_steps=25, output="images")[0]
image.save(f"{output_name}_1.png")


image = t2i_pipe(prompt=prompt, num_inference_steps=25).images[0]
image.save(f"{output_name}_2.png")

image = t2i_pipe(prompt=prompt, num_inference_steps=25).intermediates['images'][0]
image.save(f"{output_name}_3.png")

[vladmandic]

thanks @yiyixuxu

re: warnings - all good
re: output - perfect
re: image differences
the biggest reason to start using new modular pipelines is new guider concept, so i don't mind combining the two on my side
and reproducibility does not have to be exact using standard vs modular pipeline when using their respective defaults
just need to know how to map the params from standard to modular pipeline to achieve reproducibility