Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KolorsPipeline does not support from_single_file #10207

Open
Thekey756 opened this issue Dec 13, 2024 · 2 comments · May be fixed by #10215
Open

KolorsPipeline does not support from_single_file #10207

Thekey756 opened this issue Dec 13, 2024 · 2 comments · May be fixed by #10215

Comments

@Thekey756
Copy link

from diffusers import KolorsPipeline
KolorsPipeline.from_single_file("models/kolrs-8steps.safetensors")

How does KolorsPipeline load a single file model?

@hlky
Copy link
Collaborator

hlky commented Dec 13, 2024

Can you link the model you are trying to load, seems it would be this?

@hlky
Copy link
Collaborator

hlky commented Dec 13, 2024

That model is already in Diffusers format, it's just the UNet2DConditionModel, we can load it straight to pipe.unet. Alternatively a version with this UNet2DConditionModel could be uploaded to the Hub then it could be used directly with KolorsPipeline.from_pretrained.

import torch
from diffusers import KolorsPipeline, EulerAncestralDiscreteScheduler
import safetensors.torch

weights = safetensors.torch.load_file("kolors8AcceleratedVersionOf8_accelerated.safetensors")

pipe = KolorsPipeline.from_pretrained(
    "Kwai-Kolors/Kolors-diffusers", variant="fp16", torch_dtype=torch.float16
)
pipe.unet.load_state_dict(weights)
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")
prompt = (
    "A photo of a ladybug, macro, zoom, high quality, film, holding a wooden sign with the text 'KOLORS'"
)
image = pipe(prompt, num_inference_steps=10, height=720, width=1280, guidance_scale=1.0, generator=torch.Generator().manual_seed(0)).images[0]
image.save("kolors.png")

Parameters num_inference_steps=10, guidance_scale=1.0, resolution and EulerAncestral seem to be important, I get bad output with the defaults.

Common parameters:

CFG 1 / Steps 10 / Sampler Euler a / VAE sdxl-vae-fp16-fix.safetensors

Size: 1280x720 / 1536x768 / 960x1920 / 1920x960 / 2048x1024 / 1024x2048

kolors

cc @a-r-r-o-w @DN6 from_single_file won't work here, looks like we should check for some known Diffusers format key and return if it's already converted.

def convert_ldm_unet_checkpoint(checkpoint, config, extract_ema=False, **kwargs):

Failed to load UNet2DConditionModel. Weights for this component appear to be missing in the checkpoint.

edit: Just returning early breaks detection (e.g. XL and this Kolors unet are detected as v1), we also need to update CHECKPOINT_KEY_NAMES with Diffusers versions, we already do this for controlnet and flux

@hlky hlky linked a pull request Dec 13, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants