-
Notifications
You must be signed in to change notification settings - Fork 5.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add PAG support sd15 controlnet #8820
add PAG support sd15 controlnet #8820
Conversation
Hi @a-r-r-o-w I open new PR because I cannot undo deleted pipelines. I have edited it according to your comment, please review again, thank you |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking great. Just small changes remaining and this should be good for merge.
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Thank you for addressing the comments! Could we fix the failing tests? Error logs
Let me know if you need any help. We can merge as soon as these are fixed since everything else looks good to me! |
Sure can you help me on tests 🤗 |
For the first two test failures, you will have to update the expected values to what is the actual value of For the third test, you need to remove |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks!
# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.decode_latents | ||
def decode_latents(self, latents): | ||
deprecation_message = "The decode_latents method is deprecated and will be removed in 1.0.0. Please use VaeImageProcessor.postprocess(...) instead" | ||
deprecate("decode_latents", "1.0.0", deprecation_message, standard_warn=False) | ||
|
||
latents = 1 / self.vae.config.scaling_factor * latents | ||
image = self.vae.decode(latents, return_dict=False)[0] | ||
image = (image / 2 + 0.5).clamp(0, 1) | ||
# we always cast to float32 as this does not cause significant overhead and is compatible with bfloat16 | ||
image = image.cpu().permute(0, 2, 3, 1).float().numpy() | ||
return image |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.decode_latents | |
def decode_latents(self, latents): | |
deprecation_message = "The decode_latents method is deprecated and will be removed in 1.0.0. Please use VaeImageProcessor.postprocess(...) instead" | |
deprecate("decode_latents", "1.0.0", deprecation_message, standard_warn=False) | |
latents = 1 / self.vae.config.scaling_factor * latents | |
image = self.vae.decode(latents, return_dict=False)[0] | |
image = (image / 2 + 0.5).clamp(0, 1) | |
# we always cast to float32 as this does not cause significant overhead and is compatible with bfloat16 | |
image = image.cpu().permute(0, 2, 3, 1).float().numpy() | |
return image |
I don't think it is used
extra_step_kwargs["generator"] = generator | ||
return extra_step_kwargs | ||
|
||
def check_inputs( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we add a #Copied from
here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we have a version of check_inputs with callback_steps
removed, and this is the first of its kind
feel free to merge once you're happy with it! @a-r-r-o-w |
@a-r-r-o-w Can you help me with last fail of test 🤗 |
@tuanh123789 The failing test is unrelated and nothing needs to be done on your end. I'm happy to merge this once you address YiYi's comment: #8820 (comment) |
:v Sure, will be done right away |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, looking good! One other thing that I just realised - could you update the pipelines doc page with this SD15 Controlnet? https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/pag.md. Please see #7944 for an example
Done |
pip install git+https://github.com/huggingface/diffusers ImportError: cannot import name 'StableDiffusionControlNetPAGPipeline' from 'diffusers' pip list diffusers 0.30.0.dev0 how can i test? |
how do i know if its working fix seed and test?
|
Hi @tuanh123789, could you please share the colab notebook you used to test the images which you have shown in the above output. Thanks |
yes, preferably with a complex generation that you can see pag helps to fix.
It's literally copy & paste the example from this PR, there's nothing else you have to do.
You could just paste everything in one cell block, but here's notebook with the steps to test it. |
* add pag support sd15 controlnet * fix quality import * remove unecessary import * remove if state * fix tests * remove useless function * add sd1.5 controlnet pag docs --------- Co-authored-by: anhnct8 <[email protected]>
What does this PR do?
Adds PAG (Perturbed-Attention Guidance) support for SD 1.5 with controlnet models (StableDiffusionControlNetPAGPipeline)
Continuation of #8710
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@yiyixuxu
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.