-
Notifications
You must be signed in to change notification settings - Fork 5.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
adds the pipeline for pixart alpha controlnet #8857
base: main
Are you sure you want to change the base?
adds the pipeline for pixart alpha controlnet #8857
Conversation
is this the checkpoint? https://huggingface.co/PixArt-alpha/PixArt-ControlNet is this pixart alpha controlnet used a lot in the community? if not, maybe we can make a community pipeline to start with? also cc @asomoza |
@yiyixuxu that is the pixart controlnet model for HED conditioning as uploaded by the authors of pixart. they can be used with this pipeline |
why does it have its own implementation of the HED detector? It doesn't work with the regular one that everyone uses? Have you tested it with the one from the |
the sample above just used the HED class that the authors had in their repository, and that was used to train their HED controlnet. but i just checked it it seems to be the same, or better said adapted, from the controlnet_aux |
thanks, I'll give it a test later. I was asking because if it was trained with a custom HED detector which produces different results than the default one it will be really hard for people to use it. It would be nice if you could post some results (images) in the PR description. |
using the HED from control_aux it looses some quality. but i also have a training script that i am testing before creating a PR: that can be used to train further models.
will do. |
i have to correct my previous comment. i was using the default params for HED, which converted the image to 512, if i use however 1024 it works as it should. |
Thanks, the results looks nice, since we only have one controlnet, maybe do what @yiyixuxu suggested, lets start with a community pipeline first and then as it gets traction and we have more controlnets move it to core. |
ok, i will move it to the examples folder and put there the training script as well. |
@@ -0,0 +1,292 @@ | |||
from typing import Any, Dict, Optional |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think maybe pipelines can go to the /example/community folder, the training script can stay in example/pixart folder
cc @sayakpaul
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay with that plan.
@yiyixuxu |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left some comments, thanks!
from diffusers.models.modeling_utils import ModelMixin | ||
from diffusers.models.modeling_outputs import Transformer2DModelOutput | ||
|
||
class PixArtControlNetAdapterBlock(nn.Module): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need to copy paste all the model code here into the pipeline so that the pipeline will be able to run, no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the pipeline code changes the sys path, so it runs
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
@@ -0,0 +1,81 @@ | |||
import sys |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should go into README https://github.com/huggingface/diffusers/blob/main/examples/community/README.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i have added the section
this PR adds the controlnet pipeline for the pixart alpha diffusion model
the following example uses the HED edge to control the generation.
here some images: original image, control image and generated image
Who can review?
@yiyixuxu @lawrence-cj