Skip to content

Add community class StableDiffusionXL_T5Pipeline #11626

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

ppbrown
Copy link

@ppbrown ppbrown commented May 29, 2025

What does this PR do?

This adds a new community pipeline, named
StableDiffusionXL_T5Pipeline

It grafts the T5 xxl encoder onto SDXL, completely replacing TE1 and TE2.

It has been tested to produce output. However, the output isnt particularly valid.
The unet needs to be retrained.

Will initially be used with base model opendiffusionai/stablediffusionxl_t5
until a proper (re)fine-tune can be made.

Fixes # (issue)

Before submitting

Who can review?

Will be used with base model opendiffusionai/stablediffusionxl_t5
@ppbrown
Copy link
Author

ppbrown commented May 29, 2025

PS: This code was tested locally and ran, using the following test harness:

from diffusers import DiffusionPipeline
import torch.nn as nn, torch, types

SDXL_DIR = "/home/phil/git/models/t5-sdxl-model"

LOCAL_CODE = "/home/phil/git/diffusers.t5/examples/community/"

pipe = DiffusionPipeline.from_pretrained(
    SDXL_DIR, custom_pipeline=LOCAL_CODE, use_safetensors=True,
    torch_dtype=torch.bfloat16,
)

print("model initialized. Now moving to CUDA")
pipe.to("cuda")

print("Trying render now...")

images = pipe("a misty Tokyo alley at night",num_inference_steps=30).images
images[0].save("test.png")

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@asomoza
Copy link
Member

asomoza commented Jun 3, 2025

@bot /style

@asomoza
Copy link
Member

asomoza commented Jun 3, 2025

thanks @ppbrown , can you add some simple information to the README so people will know what this pipeline is, just a description and maybe a link to your experiment?

@asomoza
Copy link
Member

asomoza commented Jun 3, 2025

also the style bot didn't work, can you run make style and make quality so the test passes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants