Patch Release: Support for SDXL Kohya-style LoRAs, Fix batched inference SDXL Img2Img, Improve watermarker

We still had some bugs 🐛 in 0.19.1 some bugs, notably:

SDXL (Kohya-style) LoRA

The official SD-XL 1.0 LoRA (Kohya-styled) is now supported thanks to #4287. You can try it as follows:

from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16)
pipe.load_lora_weights("stabilityai/stable-diffusion-xl-base-1.0", weight_name="sd_xl_offset_example-lora_1.0.safetensors")
pipe.to("cuda")

prompt = "beautiful scenery nature glass bottle landscape, purple galaxy bottle"
negative_prompt = "text, watermark"

image = pipe(prompt, negative_prompt=negative_prompt, num_inference_steps=25).images[0]

In addition, a couple more SDXL LoRAs are now supported:

(SDXL 0.9:)

• https://civitai.com/models/22279?modelVersionId=118556
• https://civitai.com/models/104515/sdxlor30costumesrevue-starlight-saijoclaudine-lora
• https://civitai.com/models/108448/daiton-sdxl-test
• https://filebin.net/2ntfqqnapiu9q3zx/pixelbuildings128-v1.safetensors

To know more details and the known limitations, please check out the documentation.

Thanks to @isidentical for their sincere help in the PR.

Batched inference

@bghira found that for SDXL Img2Img batched inference led to weird artifacts. That is fixed in: #4327.

Downloads

Under some circumstances SD-XL 1.0 can download ONNX weights which is corrected in #4338.

Improved SDXL behavior

#4346 allows the user to disable the watermarker under certain circumstances to improve the usability of SDXL.

All commits:

• [SDXL Refiner] Fix refiner forward pass for batched input by @patrickvonplaten in #4327
• [ONNX] Don't download ONNX model by default by @patrickvonplaten in #4338
• [SDXL] Make watermarker optional under certain circumstances to improve usability of SDXL 1.0 by @patrickvonplaten in #4346
• [Feat] Support SDXL Kohya-style LoRA by @sayakpaul in #4287