FLUX LoRA Support

[RyanJDick]

Summary

This PR adds support for FLUX LoRA models on both quantized and non-quantized base models.

Supported formats:

• diffusers
• kohya

Full changelist:

• Consolidated LoRA handling code in invokeai/backend/lora
• Add support for FLUX kohya and FLUX diffusers LoRA model loading
• Add ability to either patch LoRAs or run as a sidecar model (the latter enables LoRAs to be applied to a wide range of quantized models).

QA Instructions

Note to reviewers: I tested everything in this checklist. Feel free to re-verify any of this, but also test any LoRAs that you have. There are many small LoRA format variations, and there's a risk of breaking one of them with this change.

FLUX LoRA

• Import / probe of kohya FLUX LoRA (https://civitai.com/models/159333/pokemon-trainer-sprite-pixelart?modelVersionId=779247)
• Import / probe of Diffusers FLUX LoRA (https://civitai.com/models/200255/hands-xl-sd-15-flux1-dev?modelVersionId=781855)
• kohya with non-quantized base model
• kohya with quantized base model (should roughly match the non-quantized case)
• diffusers with non-quantized base model
• diffusers with quantized base model (should roughly match the non-quantized case)
• Sidecar LoRA patching speed (<0.1secs after model is loaded)
• Stacking multiple fused LoRA models (i.e. on top on non-quantized model)
• Stacking multiple sidecar LoRA models (i.e. on top of quantized model)

Regression Tests

• SD1.5 LoRA (check output, speed and memory)
• SDXL LoRA (check output, speed and memory)
• USE_MODULAR_DENOISE=1 smoke test with LoRA

Test for output regression with the following LoRA formats:

• LoRA
• LoHA
• LoKr
• IA3

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)

[RyanJDick]

Note to reviewers: Please test any LoRAs that you have (SD or FLUX). There are many small LoRA format variations, and there's a risk of breaking one of them with this change.

[bghira]

i don't see how alpha=8 would work for any peft loras that aren't also rank=8

[bghira]

if you're going for lycoris support note that the support for this in peft is very lacking currently, with an open issue for community help request on updating the various modules

it's very easy to wrap the upstream lycoris library modules though, as @KohakuBlueleaf has added a functional API for this use case. it just wraps the Diffusers model and has .apply_to() and .merge_to() for sidecar or fused running

there's built in methods for extracting models and playing with them in other ways. i'd highly recommend switching from peft for lycoris stuff.

[KohakuBlueleaf]

it's very easy to wrap the upstream lycoris library modules though, as @KohakuBlueleaf has added a functional API for this use case. it just wraps the Diffusers model and has .apply_to() and .merge_to() for sidecar or fused running

.apply_to() is module API. Functional API is something designed for library to implement their own low-level wrapper.
(For example, lokr_output = layer(x) + lokr_bypass_forward(x, w1, w2a, w2b ...))

[RyanJDick]

if you're going for lycoris support note that the support for this in peft is very lacking currently, with an open issue for community help request on updating the various modules

it's very easy to wrap the upstream lycoris library modules though, as @KohakuBlueleaf has added a functional API for this use case. it just wraps the Diffusers model and has .apply_to() and .merge_to() for sidecar or fused running

there's built in methods for extracting models and playing with them in other ways. i'd highly recommend switching from peft for lycoris stuff.

Thanks for this context. We explored using PEFT at one point, but decided against using it because it didn't meet our requirements for high patching/unpatching speeds.

I'll leave the integration of the upstream lycoris lib for future work. For now, we have our own implementation of the lycoris layers for fused execution, and partial support for sidecar execution (and easy to extend).

[bghira]

i agree that the patching speed of diffusers could be improved. if you have any observations/suggestions please open an issue report