8gb Flux LoRa training - is it possible?

[Signorlimone]

as per title, is it possible to squeeze a training to fit 8gb vram?

also

what are the option to use less vram? (apart from lowering network dim and using 512 images)

[Signorlimone]

ok, so, atm this configuration works with my nvidia 2700 max-q 8gb gpu. This is bf16 mixed precision and is EXTREMELY (40s/it) slow. Sadly with fp16 mixed precision it gave a avr_loss=nan or else it would be much faster. I'll keep experimenting

8gb gpu LoRa Training.json

[remixer-dec]

There is a 4-bit model that takes ~8.5GB and its author used it for fine tuning, but it is probably not supported out of the box in kohya

[Signorlimone]

hey, thanks, I'll look into it

[Seedmanc]

I can't even have a proper sdxl training on 8gb, what do you think.
Unless you think training at 768x and only unet or TE at a time is proper.

[Signorlimone]

Flux is different, with quantized models and such it is possible to train a LoRa with 8gb. I trained some SDXL models with unet only for merging purpose and they were really useful, depends on the case. Btw, "what do you think" doesn't bring anything to the discussion, next time refrain to answer if this is your contribution. Also, a question needs a "?" at the end of the sentence. Learn how to spell properly