8gb SDXL Recommended Settings

[JJ-IsShort]

So I've been trying to train an SDXL LORA on my 3050 8GB, and I've been struggling. My training command currently is

accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train_network.py" \
--enable_bucket \
--pretrained_model_name_or_path="<path to sd_xl_base_0.9.safetensors>" \
--train_data_dir="<dataset folder>\img" \
--resolution="1024,1024" \
--output_dir="<dataset folder>\model" \
--logging_dir="<dataset folder>\log" \
--network_alpha="0.5" \
--save_model_as=safetensors \
--network_module=networks.lora \
--unet_lr=0.0001 \
--network_train_unet_only \
--network_dim=8 \
--output_name="last" \
--lr_scheduler_num_cycles="10" \
--cache_text_encoder_outputs \
--learning_rate="0.0001" \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps="260" \
--train_batch_size="1" \
--max_train_steps="2600" \
--save_every_n_epochs="1" \
--mixed_precision="fp16" \
--save_precision="fp16" \
--cache_latents \
--cache_latents_to_disk \
--optimizer_type="AdamW" \
--max_data_loader_n_workers="0" \
--max_token_length=150 \
--vae_batch_size="2" \
--bucket_reso_steps=32 \
--min_snr_gamma=5 \
--gradient_checkpointing \
--full_fp16 \
--xformers \
--bucket_no_upscale \
--vae="<path to sdxl_vae.safetensors>"

I've heard that it is possible, and I've even tried to model my settings after theirs. This has helped. Now I get OOM errors when building the optimiser. I was hoping we could all share our solutions for training SDXL on 8gb.

[kohya-ss]

Please use adamw8bit instead of adamw.

In addition, full_fp16 training seems to be unstable. Please try without the option at first.

[efoxxfiles]

work?

[efoxxfiles]

RuntimeError: NaN detected in latents: C:\IA\XL\kohya_ss\train\img\10_fxxfls man\3f0xxf1l35-(01)-0000-0000.png

[kohya-ss]

Please add --no_half_vae option (or switch to bf16) to avoid NaNs for caching latents.

[JJ-IsShort]

I use this VAE.

[efoxxfiles]

Finish but fail in SD

C:\Users\fox\miniconda3\lib\site-packages\torch\utils\checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
steps: 80%|█████████████████████████████████████████████████▌ | 400/500 [2:20:41<35:10, 21.10s/it, loss=nan]
saving checkpoint: C:/IA/XL/kohya_ss/train/model\fxxfls-000001.safetensors
epoch 2/2
steps: 100%|██████████████████████████████████████████████████████████████| 500/500 [2:51:27<00:00, 20.58s/it, loss=nan]
saving checkpoint: C:/IA/XL/kohya_ss/train/model\fxxfls.safetensors
model saved.
steps: 100%|██████████████████████████████████████████████████████████████| 500/500 [2:51:28<00:00, 20.58s/it, loss=nan]
(base) PS C:\IA\XL\kohya_ss>

AUTOMATIC1111

    return forward_call(*args, **kwargs)
  File "C:\IA\XL\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 215, in forward
    devices.test_for_nans(x_out, "unet")
  File "C:\IA\XL\stable-diffusion-webui\modules\devices.py", line 155, in test_for_nans
    raise NansException(message)
modules.devices.NansException: A tensor with all NaNs was produced in Unet. Use --disable-nan-check commandline argument to disable this check.

--no-half --medvram

y try with --disable-nan-check have Black pic

[efoxxfiles]

Finale can work with this but start to generate an preview and fail with Black Picture

accelerate launch --num_cpu_threads_per_process 8 ./sdxl_train_network.py --enable_bucket --pretrained_model_name_or_path="C:/IA/XL/stable-diffusion-webui/models/Stable-diffusion/sd_xl_base_1.0.safetensors" --train_data_dir="C:/IA/XL/kohya_ss/train/img" --resolution="1024, 1024" --output_dir="C:/IA/XL/kohya_ss/train/model" --logging_dir="C:/IA/XL/kohya_ss/train/log" --network_alpha="0.5" --save_model_as=safetensors --network_module=networks.lora --unet_lr=0.0005 --network_train_unet_only --network_dim=8 --output_name="fxxfls" --lr_scheduler_num_cycles="10" --cache_text_encoder_outputs --learning_rate="0.0005" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="1600" --save_every_n_epochs="1" --mixed_precision="fp16" --save_precision="fp16" --optimizer_type="AdamW" --max_data_loader_n_workers="0" --max_token_length=150 --vae_batch_size="2" --min_snr_gamma=5 --gradient_checkpointing --xformers --bucket_no_upscale --vae="C:\IA\XL\stable-diffusion-webui\models\VAE\sdxl_vae.safetensors" --cache_latents --cache_latents_to_disk --gradient_checkpointing

some idea?

[Vinylspider]

I'm running to completion with the SDXL branch of Kohya on an RTX3080 in Win10, but getting no apparent movement in the loss. Despite this the end results don't seem terrible. This makes me wonder if the reporting of loss to the console is not accurate. Very slow training.

My Train_network_config.toml is set to:
pretrained_model_name_or_path = "C:\Users\user_me\stable-diffusion-webui\models\Stable-diffusion\sd_xl_base_1.0.safetensors"
dataset_config = "C:\Users\user_me\Kohya\sd-scripts\dataset_config.toml"
sample_prompts = "C:\Users\user_me\Kohya\sd-scripts\prompt.txt"
output_dir = "C:\Users\user_me\Kohya\Output_Models\model\"
logging_dir = "C:\Users\user_me\Kohya\Output_Logs"
output_name = "LoraSDXL15_v1.0"
train_batch_size = 1
network_dim = 36
network_alpha = 18
max_train_steps = 7000
save_every_n_steps = 1000
sample_every_n_steps = 1000
optimizer_type = "adafactor"
optimizer_args = [ "scale_parameter=False", "relative_step=False", "warmup_init=False" ]
lr_scheduler_num_cycles = 20
lr_scheduler = "constant_with_warmup"
lr_warmup_steps = 100
network_train_unet_only = true
learning_rate = 0.0005
unet_lr = 0.0005
text_encoder_lr = 0.0001
caption_extension = ".txt"
resolution = "1024,1024"
save_precision = "bf16"
mixed_precision = "bf16"
sample_sampler = "euler_a"
network_module = "networks.lora"
max_token_length = 225
bucket_reso_steps = 32
cache_latents = true
enable_bucket = true
bucket_no_upscale = true
xformers = false
persistent_data_loader_workers = true
gradient_checkpointing = true
cache_text_encoder_outputs = true
no_half_vae = true
noise_offset = 0.0357
adaptive_noise_scale = 0.00357
cache_latents_to_disk = true

My dataset_config.toml is set to:

[[datasets]]
resolution = [1024, 1024]

[[datasets.subsets]]
image_dir = 'Z:\Downloads\Training\Clothing\LORA\Images\50_Clothing Outfit'
caption_extension = '.txt'
num_repeats = 50