multi-gpu training

[lucyknada]

I have seen a few mentions of multi-gpu especially recently, and the readme even mentions it for sdxl:

sd-scripts/README.md

Line 255 in 0a52b83

- `--ddp_gradient_as_bucket_view` and `--ddp_bucket_view`options are added to `sdxl_train.py`. Please specify these options for multi-GPU training.

actually trying to run that with main or dev branch doesnt work, it just says there's no such arguments and dumps the help page.

I have also tried to configure accelerate for non-distributed gpu training but with multiple gpu ID's or "all" in the setup wizard (as mentioned in another issue), which also just OOM'd the main GPU and didn't offload anything at all to the second.

so what is the actual way to get multi gpu training working? thanks

[kohya-ss]

python sdxl_train.py --help in main branch shows this help message:

usage: sdxl_train.py [-h] [--v2] [--v_parameterization]
                     [--pretrained_model_name_or_path PRETRAINED_MODEL_NAME_OR_PATH]
...
                     [--full_fp16] [--full_bf16] [--ddp_timeout DDP_TIMEOUT] [--ddp_gradient_as_bucket_view]
                     [--ddp_static_graph] [--clip_skip CLIP_SKIP] [--logging_dir LOGGING_DIR]
...

Please verify and update your scripts.

[lucyknada]

ddp_bucket_view still does not exist, though trying to just run with ddp_gradient_as_bucket_view and configured accelerate config for GPU 0 and 1, still just OOMs my main GPU and then crashes

can you elaborate if this needs nvlink/SLI to function or duplicate GPUs? or can you mix e.g. 4090 with 3060

[kohya-ss]

Sorry, --ddp_bucket_view is mistaken. --ddp_gradient_as_bucket_view is correct. Please try this option. I will fix README sooner.

[Isotr0py]

I don't think we can mix 4090 with 3060 in sdxl_train.py DDP training, because DDP will keep a model copy in each GPU, while 3060 will become bottleneck due to its lower VRAM.