About the GPU memory usage

[prokaryote-hgy]

Hi, wonderful work!

I wonder what's the peak memory usage when training the model with HR size of 512 and batchsize of 1 for one GPU. It seams that a 24GB GPU not enough to run the training? Or I did something wrong?

[Renzhihan]

I can train the model on 1 3090 with bs=1. But when I use 2 or more GPUs, the memory is not enough.

[prokaryote-hgy]

I can train the model on 1 3090 with bs=1. But when I use 2 or more GPUs, the memory is not enough.

The same question.

[liyuantsao]

I can train the model on 1 3090 with bs=1. But when I use 2 or more GPUs, the memory is not enough.

+1

[cswry]

I can train the model on 1 3090 with bs=1. But when I use 2 or more GPUs, the memory is not enough.

Multi-GPU parallel training consumes additional memory compared to single-GPU training, which is likely the cause.

[Moonsofang]

请问bs参数是在哪里设置啊？