RTX 4090 finetune with CUDA_OUT_OF_memory

[clpoz]

As the title, I think it should be fine with 24GB cuda_memory, when I start finetuing, it shows only take 18GB, while 3 hours later, I found it show errors with cuda_out_of_memory. Anyone have ideas? I don't know why?

[clpoz]

Ok, I found it, after did a validation and saved a checkpoint, the VRAM usage added about 3GB and never drop back, how could it happen? how could I change setting to solve it？

[little51]

try reduce block_size or per_device_train_batch_size per_device_eval_batch_size