CUDA Memory is not enough

[zhongruizhe123]

Ask the Critic training how much video memory is required at least, I used two 3090ti 24G, but to tip that video memory is not enough. Is there a way to adjust some parameters to make my program work

[fate-ubw]

24G of RAM can't fully weight finetune 7B models, which usually requires 4*80G GPUs. If you only have two 3090ti 24G I recommend you to use LoRA or QLORA, but this will lose a little bit of performance, if you don't know how to configure LORA and QLORA you can refer to the LoRA finetune script I configured https://github.com/fate-ubw/RAGLAB/blob/main/run/rag_train/script_finetune-llama3-8B-baseline-Lora.sh All you need to do is switch the training data and output model name

[fmk345]

While using the retrieval_lm/finetune.py script for full parameter fine-tuning, I noticed that the training seems to adopt a data-parallel strategy. However, when working with models larger than 7B on a setup with 4 A100 GPUs (80GB each), we encountered out-of-memory (OOM) issues. I wanted to kindly inquire if you have any guidance or code adaptations for supporting full fine-tuning of models larger than 7B in such setups. If you could provide additional code or suggestions to address this challenge, it would be of tremendous help to our research. We deeply respect your contributions to the field and would be delighted to reference your work and paper in our upcoming research based on this fine-tuning.