CUDA out of memory

[Vi-cs]

Hi,

I regularly get this error when running the app, after a couple (4/5) inferences via the GUI:

CUDA out of memory. Tried to allocate 502.00 MiB (GPU 0; 22.19 GiB total capacity; 19.98 GiB already allocated; 194.50 MiB free; 20.10 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

My current workaround is to kill the process to free some memory.

Did I miss a configuration step or something that prevent out of memory?

[zetavg]

Hi! You can try to use the --load_8bit flag when running python app.py, which will enable load_in_8bit for models and save some VRAM.

[Vi-cs]

thanks!! Le jeu. 18 mai 2023, 18:34, Pokai Chang ***@***.***> a écrit :

…

Hi! You can try to use the --load_8bit flag when running python app.py, which will enable load_in_8bit for models and save some VRAM. — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOCBXOX5LZE46MQ72WPUR23XGZFRPANCNFSM6AAAAAAX7B6RQE> . You are receiving this because you authored the thread.Message ID: ***@***.***>