Memory issue when try to build customised model

[CWYuan08]

Hi I encountered this error when trying bonito basecaller to build our own model:

RuntimeError: CUDA out of memory. Tried to allocate 1.95 GiB (GPU 0; 15.74 GiB total capacity; 2.37 GiB already allocated; 1.70 GiB free; 2.42 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I don't really understand this message, should I add max_split_size_mb to the command? Thank your very much!

Best,
CW

[CWYuan08]

Hi I have changed export 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:100’ and export 'CUDA_LAUNCH_BLOCKING=1’, but I am still getting "RuntimeError: CUDA error: out of memory", do I need to change anything in the serialization.py file?

Thank you!

[CWYuan08]

The GPU we are trying to run it on is an Nvidia A4000 with 16GB of VRAM.

[CWYuan08]

This post: #247 solved my problem