You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 23.69 GiB total capacity; 22.48 GiB already allocated; 6.06 MiB free; 22.55 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Issue 1:
运行上述命令后会在训练阶段OOM:
使用下述命令训练GLM顺利进入训练阶段,尚未发生OOM:
训练时占用:
Issue 2:
根据Readme所述,训练GLM时不能使用int8量化,但是finetune代码中没有判断后跳过此类的处理,会导致OOM:
可以注释掉这行,注释后不会在这OOM
The text was updated successfully, but these errors were encountered: