Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

跑到50轮,会爆显存有什么解决办法吗 #33

Open
yxddzho opened this issue May 24, 2024 · 1 comment
Open

跑到50轮,会爆显存有什么解决办法吗 #33

yxddzho opened this issue May 24, 2024 · 1 comment

Comments

@yxddzho
Copy link

yxddzho commented May 24, 2024

CUDA out of memory. Tried to allocate 320.00 MiB. GPU 0 has a total capacity of 8.00 GiB of which 0 bytes is free. Of the allocated memory 6.84 GiB is allocated by PyTorch, and 157.12 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

@Jalnf013516
Copy link

如果你没关Freeze_Train,那前50epoch是冻结训练,之后是UnFreeze_Train,它的显存开销很大,建议调低input_size。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants