Sudden cuda OOM issue #129
Comments
|
Pkg versions Driver version: |
|
우선 wandb를 써서 https://wandb.ai/wandb/common-ml-errors/reports/How-To-Use-GPU-with-PyTorch---VmlldzozMzAxMDk |
|
I want to test same code at my local workstation for check GPU usage Could U share code snippet or short code for run similar situation? @wheresmadog |
|
As for reproduction, you can run: However, I'm skeptical the same error will occur in different machines if usual, because the code snippet actually uses nothing but official release of HF codes. |
Despite |
|
음 그냥 이렇게 시간에 따라서 memory가 어떻게 사용되는지 체크할 수 있는데
|
이걸
이렇게 해도 에러가 날까요? |
First of all, thanks for your thoughtful answer. The scenario of the reference issue is when a model keeps running over and over for hyperparameter tuning. Which implies that it can at least successfully iterate the model a couple of times. Meanwhile, I have a problem that not even one instance is allowed to make it through. Once again, I make clear that OOM occurs in the middle of a certain layer rather than after a few iterations of entire layers of a model.
Yeap, still not working. But I found from which layer the error occurs where a GPU memory allocator suddenly strikes out. |
|
If anyone tries the code snippet and is unable to reproduce the problem I have, then it looks likely to be a H/W issue rather than a code error. (I hope my GPU is not melted down or something.) |
|
I'd like to help, but I'm sorry to say this :( I want to run the code-snippet, but I can't run the code properly because I repeatedly get errors like P.S. I also want to ask if you have experienced this import error. |
|
우선 저는 문제 없이 진행되는 것 같습니다! gpu 메모리 정보도 같이 올려보도록 하겠습니다.
|
Well, that's not the case I've faced before. Did you install from the requirement uploaded in the comment? |
|
Thanks for your confirmation. This advocates that potential suspect is H/W. |
|
As for your curiosity, I've figured out from which part of the model such memory usage spikes up. It turns out attention block tries to allocate such a massive amount of memory. 
You can see this block is not a huge one, and almost any GPU can run on it - again, batch size is not an issue as it's set to 1. |
|
위 snippet을 구글 colab에 돌렸을 때 사용 g메모리 3기가 정도라서... 재현이 안 되네요ㄷㄷ |
LOL, now that it requires over 800GiB. Am I dealing with some sort of LLM? |
|
maybe related. torch 버전 업데이트 해보심이? huggingface/diffusers#4159 |
It turns out compatibility issue. Having updated torch 2.0 resolved the issue. Thank you. P.S. But why did it work before? |
|
Context:
UNet2DModelclass from HuggingFace, see here.Things I've tried:
Can you solve it?
Detailed stats for geeks:
Summary of CUDA memory
Memory stat seems pretty stable, isn't it?
Then all of a sudden, at the fourth iteration of down sampling blocks, the model tries to allocate 56GiB to GPU.
The OOM count over the summary concerns this very issue as this summary is written at the debugging flag.
And there had been no code modification.
torch.no_grad()flag has been up. That is, no further autograd graph has been created to cause this ridiculous memory allocation.The text was updated successfully, but these errors were encountered: