Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How much VRAM is needed for this? #17

Open
mjohanning99 opened this issue Jan 12, 2022 · 2 comments
Open

How much VRAM is needed for this? #17

mjohanning99 opened this issue Jan 12, 2022 · 2 comments

Comments

@mjohanning99
Copy link

I was trying to run the sampling_ex.py, but no matter how low I set the num_candidates value (even if it's set to one or two), it always tells me that it has run out of memory. I am using an NVIDIA Quadro M5000 with 8 GB of VRAM.

@chiheonk
Copy link

In case of the released 1.3B model, total >5GB memory (1.3B * 4 bytes/per parameter) is required for placing the model on the GPU. Hence, I suspect that the model itself is already consuming the most of your GPU's VRAM (8GB), and there is not enough room to conduct an inference even with batch size 1.

One possible solution could be to cast your model to half-precision (float16) as

model.to(device=device, dtype=torch.float16)

in place of https://github.com/kakaobrain/minDALL-E/blob/0262c0281f9b34d43a751af6d593312416c85446/examples/sampling_ex.py#L36

Let me know if this works!

@mjohanning99
Copy link
Author

Hello and thank you for your response! I have added the line you suggested, but I got the following error when running the sample script:

/home/mjo/.cache/minDALL-E/1.3B/tokenizer successfully restored..
/home/mjo/.cache/minDALL-E/1.3B/stage1_last.ckpt successfully restored..
/home/mjo/.cache/minDALL-E/1.3B/stage2_last.ckpt succesfully restored..
Traceback (most recent call last):
  File "/home/mjo/AI_Stuff/mindDALL-E/minDALL-E/sampling_ex.py", line 36, in <module>
    model.to(device=device, dtype=torch.float16)
NameError: name 'torch' is not defined

I had to manually import the torch module at the top of the file for it to run the program; the resulting images, however, are now all black. The image below was creating using the standard prompt in the sampling_ex.py script with a num_candidate value of 15.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants