Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to config to train on NVIDIA GeForce RTX 3090 Ti/PCIe/SSE2 #29

Open
vtmjapandev opened this issue Mar 10, 2022 · 6 comments
Open

How to config to train on NVIDIA GeForce RTX 3090 Ti/PCIe/SSE2 #29

vtmjapandev opened this issue Mar 10, 2022 · 6 comments

Comments

@vtmjapandev
Copy link

Hi guys,

I want to train this model on(NVIDIA GeForce RTX 2080 Ti/PCIe/SSE2), RAM 16GB, How to config it.
When I ran the default setting, the bellow error has occored.

    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
@taohan10200
Copy link
Owner

see #28 (comment)

@vtmjapandev
Copy link
Author

@taohan10200
I have set CUDA_LAUNCH_BLOCKING=1 but the same error has occurred.

@vtmjapandev
Copy link
Author

@taohan10200
In the dataset setting file with NWPU, I have set the batch size to 1. But this batch size is good?
__C_NWPU.TRAIN_BATCH_SIZE = 1 #imgs

@vtmjapandev vtmjapandev changed the title How to config to train on NVIDIA GeForce RTX 2080 Ti/PCIe/SSE2 How to config to train on NVIDIA GeForce RTX 3090 Ti/PCIe/SSE2 Apr 12, 2022
@vtmjapandev
Copy link
Author

@taohan10200
I have set batch_size from 12 -> 6. But when traint, bellow error has occurred.

python3.7/site-packages/torch/nn/functional.py:3704: UserWarning: nn.functional.upsample is deprecated. Use
nn.functional.interpolate instead.
 warnings.warn("nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.")
./run_train.sh: line 6: 75217 Segmentation fault      (core dumped) python3.7 train.py

@taohan10200
Copy link
Owner

Based on my experience, this error occurs when some packages you pip install are not the right version. Please check each required package with the import xxx command to locate the error package.

@vtmjapandev
Copy link
Author

vtmjapandev commented Apr 15, 2022

@taohan10200
You can share with me your requrements.txt file.

The current I ran the project on
Ubuntu 20.04
Python3.7
GTX 3090

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants