RuntimeError: CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 6.00 GiB total capacity; 177.88 MiB already allocated; 0 bytes free; 4.71 GiB reserved in total by PyTorch)

[haomayang1126]

输入命令刚开始执行，显卡的内存就被占满了，有时候在第一轮出错，有时候在第三轮，
num_works 设置了4，2，0都是同样的问题，
环境
python 3.8
pytorch1.8.1
cuda10.1

==============================================================================
2021-07-24 17:55:22 | INFO | yolox.core.trainer:188 - ---> start train epoch1
2021-07-24 17:55:26 | INFO | apex.amp.handle:138 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 32768.0
2021-07-24 17:55:28 | INFO | apex.amp.handle:138 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0
2021-07-24 17:55:30 | INFO | apex.amp.handle:138 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 8192.0
2021-07-24 17:55:37 | INFO | yolox.core.trainer:237 - epoch: 1/30, iter: 10/40, mem: 4660Mb, iter_time: 1.570s, data_time: 0.867s, total_loss: 11.0, iou_loss: 3.0, l1_loss: 0.0, conf_
loss: 5.7, cls_loss: 2.3, lr: 1.953e-06, size: 640, ETA: 0:31:08
2021-07-24 17:55:43 | INFO | apex.amp.handle:138 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 4096.0
2021-07-24 17:55:51 | INFO | yolox.core.trainer:237 - epoch: 1/30, iter: 20/40, mem: 4660Mb, iter_time: 1.333s, data_time: 0.762s, total_loss: 10.1, iou_loss: 2.8, l1_loss: 0.0, conf_
loss: 4.5, cls_loss: 2.8, lr: 7.813e-06, size: 576, ETA: 0:28:32
2021-07-24 17:55:53 | INFO | yolox.core.trainer:183 - Training of experiment is done and the best AP is 0.00
2021-07-24 17:55:53 | ERROR | yolox.core.launch:73 - An error has been caught in function 'launch', process 'MainProcess' (5488), thread 'MainThread' (6852):
Traceback (most recent call last):

File "tools\train.py", line 111, in
launch(
└ <function launch at 0x00000126EC829E50>

File "g:\pythonproject\yolox-main\yolox\core\launch.py", line 73, in launch
main_func(*args)
│ └ (╒══════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════...
└ <function main at 0x00000126EEA76DC0>

File "tools\train.py", line 101, in main
trainer.train()
│ └ <function Trainer.train at 0x00000126EDDBCD30>
└ <yolox.core.trainer.Trainer object at 0x00000126EEAFA970>

File "g:\pythonproject\yolox-main\yolox\core\trainer.py", line 70, in train
self.train_in_epoch()
│ └ <function Trainer.train_in_epoch at 0x00000126EEA44F70>
└ <yolox.core.trainer.Trainer object at 0x00000126EEAFA970>

File "g:\pythonproject\yolox-main\yolox\core\trainer.py", line 79, in train_in_epoch
self.train_in_iter()
│ └ <function Trainer.train_in_iter at 0x00000126EEA55280>
└ <yolox.core.trainer.Trainer object at 0x00000126EEAFA970>

File "g:\pythonproject\yolox-main\yolox\core\trainer.py", line 85, in train_in_iter
self.train_one_iter()
│ └ <function Trainer.train_one_iter at 0x00000126EEA55310>
└ <yolox.core.trainer.Trainer object at 0x00000126EEAFA970>

File "g:\pythonproject\yolox-main\yolox\core\trainer.py", line 91, in train_one_iter
inps, targets = self.prefetcher.next()
│ │ └ <function DataPrefetcher.next at 0x00000126EDDBC310>
│ └ <yolox.data.data_prefetcher.DataPrefetcher object at 0x00000126F8F7D0D0>
└ <yolox.core.trainer.Trainer object at 0x00000126EEAFA970>

File "g:\pythonproject\yolox-main\yolox\data\data_prefetcher.py", line 48, in next
self.preload()
│ └ <function DataPrefetcher.preload at 0x00000126EDDBC280>
└ <yolox.data.data_prefetcher.DataPrefetcher object at 0x00000126F8F7D0D0>

File "g:\pythonproject\yolox-main\yolox\data\data_prefetcher.py", line 37, in preload
self.input_cuda()
│ └ <bound method DataPrefetcher._input_cuda_for_image of <yolox.data.data_prefetcher.DataPrefetcher object at 0x00000126F8F7D0D0>>
└ <yolox.data.data_prefetcher.DataPrefetcher object at 0x00000126F8F7D0D0>

File "g:\pythonproject\yolox-main\yolox\data\data_prefetcher.py", line 52, in _input_cuda_for_image
self.next_input = self.next_input.cuda(non_blocking=True)
│ │ │ │ └ <method 'cuda' of 'torch._C._TensorBase' objects>
│ │ │ └ tensor([[[[ 0.1426, 0.1426, 0.1254, ..., -0.5253, -0.5424, -0.5424],
│ │ │ [ 0.1426, 0.1426, 0.1254, ..., -0.5424, ...
│ │ └ <yolox.data.data_prefetcher.DataPrefetcher object at 0x00000126F8F7D0D0>
│ └ tensor([[[[ 0.1426, 0.1426, 0.1254, ..., -0.5253, -0.5424, -0.5424],
│ [ 0.1426, 0.1426, 0.1254, ..., -0.5424, ...
└ <yolox.data.data_prefetcher.DataPrefetcher object at 0x00000126F8F7D0D0>

RuntimeError: CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 6.00 GiB total capacity; 177.88 MiB already allocated; 0 bytes free; 4.71 GiB reserved in total by PyTorch)

(Swin) G:\Pythonproject\YOLOX-main>

[Joker316701882]

It's a known error #91 . We are working on it now.

[1VeniVediVeci1]

试试训练时把-o指令去掉

[haomayang1126]

试试训练时把-o指令去掉

thx~ 问题解决了

[haomayang1126]

It's a known error #91 . We are working on it now.

delete -o in command，it works

[lyp-oss]

试试训练时把-o指令去掉

我去掉-o之后还是出现这个问题：
RuntimeError: CUDA out of memory. Tried to allocate 100.00 MiB (GPU 0; 10.76 GiB total capacity; 9.62 GiB already allocated; 27.50 MiB free; 9.72 GiB reserved in total by PyTorch)

[GOATmessi8]

Then you have to reduce your batchsize or choose a small model like yolox-tiny or yolox-s

[LamnouarMohamed]

Hi,

I didn't occurred this problem according of training, but when I was tried to test our model after converted using trt, I found same problem how can I solve it ?

Thank you