issue with training yolov3tiny the given example in lava-dl

[gwgkarunasena57]

Hi,

When i am running the training for the given example for yolov3-tiny , it always gives the following error after one or two iterations. Can you please help?

poch 1: i = 21 , 1762.1063 ms elapsed
Train loss = 473.39488 (min = 458.45162) [email protected] = 0.05084 (max = 0.04602)
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.del at 0x7762b0762b90>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 4056, in del
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.del at 0x7762b0762b90>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 4056, in del
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.del at 0x7762b0762b90>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 4056, in del
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.del at 0x7762b0762b90>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 4056, in del
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.del at 0x7762b0762b90>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 4056, in del
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.del at 0x7762b0762b90>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 4056, in del
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.del at 0x7762b0447400>
Traceback (most recent call last):
File "/usr/lib/python3.10/tkinter/init.py", line 388, in del
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Tcl_AsyncDelete: async handler deleted by the wrong thread
Aborted (core dumped)

Thanks and Rgds,
Udayanga

[ZoltanV-V]

I came across this issue and I was able to fix it with this:

import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

• Make sure to add to the very top of your imports in case there are other scripts that call matplotlib too.

[gwgkarunasena57]

@ZoltanV-V thank you . It is working. However,in my case where i am using 'NVIDIA GeForce RTX 3080' with 10GB GPU memory, the only i successful batch size for training is 2 (very small batch size :D). Were you able to run with 32 or 16 batch size? Normally how many GPUs you used?.

Thanks and Rgds.
Udayanga

[ZoltanV-V]

This is a really late response to this, sorry! But I also had to set my batch-size to 1 when training using an RTX 3080 with 16GB GPU memory. I tried to parallelize with 2 GPUs but there seemed to be a bug that was causing a 'replica' error. I had to use DistributedDataParallel instead of DataParallel to get it working.

[gwgknudayanga]

Thanks for the reply @ZoltanV-V