Worker dying issue with controller training

[rohanb2018]

Hello, has anyone ran into an issue where one or more of the workers created in python traincontroller.py dies without explanation, causing the entire script to hang because it is waiting for the dead workers to finish their evaluations?

I've checked the logs created in tmp for each of the worker processes and unfortunately the .err logs seem to be uninformative or empty.

I think it might be a GPU memory issue or some issue related to some modifications I made to the CarRacing environment, but the lack of any error logging is concerning. It also does not seem to consistently happen, which is also strange.

Thanks!