LightGBM causes Optuna to go out of memory and crash

[AkshayNovacene]

I was wondering if there was a way to batch the inputs when using Optuna or reduce the load on it.

Optuna seems to go out of memory and crash on google colab when I try to run it on my big dataset. This is observed only when using LightGBM, but other models seem to work fine.

[pplonski]

Hi @AkshayNovacene,

LightGBM has some issues with memory management microsoft/LightGBM#4239

The solution would be to train each model in separate process, to always release full memory after training. I'm thinking about using Ray framework https://github.com/ray-project/ray for this (you can train each model in separate process and you can distribute training on multiple machines).

[AkshayNovacene]

Thanks for the prompt reply. I am still trying to dig into the issue because lightGBM works fine with Optuna when I use Optuna out of the box. I was using the lgb.LGBMClassifier component for training manually. It goes out of memory only when I use the optuna mode through the automl package and try to train Lightgbm.