java.lang.Exception: Dataset create call failed in LightGBM with error: vector::reserve

[jaipal1995]

Issue :

Hi, I am trying to use LightGBMRegressor. When I pass a dense vector to the model, it throws the error mentioned above. It's somehow related to native libraries' dependencies LightGBM is using after going into some details. How can we fix this?

I tried with the sparse-vector as well, it failed with this exception: java.lang.Exception: Dataset create call failed in LightGBM with error: The number of columns should be smaller than INT32_MAX. My feature vector is of size 80-90. So, this is a bit weird.

Info :

• MMLSpark Version: 0.18.1 / 0.18.0 (tried both)
• Spark Version - 2.3.1
• LightGBM Version : 2.2.350

[jaipal1995]

@mhamilton723,
Can you please provide any update on this?

[imatiach-msft]

@jaipal1995 are you still seeing this issue with latest lightgbm upgrade on master today?

#850

There was a major fix for allocating more than max int array values per partition which was in the upgrade:

microsoft/LightGBM#2859