AttributeError: Can\'t pickle local object \'_objective_function_wrapper.<locals>.inner

[pseudotensor]

LightGBM models are normally picklable, but when using a custom objective (and I assume eval_function etc.) they are not picklable.

import lightgbm as lgb
import _pickle as pickle
import numpy as np

def custom_asymmetric_objective(y_true, y_pred):
    residual = (y_true - y_pred).astype("float")
    grad = np.where(residual < 0, -2 * 10.0 * residual, -2 * residual)
    hess = np.where(residual < 0, 2 * 10.0, 2.0)
    return grad, hess

params = {}
params['objective'] = custom_asymmetric_objective
model = lgb.LGBMRegressor(**params)
X = np.array([[1,2,3,4], [2,3,4,5]]).transpose()
y = np.array([1,2,3,3])
model.fit(X, y)
preds = model.predict(X)
pickle.dump(model, open("model.pkl", "wb"))
print("Done")

Leads to:

Traceback (most recent call last):
  File "lgbm.py", line 22, in <module>
    pickle.dump(model, open("model.pkl", "wb"))
AttributeError: Can't pickle local object '_objective_function_wrapper.<locals>.inner'
jon@pseudotensor:~$ 

The problem is the def inside def way of defining the function:
https://github.com/microsoft/LightGBM/blob/master/python-package/lightgbm/sklearn.py#L18-L78

The solution is to either:

1. remove such def inside def uses and just directly make the function.
2. If using a decorator, then you need to use the standard way that allows them to be pickleable via functools.wraps

http://gael-varoquaux.info/programming/decoration-in-python-done-right-decorating-and-pickling.html

[StrikerRUS]

@pseudotensor Thank you very much for the bug catch! And special thanks for the solutions!

[pseudotensor]

Ya just change

    def inner(preds, dataset):

to

@wraps(func)
    def inner(preds, dataset):

in 2 places in python-package/lightgbm/sklearn.py Can add the above code as test.

[pseudotensor]

Unfortunately, that doesn't work, still get:

(4, 2)
[0. 0. 0. 0.]
Traceback (most recent call last):
  File "lgbm.py", line 21, in <module>
    pickle.dump(model, open("model.pkl", "wb"))
_pickle.PicklingError: Can't pickle <function custom_asymmetric_objective at 0x7fdbbf708a60>: it's not the same object as __main__.custom_asymmetric_objective

[pseudotensor]

import _pickle as pickle
import numpy as np

def custom_asymmetric_objective(y_true, y_pred):
    residual = (y_true - y_pred).astype("float")
    grad = np.where(residual < 0, -2 * 10.0 * residual, -2 * residual)
    hess = np.where(residual < 0, 2 * 10.0, 2.0)
    return grad, hess



pickle.dump(custom_asymmetric_objective, open("func.pkl", "wb"))
print("Done")

But this still works, so it's still some kind of problem inside LightGBM.

[pseudotensor]

Maybe the use of decorators needs to be avoided entirely.

[StrikerRUS]

@pseudotensor Got it work locally by converting to a new class as mentioned in http://gael-varoquaux.info/programming/decoration-in-python-done-right-decorating-and-pickling.html. Will submit a PR with tests tomorrow.

[pseudotensor]

Thanks, would be great to get this fixed soon :)

[Ludecan]

@pseudotensor , @StrikerRUS

I'm having this same issue with version 2.2.3 of LightGBM in Python 3.5.7. Running the example code above raises the AttributeError exception.

However if I install from sources version 2.2.4 the issue is gone.

Indeed it seems the fix is not merged in version 2.2.3 sources here, https://pypi.org/project/lightgbm/#history since sklearn.py still has the inner and wrapper declarations instead of the wrapper class.

Would it be possible to release a new PyPi version of the packages with the fix merged in? I can't install the sources version in my production environment because of the CMake dependency.

[guolinke]

@Ludecan refer to #2138, it is blocked by some in-review PRs.