Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error after resume training from previous non finished training #723

Open
Dadja111 opened this issue May 3, 2024 · 0 comments
Open

Error after resume training from previous non finished training #723

Dadja111 opened this issue May 3, 2024 · 0 comments

Comments

@Dadja111
Copy link

Dadja111 commented May 3, 2024

Error for 1_Default_LightGBM_SampleWeigthing_Update_1

'AutoML_1/X.data'
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/supervised/base_automl.py", line 1178, in _fit
trained = self.train_model(params)
File "/usr/local/lib/python3.10/site-packages/supervised/base_automl.py", line 384, in train_model
mf.train(results_path, model_subpath)
File "/usr/local/lib/python3.10/site-packages/supervised/model_framework.py", line 171, in train
train_data, validation_data = self.validation.get_split(k_fold, repeat)
File "/usr/local/lib/python3.10/site-packages/supervised/validation/validation_step.py", line 28, in get_split
return self.validator.get_split(k, repeat)
File "/usr/local/lib/python3.10/site-packages/supervised/validation/validator_kfold.py", line 120, in get_split
X = load_data(self._X_path)
File "/usr/local/lib/python3.10/site-packages/supervised/utils/utils.py", line 25, in load_data
return store.get(file_path)
File "/usr/local/lib/python3.10/site-packages/supervised/utils/utils.py", line 11, in get
return copy.deepcopy(Store.data[key])
KeyError: 'AutoML_1/X.data'

Indeed I observed from extraction below from your code (https://github.com/mljar/mljar-supervised/blob/master/supervised/utils/utils.py#L4C1-L29C39):

class Store:
data = {}

def set(self, key, value):
    Store.data[key] = value

def get(self, key):
    return copy.deepcopy(Store.data[key])

def dump_data(file_path, df):
store = Store()
store.set(file_path, df)
# try:
# df.to_parquet(file_path, index=False)
# except Exception as e:
# df.to_csv(file_path, index=False)

def load_data(file_path):
store = Store()
return store.get(file_path)
# try:
# return pd.read_parquet(file_path)
# except Exception as e:
# return pd.read_csv(file_path)

Calling load_data function will always result from error. The Store class instance does not containing any data after initialization. So looking for some key in store will always result in error.

I was not able to continue training due to a such error. Every new model fitting result in the same error.
I also get "'module' object is not callable" several time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant