Training stops outa nowhere

[mihailpettas]

Alright, im training a model on a code that used to work fluently and now it stops mid-train, something like this:

[flaml.automl: 12-22 15:17:03] {2344} INFO - iteration 59, current learner lgbm [flaml.automl: 12-22 15:17:03] {2532} INFO - at 10.6s, estimator lgbm's best error=0.0000, best estimator lgbm's best error=0.0000 [flaml.automl: 12-22 15:17:03] {2344} INFO - iteration 60, current learner xgboost [flaml.automl: 12-22 15:17:03] {2532} INFO - at 10.7s, estimator xgboost's best error=0.0000, best estimator lgbm's best error=0.0000 [flaml.automl: 12-22 15:17:03] {2344} INFO - iteration 61, current learner lgbm [flaml.automl: 12-22 15:17:03] {2532} INFO - at 10.8s, estimator lgbm's best error=0.0000, best estimator lgbm's best error=0.0000 [flaml.automl: 12-22 15:17:03] {2344} INFO - iteration 62, current learner xgboost

Any ideas? Early stop is on but it used to write a few stuff when it stooped, not it just returns to terminal.

[qingyun-wu]

Hi @jezerro,

Thank you for the question. Did you try running your exp on a different dataset? Does this happen just on this specific dataset/task you are working on or other datasets as well?

Several possible reasons include segmentation fault (although segmentation fault usually has an error message), or out of memory (more likely).

Can you please provide more information (e.g., code )for us to reproduce the error?

Thank you!

[qingyun-wu]

Great. We currently do not support setting certainty value. But I'd like to learn more about your needs with several clarification questions:

In 'if the system is not sure enough will just return', by 'system' are you referring to the classifier (and the certainty value is also specified for the classifier)? If so, are you using a type of learner that supports the usage of certainty value during prediction?

There is a feature request about prediction quality. Is the conformal prediction mentioned in that request what you need?

Thank you!

[mihailpettas]

Sorry if my technical terms are wrong, im not that familiar with ML. So im training a model with task = classification. I need the model to be able to recognize between hand gestures so predict() returns the right label, but i i do a new gesture it will return the one it believes its most similar. I need to make it return if its not sure about a gesture (eg < 90% certainty)

[mihailpettas]

Also can you please make the training output count minutes instead of seconds?

[qingyun-wu]

Then you can check out the nonconformist package (see this example) mentioned in the issue about prediction quality. You need to wrap your original model following their instructions to get the confidence level. It is not supported in FLAML yet. We will consider adding this feature in the future.

Thank you!

[mihailpettas]

Thanks Qingyun, i don't understand a thing from that link though, could you help me using it? I could avoid the whole confidence thing if the FLAML classifier can somehow understand as a third label everything that are not the first two.