Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

properly resolve obj alias objective #66

Open
nipnipj opened this issue Apr 20, 2023 · 3 comments
Open

properly resolve obj alias objective #66

nipnipj opened this issue Apr 20, 2023 · 3 comments
Labels
bug an unexpected problem or unintended behavior

Comments

@nipnipj
Copy link

nipnipj commented Apr 20, 2023

Error message Error in init(env): For early stopping, valids must have at least one element is produced when trying to train a multi-class model with

lgb_model <-
  boost_tree(
      stop_iter = 1000
  ) %>%
  set_engine(   
     "lightgbm",
      validation = 0.2,
      objective = 'multiclass',
      metric="none",
      eval = 'auc_mu',
      num_class = n_classes
  )
@simonpcouch
Copy link
Contributor

Thanks for the issue. Training for a multi-class task with early stopping indeed works fine:

library(bonsai)
#> Loading required package: parsnip
library(modeldata)

data(penguins)

lgb_model <-
  boost_tree("classification", stop_iter = 1000) %>%
  set_engine("lightgbm", validation = 0.2, eval = 'auc_mu')

fit(
  lgb_model,
  formula = species ~ flipper_length_mm + island, 
  data = penguins
)
#> parsnip model object
#> 
#> <lgb.Booster>
#>   Public:
#>     add_valid: function (data, name) 
#>     best_iter: 14
#>     best_score: 0.973090277777778
#>     current_iter: function () 
#>     dump_model: function (num_iteration = NULL, feature_importance_type = 0L) 
#>     eval: function (data, name, feval = NULL) 
#>     eval_train: function (feval = NULL) 
#>     eval_valid: function (feval = NULL) 
#>     finalize: function () 
#>     initialize: function (params = list(), train_set = NULL, modelfile = NULL, 
#>     lower_bound: function () 
#>     params: list
#>     predict: function (data, start_iteration = NULL, num_iteration = NULL, 
#>     raw: NA
#>     record_evals: list
#>     reset_parameter: function (params, ...) 
#>     rollback_one_iter: function () 
#>     save: function () 
#>     save_model: function (filename, num_iteration = NULL, feature_importance_type = 0L) 
#>     save_model_to_string: function (num_iteration = NULL, feature_importance_type = 0L) 
#>     set_train_data_name: function (name) 
#>     to_predictor: function () 
#>     update: function (train_set = NULL, fobj = NULL) 
#>     upper_bound: function () 
#>   Private:
#>     eval_names: auc_mu
#>     get_eval_info: function () 
#>     handle: lgb.Booster.handle
#>     higher_better_inner_eval: TRUE
#>     init_predictor: NULL
#>     inner_eval: function (data_name, data_idx, feval = NULL) 
#>     inner_predict: function (idx) 
#>     is_predicted_cur_iter: list
#>     name_train_set: training
#>     name_valid_sets: list
#>     num_class: 3
#>     num_dataset: 2
#>     predict_buffer: list
#>     set_objective_to_none: FALSE
#>     train_set: lgb.Dataset, R6
#>     train_set_version: 1
#>     valid_sets: list

Created on 2023-04-24 with reprex v2.0.2

bonsai handles setting the obj argument to lgb.train, and will warn if it's passed:

library(bonsai)
#> Loading required package: parsnip
library(modeldata)

data(penguins)

lgb_model <-
  boost_tree("classification", stop_iter = 1000) %>%
  # note `object = .`, since R partial-matches `obj` to `object`
  set_engine(object = ., "lightgbm", obj = "multiclass", validation = 0.2, eval = 'auc_mu')

fit(
  lgb_model,
  formula = species ~ flipper_length_mm + island, 
  data = penguins
)
#> Warning: The following argument(s) are guarded by bonsai and will not be passed
#> to `lgb.train`: obj
#> parsnip model object
#> ... [clipped]

Created on 2023-04-24 with reprex v2.0.2

but fails to do so when that argument is passed by its alias objective. Will leave this open to make sure we handle that alias properly in the next release.

@simonpcouch simonpcouch changed the title Early stopping doesn't work for multi-class task properly resolve obj alias objective Apr 24, 2023
@nipnipj
Copy link
Author

nipnipj commented Apr 24, 2023

Your suggestion worked. I'm getting another error, not sure if it is related, tho.

data(penguins, package= "modeldata")
data <- penguins
rec <- data %>% 
  recipe(species ~ flipper_length_mm + island) 

lgbm_model <-
  boost_tree(
    stop_iter = 100,
    trees = 1000, 
    learn_rate =  0.07, 
  ) %>%
  set_engine(
    object = .,
    "lightgbm",
    obj = 'multiclass',
    metric="none",
    eval = 'multi_logloss',
    validation = 0.2
 ) %>% 
  set_mode("classification)

learner_lgbm <- workflow() %>% 
  add_recipe(rec) %>% 
  add_model(lgbm_model)

cv_folds <- data %>%
  vfold_cv(v = 2, repeats = 1, strata  = species) 

lg <- fit_resamples(
  object = learner_lgbm,
  resamples = cv_folds,
  metrics = metric_set(roc_auc)
)

Then → A | warning: The following argument(s) are guarded by bonsai and will not be passed to lgb.train: obj There were issues with some computations A: x2 error appears.

@simonpcouch
Copy link
Contributor

See:

bonsai handles setting the obj argument to lgb.train, and will warn if it's passed

No need to pass that argument!👍

@simonpcouch simonpcouch added the bug an unexpected problem or unintended behavior label Oct 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug an unexpected problem or unintended behavior
Projects
None yet
Development

No branches or pull requests

2 participants