PipeOp to try repair predicting with unseen factor levels

[berndbischl]

problem: quite often, a learner breaks, because it sees SOME prediction in a larger table, which contains new, unseen factor levels. in such a case the predict of the underlying learner fails, completely.

see reprex here:
mlr-org/mlr3#97

this is really annoying. especially as this can happen on only a few observations, but we still 100% fail the complete prediction.

current options are: the mlr3 fallback learner. that does not really help. because this produces now fallback predictions on the complete test set.

here is MAYBE a better option.

PipOpUnseenLevels

before we go into the learner, we can on-training, store which levels are present in each factor.

PipOpUnseenLevels
train: task--stored-levels--->task
predict: task-->stored-levels-->task

train: simply stores a list, one element per factor feature, with the seen level
predict: does through all observations. for each observation where we see "unseen" levels, we create a random row, by sampling from the marginals of the columns.
that is a bit hacky, but should work?

[prockenschaub]

By "random row" you mean that for an observation with an unseen level we samplw every variable of from the marginals, even those variables for which the observed value was within the training sample, or just replace the unseen value itself with a draw from its own column marginal?

If I were to use this PipeOp, I would personally favour a more deterministic approach, either by filtering out those observations during prediction that contain a value that wasn't seen during training (ideally with warning) or by prespecifying a category that should be used if a new value is seen (the specification could say "marginals", which would include the second case in my question above)

[mb706]

Note we have the POBackupLearner for something like this: #204

[pfistfl]

This should be solved via fixfactors and imputation.
See e.g.
https://mlr3gallery.mlr-org.com/basics_pipelines_titanic/

[mb706]

together with the robustify pipeline this is probably as good as it gets.