[ENH] OWTestLearners: Cross validation by feature

[VesnaT]

Issue

Add type of CV where number of feature values determines number of folds.
Use only discrete meta features.

Description of changes

Add radio button option to select this type of CV.
Add combo box to select feature to give nr. of folds.
Documentation comes when gui is approved.

Includes

• Code changes
• Tests
• Documentation

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@51c97b6). Click here to learn what that means.
The diff coverage is 89.28%.

@@            Coverage Diff            @@
##             master    #2145   +/-   ##
=========================================
  Coverage          ?   72.08%           
=========================================
  Files             ?      318           
  Lines             ?    54653           
  Branches          ?        0           
=========================================
  Hits              ?    39397           
  Misses            ?    15256           
  Partials          ?        0

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 51c97b6...2a5bf2f. Read the comment docs.

[janezd]

I know this will come as a totally unusual surprise, but I'm disapproving the GUI. :)

The feature, if I read the code correctly, doesn't determine the number of folds, but the folds themselves. This looks like a useful feature, but I wouldn't expect it in cross-validation, and particularly not in the "Number of folds" combo. "From feature" also doesn't fit into an unexpanded combo.

The "from feature" option fundamentally changes the sampling procedure in the sense that it is no longer random.

As a minor argument against this GUI, we also discourage show/hide, and prefer enabling/disabling instead, where this could be sensibly avoided.

One solution would be to add a new radio button "Cross validation by feature" below "Cross validation". With this, this functionality would be more prominent and we'd no longer need hiding/showing of parts of the interface.

The radio button can be disable if there are no meta attributes.

If you do this, don't forget to provide appropriate migration of settings.

[markotoplak]

The main functionality is not tested (the corresponding radio button is never clicked). :)

When cross validation by feature is used for a file, which is then disconnected and connected back, the setting are reset.