[ENH] Calibration plot (add performance curves) and a new Calibrated Learner widget #3881

janezd · 2019-06-13T13:38:34Z

This PR revamps an old widget, adds a new one, adds some core functionality, and adds a few minor fixes to base classes.

Orange.evaluation.performance_curves.Curves is a class that computes performance curves (ca, f1, sens, spec...) as functions of probabilities,
Widget Calibration curves shows such curves,
A new module Orange.model.calibration contains learners and classifiers that set and user thresholds and calibrate probabilities,
Calibrated Learner is a new widget that produces such models.

Working on this uncovered some hidden bugs and glitches in other parts of the code, hence a large number of commits.

Fixes #1267.

codecov · 2019-06-13T21:37:49Z

Codecov Report

Merging #3881 into master will increase coverage by 0.24%.
The diff coverage is 96.87%.

@@            Coverage Diff             @@
##           master    #3881      +/-   ##
==========================================
+ Coverage   84.31%   84.56%   +0.24%     
==========================================
  Files         385      390       +5     
  Lines       72758    73803    +1045     
==========================================
+ Hits        61343    62408    +1065     
+ Misses      11415    11395      -20

…erformance curves

… compute curves

…derived widget to change the default name without interferring with user-changed settings.

janezd · 2019-06-20T17:55:36Z

Orange/widgets/evaluate/utils.py

@@ -47,7 +47,7 @@ def results_for_preview(data_name=""):
    from Orange.classification import \
        LogisticRegressionLearner, SVMLearner, NuSVMLearner

-    data = Table(data_name or "ionosphere")


I changed this because ionosphere is no longer present. It affects previews of widgets in Evaluation.

janezd · 2019-06-20T17:59:30Z

Orange/widgets/gui.py

@@ -1783,6 +1783,9 @@ def __init__(self, master, enableDragDrop=False, dragDropCallback=None,
    def sizeHint(self):
        return self.size_hint

+    def minimumSizeHint(self):
+        return self.size_hint
+


I added this because list view remained too large. This is the only way with which I managed to reduce its size - setMinimumSizeHint just didn't work...

janezd · 2019-06-20T18:01:37Z

Orange/evaluation/testing.py

@@ -317,7 +317,7 @@ def split_by_model(self):
                res.probabilities = self.probabilities[(i,), :, :]

            if self.models is not None:
-                res.models = self.models[:, i]
+                res.models = self.models[:, i:i+1]


I believe this was a bug because it changed self.models from 2d to 1d array. Apparently nobody needed this so far.

janezd · 2019-06-20T18:03:11Z

Orange/widgets/evaluate/owtestlearners.py

@@ -756,7 +757,8 @@ def __update(self):
                    stratified=self.shuffle_stratified,
                    random_state=rstate)
            elif self.resampling == OWTestLearners.TestOnTrain:
-                sampler = Orange.evaluation.TestOnTrainingData()
+                sampler = Orange.evaluation.TestOnTrainingData(
+                    store_models=True)


Without these changes, Orange canvas can't produce evaluation results with valid models that Calibration plot can output. This will also be useful in, say ROC Analysis that will now also be able to output a model with some specific threshold.

janezd · 2019-06-20T18:07:00Z

Orange/widgets/evaluate/owtestlearners.py

@@ -315,7 +315,7 @@ def set_learner(self, learner, key):
            # Removed
            self._invalidate([key])
            del self.learners[key]
-        else:
+        elif learner is not None:


We haven't encountered this bug before because (I suppose) no learner widget can fail to produce a learner. Now it can - Calibrated Learner outputs None if it doesn't have a learner on its input.

janezd · 2019-06-20T18:09:54Z

Orange/widgets/evaluate/contexthandlers.py

-    def __init__(self, targetAttr, selectedAttr):
-        super().__init__()
-        self.targetAttr, self.selectedAttr = targetAttr, selectedAttr
+    """Context handler for evaluation results"""


This is obviously a new context handler with the same name as an old one that nobody used because it probably didn't work and it was weird. ROC Analysis and Lift Curves currently don't have context; they can now have this one.

janezd · 2019-06-20T18:18:25Z

The potentially problematic commit (but I think it's OK) is f742ff9.

Currently, widgets for learners set the default name and the user can change it. The default name was always fixed. In Calibrated Learner however, the name should depend on the widget's input and settings to produce a meaningful name. The widget could reset the name (as a part of its context?), but with this it would override user's changes.

This commit sets the default name as editline's placeholder, so the user sees the default, (s)he can enter a different name, the widget can change the default (without having to pay attention to any user's changes) and it the user then removes hir input, (s)he gets the widget's default.

ajdapretnar · 2019-06-21T08:08:30Z

There's a glitch when changing the threshold. Try it with Logistic Regression, which has a long name. The difference between the size of the numbers makes the widget change size. I suggest a little padding to account for long model names.

Orange/widgets/evaluate/owcalibrationplot.py

janezd · 2019-06-24T19:55:37Z

Huh. I limited the model name to 20 characters (elide if longer; I can't add tooltips because Qt's rich text doesn't support them). Not the most elegant and foolproof solution, but I don't have a better one.

ajdapretnar · 2019-06-28T10:39:56Z

Exception: | AttributeError: 'OWCalibratedLearner' object has no attribute 'calibrate'

Press Report in Calibrated Learner widget.

janezd · 2019-06-28T11:51:52Z

Exception: | AttributeError: 'OWCalibratedLearner' object has no attribute 'calibrate'

Press Report in Calibrated Learner widget.

I obviously haven't tried it. :) Now it works.

ajdapretnar · 2019-06-28T12:06:44Z

To merge this PR I would suggest:

adding an icon for Calibrated Learner (at least a temporary one)
check the formatting of report in Calibration Plot
make sure Calibration Plot works with sparse (or at least doesn't crash)

janezd · 2019-07-12T10:37:02Z

Merged due to lack of activity.

janezd force-pushed the calibration-plot-curves branch from dd860da to 2312bf4 Compare June 13, 2019 21:41

janezd mentioned this pull request Jun 13, 2019

Feature request: cutoff/threshold value for classification #1267

Closed

janezd changed the title ~~Calibration plot: Add plots of ca, sens/spec, prec/recall, ppv/npv~~ [WIP] Calibration plot: Add plots of ca, sens/spec, prec/recall, ppv/npv Jun 13, 2019

janezd force-pushed the calibration-plot-curves branch 7 times, most recently from 064c928 to a67d0fd Compare June 17, 2019 21:07

janezd added 15 commits June 18, 2019 14:41

Calibration plot: Add plots of ca, sens/spec, prec/recall, ppv/npv

35a6f4b

Calibration plot: Add threshold line

2fa1750

Calibration plot: Refactor computation of metrics

d47b68b

Testing: Keep 2d array of models when splitting Results by models

585feb2

Test Learners: Store models when there is just one; properly stack them

7b876e6

classification: Add ModelWithThreshold

93b7a72

Calibration plot: Output selected model

ff67b49

Orange.evaluation.performance_curves: Add module for computation of p…

a4424fb

…erformance curves

Calibration plot: Use Orange.evaluation.testing.performance_curves to…

6024897

… compute curves

Calibration plot: Fix selected model output

1cfbeec

OWLearnerWidget: Let default name appear as placeholder. This allows …

f742ff9

…derived widget to change the default name without interferring with user-changed settings.

evaluations.testing: Minor fixes in unit tests

c5d070d

OWTestLearners: Skip inactive signals (e.g. learner widget outputs None)

557fa2e

Calibrated Learner: Add widget

1a8b013

Calibration plot: Add context settings

6ac1db1

janezd force-pushed the calibration-plot-curves branch from a67d0fd to 4cc3a54 Compare June 18, 2019 20:56

OWCalibration Plot: Unit tests and some fixes

2edcb39

janezd force-pushed the calibration-plot-curves branch from 4cc3a54 to 2edcb39 Compare June 18, 2019 23:07

Calibration plot: Test missing probabilities and single classes

2049afa

janezd force-pushed the calibration-plot-curves branch from 47308ca to 2049afa Compare June 19, 2019 11:24

janezd changed the title ~~[WIP] Calibration plot: Add plots of ca, sens/spec, prec/recall, ppv/npv~~ [ENH] Calibration plot (add performance curves) and a new Calibrated Learner widget Jun 19, 2019

janezd changed the title ~~[ENH] Calibration plot (add performance curves) and a new Calibrated Learner widget~~ Calibration plot: Add plots of ca, sens/spec, prec/recall, ppv/npv Jun 19, 2019

janezd changed the title ~~Calibration plot: Add plots of ca, sens/spec, prec/recall, ppv/npv~~ [ENH] Calibration plot (add performance curves) and a new Calibrated Learner widget Jun 19, 2019

janezd commented Jun 20, 2019

View reviewed changes

ajdapretnar reviewed Jun 21, 2019

View reviewed changes

Orange/widgets/evaluate/owcalibrationplot.py Outdated Show resolved Hide resolved

Calibration plot: Minor fixes

04d05f4

janezd force-pushed the calibration-plot-curves branch from 37323a1 to 04d05f4 Compare June 24, 2019 20:23

janezd assigned ajdapretnar Jun 28, 2019

Calibrated Learner: Fix report

6695ee9

Calibrated Learner: Add icon

65c69e2

Calibration plot: Nicer report

864d7b5

janezd merged commit 9deed46 into biolab:master Jul 12, 2019

janezd mentioned this pull request Oct 23, 2019

[FIX] ROC and LiftCurve: Store context settings #4138

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENH] Calibration plot (add performance curves) and a new Calibrated Learner widget #3881

[ENH] Calibration plot (add performance curves) and a new Calibrated Learner widget #3881

janezd commented Jun 13, 2019 •

edited

Loading

codecov bot commented Jun 13, 2019 •

edited

Loading

janezd Jun 20, 2019

janezd Jun 20, 2019

janezd Jun 20, 2019

janezd Jun 20, 2019

janezd Jun 20, 2019

janezd Jun 20, 2019

janezd commented Jun 20, 2019

ajdapretnar commented Jun 21, 2019

janezd commented Jun 24, 2019

ajdapretnar commented Jun 28, 2019

janezd commented Jun 28, 2019

ajdapretnar commented Jun 28, 2019 •

edited

Loading

janezd commented Jul 12, 2019

[ENH] Calibration plot (add performance curves) and a new Calibrated Learner widget #3881

[ENH] Calibration plot (add performance curves) and a new Calibrated Learner widget #3881

Conversation

janezd commented Jun 13, 2019 • edited Loading

codecov bot commented Jun 13, 2019 • edited Loading

Codecov Report

janezd Jun 20, 2019

Choose a reason for hiding this comment

janezd Jun 20, 2019

Choose a reason for hiding this comment

janezd Jun 20, 2019

Choose a reason for hiding this comment

janezd Jun 20, 2019

Choose a reason for hiding this comment

janezd Jun 20, 2019

Choose a reason for hiding this comment

janezd Jun 20, 2019

Choose a reason for hiding this comment

janezd commented Jun 20, 2019

ajdapretnar commented Jun 21, 2019

janezd commented Jun 24, 2019

ajdapretnar commented Jun 28, 2019

janezd commented Jun 28, 2019

ajdapretnar commented Jun 28, 2019 • edited Loading

janezd commented Jul 12, 2019

janezd commented Jun 13, 2019 •

edited

Loading

codecov bot commented Jun 13, 2019 •

edited

Loading

ajdapretnar commented Jun 28, 2019 •

edited

Loading