Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH] Implement proper Lift curve; keep Cumulative gains as an option #5075

Merged
merged 4 commits into from
Dec 8, 2020

Conversation

janezd
Copy link
Contributor

@janezd janezd commented Nov 6, 2020

Issue

Fixes #5056. The widget showed cumulative gain curve instead of lift chart. Now it can do both.

Description of changes

The function used to use sklearn's roc curve. This PR has its own, simpler function modelled by the one that was used.

Includes
  • Code changes
  • Tests
  • Documentation

@codecov
Copy link

codecov bot commented Nov 6, 2020

Codecov Report

Merging #5075 (b84cc57) into master (1277566) will increase coverage by 0.00%.
The diff coverage is 94.54%.

@@           Coverage Diff           @@
##           master    #5075   +/-   ##
=======================================
  Coverage   84.74%   84.74%           
=======================================
  Files         286      286           
  Lines       60043    60061   +18     
=======================================
+ Hits        50884    50900   +16     
- Misses       9159     9161    +2     


graph_name = "plot"

YLabels = ("Lift", "TPR")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"TPR" used to be "TP Rate" and we still have "P Rate", so I suggest we use the same naming.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was unintentional. Fixed. Thanks for noticing.

@lanzagar
Copy link
Contributor

Documentation is missing for the new widget look and functions. Not sure if you plan to add that as well or @ajdapretnar will help with that?
After that I would merge this.

@lanzagar lanzagar changed the title Implement proper Lift curve; keep Cumulative gains as an option [ENH] Implement proper Lift curve; keep Cumulative gains as an option Nov 20, 2020
@janezd janezd assigned janezd and unassigned lanzagar Nov 20, 2020
@janezd janezd force-pushed the lift-curve-cummulative branch from 46f4b6a to 6bf723e Compare November 20, 2020 10:58
@janezd
Copy link
Contributor Author

janezd commented Nov 20, 2020

@ajdapretnar, I've rewritten the documentation and I ask for stamping the picture and reviewing the text.

@janezd janezd assigned ajdapretnar and unassigned janezd Nov 20, 2020
The **Lift curve** shows the relation between the number of instances which were predicted positive and those that are indeed positive and thus measures the performance of a chosen classifier against a random classifier. The graph is constructed with the cumulative number of cases (in descending order of probability) on the x-axis and the cumulative number of true positives on the y-axis. Lift curve is often used in segmenting the population, e.g., plotting the number of responding customers against the number of all customers contacted. You can also determine the optimal classifier and its threshold from the graph.
The **Lift curve** shows to curves for analysing the proportion of true positive data instances in relation to the classifier's threshold or the number of instances that we classify as positive.

Cummulative gains chart shows the proportion of true positive instances (for example, the number of clients who accept the offer) as a function of the number of positive instances (the number of clients contacted), assuming the the instances are ordered according to the models probability of being positive (e.g. ranking of clients).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to the models --> according to the model's

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, cumulative is written with a single m.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... which I occasionally, but seldom do.

2. If test results contain more than one classifier, the user can choose which curves she or he wants to see plotted. Click on a classifier to select or deselect the curve.
3. *Show lift convex hull* plots a convex hull over lift curves for all classifiers (yellow curve). The curve shows the optimal classifier (or combination thereof) for each desired TP/P rate.
4. Press *Save Image* if you want to save the created image to your computer in a .svg or .png format.
5. Produce a report.
6. 2-D pane with **P rate** (population) as x-axis and **TP rate** (true positives) as a y-axis. The diagonal line represents the behavior of a random classifier. Click and drag to move the pane and scroll in or out to zoom. Click on the "*A*" sign at the bottom left corner to realign the pane.
6. A plot with **Lift** or **true positive rate** vs. **P rate**. The dashed line represents the behavior of a random classifier.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P rate or positive rate (it would make sense to explain this as in the previous part with Lift)
Also, the second sentence refers only to cumulative gains. I'd make this a bit more obvious.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not. The dashed line also appears for lift, but it's horizontal, at 1. It is not always at the bottom; lift curve can go below 1.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, didn't notice it! 👀


References
----------
The widgets that provide the right type of the signal needed by the **Lift Curve** (evaluation data) are [Test & Score](../evaluate/testandscore.md) and [Predictions](../evaluate/predictions.md).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Predictions part is true only for labelled data. Perhaps make this clear?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't both?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but Test & Score warns you about missing target variable, while Predictions doesn't. I don't know, I just think it would be clearer that way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand what you meant. But it's somehow clear that the data has to have a target variable, so I'd rather keep it short.


Handouts of the University of Notre Dame on Data Mining - Lift Curve. Available [here](https://www3.nd.edu/~busiforc/handouts/DataMining/Lift%20Charts.html).
In the example below, we observe the lift curve and cummulative gain for the bank marketing data, where the classification goal is to predict whether the client will accept a term deposit offer based on his age, job, education, marital status and similar data. The data set is available in the Datasets widget. We run the learning algorithms in the Test Learners widget and send the results to Lift Curve. to see their performance against a random model. Of the two algorithms tested, logistic regression outperforms the naive Bayesian classifier. The curve tells us that by picking the first 20 % of clients as ranked by the model, we are going to hit four times more positive instances than by selecting a random sample with 20 % of clients.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

results to Lift Curve. to see their performance against a random model.

? remove the full stop? Also, Test Learners is now Test and Score.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whatever I was reading after writing this documentation, it was not this documentation. Sorry that you had to check my typos.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tell me about it. I proofread my papers three times only for the second reader to find about 10 typos in them. 😒

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... with the first one being the fifth word of the abstract.

@janezd janezd force-pushed the lift-curve-cummulative branch from 6bf723e to 589216b Compare November 20, 2020 11:32
@janezd janezd force-pushed the lift-curve-cummulative branch from 589216b to b84cc57 Compare November 20, 2020 13:17
@lanzagar lanzagar merged commit d04e3cd into biolab:master Dec 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Check and correct the Lift Curve widget
3 participants