-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathindex.html
121 lines (85 loc) · 6.1 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
<!DOCTYPE html>
<html><head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="chrome=1">
<title>betacal.github.io by betacal</title>
<link href="files/css.css" rel="stylesheet" type="text/css">
<link rel="stylesheet" href="files/style.css">
<script src="files/scale.js"></script>
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no">
<!--[if lt IE 9]>
<script src="//html5shiv.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
</head>
<body>
<div class="wrapper">
<header>
<h1 class="header">Beta calibration</h1>
<p class="header"></p>
<ul>
<li><a class="buttons github" href="https://github.com/betacal/">View On GitHub</a></li>
</ul>
<ul>
<li><a class="buttons github" href="https://github.com/betacal/aistats2017/tree/master/experiments">Experiments</a></li>
</ul>
</header>
<section>
<h1 id="beta-calibration-a-well-founded-and-easily-implemented-improvement-on-logistic-calibration-for-binary-classifiers">Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers</h1>
<p><a href="http://www.bris.ac.uk/engineering/people/meelis-kull/">Meelis Kull</a>, <a href="https://www.researchgate.net/profiles/Telmo_Silva_Filho">Telmo de Menezes e Silva Filho</a> and <a href="https://www.cs.bris.ac.uk/%7Eflach/">Peter Flach</a></p>
<p>For optimal decision making under variable class distributions and
misclassification costs a classifier needs to produce well-calibrated
estimates of the posterior probability. Isotonic calibration is a
powerful non-parametric method that is however prone to overfitting on
smaller datasets; hence a parametric method based on the logistic curve
is commonly used. While logistic calibration is designed to correct for a
specific kind of distortion where classifiers tend to score on too
narrow a scale, we demonstrate experimentally that many classifiers
including naive Bayes and Adaboost suffer from the opposite distortion
where scores tend too much to the extremes. In such cases logistic
calibration can easily yield probability estimates that are worse than
the original scores. Moreover, the logistic curve family does not
include the identity function, and hence logistic calibration can easily
uncalibrate a perfectly calibrated classifier.</p>
<p>In this paper we solve all these problems with a richer class of
calibration maps based on the Beta distribution. We derive the method
from first principles and show that fitting it is as easy as fitting a
logistic curve. Extensive experiments show that beta calibration is
superior to logistic calibration for naive Bayes and Adaboost.</p>
<h1 id="packages">Packages</h1>
<p>To make it easier for practitioners to experiment with our method, we have developed packages for <a href="https://www.python.org/">Python</a> and <a href="https://www.r-project.org/">R</a>.</p>
<ul>
<li><a href="https://pypi.python.org/pypi/betacal">Python package</a></li>
<li><a href="https://cran.r-project.org/web/packages/betacal/index.html">R package</a></li>
</ul>
<h1 id="tutorials">Tutorials</h1>
<p>We provide usage tutorials for beta calibration in Python and R.</p>
<ul>
<li><a href="https://github.com/REFRAME/betacal/blob/master/python/tutorial/Python%20tutorial.ipynb">Python tutorial</a></li>
<li><a href="https://github.com/REFRAME/betacal/blob/master/R/tutorial/Rtutorial.pdf">R tutorial</a></li>
</ul>
<h1 id="tutorials">Documents</h1>
<h2 id="tutorials">AISTATS2017</h2>
<p>Click on the following links to access the paper or download the poster and the presentation slides from AISTATS2017.</p>
<ul>
<li><a href="http://proceedings.mlr.press/v54/kull17a.html">AISTATS2017 paper</a></li>
<li><a href="https://github.com/betacal/aistats2017/blob/master/aistats2017_beta_calibration_poster.pdf">AISTATS2017 poster</a></li>
<li><a href="https://github.com/betacal/aistats2017/blob/master/aistats2017_beta_calibration_slides.pdf">AISTATS2017 slides</a></li>
</ul>
<h2 id="tutorials">Electronic Journal of Statistics</h2>
<p>We have published an extended version of the AISTATS2017 paper in the Electronic Journal of Statistics. The most significant additions include additional experiments with logistic regression, random forest, multi-layer perceptron and support vector machine; an empirical investigation of the effect of dataset size on the performance of the various calibration methods; and a novel statistical test that builds on beta calibration to recognise if the model deviates from being well-calibrated.
Click <a href="https://projecteuclid.org/euclid.ejs/1513306867">here</a> to access the paper.</p>
<h1 id="citing-beta-calibration">Citing Beta Calibration</h1>
<p>If you want to cite the AISTATS work, please use the following citation format:</p>
<p><em> Meelis Kull, Telmo Silva Filho, Peter Flach; Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54:623-631, 2017. </em></p>
<p>For the EJS paper, use the following citation format:</p>
<p><em>Meelis Kull, Telmo Silva Filho, Peter Flach; Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration. Electron. J. Statist. 11 (2017), no. 2, 5052--5080. doi:10.1214/17-EJS1338SI.</em></p>
<h1 id="support-or-contact">Support or Contact</h1>
<p>If you are having problems executing the experiments or the tutorials, do not hesitate to contact us.</p>
</section>
<footer>
<p><small>Hosted on <a href="https://pages.github.com/">GitHub Pages</a> using the Dinky theme</small></p>
</footer>
</div>
<!--[if !IE]><script>fixScale(document);</script><![endif]-->
</body></html>