-
Notifications
You must be signed in to change notification settings - Fork 12
/
How_to_Evaluate_Models.html
255 lines (229 loc) · 51.3 KB
/
How_to_Evaluate_Models.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="utf-8" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="author" content="Robert Kubinec" />
<meta name="date" content="2018-10-30" />
<title>How to Evaluate Models</title>
<style type="text/css">code{white-space: pre;}</style>
<style type="text/css">
div.sourceCode { overflow-x: auto; }
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; line-height: 100%; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; }
td.sourceCode { padding-left: 5px; }
code > span.kw { color: #007020; font-weight: bold; } /* Keyword */
code > span.dt { color: #902000; } /* DataType */
code > span.dv { color: #40a070; } /* DecVal */
code > span.bn { color: #40a070; } /* BaseN */
code > span.fl { color: #40a070; } /* Float */
code > span.ch { color: #4070a0; } /* Char */
code > span.st { color: #4070a0; } /* String */
code > span.co { color: #60a0b0; font-style: italic; } /* Comment */
code > span.ot { color: #007020; } /* Other */
code > span.al { color: #ff0000; font-weight: bold; } /* Alert */
code > span.fu { color: #06287e; } /* Function */
code > span.er { color: #ff0000; font-weight: bold; } /* Error */
code > span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
code > span.cn { color: #880000; } /* Constant */
code > span.sc { color: #4070a0; } /* SpecialChar */
code > span.vs { color: #4070a0; } /* VerbatimString */
code > span.ss { color: #bb6688; } /* SpecialString */
code > span.im { } /* Import */
code > span.va { color: #19177c; } /* Variable */
code > span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code > span.op { color: #666666; } /* Operator */
code > span.bu { } /* BuiltIn */
code > span.ex { } /* Extension */
code > span.pp { color: #bc7a00; } /* Preprocessor */
code > span.at { color: #7d9029; } /* Attribute */
code > span.do { color: #ba2121; font-style: italic; } /* Documentation */
code > span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code > span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code > span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
</style>
<link href="data:text/css;charset=utf-8,body%20%7B%0Abackground%2Dcolor%3A%20%23fff%3B%0Amargin%3A%201em%20auto%3B%0Amax%2Dwidth%3A%20700px%3B%0Aoverflow%3A%20visible%3B%0Apadding%2Dleft%3A%202em%3B%0Apadding%2Dright%3A%202em%3B%0Afont%2Dfamily%3A%20%22Open%20Sans%22%2C%20%22Helvetica%20Neue%22%2C%20Helvetica%2C%20Arial%2C%20sans%2Dserif%3B%0Afont%2Dsize%3A%2014px%3B%0Aline%2Dheight%3A%201%2E35%3B%0A%7D%0A%23header%20%7B%0Atext%2Dalign%3A%20center%3B%0A%7D%0A%23TOC%20%7B%0Aclear%3A%20both%3B%0Amargin%3A%200%200%2010px%2010px%3B%0Apadding%3A%204px%3B%0Awidth%3A%20400px%3B%0Aborder%3A%201px%20solid%20%23CCCCCC%3B%0Aborder%2Dradius%3A%205px%3B%0Abackground%2Dcolor%3A%20%23f6f6f6%3B%0Afont%2Dsize%3A%2013px%3B%0Aline%2Dheight%3A%201%2E3%3B%0A%7D%0A%23TOC%20%2Etoctitle%20%7B%0Afont%2Dweight%3A%20bold%3B%0Afont%2Dsize%3A%2015px%3B%0Amargin%2Dleft%3A%205px%3B%0A%7D%0A%23TOC%20ul%20%7B%0Apadding%2Dleft%3A%2040px%3B%0Amargin%2Dleft%3A%20%2D1%2E5em%3B%0Amargin%2Dtop%3A%205px%3B%0Amargin%2Dbottom%3A%205px%3B%0A%7D%0A%23TOC%20ul%20ul%20%7B%0Amargin%2Dleft%3A%20%2D2em%3B%0A%7D%0A%23TOC%20li%20%7B%0Aline%2Dheight%3A%2016px%3B%0A%7D%0Atable%20%7B%0Amargin%3A%201em%20auto%3B%0Aborder%2Dwidth%3A%201px%3B%0Aborder%2Dcolor%3A%20%23DDDDDD%3B%0Aborder%2Dstyle%3A%20outset%3B%0Aborder%2Dcollapse%3A%20collapse%3B%0A%7D%0Atable%20th%20%7B%0Aborder%2Dwidth%3A%202px%3B%0Apadding%3A%205px%3B%0Aborder%2Dstyle%3A%20inset%3B%0A%7D%0Atable%20td%20%7B%0Aborder%2Dwidth%3A%201px%3B%0Aborder%2Dstyle%3A%20inset%3B%0Aline%2Dheight%3A%2018px%3B%0Apadding%3A%205px%205px%3B%0A%7D%0Atable%2C%20table%20th%2C%20table%20td%20%7B%0Aborder%2Dleft%2Dstyle%3A%20none%3B%0Aborder%2Dright%2Dstyle%3A%20none%3B%0A%7D%0Atable%20thead%2C%20table%20tr%2Eeven%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0A%7D%0Ap%20%7B%0Amargin%3A%200%2E5em%200%3B%0A%7D%0Ablockquote%20%7B%0Abackground%2Dcolor%3A%20%23f6f6f6%3B%0Apadding%3A%200%2E25em%200%2E75em%3B%0A%7D%0Ahr%20%7B%0Aborder%2Dstyle%3A%20solid%3B%0Aborder%3A%20none%3B%0Aborder%2Dtop%3A%201px%20solid%20%23777%3B%0Amargin%3A%2028px%200%3B%0A%7D%0Adl%20%7B%0Amargin%2Dleft%3A%200%3B%0A%7D%0Adl%20dd%20%7B%0Amargin%2Dbottom%3A%2013px%3B%0Amargin%2Dleft%3A%2013px%3B%0A%7D%0Adl%20dt%20%7B%0Afont%2Dweight%3A%20bold%3B%0A%7D%0Aul%20%7B%0Amargin%2Dtop%3A%200%3B%0A%7D%0Aul%20li%20%7B%0Alist%2Dstyle%3A%20circle%20outside%3B%0A%7D%0Aul%20ul%20%7B%0Amargin%2Dbottom%3A%200%3B%0A%7D%0Apre%2C%20code%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0Aborder%2Dradius%3A%203px%3B%0Acolor%3A%20%23333%3B%0Awhite%2Dspace%3A%20pre%2Dwrap%3B%20%0A%7D%0Apre%20%7B%0Aborder%2Dradius%3A%203px%3B%0Amargin%3A%205px%200px%2010px%200px%3B%0Apadding%3A%2010px%3B%0A%7D%0Apre%3Anot%28%5Bclass%5D%29%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0A%7D%0Acode%20%7B%0Afont%2Dfamily%3A%20Consolas%2C%20Monaco%2C%20%27Courier%20New%27%2C%20monospace%3B%0Afont%2Dsize%3A%2085%25%3B%0A%7D%0Ap%20%3E%20code%2C%20li%20%3E%20code%20%7B%0Apadding%3A%202px%200px%3B%0A%7D%0Adiv%2Efigure%20%7B%0Atext%2Dalign%3A%20center%3B%0A%7D%0Aimg%20%7B%0Abackground%2Dcolor%3A%20%23FFFFFF%3B%0Apadding%3A%202px%3B%0Aborder%3A%201px%20solid%20%23DDDDDD%3B%0Aborder%2Dradius%3A%203px%3B%0Aborder%3A%201px%20solid%20%23CCCCCC%3B%0Amargin%3A%200%205px%3B%0A%7D%0Ah1%20%7B%0Amargin%2Dtop%3A%200%3B%0Afont%2Dsize%3A%2035px%3B%0Aline%2Dheight%3A%2040px%3B%0A%7D%0Ah2%20%7B%0Aborder%2Dbottom%3A%204px%20solid%20%23f7f7f7%3B%0Apadding%2Dtop%3A%2010px%3B%0Apadding%2Dbottom%3A%202px%3B%0Afont%2Dsize%3A%20145%25%3B%0A%7D%0Ah3%20%7B%0Aborder%2Dbottom%3A%202px%20solid%20%23f7f7f7%3B%0Apadding%2Dtop%3A%2010px%3B%0Afont%2Dsize%3A%20120%25%3B%0A%7D%0Ah4%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23f7f7f7%3B%0Amargin%2Dleft%3A%208px%3B%0Afont%2Dsize%3A%20105%25%3B%0A%7D%0Ah5%2C%20h6%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23ccc%3B%0Afont%2Dsize%3A%20105%25%3B%0A%7D%0Aa%20%7B%0Acolor%3A%20%230033dd%3B%0Atext%2Ddecoration%3A%20none%3B%0A%7D%0Aa%3Ahover%20%7B%0Acolor%3A%20%236666ff%3B%20%7D%0Aa%3Avisited%20%7B%0Acolor%3A%20%23800080%3B%20%7D%0Aa%3Avisited%3Ahover%20%7B%0Acolor%3A%20%23BB00BB%3B%20%7D%0Aa%5Bhref%5E%3D%22http%3A%22%5D%20%7B%0Atext%2Ddecoration%3A%20underline%3B%20%7D%0Aa%5Bhref%5E%3D%22https%3A%22%5D%20%7B%0Atext%2Ddecoration%3A%20underline%3B%20%7D%0A%0Acode%20%3E%20span%2Ekw%20%7B%20color%3A%20%23555%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%0Acode%20%3E%20span%2Edt%20%7B%20color%3A%20%23902000%3B%20%7D%20%0Acode%20%3E%20span%2Edv%20%7B%20color%3A%20%2340a070%3B%20%7D%20%0Acode%20%3E%20span%2Ebn%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Efl%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Ech%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Est%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Eco%20%7B%20color%3A%20%23888888%3B%20font%2Dstyle%3A%20italic%3B%20%7D%20%0Acode%20%3E%20span%2Eot%20%7B%20color%3A%20%23007020%3B%20%7D%20%0Acode%20%3E%20span%2Eal%20%7B%20color%3A%20%23ff0000%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%0Acode%20%3E%20span%2Efu%20%7B%20color%3A%20%23900%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%20code%20%3E%20span%2Eer%20%7B%20color%3A%20%23a61717%3B%20background%2Dcolor%3A%20%23e3d2d2%3B%20%7D%20%0A" rel="stylesheet" type="text/css" />
</head>
<body>
<h1 class="title toc-ignore">How to Evaluate Models</h1>
<h4 class="author"><em>Robert Kubinec</em></h4>
<h4 class="date"><em>2018-10-30</em></h4>
<p>A big part of the purpose of <code>idealstan</code> is to give people different options in fitting ideal point models to diverse data. Along with that, <code>idealstan</code> makes use of Bayesian model evaluation a la <a href="https://CRAN.R-project.org/package=loo/">loo</a> and also can analyze the posterior predictive distribution using <a href="http://mc-stan.org/users/interfaces/bayesplot">bayesplot</a>. <code>loo</code> is an approximation of the predictive error of a Bayesian model via leave-one-out cross-validation (LOO-CV). True LOO-CV on Bayesian models is computationally prohibitive because it involves estimating a new model for each data point. For IRT models incorporating thousands or even millions of observations, this is practically infeasible.</p>
<p><code>bayesplot</code> allows us to analyze the data we used to estimate the model compared to data produced by the model, or what is called the posterior predictive distribution. This is very useful as a general summary of model fit to see whether there are values of the outcome that we are over or under predicting.</p>
<p><code>idealstan</code> implements functions for each ideal point model that calculate the log-posterior probability of the data, which is the necessary input to use <code>loo</code>’s model evaluation features. This vignette demonstrates the basic usage. I also discuss best practices for evaluating models.</p>
<p>We first begin by simulating data for a standard IRT 2-PL ideal point model but with strategically missing data:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">irt_2pl <-<span class="st"> </span><span class="kw">id_sim_gen</span>(<span class="dt">inflate=</span><span class="ot">TRUE</span>)</code></pre></div>
<pre><code>## Warning: package 'bindrcpp' was built under R version 3.4.4</code></pre>
<p>We can then fit two ideal point models to the same data, one that uses the correct binomial model for a 0/1 binary outcome, and the second that tries to fit a Poisson count model to this same binary data.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="co"># Because of CRAN limitations, only using 2 cores & 2 chains</span>
irt_2pl_correct <-<span class="st"> </span><span class="kw">id_estimate</span>(<span class="dt">idealdata=</span>irt_2pl,
<span class="dt">model_type=</span><span class="dv">2</span>,
<span class="dt">restrict_ind_high =</span> <span class="kw">as.character</span>(<span class="kw">sort</span>(irt_2pl<span class="op">@</span>simul_data<span class="op">$</span>true_person,
<span class="dt">decreasing=</span><span class="ot">TRUE</span>,
<span class="dt">index=</span><span class="ot">TRUE</span>)<span class="op">$</span>ix[<span class="dv">1</span>]),
<span class="dt">restrict_ind_low =</span> <span class="kw">as.character</span>(<span class="kw">sort</span>(irt_2pl<span class="op">@</span>simul_data<span class="op">$</span>true_person,
<span class="dt">decreasing=</span><span class="ot">FALSE</span>,
<span class="dt">index=</span><span class="ot">TRUE</span>)<span class="op">$</span>ix[<span class="dv">1</span>]),
<span class="dt">fixtype=</span><span class="st">'vb_partial'</span>,
<span class="dt">ncores=</span><span class="dv">2</span>,
<span class="dt">nchains=</span><span class="dv">2</span>,
<span class="dt">niters =</span> <span class="dv">500</span>)</code></pre></div>
<pre><code>## Chain 1: Gradient evaluation took 0 seconds
## Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0 seconds.
## Chain 1: Iteration: 1 / 250 [ 0%] (Adaptation)
## Chain 1: Iteration: 50 / 250 [ 20%] (Adaptation)
## Chain 1: Iteration: 100 / 250 [ 40%] (Adaptation)
## Chain 1: Iteration: 150 / 250 [ 60%] (Adaptation)
## Chain 1: Iteration: 200 / 250 [ 80%] (Adaptation)
## Chain 1: Success! Found best value [eta = 1] earlier than expected.
## Chain 1: 100 -1057.420 1.000 1.000
## Chain 1: 200 -1045.939 0.505 1.000
## Chain 1: 300 -1044.276 0.338 0.011
## Chain 1: 400 -1041.217 0.254 0.011
## Chain 1: 500 -1042.919 0.203 0.003 MEDIAN ELBO CONVERGED
## Chain 1: Drawing a sample of size 1000 from the approximate posterior...</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">irt_2pl_incorrect <-<span class="st"> </span><span class="kw">id_estimate</span>(<span class="dt">idealdata=</span>irt_2pl,
<span class="dt">model_type=</span><span class="dv">8</span>,
<span class="dt">restrict_ind_high =</span> <span class="kw">as.character</span>(<span class="kw">sort</span>(irt_2pl<span class="op">@</span>simul_data<span class="op">$</span>true_person,
<span class="dt">decreasing=</span><span class="ot">TRUE</span>,
<span class="dt">index=</span><span class="ot">TRUE</span>)<span class="op">$</span>ix[<span class="dv">1</span>]),
<span class="dt">restrict_ind_low =</span> <span class="kw">as.character</span>(<span class="kw">sort</span>(irt_2pl<span class="op">@</span>simul_data<span class="op">$</span>true_person,
<span class="dt">decreasing=</span><span class="ot">FALSE</span>,
<span class="dt">index=</span><span class="ot">TRUE</span>)<span class="op">$</span>ix[<span class="dv">1</span>]),
<span class="dt">fixtype=</span><span class="st">'vb_partial'</span>,
<span class="dt">ncores=</span><span class="dv">2</span>,
<span class="dt">nchains=</span><span class="dv">2</span>,
<span class="dt">niters=</span><span class="dv">500</span>)</code></pre></div>
<pre><code>## Chain 1: Gradient evaluation took 0 seconds
## Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0 seconds.
## Chain 1: Iteration: 1 / 250 [ 0%] (Adaptation)
## Chain 1: Iteration: 50 / 250 [ 20%] (Adaptation)
## Chain 1: Iteration: 100 / 250 [ 40%] (Adaptation)
## Chain 1: Iteration: 150 / 250 [ 60%] (Adaptation)
## Chain 1: Iteration: 200 / 250 [ 80%] (Adaptation)
## Chain 1: Success! Found best value [eta = 1] earlier than expected.
## Chain 1: 100 -4154172.311 1.000 1.000
## Chain 1: 200 -462000.315 4.496 7.992
## Chain 1: 300 -1605796.824 3.235 1.000
## Chain 1: 400 -9960842.290 2.636 1.000
## Chain 1: 500 -3660771.278 2.453 1.000
## Chain 1: 600 -208653028151.058 2.211 1.000
## Chain 1: 700 -871621239.323 35.950 1.000
## Chain 1: 800 -25862319.457 35.544 1.721
## Chain 1: 900 -47171459.169 31.645 1.000
## Chain 1: 1000 -13591951.336 28.727 1.721
## Chain 1: 1100 -1871355.503 29.254 2.471 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 1200 -1104070282.330 28.554 1.721 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 1300 -92502391119.340 28.582 1.721 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 1400 -563994.875 16429.680 2.471 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 1500 -49620069.571 16429.607 2.471 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 1600 -8661.581 17002.282 6.263 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 1700 -1810.391 16978.822 3.784 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 1800 -2552.708 16975.581 2.471 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 1900 -1682.214 16975.588 2.471 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 2000 -1659.393 16975.342 0.998 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 2100 -1588.202 16974.720 0.989 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 2200 -1558.193 16974.622 0.988 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 2300 -1547.898 16974.524 0.517 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 2400 -1546.471 573.342 0.291 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 2500 -1547.006 573.243 0.045 MAY BE DIVERGING... INSPECT ELBO
## Chain 1: 2600 -1548.441 0.468 0.019
## Chain 1: 2700 -1547.860 0.090 0.014
## Chain 1: 2800 -1546.677 0.061 0.007 MEDIAN ELBO CONVERGED
## Chain 1: Drawing a sample of size 1000 from the approximate posterior...</code></pre>
<p>The first thing we want to check with any MCMC model is convergence. An easy way to check is by looking at the Rhat distributions. If all these values are below 1.1, then we have good reason to believe that the model converged, and we can get these distributions with the <code>id_plot_rhats</code> function:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">id_plot_rhats</span>(irt_2pl_correct)</code></pre></div>
<pre><code>## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.</code></pre>
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAkAAAAHgCAMAAAB6sCJ3AAAAt1BMVEUAAAAAADoAAGYAOpAAZrY6AAA6ADo6AGY6kNtNTU1NTW5NTY5NbqtNjshZWVlmAABmtv9uTU1uTW5uTY5ubo5ubqtuq+SOTU2OTW6OTY6Obk2ObquOyP+QOgCQkGaQ27aQ2/+rbk2rbm6rbo6rjk2ryKur5OSr5P+2ZgC22/+2///Ijk3I///bkDrb/7bb///kq27k////tmb/yI7/25D/27b/29v/5Kv//7b//8j//9v//+T///9oZZr7AAAACXBIWXMAAA7DAAAOwwHHb6hkAAAK3UlEQVR4nO3de3vT5gHG4QTCitsg0jJOW9hIdyCDpE0CmwnR9/9csx0HsLEO9hNiqbrvP3CvSJciy7++fiVhs1NCYGfbO0C/CYiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICIfKeAikXf55fQAQIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiNQFdHY1+fFOWl89H++frblZAQ1ET0LvD8v3++dXRYXn2aN3NCmgoqgO6fHly8zB+crLmZgU0FNUBjZ/+c/oWNn56Xl6+eLPmZgU0FDUB/Xw4rWfyLiYgKtWNQLNyjEDUqZkD/XVWjjkQderPwibDz9XRM2dhVKoJ6PL56KcT14Go5Uo0EQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEbmbgJZ9n1/KFgiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiMh2AmryfXaK70BARAREREBEBEREQERuKaAwGAH1loCICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIhIbUBXR4dlefl8tH/etBkBDVVtQGejw1lEZ4+aNiOgoaoLaPznvxyWly9PyvGTk4bNCGioagK6+vXfk9Fn/PS8vHzxpmEzAhqqmoDOnk3fvt7vC4hq1QFNhp4rIxANqgM6G009MweiTuNp/NXRM2dhVHIdiIgr0UQWAvr4+ODj4517b9ffjICGaiGg473y9N7b0731NyOgofo6oMkA9OnVXnmxwRAkoKFaCujj44cCYg1fB/Tp1cOL3dfTN7K1CWioFuZAHx7s7JXH939ffzMCGiqn8UQW38IONt2MgIZqaRK96WYENFQLb2Gnm0x/ZgQ0VIsj0M6M03haM4kmIiAiiwGd7uwcbDQREtBQLd5Mvf/b9e2wtQloqL65F3bgXhhrEBCRpetAv13fkF+bgIZqcRJ9Mb0MtEE/Ahosp/FEVtwLMweiPQER+RLQ6c4N14FozV/nIGISTcS9MCLuhRFxK4OIgIi4F0bEvTAiTuOJCIjI8mfjfayHtSx+tHmDK0DXBDRU7oUR8eUKRBbmQJtcQrwmoKEyiSay9BV3m25GQENlEk3EJJrI4hzoB5No1uMLpoi4F0ZEQERcByKydB1ociK20cm8gIZq+TrQ8cPyYoPP9QhoqJYDOvXPPbGO5X9wblLPJp8sFNBQLQQ0vRl2vLP7ev3NCGiounkaL6je6Oa9MAH1RjfvxguoN7r5r/UIqDe6eTNVQL1hEk1EQES6eTNVQL3RzZupAuqNbt5MFVBvdPNmqoB6o5s3UwXUG928mSqg3nAaT0RARL4K6HijK0DXBDRUXwI6nsydTzctSEBD9Tmg2V8G2vhvBAloqD4HNLsAvfEXvAhoqAREREBEBETkq4A+/5up7oXRmguJRAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARP4QAeltewREREBEagIa/zwaHZbl5fPR/nnTZgQ0VNUBXb54U45/eXN1dFiePWrajICGqjqg99Nq3h1evjwpx09OGjYjoKGqnwNNRqHx0/PZYFTvjgNac3W+n9qAro6ele/3BUS1uoAunz+bTKWNQNSoPQs7nPzZyTmQgDqjOqDrfmZvY507CxNQZ1QHdDaaOuzkdSABdUY/r0QLqDMERERARAREREBEBESkHwGt6XaeE20IiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIjIIANad32qCajF+lQTUIv1qSagFutTTUAt1qeagFqsTzUBtVifagJqsT7VBNRifaoJqMX6VBNQi/WpJqAW61NNQC3Wp5qAWqxPNQG1WJ9qAmqxPtUE1GJ9qgmoxfpU+0MGtKxpb2/nGAyTgFYspz0BrVhOewJasZz2BLRiOe0JaMVy2hPQiuW0J6AVy2lPQCuW056AViynPQGtWE57AlqxnPYEtGJ5+OTDzfWLgFYsD598uLl+EdCK5eGTDzfXLwJasTx88uHm+kVAK5aHTz7cXL8IaMXy8MmHm+uXQQTU5Jaf/K0c0b4QUCGghIAKASUEVAgoIaBCQAkBFQJKCKhofMWbVl9zc38sAioElBBQIaCEgAoBJQRUCCghoEJACQEVAkoIqBBQQkCFgBICKgSUEFAhoISACgElBFTcdUBrrt5xAioElBBQIaCEgAoBJQRUCCghoEJACQEV3z2g7Ld3nIAKASUEVAgoIaBCQAkBFQJKCKgQUEJAxa0HdLu/veOn/QIqBJQQUCGghIAKASUEVAgoIaBCQAkBFd+8JN3am9u+V7vm8qZXXkCFgOqWN73yAioEVLe86ZVvDujy+Wj/vGmlWzx+W9CtJ9N0bNd8KcLlTa98Y0BXR4fl2aOmtW7x+G1Bt55M07Fd86UIlze98o0BXb48KcdPTtbc637p1pNpOrZrvhTh8obf1hzQ+Ol5efniTdNqDFRjQO/3BUQ1IxCRW5oDMVQtzsKetTgLY6hu6ToQQ3VLV6IZKgERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARARHpUUA3H2+cf85o8aET+3I2Go1+uvvPYC7tzPwHd3Rg+hPQ+/lrM/++mcWHTuxL+e7wrndkxc7Mf3BXB6Y3Ab378V/X/6PNP2u9+NCJfbn6dRtfIbC8M/Mf3NWB6U1An0fq+bc9LD50Yl8mbxqj0eFd78ryzsx/cFcHpn8Bzb9vZvGhE/sy/uVNuY1RaHFn5j+4qwPTv4C6OwLNlmxhHmQEamfcmTlQxb7MlmwvoC97MTYHWmV+OObfN7P40Il9mb5rXP1ja6fxXw7F9Ad3dWB6FtDs/62OXAf6dl/ORqMft3AitrQzrgPRIwIiIiAiAiIiICICIiIgIgKq9+nVzszDsvzwp9eLy/77n+3sU6cIqN6nVw+nDx8eHHwT0DdBDZKA6s0DKo/3BLSSgOotBPS3yXvZwXQ4mr6nTf98eP3fB1veyW0SUL15QBeTSD48uP97eXrv7cfHk2Amj9MRaDYKTd/fBktA9W4m0QfzTibF/O/3cvY4C+iHt9vewy0TUL3ZCPThwXQYuh5tpn9cTIranQVUHu/s7G17H7dKQPWu38Iudg6+BPTx8e7rmxFo4uPjnXsDHoYEVG8+BzqeNHIT0MVkKlRe7L7+fBY2mxQNlYDqfb4OtPcloOkA9GD39bSbi+ngM+jzeQHVuzmNP52ct9/MgSbznt2/T+o5nkx/rqdDW97JbRIQEQERERARAREREBEBEREQEQERERARARH5PzAjDrWwmacYAAAAAElFTkSuQmCC" style="display: block; margin: auto;" /></p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">id_plot_rhats</span>(irt_2pl_incorrect)</code></pre></div>
<pre><code>## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.</code></pre>
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAkAAAAHgCAMAAAB6sCJ3AAAAulBMVEUAAAAAADoAAGYAOpAAZrY6AAA6ADo6AGY6kNtNTU1NTW5NTY5NbqtNjshZWVlmAABmtv9uTU1uTW5uTY5ubo5ubqtuq8huq+SOTU2OTW6OTY6Obk2ObquOyP+QOgCQkGaQ27aQ2/+rbk2rbm6rbo6rjk2ryKur5OSr5P+2ZgC22/+2///Ijk3I///bkDrb/7bb///kq27k////tmb/yI7/25D/27b/29v/5Kv//7b//8j//9v//+T////OtEjUAAAACXBIWXMAAA7DAAAOwwHHb6hkAAALgklEQVR4nO3de18T6R3GYdzVrrGo7FrrocVWtgeowi6gLSLz/t9WkxDEZMyEfO4nTpLnuv5wLNLJ5LdfJ3MgcaeBwE7fG8BmExARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQER6SegJzN62QhKEBARAREREJGOgK4OBo8Om+by5WD3rPCjCmhrdAT0fr/5sHt2dbDfnD4u/KgC2hrzA7p8fXyzuHh2XPZRBbQ15gd08fyfo5ewi+dnzeWrw7KPKqCt0RHQz/ujeoavYgJirq490LgceyC6dBwD/XVcjmMgunSfhQ13P1cHL5yFMVdHQJcvB388dh2ITt/nSvRsMALaGgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiKyHgEJamMJiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIjIegYkqI0hICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAinQFdHew3zeXLwe5Z+CgC2lqdAZ0O9scRnT4OH0VAW6sroIs//WW/uXx93Fw8O84eRUBbqyOgq1//Pdz7XDw/ay5fHWaPIqCt1RHQ6YvRy9eHXQEx3/yAhrueK3sgFpgf0Olg5IVjILosPI2/OnjhLIy5XAci4ko0EQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREJGpgD493fv0dOeHd8UfRUBbayqgo/vNyQ/vTu4XfxQBba2vAxrugD6/ud+cl98FCWhrzQT06elDAbGErwP6/Obh+b23oxey0gS0taaOgT4+2LnfHP34e/FHEdDWchpPZPolbG9FjyKgrTVzEL2iRxHQ1pp6CTtZweHPmIC21vQeaGfMaTx35iCaiICITAd0srOzt4oDIQFtrembqT/+dn07rDQBba3WvbA998JYgoCIzFwH+u36hnxpAtpa0wfR56PLQOX7EdD2chpP5Bv3whwDcXcrCigtRkCb4jagk50bBa4DCagWK/pxDgHVYkUH0QKqxYruhQmoFiu6FyagWqzoVoaAaiEgIiu6FyagWqzoXpiAauE0noiAiMy+N77Q23oEVIvptzYX+2loAdXCvTAiK/pwBQHVYuoYqNyPkgmoFg6iicx8xF2p1QqoFg6iiTiIJjJ9DPSTg2iWs6IPmBJQLdwLIyIgIq4DEZm5DjQ8EStyMi+gWsxeBzp62JwXeF+PgGoxG9BJmX/uSUC1mP0H54b1lHhnoYBqMRXQ6GbY0c69t/lqBVQLp/FENvNemL7WxmbejRfQ2ljRv9YjoFps5s1UAa2NzTyIFtDaEBCRzbyZKqC1sRk3U5fsi+9nM26mCmhtbcbNVAGtrc24mSqgtbUZN1MFtLY24zReQGtLQES+CuioyBWgawKqxW1AR8Nj55NSBQmoFl8CGv8wULGfCBJQLb4ENL4AXewDXgRUCwERERARARH5KqAv/2aqe2HcmQuJRAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREpCOgi58Hg/2muXw52D1bdrUCqsX8gC5fHTYXvxxeHew3p4+XXa2AajE/oA+jat7vX74+bi6eHS+5WgHVovsYaLgXunh+Nt4ZLUdAtegM6OrgRfNhV0DM1xXQ5csXw0NpeyA6dJ6F7Q9/dQxEl/kBXfczfhlzFsY88wM6HYzsuw5EF1eiiQiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiGxFQILqj4CICIiIgIgIiIiAiAiISKGA+g1mVpnnxF0IiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIhsZUCzyjxHvkVARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERKSKgGaVec6MCIiIgIgIiIiAiAiISJUBzSozgzoJ6ImAEgJ6IqCEgJ4IKCGgJwJKCOiJgBICeiKghIDayoykEgJqKzOSSgiorcxIKiGgtjIjqYSA2sqMpBICaiszkkoIqK3MSCohoLYyI6mEgNrKjKQSAmorM5JKCKitzEgqIaC2MiOphIDayoykEgJqKzOSSgiorcxIKiGgtjIjqYSA2sqMpBICaiszkkoIqK3MSCohoLYyI6mEgNrKjKQSAmorM5JKCKitzEgqIaC2MiOphIDayoykEgJqS598mZFuCAG1pU++zEg3hIDa0idfZqQbQkBt6ZMvM9INIaC29MmXGemGEFBb+uTLjHRDCGh5i558mZFuCAEtb9GTLzPSDSGg5S168mVGuiEEtLxFT77MSDeEgJa36MmXGemGENDyFj35MiPdEAJa3qInX2akG0JA5S0YTjrbwqsPCai8BcNJZ1t49SEBlbdgOOlsC68+JKDyFgwnnW3h1YcEVN6C4aSzLbz6kIDKWzCcdLaFVx8S0HeXznLB9y+7+nB1Avru0lku+P5lVx+uTkDfXTrLBd+/7OrD1Qnou0tnueD7l119uLrFAV2+HOyeLbtVzJfOcsH3L7v6cHULA7o62G9OHy+7VcyXznLB9y+7+nB1CwO6fH3cXDw7XnKrmG/RxJf9/4erD1e3MKCL52fN5avDRd9GpRYG9GFXQMxnD0Sk0DEQtbrDWdiLO5yFUatC14GoVaEr0dRKQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARO4U0O07ey5+Hvzx+MvidDAYL7+/m7c6TrZserE229PbfGY2Z/KFlYznLgHdfsLL5cvh73bPJovm/X7pzbmjD5P/LpMtm16szfb0Np+ZzZl8YTXjuUtAt+9unrxRfrK4+rWnt8y/f/Sv679iky2bXqzN9vQ1n9nNmXxhNeO5S0C3n68wHdBwlzgY7BfeoLuZDGJ6e3r8HIhvbk9/85nenMkXVjOeuwR0+wkv49euR4eTxcUvh01Pf8smE5ps2fRibbanv/lMb87kC6sZz3J7oNHR85+HI5ksRvp5nd+IPdD4T3qZz5rtgaZfPEf/63bRb0Brcgw0Z3vGf9JnQF8dvPZ6DHT7CS+jgIe/myxG+8Srf/T5H2yyZdOLtdme/uYzvTmTL6xmPEtcBxq/jg7GlxImi9PB4FE/JxrjXfLob9UaXQdqb09v85nZnL6vA8FcAiIiICICIiIgIgIiIiAiAur2+c3O2MOm+fiHt9N/9t//9LNNa0VA3T6/eThafHyw1wqoFVSVBNRtElBzdF9A3ySgblMB/W34WrY32h2NXtNGvz68/v1ezxvZJwF1mwR0Pozk44Mff29Ofnj36ekwmOFytAca74VGr2/VElC3m4PovUknw2L+93szXo4D+uld31vYMwF1G++BPj4Y7Yau9zajX86HRd0bB9Qc7ezc73sbeyWgbtcvYec7e7cBfXp67+3NHmjo09OdHyreDQmo2+QY6GjYyE1A58NDoeb83tsvZ2Hjg6JaCajbl+tA928DGu2AHtx7O+rmfLTzqfp8XkDdbk7jT4bn7TfHQMPjnnt/H9ZzNDz8uT4c6nkj+yQgIgIiIiAiAiIiICICIiIgIgIiIiAiAiLyf5oBtR2hanqmAAAAAElFTkSuQmCC" style="display: block; margin: auto;" /></p>
<p>We can only assume that the first model will converge. If the second model failed the Rhat test, at this point we should go back and see if the model is miss-specified or there is something wrong with the data. But for the sake of illustration, we will look at other diagnostics. We can also examine whether 1) the models are able to replicate the data they were fitted on accurately and 2) overall measures of model fit.</p>
<p>We can first look at how well the model reproduces the data, which is called the posterior predictive distribution. We can obtain these distributions using the <code>id_post_pred</code> function:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">post_correct <-<span class="st"> </span><span class="kw">id_post_pred</span>(irt_2pl_correct)</code></pre></div>
<pre><code>## [1] "Processing posterior replications for 1000 scores using 100 posterior samples out of a total of 500 samples."</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">post_incorrect <-<span class="st"> </span><span class="kw">id_post_pred</span>(irt_2pl_incorrect)</code></pre></div>
<pre><code>## [1] "Processing posterior replications for 1000 scores using 100 posterior samples out of a total of 500 samples."</code></pre>
<p>What we can do is the use a wrapper around the <code>bayesplot</code> package called <code>id_plot_ppc</code> to see how well these models replicate their own data. These plots show the original data as blue bars and the posterior predictive estimate as an uncertainty interval. If the interval overlaps with the observed data, then the model was able to replicate the observed data, which is a basic and important validity test for the model, i.e., could the model have generated the data that was used to fit it?</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">id_plot_ppc</span>(irt_2pl_correct,<span class="dt">ppc_pred=</span>post_correct)</code></pre></div>
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAkAAAAHgCAMAAAB6sCJ3AAAA9lBMVEUAAAAAADoAAGYAOpAAZrYBH0szMzM6AAA6ADo6AGY6OmY6OpA6ZrY6kLw6kNtNTU1NTW5NTY5NbqtNjshmAABmADpmAGZmOpBmZrZmkNtmtv9uTU1uTY5ubqtuq+SOTU2OTW6OTY6ObquOq+SOyP+QOgCQOjqQOmaQZpCQkDqQkJCQkLaQkNuQ2/+rbk2rbo6rjk2rq8ir5P+zzeC2ZgC2Zjq2Zma2tma2///Ijk3Ijo7Iq6vI///R4ezbkDrbkGbbkJDb25Db/9vb///kq27kq47k////tmb/tpD/yI7/yKv/25D/5Kv//7b//8j//9v//+T////fOXdpAAAACXBIWXMAAA7DAAAOwwHHb6hkAAAPQElEQVR4nO2dC3fc1BVGHRKlbyZgSilJhgBOn5AWJ20xNekjg42bsWP9/z/TmZHGj2Qs3blH9/PVuXuvhQhZHLGt2chX0niyUwMY2LltARg3BAQmCAhMEBCYICAwQUBgIiogqoM1BAQmCAhMEBCYICAwQUBggoDABAGBCQICEwQEJggITBAQmCAgMEFAYIKAwAQBgQkCAhMEBCYICEwQEJggIDBBQGCCgPKiqm7bYEsIKC8ICEwQEJggIDBBQGCCgMAEAYEJAgITBAQmCAhMEBCYICAwQUBggoDABAGBCQICEwQEJggITBAQWKiqsRVEQDlRVaMriIAyoqrGVxAB5UNVjbAgAsoHAgITBAQmCAhsjLAfAsqK8fVDQHkxun4IKDPG1g8BZQYBgQkCAhMEBCYICEwQEJggIDBBQGCCgMAEAYEJAgITBAQmCAhMOAvoYK+u5w8f7K+3QUNgwFdA84d79dmT/fmjw2YbNAQWfAX0z7/u1Se79fmzvWYbNAQWXAV0srf4FjbbXXwnmzbb5W/eX0BAyfAU0Pk3yzXQbLoKaLoOqGcIbHgK6H/7BBTFcwNVZZm+hS+2o4WDyYIpa6Ctef59PFVlGM4soHp1Gb+8/np81GzDhorHEpCJLAPiPtDWEFCSoXIgoCRD5UBASYbKgYCSDJUDASUZKgcCSjJUDgSUZKgcCCjJUDkQUJKhciCgJEPlQEBJhsqBgJIMlQMBJRkqBwJKMlQOBJRkqBwIKMlQORBQkqFyIKAkQ+VAQEmGyoGAkgyVAwElGSoHAkoyVA4ElGSoHAgoyVA5EFCSoXIgoCRD5UBASYbKgYCSDJUDASUZKgcCSjJUDgSUZKgcCCjJUDkQUJKhciCgJEPlQEBJhsqBgJIMlQMBJRkqBwJKMlQOBJRkqBwIKMlQORBQkqFyIKAkQ+VAQEmGyoGAkgyVAwElGSoHAkoyVA4ElGSoHAgoyVA5EFCSoXIgoCRD5UBASYbKgYCSDJUDASUZKgcCSjJUDgSUZKgcCCjJUDkQUJKhciCgJEPlQEBJhjKnqgbbFQElGcocAoqDgFoIKA4CaiGgOAiohYDi8BXQ83iqyjD8/NpLR0BJhhQYXrqqGuylI6AkQwoISA4BtRBQHATUQkBxEFALAcVBQC0EFAcBtRBQHATUQkBxEFALAcVBQC0EFAcBtRBQHATUQkBxEFALAcVBQC0EFAcBNVSVqSACSj+kIPqlqypbQQSUfkhB7EtXVcaCCCj9kILIl66qrAURUPohBQQkh4C+JyALBPQ9AVnoamE2ebBf1/OHl9uAoVuFRbScjhbmX9az3frsyf780WGzDRiKIIefyOIyPpbuFk6m9cluff5sr9mGDW1LDgFxIzGWzhbOv6mXJ6H6YNpsl793f4HDgHiUEUlXC2efTab1bLoKaLoOqG9oewhoKHILaHH6+eCQgLZ+6QjogrPPDwtZAxFQJD2L6OYq7PFRsw0b2hYCGorMAppNJoulc/r7QAQ0FJkFNPDQjRDQUBCQFQKSQ0AtBBQHAbUQUBwE1EJAcRBQCwHFQUAtBBQHAbUQUBwE1EJAcRBQCwHFQUAtBBQHAbUQUBwE1EJAcRBQCwHFQUAtBBTH7Qe0/HmIofZFQHJuPaDmJ7IG2hkByRkqoPg/pqvh1v+kLgKKY7CA4r7kfH4qnYDiIKALk/hZArIPEdAgFjYIyHrQ8njp8rAQccsBufxglTwsRNx2QB4/WCUPCxG3HpDDD1bJw0LE7Qfkb/mah4UIAhqEDC1EENAgZGghgoAGIUMLEQQ0CBlaiCCgQcjQIhjbeyEIaBAytAiGgAaxsJGhRTAENIiFjQwtgiGgQSxsZGgRysZ3hL7YufN1Xb+8+6p3/FoLp18tty8/3WqogYByswik2vye4tOP3q/rH3/+be/8hoBOP+4bI6ARWIRx8Xaat37/zdN7ixL+3L+DKy0c77Tc22JoDQHlZhHE5fux3i7oxaKCf/efgK638Pq3Yf9ZAhqBRQhVdWNBL+6+CsqBRfQgZGgRQkdAL+/+d/UN7PTjP975+sXO3VenH/9jZ+f9kBb6z1wENAKLELoCeu9PywzePF1Ec/x+/eJXTxdLm9c/eecC63oLL1droPcIyIFFCB0BHS+v4+vmkmqZxb3VtdXLd05B16/Cfr0a4gzkwSKEroDac80qoE/bX/QF9Ob3Acvut4caCCg3iyBu7OdiAb3s5vheffrJ8uzSnmFubqG5EcQZyINFGBv7Of7Ff75a/3q5fl5s3vv29KOdnTvv9PPWGejpbayBBjxoWGzNpvPP8Z0NF/A33F++3sK/bmMNNOBBw2J7Nj7J2EBQQIEQ0Agsggl7Gr/47rTx0eqGFvpvQBLQCCyCGfDtHMt10oLeh/gENAILERuexvc/QiOgEViI2NDC6W84AzmwELFpDfQzrsIcWIjYtAZ695Fr11BDHgcNCzkb1kBbDjXkcdCwkMN9IK8WIt5qYfngvvc99QQ0BgsR11tYvhG2fsEayIOFiOtroE9Wf+NZmAcLEZsW0X8nIAcWIt59Gr/6iaBthlbkcdCwkHO9hdUbgnqXQAQ0BgsRXMZ7tRBBQF4tRFy0sHyMsVz8vPndu+97vXHokjwOGhbB9P/ZWWFctnDc3kF8/dPegghoBBY9bJQ0BfS3dTevfxk+1O2jIMOXLg+LHgYPqL2JWIf8dBgBjcCih+EDunwSz41EDxY9DB7Qm4sPE3rzB96R6MCih+HXQD+u10DHPEz1YNFDd0Axn5G4foTxovdJBgGNwaKHnjNQ1GckNp9x1/92IAIag0UPPQHFfEZiOAQ0Aose+tZAUZ+RGAoBjcCih96AxvMZiTYyfOnysOihL6D1ZyRGtBA1lMdBwyKY3oCaz0iMaSFqKI+DhkUwfQEdb/gwqcAWoobyOGhYBNMbUMDV+A0tRA3lcdCwCKYnoNDPnCcgtxY9dAV09TMSI1qIGsrjoGERTGdAmz4jMbyFqKE8DhoWwQz/MNU2lMdBwyIYAtr0NWMRjCCg82eTB/t1PX94ub15KI+DhkUww7+p/h1+2K8PPjw6e7I/f3TYbDuG8jhoWMjp/ha2iOZkd3Eq2mu2HUN5HDQs5PQE9PhotlvXB9Nmu/yt+wsIaAQWIroDOpnWs+kqoOk6oBuG8jhoWMjpDOjsiyMCGquFiM6Avlssm1kDjdRCRFdAs716/uXy+uvxUbPtGMrjoGEhpyOgg8lkeSOI+0DjtBDBnWivFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiI6A5rtLjbzhw/219ubh/I4aFjI6QpoNlkEdPZkf/7osNl2DOVx0LCQ03sGOtmtz5/tNduOoTwOGhZyegNa/nUwbbbL37u/gIBGYCGiP6DpKqDpOqAbhvI4aFjIISCvFiJYA3m1ENEb0PL66/FRs+0YyuOgYSGnK6CTyfI6nvtA47QQwZ1orxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiCMirhQgC8mohgoC8WoggIK8WIgjIq4UIAvJqIYKAvFqIICCvFiIIyKuFCALyaiGCgLxaiCAgrxYiQgOaP3yw3zWUx0HDQk5gQGdP9uePDjuG8jhoWMgJDOhktz5/ttcxlMdBw0JOYECz3bo+mC5/dX/BzhWu/UMc7GLYfVzdRcp2VoQGNF0H9Db37Q7sYth9DKERDAHltYtcNIKJWgNdJY8v2M0uctEIZoursMdHaVVgjETdBwJYk36ZDq4hIDBBQGCCgMBEfEAXy+qzzyYfHPb8yzewvMF9bV/xu4i2OH82af/j0RaXu7AcC7PG5S4MGtsSHdDl49Ufoi/PZpPdt/YVvYt4i8XgwYdHJouLXRiOxfzL9v+FeI2LXRg0tiY6oItbi4vaN96iDqH9em++TRm8C4tF3bxgFot2FyaL+mQ1a9JodmHT2I7ogC4fry7OubG67at/ZV+xu7BY1M09UotFvb7NarA4/2b1N4tGuwuTxpbEB3Tl6djZ57Y10M1P2oJ3YbFo/7+1WLS7sFisTxsGjStnnviDsSWDBFR/l0NA0RZnXxxZLdpdWCwWAquFr6nj2cXaOV5jO+xroAXnf7EFNMAayGLRHmuLxeXLFW2xPmuY1kAXJx6DxnaYrsLWj1dPjGsgy6Pay4BiLWZ7y+sXk8V6FwaLepWOTWO9C5vGdhjvA80f7p1MJrv9//ZGVqOLXRhufax3EW9xMJlMHuybLC52YTgWs9WoSeNiFwaNreFONJggIDBBQGCCgMAEAYEJAgITBAQmCAhMEBCYICAw8X/V1M02m5WRBwAAAABJRU5ErkJggg==" style="display: block; margin: auto;" /></p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">id_plot_ppc</span>(irt_2pl_incorrect,<span class="dt">ppc_pred=</span>post_incorrect)</code></pre></div>
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAkAAAAHgCAMAAAB6sCJ3AAABAlBMVEUAAAAAADoAAGYAOpAAZrYBH0szMzM6AAA6ADo6AGY6OmY6OpA6ZrY6kLw6kNtNTU1NTW5NTY5NbqtNjo5NjshmAABmADpmAGZmOpBmZrZmkNtmtv9uTU1uTW5uTY5ubqtuq+SOTU2OTW6OTY6ObquOjk2Oq+SOyP+QOgCQOjqQOmaQZpCQkDqQkJCQkLaQkNuQ2/+rbk2rjk2rq8ir5OSr5P+zzeC2ZgC2Zjq2Zma2tma2///Ijk3Ijo7Iq6vIyP/I///R4ezbkDrbkGbbkJDb25Db/9vb///kq27kq47k////tmb/tpD/yI7/yKv/25D/5Kv//7b//8j//9v//+T///+GLg3uAAAACXBIWXMAAA7DAAAOwwHHb6hkAAALaklEQVR4nO3dDXfT1gHH4VAQe69Zl67ryiBlXWGvbdalbGu6UfbSlDQjCdH3/yqzYgwNsSU5fylxpOc5p4KT4yt0wq/y1ZUjNkoIbFz1AXC9CYiIgIgIiIiAiAiIyIUCUh1zAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIrJGAX3eUi9/OBe0TgF91YqA1oqAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiEhdC3uTOztleXj39bbFoIsT0HVU08Lhx+XeZnn8YOfw3pPZtsWggICuo/oWDrbKg83yZPvhbNtu0EUJ6DqqbeHks7I6CZW7W7Nt9bXbUwJirq6F4w8nW+Xe1mlAW/OAmgZdnICuo/oW9n76REDUqW/h+NdPzIGo0zCJnl2F3X8227YbdFECuo5qWtibTKZTZ+tA1LESTURARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARPoO6POWSgFdT70H1L4KAV1HAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICI1LZxsT+7slOXh3dfb5kFvEtDA1bTwzU65+86z4wc7h/eezLYtBr1JQANX38I0moPN6ano4WzbbtAZAhq4hoDuP9vbLMvdrdm2+tLtKQExV9/CwVa5t3Ua0NY8oOZBZwho4GpbOP7omYCoVdvCl9NpszkQdepa2HtYHn5cXX/dfzbbthn0hvNVFIWABqSmhd3JpFoI6nodSECDcvkr0QIaFAERERARAREREBEBERHQ6BVFMlpAoycgIgIiIiASReXNLz7euPFpWT69+XXj8DMtHH1SbZ9+sNKgegJac0WxsKCjd98uy29/+EXj+AUBHb3XNExAg1EUiwt68ejWtIQ/Ne/gOy3sb7x0a4VBTc5VUR2sgNZFUSwr6PG0gn83n4DOtvD8l+3+2CCg2cEKaD0UxdKCHt/8ulUOlzyJnh+sgNZCTUBPb/739A3s6L0/3Pj08cbNr4/e+/vGxtvndrGohcYz14UDen20AloHdQG99ccqgxePptHsv10+/smj6dTm+ffOXWCdbeHp6RzoLQGNRE1A+9V1fDm7pKqyuHV6bfX03Cno7FXYz08HOQONRV1AL881pwF98PI3TQG9+F2Lafebg+oJaK0t7efVBLrqZv9WefR+dXZ5eYb5rrMtzBaC+jsDmUSvm4X97P/oP5/Mf1/Nn6ebt744endj48a5ft44Az3qeQ7kMn7tLDr/7N9YcAG/ZH35bAv/6nkO9JWFxLWz4P1roVYBteRWxpC0uxs/fXdaeGt1QQvNC5CXFdDSl9KhDj/OUc2Tphpv4guIuQV345tvoQmIuQUtHP3CGYi2Fs2BfrAunwcS0PpbNAc6f8u1blA9AQ3cgjnQioPqCWjg1nodSEDr740Wqhv3jZ+pFxCvnG2h+iBs+dgciNbOzoHeP/2l13thAhqWRZPovwmIts7fjT/9iaBVBtUS0MCdbeH0A0GNUyAB8YrLeCICIvKqheo2RjX5efHb8597XTqomYDW1OcLrb6f1y3sv1xBfP79xoIuKaDln34ltvDzn1FAf5138/zH7Qc1CgKq+fw9sc4DermIWLb56bBL+dd66n4CiFj3Ab2+E9/vQmLLgGp/BpFY5wG9ePUwoRe/7/UTiQJaC93Pgb6dz4H2+72ZKqC1UB/QRZ6ROL+F8bjxToaABqDhDHShZyTOnnHX/HEgk+gBaAjoIs9IbO9y/s1Ul/F9apoDXegZiW1d0j+6ayGxR40BreczElcKyK2MHjUFNH9GYhMBjVRjQLNnJDYS0Eg1BbS/4GFSiwhopBoDanE1XhHQSDUE1PaZ8wIaq7qAvvuMxCYCGqnagBY9I3EJAY1U9zdT+xkkoDUlICICItL9h+r7GSSggRMQEQERERARAREREBEBEREQkbUOaOlLWRsCIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiIiICICIiIgIgIiIiAiAiLSUUBFseSlAho4AREREBEBEREQEQERGUxAS4+AXgmIiICICIiIgIgIiIiAiAiIiICICIiIgIh0E1BRLPv7E9DAdRJQUSwtSEAD10VARbG8IAENXG1Ae5vTzeHdOzvz7cJBRVFTkIAGri6gvck0oOMHO4f3nsy2iwcJaMwaz0AHm+XJ9sPZdvEgAY1ZY0DVf7tbs231tdtTAmKuOaCt04C25gEtGLQOk+jlCwn0qouA1uAyvuYI6FUHc6Dy6hcS686B9KoxoOr66/6z2Xb5oKu9lVE7C6NXdQEdTKrr+MZ1oIqAxmoQN1MFdHUERGQQAZlEX51hBOQy/soMJCALiVdlKAG5lXFFBEREQEQERERARAbznGgBXQ0BEREQEQERERARAREREJHBBMTVEBARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQERERARAREREBEBERljQB6n2CEBEREQEQERERARAREREBEBEREQEQERERCREQZUFArqzvgCKgoFdWh0ARWFgro0toCKQkGdEhARAREREJGxBWQS3bHRBeQyvlvjC8hCYqdGGJBbGV0SEBEBEREQEQERERARAREREJExBkSHBEREQEQERERARAREREBEBEREQEQERERARAREREBEBEREQEQEVMtHh5oIqJaAmgioloCaCKiOn4JuJKAansPQTEDLeRJMCwJayrOo2hDQUgJqQ0BLCagNAS0loDYEtJx+WhBQDf00E1Ad/TQSUC39NBEQEQEREVBHxjpdElA3RnvBJqBOjHfJSEBdGPGitYC6IKDeBglo8atX2XP3L+2QgLqwUkCrnKpWeO0VnQEF1ImV+2l/rupht11q28Lh3Ts7Kw8qRxNQ+///VzlZrfDaK5uFtWzh+MHO4b0nKw6qjCWg1jOQkQZ0sFmebD9ccVBlNAG1NdKA9jbLcner+t3tqY3zFn0t189ex3SwfbZzqm1AW/OAlrjdydFczl4dbIcEtOZ7HUhAZ+dAC/g7uV4H25kVrsLuP+v3ULiOLrQOBHP9T9MZNAERERARARHpJqBeptgn25OeJu67dQsSFzU93B52uzeZ1C2/Xb1OAjp7q7Ur3+yUu+/0sXRweLeHv+njDze73+npd/ZXOz3suDOdBNS4zHhRPWQ59Y8/d3+sJ9u9nCim34BpQ33suSudBPT6VmvHelm8PHjYw1vY4b3fTHo4BZ1sbx6M4C2s8U7ZRfXxzTv5rI850N7P/nmN3hk7tNYBHX/Uwwnofzu9BNTTt+Cbv2z3Mg/szFrPgb7sYwa0O5n0cGXT07v49D28n9llZzq7CuthtrL3sDz8uPO9lr1cxk+vlfqY7VaT6A+HH1A/60DVqaKfhaA+1oEOJn0sA41kHYjxEhARAREREBEBEREQEQERERARAREREJH/A1UpxFfEQr9rAAAAAElFTkSuQmCC" style="display: block; margin: auto;" /></p>
<p>It is stupidly obvious from these plots that one should not use a Poisson distribution for a Bernoulli (binary) outcome as it will predict values as high as 10 (although it still puts the majority of the density on 1 and 2). Only two observed values are showing on the Poisson predictive distribution because the missing data model must be shown separately if the outcome is unbounded. To view the missing data model (i.e., a binary response), simply change the <code>output</code> option in <code>id_post_pred</code> to <code>'missing'</code>. We can also look at particular persons or items to see how well the models predict those persons or items. For example, let’s look at the first two persons in the simulated data for which we incorrectly used the Poisson model by passing their IDs as a character vector to the <code>group</code> option:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">id_plot_ppc</span>(irt_2pl_incorrect,<span class="dt">ppc_pred=</span>post_incorrect,<span class="dt">group=</span><span class="kw">c</span>(<span class="st">'1'</span>,<span class="st">'2'</span>))</code></pre></div>
<p><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAkAAAAHgCAMAAAB6sCJ3AAABRFBMVEUAAAAAADoAAGYAOpAAZrYBH0sZGUgZGXEZSJcZcboaGhozMzM6AAA6ADo6AGY6OmY6OpA6ZrY6kLw6kNtIGRlIGUhIcbpIl91NTU1NTW5NTY5NbqtNjo5NjshmAABmADpmAGZmOpBmZrZmkNtmtv9uTU1uTW5uTY5ubqtuq+RxGUhxSJdxuv+OTU2OTW6OTY6ObquOjk2Oq+SOyP+QOgCQOjqQOmaQZpCQkDqQkJCQkLaQkNuQ2/+X3f+rbk2rbo6rjk2rq8ir5OSr5P+zzeC2ZgC2Zjq2Zma2tma2//+6cXG6///Ijk3Ijo7Iq6vIyP/I///R4ezbkDrbkGbbkJDb25Db/9vb///dl0jdurrd///kq27kq47k////tmb/tpD/unH/yI7/yKv/25D/3Zf/5Kv//7b//7r//8j//9v//+T///9SRIaiAAAACXBIWXMAAA7DAAAOwwHHb6hkAAAPsklEQVR4nO3d/ZsbZRWH8S1tfGl9BzWgi4gUqFpQi69gQbdVkaKlolS2LF3bXbab//93k33LTnLmnG9yJs3MPPd9XZY212w2Z/w0eWYym26MiBJtrPsBULcDEKUCEKUCEKUCEKUCEKUCEKUCEKUCEKUCEKUCEKUCEKUCEKUCEKUCEKUCEKVqH6DH3/3juh/CU+jh5ctfXfdjaKTWAXp4+UsFAHr8vdHj576/7kfRRK0DNPripwUA+u+no9H9r3y67ofRQABaWw8BtJKKAXSfl7CVVAqgL37WhycgAK2tv3+w7kfQSABaU/d/vu5H0EwAWk8Pxwugh30w1DpAj5+7fLkPO9bv/uVxX/5g3Q+jgVoHiLoVgCgVgCgVgCgVgCgVgCgVgCjVgoDwRtUARKkARKkARKkARKkARKkARKkARKkARKkARKkARKkARKkARKkARKkARKkARKkARKkARKkARKkARKkAlOrWfOt+SE85AKW69dFsAGpy894HIAClAhCAUgEIQKkABKBUAAJQKgABKBWAAJQKQABKBSAApQIQgFIBCECpAASgVAACUCoAASgVgACUCkAASgUgAKUCEIBSAQhAqQAEoFQAAlAqAAEoFYAAlApAAEoFIAClAhCAUgEIQKkABKBUAAJQKgABKBWAAJQKQABKBSAApQIQgFIBCECpAASgVAACUCoAASgVgACUCkAASgUgAKUC0KoAFfKPSABoZYDK2LOFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOkEoFSFjOnkitjeHP9ycH34wl1p8/MVsmcLGdPJE7E9nAB6sCVuXqmQPVvImE7hM9D4CeiauPn5CtmzhYzpFL+EjfauHgu6Ms7a3PwBjEL2bCFjOgmARgdvumsgcycWsmcLGdNJATT6EEA1FTKmkwLo8F0A1VTImE4KoN3pKhpA1QoZ08kDtDscH8cf/eJuDqDej+nUwJloAPV+TCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcApSpkTCcAqZ39yMlgMP3pk/6NuWgAUjubaDCYjtS/MRcNQGoAMgOQGoDMAKQGIDMAqQHIDEBqADIDkBqAzACkBiAzAKkByAxAagAyA5AagMwApAYgMwCpAcgMQGoAMgOQGoDMAKQGIDMAqQHIDEBqADIDkBqAzACkBiAzAKkByKxBQOd3bA/3LIDMAKQGIDMAqQHIDEBqADIDkBqAzACkBiAzAKkByAxAagAyA5AagMwApAYgMwCpAcgMQGoAMgOQGoDMAKQGIDMAqQHIDEBqADIDkBqAzACkBiAzAKkByAxAagAyA5AagMxWDahyY6dzAfVnzEUDkBqAzACkBiAzAKkByAxAagAyA5AagMwApAYgMwCpAcgMQGoAMgOQGoDMAKQGIDMAqQHIDEBqADIDkBqAzACkBiAzAKkByAxAagAyA5AagMwApAYgMwCpAcgMQGoAMgOQGoDMAKQGIDMAqQHIDEBqADIDkBqAzACkBiAzAKkByAxAagAyA5AagMwApAYgMwCpAcgMQGoAMgOQGoDMAKTWW0CDQearAaQGIDMAqQHIDEBqfQU0mDR74+2NC++MRvcufhJ+OYDUegpoMDAF7b/47Gj0+TffD7++ImL/7cmv914RNz8JQB0eczCwBT1569LYwx/iOzAA7b9U7w5A/QI0GNQJuj0G9HH8BHRexM7GSZekzc8qDNBkZ/cF0GBQK+j2xU8e/US4i4qI+CsAdLyz+w/o3sX/CC9gwSJ6e3P8y97V57fczYsCdLq3+w/omd8fvYDtv/S7C+/c3rj4yf5Lf9vYeHbuLiwRpy9928MxoIM3tvZeu+ttXhKg6e7uO6CdyXH8ZDE9RrPz7Oj2d94aL20efW3uAKsq4t7RGuiZ07XT5Blod3N0ePOGvflxAOrsmB6gEyqTQ6oJi0tHx1b35p6CqkdhPzpS9/F5QJP/3bk2+dOVcQDqFaD6o7Cz5fARoFdOfhMBevKb6nHbEaBrp4DmNj8JQB0e0/Sz861/v336+4mbnUuj/Zcnzy4nzzDnq4o4PhFUfQYC0HF9XESP7DPROxemh+OT9fP4l2fe339xY+PCnJ+ZZ6C3WAPV1sfD+EnG65dZzfnlqoh/za+BJkdhr39mb35cWYB6dSLxKPHdeAlQtd3h5Die80DH9fKtjEkaoPGrk/nevCHCOx8NoP4BylU9jH/xaA3kXAUCIABVM96N996DBRCAqhki9n/MM5ARgMysNdA3uB7ICEBm1hpo/i1Xe/OTANSnMRfNWAPJm58EoD6NuWhcVK8GILMZEZM37p1r6gEEoJmqIiZXUo9uswayApBZdQ308tF/nBNBAAJQNWsR/delAFXeZOzhngWQ2fy78Uc/UqZtftz5C2UA1PUxF60q4uiCIGcJVA9o5kKrHu5ZAJk1dBg/e6lnD/csgMwApAYgszMRk7cxJoufJ7+ev+51fvPzFyEByB4z97k7q+6W2eL3M31K2Tk5g/jo644gAPUG0EdGKUB/OXXz6Nvx5rOAWEQXD+jkJOJo/qfDrM3nAHEYXzyg6TvxwonEeUCcSCwc0JOzD/N48tv4ikQDEG9llA1o9PnpGmhHeDMVQP0GtMyHbJ6+hXHbeScDQIUAWu5DNo8/4867HAhApQBa7kM29c0B1HNAy3zIphKAigG0zIds6psDqO+AGvmQzfrNAdR7QCcfshkFILXCAO0YHyZlBSC10gC5R+PTAKRWFiBpAT0JQGoFATr/IZtRKwZUfYe103mAzDE7DOiC+vyzakAz13h0OgeQPWZ3AS3SSgHNXmXW6eoB1YwJIGdzCdDcda6drhZQ3ZgAcjYvBJA1Zl8ANX5R/UKbpwC1e8+er8eAmgpAXgAKW8ciujt7VgLU/TEzreMwvjt7VgPU+TEzreNEYnf2rAio62NmWsdbGd3Zsyqgjo+ZCUBeAAoDkBeAwgDkBaAwAHkBKAxAXgAKA5AXgMIA5AWgMAB5ASgMQF4ACgOQF4DCAOQFoDAAeQEoDEBeAAoDkBeAwgDkBaAwAHkBKAxAXgAKA5AXgMIA5AWgMAB5ASgMQF4ACgOQF4DCAOQFoDAAeQEoDEBeAAoDkBeAwgDkBaAwAHkBKAxAXgAKA5AXgMIA5AWgMAB5ASgMQF4ACgOQF4DCAOQFoDAAeQEoDEBeAAoDkBeAwgDkBaAwAHkBKAxAXgAKA5AXgMIA5AWgMAB5ASgMQE6Tf4Tn7A8AMgNQfcf/DNjpnwBkBqDaTv8hwpM/AsgMQHVN/ynU4z8DyAxAdQFICkB1AUgKQHUBSApAtbGIVgJQfRzGCwHIiROJcQDy4q2MsOUAmX81VUCVL253GUAdGjPTUoDsxYEIqPrF7S4BqEtjZooBHVwfvnC3snnN4YkGaOaL293ygDo1ZqYY0IOtmc3rTpBIgGa/uN0tDahbY2YKAY2fgK5VNwcQgKYJa6C9q8eCrozzAE27VXOj8cVntXJXm4DOj9SPMTMpi+iDNytrIAABaJp0FPahtIiWANWuLlu5Z5cG1K0xMymADt+tAqo7xy8Bqju+beWeXR5Qp8bMpADana6ivROJIqCaM2yt3LMJQF0aM1MIaHc43Jzf3N+zPiB7J7Zyz2YAdWjMTA2+Fwagro6ZCUBeAAoDkBeAwgDkBaAwAHkBKAxAXgAKA5AXgMIA5AWgMAB5ASgMQF4ACgOQF4DCAOQFoDAAeQEoDEBeAAoDkBeAwgDkBaAwAHkBKAxAXgAKA5AXgMIA5AWgMAB5ASgMQF4ACgOQF4DCAOQFoDAAeQEoDEBeAAoDkBeAwgDkBaAwAHkBKAxAXgAKA5AXgMIA5AWgMAB5ASisLYDa+ZHKAAprCaCWfig3gMLaAaitH+sOoLBWAGrtPwwAoDAAeQEobB2A5qoDtPa9DaAwAHkBKKwVgOoW0Wvf2wAKawegmsP4te9tAIW1BJB9InHtextAYW0BVHN6erFH13gACgOQF4DCAOQFoDAAeQEoDEBeAAoDkBeAwgDkBaAwAHkBKAxAXgAKA5AXgMIA5AWgMAB5ASgMQF6NA2rhVZfJAOTVNKBWXrebC0BeDQNq6ZXfqQDk1Syg1v7sQCYAeQEobElA5+sxoPMtDGiu1v7sQCYAqQHIDEBqeUB1i+hWjbloAFJrAFBbf/gkE4DUmgC0wA+ftGp2JwCpNQJIH7NVszsBSA1AZgBSA5AZgNQAZAYgNQCZAUgNQGYAUgOQGYDUAGQGIDUAmQFIDUBmAFIDkBmA1ABk1mZANe882tcaL/3FaisDlBpT33BFtRhQzbUz1gVZ2m25H4pYFaDUmPqGq6q9gMyrr6wb1duSPxSxIkCpMfUNV1ZrAQ0Gxo6wblRvy17TvhpAqTH1DVcXgNQAZNYaQLP1FNBsAAIQgJKbrwZQTxfRc6XGZBHtZe4G60b1tjYcxs+XGpPDeC9zN1g3qre14ESiUWpMTiR2ppUB6nYAUgOQGYDUAGQGIDUAmQFIDUBmAFIDkBmA1ABkBiA1AJkBSA1AZgBSA5AZgNQAZAYgNQCZAUgNQGYAUgOQWQxo7+rzW+7mAOrTmIsWAjp4Y2vvtbve5gDq05iLFgLa3Rwd3rzhbQ6gPo25aCGg7c3R6M61ye+ujNs4bfq7oPVt6G/n75WVPainvWFDTJxdFW2wfe0UUKUr6jdY34byHTZxH13YcCUBqJn76MKGK2nBNdBZV9RvsL4N5Tts4j66sOFKko7CXv/saTwU6mILngciqrb6ZTr1OgBRKgBRKgBRquUAqQvrw5tDeQV+Z/5cQd19KltuD4fzZ68WrANjNjJnpqUAVd9gdXqwNbrzA+0cwN5Vbc8eXN+UNhs/xFeTR48dGLOROVMtBajm5KKZ9n/BaPSP96Q7PLyp/X0bf9vxvpU2ra0DYzYyZ6qlAE3fYI0Tz0Lu3tCe2/de++VQ+bt5eHNzN/vU3oExG5kz1XKA7PfHzLTxDv8kLg62f/hP7VVAfQ3wvlcHxmxizlSrBnTwC+lv5v+21D2rfusHf74pLkvy32uNYzYxZ6pVr4E+1JYGd4ZD7WhCfVkZv6TI65e6OjBmI3OmWvooTHvN374x2vuVdqfi4uDVLWnROFlcXk/u2A6M2cicqVZ7HmjyN049QyKeINkdPr3zIx0Ys5vngYhOAxClAhClAhClAhClAhClAhClAhClAhClAhCl+j97Sb7dcj6qpgAAAABJRU5ErkJggg==" style="display: block; margin: auto;" /></p>
<p>Finally, we can turn to summary measures of model fit that also allow us to compare models directly to each other (if they were fit on the same data). To do so, I first employ the <code>log_lik</code> option of the <code>id_post_pred</code> function to generate log-likelihood values for each of these models.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">log_lik_irt_2pl_correct <-<span class="st"> </span><span class="kw">id_post_pred</span>(irt_2pl_correct,<span class="dt">type=</span><span class="st">'log_lik'</span>)</code></pre></div>
<pre><code>## [1] "Processing posterior replications for 1000 scores using 500 posterior samples out of a total of 500 samples."</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">log_lik_irt_2pl_incorrect <-<span class="st"> </span><span class="kw">id_post_pred</span>(irt_2pl_incorrect,<span class="dt">type=</span><span class="st">'log_lik'</span>)</code></pre></div>
<pre><code>## [1] "Processing posterior replications for 1000 scores using 500 posterior samples out of a total of 500 samples."</code></pre>
<p>Note that we must also specify the <code>relative_eff</code> function in order to calculate degrees of freedom. The first argument to <code>relative_eff</code> is the exponentiated log-likelihood matrix we calculated previously. The second argument uses the function <code>derive_chain</code> and also takes the log-likelihood matrix as its argument.</p>
<p>I put in an option for two cores, but a larger number could be used which would improve the speed of the calculations.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">correct_loo <-<span class="st"> </span><span class="kw">loo</span>(log_lik_irt_2pl_correct,
<span class="dt">cores=</span><span class="dv">2</span>,
<span class="dt">r_eff=</span><span class="kw">relative_eff</span>(<span class="kw">exp</span>(log_lik_irt_2pl_correct),
<span class="dt">chain_id=</span><span class="kw">derive_chain</span>(log_lik_irt_2pl_correct)))</code></pre></div>
<pre><code>## Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">incorrect_loo <-<span class="st"> </span><span class="kw">loo</span>(log_lik_irt_2pl_incorrect,
<span class="dt">cores=</span><span class="dv">2</span>,
<span class="dt">r_eff=</span><span class="kw">relative_eff</span>(<span class="kw">exp</span>(log_lik_irt_2pl_incorrect),
<span class="dt">chain_id=</span><span class="kw">derive_chain</span>(log_lik_irt_2pl_incorrect)))</code></pre></div>
<pre><code>## Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">print</span>(correct_loo)</code></pre></div>
<pre><code>##
## Computed from 500 by 1000 log-likelihood matrix
##
## Estimate SE
## elpd_loo -630.2 14.8
## p_loo 105.4 3.8
## looic 1260.4 29.6
## ------
## Monte Carlo SE of elpd_loo is NA.
##
## Pareto k diagnostic values:
## Count Pct. Min. n_eff
## (-Inf, 0.5] (good) 936 93.6% 106
## (0.5, 0.7] (ok) 59 5.9% 73
## (0.7, 1] (bad) 5 0.5% 48
## (1, Inf) (very bad) 0 0.0% <NA>
## See help('pareto-k-diagnostic') for details.</code></pre>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">print</span>(incorrect_loo)</code></pre></div>
<pre><code>##
## Computed from 500 by 1000 log-likelihood matrix
##
## Estimate SE
## elpd_loo -1042.5 12.2
## p_loo 39.3 2.2
## looic 2084.9 24.4
## ------
## Monte Carlo SE of elpd_loo is NA.
##
## Pareto k diagnostic values:
## Count Pct. Min. n_eff
## (-Inf, 0.5] (good) 992 99.2% 172
## (0.5, 0.7] (ok) 7 0.7% 227
## (0.7, 1] (bad) 1 0.1% 379
## (1, Inf) (very bad) 0 0.0% <NA>
## See help('pareto-k-diagnostic') for details.</code></pre>
<p>With this calculation we can examine the models’ <code>loo</code> values, which shows the relative predictive performance of the model to the data. Overall, model performance seems quite good, as the Pareto k values show that there are only a few dozen observations in the dataset that aren’t well predicted. The LOO-IC, or the leave-one-out information criterion (think AIC or BIC), is much lower (i.e. better) for the correct model, as we would expect.</p>
<p>We can also compare the LOOIC of the two models explicitly using a second <code>loo</code> function that will even give us a confidence interval around the difference. If the difference is negative, then the first (correct) model has higher predictive accuracy:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">compare</span>(correct_loo,
incorrect_loo)</code></pre></div>
<pre><code>## elpd_diff se
## -412.3 14.8</code></pre>
<p>The first (correct) model is clearly preferred, as we would certainly hope in this extreme example. In less obvious cases, the <code>elpd</code> values can help arbitrate between model choices when a theoretical reason for choosing a model does not exist.</p>
<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
var script = document.createElement("script");
script.type = "text/javascript";
script.src = "https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML";
document.getElementsByTagName("head")[0].appendChild(script);
})();
</script>
</body>
</html>