diff --git a/01-introduction.Rmd b/01-introduction.Rmd
index 5f97cb7..b8d5a63 100644
--- a/01-introduction.Rmd
+++ b/01-introduction.Rmd
@@ -5,6 +5,7 @@
 Various people have helped in the process of putting together these materials (either knowingly, or unknowingly). Big thanks go to: 
 
 - [Alexandra Chouldechova](https://www.andrew.cmu.edu/user/achoulde/)
+- [Allison Horst](https://www.allisonhorst.com/)
 - [Andrew Heiss](https://www.andrewheiss.com/)
 - [Ben Baumer](https://www.smith.edu/academics/faculty/ben-baumer)
 - [Benoit Monin](https://www.gsb.stanford.edu/faculty-research/faculty/benoit-monin)
diff --git a/02-visualization1.Rmd b/02-visualization1.Rmd
index 7b1b5ac..dfab170 100644
--- a/02-visualization1.Rmd
+++ b/02-visualization1.Rmd
@@ -780,6 +780,6 @@ Here, I've set `color = "black"` within the `geom_smooth()` function, and now on
 sessionInfo()
 ```
 
-```{r, echo=FALSE, fig.cap="Defense at the reproducibility court.", out.width="95%"}
+```{r, echo=FALSE, fig.cap="Defense at the reproducibility court (graphic by [Allison Horst](https://github.com/allisonhorst)).", out.width="95%"}
 include_graphics("figures/reproducibility_court.jpg")
 ```
diff --git a/15-model_comparison.Rmd b/15-model_comparison.Rmd
index 5866ade..b1bca40 100644
--- a/15-model_comparison.Rmd
+++ b/15-model_comparison.Rmd
@@ -36,7 +36,7 @@ In general, we want our models to explain the data we observed, and correctly pr
 
 ### Fitting vs. predicting
 
-Let's illustrate the trad-off between complexity and simplicty for fitting vs. prediction. We generate data from a model of the following form: 
+Let's illustrate the trade-off between complexity and simplicty for fitting vs. prediction. We generate data from a model of the following form: 
 
 $$
 Y_i = \beta_0 + \beta_1 \cdot X_i + \beta_2 + X_i^2 + \epsilon_i
diff --git a/_bookdown_files/psych252_files/figure-html/bad-figure-1.png b/_bookdown_files/psych252_files/figure-html/bad-figure-1.png
index 85fc645..2500dc4 100644
Binary files a/_bookdown_files/psych252_files/figure-html/bad-figure-1.png and b/_bookdown_files/psych252_files/figure-html/bad-figure-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/errorbars-normal-1.png b/_bookdown_files/psych252_files/figure-html/errorbars-normal-1.png
index 5e5cd59..138783c 100644
Binary files a/_bookdown_files/psych252_files/figure-html/errorbars-normal-1.png and b/_bookdown_files/psych252_files/figure-html/errorbars-normal-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/good-figure-1.png b/_bookdown_files/psych252_files/figure-html/good-figure-1.png
index 5b6b72b..e33c2f0 100644
Binary files a/_bookdown_files/psych252_files/figure-html/good-figure-1.png and b/_bookdown_files/psych252_files/figure-html/good-figure-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/stretching-it-1.png b/_bookdown_files/psych252_files/figure-html/stretching-it-1.png
index 8763fa1..e72ef3a 100644
Binary files a/_bookdown_files/psych252_files/figure-html/stretching-it-1.png and b/_bookdown_files/psych252_files/figure-html/stretching-it-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-11-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-11-1.png
index 987bdf4..537f034 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-11-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-11-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-12-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-12-1.png
index 3f5ce47..f41199d 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-12-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-12-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-13-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-13-1.png
index e6ec9cc..f63e7b2 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-13-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-13-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-15-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-15-1.png
index 0d3c0c1..4346651 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-15-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-15-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-16-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-16-1.png
index 910c50e..c6ca65d 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-16-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-16-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-17-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-17-1.png
index c410901..0b8ce69 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-17-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-17-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-20-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-20-1.png
index 9f2bc45..1b47727 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-20-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-20-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-21-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-21-1.png
index 97757d8..948d191 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-21-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-21-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-22-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-22-1.png
index 1985a9a..58e8265 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-22-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-22-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-26-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-26-1.png
index a05755d..eb139cb 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-26-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-26-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-27-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-27-1.png
index 472bd6a..9457613 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-27-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-27-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-28-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-28-1.png
index 4a2b739..ffda002 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-28-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-28-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-29-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-29-1.png
index c486791..9c072c4 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-29-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-29-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-3-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-3-1.png
index 95bac07..507d838 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-3-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-3-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-30-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-30-1.png
index 4a4e3b5..fd32d86 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-30-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-30-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-34-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-34-1.png
index 458ceac..ff3aa65 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-34-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-34-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-35-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-35-1.png
index 38cbe30..905f88c 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-35-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-35-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-36-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-36-1.png
index ffe26e6..b487769 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-36-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-36-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-37-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-37-1.png
index 9637cec..dbf6f2a 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-37-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-37-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-4-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-4-1.png
index bac90d1..d966a09 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-4-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-4-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-6-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-6-1.png
index e7c94ec..59a3cfa 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-6-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-6-1.png differ
diff --git a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-8-1.png b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-8-1.png
index 52d8146..44302c0 100644
Binary files a/_bookdown_files/psych252_files/figure-html/unnamed-chunk-8-1.png and b/_bookdown_files/psych252_files/figure-html/unnamed-chunk-8-1.png differ
diff --git a/docs/404.html b/docs/404.html
index bc1ea30..8e822fb 100644
--- a/docs/404.html
+++ b/docs/404.html
@@ -51,19 +51,6 @@
 <link href="libs/anchor-sections-1.1.0/anchor-sections.css" rel="stylesheet" />
 <link href="libs/anchor-sections-1.1.0/anchor-sections-hash.css" rel="stylesheet" />
 <script src="libs/anchor-sections-1.1.0/anchor-sections.js"></script>
-<script src="libs/htmlwidgets-1.5.4/htmlwidgets.js"></script>
-<link href="libs/datatables-css-0.0.0/datatables-crosstalk.css" rel="stylesheet" />
-<script src="libs/datatables-binding-0.22/datatables.js"></script>
-<link href="libs/dt-core-1.11.3/css/jquery.dataTables.min.css" rel="stylesheet" />
-<link href="libs/dt-core-1.11.3/css/jquery.dataTables.extra.css" rel="stylesheet" />
-<script src="libs/dt-core-1.11.3/js/jquery.dataTables.min.js"></script>
-<link href="libs/crosstalk-1.2.0/css/crosstalk.min.css" rel="stylesheet" />
-<script src="libs/crosstalk-1.2.0/js/crosstalk.min.js"></script>
-<script src="libs/viz-1.8.2/viz.js"></script>
-<link href="libs/DiagrammeR-styles-0.2/styles.css" rel="stylesheet" />
-<script src="libs/grViz-binding-1.0.9/grViz.js"></script>
-<script src="libs/kePrint-0.0.1/kePrint.js"></script>
-<link href="libs/lightable-0.0.1/lightable.css" rel="stylesheet" />
 
 
 <style type="text/css">
@@ -131,28 +118,6 @@
 code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
 </style>
 
-<style type="text/css">
-/* Used with Pandoc 2.11+ new --citeproc when CSL is used */
-div.csl-bib-body { }
-div.csl-entry {
-  clear: both;
-}
-.hanging div.csl-entry {
-  margin-left:2em;
-  text-indent:-2em;
-}
-div.csl-left-margin {
-  min-width:2em;
-  float:left;
-}
-div.csl-right-inline {
-  margin-left:2em;
-  padding-left:1em;
-}
-div.csl-indent {
-  margin-left: 2em;
-}
-</style>
 
 <link rel="stylesheet" href="style.css" type="text/css" />
 </head>
@@ -369,20 +334,23 @@
 <li class="chapter" data-level="7.3.4" data-path="simulation-1.html"><a href="simulation-1.html#cumulative-probability-distribution"><i class="fa fa-check"></i><b>7.3.4</b> Cumulative probability distribution</a></li>
 <li class="chapter" data-level="7.3.5" data-path="simulation-1.html"><a href="simulation-1.html#inverse-cumulative-distribution"><i class="fa fa-check"></i><b>7.3.5</b> Inverse cumulative distribution</a></li>
 <li class="chapter" data-level="7.3.6" data-path="simulation-1.html"><a href="simulation-1.html#computing-probabilities"><i class="fa fa-check"></i><b>7.3.6</b> Computing probabilities</a></li>
-<li class="chapter" data-level="7.3.7" data-path="simulation-1.html"><a href="simulation-1.html#make-the-plot"><i class="fa fa-check"></i><b>7.3.7</b> Make the plot</a></li>
-<li class="chapter" data-level="7.3.8" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solutions"><i class="fa fa-check"></i><b>7.3.8</b> Analytic solutions</a></li>
-<li class="chapter" data-level="7.3.9" data-path="simulation-1.html"><a href="simulation-1.html#sampling-solution"><i class="fa fa-check"></i><b>7.3.9</b> Sampling solution</a></li>
 </ul></li>
-<li class="chapter" data-level="7.4" data-path="simulation-1.html"><a href="simulation-1.html#bayesian-inference-with-the-normal-distribution"><i class="fa fa-check"></i><b>7.4</b> Bayesian inference with the normal distribution</a>
+<li class="chapter" data-level="7.4" data-path="simulation-1.html"><a href="simulation-1.html#breakout-room-exercise"><i class="fa fa-check"></i><b>7.4</b> Breakout room exercise</a>
+<ul>
+<li class="chapter" data-level="7.4.1" data-path="simulation-1.html"><a href="simulation-1.html#make-the-plot"><i class="fa fa-check"></i><b>7.4.1</b> Make the plot</a></li>
+<li class="chapter" data-level="7.4.2" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solutions"><i class="fa fa-check"></i><b>7.4.2</b> Analytic solutions</a></li>
+<li class="chapter" data-level="7.4.3" data-path="simulation-1.html"><a href="simulation-1.html#sampling-solution"><i class="fa fa-check"></i><b>7.4.3</b> Sampling solution</a></li>
+</ul></li>
+<li class="chapter" data-level="7.5" data-path="simulation-1.html"><a href="simulation-1.html#bayesian-inference-with-the-normal-distribution"><i class="fa fa-check"></i><b>7.5</b> Bayesian inference with the normal distribution</a>
 <ul>
-<li class="chapter" data-level="7.4.1" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solution"><i class="fa fa-check"></i><b>7.4.1</b> Analytic solution</a></li>
-<li class="chapter" data-level="7.4.2" data-path="simulation-1.html"><a href="simulation-1.html#solution-via-sampling"><i class="fa fa-check"></i><b>7.4.2</b> Solution via sampling</a></li>
+<li class="chapter" data-level="7.5.1" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solution"><i class="fa fa-check"></i><b>7.5.1</b> Analytic solution</a></li>
+<li class="chapter" data-level="7.5.2" data-path="simulation-1.html"><a href="simulation-1.html#solution-via-sampling"><i class="fa fa-check"></i><b>7.5.2</b> Solution via sampling</a></li>
 </ul></li>
-<li class="chapter" data-level="7.5" data-path="simulation-1.html"><a href="simulation-1.html#additional-resources-5"><i class="fa fa-check"></i><b>7.5</b> Additional resources</a>
+<li class="chapter" data-level="7.6" data-path="simulation-1.html"><a href="simulation-1.html#additional-resources-5"><i class="fa fa-check"></i><b>7.6</b> Additional resources</a>
 <ul>
-<li class="chapter" data-level="7.5.1" data-path="simulation-1.html"><a href="simulation-1.html#datacamp"><i class="fa fa-check"></i><b>7.5.1</b> Datacamp</a></li>
+<li class="chapter" data-level="7.6.1" data-path="simulation-1.html"><a href="simulation-1.html#datacamp"><i class="fa fa-check"></i><b>7.6.1</b> Datacamp</a></li>
 </ul></li>
-<li class="chapter" data-level="7.6" data-path="simulation-1.html"><a href="simulation-1.html#session-info-6"><i class="fa fa-check"></i><b>7.6</b> Session info</a></li>
+<li class="chapter" data-level="7.7" data-path="simulation-1.html"><a href="simulation-1.html#session-info-6"><i class="fa fa-check"></i><b>7.7</b> Session info</a></li>
 </ul></li>
 <li class="chapter" data-level="8" data-path="simulation-2.html"><a href="simulation-2.html"><i class="fa fa-check"></i><b>8</b> Simulation 2</a>
 <ul>
@@ -911,20 +879,6 @@ <h1>Page not found</h1>
 });
 </script>
 
-<!-- dynamically load mathjax for compatibility with self-contained -->
-<script>
-  (function () {
-    var script = document.createElement("script");
-    script.type = "text/javascript";
-    var src = "true";
-    if (src === "" || src === "true") src = "https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-MML-AM_CHTML";
-    if (location.protocol !== "file:")
-      if (/^https?:/.test(src))
-        src = src.replace(/^https?:/, '');
-    script.src = src;
-    document.getElementsByTagName("head")[0].appendChild(script);
-  })();
-</script>
 </body>
 
 </html>
diff --git a/docs/introduction.html b/docs/introduction.html
index f6308aa..9b1526d 100644
--- a/docs/introduction.html
+++ b/docs/introduction.html
@@ -51,19 +51,6 @@
 <link href="libs/anchor-sections-1.1.0/anchor-sections.css" rel="stylesheet" />
 <link href="libs/anchor-sections-1.1.0/anchor-sections-hash.css" rel="stylesheet" />
 <script src="libs/anchor-sections-1.1.0/anchor-sections.js"></script>
-<script src="libs/htmlwidgets-1.5.4/htmlwidgets.js"></script>
-<link href="libs/datatables-css-0.0.0/datatables-crosstalk.css" rel="stylesheet" />
-<script src="libs/datatables-binding-0.22/datatables.js"></script>
-<link href="libs/dt-core-1.11.3/css/jquery.dataTables.min.css" rel="stylesheet" />
-<link href="libs/dt-core-1.11.3/css/jquery.dataTables.extra.css" rel="stylesheet" />
-<script src="libs/dt-core-1.11.3/js/jquery.dataTables.min.js"></script>
-<link href="libs/crosstalk-1.2.0/css/crosstalk.min.css" rel="stylesheet" />
-<script src="libs/crosstalk-1.2.0/js/crosstalk.min.js"></script>
-<script src="libs/viz-1.8.2/viz.js"></script>
-<link href="libs/DiagrammeR-styles-0.2/styles.css" rel="stylesheet" />
-<script src="libs/grViz-binding-1.0.9/grViz.js"></script>
-<script src="libs/kePrint-0.0.1/kePrint.js"></script>
-<link href="libs/lightable-0.0.1/lightable.css" rel="stylesheet" />
 
 
 <style type="text/css">
@@ -131,28 +118,6 @@
 code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
 </style>
 
-<style type="text/css">
-/* Used with Pandoc 2.11+ new --citeproc when CSL is used */
-div.csl-bib-body { }
-div.csl-entry {
-  clear: both;
-}
-.hanging div.csl-entry {
-  margin-left:2em;
-  text-indent:-2em;
-}
-div.csl-left-margin {
-  min-width:2em;
-  float:left;
-}
-div.csl-right-inline {
-  margin-left:2em;
-  padding-left:1em;
-}
-div.csl-indent {
-  margin-left: 2em;
-}
-</style>
 
 <link rel="stylesheet" href="style.css" type="text/css" />
 </head>
@@ -369,20 +334,23 @@
 <li class="chapter" data-level="7.3.4" data-path="simulation-1.html"><a href="simulation-1.html#cumulative-probability-distribution"><i class="fa fa-check"></i><b>7.3.4</b> Cumulative probability distribution</a></li>
 <li class="chapter" data-level="7.3.5" data-path="simulation-1.html"><a href="simulation-1.html#inverse-cumulative-distribution"><i class="fa fa-check"></i><b>7.3.5</b> Inverse cumulative distribution</a></li>
 <li class="chapter" data-level="7.3.6" data-path="simulation-1.html"><a href="simulation-1.html#computing-probabilities"><i class="fa fa-check"></i><b>7.3.6</b> Computing probabilities</a></li>
-<li class="chapter" data-level="7.3.7" data-path="simulation-1.html"><a href="simulation-1.html#make-the-plot"><i class="fa fa-check"></i><b>7.3.7</b> Make the plot</a></li>
-<li class="chapter" data-level="7.3.8" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solutions"><i class="fa fa-check"></i><b>7.3.8</b> Analytic solutions</a></li>
-<li class="chapter" data-level="7.3.9" data-path="simulation-1.html"><a href="simulation-1.html#sampling-solution"><i class="fa fa-check"></i><b>7.3.9</b> Sampling solution</a></li>
 </ul></li>
-<li class="chapter" data-level="7.4" data-path="simulation-1.html"><a href="simulation-1.html#bayesian-inference-with-the-normal-distribution"><i class="fa fa-check"></i><b>7.4</b> Bayesian inference with the normal distribution</a>
+<li class="chapter" data-level="7.4" data-path="simulation-1.html"><a href="simulation-1.html#breakout-room-exercise"><i class="fa fa-check"></i><b>7.4</b> Breakout room exercise</a>
+<ul>
+<li class="chapter" data-level="7.4.1" data-path="simulation-1.html"><a href="simulation-1.html#make-the-plot"><i class="fa fa-check"></i><b>7.4.1</b> Make the plot</a></li>
+<li class="chapter" data-level="7.4.2" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solutions"><i class="fa fa-check"></i><b>7.4.2</b> Analytic solutions</a></li>
+<li class="chapter" data-level="7.4.3" data-path="simulation-1.html"><a href="simulation-1.html#sampling-solution"><i class="fa fa-check"></i><b>7.4.3</b> Sampling solution</a></li>
+</ul></li>
+<li class="chapter" data-level="7.5" data-path="simulation-1.html"><a href="simulation-1.html#bayesian-inference-with-the-normal-distribution"><i class="fa fa-check"></i><b>7.5</b> Bayesian inference with the normal distribution</a>
 <ul>
-<li class="chapter" data-level="7.4.1" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solution"><i class="fa fa-check"></i><b>7.4.1</b> Analytic solution</a></li>
-<li class="chapter" data-level="7.4.2" data-path="simulation-1.html"><a href="simulation-1.html#solution-via-sampling"><i class="fa fa-check"></i><b>7.4.2</b> Solution via sampling</a></li>
+<li class="chapter" data-level="7.5.1" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solution"><i class="fa fa-check"></i><b>7.5.1</b> Analytic solution</a></li>
+<li class="chapter" data-level="7.5.2" data-path="simulation-1.html"><a href="simulation-1.html#solution-via-sampling"><i class="fa fa-check"></i><b>7.5.2</b> Solution via sampling</a></li>
 </ul></li>
-<li class="chapter" data-level="7.5" data-path="simulation-1.html"><a href="simulation-1.html#additional-resources-5"><i class="fa fa-check"></i><b>7.5</b> Additional resources</a>
+<li class="chapter" data-level="7.6" data-path="simulation-1.html"><a href="simulation-1.html#additional-resources-5"><i class="fa fa-check"></i><b>7.6</b> Additional resources</a>
 <ul>
-<li class="chapter" data-level="7.5.1" data-path="simulation-1.html"><a href="simulation-1.html#datacamp"><i class="fa fa-check"></i><b>7.5.1</b> Datacamp</a></li>
+<li class="chapter" data-level="7.6.1" data-path="simulation-1.html"><a href="simulation-1.html#datacamp"><i class="fa fa-check"></i><b>7.6.1</b> Datacamp</a></li>
 </ul></li>
-<li class="chapter" data-level="7.6" data-path="simulation-1.html"><a href="simulation-1.html#session-info-6"><i class="fa fa-check"></i><b>7.6</b> Session info</a></li>
+<li class="chapter" data-level="7.7" data-path="simulation-1.html"><a href="simulation-1.html#session-info-6"><i class="fa fa-check"></i><b>7.7</b> Session info</a></li>
 </ul></li>
 <li class="chapter" data-level="8" data-path="simulation-2.html"><a href="simulation-2.html"><i class="fa fa-check"></i><b>8</b> Simulation 2</a>
 <ul>
@@ -852,6 +820,7 @@ <h2><span class="header-section-number">1.1</span> Thanks<a href="introduction.h
 <p>Various people have helped in the process of putting together these materials (either knowingly, or unknowingly). Big thanks go to:</p>
 <ul>
 <li><a href="https://www.andrew.cmu.edu/user/achoulde/">Alexandra Chouldechova</a></li>
+<li><a href="https://www.allisonhorst.com/">Allison Horst</a></li>
 <li><a href="https://www.andrewheiss.com/">Andrew Heiss</a></li>
 <li><a href="https://www.smith.edu/academics/faculty/ben-baumer">Ben Baumer</a></li>
 <li><a href="https://www.gsb.stanford.edu/faculty-research/faculty/benoit-monin">Benoit Monin</a></li>
@@ -986,10 +955,10 @@ <h2><span class="header-section-number">1.3</span> Session info<a href="introduc
 ## loaded via a namespace (and not attached):
 ##  [1] bookdown_0.26   digest_0.6.29   R6_2.5.1        jsonlite_1.8.0 
 ##  [5] magrittr_2.0.3  evaluate_0.15   stringi_1.7.6   rlang_1.0.2    
-##  [9] cli_3.3.0       jquerylib_0.1.4 bslib_0.3.1     rmarkdown_2.14 
-## [13] tools_4.1.2     stringr_1.4.0   xfun_0.30       yaml_2.3.5     
-## [17] fastmap_1.1.0   compiler_4.1.2  htmltools_0.5.2 knitr_1.39     
-## [21] sass_0.4.1</code></pre>
+##  [9] cli_3.3.0       rstudioapi_0.13 jquerylib_0.1.4 bslib_0.3.1    
+## [13] rmarkdown_2.14  tools_4.1.2     stringr_1.4.0   xfun_0.30      
+## [17] yaml_2.3.5      fastmap_1.1.0   compiler_4.1.2  htmltools_0.5.2
+## [21] knitr_1.39      sass_0.4.1</code></pre>
 
 </div>
 </div>
@@ -1053,20 +1022,6 @@ <h2><span class="header-section-number">1.3</span> Session info<a href="introduc
 });
 </script>
 
-<!-- dynamically load mathjax for compatibility with self-contained -->
-<script>
-  (function () {
-    var script = document.createElement("script");
-    script.type = "text/javascript";
-    var src = "true";
-    if (src === "" || src === "true") src = "https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-MML-AM_CHTML";
-    if (location.protocol !== "file:")
-      if (/^https?:/.test(src))
-        src = src.replace(/^https?:/, '');
-    script.src = src;
-    document.getElementsByTagName("head")[0].appendChild(script);
-  })();
-</script>
 </body>
 
 </html>
diff --git a/docs/model-comparison.html b/docs/model-comparison.html
index fc1b183..732b84d 100644
--- a/docs/model-comparison.html
+++ b/docs/model-comparison.html
@@ -51,19 +51,6 @@
 <link href="libs/anchor-sections-1.1.0/anchor-sections.css" rel="stylesheet" />
 <link href="libs/anchor-sections-1.1.0/anchor-sections-hash.css" rel="stylesheet" />
 <script src="libs/anchor-sections-1.1.0/anchor-sections.js"></script>
-<script src="libs/htmlwidgets-1.5.4/htmlwidgets.js"></script>
-<link href="libs/datatables-css-0.0.0/datatables-crosstalk.css" rel="stylesheet" />
-<script src="libs/datatables-binding-0.22/datatables.js"></script>
-<link href="libs/dt-core-1.11.3/css/jquery.dataTables.min.css" rel="stylesheet" />
-<link href="libs/dt-core-1.11.3/css/jquery.dataTables.extra.css" rel="stylesheet" />
-<script src="libs/dt-core-1.11.3/js/jquery.dataTables.min.js"></script>
-<link href="libs/crosstalk-1.2.0/css/crosstalk.min.css" rel="stylesheet" />
-<script src="libs/crosstalk-1.2.0/js/crosstalk.min.js"></script>
-<script src="libs/viz-1.8.2/viz.js"></script>
-<link href="libs/DiagrammeR-styles-0.2/styles.css" rel="stylesheet" />
-<script src="libs/grViz-binding-1.0.9/grViz.js"></script>
-<script src="libs/kePrint-0.0.1/kePrint.js"></script>
-<link href="libs/lightable-0.0.1/lightable.css" rel="stylesheet" />
 
 
 <style type="text/css">
@@ -131,28 +118,6 @@
 code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
 </style>
 
-<style type="text/css">
-/* Used with Pandoc 2.11+ new --citeproc when CSL is used */
-div.csl-bib-body { }
-div.csl-entry {
-  clear: both;
-}
-.hanging div.csl-entry {
-  margin-left:2em;
-  text-indent:-2em;
-}
-div.csl-left-margin {
-  min-width:2em;
-  float:left;
-}
-div.csl-right-inline {
-  margin-left:2em;
-  padding-left:1em;
-}
-div.csl-indent {
-  margin-left: 2em;
-}
-</style>
 
 <link rel="stylesheet" href="style.css" type="text/css" />
 </head>
@@ -369,20 +334,23 @@
 <li class="chapter" data-level="7.3.4" data-path="simulation-1.html"><a href="simulation-1.html#cumulative-probability-distribution"><i class="fa fa-check"></i><b>7.3.4</b> Cumulative probability distribution</a></li>
 <li class="chapter" data-level="7.3.5" data-path="simulation-1.html"><a href="simulation-1.html#inverse-cumulative-distribution"><i class="fa fa-check"></i><b>7.3.5</b> Inverse cumulative distribution</a></li>
 <li class="chapter" data-level="7.3.6" data-path="simulation-1.html"><a href="simulation-1.html#computing-probabilities"><i class="fa fa-check"></i><b>7.3.6</b> Computing probabilities</a></li>
-<li class="chapter" data-level="7.3.7" data-path="simulation-1.html"><a href="simulation-1.html#make-the-plot"><i class="fa fa-check"></i><b>7.3.7</b> Make the plot</a></li>
-<li class="chapter" data-level="7.3.8" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solutions"><i class="fa fa-check"></i><b>7.3.8</b> Analytic solutions</a></li>
-<li class="chapter" data-level="7.3.9" data-path="simulation-1.html"><a href="simulation-1.html#sampling-solution"><i class="fa fa-check"></i><b>7.3.9</b> Sampling solution</a></li>
 </ul></li>
-<li class="chapter" data-level="7.4" data-path="simulation-1.html"><a href="simulation-1.html#bayesian-inference-with-the-normal-distribution"><i class="fa fa-check"></i><b>7.4</b> Bayesian inference with the normal distribution</a>
+<li class="chapter" data-level="7.4" data-path="simulation-1.html"><a href="simulation-1.html#breakout-room-exercise"><i class="fa fa-check"></i><b>7.4</b> Breakout room exercise</a>
+<ul>
+<li class="chapter" data-level="7.4.1" data-path="simulation-1.html"><a href="simulation-1.html#make-the-plot"><i class="fa fa-check"></i><b>7.4.1</b> Make the plot</a></li>
+<li class="chapter" data-level="7.4.2" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solutions"><i class="fa fa-check"></i><b>7.4.2</b> Analytic solutions</a></li>
+<li class="chapter" data-level="7.4.3" data-path="simulation-1.html"><a href="simulation-1.html#sampling-solution"><i class="fa fa-check"></i><b>7.4.3</b> Sampling solution</a></li>
+</ul></li>
+<li class="chapter" data-level="7.5" data-path="simulation-1.html"><a href="simulation-1.html#bayesian-inference-with-the-normal-distribution"><i class="fa fa-check"></i><b>7.5</b> Bayesian inference with the normal distribution</a>
 <ul>
-<li class="chapter" data-level="7.4.1" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solution"><i class="fa fa-check"></i><b>7.4.1</b> Analytic solution</a></li>
-<li class="chapter" data-level="7.4.2" data-path="simulation-1.html"><a href="simulation-1.html#solution-via-sampling"><i class="fa fa-check"></i><b>7.4.2</b> Solution via sampling</a></li>
+<li class="chapter" data-level="7.5.1" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solution"><i class="fa fa-check"></i><b>7.5.1</b> Analytic solution</a></li>
+<li class="chapter" data-level="7.5.2" data-path="simulation-1.html"><a href="simulation-1.html#solution-via-sampling"><i class="fa fa-check"></i><b>7.5.2</b> Solution via sampling</a></li>
 </ul></li>
-<li class="chapter" data-level="7.5" data-path="simulation-1.html"><a href="simulation-1.html#additional-resources-5"><i class="fa fa-check"></i><b>7.5</b> Additional resources</a>
+<li class="chapter" data-level="7.6" data-path="simulation-1.html"><a href="simulation-1.html#additional-resources-5"><i class="fa fa-check"></i><b>7.6</b> Additional resources</a>
 <ul>
-<li class="chapter" data-level="7.5.1" data-path="simulation-1.html"><a href="simulation-1.html#datacamp"><i class="fa fa-check"></i><b>7.5.1</b> Datacamp</a></li>
+<li class="chapter" data-level="7.6.1" data-path="simulation-1.html"><a href="simulation-1.html#datacamp"><i class="fa fa-check"></i><b>7.6.1</b> Datacamp</a></li>
 </ul></li>
-<li class="chapter" data-level="7.6" data-path="simulation-1.html"><a href="simulation-1.html#session-info-6"><i class="fa fa-check"></i><b>7.6</b> Session info</a></li>
+<li class="chapter" data-level="7.7" data-path="simulation-1.html"><a href="simulation-1.html#session-info-6"><i class="fa fa-check"></i><b>7.7</b> Session info</a></li>
 </ul></li>
 <li class="chapter" data-level="8" data-path="simulation-2.html"><a href="simulation-2.html"><i class="fa fa-check"></i><b>8</b> Simulation 2</a>
 <ul>
@@ -863,25 +831,25 @@ <h2><span class="header-section-number">15.1</span> Learning goals<a href="model
 </div>
 <div id="load-packages-and-set-plotting-theme-8" class="section level2 hasAnchor" number="15.2">
 <h2><span class="header-section-number">15.2</span> Load packages and set plotting theme<a href="model-comparison.html#load-packages-and-set-plotting-theme-8" class="anchor-section" aria-label="Anchor link to header"></a></h2>
-<div class="sourceCode" id="cb805"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb805-1"><a href="model-comparison.html#cb805-1" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;knitr&quot;</span>)      <span class="co"># for knitting RMarkdown </span></span>
-<span id="cb805-2"><a href="model-comparison.html#cb805-2" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;kableExtra&quot;</span>) <span class="co"># for making nice tables</span></span>
-<span id="cb805-3"><a href="model-comparison.html#cb805-3" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;janitor&quot;</span>)    <span class="co"># for cleaning column names</span></span>
-<span id="cb805-4"><a href="model-comparison.html#cb805-4" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;broom&quot;</span>)      <span class="co"># for tidying up linear models </span></span>
-<span id="cb805-5"><a href="model-comparison.html#cb805-5" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;patchwork&quot;</span>)  <span class="co"># for figure panels</span></span>
-<span id="cb805-6"><a href="model-comparison.html#cb805-6" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;modelr&quot;</span>)     <span class="co"># for cross-validation</span></span>
-<span id="cb805-7"><a href="model-comparison.html#cb805-7" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;tidyverse&quot;</span>)  <span class="co"># for wrangling, plotting, etc. </span></span></code></pre></div>
-<div class="sourceCode" id="cb806"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb806-1"><a href="model-comparison.html#cb806-1" aria-hidden="true" tabindex="-1"></a><span class="fu">theme_set</span>(<span class="fu">theme_classic</span>() <span class="sc">+</span> <span class="co">#set the theme </span></span>
-<span id="cb806-2"><a href="model-comparison.html#cb806-2" aria-hidden="true" tabindex="-1"></a>            <span class="fu">theme</span>(<span class="at">text =</span> <span class="fu">element_text</span>(<span class="at">size =</span> <span class="dv">20</span>))) <span class="co">#set the default text size</span></span>
-<span id="cb806-3"><a href="model-comparison.html#cb806-3" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb806-4"><a href="model-comparison.html#cb806-4" aria-hidden="true" tabindex="-1"></a>opts_chunk<span class="sc">$</span><span class="fu">set</span>(<span class="at">comment =</span> <span class="st">&quot;&quot;</span>,</span>
-<span id="cb806-5"><a href="model-comparison.html#cb806-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">fig.show =</span> <span class="st">&quot;hold&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb1"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb1-1"><a href="model-comparison.html#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;knitr&quot;</span>)      <span class="co"># for knitting RMarkdown </span></span>
+<span id="cb1-2"><a href="model-comparison.html#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;kableExtra&quot;</span>) <span class="co"># for making nice tables</span></span>
+<span id="cb1-3"><a href="model-comparison.html#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;janitor&quot;</span>)    <span class="co"># for cleaning column names</span></span>
+<span id="cb1-4"><a href="model-comparison.html#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;broom&quot;</span>)      <span class="co"># for tidying up linear models </span></span>
+<span id="cb1-5"><a href="model-comparison.html#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;patchwork&quot;</span>)  <span class="co"># for figure panels</span></span>
+<span id="cb1-6"><a href="model-comparison.html#cb1-6" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;modelr&quot;</span>)     <span class="co"># for cross-validation</span></span>
+<span id="cb1-7"><a href="model-comparison.html#cb1-7" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;tidyverse&quot;</span>)  <span class="co"># for wrangling, plotting, etc. </span></span></code></pre></div>
+<div class="sourceCode" id="cb2"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb2-1"><a href="model-comparison.html#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="fu">theme_set</span>(<span class="fu">theme_classic</span>() <span class="sc">+</span> <span class="co">#set the theme </span></span>
+<span id="cb2-2"><a href="model-comparison.html#cb2-2" aria-hidden="true" tabindex="-1"></a>            <span class="fu">theme</span>(<span class="at">text =</span> <span class="fu">element_text</span>(<span class="at">size =</span> <span class="dv">20</span>))) <span class="co">#set the default text size</span></span>
+<span id="cb2-3"><a href="model-comparison.html#cb2-3" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb2-4"><a href="model-comparison.html#cb2-4" aria-hidden="true" tabindex="-1"></a>opts_chunk<span class="sc">$</span><span class="fu">set</span>(<span class="at">comment =</span> <span class="st">&quot;&quot;</span>,</span>
+<span id="cb2-5"><a href="model-comparison.html#cb2-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">fig.show =</span> <span class="st">&quot;hold&quot;</span>)</span></code></pre></div>
 </div>
 <div id="model-comparison-1" class="section level2 hasAnchor" number="15.3">
 <h2><span class="header-section-number">15.3</span> Model comparison<a href="model-comparison.html#model-comparison-1" class="anchor-section" aria-label="Anchor link to header"></a></h2>
 <p>In general, we want our models to explain the data we observed, and correctly predict future data. Often, there is a trade-off between how well the model fits the data we have (e.g. how much of the variance it explains), and how well the model will predict future data. If our model is too complex, then it will not only capture the systematicity in the data but also fit to the noise in the data. If our mdoel is too simple, however, it will not capture some of the systematicity that’s actually present in the data. The goal, as always in statistical modeling, is to find a model that finds the sweet spot between simplicity and complexity.</p>
 <div id="fitting-vs.-predicting" class="section level3 hasAnchor" number="15.3.1">
 <h3><span class="header-section-number">15.3.1</span> Fitting vs. predicting<a href="model-comparison.html#fitting-vs.-predicting" class="anchor-section" aria-label="Anchor link to header"></a></h3>
-<p>Let’s illustrate the trad-off between complexity and simplicty for fitting vs. prediction. We generate data from a model of the following form:</p>
+<p>Let’s illustrate the trade-off between complexity and simplicty for fitting vs. prediction. We generate data from a model of the following form:</p>
 <p><span class="math display">\[
 Y_i = \beta_0 + \beta_1 \cdot X_i + \beta_2 + X_i^2 + \epsilon_i
 \]</span>
@@ -890,149 +858,149 @@ <h3><span class="header-section-number">15.3.1</span> Fitting vs. predicting<a
 \epsilon_i \sim \mathcal{N}(\text{mean} = 0, ~\text{sd} = 20)
 \]</span>
 Here, I’ll use the following parameters: <span class="math inline">\(\beta_0 = 10\)</span>, <span class="math inline">\(\beta_1 = 3\)</span>, and <span class="math inline">\(\beta_2 = 2\)</span> to generate the data:</p>
-<div class="sourceCode" id="cb807"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb807-1"><a href="model-comparison.html#cb807-1" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">1</span>)</span>
-<span id="cb807-2"><a href="model-comparison.html#cb807-2" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb807-3"><a href="model-comparison.html#cb807-3" aria-hidden="true" tabindex="-1"></a>n_plots <span class="ot">=</span> <span class="dv">3</span></span>
-<span id="cb807-4"><a href="model-comparison.html#cb807-4" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb807-5"><a href="model-comparison.html#cb807-5" aria-hidden="true" tabindex="-1"></a><span class="co"># sample size </span></span>
-<span id="cb807-6"><a href="model-comparison.html#cb807-6" aria-hidden="true" tabindex="-1"></a>n_samples <span class="ot">=</span> <span class="dv">20</span> </span>
-<span id="cb807-7"><a href="model-comparison.html#cb807-7" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb807-8"><a href="model-comparison.html#cb807-8" aria-hidden="true" tabindex="-1"></a><span class="co"># number of parameters in the polynomial regression</span></span>
-<span id="cb807-9"><a href="model-comparison.html#cb807-9" aria-hidden="true" tabindex="-1"></a>n_parameters <span class="ot">=</span> <span class="fu">c</span>(<span class="dv">1</span><span class="sc">:</span><span class="dv">4</span>, <span class="fu">seq</span>(<span class="dv">7</span>, <span class="dv">19</span>, <span class="at">length.out =</span> <span class="dv">5</span>)) </span>
-<span id="cb807-10"><a href="model-comparison.html#cb807-10" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb807-11"><a href="model-comparison.html#cb807-11" aria-hidden="true" tabindex="-1"></a><span class="co"># generate data </span></span>
-<span id="cb807-12"><a href="model-comparison.html#cb807-12" aria-hidden="true" tabindex="-1"></a>df.data <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">x =</span> <span class="fu">runif</span>(n_samples, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">10</span>), </span>
-<span id="cb807-13"><a href="model-comparison.html#cb807-13" aria-hidden="true" tabindex="-1"></a>                 <span class="at">y =</span> <span class="dv">10</span> <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x<span class="sc">^</span><span class="dv">2</span> <span class="sc">+</span> <span class="fu">rnorm</span>(n_samples, <span class="at">sd =</span> <span class="dv">20</span>))</span>
-<span id="cb807-14"><a href="model-comparison.html#cb807-14" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb807-15"><a href="model-comparison.html#cb807-15" aria-hidden="true" tabindex="-1"></a><span class="co"># plotting function</span></span>
-<span id="cb807-16"><a href="model-comparison.html#cb807-16" aria-hidden="true" tabindex="-1"></a>plot_fit <span class="ot">=</span> <span class="cf">function</span>(i){</span>
-<span id="cb807-17"><a href="model-comparison.html#cb807-17" aria-hidden="true" tabindex="-1"></a>  <span class="co"># calculate RMSE</span></span>
-<span id="cb807-18"><a href="model-comparison.html#cb807-18" aria-hidden="true" tabindex="-1"></a>  rmse <span class="ot">=</span> <span class="fu">lm</span>(<span class="at">formula =</span> y <span class="sc">~</span> <span class="fu">poly</span>(x, <span class="at">degree =</span> i, <span class="at">raw =</span> <span class="cn">TRUE</span>),</span>
-<span id="cb807-19"><a href="model-comparison.html#cb807-19" aria-hidden="true" tabindex="-1"></a>            <span class="at">data =</span> df.data) <span class="sc">%&gt;%</span> </span>
-<span id="cb807-20"><a href="model-comparison.html#cb807-20" aria-hidden="true" tabindex="-1"></a>    <span class="fu">rmse</span>(<span class="at">data =</span> df.data)</span>
-<span id="cb807-21"><a href="model-comparison.html#cb807-21" aria-hidden="true" tabindex="-1"></a>  </span>
-<span id="cb807-22"><a href="model-comparison.html#cb807-22" aria-hidden="true" tabindex="-1"></a>  <span class="co"># make a plot</span></span>
-<span id="cb807-23"><a href="model-comparison.html#cb807-23" aria-hidden="true" tabindex="-1"></a>  <span class="fu">ggplot</span>(<span class="at">data =</span> df.data,</span>
-<span id="cb807-24"><a href="model-comparison.html#cb807-24" aria-hidden="true" tabindex="-1"></a>         <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> x,</span>
-<span id="cb807-25"><a href="model-comparison.html#cb807-25" aria-hidden="true" tabindex="-1"></a>                       <span class="at">y =</span> y)) <span class="sc">+</span></span>
-<span id="cb807-26"><a href="model-comparison.html#cb807-26" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_point</span>(<span class="at">size =</span> <span class="dv">2</span>) <span class="sc">+</span></span>
-<span id="cb807-27"><a href="model-comparison.html#cb807-27" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>, <span class="at">se =</span> F,</span>
-<span id="cb807-28"><a href="model-comparison.html#cb807-28" aria-hidden="true" tabindex="-1"></a>                <span class="at">formula =</span> y <span class="sc">~</span> <span class="fu">poly</span>(x, <span class="at">degree =</span> i, <span class="at">raw =</span> <span class="cn">TRUE</span>)) <span class="sc">+</span></span>
-<span id="cb807-29"><a href="model-comparison.html#cb807-29" aria-hidden="true" tabindex="-1"></a>    <span class="fu">annotate</span>(<span class="at">geom =</span> <span class="st">&quot;text&quot;</span>,</span>
-<span id="cb807-30"><a href="model-comparison.html#cb807-30" aria-hidden="true" tabindex="-1"></a>             <span class="at">x =</span> <span class="cn">Inf</span>,</span>
-<span id="cb807-31"><a href="model-comparison.html#cb807-31" aria-hidden="true" tabindex="-1"></a>             <span class="at">y =</span> <span class="sc">-</span><span class="cn">Inf</span>,</span>
-<span id="cb807-32"><a href="model-comparison.html#cb807-32" aria-hidden="true" tabindex="-1"></a>             <span class="at">label =</span> <span class="fu">str_c</span>(<span class="st">&quot;RMSE = &quot;</span>, <span class="fu">round</span>(rmse, <span class="dv">2</span>)),</span>
-<span id="cb807-33"><a href="model-comparison.html#cb807-33" aria-hidden="true" tabindex="-1"></a>             <span class="at">hjust =</span> <span class="fl">1.1</span>,</span>
-<span id="cb807-34"><a href="model-comparison.html#cb807-34" aria-hidden="true" tabindex="-1"></a>             <span class="at">vjust =</span> <span class="sc">-</span><span class="fl">0.3</span>) <span class="sc">+</span> </span>
-<span id="cb807-35"><a href="model-comparison.html#cb807-35" aria-hidden="true" tabindex="-1"></a>    <span class="fu">theme</span>(<span class="at">axis.ticks =</span> <span class="fu">element_blank</span>(),</span>
-<span id="cb807-36"><a href="model-comparison.html#cb807-36" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.title =</span> <span class="fu">element_blank</span>(),</span>
-<span id="cb807-37"><a href="model-comparison.html#cb807-37" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.text =</span> <span class="fu">element_blank</span>())</span>
-<span id="cb807-38"><a href="model-comparison.html#cb807-38" aria-hidden="true" tabindex="-1"></a>}</span>
-<span id="cb807-39"><a href="model-comparison.html#cb807-39" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb807-40"><a href="model-comparison.html#cb807-40" aria-hidden="true" tabindex="-1"></a><span class="co"># save plots in a list</span></span>
-<span id="cb807-41"><a href="model-comparison.html#cb807-41" aria-hidden="true" tabindex="-1"></a>l.p <span class="ot">=</span> <span class="fu">map</span>(<span class="at">.x =</span> n_parameters,</span>
-<span id="cb807-42"><a href="model-comparison.html#cb807-42" aria-hidden="true" tabindex="-1"></a>          <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">plot_fit</span>(.))</span>
-<span id="cb807-43"><a href="model-comparison.html#cb807-43" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb807-44"><a href="model-comparison.html#cb807-44" aria-hidden="true" tabindex="-1"></a><span class="co"># make figure panel </span></span>
-<span id="cb807-45"><a href="model-comparison.html#cb807-45" aria-hidden="true" tabindex="-1"></a><span class="fu">wrap_plots</span>(<span class="at">plotlist =</span> l.p, <span class="at">ncol =</span> <span class="dv">3</span>)</span></code></pre></div>
-<p><img src="15-model_comparison_files/figure-html/unnamed-chunk-3-1.png" width="672" /></p>
+<div class="sourceCode" id="cb3"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb3-1"><a href="model-comparison.html#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">1</span>)</span>
+<span id="cb3-2"><a href="model-comparison.html#cb3-2" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb3-3"><a href="model-comparison.html#cb3-3" aria-hidden="true" tabindex="-1"></a>n_plots <span class="ot">=</span> <span class="dv">3</span></span>
+<span id="cb3-4"><a href="model-comparison.html#cb3-4" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb3-5"><a href="model-comparison.html#cb3-5" aria-hidden="true" tabindex="-1"></a><span class="co"># sample size </span></span>
+<span id="cb3-6"><a href="model-comparison.html#cb3-6" aria-hidden="true" tabindex="-1"></a>n_samples <span class="ot">=</span> <span class="dv">20</span> </span>
+<span id="cb3-7"><a href="model-comparison.html#cb3-7" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb3-8"><a href="model-comparison.html#cb3-8" aria-hidden="true" tabindex="-1"></a><span class="co"># number of parameters in the polynomial regression</span></span>
+<span id="cb3-9"><a href="model-comparison.html#cb3-9" aria-hidden="true" tabindex="-1"></a>n_parameters <span class="ot">=</span> <span class="fu">c</span>(<span class="dv">1</span><span class="sc">:</span><span class="dv">4</span>, <span class="fu">seq</span>(<span class="dv">7</span>, <span class="dv">19</span>, <span class="at">length.out =</span> <span class="dv">5</span>)) </span>
+<span id="cb3-10"><a href="model-comparison.html#cb3-10" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb3-11"><a href="model-comparison.html#cb3-11" aria-hidden="true" tabindex="-1"></a><span class="co"># generate data </span></span>
+<span id="cb3-12"><a href="model-comparison.html#cb3-12" aria-hidden="true" tabindex="-1"></a>df.data <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">x =</span> <span class="fu">runif</span>(n_samples, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">10</span>), </span>
+<span id="cb3-13"><a href="model-comparison.html#cb3-13" aria-hidden="true" tabindex="-1"></a>                 <span class="at">y =</span> <span class="dv">10</span> <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x<span class="sc">^</span><span class="dv">2</span> <span class="sc">+</span> <span class="fu">rnorm</span>(n_samples, <span class="at">sd =</span> <span class="dv">20</span>))</span>
+<span id="cb3-14"><a href="model-comparison.html#cb3-14" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb3-15"><a href="model-comparison.html#cb3-15" aria-hidden="true" tabindex="-1"></a><span class="co"># plotting function</span></span>
+<span id="cb3-16"><a href="model-comparison.html#cb3-16" aria-hidden="true" tabindex="-1"></a>plot_fit <span class="ot">=</span> <span class="cf">function</span>(i){</span>
+<span id="cb3-17"><a href="model-comparison.html#cb3-17" aria-hidden="true" tabindex="-1"></a>  <span class="co"># calculate RMSE</span></span>
+<span id="cb3-18"><a href="model-comparison.html#cb3-18" aria-hidden="true" tabindex="-1"></a>  rmse <span class="ot">=</span> <span class="fu">lm</span>(<span class="at">formula =</span> y <span class="sc">~</span> <span class="fu">poly</span>(x, <span class="at">degree =</span> i, <span class="at">raw =</span> <span class="cn">TRUE</span>),</span>
+<span id="cb3-19"><a href="model-comparison.html#cb3-19" aria-hidden="true" tabindex="-1"></a>            <span class="at">data =</span> df.data) <span class="sc">%&gt;%</span> </span>
+<span id="cb3-20"><a href="model-comparison.html#cb3-20" aria-hidden="true" tabindex="-1"></a>    <span class="fu">rmse</span>(<span class="at">data =</span> df.data)</span>
+<span id="cb3-21"><a href="model-comparison.html#cb3-21" aria-hidden="true" tabindex="-1"></a>  </span>
+<span id="cb3-22"><a href="model-comparison.html#cb3-22" aria-hidden="true" tabindex="-1"></a>  <span class="co"># make a plot</span></span>
+<span id="cb3-23"><a href="model-comparison.html#cb3-23" aria-hidden="true" tabindex="-1"></a>  <span class="fu">ggplot</span>(<span class="at">data =</span> df.data,</span>
+<span id="cb3-24"><a href="model-comparison.html#cb3-24" aria-hidden="true" tabindex="-1"></a>         <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> x,</span>
+<span id="cb3-25"><a href="model-comparison.html#cb3-25" aria-hidden="true" tabindex="-1"></a>                       <span class="at">y =</span> y)) <span class="sc">+</span></span>
+<span id="cb3-26"><a href="model-comparison.html#cb3-26" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_point</span>(<span class="at">size =</span> <span class="dv">2</span>) <span class="sc">+</span></span>
+<span id="cb3-27"><a href="model-comparison.html#cb3-27" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>, <span class="at">se =</span> F,</span>
+<span id="cb3-28"><a href="model-comparison.html#cb3-28" aria-hidden="true" tabindex="-1"></a>                <span class="at">formula =</span> y <span class="sc">~</span> <span class="fu">poly</span>(x, <span class="at">degree =</span> i, <span class="at">raw =</span> <span class="cn">TRUE</span>)) <span class="sc">+</span></span>
+<span id="cb3-29"><a href="model-comparison.html#cb3-29" aria-hidden="true" tabindex="-1"></a>    <span class="fu">annotate</span>(<span class="at">geom =</span> <span class="st">&quot;text&quot;</span>,</span>
+<span id="cb3-30"><a href="model-comparison.html#cb3-30" aria-hidden="true" tabindex="-1"></a>             <span class="at">x =</span> <span class="cn">Inf</span>,</span>
+<span id="cb3-31"><a href="model-comparison.html#cb3-31" aria-hidden="true" tabindex="-1"></a>             <span class="at">y =</span> <span class="sc">-</span><span class="cn">Inf</span>,</span>
+<span id="cb3-32"><a href="model-comparison.html#cb3-32" aria-hidden="true" tabindex="-1"></a>             <span class="at">label =</span> <span class="fu">str_c</span>(<span class="st">&quot;RMSE = &quot;</span>, <span class="fu">round</span>(rmse, <span class="dv">2</span>)),</span>
+<span id="cb3-33"><a href="model-comparison.html#cb3-33" aria-hidden="true" tabindex="-1"></a>             <span class="at">hjust =</span> <span class="fl">1.1</span>,</span>
+<span id="cb3-34"><a href="model-comparison.html#cb3-34" aria-hidden="true" tabindex="-1"></a>             <span class="at">vjust =</span> <span class="sc">-</span><span class="fl">0.3</span>) <span class="sc">+</span> </span>
+<span id="cb3-35"><a href="model-comparison.html#cb3-35" aria-hidden="true" tabindex="-1"></a>    <span class="fu">theme</span>(<span class="at">axis.ticks =</span> <span class="fu">element_blank</span>(),</span>
+<span id="cb3-36"><a href="model-comparison.html#cb3-36" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.title =</span> <span class="fu">element_blank</span>(),</span>
+<span id="cb3-37"><a href="model-comparison.html#cb3-37" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.text =</span> <span class="fu">element_blank</span>())</span>
+<span id="cb3-38"><a href="model-comparison.html#cb3-38" aria-hidden="true" tabindex="-1"></a>}</span>
+<span id="cb3-39"><a href="model-comparison.html#cb3-39" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb3-40"><a href="model-comparison.html#cb3-40" aria-hidden="true" tabindex="-1"></a><span class="co"># save plots in a list</span></span>
+<span id="cb3-41"><a href="model-comparison.html#cb3-41" aria-hidden="true" tabindex="-1"></a>l.p <span class="ot">=</span> <span class="fu">map</span>(<span class="at">.x =</span> n_parameters,</span>
+<span id="cb3-42"><a href="model-comparison.html#cb3-42" aria-hidden="true" tabindex="-1"></a>          <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">plot_fit</span>(.))</span>
+<span id="cb3-43"><a href="model-comparison.html#cb3-43" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb3-44"><a href="model-comparison.html#cb3-44" aria-hidden="true" tabindex="-1"></a><span class="co"># make figure panel </span></span>
+<span id="cb3-45"><a href="model-comparison.html#cb3-45" aria-hidden="true" tabindex="-1"></a><span class="fu">wrap_plots</span>(<span class="at">plotlist =</span> l.p, <span class="at">ncol =</span> <span class="dv">3</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-3-1.png" width="672" /></p>
 <p>As we can see, RMSE becomes smaller and smaller the more parameters the model has to fit the data. But how does the RMSE look like for new data that is generated from the same underlying ground truth?</p>
-<div class="sourceCode" id="cb808"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb808-1"><a href="model-comparison.html#cb808-1" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">1</span>)</span>
-<span id="cb808-2"><a href="model-comparison.html#cb808-2" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb808-3"><a href="model-comparison.html#cb808-3" aria-hidden="true" tabindex="-1"></a>n_plots <span class="ot">=</span> <span class="dv">3</span></span>
-<span id="cb808-4"><a href="model-comparison.html#cb808-4" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb808-5"><a href="model-comparison.html#cb808-5" aria-hidden="true" tabindex="-1"></a><span class="co"># sample size </span></span>
-<span id="cb808-6"><a href="model-comparison.html#cb808-6" aria-hidden="true" tabindex="-1"></a>n_samples <span class="ot">=</span> <span class="dv">20</span> </span>
-<span id="cb808-7"><a href="model-comparison.html#cb808-7" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb808-8"><a href="model-comparison.html#cb808-8" aria-hidden="true" tabindex="-1"></a><span class="co"># number of parameters in the polynomial regression</span></span>
-<span id="cb808-9"><a href="model-comparison.html#cb808-9" aria-hidden="true" tabindex="-1"></a>n_parameters <span class="ot">=</span> <span class="fu">c</span>(<span class="dv">1</span><span class="sc">:</span><span class="dv">4</span>, <span class="fu">seq</span>(<span class="dv">7</span>, <span class="dv">19</span>, <span class="at">length.out =</span> <span class="dv">5</span>)) </span>
-<span id="cb808-10"><a href="model-comparison.html#cb808-10" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb808-11"><a href="model-comparison.html#cb808-11" aria-hidden="true" tabindex="-1"></a><span class="co"># generate data </span></span>
-<span id="cb808-12"><a href="model-comparison.html#cb808-12" aria-hidden="true" tabindex="-1"></a>df.data <span class="ot">=</span> <span class="fu">tibble</span>(</span>
-<span id="cb808-13"><a href="model-comparison.html#cb808-13" aria-hidden="true" tabindex="-1"></a>  <span class="at">x =</span> <span class="fu">runif</span>(n_samples, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">10</span>), </span>
-<span id="cb808-14"><a href="model-comparison.html#cb808-14" aria-hidden="true" tabindex="-1"></a>  <span class="at">y =</span> <span class="dv">10</span> <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x<span class="sc">^</span><span class="dv">2</span> <span class="sc">+</span> <span class="fu">rnorm</span>(n_samples, <span class="at">sd =</span> <span class="dv">20</span>)</span>
-<span id="cb808-15"><a href="model-comparison.html#cb808-15" aria-hidden="true" tabindex="-1"></a>)</span>
-<span id="cb808-16"><a href="model-comparison.html#cb808-16" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb808-17"><a href="model-comparison.html#cb808-17" aria-hidden="true" tabindex="-1"></a><span class="co"># generate some more data </span></span>
-<span id="cb808-18"><a href="model-comparison.html#cb808-18" aria-hidden="true" tabindex="-1"></a>df.more_data <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">x =</span> <span class="fu">runif</span>(<span class="dv">50</span>, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">10</span>), </span>
-<span id="cb808-19"><a href="model-comparison.html#cb808-19" aria-hidden="true" tabindex="-1"></a>                      <span class="at">y =</span> <span class="dv">10</span> <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x<span class="sc">^</span><span class="dv">2</span> <span class="sc">+</span> <span class="fu">rnorm</span>(<span class="dv">50</span>, <span class="at">sd =</span> <span class="dv">20</span>))</span>
-<span id="cb808-20"><a href="model-comparison.html#cb808-20" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb808-21"><a href="model-comparison.html#cb808-21" aria-hidden="true" tabindex="-1"></a><span class="co"># list for plots </span></span>
-<span id="cb808-22"><a href="model-comparison.html#cb808-22" aria-hidden="true" tabindex="-1"></a>l.p <span class="ot">=</span> <span class="fu">list</span>()</span>
-<span id="cb808-23"><a href="model-comparison.html#cb808-23" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb808-24"><a href="model-comparison.html#cb808-24" aria-hidden="true" tabindex="-1"></a><span class="co"># plotting function</span></span>
-<span id="cb808-25"><a href="model-comparison.html#cb808-25" aria-hidden="true" tabindex="-1"></a>plot_fit <span class="ot">=</span> <span class="cf">function</span>(i){</span>
-<span id="cb808-26"><a href="model-comparison.html#cb808-26" aria-hidden="true" tabindex="-1"></a>  <span class="co"># calculate RMSE for fitted data </span></span>
-<span id="cb808-27"><a href="model-comparison.html#cb808-27" aria-hidden="true" tabindex="-1"></a>  fit <span class="ot">=</span> <span class="fu">lm</span>(<span class="at">formula =</span> y <span class="sc">~</span> <span class="fu">poly</span>(x, <span class="at">degree =</span> i, <span class="at">raw =</span> <span class="cn">TRUE</span>),</span>
-<span id="cb808-28"><a href="model-comparison.html#cb808-28" aria-hidden="true" tabindex="-1"></a>           <span class="at">data =</span> df.data)</span>
-<span id="cb808-29"><a href="model-comparison.html#cb808-29" aria-hidden="true" tabindex="-1"></a>  </span>
-<span id="cb808-30"><a href="model-comparison.html#cb808-30" aria-hidden="true" tabindex="-1"></a>  <span class="co"># calculate RMSE for training data</span></span>
-<span id="cb808-31"><a href="model-comparison.html#cb808-31" aria-hidden="true" tabindex="-1"></a>  rmse <span class="ot">=</span> fit <span class="sc">%&gt;%</span> </span>
-<span id="cb808-32"><a href="model-comparison.html#cb808-32" aria-hidden="true" tabindex="-1"></a>    <span class="fu">rmse</span>(<span class="at">data =</span> df.data)</span>
-<span id="cb808-33"><a href="model-comparison.html#cb808-33" aria-hidden="true" tabindex="-1"></a>  </span>
-<span id="cb808-34"><a href="model-comparison.html#cb808-34" aria-hidden="true" tabindex="-1"></a>  <span class="co"># calculate RMSE for new data </span></span>
-<span id="cb808-35"><a href="model-comparison.html#cb808-35" aria-hidden="true" tabindex="-1"></a>  rmse_new <span class="ot">=</span> fit <span class="sc">%&gt;%</span> </span>
-<span id="cb808-36"><a href="model-comparison.html#cb808-36" aria-hidden="true" tabindex="-1"></a>    <span class="fu">rmse</span>(<span class="at">data =</span> df.more_data)</span>
-<span id="cb808-37"><a href="model-comparison.html#cb808-37" aria-hidden="true" tabindex="-1"></a>  </span>
-<span id="cb808-38"><a href="model-comparison.html#cb808-38" aria-hidden="true" tabindex="-1"></a>  <span class="co"># make a plot</span></span>
-<span id="cb808-39"><a href="model-comparison.html#cb808-39" aria-hidden="true" tabindex="-1"></a>  <span class="fu">ggplot</span>(<span class="at">data =</span> df.data,</span>
-<span id="cb808-40"><a href="model-comparison.html#cb808-40" aria-hidden="true" tabindex="-1"></a>         <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> x,</span>
-<span id="cb808-41"><a href="model-comparison.html#cb808-41" aria-hidden="true" tabindex="-1"></a>                       <span class="at">y =</span> y)) <span class="sc">+</span></span>
-<span id="cb808-42"><a href="model-comparison.html#cb808-42" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_point</span>(<span class="at">size =</span> <span class="dv">2</span>) <span class="sc">+</span></span>
-<span id="cb808-43"><a href="model-comparison.html#cb808-43" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_point</span>(<span class="at">data =</span> df.more_data,</span>
-<span id="cb808-44"><a href="model-comparison.html#cb808-44" aria-hidden="true" tabindex="-1"></a>               <span class="at">size =</span> <span class="dv">2</span>, </span>
-<span id="cb808-45"><a href="model-comparison.html#cb808-45" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;red&quot;</span>) <span class="sc">+</span></span>
-<span id="cb808-46"><a href="model-comparison.html#cb808-46" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>, <span class="at">se =</span> F,</span>
-<span id="cb808-47"><a href="model-comparison.html#cb808-47" aria-hidden="true" tabindex="-1"></a>                <span class="at">formula =</span> y <span class="sc">~</span> <span class="fu">poly</span>(x, <span class="at">degree =</span> i, <span class="at">raw =</span> <span class="cn">TRUE</span>)) <span class="sc">+</span></span>
-<span id="cb808-48"><a href="model-comparison.html#cb808-48" aria-hidden="true" tabindex="-1"></a>    <span class="fu">annotate</span>(<span class="at">geom =</span> <span class="st">&quot;text&quot;</span>,</span>
-<span id="cb808-49"><a href="model-comparison.html#cb808-49" aria-hidden="true" tabindex="-1"></a>             <span class="at">x =</span> <span class="cn">Inf</span>,</span>
-<span id="cb808-50"><a href="model-comparison.html#cb808-50" aria-hidden="true" tabindex="-1"></a>             <span class="at">y =</span> <span class="sc">-</span><span class="cn">Inf</span>,</span>
-<span id="cb808-51"><a href="model-comparison.html#cb808-51" aria-hidden="true" tabindex="-1"></a>             <span class="at">label =</span> <span class="fu">str_c</span>(<span class="st">&quot;RMSE = &quot;</span>, <span class="fu">round</span>(rmse, <span class="dv">2</span>)),</span>
-<span id="cb808-52"><a href="model-comparison.html#cb808-52" aria-hidden="true" tabindex="-1"></a>             <span class="at">hjust =</span> <span class="fl">1.1</span>,</span>
-<span id="cb808-53"><a href="model-comparison.html#cb808-53" aria-hidden="true" tabindex="-1"></a>             <span class="at">vjust =</span> <span class="sc">-</span><span class="fl">0.3</span>) <span class="sc">+</span> </span>
-<span id="cb808-54"><a href="model-comparison.html#cb808-54" aria-hidden="true" tabindex="-1"></a>    <span class="fu">annotate</span>(<span class="at">geom =</span> <span class="st">&quot;text&quot;</span>,</span>
-<span id="cb808-55"><a href="model-comparison.html#cb808-55" aria-hidden="true" tabindex="-1"></a>             <span class="at">x =</span> <span class="cn">Inf</span>,</span>
-<span id="cb808-56"><a href="model-comparison.html#cb808-56" aria-hidden="true" tabindex="-1"></a>             <span class="at">y =</span> <span class="sc">-</span><span class="cn">Inf</span>,</span>
-<span id="cb808-57"><a href="model-comparison.html#cb808-57" aria-hidden="true" tabindex="-1"></a>             <span class="at">label =</span> <span class="fu">str_c</span>(<span class="st">&quot;RMSE = &quot;</span>, <span class="fu">round</span>(rmse_new, <span class="dv">2</span>)),</span>
-<span id="cb808-58"><a href="model-comparison.html#cb808-58" aria-hidden="true" tabindex="-1"></a>             <span class="at">hjust =</span> <span class="fl">1.1</span>,</span>
-<span id="cb808-59"><a href="model-comparison.html#cb808-59" aria-hidden="true" tabindex="-1"></a>             <span class="at">vjust =</span> <span class="sc">-</span><span class="dv">2</span>,</span>
-<span id="cb808-60"><a href="model-comparison.html#cb808-60" aria-hidden="true" tabindex="-1"></a>             <span class="at">color =</span> <span class="st">&quot;red&quot;</span>) <span class="sc">+</span> </span>
-<span id="cb808-61"><a href="model-comparison.html#cb808-61" aria-hidden="true" tabindex="-1"></a>    <span class="fu">theme</span>(<span class="at">axis.ticks =</span> <span class="fu">element_blank</span>(),</span>
-<span id="cb808-62"><a href="model-comparison.html#cb808-62" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.title =</span> <span class="fu">element_blank</span>(),</span>
-<span id="cb808-63"><a href="model-comparison.html#cb808-63" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.text =</span> <span class="fu">element_blank</span>())</span>
-<span id="cb808-64"><a href="model-comparison.html#cb808-64" aria-hidden="true" tabindex="-1"></a>}</span>
-<span id="cb808-65"><a href="model-comparison.html#cb808-65" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb808-66"><a href="model-comparison.html#cb808-66" aria-hidden="true" tabindex="-1"></a><span class="co"># map over the parameters</span></span>
-<span id="cb808-67"><a href="model-comparison.html#cb808-67" aria-hidden="true" tabindex="-1"></a>l.p <span class="ot">=</span> <span class="fu">map</span>(<span class="at">.x =</span> n_parameters,</span>
-<span id="cb808-68"><a href="model-comparison.html#cb808-68" aria-hidden="true" tabindex="-1"></a>          <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">plot_fit</span>(.))</span>
-<span id="cb808-69"><a href="model-comparison.html#cb808-69" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb808-70"><a href="model-comparison.html#cb808-70" aria-hidden="true" tabindex="-1"></a><span class="co"># make figure panel </span></span>
-<span id="cb808-71"><a href="model-comparison.html#cb808-71" aria-hidden="true" tabindex="-1"></a><span class="fu">wrap_plots</span>(<span class="at">plotlist =</span> l.p, <span class="at">ncol =</span> <span class="dv">3</span>)</span></code></pre></div>
-<p><img src="15-model_comparison_files/figure-html/unnamed-chunk-4-1.png" width="672" /></p>
+<div class="sourceCode" id="cb4"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb4-1"><a href="model-comparison.html#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">1</span>)</span>
+<span id="cb4-2"><a href="model-comparison.html#cb4-2" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb4-3"><a href="model-comparison.html#cb4-3" aria-hidden="true" tabindex="-1"></a>n_plots <span class="ot">=</span> <span class="dv">3</span></span>
+<span id="cb4-4"><a href="model-comparison.html#cb4-4" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb4-5"><a href="model-comparison.html#cb4-5" aria-hidden="true" tabindex="-1"></a><span class="co"># sample size </span></span>
+<span id="cb4-6"><a href="model-comparison.html#cb4-6" aria-hidden="true" tabindex="-1"></a>n_samples <span class="ot">=</span> <span class="dv">20</span> </span>
+<span id="cb4-7"><a href="model-comparison.html#cb4-7" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb4-8"><a href="model-comparison.html#cb4-8" aria-hidden="true" tabindex="-1"></a><span class="co"># number of parameters in the polynomial regression</span></span>
+<span id="cb4-9"><a href="model-comparison.html#cb4-9" aria-hidden="true" tabindex="-1"></a>n_parameters <span class="ot">=</span> <span class="fu">c</span>(<span class="dv">1</span><span class="sc">:</span><span class="dv">4</span>, <span class="fu">seq</span>(<span class="dv">7</span>, <span class="dv">19</span>, <span class="at">length.out =</span> <span class="dv">5</span>)) </span>
+<span id="cb4-10"><a href="model-comparison.html#cb4-10" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb4-11"><a href="model-comparison.html#cb4-11" aria-hidden="true" tabindex="-1"></a><span class="co"># generate data </span></span>
+<span id="cb4-12"><a href="model-comparison.html#cb4-12" aria-hidden="true" tabindex="-1"></a>df.data <span class="ot">=</span> <span class="fu">tibble</span>(</span>
+<span id="cb4-13"><a href="model-comparison.html#cb4-13" aria-hidden="true" tabindex="-1"></a>  <span class="at">x =</span> <span class="fu">runif</span>(n_samples, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">10</span>), </span>
+<span id="cb4-14"><a href="model-comparison.html#cb4-14" aria-hidden="true" tabindex="-1"></a>  <span class="at">y =</span> <span class="dv">10</span> <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x<span class="sc">^</span><span class="dv">2</span> <span class="sc">+</span> <span class="fu">rnorm</span>(n_samples, <span class="at">sd =</span> <span class="dv">20</span>)</span>
+<span id="cb4-15"><a href="model-comparison.html#cb4-15" aria-hidden="true" tabindex="-1"></a>)</span>
+<span id="cb4-16"><a href="model-comparison.html#cb4-16" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb4-17"><a href="model-comparison.html#cb4-17" aria-hidden="true" tabindex="-1"></a><span class="co"># generate some more data </span></span>
+<span id="cb4-18"><a href="model-comparison.html#cb4-18" aria-hidden="true" tabindex="-1"></a>df.more_data <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">x =</span> <span class="fu">runif</span>(<span class="dv">50</span>, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">10</span>), </span>
+<span id="cb4-19"><a href="model-comparison.html#cb4-19" aria-hidden="true" tabindex="-1"></a>                      <span class="at">y =</span> <span class="dv">10</span> <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x<span class="sc">^</span><span class="dv">2</span> <span class="sc">+</span> <span class="fu">rnorm</span>(<span class="dv">50</span>, <span class="at">sd =</span> <span class="dv">20</span>))</span>
+<span id="cb4-20"><a href="model-comparison.html#cb4-20" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb4-21"><a href="model-comparison.html#cb4-21" aria-hidden="true" tabindex="-1"></a><span class="co"># list for plots </span></span>
+<span id="cb4-22"><a href="model-comparison.html#cb4-22" aria-hidden="true" tabindex="-1"></a>l.p <span class="ot">=</span> <span class="fu">list</span>()</span>
+<span id="cb4-23"><a href="model-comparison.html#cb4-23" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb4-24"><a href="model-comparison.html#cb4-24" aria-hidden="true" tabindex="-1"></a><span class="co"># plotting function</span></span>
+<span id="cb4-25"><a href="model-comparison.html#cb4-25" aria-hidden="true" tabindex="-1"></a>plot_fit <span class="ot">=</span> <span class="cf">function</span>(i){</span>
+<span id="cb4-26"><a href="model-comparison.html#cb4-26" aria-hidden="true" tabindex="-1"></a>  <span class="co"># calculate RMSE for fitted data </span></span>
+<span id="cb4-27"><a href="model-comparison.html#cb4-27" aria-hidden="true" tabindex="-1"></a>  fit <span class="ot">=</span> <span class="fu">lm</span>(<span class="at">formula =</span> y <span class="sc">~</span> <span class="fu">poly</span>(x, <span class="at">degree =</span> i, <span class="at">raw =</span> <span class="cn">TRUE</span>),</span>
+<span id="cb4-28"><a href="model-comparison.html#cb4-28" aria-hidden="true" tabindex="-1"></a>           <span class="at">data =</span> df.data)</span>
+<span id="cb4-29"><a href="model-comparison.html#cb4-29" aria-hidden="true" tabindex="-1"></a>  </span>
+<span id="cb4-30"><a href="model-comparison.html#cb4-30" aria-hidden="true" tabindex="-1"></a>  <span class="co"># calculate RMSE for training data</span></span>
+<span id="cb4-31"><a href="model-comparison.html#cb4-31" aria-hidden="true" tabindex="-1"></a>  rmse <span class="ot">=</span> fit <span class="sc">%&gt;%</span> </span>
+<span id="cb4-32"><a href="model-comparison.html#cb4-32" aria-hidden="true" tabindex="-1"></a>    <span class="fu">rmse</span>(<span class="at">data =</span> df.data)</span>
+<span id="cb4-33"><a href="model-comparison.html#cb4-33" aria-hidden="true" tabindex="-1"></a>  </span>
+<span id="cb4-34"><a href="model-comparison.html#cb4-34" aria-hidden="true" tabindex="-1"></a>  <span class="co"># calculate RMSE for new data </span></span>
+<span id="cb4-35"><a href="model-comparison.html#cb4-35" aria-hidden="true" tabindex="-1"></a>  rmse_new <span class="ot">=</span> fit <span class="sc">%&gt;%</span> </span>
+<span id="cb4-36"><a href="model-comparison.html#cb4-36" aria-hidden="true" tabindex="-1"></a>    <span class="fu">rmse</span>(<span class="at">data =</span> df.more_data)</span>
+<span id="cb4-37"><a href="model-comparison.html#cb4-37" aria-hidden="true" tabindex="-1"></a>  </span>
+<span id="cb4-38"><a href="model-comparison.html#cb4-38" aria-hidden="true" tabindex="-1"></a>  <span class="co"># make a plot</span></span>
+<span id="cb4-39"><a href="model-comparison.html#cb4-39" aria-hidden="true" tabindex="-1"></a>  <span class="fu">ggplot</span>(<span class="at">data =</span> df.data,</span>
+<span id="cb4-40"><a href="model-comparison.html#cb4-40" aria-hidden="true" tabindex="-1"></a>         <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> x,</span>
+<span id="cb4-41"><a href="model-comparison.html#cb4-41" aria-hidden="true" tabindex="-1"></a>                       <span class="at">y =</span> y)) <span class="sc">+</span></span>
+<span id="cb4-42"><a href="model-comparison.html#cb4-42" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_point</span>(<span class="at">size =</span> <span class="dv">2</span>) <span class="sc">+</span></span>
+<span id="cb4-43"><a href="model-comparison.html#cb4-43" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_point</span>(<span class="at">data =</span> df.more_data,</span>
+<span id="cb4-44"><a href="model-comparison.html#cb4-44" aria-hidden="true" tabindex="-1"></a>               <span class="at">size =</span> <span class="dv">2</span>, </span>
+<span id="cb4-45"><a href="model-comparison.html#cb4-45" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;red&quot;</span>) <span class="sc">+</span></span>
+<span id="cb4-46"><a href="model-comparison.html#cb4-46" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>, <span class="at">se =</span> F,</span>
+<span id="cb4-47"><a href="model-comparison.html#cb4-47" aria-hidden="true" tabindex="-1"></a>                <span class="at">formula =</span> y <span class="sc">~</span> <span class="fu">poly</span>(x, <span class="at">degree =</span> i, <span class="at">raw =</span> <span class="cn">TRUE</span>)) <span class="sc">+</span></span>
+<span id="cb4-48"><a href="model-comparison.html#cb4-48" aria-hidden="true" tabindex="-1"></a>    <span class="fu">annotate</span>(<span class="at">geom =</span> <span class="st">&quot;text&quot;</span>,</span>
+<span id="cb4-49"><a href="model-comparison.html#cb4-49" aria-hidden="true" tabindex="-1"></a>             <span class="at">x =</span> <span class="cn">Inf</span>,</span>
+<span id="cb4-50"><a href="model-comparison.html#cb4-50" aria-hidden="true" tabindex="-1"></a>             <span class="at">y =</span> <span class="sc">-</span><span class="cn">Inf</span>,</span>
+<span id="cb4-51"><a href="model-comparison.html#cb4-51" aria-hidden="true" tabindex="-1"></a>             <span class="at">label =</span> <span class="fu">str_c</span>(<span class="st">&quot;RMSE = &quot;</span>, <span class="fu">round</span>(rmse, <span class="dv">2</span>)),</span>
+<span id="cb4-52"><a href="model-comparison.html#cb4-52" aria-hidden="true" tabindex="-1"></a>             <span class="at">hjust =</span> <span class="fl">1.1</span>,</span>
+<span id="cb4-53"><a href="model-comparison.html#cb4-53" aria-hidden="true" tabindex="-1"></a>             <span class="at">vjust =</span> <span class="sc">-</span><span class="fl">0.3</span>) <span class="sc">+</span> </span>
+<span id="cb4-54"><a href="model-comparison.html#cb4-54" aria-hidden="true" tabindex="-1"></a>    <span class="fu">annotate</span>(<span class="at">geom =</span> <span class="st">&quot;text&quot;</span>,</span>
+<span id="cb4-55"><a href="model-comparison.html#cb4-55" aria-hidden="true" tabindex="-1"></a>             <span class="at">x =</span> <span class="cn">Inf</span>,</span>
+<span id="cb4-56"><a href="model-comparison.html#cb4-56" aria-hidden="true" tabindex="-1"></a>             <span class="at">y =</span> <span class="sc">-</span><span class="cn">Inf</span>,</span>
+<span id="cb4-57"><a href="model-comparison.html#cb4-57" aria-hidden="true" tabindex="-1"></a>             <span class="at">label =</span> <span class="fu">str_c</span>(<span class="st">&quot;RMSE = &quot;</span>, <span class="fu">round</span>(rmse_new, <span class="dv">2</span>)),</span>
+<span id="cb4-58"><a href="model-comparison.html#cb4-58" aria-hidden="true" tabindex="-1"></a>             <span class="at">hjust =</span> <span class="fl">1.1</span>,</span>
+<span id="cb4-59"><a href="model-comparison.html#cb4-59" aria-hidden="true" tabindex="-1"></a>             <span class="at">vjust =</span> <span class="sc">-</span><span class="dv">2</span>,</span>
+<span id="cb4-60"><a href="model-comparison.html#cb4-60" aria-hidden="true" tabindex="-1"></a>             <span class="at">color =</span> <span class="st">&quot;red&quot;</span>) <span class="sc">+</span> </span>
+<span id="cb4-61"><a href="model-comparison.html#cb4-61" aria-hidden="true" tabindex="-1"></a>    <span class="fu">theme</span>(<span class="at">axis.ticks =</span> <span class="fu">element_blank</span>(),</span>
+<span id="cb4-62"><a href="model-comparison.html#cb4-62" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.title =</span> <span class="fu">element_blank</span>(),</span>
+<span id="cb4-63"><a href="model-comparison.html#cb4-63" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.text =</span> <span class="fu">element_blank</span>())</span>
+<span id="cb4-64"><a href="model-comparison.html#cb4-64" aria-hidden="true" tabindex="-1"></a>}</span>
+<span id="cb4-65"><a href="model-comparison.html#cb4-65" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb4-66"><a href="model-comparison.html#cb4-66" aria-hidden="true" tabindex="-1"></a><span class="co"># map over the parameters</span></span>
+<span id="cb4-67"><a href="model-comparison.html#cb4-67" aria-hidden="true" tabindex="-1"></a>l.p <span class="ot">=</span> <span class="fu">map</span>(<span class="at">.x =</span> n_parameters,</span>
+<span id="cb4-68"><a href="model-comparison.html#cb4-68" aria-hidden="true" tabindex="-1"></a>          <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">plot_fit</span>(.))</span>
+<span id="cb4-69"><a href="model-comparison.html#cb4-69" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb4-70"><a href="model-comparison.html#cb4-70" aria-hidden="true" tabindex="-1"></a><span class="co"># make figure panel </span></span>
+<span id="cb4-71"><a href="model-comparison.html#cb4-71" aria-hidden="true" tabindex="-1"></a><span class="fu">wrap_plots</span>(<span class="at">plotlist =</span> l.p, <span class="at">ncol =</span> <span class="dv">3</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-4-1.png" width="672" /></p>
 <p>The RMSE in black shows the root mean squared error for the data that the model was fit on. The RMSE in red shows the RMSE on the new data. As you can see, the complex models do really poorly. They overfit the noise in the original data which leads to make poor predictions for new data. The simplest model (with two parameters) doesn’t do particularly well either since it misses out on the quadratic trend in the data. Both the model with the quadratic term (top middle) and a model that includes a cubic term (top right) provide a good balance – their RMSE on the new data is lowest.</p>
 <p>Let’s generate another data set:</p>
-<div class="sourceCode" id="cb809"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb809-1"><a href="model-comparison.html#cb809-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make example reproducible </span></span>
-<span id="cb809-2"><a href="model-comparison.html#cb809-2" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">1</span>)</span>
-<span id="cb809-3"><a href="model-comparison.html#cb809-3" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb809-4"><a href="model-comparison.html#cb809-4" aria-hidden="true" tabindex="-1"></a><span class="co"># parameters</span></span>
-<span id="cb809-5"><a href="model-comparison.html#cb809-5" aria-hidden="true" tabindex="-1"></a>sample_size <span class="ot">=</span> <span class="dv">100</span></span>
-<span id="cb809-6"><a href="model-comparison.html#cb809-6" aria-hidden="true" tabindex="-1"></a>b0 <span class="ot">=</span> <span class="dv">1</span></span>
-<span id="cb809-7"><a href="model-comparison.html#cb809-7" aria-hidden="true" tabindex="-1"></a>b1 <span class="ot">=</span> <span class="dv">2</span></span>
-<span id="cb809-8"><a href="model-comparison.html#cb809-8" aria-hidden="true" tabindex="-1"></a>b2 <span class="ot">=</span> <span class="dv">3</span></span>
-<span id="cb809-9"><a href="model-comparison.html#cb809-9" aria-hidden="true" tabindex="-1"></a>sd <span class="ot">=</span> <span class="fl">0.5</span></span>
-<span id="cb809-10"><a href="model-comparison.html#cb809-10" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb809-11"><a href="model-comparison.html#cb809-11" aria-hidden="true" tabindex="-1"></a><span class="co"># sample</span></span>
-<span id="cb809-12"><a href="model-comparison.html#cb809-12" aria-hidden="true" tabindex="-1"></a>df.data <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">participant =</span> <span class="dv">1</span><span class="sc">:</span>sample_size,</span>
-<span id="cb809-13"><a href="model-comparison.html#cb809-13" aria-hidden="true" tabindex="-1"></a>                 <span class="at">x =</span> <span class="fu">runif</span>(sample_size, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">1</span>),</span>
-<span id="cb809-14"><a href="model-comparison.html#cb809-14" aria-hidden="true" tabindex="-1"></a>                 <span class="at">y =</span> b0 <span class="sc">+</span> b1<span class="sc">*</span>x <span class="sc">+</span> b2<span class="sc">*</span>x<span class="sc">^</span><span class="dv">2</span> <span class="sc">+</span> <span class="fu">rnorm</span>(sample_size, <span class="at">sd =</span> sd)) </span></code></pre></div>
+<div class="sourceCode" id="cb5"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb5-1"><a href="model-comparison.html#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make example reproducible </span></span>
+<span id="cb5-2"><a href="model-comparison.html#cb5-2" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">1</span>)</span>
+<span id="cb5-3"><a href="model-comparison.html#cb5-3" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb5-4"><a href="model-comparison.html#cb5-4" aria-hidden="true" tabindex="-1"></a><span class="co"># parameters</span></span>
+<span id="cb5-5"><a href="model-comparison.html#cb5-5" aria-hidden="true" tabindex="-1"></a>sample_size <span class="ot">=</span> <span class="dv">100</span></span>
+<span id="cb5-6"><a href="model-comparison.html#cb5-6" aria-hidden="true" tabindex="-1"></a>b0 <span class="ot">=</span> <span class="dv">1</span></span>
+<span id="cb5-7"><a href="model-comparison.html#cb5-7" aria-hidden="true" tabindex="-1"></a>b1 <span class="ot">=</span> <span class="dv">2</span></span>
+<span id="cb5-8"><a href="model-comparison.html#cb5-8" aria-hidden="true" tabindex="-1"></a>b2 <span class="ot">=</span> <span class="dv">3</span></span>
+<span id="cb5-9"><a href="model-comparison.html#cb5-9" aria-hidden="true" tabindex="-1"></a>sd <span class="ot">=</span> <span class="fl">0.5</span></span>
+<span id="cb5-10"><a href="model-comparison.html#cb5-10" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb5-11"><a href="model-comparison.html#cb5-11" aria-hidden="true" tabindex="-1"></a><span class="co"># sample</span></span>
+<span id="cb5-12"><a href="model-comparison.html#cb5-12" aria-hidden="true" tabindex="-1"></a>df.data <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">participant =</span> <span class="dv">1</span><span class="sc">:</span>sample_size,</span>
+<span id="cb5-13"><a href="model-comparison.html#cb5-13" aria-hidden="true" tabindex="-1"></a>                 <span class="at">x =</span> <span class="fu">runif</span>(sample_size, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">1</span>),</span>
+<span id="cb5-14"><a href="model-comparison.html#cb5-14" aria-hidden="true" tabindex="-1"></a>                 <span class="at">y =</span> b0 <span class="sc">+</span> b1<span class="sc">*</span>x <span class="sc">+</span> b2<span class="sc">*</span>x<span class="sc">^</span><span class="dv">2</span> <span class="sc">+</span> <span class="fu">rnorm</span>(sample_size, <span class="at">sd =</span> sd)) </span></code></pre></div>
 <p>And plot it:</p>
-<div class="sourceCode" id="cb810"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb810-1"><a href="model-comparison.html#cb810-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.data,</span>
-<span id="cb810-2"><a href="model-comparison.html#cb810-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> x,</span>
-<span id="cb810-3"><a href="model-comparison.html#cb810-3" aria-hidden="true" tabindex="-1"></a>                    <span class="at">y =</span> y)) <span class="sc">+</span> </span>
-<span id="cb810-4"><a href="model-comparison.html#cb810-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>,</span>
-<span id="cb810-5"><a href="model-comparison.html#cb810-5" aria-hidden="true" tabindex="-1"></a>              <span class="at">formula =</span> y <span class="sc">~</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>)) <span class="sc">+</span></span>
-<span id="cb810-6"><a href="model-comparison.html#cb810-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>()</span></code></pre></div>
-<p><img src="15-model_comparison_files/figure-html/unnamed-chunk-6-1.png" width="672" /></p>
+<div class="sourceCode" id="cb6"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb6-1"><a href="model-comparison.html#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.data,</span>
+<span id="cb6-2"><a href="model-comparison.html#cb6-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> x,</span>
+<span id="cb6-3"><a href="model-comparison.html#cb6-3" aria-hidden="true" tabindex="-1"></a>                    <span class="at">y =</span> y)) <span class="sc">+</span> </span>
+<span id="cb6-4"><a href="model-comparison.html#cb6-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>,</span>
+<span id="cb6-5"><a href="model-comparison.html#cb6-5" aria-hidden="true" tabindex="-1"></a>              <span class="at">formula =</span> y <span class="sc">~</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>)) <span class="sc">+</span></span>
+<span id="cb6-6"><a href="model-comparison.html#cb6-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>()</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-6-1.png" width="672" /></p>
 </div>
 <div id="f-test" class="section level3 hasAnchor" number="15.3.2">
 <h3><span class="header-section-number">15.3.2</span> F-test<a href="model-comparison.html#f-test" class="anchor-section" aria-label="Anchor link to header"></a></h3>
@@ -1040,13 +1008,13 @@ <h3><span class="header-section-number">15.3.2</span> F-test<a href="model-compa
 <p><span class="math display">\[
 \widehat Y_i = b_0 + b_1 \cdot X_i + b_2 \cdot X_i^2
 \]</span></p>
-<div class="sourceCode" id="cb811"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb811-1"><a href="model-comparison.html#cb811-1" aria-hidden="true" tabindex="-1"></a><span class="co"># fit models to the data </span></span>
-<span id="cb811-2"><a href="model-comparison.html#cb811-2" aria-hidden="true" tabindex="-1"></a>fit_simple <span class="ot">=</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x, <span class="at">data =</span> df.data)</span>
-<span id="cb811-3"><a href="model-comparison.html#cb811-3" aria-hidden="true" tabindex="-1"></a>fit_correct <span class="ot">=</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>), <span class="at">data =</span> df.data)</span>
-<span id="cb811-4"><a href="model-comparison.html#cb811-4" aria-hidden="true" tabindex="-1"></a>fit_complex <span class="ot">=</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>) <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">3</span>), <span class="at">data =</span> df.data)</span>
-<span id="cb811-5"><a href="model-comparison.html#cb811-5" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb811-6"><a href="model-comparison.html#cb811-6" aria-hidden="true" tabindex="-1"></a><span class="co"># compare the models using an F-test </span></span>
-<span id="cb811-7"><a href="model-comparison.html#cb811-7" aria-hidden="true" tabindex="-1"></a><span class="fu">anova</span>(fit_simple, fit_correct)</span></code></pre></div>
+<div class="sourceCode" id="cb7"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb7-1"><a href="model-comparison.html#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="co"># fit models to the data </span></span>
+<span id="cb7-2"><a href="model-comparison.html#cb7-2" aria-hidden="true" tabindex="-1"></a>fit_simple <span class="ot">=</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x, <span class="at">data =</span> df.data)</span>
+<span id="cb7-3"><a href="model-comparison.html#cb7-3" aria-hidden="true" tabindex="-1"></a>fit_correct <span class="ot">=</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>), <span class="at">data =</span> df.data)</span>
+<span id="cb7-4"><a href="model-comparison.html#cb7-4" aria-hidden="true" tabindex="-1"></a>fit_complex <span class="ot">=</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>) <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">3</span>), <span class="at">data =</span> df.data)</span>
+<span id="cb7-5"><a href="model-comparison.html#cb7-5" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb7-6"><a href="model-comparison.html#cb7-6" aria-hidden="true" tabindex="-1"></a><span class="co"># compare the models using an F-test </span></span>
+<span id="cb7-7"><a href="model-comparison.html#cb7-7" aria-hidden="true" tabindex="-1"></a><span class="fu">anova</span>(fit_simple, fit_correct)</span></code></pre></div>
 <pre><code>Analysis of Variance Table
 
 Model 1: y ~ 1 + x
@@ -1056,7 +1024,7 @@ <h3><span class="header-section-number">15.3.2</span> F-test<a href="model-compa
 2     97 21.693  1    3.6039 16.115 0.0001175 ***
 ---
 Signif. codes:  0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1</code></pre>
-<div class="sourceCode" id="cb813"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb813-1"><a href="model-comparison.html#cb813-1" aria-hidden="true" tabindex="-1"></a><span class="fu">anova</span>(fit_correct, fit_complex)</span></code></pre></div>
+<div class="sourceCode" id="cb9"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb9-1"><a href="model-comparison.html#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="fu">anova</span>(fit_correct, fit_complex)</span></code></pre></div>
 <pre><code>Analysis of Variance Table
 
 Model 1: y ~ 1 + x + I(x^2)
@@ -1085,92 +1053,92 @@ <h3><span class="header-section-number">15.3.3</span> Cross-validation<a href="m
 <div id="leave-one-out-cross-validation" class="section level4 hasAnchor" number="15.3.3.1">
 <h4><span class="header-section-number">15.3.3.1</span> Leave-one-out cross-validation<a href="model-comparison.html#leave-one-out-cross-validation" class="anchor-section" aria-label="Anchor link to header"></a></h4>
 <p>I’ve used code similar to this one to illustrate how LOO works in class. Here is a simple data set with 9 data points. We fit 9 models, where for each model, the training set includes one of the data points, and then we look at how well the model captures the held-out data point. We can then characterize the model’s performance by calculating the mean squared error across the 9 runs.</p>
-<div class="sourceCode" id="cb815"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb815-1"><a href="model-comparison.html#cb815-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make example reproducible </span></span>
-<span id="cb815-2"><a href="model-comparison.html#cb815-2" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">1</span>)</span>
-<span id="cb815-3"><a href="model-comparison.html#cb815-3" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb815-4"><a href="model-comparison.html#cb815-4" aria-hidden="true" tabindex="-1"></a><span class="co"># sample</span></span>
-<span id="cb815-5"><a href="model-comparison.html#cb815-5" aria-hidden="true" tabindex="-1"></a>df.loo <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">x =</span> <span class="dv">1</span><span class="sc">:</span><span class="dv">9</span>,</span>
-<span id="cb815-6"><a href="model-comparison.html#cb815-6" aria-hidden="true" tabindex="-1"></a>                <span class="at">y =</span> <span class="fu">c</span>(<span class="dv">5</span>, <span class="dv">2</span>, <span class="dv">4</span>, <span class="dv">10</span>, <span class="dv">3</span>, <span class="dv">4</span>, <span class="dv">10</span>, <span class="dv">2</span>, <span class="dv">8</span>)) </span>
-<span id="cb815-7"><a href="model-comparison.html#cb815-7" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb815-8"><a href="model-comparison.html#cb815-8" aria-hidden="true" tabindex="-1"></a>df.loo_cross <span class="ot">=</span> df.loo <span class="sc">%&gt;%</span> </span>
-<span id="cb815-9"><a href="model-comparison.html#cb815-9" aria-hidden="true" tabindex="-1"></a>  <span class="fu">crossv_loo</span>() <span class="sc">%&gt;%</span> </span>
-<span id="cb815-10"><a href="model-comparison.html#cb815-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">fit =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
-<span id="cb815-11"><a href="model-comparison.html#cb815-11" aria-hidden="true" tabindex="-1"></a>                   <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> x, <span class="at">data =</span> .)),</span>
-<span id="cb815-12"><a href="model-comparison.html#cb815-12" aria-hidden="true" tabindex="-1"></a>         <span class="at">tidy =</span> <span class="fu">map</span>(<span class="at">.x =</span> fit,</span>
-<span id="cb815-13"><a href="model-comparison.html#cb815-13" aria-hidden="true" tabindex="-1"></a>                    <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">tidy</span>(.))) <span class="sc">%&gt;%</span> </span>
-<span id="cb815-14"><a href="model-comparison.html#cb815-14" aria-hidden="true" tabindex="-1"></a>  <span class="fu">unnest</span>(tidy)</span>
-<span id="cb815-15"><a href="model-comparison.html#cb815-15" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb815-16"><a href="model-comparison.html#cb815-16" aria-hidden="true" tabindex="-1"></a><span class="co"># original plot </span></span>
-<span id="cb815-17"><a href="model-comparison.html#cb815-17" aria-hidden="true" tabindex="-1"></a>df.plot <span class="ot">=</span> df.loo <span class="sc">%&gt;%</span> </span>
-<span id="cb815-18"><a href="model-comparison.html#cb815-18" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">color =</span> <span class="dv">1</span>)</span>
-<span id="cb815-19"><a href="model-comparison.html#cb815-19" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb815-20"><a href="model-comparison.html#cb815-20" aria-hidden="true" tabindex="-1"></a><span class="co"># fit to all data except one </span></span>
-<span id="cb815-21"><a href="model-comparison.html#cb815-21" aria-hidden="true" tabindex="-1"></a>fun.cv_plot <span class="ot">=</span> <span class="cf">function</span>(data_point){</span>
-<span id="cb815-22"><a href="model-comparison.html#cb815-22" aria-hidden="true" tabindex="-1"></a>  </span>
-<span id="cb815-23"><a href="model-comparison.html#cb815-23" aria-hidden="true" tabindex="-1"></a>  <span class="co"># determine which point to leave out </span></span>
-<span id="cb815-24"><a href="model-comparison.html#cb815-24" aria-hidden="true" tabindex="-1"></a>  df.plot <span class="ot">=</span> df.plot <span class="sc">%&gt;%</span> </span>
-<span id="cb815-25"><a href="model-comparison.html#cb815-25" aria-hidden="true" tabindex="-1"></a>    <span class="fu">mutate</span>(<span class="at">color =</span> <span class="fu">ifelse</span>(<span class="fu">row_number</span>() <span class="sc">==</span> data_point, <span class="dv">2</span>, color))</span>
-<span id="cb815-26"><a href="model-comparison.html#cb815-26" aria-hidden="true" tabindex="-1"></a>  </span>
-<span id="cb815-27"><a href="model-comparison.html#cb815-27" aria-hidden="true" tabindex="-1"></a>  <span class="co"># fit </span></span>
-<span id="cb815-28"><a href="model-comparison.html#cb815-28" aria-hidden="true" tabindex="-1"></a>  df.fit <span class="ot">=</span> df.plot <span class="sc">%&gt;%</span> </span>
-<span id="cb815-29"><a href="model-comparison.html#cb815-29" aria-hidden="true" tabindex="-1"></a>    <span class="fu">filter</span>(color <span class="sc">!=</span> <span class="dv">2</span>) <span class="sc">%&gt;%</span> </span>
-<span id="cb815-30"><a href="model-comparison.html#cb815-30" aria-hidden="true" tabindex="-1"></a>    <span class="fu">lm</span>(<span class="at">formula =</span> y <span class="sc">~</span> x, <span class="at">data =</span> .) <span class="sc">%&gt;%</span> </span>
-<span id="cb815-31"><a href="model-comparison.html#cb815-31" aria-hidden="true" tabindex="-1"></a>    <span class="fu">augment</span>(<span class="at">newdata =</span> df.plot <span class="sc">%&gt;%</span> </span>
-<span id="cb815-32"><a href="model-comparison.html#cb815-32" aria-hidden="true" tabindex="-1"></a>              <span class="fu">filter</span>(color <span class="sc">==</span> <span class="dv">2</span>)) <span class="sc">%&gt;%</span> </span>
-<span id="cb815-33"><a href="model-comparison.html#cb815-33" aria-hidden="true" tabindex="-1"></a>    <span class="fu">clean_names</span>()</span>
-<span id="cb815-34"><a href="model-comparison.html#cb815-34" aria-hidden="true" tabindex="-1"></a>  </span>
-<span id="cb815-35"><a href="model-comparison.html#cb815-35" aria-hidden="true" tabindex="-1"></a>  p <span class="ot">=</span> <span class="fu">ggplot</span>(df.plot,</span>
-<span id="cb815-36"><a href="model-comparison.html#cb815-36" aria-hidden="true" tabindex="-1"></a>             <span class="fu">aes</span>(x, y, <span class="at">color =</span> <span class="fu">as.factor</span>(color))) <span class="sc">+</span> </span>
-<span id="cb815-37"><a href="model-comparison.html#cb815-37" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_segment</span>(<span class="fu">aes</span>(<span class="at">xend =</span> x,</span>
-<span id="cb815-38"><a href="model-comparison.html#cb815-38" aria-hidden="true" tabindex="-1"></a>                     <span class="at">yend =</span> fitted),</span>
-<span id="cb815-39"><a href="model-comparison.html#cb815-39" aria-hidden="true" tabindex="-1"></a>                 <span class="at">data =</span> df.fit,</span>
-<span id="cb815-40"><a href="model-comparison.html#cb815-40" aria-hidden="true" tabindex="-1"></a>                 <span class="at">color =</span> <span class="st">&quot;red&quot;</span>,</span>
-<span id="cb815-41"><a href="model-comparison.html#cb815-41" aria-hidden="true" tabindex="-1"></a>                 <span class="at">size =</span> <span class="dv">1</span>) <span class="sc">+</span></span>
-<span id="cb815-42"><a href="model-comparison.html#cb815-42" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_point</span>(<span class="at">size =</span> <span class="dv">2</span>) <span class="sc">+</span></span>
-<span id="cb815-43"><a href="model-comparison.html#cb815-43" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>,</span>
-<span id="cb815-44"><a href="model-comparison.html#cb815-44" aria-hidden="true" tabindex="-1"></a>                <span class="at">formula =</span> <span class="st">&quot;y ~ x&quot;</span>,</span>
-<span id="cb815-45"><a href="model-comparison.html#cb815-45" aria-hidden="true" tabindex="-1"></a>                <span class="at">se =</span> F,</span>
-<span id="cb815-46"><a href="model-comparison.html#cb815-46" aria-hidden="true" tabindex="-1"></a>                <span class="at">color =</span> <span class="st">&quot;black&quot;</span>, </span>
-<span id="cb815-47"><a href="model-comparison.html#cb815-47" aria-hidden="true" tabindex="-1"></a>                <span class="at">fullrange =</span> T,</span>
-<span id="cb815-48"><a href="model-comparison.html#cb815-48" aria-hidden="true" tabindex="-1"></a>                <span class="at">data =</span> df.plot <span class="sc">%&gt;%</span> <span class="fu">filter</span>(color <span class="sc">!=</span> <span class="dv">2</span>))  <span class="sc">+</span></span>
-<span id="cb815-49"><a href="model-comparison.html#cb815-49" aria-hidden="true" tabindex="-1"></a>    <span class="fu">scale_color_manual</span>(<span class="at">values =</span> <span class="fu">c</span>(<span class="st">&quot;black&quot;</span>, <span class="st">&quot;red&quot;</span>)) <span class="sc">+</span> </span>
-<span id="cb815-50"><a href="model-comparison.html#cb815-50" aria-hidden="true" tabindex="-1"></a>    <span class="fu">theme</span>(<span class="at">legend.position =</span> <span class="st">&quot;none&quot;</span>,</span>
-<span id="cb815-51"><a href="model-comparison.html#cb815-51" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.title =</span> <span class="fu">element_blank</span>(),</span>
-<span id="cb815-52"><a href="model-comparison.html#cb815-52" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.ticks =</span> <span class="fu">element_blank</span>(),</span>
-<span id="cb815-53"><a href="model-comparison.html#cb815-53" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.text =</span> <span class="fu">element_blank</span>())</span>
-<span id="cb815-54"><a href="model-comparison.html#cb815-54" aria-hidden="true" tabindex="-1"></a>  <span class="fu">return</span>(p)</span>
-<span id="cb815-55"><a href="model-comparison.html#cb815-55" aria-hidden="true" tabindex="-1"></a>}</span>
-<span id="cb815-56"><a href="model-comparison.html#cb815-56" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb815-57"><a href="model-comparison.html#cb815-57" aria-hidden="true" tabindex="-1"></a><span class="co"># save plots in list </span></span>
-<span id="cb815-58"><a href="model-comparison.html#cb815-58" aria-hidden="true" tabindex="-1"></a>l.plots <span class="ot">=</span> <span class="fu">map</span>(<span class="at">.x =</span> <span class="dv">1</span><span class="sc">:</span><span class="dv">9</span>,</span>
-<span id="cb815-59"><a href="model-comparison.html#cb815-59" aria-hidden="true" tabindex="-1"></a>              <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">fun.cv_plot</span>(.))</span>
-<span id="cb815-60"><a href="model-comparison.html#cb815-60" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb815-61"><a href="model-comparison.html#cb815-61" aria-hidden="true" tabindex="-1"></a><span class="co"># make figure panel </span></span>
-<span id="cb815-62"><a href="model-comparison.html#cb815-62" aria-hidden="true" tabindex="-1"></a><span class="fu">wrap_plots</span>(<span class="at">plotlist =</span> l.plots, <span class="at">ncol =</span> <span class="dv">3</span>)</span></code></pre></div>
-<p><img src="15-model_comparison_files/figure-html/unnamed-chunk-8-1.png" width="672" /></p>
+<div class="sourceCode" id="cb11"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb11-1"><a href="model-comparison.html#cb11-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make example reproducible </span></span>
+<span id="cb11-2"><a href="model-comparison.html#cb11-2" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">1</span>)</span>
+<span id="cb11-3"><a href="model-comparison.html#cb11-3" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb11-4"><a href="model-comparison.html#cb11-4" aria-hidden="true" tabindex="-1"></a><span class="co"># sample</span></span>
+<span id="cb11-5"><a href="model-comparison.html#cb11-5" aria-hidden="true" tabindex="-1"></a>df.loo <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">x =</span> <span class="dv">1</span><span class="sc">:</span><span class="dv">9</span>,</span>
+<span id="cb11-6"><a href="model-comparison.html#cb11-6" aria-hidden="true" tabindex="-1"></a>                <span class="at">y =</span> <span class="fu">c</span>(<span class="dv">5</span>, <span class="dv">2</span>, <span class="dv">4</span>, <span class="dv">10</span>, <span class="dv">3</span>, <span class="dv">4</span>, <span class="dv">10</span>, <span class="dv">2</span>, <span class="dv">8</span>)) </span>
+<span id="cb11-7"><a href="model-comparison.html#cb11-7" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb11-8"><a href="model-comparison.html#cb11-8" aria-hidden="true" tabindex="-1"></a>df.loo_cross <span class="ot">=</span> df.loo <span class="sc">%&gt;%</span> </span>
+<span id="cb11-9"><a href="model-comparison.html#cb11-9" aria-hidden="true" tabindex="-1"></a>  <span class="fu">crossv_loo</span>() <span class="sc">%&gt;%</span> </span>
+<span id="cb11-10"><a href="model-comparison.html#cb11-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">fit =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
+<span id="cb11-11"><a href="model-comparison.html#cb11-11" aria-hidden="true" tabindex="-1"></a>                   <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> x, <span class="at">data =</span> .)),</span>
+<span id="cb11-12"><a href="model-comparison.html#cb11-12" aria-hidden="true" tabindex="-1"></a>         <span class="at">tidy =</span> <span class="fu">map</span>(<span class="at">.x =</span> fit,</span>
+<span id="cb11-13"><a href="model-comparison.html#cb11-13" aria-hidden="true" tabindex="-1"></a>                    <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">tidy</span>(.))) <span class="sc">%&gt;%</span> </span>
+<span id="cb11-14"><a href="model-comparison.html#cb11-14" aria-hidden="true" tabindex="-1"></a>  <span class="fu">unnest</span>(tidy)</span>
+<span id="cb11-15"><a href="model-comparison.html#cb11-15" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb11-16"><a href="model-comparison.html#cb11-16" aria-hidden="true" tabindex="-1"></a><span class="co"># original plot </span></span>
+<span id="cb11-17"><a href="model-comparison.html#cb11-17" aria-hidden="true" tabindex="-1"></a>df.plot <span class="ot">=</span> df.loo <span class="sc">%&gt;%</span> </span>
+<span id="cb11-18"><a href="model-comparison.html#cb11-18" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">color =</span> <span class="dv">1</span>)</span>
+<span id="cb11-19"><a href="model-comparison.html#cb11-19" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb11-20"><a href="model-comparison.html#cb11-20" aria-hidden="true" tabindex="-1"></a><span class="co"># fit to all data except one </span></span>
+<span id="cb11-21"><a href="model-comparison.html#cb11-21" aria-hidden="true" tabindex="-1"></a>fun.cv_plot <span class="ot">=</span> <span class="cf">function</span>(data_point){</span>
+<span id="cb11-22"><a href="model-comparison.html#cb11-22" aria-hidden="true" tabindex="-1"></a>  </span>
+<span id="cb11-23"><a href="model-comparison.html#cb11-23" aria-hidden="true" tabindex="-1"></a>  <span class="co"># determine which point to leave out </span></span>
+<span id="cb11-24"><a href="model-comparison.html#cb11-24" aria-hidden="true" tabindex="-1"></a>  df.plot <span class="ot">=</span> df.plot <span class="sc">%&gt;%</span> </span>
+<span id="cb11-25"><a href="model-comparison.html#cb11-25" aria-hidden="true" tabindex="-1"></a>    <span class="fu">mutate</span>(<span class="at">color =</span> <span class="fu">ifelse</span>(<span class="fu">row_number</span>() <span class="sc">==</span> data_point, <span class="dv">2</span>, color))</span>
+<span id="cb11-26"><a href="model-comparison.html#cb11-26" aria-hidden="true" tabindex="-1"></a>  </span>
+<span id="cb11-27"><a href="model-comparison.html#cb11-27" aria-hidden="true" tabindex="-1"></a>  <span class="co"># fit </span></span>
+<span id="cb11-28"><a href="model-comparison.html#cb11-28" aria-hidden="true" tabindex="-1"></a>  df.fit <span class="ot">=</span> df.plot <span class="sc">%&gt;%</span> </span>
+<span id="cb11-29"><a href="model-comparison.html#cb11-29" aria-hidden="true" tabindex="-1"></a>    <span class="fu">filter</span>(color <span class="sc">!=</span> <span class="dv">2</span>) <span class="sc">%&gt;%</span> </span>
+<span id="cb11-30"><a href="model-comparison.html#cb11-30" aria-hidden="true" tabindex="-1"></a>    <span class="fu">lm</span>(<span class="at">formula =</span> y <span class="sc">~</span> x, <span class="at">data =</span> .) <span class="sc">%&gt;%</span> </span>
+<span id="cb11-31"><a href="model-comparison.html#cb11-31" aria-hidden="true" tabindex="-1"></a>    <span class="fu">augment</span>(<span class="at">newdata =</span> df.plot <span class="sc">%&gt;%</span> </span>
+<span id="cb11-32"><a href="model-comparison.html#cb11-32" aria-hidden="true" tabindex="-1"></a>              <span class="fu">filter</span>(color <span class="sc">==</span> <span class="dv">2</span>)) <span class="sc">%&gt;%</span> </span>
+<span id="cb11-33"><a href="model-comparison.html#cb11-33" aria-hidden="true" tabindex="-1"></a>    <span class="fu">clean_names</span>()</span>
+<span id="cb11-34"><a href="model-comparison.html#cb11-34" aria-hidden="true" tabindex="-1"></a>  </span>
+<span id="cb11-35"><a href="model-comparison.html#cb11-35" aria-hidden="true" tabindex="-1"></a>  p <span class="ot">=</span> <span class="fu">ggplot</span>(df.plot,</span>
+<span id="cb11-36"><a href="model-comparison.html#cb11-36" aria-hidden="true" tabindex="-1"></a>             <span class="fu">aes</span>(x, y, <span class="at">color =</span> <span class="fu">as.factor</span>(color))) <span class="sc">+</span> </span>
+<span id="cb11-37"><a href="model-comparison.html#cb11-37" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_segment</span>(<span class="fu">aes</span>(<span class="at">xend =</span> x,</span>
+<span id="cb11-38"><a href="model-comparison.html#cb11-38" aria-hidden="true" tabindex="-1"></a>                     <span class="at">yend =</span> fitted),</span>
+<span id="cb11-39"><a href="model-comparison.html#cb11-39" aria-hidden="true" tabindex="-1"></a>                 <span class="at">data =</span> df.fit,</span>
+<span id="cb11-40"><a href="model-comparison.html#cb11-40" aria-hidden="true" tabindex="-1"></a>                 <span class="at">color =</span> <span class="st">&quot;red&quot;</span>,</span>
+<span id="cb11-41"><a href="model-comparison.html#cb11-41" aria-hidden="true" tabindex="-1"></a>                 <span class="at">size =</span> <span class="dv">1</span>) <span class="sc">+</span></span>
+<span id="cb11-42"><a href="model-comparison.html#cb11-42" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_point</span>(<span class="at">size =</span> <span class="dv">2</span>) <span class="sc">+</span></span>
+<span id="cb11-43"><a href="model-comparison.html#cb11-43" aria-hidden="true" tabindex="-1"></a>    <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>,</span>
+<span id="cb11-44"><a href="model-comparison.html#cb11-44" aria-hidden="true" tabindex="-1"></a>                <span class="at">formula =</span> <span class="st">&quot;y ~ x&quot;</span>,</span>
+<span id="cb11-45"><a href="model-comparison.html#cb11-45" aria-hidden="true" tabindex="-1"></a>                <span class="at">se =</span> F,</span>
+<span id="cb11-46"><a href="model-comparison.html#cb11-46" aria-hidden="true" tabindex="-1"></a>                <span class="at">color =</span> <span class="st">&quot;black&quot;</span>, </span>
+<span id="cb11-47"><a href="model-comparison.html#cb11-47" aria-hidden="true" tabindex="-1"></a>                <span class="at">fullrange =</span> T,</span>
+<span id="cb11-48"><a href="model-comparison.html#cb11-48" aria-hidden="true" tabindex="-1"></a>                <span class="at">data =</span> df.plot <span class="sc">%&gt;%</span> <span class="fu">filter</span>(color <span class="sc">!=</span> <span class="dv">2</span>))  <span class="sc">+</span></span>
+<span id="cb11-49"><a href="model-comparison.html#cb11-49" aria-hidden="true" tabindex="-1"></a>    <span class="fu">scale_color_manual</span>(<span class="at">values =</span> <span class="fu">c</span>(<span class="st">&quot;black&quot;</span>, <span class="st">&quot;red&quot;</span>)) <span class="sc">+</span> </span>
+<span id="cb11-50"><a href="model-comparison.html#cb11-50" aria-hidden="true" tabindex="-1"></a>    <span class="fu">theme</span>(<span class="at">legend.position =</span> <span class="st">&quot;none&quot;</span>,</span>
+<span id="cb11-51"><a href="model-comparison.html#cb11-51" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.title =</span> <span class="fu">element_blank</span>(),</span>
+<span id="cb11-52"><a href="model-comparison.html#cb11-52" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.ticks =</span> <span class="fu">element_blank</span>(),</span>
+<span id="cb11-53"><a href="model-comparison.html#cb11-53" aria-hidden="true" tabindex="-1"></a>          <span class="at">axis.text =</span> <span class="fu">element_blank</span>())</span>
+<span id="cb11-54"><a href="model-comparison.html#cb11-54" aria-hidden="true" tabindex="-1"></a>  <span class="fu">return</span>(p)</span>
+<span id="cb11-55"><a href="model-comparison.html#cb11-55" aria-hidden="true" tabindex="-1"></a>}</span>
+<span id="cb11-56"><a href="model-comparison.html#cb11-56" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb11-57"><a href="model-comparison.html#cb11-57" aria-hidden="true" tabindex="-1"></a><span class="co"># save plots in list </span></span>
+<span id="cb11-58"><a href="model-comparison.html#cb11-58" aria-hidden="true" tabindex="-1"></a>l.plots <span class="ot">=</span> <span class="fu">map</span>(<span class="at">.x =</span> <span class="dv">1</span><span class="sc">:</span><span class="dv">9</span>,</span>
+<span id="cb11-59"><a href="model-comparison.html#cb11-59" aria-hidden="true" tabindex="-1"></a>              <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">fun.cv_plot</span>(.))</span>
+<span id="cb11-60"><a href="model-comparison.html#cb11-60" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb11-61"><a href="model-comparison.html#cb11-61" aria-hidden="true" tabindex="-1"></a><span class="co"># make figure panel </span></span>
+<span id="cb11-62"><a href="model-comparison.html#cb11-62" aria-hidden="true" tabindex="-1"></a><span class="fu">wrap_plots</span>(<span class="at">plotlist =</span> l.plots, <span class="at">ncol =</span> <span class="dv">3</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-8-1.png" width="672" /></p>
 <p>As you can see, the regression line changes quite a bit depending on which data point is in the test set.</p>
 <p>Now, let’s use LOO to evaluate the models on the data set I’ve created above:</p>
-<div class="sourceCode" id="cb816"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb816-1"><a href="model-comparison.html#cb816-1" aria-hidden="true" tabindex="-1"></a><span class="co"># fit the models and calculate the RMSE for each model on the test set </span></span>
-<span id="cb816-2"><a href="model-comparison.html#cb816-2" aria-hidden="true" tabindex="-1"></a>df.cross <span class="ot">=</span> df.data <span class="sc">%&gt;%</span> </span>
-<span id="cb816-3"><a href="model-comparison.html#cb816-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">crossv_loo</span>() <span class="sc">%&gt;%</span> <span class="co"># function which generates training and test data sets </span></span>
-<span id="cb816-4"><a href="model-comparison.html#cb816-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">model_simple =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
-<span id="cb816-5"><a href="model-comparison.html#cb816-5" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x, <span class="at">data =</span> .)),</span>
-<span id="cb816-6"><a href="model-comparison.html#cb816-6" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_correct =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
-<span id="cb816-7"><a href="model-comparison.html#cb816-7" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>), <span class="at">data =</span> .)),</span>
-<span id="cb816-8"><a href="model-comparison.html#cb816-8" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_complex =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
-<span id="cb816-9"><a href="model-comparison.html#cb816-9" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>) <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">3</span>), <span class="at">data =</span> .))) <span class="sc">%&gt;%</span> </span>
-<span id="cb816-10"><a href="model-comparison.html#cb816-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">pivot_longer</span>(<span class="at">cols =</span> <span class="fu">contains</span>(<span class="st">&quot;model&quot;</span>),</span>
-<span id="cb816-11"><a href="model-comparison.html#cb816-11" aria-hidden="true" tabindex="-1"></a>               <span class="at">names_to =</span> <span class="st">&quot;model&quot;</span>,</span>
-<span id="cb816-12"><a href="model-comparison.html#cb816-12" aria-hidden="true" tabindex="-1"></a>               <span class="at">values_to =</span> <span class="st">&quot;fit&quot;</span>) <span class="sc">%&gt;%</span> </span>
-<span id="cb816-13"><a href="model-comparison.html#cb816-13" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">rmse =</span> <span class="fu">map2_dbl</span>(<span class="at">.x =</span> fit,</span>
-<span id="cb816-14"><a href="model-comparison.html#cb816-14" aria-hidden="true" tabindex="-1"></a>                         <span class="at">.y =</span> test,</span>
-<span id="cb816-15"><a href="model-comparison.html#cb816-15" aria-hidden="true" tabindex="-1"></a>                         <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">rmse</span>(.x, .y)))</span>
-<span id="cb816-16"><a href="model-comparison.html#cb816-16" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb816-17"><a href="model-comparison.html#cb816-17" aria-hidden="true" tabindex="-1"></a><span class="co"># show the average RMSE for each model </span></span>
-<span id="cb816-18"><a href="model-comparison.html#cb816-18" aria-hidden="true" tabindex="-1"></a>df.cross <span class="sc">%&gt;%</span> </span>
-<span id="cb816-19"><a href="model-comparison.html#cb816-19" aria-hidden="true" tabindex="-1"></a>  <span class="fu">group_by</span>(model) <span class="sc">%&gt;%</span> </span>
-<span id="cb816-20"><a href="model-comparison.html#cb816-20" aria-hidden="true" tabindex="-1"></a>  <span class="fu">summarize</span>(<span class="at">mean_rmse =</span> <span class="fu">mean</span>(rmse) <span class="sc">%&gt;%</span> </span>
-<span id="cb816-21"><a href="model-comparison.html#cb816-21" aria-hidden="true" tabindex="-1"></a>              <span class="fu">round</span>(<span class="dv">3</span>))</span></code></pre></div>
+<div class="sourceCode" id="cb12"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb12-1"><a href="model-comparison.html#cb12-1" aria-hidden="true" tabindex="-1"></a><span class="co"># fit the models and calculate the RMSE for each model on the test set </span></span>
+<span id="cb12-2"><a href="model-comparison.html#cb12-2" aria-hidden="true" tabindex="-1"></a>df.cross <span class="ot">=</span> df.data <span class="sc">%&gt;%</span> </span>
+<span id="cb12-3"><a href="model-comparison.html#cb12-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">crossv_loo</span>() <span class="sc">%&gt;%</span> <span class="co"># function which generates training and test data sets </span></span>
+<span id="cb12-4"><a href="model-comparison.html#cb12-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">model_simple =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
+<span id="cb12-5"><a href="model-comparison.html#cb12-5" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x, <span class="at">data =</span> .)),</span>
+<span id="cb12-6"><a href="model-comparison.html#cb12-6" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_correct =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
+<span id="cb12-7"><a href="model-comparison.html#cb12-7" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>), <span class="at">data =</span> .)),</span>
+<span id="cb12-8"><a href="model-comparison.html#cb12-8" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_complex =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
+<span id="cb12-9"><a href="model-comparison.html#cb12-9" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>) <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">3</span>), <span class="at">data =</span> .))) <span class="sc">%&gt;%</span> </span>
+<span id="cb12-10"><a href="model-comparison.html#cb12-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">pivot_longer</span>(<span class="at">cols =</span> <span class="fu">contains</span>(<span class="st">&quot;model&quot;</span>),</span>
+<span id="cb12-11"><a href="model-comparison.html#cb12-11" aria-hidden="true" tabindex="-1"></a>               <span class="at">names_to =</span> <span class="st">&quot;model&quot;</span>,</span>
+<span id="cb12-12"><a href="model-comparison.html#cb12-12" aria-hidden="true" tabindex="-1"></a>               <span class="at">values_to =</span> <span class="st">&quot;fit&quot;</span>) <span class="sc">%&gt;%</span> </span>
+<span id="cb12-13"><a href="model-comparison.html#cb12-13" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">rmse =</span> <span class="fu">map2_dbl</span>(<span class="at">.x =</span> fit,</span>
+<span id="cb12-14"><a href="model-comparison.html#cb12-14" aria-hidden="true" tabindex="-1"></a>                         <span class="at">.y =</span> test,</span>
+<span id="cb12-15"><a href="model-comparison.html#cb12-15" aria-hidden="true" tabindex="-1"></a>                         <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">rmse</span>(.x, .y)))</span>
+<span id="cb12-16"><a href="model-comparison.html#cb12-16" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb12-17"><a href="model-comparison.html#cb12-17" aria-hidden="true" tabindex="-1"></a><span class="co"># show the average RMSE for each model </span></span>
+<span id="cb12-18"><a href="model-comparison.html#cb12-18" aria-hidden="true" tabindex="-1"></a>df.cross <span class="sc">%&gt;%</span> </span>
+<span id="cb12-19"><a href="model-comparison.html#cb12-19" aria-hidden="true" tabindex="-1"></a>  <span class="fu">group_by</span>(model) <span class="sc">%&gt;%</span> </span>
+<span id="cb12-20"><a href="model-comparison.html#cb12-20" aria-hidden="true" tabindex="-1"></a>  <span class="fu">summarize</span>(<span class="at">mean_rmse =</span> <span class="fu">mean</span>(rmse) <span class="sc">%&gt;%</span> </span>
+<span id="cb12-21"><a href="model-comparison.html#cb12-21" aria-hidden="true" tabindex="-1"></a>              <span class="fu">round</span>(<span class="dv">3</span>))</span></code></pre></div>
 <pre><code># A tibble: 3 × 2
   model         mean_rmse
   &lt;chr&gt;             &lt;dbl&gt;
@@ -1184,25 +1152,25 @@ <h4><span class="header-section-number">15.3.3.1</span> Leave-one-out cross-vali
 <h4><span class="header-section-number">15.3.3.2</span> k-fold cross-validation<a href="model-comparison.html#k-fold-cross-validation" class="anchor-section" aria-label="Anchor link to header"></a></h4>
 <p>For k-fold cross-validation, we split the data set in k folds, and then use k-1 folds as the training set, and the remaining fold as the test set.</p>
 <p>The code is almost identical as before. Instead of <code>crossv_loo()</code>, we use the <code>crossv_kfold()</code> function instead and say how many times we want to “fold” the data.</p>
-<div class="sourceCode" id="cb818"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb818-1"><a href="model-comparison.html#cb818-1" aria-hidden="true" tabindex="-1"></a><span class="co"># crossvalidation scheme </span></span>
-<span id="cb818-2"><a href="model-comparison.html#cb818-2" aria-hidden="true" tabindex="-1"></a>df.cross <span class="ot">=</span> df.data <span class="sc">%&gt;%</span> </span>
-<span id="cb818-3"><a href="model-comparison.html#cb818-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">crossv_kfold</span>(<span class="at">k =</span> <span class="dv">10</span>) <span class="sc">%&gt;%</span> </span>
-<span id="cb818-4"><a href="model-comparison.html#cb818-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">model_simple =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
-<span id="cb818-5"><a href="model-comparison.html#cb818-5" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x, <span class="at">data =</span> .)),</span>
-<span id="cb818-6"><a href="model-comparison.html#cb818-6" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_correct =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
-<span id="cb818-7"><a href="model-comparison.html#cb818-7" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>), <span class="at">data =</span> .)),</span>
-<span id="cb818-8"><a href="model-comparison.html#cb818-8" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_complex =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
-<span id="cb818-9"><a href="model-comparison.html#cb818-9" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>) <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">3</span>), <span class="at">data =</span> .))) <span class="sc">%&gt;%</span> </span>
-<span id="cb818-10"><a href="model-comparison.html#cb818-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">pivot_longer</span>(<span class="at">cols =</span> <span class="fu">contains</span>(<span class="st">&quot;model&quot;</span>),</span>
-<span id="cb818-11"><a href="model-comparison.html#cb818-11" aria-hidden="true" tabindex="-1"></a>               <span class="at">names_to =</span> <span class="st">&quot;model&quot;</span>,</span>
-<span id="cb818-12"><a href="model-comparison.html#cb818-12" aria-hidden="true" tabindex="-1"></a>               <span class="at">values_to =</span> <span class="st">&quot;fit&quot;</span>) <span class="sc">%&gt;%</span> </span>
-<span id="cb818-13"><a href="model-comparison.html#cb818-13" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">rsquare =</span> <span class="fu">map2_dbl</span>(<span class="at">.x =</span> fit,</span>
-<span id="cb818-14"><a href="model-comparison.html#cb818-14" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.y =</span> test,</span>
-<span id="cb818-15"><a href="model-comparison.html#cb818-15" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">rsquare</span>(.x, .y)))</span>
-<span id="cb818-16"><a href="model-comparison.html#cb818-16" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb818-17"><a href="model-comparison.html#cb818-17" aria-hidden="true" tabindex="-1"></a>df.cross <span class="sc">%&gt;%</span> </span>
-<span id="cb818-18"><a href="model-comparison.html#cb818-18" aria-hidden="true" tabindex="-1"></a>  <span class="fu">group_by</span>(model) <span class="sc">%&gt;%</span> </span>
-<span id="cb818-19"><a href="model-comparison.html#cb818-19" aria-hidden="true" tabindex="-1"></a>  <span class="fu">summarize</span>(<span class="at">median_rsquare =</span> <span class="fu">median</span>(rsquare))</span></code></pre></div>
+<div class="sourceCode" id="cb14"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb14-1"><a href="model-comparison.html#cb14-1" aria-hidden="true" tabindex="-1"></a><span class="co"># crossvalidation scheme </span></span>
+<span id="cb14-2"><a href="model-comparison.html#cb14-2" aria-hidden="true" tabindex="-1"></a>df.cross <span class="ot">=</span> df.data <span class="sc">%&gt;%</span> </span>
+<span id="cb14-3"><a href="model-comparison.html#cb14-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">crossv_kfold</span>(<span class="at">k =</span> <span class="dv">10</span>) <span class="sc">%&gt;%</span> </span>
+<span id="cb14-4"><a href="model-comparison.html#cb14-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">model_simple =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
+<span id="cb14-5"><a href="model-comparison.html#cb14-5" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x, <span class="at">data =</span> .)),</span>
+<span id="cb14-6"><a href="model-comparison.html#cb14-6" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_correct =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
+<span id="cb14-7"><a href="model-comparison.html#cb14-7" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>), <span class="at">data =</span> .)),</span>
+<span id="cb14-8"><a href="model-comparison.html#cb14-8" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_complex =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
+<span id="cb14-9"><a href="model-comparison.html#cb14-9" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>) <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">3</span>), <span class="at">data =</span> .))) <span class="sc">%&gt;%</span> </span>
+<span id="cb14-10"><a href="model-comparison.html#cb14-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">pivot_longer</span>(<span class="at">cols =</span> <span class="fu">contains</span>(<span class="st">&quot;model&quot;</span>),</span>
+<span id="cb14-11"><a href="model-comparison.html#cb14-11" aria-hidden="true" tabindex="-1"></a>               <span class="at">names_to =</span> <span class="st">&quot;model&quot;</span>,</span>
+<span id="cb14-12"><a href="model-comparison.html#cb14-12" aria-hidden="true" tabindex="-1"></a>               <span class="at">values_to =</span> <span class="st">&quot;fit&quot;</span>) <span class="sc">%&gt;%</span> </span>
+<span id="cb14-13"><a href="model-comparison.html#cb14-13" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">rsquare =</span> <span class="fu">map2_dbl</span>(<span class="at">.x =</span> fit,</span>
+<span id="cb14-14"><a href="model-comparison.html#cb14-14" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.y =</span> test,</span>
+<span id="cb14-15"><a href="model-comparison.html#cb14-15" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">rsquare</span>(.x, .y)))</span>
+<span id="cb14-16"><a href="model-comparison.html#cb14-16" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb14-17"><a href="model-comparison.html#cb14-17" aria-hidden="true" tabindex="-1"></a>df.cross <span class="sc">%&gt;%</span> </span>
+<span id="cb14-18"><a href="model-comparison.html#cb14-18" aria-hidden="true" tabindex="-1"></a>  <span class="fu">group_by</span>(model) <span class="sc">%&gt;%</span> </span>
+<span id="cb14-19"><a href="model-comparison.html#cb14-19" aria-hidden="true" tabindex="-1"></a>  <span class="fu">summarize</span>(<span class="at">median_rsquare =</span> <span class="fu">median</span>(rsquare))</span></code></pre></div>
 <pre><code># A tibble: 3 × 2
   model         median_rsquare
   &lt;chr&gt;                  &lt;dbl&gt;
@@ -1214,25 +1182,25 @@ <h4><span class="header-section-number">15.3.3.2</span> k-fold cross-validation<
 <div id="monte-carlo-cross-validation" class="section level4 hasAnchor" number="15.3.3.3">
 <h4><span class="header-section-number">15.3.3.3</span> Monte Carlo cross-validation<a href="model-comparison.html#monte-carlo-cross-validation" class="anchor-section" aria-label="Anchor link to header"></a></h4>
 <p>Finally, let’s consider another very flexible version of cross-validation. For this version of cross-validation, we determine how many random splits into training set and test set we would like to do, and what proportion of the data should be in the test set.</p>
-<div class="sourceCode" id="cb820"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb820-1"><a href="model-comparison.html#cb820-1" aria-hidden="true" tabindex="-1"></a><span class="co"># crossvalidation scheme </span></span>
-<span id="cb820-2"><a href="model-comparison.html#cb820-2" aria-hidden="true" tabindex="-1"></a>df.cross <span class="ot">=</span> df.data <span class="sc">%&gt;%</span> </span>
-<span id="cb820-3"><a href="model-comparison.html#cb820-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">crossv_mc</span>(<span class="at">n =</span> <span class="dv">50</span>, <span class="at">test =</span> <span class="fl">0.5</span>) <span class="sc">%&gt;%</span> <span class="co"># number of samples, and percentage of test </span></span>
-<span id="cb820-4"><a href="model-comparison.html#cb820-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">model_simple =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
-<span id="cb820-5"><a href="model-comparison.html#cb820-5" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x, <span class="at">data =</span> .x)),</span>
-<span id="cb820-6"><a href="model-comparison.html#cb820-6" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_correct =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
-<span id="cb820-7"><a href="model-comparison.html#cb820-7" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>), <span class="at">data =</span> .x)),</span>
-<span id="cb820-8"><a href="model-comparison.html#cb820-8" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_complex =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
-<span id="cb820-9"><a href="model-comparison.html#cb820-9" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>) <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">3</span>), <span class="at">data =</span> .))) <span class="sc">%&gt;%</span> </span>
-<span id="cb820-10"><a href="model-comparison.html#cb820-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">pivot_longer</span>(<span class="at">cols =</span> <span class="fu">contains</span>(<span class="st">&quot;model&quot;</span>),</span>
-<span id="cb820-11"><a href="model-comparison.html#cb820-11" aria-hidden="true" tabindex="-1"></a>               <span class="at">names_to =</span> <span class="st">&quot;model&quot;</span>,</span>
-<span id="cb820-12"><a href="model-comparison.html#cb820-12" aria-hidden="true" tabindex="-1"></a>               <span class="at">values_to =</span> <span class="st">&quot;fit&quot;</span>) <span class="sc">%&gt;%</span> </span>
-<span id="cb820-13"><a href="model-comparison.html#cb820-13" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">rmse =</span> <span class="fu">map2_dbl</span>(<span class="at">.x =</span> fit,</span>
-<span id="cb820-14"><a href="model-comparison.html#cb820-14" aria-hidden="true" tabindex="-1"></a>                         <span class="at">.y =</span> test,</span>
-<span id="cb820-15"><a href="model-comparison.html#cb820-15" aria-hidden="true" tabindex="-1"></a>                         <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">rmse</span>(.x, .y)))</span>
-<span id="cb820-16"><a href="model-comparison.html#cb820-16" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb820-17"><a href="model-comparison.html#cb820-17" aria-hidden="true" tabindex="-1"></a>df.cross <span class="sc">%&gt;%</span> </span>
-<span id="cb820-18"><a href="model-comparison.html#cb820-18" aria-hidden="true" tabindex="-1"></a>  <span class="fu">group_by</span>(model) <span class="sc">%&gt;%</span> </span>
-<span id="cb820-19"><a href="model-comparison.html#cb820-19" aria-hidden="true" tabindex="-1"></a>  <span class="fu">summarize</span>(<span class="at">mean_rmse =</span> <span class="fu">mean</span>(rmse))</span></code></pre></div>
+<div class="sourceCode" id="cb16"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb16-1"><a href="model-comparison.html#cb16-1" aria-hidden="true" tabindex="-1"></a><span class="co"># crossvalidation scheme </span></span>
+<span id="cb16-2"><a href="model-comparison.html#cb16-2" aria-hidden="true" tabindex="-1"></a>df.cross <span class="ot">=</span> df.data <span class="sc">%&gt;%</span> </span>
+<span id="cb16-3"><a href="model-comparison.html#cb16-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">crossv_mc</span>(<span class="at">n =</span> <span class="dv">50</span>, <span class="at">test =</span> <span class="fl">0.5</span>) <span class="sc">%&gt;%</span> <span class="co"># number of samples, and percentage of test </span></span>
+<span id="cb16-4"><a href="model-comparison.html#cb16-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">model_simple =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
+<span id="cb16-5"><a href="model-comparison.html#cb16-5" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x, <span class="at">data =</span> .x)),</span>
+<span id="cb16-6"><a href="model-comparison.html#cb16-6" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_correct =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
+<span id="cb16-7"><a href="model-comparison.html#cb16-7" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>), <span class="at">data =</span> .x)),</span>
+<span id="cb16-8"><a href="model-comparison.html#cb16-8" aria-hidden="true" tabindex="-1"></a>         <span class="at">model_complex =</span> <span class="fu">map</span>(<span class="at">.x =</span> train,</span>
+<span id="cb16-9"><a href="model-comparison.html#cb16-9" aria-hidden="true" tabindex="-1"></a>                             <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">lm</span>(y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">2</span>) <span class="sc">+</span> <span class="fu">I</span>(x<span class="sc">^</span><span class="dv">3</span>), <span class="at">data =</span> .))) <span class="sc">%&gt;%</span> </span>
+<span id="cb16-10"><a href="model-comparison.html#cb16-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">pivot_longer</span>(<span class="at">cols =</span> <span class="fu">contains</span>(<span class="st">&quot;model&quot;</span>),</span>
+<span id="cb16-11"><a href="model-comparison.html#cb16-11" aria-hidden="true" tabindex="-1"></a>               <span class="at">names_to =</span> <span class="st">&quot;model&quot;</span>,</span>
+<span id="cb16-12"><a href="model-comparison.html#cb16-12" aria-hidden="true" tabindex="-1"></a>               <span class="at">values_to =</span> <span class="st">&quot;fit&quot;</span>) <span class="sc">%&gt;%</span> </span>
+<span id="cb16-13"><a href="model-comparison.html#cb16-13" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">rmse =</span> <span class="fu">map2_dbl</span>(<span class="at">.x =</span> fit,</span>
+<span id="cb16-14"><a href="model-comparison.html#cb16-14" aria-hidden="true" tabindex="-1"></a>                         <span class="at">.y =</span> test,</span>
+<span id="cb16-15"><a href="model-comparison.html#cb16-15" aria-hidden="true" tabindex="-1"></a>                         <span class="at">.f =</span> <span class="sc">~</span> <span class="fu">rmse</span>(.x, .y)))</span>
+<span id="cb16-16"><a href="model-comparison.html#cb16-16" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb16-17"><a href="model-comparison.html#cb16-17" aria-hidden="true" tabindex="-1"></a>df.cross <span class="sc">%&gt;%</span> </span>
+<span id="cb16-18"><a href="model-comparison.html#cb16-18" aria-hidden="true" tabindex="-1"></a>  <span class="fu">group_by</span>(model) <span class="sc">%&gt;%</span> </span>
+<span id="cb16-19"><a href="model-comparison.html#cb16-19" aria-hidden="true" tabindex="-1"></a>  <span class="fu">summarize</span>(<span class="at">mean_rmse =</span> <span class="fu">mean</span>(rmse))</span></code></pre></div>
 <pre><code># A tibble: 3 × 2
   model         mean_rmse
   &lt;chr&gt;             &lt;dbl&gt;
@@ -1246,29 +1214,29 @@ <h4><span class="header-section-number">15.3.3.3</span> Monte Carlo cross-valida
 <h3><span class="header-section-number">15.3.4</span> Bootstrap<a href="model-comparison.html#bootstrap" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>We can also use the <code>modelr</code> package for bootstrapping. The idea is the same as when we did cross-validation. We create a number of data sets from our original data set. Instead of splitting the data set in a training and test data set, for bootstrapping, we sample values from the original data set with replacement. Doing so, we can, for example, calculate the confidence interval of different statistics of interest.</p>
 <p>Here is an example for how to boostrap confidence intervals for a mean.</p>
-<div class="sourceCode" id="cb822"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb822-1"><a href="model-comparison.html#cb822-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make example reproducible </span></span>
-<span id="cb822-2"><a href="model-comparison.html#cb822-2" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">1</span>)</span>
-<span id="cb822-3"><a href="model-comparison.html#cb822-3" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb822-4"><a href="model-comparison.html#cb822-4" aria-hidden="true" tabindex="-1"></a>sample_size <span class="ot">=</span> <span class="dv">10</span> </span>
-<span id="cb822-5"><a href="model-comparison.html#cb822-5" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb822-6"><a href="model-comparison.html#cb822-6" aria-hidden="true" tabindex="-1"></a><span class="co"># sample</span></span>
-<span id="cb822-7"><a href="model-comparison.html#cb822-7" aria-hidden="true" tabindex="-1"></a>df.data <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">participant =</span> <span class="dv">1</span><span class="sc">:</span>sample_size,</span>
-<span id="cb822-8"><a href="model-comparison.html#cb822-8" aria-hidden="true" tabindex="-1"></a>                 <span class="at">x =</span> <span class="fu">runif</span>(sample_size, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">1</span>)) </span>
-<span id="cb822-9"><a href="model-comparison.html#cb822-9" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb822-10"><a href="model-comparison.html#cb822-10" aria-hidden="true" tabindex="-1"></a><span class="co"># mean of the actual sample</span></span>
-<span id="cb822-11"><a href="model-comparison.html#cb822-11" aria-hidden="true" tabindex="-1"></a><span class="fu">mean</span>(df.data<span class="sc">$</span>x)</span></code></pre></div>
+<div class="sourceCode" id="cb18"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb18-1"><a href="model-comparison.html#cb18-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make example reproducible </span></span>
+<span id="cb18-2"><a href="model-comparison.html#cb18-2" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">1</span>)</span>
+<span id="cb18-3"><a href="model-comparison.html#cb18-3" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb18-4"><a href="model-comparison.html#cb18-4" aria-hidden="true" tabindex="-1"></a>sample_size <span class="ot">=</span> <span class="dv">10</span> </span>
+<span id="cb18-5"><a href="model-comparison.html#cb18-5" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb18-6"><a href="model-comparison.html#cb18-6" aria-hidden="true" tabindex="-1"></a><span class="co"># sample</span></span>
+<span id="cb18-7"><a href="model-comparison.html#cb18-7" aria-hidden="true" tabindex="-1"></a>df.data <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">participant =</span> <span class="dv">1</span><span class="sc">:</span>sample_size,</span>
+<span id="cb18-8"><a href="model-comparison.html#cb18-8" aria-hidden="true" tabindex="-1"></a>                 <span class="at">x =</span> <span class="fu">runif</span>(sample_size, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">1</span>)) </span>
+<span id="cb18-9"><a href="model-comparison.html#cb18-9" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb18-10"><a href="model-comparison.html#cb18-10" aria-hidden="true" tabindex="-1"></a><span class="co"># mean of the actual sample</span></span>
+<span id="cb18-11"><a href="model-comparison.html#cb18-11" aria-hidden="true" tabindex="-1"></a><span class="fu">mean</span>(df.data<span class="sc">$</span>x)</span></code></pre></div>
 <pre><code>[1] 0.5515139</code></pre>
-<div class="sourceCode" id="cb824"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb824-1"><a href="model-comparison.html#cb824-1" aria-hidden="true" tabindex="-1"></a><span class="co"># bootstrap to get confidence intervals around the mean </span></span>
-<span id="cb824-2"><a href="model-comparison.html#cb824-2" aria-hidden="true" tabindex="-1"></a>df.data <span class="sc">%&gt;%</span></span>
-<span id="cb824-3"><a href="model-comparison.html#cb824-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">bootstrap</span>(<span class="at">n =</span> <span class="dv">1000</span>) <span class="sc">%&gt;%</span> <span class="co"># create 1000 bootstrapped samples</span></span>
-<span id="cb824-4"><a href="model-comparison.html#cb824-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">estimate =</span> <span class="fu">map_dbl</span>(<span class="at">.x =</span> strap,</span>
-<span id="cb824-5"><a href="model-comparison.html#cb824-5" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.f =</span> <span class="sc">~</span> .x <span class="sc">%&gt;%</span> </span>
-<span id="cb824-6"><a href="model-comparison.html#cb824-6" aria-hidden="true" tabindex="-1"></a>                              <span class="fu">as_tibble</span>() <span class="sc">%&gt;%</span> </span>
-<span id="cb824-7"><a href="model-comparison.html#cb824-7" aria-hidden="true" tabindex="-1"></a>                              <span class="fu">pull</span>(x) <span class="sc">%&gt;%</span> </span>
-<span id="cb824-8"><a href="model-comparison.html#cb824-8" aria-hidden="true" tabindex="-1"></a>                              <span class="fu">mean</span>())) <span class="sc">%&gt;%</span> </span>
-<span id="cb824-9"><a href="model-comparison.html#cb824-9" aria-hidden="true" tabindex="-1"></a>  <span class="fu">summarize</span>(<span class="at">mean =</span> <span class="fu">mean</span>(estimate),</span>
-<span id="cb824-10"><a href="model-comparison.html#cb824-10" aria-hidden="true" tabindex="-1"></a>            <span class="at">low =</span> <span class="fu">quantile</span>(estimate, <span class="fl">0.025</span>), <span class="co"># calculate the 2.5 / 97.5 percentiles</span></span>
-<span id="cb824-11"><a href="model-comparison.html#cb824-11" aria-hidden="true" tabindex="-1"></a>            <span class="at">high =</span> <span class="fu">quantile</span>(estimate, <span class="fl">0.975</span>))</span></code></pre></div>
+<div class="sourceCode" id="cb20"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb20-1"><a href="model-comparison.html#cb20-1" aria-hidden="true" tabindex="-1"></a><span class="co"># bootstrap to get confidence intervals around the mean </span></span>
+<span id="cb20-2"><a href="model-comparison.html#cb20-2" aria-hidden="true" tabindex="-1"></a>df.data <span class="sc">%&gt;%</span></span>
+<span id="cb20-3"><a href="model-comparison.html#cb20-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">bootstrap</span>(<span class="at">n =</span> <span class="dv">1000</span>) <span class="sc">%&gt;%</span> <span class="co"># create 1000 bootstrapped samples</span></span>
+<span id="cb20-4"><a href="model-comparison.html#cb20-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">estimate =</span> <span class="fu">map_dbl</span>(<span class="at">.x =</span> strap,</span>
+<span id="cb20-5"><a href="model-comparison.html#cb20-5" aria-hidden="true" tabindex="-1"></a>                            <span class="at">.f =</span> <span class="sc">~</span> .x <span class="sc">%&gt;%</span> </span>
+<span id="cb20-6"><a href="model-comparison.html#cb20-6" aria-hidden="true" tabindex="-1"></a>                              <span class="fu">as_tibble</span>() <span class="sc">%&gt;%</span> </span>
+<span id="cb20-7"><a href="model-comparison.html#cb20-7" aria-hidden="true" tabindex="-1"></a>                              <span class="fu">pull</span>(x) <span class="sc">%&gt;%</span> </span>
+<span id="cb20-8"><a href="model-comparison.html#cb20-8" aria-hidden="true" tabindex="-1"></a>                              <span class="fu">mean</span>())) <span class="sc">%&gt;%</span> </span>
+<span id="cb20-9"><a href="model-comparison.html#cb20-9" aria-hidden="true" tabindex="-1"></a>  <span class="fu">summarize</span>(<span class="at">mean =</span> <span class="fu">mean</span>(estimate),</span>
+<span id="cb20-10"><a href="model-comparison.html#cb20-10" aria-hidden="true" tabindex="-1"></a>            <span class="at">low =</span> <span class="fu">quantile</span>(estimate, <span class="fl">0.025</span>), <span class="co"># calculate the 2.5 / 97.5 percentiles</span></span>
+<span id="cb20-11"><a href="model-comparison.html#cb20-11" aria-hidden="true" tabindex="-1"></a>            <span class="at">high =</span> <span class="fu">quantile</span>(estimate, <span class="fl">0.975</span>))</span></code></pre></div>
 <pre><code># A tibble: 1 × 3
    mean   low  high
   &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
@@ -1285,25 +1253,25 @@ <h3><span class="header-section-number">15.3.5</span> AIC and BIC<a href="model-
 \]</span></p>
 <p>where <span class="math inline">\(k\)</span> is the number of parameters in the model, <span class="math inline">\(n\)</span> is the number of observations, and <span class="math inline">\(\hat L\)</span> is the maximized value of the likelihood function of the model. Both AIC and BIC trade off model fit (as measured by the maximum likelihood of the data <span class="math inline">\(\hat L\)</span>) and the number of parameters in the model.</p>
 <p>Calculating AIC and BIC in R is straightforward. We simply need to fit a linear model, and then call the <code>AIC()</code> or <code>BIC()</code> functions on the fitted model like so:</p>
-<div class="sourceCode" id="cb826"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb826-1"><a href="model-comparison.html#cb826-1" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">0</span>)</span>
-<span id="cb826-2"><a href="model-comparison.html#cb826-2" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb826-3"><a href="model-comparison.html#cb826-3" aria-hidden="true" tabindex="-1"></a><span class="co"># let&#39;s generate some data </span></span>
-<span id="cb826-4"><a href="model-comparison.html#cb826-4" aria-hidden="true" tabindex="-1"></a>df.example <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">x =</span> <span class="fu">runif</span>(<span class="dv">20</span>, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">1</span>),</span>
-<span id="cb826-5"><a href="model-comparison.html#cb826-5" aria-hidden="true" tabindex="-1"></a>                    <span class="at">y =</span> <span class="dv">1</span> <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x <span class="sc">+</span> <span class="fu">rnorm</span>(<span class="dv">20</span>, <span class="at">sd =</span> <span class="dv">2</span>))</span>
-<span id="cb826-6"><a href="model-comparison.html#cb826-6" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb826-7"><a href="model-comparison.html#cb826-7" aria-hidden="true" tabindex="-1"></a><span class="co"># fit a linear model </span></span>
-<span id="cb826-8"><a href="model-comparison.html#cb826-8" aria-hidden="true" tabindex="-1"></a>fit <span class="ot">=</span> <span class="fu">lm</span>(<span class="at">formula =</span> y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x,</span>
-<span id="cb826-9"><a href="model-comparison.html#cb826-9" aria-hidden="true" tabindex="-1"></a>         <span class="at">data =</span> df.example)</span>
-<span id="cb826-10"><a href="model-comparison.html#cb826-10" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb826-11"><a href="model-comparison.html#cb826-11" aria-hidden="true" tabindex="-1"></a><span class="co"># get AIC </span></span>
-<span id="cb826-12"><a href="model-comparison.html#cb826-12" aria-hidden="true" tabindex="-1"></a><span class="fu">AIC</span>(fit)</span></code></pre></div>
+<div class="sourceCode" id="cb22"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb22-1"><a href="model-comparison.html#cb22-1" aria-hidden="true" tabindex="-1"></a><span class="fu">set.seed</span>(<span class="dv">0</span>)</span>
+<span id="cb22-2"><a href="model-comparison.html#cb22-2" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb22-3"><a href="model-comparison.html#cb22-3" aria-hidden="true" tabindex="-1"></a><span class="co"># let&#39;s generate some data </span></span>
+<span id="cb22-4"><a href="model-comparison.html#cb22-4" aria-hidden="true" tabindex="-1"></a>df.example <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">x =</span> <span class="fu">runif</span>(<span class="dv">20</span>, <span class="at">min =</span> <span class="dv">0</span>, <span class="at">max =</span> <span class="dv">1</span>),</span>
+<span id="cb22-5"><a href="model-comparison.html#cb22-5" aria-hidden="true" tabindex="-1"></a>                    <span class="at">y =</span> <span class="dv">1</span> <span class="sc">+</span> <span class="dv">3</span> <span class="sc">*</span> x <span class="sc">+</span> <span class="fu">rnorm</span>(<span class="dv">20</span>, <span class="at">sd =</span> <span class="dv">2</span>))</span>
+<span id="cb22-6"><a href="model-comparison.html#cb22-6" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb22-7"><a href="model-comparison.html#cb22-7" aria-hidden="true" tabindex="-1"></a><span class="co"># fit a linear model </span></span>
+<span id="cb22-8"><a href="model-comparison.html#cb22-8" aria-hidden="true" tabindex="-1"></a>fit <span class="ot">=</span> <span class="fu">lm</span>(<span class="at">formula =</span> y <span class="sc">~</span> <span class="dv">1</span> <span class="sc">+</span> x,</span>
+<span id="cb22-9"><a href="model-comparison.html#cb22-9" aria-hidden="true" tabindex="-1"></a>         <span class="at">data =</span> df.example)</span>
+<span id="cb22-10"><a href="model-comparison.html#cb22-10" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb22-11"><a href="model-comparison.html#cb22-11" aria-hidden="true" tabindex="-1"></a><span class="co"># get AIC </span></span>
+<span id="cb22-12"><a href="model-comparison.html#cb22-12" aria-hidden="true" tabindex="-1"></a><span class="fu">AIC</span>(fit)</span></code></pre></div>
 <pre><code>[1] 75.47296</code></pre>
-<div class="sourceCode" id="cb828"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb828-1"><a href="model-comparison.html#cb828-1" aria-hidden="true" tabindex="-1"></a><span class="co"># get BIC</span></span>
-<span id="cb828-2"><a href="model-comparison.html#cb828-2" aria-hidden="true" tabindex="-1"></a><span class="fu">BIC</span>(fit)</span></code></pre></div>
+<div class="sourceCode" id="cb24"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb24-1"><a href="model-comparison.html#cb24-1" aria-hidden="true" tabindex="-1"></a><span class="co"># get BIC</span></span>
+<span id="cb24-2"><a href="model-comparison.html#cb24-2" aria-hidden="true" tabindex="-1"></a><span class="fu">BIC</span>(fit)</span></code></pre></div>
 <pre><code>[1] 78.46016</code></pre>
 <p>We can also just use the <code>broom</code> package to get that information:</p>
-<div class="sourceCode" id="cb830"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb830-1"><a href="model-comparison.html#cb830-1" aria-hidden="true" tabindex="-1"></a>fit <span class="sc">%&gt;%</span> </span>
-<span id="cb830-2"><a href="model-comparison.html#cb830-2" aria-hidden="true" tabindex="-1"></a>  <span class="fu">glance</span>()</span></code></pre></div>
+<div class="sourceCode" id="cb26"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb26-1"><a href="model-comparison.html#cb26-1" aria-hidden="true" tabindex="-1"></a>fit <span class="sc">%&gt;%</span> </span>
+<span id="cb26-2"><a href="model-comparison.html#cb26-2" aria-hidden="true" tabindex="-1"></a>  <span class="fu">glance</span>()</span></code></pre></div>
 <pre><code># A tibble: 1 × 12
   r.squared adj.r.squared sigma statistic p.value    df logLik   AIC   BIC
       &lt;dbl&gt;         &lt;dbl&gt; &lt;dbl&gt;     &lt;dbl&gt;   &lt;dbl&gt; &lt;dbl&gt;  &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
@@ -1311,83 +1279,83 @@ <h3><span class="header-section-number">15.3.5</span> AIC and BIC<a href="model-
 # … with 3 more variables: deviance &lt;dbl&gt;, df.residual &lt;int&gt;, nobs &lt;int&gt;</code></pre>
 <p>Both AIC and BIC take the number of parameters and the model’s likelihood into account. BIC additionally considers the number of observations. But how is the likelihood of a linear model determined?</p>
 <p>Let’s visualize the data first:</p>
-<div class="sourceCode" id="cb832"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb832-1"><a href="model-comparison.html#cb832-1" aria-hidden="true" tabindex="-1"></a><span class="co"># plot the data with a linear model fit  </span></span>
-<span id="cb832-2"><a href="model-comparison.html#cb832-2" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.example,</span>
-<span id="cb832-3"><a href="model-comparison.html#cb832-3" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> x,</span>
-<span id="cb832-4"><a href="model-comparison.html#cb832-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> y)) <span class="sc">+</span> </span>
-<span id="cb832-5"><a href="model-comparison.html#cb832-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>(<span class="at">size =</span> <span class="dv">2</span>) <span class="sc">+</span></span>
-<span id="cb832-6"><a href="model-comparison.html#cb832-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>,</span>
-<span id="cb832-7"><a href="model-comparison.html#cb832-7" aria-hidden="true" tabindex="-1"></a>              <span class="at">color =</span> <span class="st">&quot;black&quot;</span>)</span></code></pre></div>
-<p><img src="15-model_comparison_files/figure-html/unnamed-chunk-15-1.png" width="672" /></p>
+<div class="sourceCode" id="cb28"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb28-1"><a href="model-comparison.html#cb28-1" aria-hidden="true" tabindex="-1"></a><span class="co"># plot the data with a linear model fit  </span></span>
+<span id="cb28-2"><a href="model-comparison.html#cb28-2" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.example,</span>
+<span id="cb28-3"><a href="model-comparison.html#cb28-3" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> x,</span>
+<span id="cb28-4"><a href="model-comparison.html#cb28-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> y)) <span class="sc">+</span> </span>
+<span id="cb28-5"><a href="model-comparison.html#cb28-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>(<span class="at">size =</span> <span class="dv">2</span>) <span class="sc">+</span></span>
+<span id="cb28-6"><a href="model-comparison.html#cb28-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>,</span>
+<span id="cb28-7"><a href="model-comparison.html#cb28-7" aria-hidden="true" tabindex="-1"></a>              <span class="at">color =</span> <span class="st">&quot;black&quot;</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-15-1.png" width="672" /></p>
 <p>Now, let’s take a look at the residuals by plotting the fitted values on the x axis, and the residuals on the y axis.</p>
-<div class="sourceCode" id="cb833"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb833-1"><a href="model-comparison.html#cb833-1" aria-hidden="true" tabindex="-1"></a><span class="co"># residual plot </span></span>
-<span id="cb833-2"><a href="model-comparison.html#cb833-2" aria-hidden="true" tabindex="-1"></a>df.plot <span class="ot">=</span> df.example <span class="sc">%&gt;%</span> </span>
-<span id="cb833-3"><a href="model-comparison.html#cb833-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">lm</span>(<span class="at">formula =</span> y <span class="sc">~</span> x,</span>
-<span id="cb833-4"><a href="model-comparison.html#cb833-4" aria-hidden="true" tabindex="-1"></a>     <span class="at">data =</span> .) <span class="sc">%&gt;%</span> </span>
-<span id="cb833-5"><a href="model-comparison.html#cb833-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">augment</span>() <span class="sc">%&gt;%</span> </span>
-<span id="cb833-6"><a href="model-comparison.html#cb833-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">clean_names</span>()</span>
-<span id="cb833-7"><a href="model-comparison.html#cb833-7" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb833-8"><a href="model-comparison.html#cb833-8" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.plot,</span>
-<span id="cb833-9"><a href="model-comparison.html#cb833-9" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> fitted, </span>
-<span id="cb833-10"><a href="model-comparison.html#cb833-10" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> resid)) <span class="sc">+</span> </span>
-<span id="cb833-11"><a href="model-comparison.html#cb833-11" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>(<span class="at">size =</span> <span class="dv">2</span>)</span></code></pre></div>
-<p><img src="15-model_comparison_files/figure-html/unnamed-chunk-16-1.png" width="672" /></p>
+<div class="sourceCode" id="cb29"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb29-1"><a href="model-comparison.html#cb29-1" aria-hidden="true" tabindex="-1"></a><span class="co"># residual plot </span></span>
+<span id="cb29-2"><a href="model-comparison.html#cb29-2" aria-hidden="true" tabindex="-1"></a>df.plot <span class="ot">=</span> df.example <span class="sc">%&gt;%</span> </span>
+<span id="cb29-3"><a href="model-comparison.html#cb29-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">lm</span>(<span class="at">formula =</span> y <span class="sc">~</span> x,</span>
+<span id="cb29-4"><a href="model-comparison.html#cb29-4" aria-hidden="true" tabindex="-1"></a>     <span class="at">data =</span> .) <span class="sc">%&gt;%</span> </span>
+<span id="cb29-5"><a href="model-comparison.html#cb29-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">augment</span>() <span class="sc">%&gt;%</span> </span>
+<span id="cb29-6"><a href="model-comparison.html#cb29-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">clean_names</span>()</span>
+<span id="cb29-7"><a href="model-comparison.html#cb29-7" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb29-8"><a href="model-comparison.html#cb29-8" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.plot,</span>
+<span id="cb29-9"><a href="model-comparison.html#cb29-9" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> fitted, </span>
+<span id="cb29-10"><a href="model-comparison.html#cb29-10" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> resid)) <span class="sc">+</span> </span>
+<span id="cb29-11"><a href="model-comparison.html#cb29-11" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>(<span class="at">size =</span> <span class="dv">2</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-16-1.png" width="672" /></p>
 <p>Remember that the linear model makes the assumption that the residuals are normally distributed with mean 0 (which is always the case if we fit a linear model) and some fitted standard deviation. In fact, the standard deviation of the normal distribution is fitted such that the overall likelihood of the data is maximized.</p>
 <p>Let’s make a plot that shows a normal distribution alongside the residuals:</p>
-<div class="sourceCode" id="cb834"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb834-1"><a href="model-comparison.html#cb834-1" aria-hidden="true" tabindex="-1"></a><span class="co"># define a normal distribution </span></span>
-<span id="cb834-2"><a href="model-comparison.html#cb834-2" aria-hidden="true" tabindex="-1"></a>df.normal <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">y =</span> <span class="fu">seq</span>(<span class="sc">-</span><span class="dv">5</span>, <span class="dv">5</span>, <span class="fl">0.1</span>),</span>
-<span id="cb834-3"><a href="model-comparison.html#cb834-3" aria-hidden="true" tabindex="-1"></a>                   <span class="at">x =</span> <span class="fu">dnorm</span>(y, <span class="at">sd =</span> <span class="dv">2</span>) <span class="sc">+</span> <span class="fl">3.75</span>)</span>
-<span id="cb834-4"><a href="model-comparison.html#cb834-4" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb834-5"><a href="model-comparison.html#cb834-5" aria-hidden="true" tabindex="-1"></a><span class="co"># show the residual plot together with the normal distribution</span></span>
-<span id="cb834-6"><a href="model-comparison.html#cb834-6" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.plot ,</span>
-<span id="cb834-7"><a href="model-comparison.html#cb834-7" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> fitted, <span class="at">y =</span> resid)) <span class="sc">+</span> </span>
-<span id="cb834-8"><a href="model-comparison.html#cb834-8" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>() <span class="sc">+</span></span>
-<span id="cb834-9"><a href="model-comparison.html#cb834-9" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_path</span>(<span class="at">data =</span> df.normal,</span>
-<span id="cb834-10"><a href="model-comparison.html#cb834-10" aria-hidden="true" tabindex="-1"></a>            <span class="fu">aes</span>(<span class="at">x =</span> x, <span class="at">y =</span> y),</span>
-<span id="cb834-11"><a href="model-comparison.html#cb834-11" aria-hidden="true" tabindex="-1"></a>            <span class="at">size =</span> <span class="dv">2</span>)</span></code></pre></div>
-<p><img src="15-model_comparison_files/figure-html/unnamed-chunk-17-1.png" width="672" /></p>
+<div class="sourceCode" id="cb30"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb30-1"><a href="model-comparison.html#cb30-1" aria-hidden="true" tabindex="-1"></a><span class="co"># define a normal distribution </span></span>
+<span id="cb30-2"><a href="model-comparison.html#cb30-2" aria-hidden="true" tabindex="-1"></a>df.normal <span class="ot">=</span> <span class="fu">tibble</span>(<span class="at">y =</span> <span class="fu">seq</span>(<span class="sc">-</span><span class="dv">5</span>, <span class="dv">5</span>, <span class="fl">0.1</span>),</span>
+<span id="cb30-3"><a href="model-comparison.html#cb30-3" aria-hidden="true" tabindex="-1"></a>                   <span class="at">x =</span> <span class="fu">dnorm</span>(y, <span class="at">sd =</span> <span class="dv">2</span>) <span class="sc">+</span> <span class="fl">3.75</span>)</span>
+<span id="cb30-4"><a href="model-comparison.html#cb30-4" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb30-5"><a href="model-comparison.html#cb30-5" aria-hidden="true" tabindex="-1"></a><span class="co"># show the residual plot together with the normal distribution</span></span>
+<span id="cb30-6"><a href="model-comparison.html#cb30-6" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.plot ,</span>
+<span id="cb30-7"><a href="model-comparison.html#cb30-7" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> fitted, <span class="at">y =</span> resid)) <span class="sc">+</span> </span>
+<span id="cb30-8"><a href="model-comparison.html#cb30-8" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>() <span class="sc">+</span></span>
+<span id="cb30-9"><a href="model-comparison.html#cb30-9" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_path</span>(<span class="at">data =</span> df.normal,</span>
+<span id="cb30-10"><a href="model-comparison.html#cb30-10" aria-hidden="true" tabindex="-1"></a>            <span class="fu">aes</span>(<span class="at">x =</span> x, <span class="at">y =</span> y),</span>
+<span id="cb30-11"><a href="model-comparison.html#cb30-11" aria-hidden="true" tabindex="-1"></a>            <span class="at">size =</span> <span class="dv">2</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-17-1.png" width="672" /></p>
 <p>To determine the likelihood of the data given the model <span class="math inline">\(\hat L\)</span>, we now calculate the likelihood of each point (with the <code>dnorm()</code> function), and then multiply the likelihood of each data point to get the overall likelihood. We can simply multiply the data points since we also assume that the data points are independent.
 Instead of multiplying likelihoods, we often sum the log likelihoods instead. This is because if we multiply many small values, the overall value gets to close to 0 so that computers get confused. By taking logs instead, we avoid these nasty precision errors.</p>
 <p>To better understand AIC and BIC, let’s calculate them by hand:</p>
-<div class="sourceCode" id="cb835"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb835-1"><a href="model-comparison.html#cb835-1" aria-hidden="true" tabindex="-1"></a><span class="co"># we first get the estimate of the standard deviation of the residuals </span></span>
-<span id="cb835-2"><a href="model-comparison.html#cb835-2" aria-hidden="true" tabindex="-1"></a>sigma <span class="ot">=</span> fit <span class="sc">%&gt;%</span> </span>
-<span id="cb835-3"><a href="model-comparison.html#cb835-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">glance</span>() <span class="sc">%&gt;%</span> </span>
-<span id="cb835-4"><a href="model-comparison.html#cb835-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">pull</span>(sigma)</span>
-<span id="cb835-5"><a href="model-comparison.html#cb835-5" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb835-6"><a href="model-comparison.html#cb835-6" aria-hidden="true" tabindex="-1"></a><span class="co"># then we calculate the log likelihood of the model </span></span>
-<span id="cb835-7"><a href="model-comparison.html#cb835-7" aria-hidden="true" tabindex="-1"></a>log_likelihood <span class="ot">=</span> fit <span class="sc">%&gt;%</span> </span>
-<span id="cb835-8"><a href="model-comparison.html#cb835-8" aria-hidden="true" tabindex="-1"></a>  <span class="fu">augment</span>() <span class="sc">%&gt;%</span> </span>
-<span id="cb835-9"><a href="model-comparison.html#cb835-9" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">likelihood =</span> <span class="fu">dnorm</span>(.resid, <span class="at">sd =</span> sigma)) <span class="sc">%&gt;%</span> </span>
-<span id="cb835-10"><a href="model-comparison.html#cb835-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">summarize</span>(<span class="at">logLik =</span> <span class="fu">sum</span>(<span class="fu">log</span>(likelihood))) <span class="sc">%&gt;%</span> </span>
-<span id="cb835-11"><a href="model-comparison.html#cb835-11" aria-hidden="true" tabindex="-1"></a>  <span class="fu">as.numeric</span>()</span>
-<span id="cb835-12"><a href="model-comparison.html#cb835-12" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb835-13"><a href="model-comparison.html#cb835-13" aria-hidden="true" tabindex="-1"></a><span class="co"># then we calculate AIC and BIC using the formulas introduced above</span></span>
-<span id="cb835-14"><a href="model-comparison.html#cb835-14" aria-hidden="true" tabindex="-1"></a>aic <span class="ot">=</span> <span class="dv">2</span><span class="sc">*</span><span class="dv">3</span> <span class="sc">-</span> <span class="dv">2</span> <span class="sc">*</span> log_likelihood</span>
-<span id="cb835-15"><a href="model-comparison.html#cb835-15" aria-hidden="true" tabindex="-1"></a>bic <span class="ot">=</span> <span class="fu">log</span>(<span class="fu">nrow</span>(df.example)) <span class="sc">*</span> <span class="dv">3</span> <span class="sc">-</span> <span class="dv">2</span> <span class="sc">*</span> log_likelihood</span>
-<span id="cb835-16"><a href="model-comparison.html#cb835-16" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb835-17"><a href="model-comparison.html#cb835-17" aria-hidden="true" tabindex="-1"></a><span class="fu">print</span>(aic)</span></code></pre></div>
+<div class="sourceCode" id="cb31"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb31-1"><a href="model-comparison.html#cb31-1" aria-hidden="true" tabindex="-1"></a><span class="co"># we first get the estimate of the standard deviation of the residuals </span></span>
+<span id="cb31-2"><a href="model-comparison.html#cb31-2" aria-hidden="true" tabindex="-1"></a>sigma <span class="ot">=</span> fit <span class="sc">%&gt;%</span> </span>
+<span id="cb31-3"><a href="model-comparison.html#cb31-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">glance</span>() <span class="sc">%&gt;%</span> </span>
+<span id="cb31-4"><a href="model-comparison.html#cb31-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">pull</span>(sigma)</span>
+<span id="cb31-5"><a href="model-comparison.html#cb31-5" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb31-6"><a href="model-comparison.html#cb31-6" aria-hidden="true" tabindex="-1"></a><span class="co"># then we calculate the log likelihood of the model </span></span>
+<span id="cb31-7"><a href="model-comparison.html#cb31-7" aria-hidden="true" tabindex="-1"></a>log_likelihood <span class="ot">=</span> fit <span class="sc">%&gt;%</span> </span>
+<span id="cb31-8"><a href="model-comparison.html#cb31-8" aria-hidden="true" tabindex="-1"></a>  <span class="fu">augment</span>() <span class="sc">%&gt;%</span> </span>
+<span id="cb31-9"><a href="model-comparison.html#cb31-9" aria-hidden="true" tabindex="-1"></a>  <span class="fu">mutate</span>(<span class="at">likelihood =</span> <span class="fu">dnorm</span>(.resid, <span class="at">sd =</span> sigma)) <span class="sc">%&gt;%</span> </span>
+<span id="cb31-10"><a href="model-comparison.html#cb31-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">summarize</span>(<span class="at">logLik =</span> <span class="fu">sum</span>(<span class="fu">log</span>(likelihood))) <span class="sc">%&gt;%</span> </span>
+<span id="cb31-11"><a href="model-comparison.html#cb31-11" aria-hidden="true" tabindex="-1"></a>  <span class="fu">as.numeric</span>()</span>
+<span id="cb31-12"><a href="model-comparison.html#cb31-12" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb31-13"><a href="model-comparison.html#cb31-13" aria-hidden="true" tabindex="-1"></a><span class="co"># then we calculate AIC and BIC using the formulas introduced above</span></span>
+<span id="cb31-14"><a href="model-comparison.html#cb31-14" aria-hidden="true" tabindex="-1"></a>aic <span class="ot">=</span> <span class="dv">2</span><span class="sc">*</span><span class="dv">3</span> <span class="sc">-</span> <span class="dv">2</span> <span class="sc">*</span> log_likelihood</span>
+<span id="cb31-15"><a href="model-comparison.html#cb31-15" aria-hidden="true" tabindex="-1"></a>bic <span class="ot">=</span> <span class="fu">log</span>(<span class="fu">nrow</span>(df.example)) <span class="sc">*</span> <span class="dv">3</span> <span class="sc">-</span> <span class="dv">2</span> <span class="sc">*</span> log_likelihood</span>
+<span id="cb31-16"><a href="model-comparison.html#cb31-16" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb31-17"><a href="model-comparison.html#cb31-17" aria-hidden="true" tabindex="-1"></a><span class="fu">print</span>(aic)</span></code></pre></div>
 <pre><code>[1] 75.58017</code></pre>
-<div class="sourceCode" id="cb837"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb837-1"><a href="model-comparison.html#cb837-1" aria-hidden="true" tabindex="-1"></a><span class="fu">print</span>(bic)</span></code></pre></div>
+<div class="sourceCode" id="cb33"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb33-1"><a href="model-comparison.html#cb33-1" aria-hidden="true" tabindex="-1"></a><span class="fu">print</span>(bic)</span></code></pre></div>
 <pre><code>[1] 78.56737</code></pre>
 <p>Cool! The values are the same as when we use the <code>glance()</code> function like so (except for a small difference due to rounding):</p>
-<div class="sourceCode" id="cb839"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb839-1"><a href="model-comparison.html#cb839-1" aria-hidden="true" tabindex="-1"></a>fit <span class="sc">%&gt;%</span> </span>
-<span id="cb839-2"><a href="model-comparison.html#cb839-2" aria-hidden="true" tabindex="-1"></a>  <span class="fu">glance</span>() <span class="sc">%&gt;%</span> </span>
-<span id="cb839-3"><a href="model-comparison.html#cb839-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">select</span>(AIC, BIC)</span></code></pre></div>
+<div class="sourceCode" id="cb35"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb35-1"><a href="model-comparison.html#cb35-1" aria-hidden="true" tabindex="-1"></a>fit <span class="sc">%&gt;%</span> </span>
+<span id="cb35-2"><a href="model-comparison.html#cb35-2" aria-hidden="true" tabindex="-1"></a>  <span class="fu">glance</span>() <span class="sc">%&gt;%</span> </span>
+<span id="cb35-3"><a href="model-comparison.html#cb35-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">select</span>(AIC, BIC)</span></code></pre></div>
 <pre><code># A tibble: 1 × 2
     AIC   BIC
   &lt;dbl&gt; &lt;dbl&gt;
 1  75.5  78.5</code></pre>
 <div id="log-is-your-friend" class="section level4 hasAnchor" number="15.3.5.1">
 <h4><span class="header-section-number">15.3.5.1</span> log() is your friend<a href="model-comparison.html#log-is-your-friend" class="anchor-section" aria-label="Anchor link to header"></a></h4>
-<div class="sourceCode" id="cb841"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb841-1"><a href="model-comparison.html#cb841-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> <span class="fu">tibble</span>(<span class="at">x =</span> <span class="fu">c</span>(<span class="dv">0</span>, <span class="dv">1</span>)),</span>
-<span id="cb841-2"><a href="model-comparison.html#cb841-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> x)) <span class="sc">+</span> </span>
-<span id="cb841-3"><a href="model-comparison.html#cb841-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_function</span>(<span class="at">fun =</span> <span class="st">&quot;log&quot;</span>,</span>
-<span id="cb841-4"><a href="model-comparison.html#cb841-4" aria-hidden="true" tabindex="-1"></a>                <span class="at">size =</span> <span class="dv">1</span>) <span class="sc">+</span></span>
-<span id="cb841-5"><a href="model-comparison.html#cb841-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">labs</span>(<span class="at">x =</span> <span class="st">&quot;probability&quot;</span>,</span>
-<span id="cb841-6"><a href="model-comparison.html#cb841-6" aria-hidden="true" tabindex="-1"></a>       <span class="at">y =</span> <span class="st">&quot;log(probability)&quot;</span>) <span class="sc">+</span></span>
-<span id="cb841-7"><a href="model-comparison.html#cb841-7" aria-hidden="true" tabindex="-1"></a>  <span class="fu">theme</span>(<span class="at">axis.text =</span> <span class="fu">element_text</span>(<span class="at">size =</span> <span class="dv">24</span>),</span>
-<span id="cb841-8"><a href="model-comparison.html#cb841-8" aria-hidden="true" tabindex="-1"></a>        <span class="at">axis.title =</span> <span class="fu">element_text</span>(<span class="at">size =</span> <span class="dv">26</span>))</span></code></pre></div>
-<p><img src="15-model_comparison_files/figure-html/unnamed-chunk-20-1.png" width="672" /></p>
+<div class="sourceCode" id="cb37"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb37-1"><a href="model-comparison.html#cb37-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> <span class="fu">tibble</span>(<span class="at">x =</span> <span class="fu">c</span>(<span class="dv">0</span>, <span class="dv">1</span>)),</span>
+<span id="cb37-2"><a href="model-comparison.html#cb37-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> x)) <span class="sc">+</span> </span>
+<span id="cb37-3"><a href="model-comparison.html#cb37-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_function</span>(<span class="at">fun =</span> <span class="st">&quot;log&quot;</span>,</span>
+<span id="cb37-4"><a href="model-comparison.html#cb37-4" aria-hidden="true" tabindex="-1"></a>                <span class="at">size =</span> <span class="dv">1</span>) <span class="sc">+</span></span>
+<span id="cb37-5"><a href="model-comparison.html#cb37-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">labs</span>(<span class="at">x =</span> <span class="st">&quot;probability&quot;</span>,</span>
+<span id="cb37-6"><a href="model-comparison.html#cb37-6" aria-hidden="true" tabindex="-1"></a>       <span class="at">y =</span> <span class="st">&quot;log(probability)&quot;</span>) <span class="sc">+</span></span>
+<span id="cb37-7"><a href="model-comparison.html#cb37-7" aria-hidden="true" tabindex="-1"></a>  <span class="fu">theme</span>(<span class="at">axis.text =</span> <span class="fu">element_text</span>(<span class="at">size =</span> <span class="dv">24</span>),</span>
+<span id="cb37-8"><a href="model-comparison.html#cb37-8" aria-hidden="true" tabindex="-1"></a>        <span class="at">axis.title =</span> <span class="fu">element_text</span>(<span class="at">size =</span> <span class="dv">26</span>))</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-20-1.png" width="672" /></p>
 </div>
 </div>
 </div>
@@ -1410,7 +1378,7 @@ <h3><span class="header-section-number">15.4.2</span> Reading<a href="model-comp
 <div id="session-info-14" class="section level2 hasAnchor" number="15.5">
 <h2><span class="header-section-number">15.5</span> Session info<a href="model-comparison.html#session-info-14" class="anchor-section" aria-label="Anchor link to header"></a></h2>
 <p>Information about this R session including which version of R was used, and what packages were loaded.</p>
-<div class="sourceCode" id="cb842"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb842-1"><a href="model-comparison.html#cb842-1" aria-hidden="true" tabindex="-1"></a><span class="fu">sessionInfo</span>()</span></code></pre></div>
+<div class="sourceCode" id="cb38"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb38-1"><a href="model-comparison.html#cb38-1" aria-hidden="true" tabindex="-1"></a><span class="fu">sessionInfo</span>()</span></code></pre></div>
 <pre><code>R version 4.1.2 (2021-11-01)
 Platform: x86_64-apple-darwin17.0 (64-bit)
 Running under: macOS Big Sur 10.16
diff --git a/docs/psych252_files/figure-html/bad-figure-1.png b/docs/psych252_files/figure-html/bad-figure-1.png
index 85fc645..2500dc4 100644
Binary files a/docs/psych252_files/figure-html/bad-figure-1.png and b/docs/psych252_files/figure-html/bad-figure-1.png differ
diff --git a/docs/psych252_files/figure-html/errorbars-normal-1.png b/docs/psych252_files/figure-html/errorbars-normal-1.png
index 5e5cd59..138783c 100644
Binary files a/docs/psych252_files/figure-html/errorbars-normal-1.png and b/docs/psych252_files/figure-html/errorbars-normal-1.png differ
diff --git a/docs/psych252_files/figure-html/good-figure-1.png b/docs/psych252_files/figure-html/good-figure-1.png
index 5b6b72b..e33c2f0 100644
Binary files a/docs/psych252_files/figure-html/good-figure-1.png and b/docs/psych252_files/figure-html/good-figure-1.png differ
diff --git a/docs/psych252_files/figure-html/stretching-it-1.png b/docs/psych252_files/figure-html/stretching-it-1.png
index 8763fa1..e72ef3a 100644
Binary files a/docs/psych252_files/figure-html/stretching-it-1.png and b/docs/psych252_files/figure-html/stretching-it-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-11-1.png b/docs/psych252_files/figure-html/unnamed-chunk-11-1.png
index 987bdf4..537f034 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-11-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-11-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-12-1.png b/docs/psych252_files/figure-html/unnamed-chunk-12-1.png
index 3f5ce47..f41199d 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-12-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-12-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-13-1.png b/docs/psych252_files/figure-html/unnamed-chunk-13-1.png
index e6ec9cc..f63e7b2 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-13-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-13-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-15-1.png b/docs/psych252_files/figure-html/unnamed-chunk-15-1.png
index 0d3c0c1..4346651 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-15-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-15-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-16-1.png b/docs/psych252_files/figure-html/unnamed-chunk-16-1.png
index 910c50e..c6ca65d 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-16-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-16-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-17-1.png b/docs/psych252_files/figure-html/unnamed-chunk-17-1.png
index c410901..0b8ce69 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-17-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-17-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-20-1.png b/docs/psych252_files/figure-html/unnamed-chunk-20-1.png
index 9f2bc45..1b47727 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-20-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-20-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-21-1.png b/docs/psych252_files/figure-html/unnamed-chunk-21-1.png
index 97757d8..948d191 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-21-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-21-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-22-1.png b/docs/psych252_files/figure-html/unnamed-chunk-22-1.png
index 1985a9a..58e8265 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-22-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-22-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-26-1.png b/docs/psych252_files/figure-html/unnamed-chunk-26-1.png
index a05755d..eb139cb 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-26-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-26-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-27-1.png b/docs/psych252_files/figure-html/unnamed-chunk-27-1.png
index 472bd6a..9457613 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-27-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-27-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-28-1.png b/docs/psych252_files/figure-html/unnamed-chunk-28-1.png
index 4a2b739..ffda002 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-28-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-28-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-29-1.png b/docs/psych252_files/figure-html/unnamed-chunk-29-1.png
index c486791..9c072c4 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-29-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-29-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-3-1.png b/docs/psych252_files/figure-html/unnamed-chunk-3-1.png
index 95bac07..507d838 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-3-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-3-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-30-1.png b/docs/psych252_files/figure-html/unnamed-chunk-30-1.png
index 4a4e3b5..fd32d86 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-30-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-30-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-34-1.png b/docs/psych252_files/figure-html/unnamed-chunk-34-1.png
index 458ceac..ff3aa65 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-34-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-34-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-35-1.png b/docs/psych252_files/figure-html/unnamed-chunk-35-1.png
index 38cbe30..905f88c 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-35-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-35-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-36-1.png b/docs/psych252_files/figure-html/unnamed-chunk-36-1.png
index ffe26e6..b487769 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-36-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-36-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-37-1.png b/docs/psych252_files/figure-html/unnamed-chunk-37-1.png
index 9637cec..dbf6f2a 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-37-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-37-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-4-1.png b/docs/psych252_files/figure-html/unnamed-chunk-4-1.png
index bac90d1..d966a09 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-4-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-4-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-6-1.png b/docs/psych252_files/figure-html/unnamed-chunk-6-1.png
index e7c94ec..59a3cfa 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-6-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-6-1.png differ
diff --git a/docs/psych252_files/figure-html/unnamed-chunk-8-1.png b/docs/psych252_files/figure-html/unnamed-chunk-8-1.png
index 52d8146..44302c0 100644
Binary files a/docs/psych252_files/figure-html/unnamed-chunk-8-1.png and b/docs/psych252_files/figure-html/unnamed-chunk-8-1.png differ
diff --git a/docs/reference-keys.txt b/docs/reference-keys.txt
index 2852d68..13d3026 100644
--- a/docs/reference-keys.txt
+++ b/docs/reference-keys.txt
@@ -711,3 +711,5 @@ session-info-25
 cheatsheets-6
 statistics
 r
+breakout-room-exercise
+credible-interval-vs.-confidence-interval
diff --git a/docs/search_index.json b/docs/search_index.json
index 56b5375..0777f07 100644
--- a/docs/search_index.json
+++ b/docs/search_index.json
@@ -1 +1 @@
-[["index.html", "Psych 252: Statistical Methods for Behavioral and Social Sciences Preface Course description Course homepage", " Psych 252: Statistical Methods for Behavioral and Social Sciences Tobias Gerstenberg 2022-05-10 Preface This book contains the course notes for Psych 252. The book is not intended to be self-explanatory and instead should be used in combination with the course lectures posted here. If you have any questions about the notes, please feel free to contact me at: gerstenberg@stanford.edu or post an issue on the book’s github repository. Course description This course offers an introduction to advanced topics in statistics with the focus of understanding data in the behavioral and social sciences. It is a practical course in which learning statistical concepts and building models in R go hand in hand. The course is organized into three parts: In the first part, we will learn how to visualize, wrangle, and simulate data in R. In the second part, we will cover topics in frequentist statistics (such as multiple regression, logistic regression, and mixed effects models) using the general linear model as an organizing framework. We will learn how to compare models using simulation methods such as bootstrapping and cross-validation. In the third part, we will focus on Bayesian data analysis as an alternative framework for answering statistical questions. Course homepage https://psych252.github.io/ "],["introduction.html", "Chapter 1 Introduction 1.1 Thanks 1.2 List of R packages used in this book 1.3 Session info", " Chapter 1 Introduction 1.1 Thanks Various people have helped in the process of putting together these materials (either knowingly, or unknowingly). Big thanks go to: Alexandra Chouldechova Andrew Heiss Ben Baumer Benoit Monin David Lagnado Ewart Thomas Henrik Singmann Julian Jara-Ettinger Justin Gardner Kevin Smith Lisa DeBruine Maarten Speekenbrink Matthew Kay Matthew Salganik Mika Braginsky Mike Frank Mine Çetinkaya-Rundel Nick C. Huntington-Klein Patrick Mair Paul-Christian Bürkner Peter Cushner Mohanty Richard McElreath Russ Poldrack Stephen Dewitt Solomon Kurz Tom Hardwicke Tristan Mahr Special thanks go to my teaching teams: 2022: Ari Beller Sarah Wu Chengxu Zhuang 2021: Andrew Nam Catherine Thomas Jon Walters Dan Yamins 2020: Tyler Bonnen Andrew Nam Jinxiao Zhang 2019: Andrew Lampinen Mona Rosenke Shao-Fang (Pam) Wang 1.2 List of R packages used in this book # RMarkdown library(&quot;knitr&quot;) # markdown things library(&quot;kableExtra&quot;) # for nicely formatted tables # Datasets library(&quot;gapminder&quot;) # data available from Gapminder.org library(&quot;NHANES&quot;) # data set library(&quot;titanic&quot;) # titanic dataset # Data manipulation library(&quot;arrangements&quot;) # fast generators and iterators for permutations, combinations and partitions library(&quot;magrittr&quot;) # for wrangling library(&quot;tidyverse&quot;) # everything else # Visualization library(&quot;patchwork&quot;) # making figure panels library(&quot;ggpol&quot;) # for making fancy boxplots library(&quot;ggridges&quot;) # for making joyplots library(&quot;gganimate&quot;) # for making animations library(&quot;GGally&quot;) # for pairs plot library(&quot;ggrepel&quot;) # for labels in ggplots library(&quot;corrr&quot;) # for calculating correlations between many variables library(&quot;corrplot&quot;) # for plotting correlations library(&quot;DiagrammeR&quot;) # for drawing diagrams library(&quot;ggeffects&quot;) # for visualizing effects library(&quot;bayesplot&quot;) # for visualization of Bayesian model fits # Modeling library(&quot;afex&quot;) # also for running ANOVAs library(&quot;lme4&quot;) # mixed effects models library(&quot;emmeans&quot;) # comparing estimated marginal means library(&quot;broom.mixed&quot;) # getting tidy mixed model summaries library(&quot;janitor&quot;) # cleaning variable names library(&quot;car&quot;) # for running ANOVAs library(&quot;rstanarm&quot;) # for Bayesian models library(&quot;greta&quot;) # Bayesian models library(&quot;tidybayes&quot;) # tidying up results from Bayesian models library(&quot;boot&quot;) # bootstrapping library(&quot;modelr&quot;) # cross-validation and bootstrapping library(&quot;mediation&quot;) # for mediation and moderation analysis library(&quot;multilevel&quot;) # Sobel test library(&quot;extraDistr&quot;) # additional probability distributions library(&quot;effects&quot;) # for showing effects in linear, generalized linear, and other models library(&quot;brms&quot;) # Bayesian regression library(&quot;parameters&quot;) # For extracting parameters # Misc library(&quot;tictoc&quot;) # timing things library(&quot;MASS&quot;) # various useful functions (e.g. bootstrapped confidence intervals) library(&quot;lsr&quot;) # for computing effect size measures library(&quot;extrafont&quot;) # additional fonts library(&quot;pwr&quot;) # for power calculations library(&quot;arrangements&quot;) # fast generators and iterators for permutations, combinations and partitions library(&quot;stargazer&quot;) # for regression tables library(&quot;sjPlot&quot;) # for regression tables library(&quot;xtable&quot;) # for tables 1.3 Session info ## R version 4.1.2 (2021-11-01) ## Platform: x86_64-apple-darwin17.0 (64-bit) ## Running under: macOS Big Sur 10.16 ## ## Matrix products: default ## BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib ## LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib ## ## locale: ## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 ## ## attached base packages: ## [1] stats graphics grDevices utils datasets methods base ## ## loaded via a namespace (and not attached): ## [1] bookdown_0.26 digest_0.6.29 R6_2.5.1 jsonlite_1.8.0 ## [5] magrittr_2.0.3 evaluate_0.15 stringi_1.7.6 rlang_1.0.2 ## [9] cli_3.3.0 jquerylib_0.1.4 bslib_0.3.1 rmarkdown_2.14 ## [13] tools_4.1.2 stringr_1.4.0 xfun_0.30 yaml_2.3.5 ## [17] fastmap_1.1.0 compiler_4.1.2 htmltools_0.5.2 knitr_1.39 ## [21] sass_0.4.1 "],["visualization-1.html", "Chapter 2 Visualization 1 2.1 Learning goals 2.2 Load packages 2.3 Why visualize data? 2.4 Some basics 2.5 Data visualization 2.6 Additional resources 2.7 Session info", " Chapter 2 Visualization 1 In this lecture, we will take a look at how to visualize data using the powerful ggplot2 package. We will use ggplot2 a lot throughout the rest of the course! 2.1 Learning goals Take a look at some suboptimal plots, and think about how to make them better. Get familiar with the RStudio interface. Understand the general philosophy behind ggplot2 – a grammar of graphics. Understand the mapping from data to geoms in ggplot2. Create informative figures using grouping and facets. 2.2 Load packages Let’s first load the packages that we need for this chapter. You can click on the green arrow to execute the code chunk below. library(&quot;knitr&quot;) # for rendering the RMarkdown file library(&quot;tidyverse&quot;) # for plotting (and many more cool things we&#39;ll discover later) # these options here change the formatting of how comments are rendered # in RMarkdown opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) The tidyverse is a collection of packages that includes ggplot2. 2.3 Why visualize data? The greatest value of a picture is when it forces us to notice what we never expected to see. — John Tukey There is no single statistical tool that is as powerful as a well‐chosen graph. (Chambers et al. 1983) …make both calculations and graphs. Both sorts of output should be studied; each will contribute to understanding. (Anscombe 1973) Figure 2.1: Anscombe’s quartet. Anscombe’s quartet in Figure 2.1 (left side) illustrates the importance of visualizing data. Even though the datasets I-IV have the same summary statistics (mean, standard deviation, correlation), they are importantly different from each other. On the right side, we have four data sets with the same summary statistics that are very similar to each other. Figure 2.2: The Pearson’s \\(r\\) correlation coefficient is the same for all of these datasets. Source: Data Visualization – A practical introduction by Kieran Healy All the datasets in Figure 2.2 share the same correlation coefficient. However, again, they are very different from each other. Figure 2.3: The Datasaurus Dozen. While different in appearance, each dataset has the same summary statistics to two decimal places (mean, standard deviation, and Pearson’s correlation). The data sets in Figure 2.3 all share the same summary statistics. Clearly, the data sets are not the same though. Tip: Always plot the data first! Here is the paper from which I took Figure 2.3. It explains how the figures were generated and shows more examples for how summary statistics and some kinds of plots are insufficient to get a good sense for what’s going on in the data. 2.4 Some basics 2.4.1 Setting up RStudio Figure 2.4: General preferences. Make sure that: Restore .RData into workspace at startup is unselected Save workspace to .RData on exit is set to Never This can otherwise cause problems with reproducibility and weird behavior between R sessions because certain things may still be saved in your workspace. Figure 2.5: Code window preferences. Make sure that: Soft-wrap R source files is selected This way you don’t have to scroll horizontally. At the same time, avoid writing long single lines of code. For example, instead of writing code like so: ggplot(data = diamonds, mapping = aes(x = cut, y = price)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;, fill = &quot;lightblue&quot;, width = 0.85) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1.5) + labs(title = &quot;Price as a function of quality of cut&quot;, subtitle = &quot;Note: The price is in US dollars&quot;, tag = &quot;A&quot;, x = &quot;Quality of the cut&quot;, y = &quot;Price&quot;) You may want to write it this way instead: ggplot(data = diamonds, mapping = aes(x = cut, y = price)) + # display the means stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;, fill = &quot;lightblue&quot;, width = 0.85) + # display the error bars stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1.5) + # change labels labs(title = &quot;Price as a function of quality of cut&quot;, subtitle = &quot;Note: The price is in US dollars&quot;, # we might want to change this later tag = &quot;A&quot;, x = &quot;Quality of the cut&quot;, y = &quot;Price&quot;) This makes it much easier to see what’s going on, and you can easily add comments to individual lines of code. Tip: If a function has more than two arguments put each argument on a new line. RStudio makes it easy to write nice code. It figures out where to put the next line of code when you press ENTER. And if things ever get messy, just select the code of interest and hit cmd + i to re-indent the code. Here are some more resources with tips for how to write nice code in R: Advanced R style guide Tip: Use a consistent coding style. This makes reading code and debugging much easier! 2.4.2 Getting help There are three simple ways to get help in R. You can either put a ? in front of the function you’d like to learn more about, or use the help() function. ?print help(&quot;print&quot;) Tip: To see the help file, hover over a function (or dataset) with the mouse (or select the text) and then press F1. I recommend using F1 to get to help files – it’s the fastest way! R help files can sometimes look a little cryptic. Most R help files have the following sections (copied from here): Title: A one-sentence overview of the function. Description: An introduction to the high-level objectives of the function. Usage: A description of the syntax of the function (in other words, how the function is called). This is where you find all the arguments that you can supply to the function, as well as any default values of these arguments. Arguments: A description of each argument. Usually this includes a specification of the class (for example, character, numeric, list, and so on). This section is an important one to understand, because arguments are frequently a cause of errors in R. Details: Extended details about how the function works, provides longer descriptions of the various ways to call the function (if applicable), and a longer discussion of the arguments. Value: A description of the class of the value returned by the function. See also: Links to other relevant functions. In most of the R editors, you can click these links to read the Help files for these functions. Examples: Worked examples of real R code that you can paste into your console and run. Here is the help file for the print() function: Figure 2.6: Help file for the print() function. 2.4.3 R Markdown infos An RMarkdown file has four key components: YAML header Headings to structure the document Text Code chunks The YAML (Yet Another Markdown Language) header specifies general options such as whether you’d like to have a table of content displayed, and in what output format you want to create your report (e.g. html or pdf). Notice that the YAML header cares about indentation, so make sure to get that right! Headings are very useful for structuring your RMarkdown file. For your reports, it’s often a good idea to have one header for each code chunk. The outline viewer here on the right is great for navigating large analysis files. Text is self-explanatory. Code chunks is where the coding happens. You can add one via the Insert button above, or via the shortcut cmd + option + i (the much cooler way of doing it!) Code chunks can have code chunk options which we can set by clicking on the cog symbol on the right. You can also give code chunks a name, so that we can refer to it in text. I’ve named the one above “another-code-chunk”. Make sure to have no white space or underscore in a code chunk name. 2.4.4 Helpful keyboard shortcuts cmd + enter: run selected code cmd + shift + enter: run code chunk cmd + i: re-indent selected code cmd + shift + c: comment/uncomment several lines of code cmd + shift + d: duplicate line underneath set up your own shortcuts to do useful things like switch tabs jump up and down between code chunks … 2.5 Data visualization We will use the ggplot2 package to visualize data. By the end of next class, you’ll be able to make a figure like this: Figure 2.7: What a nice figure! Now let’s figure out (pun intended!) how to get there. 2.5.1 Setting up a plot Let’s first get some data. df.diamonds = diamonds The diamonds dataset comes with the ggplot2 package. We can get a description of the dataset by running the following command: ?diamonds Above, we assigned the diamonds dataset to the variable df.diamonds so that we can see it in the data explorer. Let’s take a look at the full dataset by clicking on it in the explorer. Tip: You can view a data frame by highlighting the text in the editor (or simply moving the mouse above the text), and then pressing F2. The df.diamonds data frame contains information about almost 60,000 diamonds, including their price, carat value, size, etc. Let’s use visualization to get a better sense for this dataset. We start by setting up the plot. To do so, we pass a data frame to the function ggplot() in the following way. ggplot(data = df.diamonds) This, by itself, won’t do anything yet. We also need to specify what to plot. Let’s take a look at how much diamonds of different color cost. The help file says that diamonds labeled D have the best color, and diamonds labeled J the worst color. Let’s make a bar plot that shows the average price of diamonds for different colors. We do so via specifying a mapping from the data to the plot aesthetics with the function aes(). We need to tell aes() what we would like to display on the x-axis, and the y-axis of the plot. ggplot(data = df.diamonds, mapping = aes(x = color, y = price)) Here, we specified that we want to plot color on the x-axis, and price on the y-axis. As you can see, ggplot2 has already figured out how to label the axes. However, we still need to specify how to plot it. 2.5.2 Bar plot Let’s make a bar graph: ggplot(data = df.diamonds, mapping = aes(x = color, y = price)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;) Neat! Three lines of code produce an almost-publication-ready plot (to be published in the Proceedings of Unnecessary Diamonds)! Note how we used a + at the end of the first line of code to specify that there will be more. This is a very powerful idea underlying ggplot2. We can start simple and keep adding things to the plot step by step. We used the stat_summary() function to define what we want to plot (the “mean”), and how (as a “bar” chart). Let’s take a closer look at that function. help(stat_summary) Not the the easiest help file … We supplied two arguments to the function, fun = and geom =. The fun argument specifies what function we’d like to apply to the data for each value of x. Here, we said that we would like to take the mean and we specified that as a string. The geom (= geometric object) argument specifies how we would like to plot the result, namely as a “bar” plot. Instead of showing the “mean”, we could also show the “median” instead. ggplot(data = df.diamonds, mapping = aes(x = color, y = price)) + stat_summary(fun = &quot;median&quot;, geom = &quot;bar&quot;) And instead of making a bar plot, we could plot some points. ggplot(df.diamonds, aes(x = color, y = price)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;) Tip: Take a look here to see what other geoms ggplot2 supports. Somewhat surprisingly, diamonds with the best color (D) are not the most expensive ones. What’s going on here? We’ll need to do some more exploration to figure this out. 2.5.3 Setting the default plot theme Before moving on, let’s set a different default theme for our plots. Personally, I’m not a big fan of the gray background and the white grid lines. Also, the default size of the text should be bigger. We can change the default theme using the theme_set() function like so: theme_set(theme_classic() + # set the theme theme(text = element_text(size = 20))) # set the default text size From now on, all our plots will use what’s specified in theme_classic(), and the default text size will be larger, too. For any individual plot, we can still override these settings. 2.5.4 Scatter plot I don’t know much about diamonds, but I do know that diamonds with a higher carat value tend to be more expensive. color was a discrete variable with seven different values. carat, however, is a continuous variable. We want to see how the price of diamonds differs as a function of the carat value. Since we are interested in the relationship between two continuous variables, plotting a bar graph won’t work. Instead, let’s make a scatter plot. Let’s put the carat value on the x-axis, and the price on the y-axis. ggplot(data = df.diamonds, mapping = aes(x = carat, y = price)) + geom_point() Figure 2.8: Scatterplot. Cool! That looks sensible. Diamonds with a higher carat value tend to have a higher price. Our dataset has 53940 rows. So the plot actually shows 53940 circles even though we can’t see all of them since they overlap. Let’s make some progress on trying to figure out why the diamonds with the better color weren’t the most expensive ones on average. We’ll add some color to the scatter plot in Figure 2.8. We color each of the points based on the diamond’s color. To do so, we pass another argument to the aesthetics of the plot via aes(). ggplot(data = df.diamonds, mapping = aes(x = carat, y = price, color = color)) + geom_point() Figure 2.9: Scatterplot with color. Aha! Now we’ve got some color. Notice how in Figure 2.9 ggplot2 added a legend for us, thanks! We’ll see later how to play around with legends. Form just eye-balling the plot, it looks like the diamonds with the best color (D) tended to have a lower carat value, and the ones with the worst color (J), tended to have the highest carat values. So this is why diamonds with better colors are less expensive – these diamonds have a lower carat value overall. There are many other things that we can define in aes(). Take a quick look at the vignette: vignette(&quot;ggplot2-specs&quot;) 2.5.4.1 Practice plot 1 Make a scatter plot that shows the relationship between the variables depth (on the x-axis), and table (on the y-axis). Take a look at the description for the diamonds dataset so you know what these different variables mean. Your plot should look like the one shown in Figure 2.10. # make practice plot 1 here include_graphics(&quot;figures/vis1_practice_plot1.png&quot;) Figure 2.10: Practice plot 1. Advanced: A neat trick to get a better sense for the data here is to add transparency. Your plot should look like the one shown in Figure 2.11. # make advanced practice plot 1 here include_graphics(&quot;figures/vis1_practice_plot1a.png&quot;) Figure 2.11: Practice plot 1. 2.5.5 Line plot What else do we know about the diamonds? We actually know the quality of how they were cut. The cut variable ranges from “Fair” to “Ideal”. First, let’s take a look at the relationship between cut and price. This time, we’ll make a line plot instead of a bar plot (just because we can). ggplot(data = df.diamonds, mapping = aes(x = cut, y = price)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;line&quot;) geom_path: Each group consists of only one observation. Do you need to adjust the group aesthetic? Oops! All we did is that we replaced x = color with x = cut, and geom = \"bar\" with geom = \"line\". However, the plot doesn’t look like expected (i.e. there is no real plot). What happened here? The reason is that the line plot needs to know which points to connect. The error message tells us that each group consists of only one observation. Let’s adjust the group aesthetic to fix this. ggplot(data = df.diamonds, mapping = aes(x = cut, y = price, group = 1)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;line&quot;) By adding the parameter group = 1 to mapping = aes(), we specify that we would like all the levels in x = cut to be treated as coming from the same group. The reason for this is that cut (our x-axis variable) is a factor (and not a numeric variable), so, by default, ggplot2 tries to draw a separate line for each factor level. We’ll learn more about grouping below (and about factors later). Interestingly, there is no simple relationship between the quality of the cut and the price of the diamond. In fact, “Ideal” diamonds tend to be cheapest. 2.5.6 Adding error bars We often don’t just want to show the means but also give a sense for how much the data varies. ggplot2 has some convenient ways of specifying error bars. Let’s take a look at how much price varies as a function of clarity (another variable in our diamonds data frame). ggplot(data = df.diamonds, mapping = aes(x = clarity, y = price)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;pointrange&quot;) Figure 2.12: Relationship between diamond clarity and price. Error bars indicate 95% bootstrapped confidence intervals. Here we have it. The average price of our diamonds for different levels of clarity together with bootstrapped 95% confidence intervals. How do we know that we have 95% confidence intervals? That’s what mean_cl_boot() computes as a default. Let’s take a look at that function: help(mean_cl_boot) Note that I had to use the fun.data = argument here instead of fun = because the mean_cl_boot() function produces three data points for each value of the x-axis (the mean, lower, and upper confidence interval). 2.5.7 Order matters The order in which we add geoms to a ggplot matters! Generally, we want to plot error bars before the points that represent the means. To illustrate, let’s set the color in which we show the means to “red”. ggplot(data = df.diamonds, mapping = aes(x = clarity, y = price)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;, color = &quot;red&quot;) Figure 2.13: This figure looks good. Error bars and means are drawn in the correct order. Figure 2.13 looks good. # I&#39;ve changed the order in which the means and error bars are drawn. ggplot(df.diamonds, aes(x = clarity, y = price)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;, color = &quot;red&quot;) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;) Figure 2.14: This figure looks bad. Error bars and means are drawn in the incorrect order. Figure 2.14 doesn’t look good. The error bars are on top of the points that represent the means. One cool feature about using stat_summary() is that we did not have to change anything about the data frame that we used to make the plots. We directly used our raw data instead of having to make separate data frames that contain the relevant information (such as the means and the confidence intervals). You may not remember exactly what confidence intervals actually are. Don’t worry! We’ll have a recap later in class. Let’s take a look at two more principles for plotting data that are extremely helpful: groups and facets. But before, another practice plot. 2.5.7.1 Practice plot 2 Make a bar plot that shows the average price of diamonds (on the y-axis) as a function of their clarity (on the x-axis). Also add error bars. Your plot should look like the one shown in Figure 2.15. # make practice plot 2 here include_graphics(&quot;figures/vis1_practice_plot2.png&quot;) Figure 2.15: Practice plot 2. Advanced: Try to make the plot shown in Figure 2.16. # make advanced practice plot 2 here include_graphics(&quot;figures/vis1_practice_plot2a.png&quot;) Figure 2.16: Practice plot 2. 2.5.8 Grouping data Grouping in ggplot2 is a very powerful idea. It allows us to plot subsets of the data – again without the need to make separate data frames first. Let’s make a plot that shows the relationship between price and color separately for the different qualities of cut. ggplot(data = df.diamonds, mapping = aes(x = color, y = price, group = cut)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;line&quot;) Well, we got some separate lines here but we don’t know which line corresponds to which cut. Let’s add some color! ggplot(data = df.diamonds, mapping = aes(x = color, y = price, group = cut, color = cut)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;line&quot;, size = 2) Nice! In addition to adding color, I’ve made the lines a little thicker here by setting the size argument to 2. Grouping is very useful for bar plots. Let’s take a look at how the average price of diamonds looks like taking into account both cut and color (I know – exciting times!). Let’s put the color on the x-axis and then group by the cut. ggplot(data = df.diamonds, mapping = aes(x = color, y = price, group = cut, color = cut)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;) That’s a fail! Several things went wrong here. All the bars are gray and only their outline is colored differently. Instead we want the bars to have a different color. For that we need to specify the fill argument rather than the color argument! But things are worse. The bars currently are shown on top of each other. Instead, we’d like to put them next to each other. Here is how we can do that: ggplot(data = df.diamonds, mapping = aes(x = color, y = price, group = cut, fill = cut)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, position = position_dodge()) Neato! We’ve changed the color argument to fill, and have added the position = position_dodge() argument to the stat_summary() call. This argument makes it such that the bars are nicely dodged next to each other. Let’s add some error bars just for kicks. ggplot(data = df.diamonds, mapping = aes(x = color, y = price, group = cut, fill = cut)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, position = position_dodge(width = 0.9), color = &quot;black&quot;) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, position = position_dodge(width = 0.9)) Voila! Now with error bars. Note that we’ve added the width = 0.9 argument to position_dodge(). Somehow R was complaining when this was not defined for geom “linerange”. I’ve also added some outline to the bars by including the argument color = \"black\". I think it looks nicer this way. So, still somewhat surprisingly, diamonds with the worst color (J) are more expensive than dimanods with the best color (D), and diamonds with better cuts are not necessarily more expensive. 2.5.8.1 Practice plot 3 Recreate the plot shown in Figure 2.17. # make practice plot 3 here include_graphics(&quot;figures/vis1_practice_plot3.png&quot;) Figure 2.17: Practice plot 3. Advanced: Try to recreate the plot show in in Figure 2.18. # make advanced practice plot 3 here include_graphics(&quot;figures/vis1_practice_plot3a.png&quot;) Figure 2.18: Practice plot 3. 2.5.9 Making facets Having too much information in a single plot can be overwhelming. The previous plot is already pretty busy. Facets are a nice way of splitting up plots and showing information in separate panels. Let’s take a look at how wide these diamonds tend to be. The width in mm is given in the y column of the diamonds data frame. We’ll make a histogram first. To make a histogram, the only aesthetic we needed to specify is x. ggplot(data = df.diamonds, mapping = aes(x = y)) + geom_histogram() `stat_bin()` using `bins = 30`. Pick better value with `binwidth`. That looks bad! Let’s pick a different value for the width of the bins in the histogram. ggplot(data = df.diamonds, mapping = aes(x = y)) + geom_histogram(binwidth = 0.1) Still bad. There seems to be an outlier diamond that happens to be almost 60 mm wide, while most of the rest is much narrower. One option would be to remove the outlier from the data before plotting it. But generally, we don’t want to make new data frames. Instead, let’s just limit what data we show in the plot. ggplot(data = df.diamonds, mapping = aes(x = y)) + geom_histogram(binwidth = 0.1) + coord_cartesian(xlim = c(3, 10)) I’ve used the coord_cartesian() function to restrict the range of data to show by passing a minimum and maximum to the xlim argument. This looks better now. Instead of histograms, we can also plot a density fitted to the distribution. ggplot(data = df.diamonds, mapping = aes(x = y)) + geom_density() + coord_cartesian(xlim = c(3, 10)) Looks pretty similar to our histogram above! Just like we can play around with the binwidth of the histogram, we can change the smoothing bandwidth of the kernel that is used to create the histogram. Here is a histogram with a much wider bandwidth: ggplot(data = df.diamonds, mapping = aes(x = y)) + geom_density(bw = 0.5) + coord_cartesian(xlim = c(3, 10)) We’ll learn more about how these densities are determined later in class. I promised that this section was about making facets, right? We’re getting there! Let’s first take a look at how wide diamonds of different color are. We can use grouping to make this happen. ggplot(data = df.diamonds, mapping = aes(x = y, group = color, fill = color)) + geom_density(bw = 0.2, alpha = 0.2) + coord_cartesian(xlim = c(3, 10)) OK! That’s a little tricky to tell apart. Notice that I’ve specified the alpha argument in the geom_density() function so that the densities in the front don’t completely hide the densities in the back. But this plot still looks too busy. Instead of grouping, let’s put the densities for the different colors, in separate panels. That’s what facetting allows you to do. ggplot(data = df.diamonds, mapping = aes(x = y, fill = color)) + geom_density(bw = 0.2) + facet_grid(cols = vars(color)) + coord_cartesian(xlim = c(3, 10)) Now we have the densities next to each other in separate panels. I’ve removed the alpha argument since the densities aren’t overlapping anymore. To make the different panels, I used the facet_grid() function and specified that I want separate columns for the different colors (cols = vars(color)). What’s the deal with vars()? Why couldn’t we just write facet_grid(cols = color) instead? The short answer is: that’s what the function wants. The long answer is: long. (We’ll learn more about this later in the course.) To show the facets in different rows instead of columns we simply replace cols = vars(color) with rows = vars(color). ggplot(data = df.diamonds, mapping = aes(x = y, fill = color)) + geom_density(bw = 0.2) + facet_grid(rows = vars(color)) + coord_cartesian(xlim = c(3, 10)) Several aspects about this plot should be improved: the y-axis text is overlapping having both a legend and separate facet labels is redundant having separate fills is not really necessary here So, what does this plot actually show us? Well, J-colored diamonds tend to be wider than D-colored diamonds. Fascinating! Of course, we could go completely overboard with facets and groups. So let’s do it! Let’s look at how the average price (somewhat more interesting) varies as a function of color, cut, and clarity. We’ll put color on the x-axis, and make separate rows for cut and columns for clarity. ggplot(data = df.diamonds, mapping = aes(y = price, x = color, fill = color)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;) + facet_grid(rows = vars(cut), cols = vars(clarity)) Warning: Removed 1 rows containing missing values (geom_segment). Warning: Removed 3 rows containing missing values (geom_segment). Warning: Removed 1 rows containing missing values (geom_segment). Figure 2.19: A figure that is stretching it in terms of information. Figure 2.19 is stretching it in terms of how much information it presents. But it gives you a sense for how to combine the different bits and pieces we’ve learned so far. 2.5.9.1 Practice plot 4 Recreate the plot shown in Figure 2.20. # make practice plot 4 here include_graphics(&quot;figures/vis1_practice_plot4.png&quot;) Figure 2.20: Practice plot 4. 2.5.10 Global, local, and setting aes() ggplot2 allows you to specify the plot aesthetics in different ways. ggplot(data = df.diamonds, mapping = aes(x = carat, y = price, color = color)) + geom_point() + geom_smooth(method = &quot;lm&quot;, se = F) `geom_smooth()` using formula &#39;y ~ x&#39; Here, I’ve drawn a scatter plot of the relationship between carat and price, and I have added the best-fitting regression lines via the geom_smooth(method = \"lm\") call. (We will learn more about what these regression lines mean later in class.) Because I have defined all the aesthetics at the top level (i.e. directly within the ggplot() function), the aesthetics apply to all the functions afterwards. Aesthetics defined in the ggplot() call are global. In this case, the geom_point() and the geom_smooth() functions. The geom_smooth() function produces separate best-fit regression lines for each different color. But what if we only wanted to show one regression line instead that applies to all the data? Here is one way of doing so: ggplot(data = df.diamonds, mapping = aes(x = carat, y = price)) + geom_point(mapping = aes(color = color)) + geom_smooth(method = &quot;lm&quot;) `geom_smooth()` using formula &#39;y ~ x&#39; Here, I’ve moved the color aesthetic into the geom_point() function call. Now, the x and y aesthetics still apply to both the geom_point() and the geom_smooth() function call (they are global), but the color aesthetic applies only to geom_point() (it is local). Alternatively, we can simply overwrite global aesthetics within local function calls. ggplot(data = df.diamonds, mapping = aes(x = carat, y = price, color = color)) + geom_point() + geom_smooth(method = &quot;lm&quot;, color = &quot;black&quot;) `geom_smooth()` using formula &#39;y ~ x&#39; Here, I’ve set color = \"black\" within the geom_smooth() function, and now only one overall regression line is displayed since the global color aesthetic was overwritten in the local function call. 2.6 Additional resources 2.6.1 Cheatsheets RStudio IDE –&gt; information about RStudio RMarkdown –&gt; information about writing in RMarkdown RMarkdown reference –&gt; RMarkdown reference sheet Data visualization –&gt; general principles of effective graphic design ggplot2 –&gt; specific information about ggplot 2.6.2 Datacamp courses Introduction to R ggplot (intro) Reporting visualization best practices 2.6.3 Books and chapters R graphics cookbook –&gt; quick intro to the the most common graphs ggplot2 book R for Data Science book Data visualization Graphics for communication Data Visualization – A practical introduction (by Kieran Healy) Look at data Make a plot Show the right numbers Fundamentals of Data Visualization –&gt; very nice resource that goes beyond basic functionality of ggplot and focuses on how to make good figures (e.g. how to choose colors, axes, …) 2.6.4 Misc nice online ggplot tutorial how to read R help files ggplot2 extensions –&gt; gallery of ggplot2 extension packages ggplot2 visualizations with code –&gt; gallery of plots with code 2.7 Session info R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 knitr_1.39 loaded via a namespace (and not attached): [1] nlme_3.1-157 fs_1.5.2 lubridate_1.8.0 [4] RColorBrewer_1.1-3 httr_1.4.3 tools_4.1.2 [7] backports_1.4.1 bslib_0.3.1 utf8_1.2.2 [10] R6_2.5.1 rpart_4.1.16 Hmisc_4.7-0 [13] DBI_1.1.2 mgcv_1.8-40 colorspace_2.0-3 [16] nnet_7.3-17 withr_2.5.0 tidyselect_1.1.2 [19] gridExtra_2.3 compiler_4.1.2 cli_3.3.0 [22] rvest_1.0.2 htmlTable_2.4.0 xml2_1.3.3 [25] labeling_0.4.2 bookdown_0.26 sass_0.4.1 [28] scales_1.2.0 checkmate_2.1.0 digest_0.6.29 [31] foreign_0.8-82 rmarkdown_2.14 base64enc_0.1-3 [34] jpeg_0.1-9 pkgconfig_2.0.3 htmltools_0.5.2 [37] dbplyr_2.1.1 fastmap_1.1.0 highr_0.9 [40] htmlwidgets_1.5.4 rlang_1.0.2 readxl_1.4.0 [43] rstudioapi_0.13 jquerylib_0.1.4 farver_2.1.0 [46] generics_0.1.2 jsonlite_1.8.0 magrittr_2.0.3 [49] Formula_1.2-4 Matrix_1.4-1 munsell_0.5.0 [52] fansi_1.0.3 lifecycle_1.0.1 stringi_1.7.6 [55] yaml_2.3.5 grid_4.1.2 crayon_1.5.1 [58] lattice_0.20-45 haven_2.5.0 splines_4.1.2 [61] hms_1.1.1 pillar_1.7.0 reprex_2.0.1 [64] glue_1.6.2 evaluate_0.15 latticeExtra_0.6-29 [67] data.table_1.14.2 modelr_0.1.8 png_0.1-7 [70] vctrs_0.4.1 tzdb_0.3.0 cellranger_1.1.0 [73] gtable_0.3.0 assertthat_0.2.1 xfun_0.30 [76] broom_0.8.0 survival_3.3-1 viridisLite_0.4.0 [79] cluster_2.1.3 ellipsis_0.3.2 Figure 2.21: Defense at the reproducibility court. References "],["visualization-2.html", "Chapter 3 Visualization 2 3.1 Learning objectives 3.2 Install and load packages, load data, set theme 3.3 Overview of different plot types for different things 3.4 Customizing plots 3.5 Saving plots 3.6 Creating figure panels 3.7 Peeking behind the scenes 3.8 Making animations 3.9 Shiny apps 3.10 Defining snippets 3.11 Additional resources 3.12 Session info", " Chapter 3 Visualization 2 In this lecture, we will lift our ggplot2 skills to the next level! 3.1 Learning objectives Deciding what plot is appropriate for what kind of data. Customizing plots: Take a sad plot and make it better. Saving plots. Making figure panels. Debugging. Making animations. Defining snippets. 3.2 Install and load packages, load data, set theme Let’s first install the new packages that you might not have yet. install.packages(c(&quot;gganimate&quot;, &quot;gapminder&quot;, &quot;ggridges&quot;, &quot;devtools&quot;, &quot;png&quot;, &quot;gifski&quot;, &quot;patchwork&quot;)) Now, let’s load the packages that we need for this chapter. library(&quot;knitr&quot;) # for rendering the RMarkdown file library(&quot;patchwork&quot;) # for making figure panels library(&quot;ggridges&quot;) # for making joyplots library(&quot;gganimate&quot;) # for making animations library(&quot;gapminder&quot;) # data available from Gapminder.org library(&quot;tidyverse&quot;) # for plotting (and many more cool things we&#39;ll discover later) And set some settings: # these options here change the formatting of how comments are rendered opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) # this just suppresses an unnecessary message about grouping options(dplyr.summarise.inform = F) # set the default plotting theme theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size And let’s load the diamonds data again. df.diamonds = diamonds 3.3 Overview of different plot types for different things Different plots work best for different kinds of data. Let’s take a look at some. 3.3.1 Proportions 3.3.1.1 Stacked bar charts ggplot(data = df.diamonds, mapping = aes(x = cut, fill = color)) + geom_bar(color = &quot;black&quot;) This bar chart shows for the different cuts (x-axis), the number of diamonds of different color. Stacked bar charts give a good general impression of the data. However, it’s difficult to precisely compare different proportions. 3.3.1.2 Pie charts Figure 3.1: Finally a pie chart that makes sense. Pie charts have a bad reputation. And there are indeed a number of problems with pie charts: proportions are difficult to compare don’t look good when there are many categories ggplot(data = df.diamonds, mapping = aes(x = 1, fill = cut)) + geom_bar() + coord_polar(&quot;y&quot;, start = 0) + theme_void() We can create a pie chart with ggplot2 by changing the coordinate system using coord_polar(). If we are interested in comparing proportions and we don’t have too many data points, then tables are a good alternative to showing figures. 3.3.2 Comparisons Often we want to compare the data from many different conditions. And sometimes, it’s also useful to get a sense for what the individual participant data look like. Here is a plot that achieves both. 3.3.2.1 Means and individual data points ggplot(data = df.diamonds[1:150, ], mapping = aes(x = color, y = price)) + # means with confidence intervals stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;pointrange&quot;, color = &quot;black&quot;, fill = &quot;yellow&quot;, shape = 21, size = 1) + # individual data points (jittered horizontally) geom_point(alpha = 0.2, color = &quot;blue&quot;, position = position_jitter(width = 0.1, height = 0), size = 2) Figure 3.2: Price of differently colored diamonds. Large yellow circles are means, small black circles are individual data poins, and the error bars are 95% bootstrapped confidence intervals. Note that I’m only plotting the first 150 entries of the data here by setting data = df.diamonds[1:150,] in gpplot(). This plot shows means, bootstrapped confidence intervals, and individual data points. I’ve used two tricks to make the individual data points easier to see. 1. I’ve set the alpha attribute to make the points somewhat transparent. 2. I’ve used the position_jitter() function to jitter the points horizontally. 3. I’ve used shape = 21 for displaying the mean. For this circle shape, we can set a color and fill (see Figure 3.3) Figure 3.3: Different shapes that can be used for plotting. Here is an example of an actual plot that I’ve made for a paper that I’m working on (using the same techniques). Figure 2.7: Participants’ preference for the conjunctive (top) versus dis-junctive (bottom) structure as a function of the explanation (abnormal cause vs. normalcause) and the type of norm (statistical vs. prescriptive). Note: Large circles are groupmeans. Error bars are bootstrapped 95% confidence intervals. Small circles are individualparticipants’ judgments (jittered along the x-axis for visibility) 3.3.2.2 Boxplots Another way to get a sense for the distribution of the data is to use box plots. ggplot(data = df.diamonds[1:500,], mapping = aes(x = color, y = price)) + geom_boxplot() What do boxplots show? Here adapted from help(geom_boxplot()): The boxplots show the median as a horizontal black line. The lower and upper hinges correspond to the first and third quartiles (the 25th and 75th percentiles) of the data. The whiskers (= black vertical lines) extend from the top or bottom of the hinge by at most 1.5 * IQR (where IQR is the inter-quartile range, or distance between the first and third quartiles). Data beyond the end of the whiskers are called “outlying” points and are plotted individually. Personally, I’m not a big fan of boxplots. Many data sets are consistent with the same boxplot. Figure 3.4: Box plot distributions. Source: https://www.autodeskresearch.com/publications/samestats Figure 3.4 shows three different distributions that each correspond to the same boxplot. If there is not too much data, I recommend to plot jittered individual data points instead. If you do have a lot of data points, then violin plots can be helpful. Figure 3.5: Boxplot distributions. Source: https://www.autodeskresearch.com/publications/samestats Figure 3.5 shows the same raw data represented as jittered dots, boxplots, and violin plots. 3.3.2.3 Violin plots We make violin plots like so: ggplot(data = df.diamonds, mapping = aes(x = color, y = price)) + geom_violin() Violin plots are good for detecting bimodal distributions. They work well when: You have many data points. The data is continuous. Violin plots don’t work well for Likert-scale data (e.g. ratings on a discrete scale from 1 to 7). Here is a simple example: set.seed(1) data = tibble(rating = sample(x = 1:7, prob = c(0.1, 0.4, 0.1, 0.1, 0.2, 0, 0.1), size = 500, replace = T)) ggplot(data = data, mapping = aes(x = &quot;Likert&quot;, y = rating)) + geom_violin() + geom_point(position = position_jitter(width = 0.05, height = 0.1), alpha = 0.05) This represents a vase much better than it represents the data. 3.3.2.4 Joy plots We can also show the distributions along the x-axis using the geom_density_ridges() function from the ggridges package. ggplot(data = df.diamonds, mapping = aes(x = price, y = color)) + ggridges::geom_density_ridges(scale = 1.5) Picking joint bandwidth of 535 3.3.2.5 Practice plot 1 Try to make the plot shown in Figure 3.6. Here is a tip: For the data argument in ggplot() use: df.diamonds[1:10000, ] (this selects the first 10000 rows). # write your code here Figure 3.6: Practice plot 1. 3.3.3 Relationships 3.3.3.1 Scatter plots Scatter plots are great for looking at the relationship between two continuous variables. ggplot(data = df.diamonds, mapping = aes(x = carat, y = price, color = color)) + geom_point() 3.3.3.2 Raster plots These are useful for looking how a variable of interest varies as a function of two other variables. For example, when we are trying to fit a model with two parameters, we might be interested to see how well the model does for different combinations of these two parameters. Here, we’ll plot what carat values diamonds of different color and clarity have. ggplot(data = df.diamonds, mapping = aes(x = color, y = clarity, z = carat)) + stat_summary_2d(fun = &quot;mean&quot;, geom = &quot;tile&quot;) Not too bad. Let’s add a few tweaks to make it look nicer. ggplot(data = df.diamonds, mapping = aes(x = color, y = clarity, z = carat)) + stat_summary_2d(fun = &quot;mean&quot;, geom = &quot;tile&quot;, color = &quot;black&quot;) + scale_fill_gradient(low = &quot;white&quot;, high = &quot;black&quot;) + labs(fill = &quot;carat&quot;) I’ve added some outlines to the tiles by specifying color = \"black\" in geom_tile() and I’ve changed the scale for the fill gradient. I’ve defined the color for the low value to be “white”, and for the high value to be “black.” Finally, I’ve changed the lower and upper limit of the scale via the limits argument. Looks much better now! We see that diamonds with clarity I1 and color J tend to have the highest carat values on average. 3.3.4 Temporal data Line plots are a good choice for temporal data. Here, I’ll use the txhousing data that comes with the ggplot2 package. The dataset contains information about housing sales in Texas. # ignore this part for now (we&#39;ll learn about data wrangling soon) df.plot = txhousing %&gt;% filter(city %in% c(&quot;Dallas&quot;, &quot;Fort Worth&quot;, &quot;San Antonio&quot;, &quot;Houston&quot;)) %&gt;% mutate(city = factor(city, levels = c(&quot;Dallas&quot;, &quot;Houston&quot;, &quot;San Antonio&quot;, &quot;Fort Worth&quot;))) ggplot(data = df.plot, mapping = aes(x = year, y = median, color = city, fill = city)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;ribbon&quot;, alpha = 0.2, linetype = 0) + stat_summary(fun = &quot;mean&quot;, geom = &quot;line&quot;) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;) Ignore the top part where I’m defining df.plot for now (we’ll look into this in the next class). The other part is fairly straightforward. I’ve used stat_summary() three times: First, to define the confidence interval as a geom = \"ribbon\". Second, to show the lines connecting the means, and third to put the means as data points points on top of the lines. Let’s tweak the figure some more to make it look real good. df.plot = txhousing %&gt;% filter(city %in% c(&quot;Dallas&quot;, &quot;Fort Worth&quot;, &quot;San Antonio&quot;, &quot;Houston&quot;)) %&gt;% mutate(city = factor(city, levels = c(&quot;Dallas&quot;, &quot;Houston&quot;, &quot;San Antonio&quot;, &quot;Fort Worth&quot;))) df.text = df.plot %&gt;% filter(year == max(year)) %&gt;% group_by(city) %&gt;% summarize(year = mean(year) + 0.2, median = mean(median)) ggplot(data = df.plot, mapping = aes(x = year, y = median, color = city, fill = city)) + # draw dashed horizontal lines in the background geom_hline(yintercept = seq(from = 100000, to = 250000, by = 50000), linetype = 2, alpha = 0.2) + # draw ribbon stat_summary(fun.data = mean_cl_boot, geom = &quot;ribbon&quot;, alpha = 0.2, linetype = 0) + # draw lines connecting the means stat_summary(fun = &quot;mean&quot;, geom = &quot;line&quot;) + # draw means as points stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;) + # add the city names geom_text(data = df.text, mapping = aes(label = city), hjust = 0, size = 5) + # set the limits for the coordinates coord_cartesian(xlim = c(1999, 2015), clip = &quot;off&quot;, expand = F) + # set the x-axis labels scale_x_continuous(breaks = seq(from = 2000, to = 2015, by = 5)) + # set the y-axis labels scale_y_continuous(breaks = seq(from = 100000, to = 250000, by = 50000), labels = str_c(&quot;$&quot;, seq(from = 100, to = 250, by = 50), &quot;K&quot;)) + # set the plot title and axes titles labs(title = &quot;Change of median house sale price in Texas&quot;, x = &quot;Year&quot;, y = &quot;Median house sale price&quot;, fill = &quot;&quot;, color = &quot;&quot;) + theme(title = element_text(size = 16), legend.position = &quot;none&quot;, plot.margin = margin(r = 1, unit = &quot;in&quot;)) 3.4 Customizing plots So far, we’ve seen a number of different ways of plotting data. Now, let’s look into how to customize the plots. For example, we may want to change the axis labels, add a title, increase the font size. ggplot2 let’s you customize almost anything. Let’s start simple. ggplot(data = df.diamonds, mapping = aes(x = cut, y = price)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;) This plot shows the average price for diamonds with a different quality of the cut, as well as the bootstrapped confidence intervals. Here are some things we can do to make it look nicer. ggplot(data = df.diamonds, mapping = aes(x = cut, y = price)) + # change color of the fill, make a little more space between bars by setting their width stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;, fill = &quot;lightblue&quot;, width = 0.85) + # adjust the range of both axes coord_cartesian(xlim = c(0.25, 5.75), ylim = c(0, 5000), expand = F) + # make error bars thicker stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1.5) + # adjust what to show on the y-axis scale_y_continuous(breaks = seq(from = 0, to = 4000, by = 2000), labels = seq(from = 0, to = 4000, by = 2000)) + # add a title, subtitle, and changed axis labels labs(title = &quot;Price as a function of quality of cut&quot;, subtitle = &quot;Note: The price is in US dollars&quot;, tag = &quot;A&quot;, x = &quot;Quality of the cut&quot;, y = &quot;Price&quot;) + theme( # adjust the text size text = element_text(size = 20), # add some space at top of x-title axis.title.x = element_text(margin = margin(t = 0.2, unit = &quot;inch&quot;)), # add some space t the right of y-title axis.title.y = element_text(margin = margin(r = 0.1, unit = &quot;inch&quot;)), # add some space underneath the subtitle and make it gray plot.subtitle = element_text(margin = margin(b = 0.3, unit = &quot;inch&quot;), color = &quot;gray70&quot;), # make the plot tag bold plot.tag = element_text(face = &quot;bold&quot;), # move the plot tag a little plot.tag.position = c(0.05, 0.99) ) I’ve tweaked quite a few things here (and I’ve added comments to explain what’s happening). Take a quick look at the theme() function to see all the things you can change. 3.4.1 Anatomy of a ggplot I suggest to use this general skeleton for creating a ggplot: # ggplot call with global aesthetics ggplot(data = data, mapping = aes(x = cause, y = effect)) + # add geometric objects (geoms) geom_point() + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;) + ... + # add text objects geom_text() + annotate() + # adjust axes and coordinates coord_cartesian() + scale_x_continuous() + scale_y_continuous() + # define plot title, and axis titles labs(title = &quot;Title&quot;, x = &quot;Cause&quot;, y = &quot;Effect&quot;) + # change global aspects of the plot theme(text = element_text(size = 20), plot.margin = margin(t = 1, b = 1, l = 0.5, r = 0.5, unit = &quot;cm&quot;)) + # save the plot ggsave(filename = &quot;super_nice_plot.pdf&quot;, width = 8, height = 6) 3.4.2 Changing the order of things Sometimes we don’t have a natural ordering of our independent variable. In that case, it’s nice to show the data in order. ggplot(data = df.diamonds, mapping = aes(x = reorder(cut, price), y = price)) + # mapping = aes(x = cut, y = price)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;, fill = &quot;lightblue&quot;, width = 0.85) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1.5) + labs(x = &quot;cut&quot;) The reorder() function helps us to do just that. Now, the results are ordered according to price. To show the results in descending order, I would simply need to write reorder(cut, -price) instead. 3.4.3 Dealing with legends Legends form an important part of many figures. However, it is often better to avoid legends if possible, and directly label the data. This way, the reader doesn’t have to look back and forth between the plot and the legend to understand what’s going on. Here, we’ll look into a few aspects that come up quite often. There are two main functions to manipulate legends with ggplot2 1. theme() (there are a number of arguments starting with legend.) 2. guide_legend() Let’s make a plot with a legend. ggplot(data = df.diamonds, mapping = aes(x = color, y = price, color = clarity)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;) Let’s move the legend to the bottom of the plot: ggplot(data = df.diamonds, mapping = aes(x = color, y = price, color = clarity)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;) + theme(legend.position = &quot;bottom&quot;) Let’s change a few more things in the legend using the guides() function: have 3 rows reverse the legend order make the points in the legend larger ggplot(data = df.diamonds, mapping = aes(x = color, y = price, color = clarity)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;, size = 2) + theme(legend.position = &quot;bottom&quot;) + guides(color = guide_legend(nrow = 3, # 3 rows reverse = TRUE, # reversed order override.aes = list(size = 6))) # point size 3.4.4 Choosing good colors Color brewer helps with finding colors that are colorblind safe and printfriendly. For more information on how to use color effectively see here. 3.4.5 Customizing themes For a given project, I often want all of my plots to share certain visual features such as the font type, font size, how the axes are displayed, etc. Instead of defining these for each individual plot, I can set a theme at the beginning of my project so that it applies to all the plots in this file. To do so, I use the theme_set() command: theme_set(theme_classic() + #classic theme theme(text = element_text(size = 20))) #text size Here, I’ve just defined that I want to use theme_classic() for all my plots, and that the text size should be 20. For any individual plot, I can still overwrite any of these defaults. 3.5 Saving plots To save plots, use the ggsave() command. Personally, I prefer to save my plots as pdf files. This way, the plot looks good no matter what size you need it to be. This means it’ll look good both in presentations as well as in a paper. You can save the plot in any format that you like. I strongly recommend to use a relative path to specify where the figure should be saved. This way, if you are sharing the project with someone else via Stanford Box, Dropbox, or Github, they will be able to run the code without errors. Here is an example for how to save one of the plots that we’ve created above. p1 = ggplot(data = df.diamonds, mapping = aes(x = cut, y = price)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;, fill = &quot;lightblue&quot;) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1) print(p1) p2 = ggplot(data = df.diamonds, mapping = aes(x = cut, y = price)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;, fill = &quot;lightblue&quot;) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1) ggsave(filename = &quot;figures/diamond_plot.pdf&quot;, plot = p1, width = 8, height = 6) Here, I’m saving the plot in the figures folder and it’s name is diamond_plot.pdf. I also specify the width and height as the plot in inches (which is the default unit). 3.6 Creating figure panels Sometimes, we want to create a figure with several subfigures, each of which is labeled with a), b), etc. We have already learned how to make separate panels using facet_wrap() or facet_grid(). The R package patchwork makes it very easy to combine multiple plots. You can find out more about the package here. Let’s combine a few plots that we’ve made above into one. # first plot p1 = ggplot(data = df.diamonds, mapping = aes(x = y, fill = color)) + geom_density(bw = 0.2, show.legend = F) + facet_grid(cols = vars(color)) + labs(title = &quot;Width of differently colored diamonds&quot;) + coord_cartesian(xlim = c(3, 10), expand = F) #setting expand to FALSE removes any padding on x and y axes # second plot p2 = ggplot(data = df.diamonds, mapping = aes(x = color, y = clarity, z = carat)) + stat_summary_2d(fun = &quot;mean&quot;, geom = &quot;tile&quot;) + labs(title = &quot;Carat values&quot;, subtitle = &quot;For different color and clarity&quot;, x = &quot;Color&quot;) # third plot p3 = ggplot(data = df.diamonds, mapping = aes(x = cut, y = price)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;, fill = &quot;lightblue&quot;, width = 0.85) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1.5) + scale_x_discrete(labels = c(&quot;fair&quot;, &quot;good&quot;, &quot;very\\ngood&quot;, &quot;premium&quot;, &quot;ideal&quot;)) + labs(title = &quot;Price as a function of cut&quot;, subtitle = &quot;Note: The price is in US dollars&quot;, x = &quot;Quality of the cut&quot;, y = &quot;Price&quot;) + coord_cartesian(xlim = c(0.25, 5.75), ylim = c(0, 5000), expand = F) # combine the plots p1 + (p2 + p3) + plot_layout(ncol = 1) &amp; plot_annotation(tag_levels = &quot;A&quot;) &amp; theme_classic() &amp; theme(plot.tag = element_text(face = &quot;bold&quot;, size = 20)) # ggsave(&quot;figures/combined_plot.png&quot;, width = 10, height = 6) Not a perfect plot yet, but you get the idea. To combine the plots, we defined that we would like p2 and p3 to be displayed in the same row using the () syntax. And we specified that we only want one column via the plot_layout() function. We also applied the same theme_classic() to all the plots using the &amp; operator, and formatted how the plot tags should be displayed. For more info on how to use patchwork, take a look at the readme on the github page. Other packages that provide additional functionality for combining multiple plots into one are gridExtra and cowplot. You can find more information on how to lay out multiple plots here. An alternative way for making these plots is to use Adobe Illustrator, Powerpoint, or Keynote. However, you want to make changing plots as easy as possible. Adobe Illustrator has a feature that allows you to link to files. This way, if you change the plot, the plot within the illustrator file gets updated automatically as well. If possible, it’s much better to do everything in R though so that your plot can easily be reproduced by someone else. 3.7 Peeking behind the scenes Sometimes it can be helpful for debugging to take a look behind the scenes. Silently, ggplot() computes a data frame based on the information you pass to it. We can take a look at the data frame that’s underlying the plot. p = ggplot(data = df.diamonds, mapping = aes(x = color, y = clarity, z = carat)) + stat_summary_2d(fun = &quot;mean&quot;, geom = &quot;tile&quot;, color = &quot;black&quot;) + scale_fill_gradient(low = &quot;white&quot;, high = &quot;black&quot;) print(p) build = ggplot_build(p) df.plot_info = build$data[[1]] dim(df.plot_info) # data frame dimensions [1] 56 18 I’ve called the ggplot_build() function on the ggplot2 object that we saved as p. I’ve then printed out the data associated with that plot object. The first thing we note about the data frame is how many entries it has, 56. That’s good. This means there is one value for each of the 7 x 8 grids. The columns tell us what color was used for the fill, the value associated with each row, where each row is being displayed (x and y), etc. If a plot looks weird, it’s worth taking a look behind the scenes. For example, something we thing we could have tried is the following (in fact, this is what I tried first): p = ggplot(data = df.diamonds, mapping = aes(x = color, y = clarity, fill = carat)) + geom_tile(color = &quot;black&quot;) + scale_fill_gradient(low = &quot;white&quot;, high = &quot;black&quot;) print(p) build = ggplot_build(p) df.plot_info = build$data[[1]] dim(df.plot_info) # data frame dimensions [1] 53940 15 Why does this plot look different from the one before? What went wrong here? Notice that the data frame associated with the ggplot2 object has 53940 rows. So instead of plotting means here, we plotted all the individual data points. So what we are seeing here is just the top layer of many, many layers. 3.8 Making animations Animated plots can be a great way to illustrate your data in presentations. The R package gganimate lets you do just that. Here is an example showing how to use it. ggplot(data = gapminder, mapping = aes(x = gdpPercap, y = lifeExp, size = pop, colour = country)) + geom_point(alpha = 0.7, show.legend = FALSE) + geom_text(data = gapminder %&gt;% filter(country %in% c(&quot;United States&quot;, &quot;China&quot;, &quot;India&quot;)), mapping = aes(label = country), color = &quot;black&quot;, vjust = -0.75, show.legend = FALSE) + scale_colour_manual(values = country_colors) + scale_size(range = c(2, 12)) + scale_x_log10(breaks = c(1e3, 1e4, 1e5), labels = c(&quot;1,000&quot;, &quot;10,000&quot;, &quot;100,000&quot;)) + theme_classic() + theme(text = element_text(size = 23)) + # Here come the gganimate specific bits labs(title = &quot;Year: {frame_time}&quot;, x = &quot;GDP per capita&quot;, y = &quot;life expectancy&quot;) + transition_time(year) + ease_aes(&quot;linear&quot;) Rendering [&gt;--------------------------------------------] at 10 fps ~ eta: 9s Rendering [=&gt;-------------------------------------------] at 10 fps ~ eta: 9s Rendering [==&gt;-----------------------------------------] at 9.9 fps ~ eta: 10s Rendering [==&gt;-----------------------------------------] at 9.9 fps ~ eta: 9s Rendering [===&gt;----------------------------------------] at 9.8 fps ~ eta: 9s Rendering [===&gt;----------------------------------------] at 9.9 fps ~ eta: 9s Rendering [====&gt;---------------------------------------] at 9.8 fps ~ eta: 9s Rendering [=====&gt;--------------------------------------] at 9.7 fps ~ eta: 9s Rendering [======&gt;-------------------------------------] at 9.7 fps ~ eta: 9s Rendering [======&gt;-------------------------------------] at 9.6 fps ~ eta: 9s Rendering [=======&gt;------------------------------------] at 9.6 fps ~ eta: 9s Rendering [=======&gt;------------------------------------] at 9.6 fps ~ eta: 8s Rendering [========&gt;-----------------------------------] at 9.5 fps ~ eta: 8s Rendering [=========&gt;----------------------------------] at 9.5 fps ~ eta: 8s Rendering [==========&gt;---------------------------------] at 9.5 fps ~ eta: 8s Rendering [==========&gt;---------------------------------] at 9.4 fps ~ eta: 8s Rendering [===========&gt;--------------------------------] at 9.5 fps ~ eta: 8s Rendering [============&gt;-------------------------------] at 9.4 fps ~ eta: 8s Rendering [============&gt;-------------------------------] at 9.4 fps ~ eta: 7s Rendering [=============&gt;------------------------------] at 9.3 fps ~ eta: 7s Rendering [==============&gt;-----------------------------] at 9.4 fps ~ eta: 7s Rendering [===============&gt;----------------------------] at 9.4 fps ~ eta: 7s Rendering [================&gt;---------------------------] at 9.4 fps ~ eta: 7s Rendering [================&gt;---------------------------] at 9.3 fps ~ eta: 7s Rendering [=================&gt;--------------------------] at 9.3 fps ~ eta: 6s Rendering [=================&gt;--------------------------] at 9.4 fps ~ eta: 6s Rendering [==================&gt;-------------------------] at 9.4 fps ~ eta: 6s Rendering [===================&gt;------------------------] at 9.4 fps ~ eta: 6s Rendering [====================&gt;-----------------------] at 9.4 fps ~ eta: 6s Rendering [=====================&gt;----------------------] at 9.4 fps ~ eta: 5s Rendering [======================&gt;---------------------] at 9.2 fps ~ eta: 5s Rendering [=======================&gt;--------------------] at 9.2 fps ~ eta: 5s Rendering [========================&gt;-------------------] at 9.2 fps ~ eta: 5s Rendering [=========================&gt;------------------] at 9.2 fps ~ eta: 5s Rendering [=========================&gt;------------------] at 9.2 fps ~ eta: 4s Rendering [==========================&gt;-----------------] at 9.3 fps ~ eta: 4s Rendering [===========================&gt;----------------] at 9.3 fps ~ eta: 4s Rendering [============================&gt;---------------] at 9.3 fps ~ eta: 4s Rendering [=============================&gt;--------------] at 9.3 fps ~ eta: 3s Rendering [==============================&gt;-------------] at 9.3 fps ~ eta: 3s Rendering [===============================&gt;------------] at 9.4 fps ~ eta: 3s Rendering [================================&gt;-----------] at 9.4 fps ~ eta: 3s Rendering [=================================&gt;----------] at 9.4 fps ~ eta: 2s Rendering [==================================&gt;---------] at 9.4 fps ~ eta: 2s Rendering [===================================&gt;--------] at 9.4 fps ~ eta: 2s Rendering [====================================&gt;-------] at 9.4 fps ~ eta: 2s Rendering [====================================&gt;-------] at 9.5 fps ~ eta: 2s Rendering [=====================================&gt;------] at 9.5 fps ~ eta: 1s Rendering [======================================&gt;-----] at 9.5 fps ~ eta: 1s Rendering [=======================================&gt;----] at 9.5 fps ~ eta: 1s Rendering [========================================&gt;---] at 9.5 fps ~ eta: 1s Rendering [=========================================&gt;--] at 9.5 fps ~ eta: 1s Rendering [=========================================&gt;--] at 9.4 fps ~ eta: 0s Rendering [==========================================&gt;-] at 9.4 fps ~ eta: 0s Rendering [===========================================&gt;] at 9.4 fps ~ eta: 0s Rendering [============================================] at 9.4 fps ~ eta: 0s # anim_save(filename = &quot;figures/life_gdp_animation.gif&quot;) # to save the animation This takes a while to run but it’s worth the wait. The plot shows the relationship between GDP per capita (on the x-axis) and life expectancy (on the y-axis) changes across different years for the countries of different continents. The size of each dot represents the population size of the respective country. And different countries are shown in different colors. This animation is not super useful yet in that we don’t know which continents and countries the different dots represent. I’ve added a label to the United States, China, and India. Note how little is required to define the gganimate-specific information! The {frame_time} variable changes the title for each frame. The transition_time() variable is set to year, and the kind of transition is set as ‘linear’ in ease_aes(). I’ve saved the animation as a gif in the figures folder. We won’t have time to go into more detail here but I encourage you to play around with gganimate. It’s fun, looks cool, and (if done well) makes for a great slide in your next presentation! 3.9 Shiny apps The package shiny makes it relatively easy to create interactive plots that can be hosted online. Here is a gallery with some examples. 3.10 Defining snippets Often, we want to create similar plots over and over again. One way to achieve this is by finding the original plot, copy and pasting it, and changing the bits that need changing. Another more flexible and faster way to do this is by using snippets. Snippets are short pieces of code that Here are some snippets I use: snippet sngg ggplot(data = ${1:data}, mapping = aes(${2:aes})) + ${0} snippet sndf ${1:data} = ${1:data} %&gt;% ${0} To make a bar plot, I now only need to type snbar and then hit TAB to activate the snippet. I can then cycle through the bits in the code that are marked with ${Number:word} by hitting TAB again. In RStudio, you can change and add snippets by going to Tools –&gt; Global Options… –&gt; Code –&gt; Edit Snippets. Make sure to set the tick mark in front of Enable Code Snippets (see Figure 3.7). ). Figure 3.7: Enable code snippets. To edit code snippets faster, run this command from the usethis package. Make sure to install the package first if you don’t have it yet. # install.packages(&quot;usethis&quot;) usethis::edit_rstudio_snippets() This command opens up a separate tab in RStudio called r.snippets so that you can make new snippets and adapt old ones more quickly. Take a look at the snippets that RStudio already comes with. And then, make some new ones! By using snippets you will be able to avoid typing the same code over and over again, and you won’t have to memorize as much, too. 3.11 Additional resources 3.11.1 Cheatsheets shiny –&gt; interactive plots 3.11.2 Data camp courses shiny 3.11.3 Books and chapters R for Data Science book Data visualization Graphics for communication Data Visualization – A practical introduction (by Kieran Healy) Refine your plots 3.11.4 Misc ggplot2 extensions –&gt; gallery of ggplot2 extension packages ggplot2 gui –&gt; ggplot2 extension package ggplot2 visualizations with code –&gt; gallery of plots with code Color brewer –&gt; for finding colors shiny apps examples –&gt; shiny apps examples that focus on statistics teaching (made by students at PennState) 3.12 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 tidyverse_1.3.1 [9] gapminder_0.3.0 gganimate_1.0.7 ggplot2_3.3.6 ggridges_0.5.3 [13] patchwork_1.1.1 knitr_1.39 loaded via a namespace (and not attached): [1] fs_1.5.2 lubridate_1.8.0 RColorBrewer_1.1-3 [4] progress_1.2.2 httr_1.4.3 tools_4.1.2 [7] backports_1.4.1 bslib_0.3.1 utf8_1.2.2 [10] R6_2.5.1 rpart_4.1.16 Hmisc_4.7-0 [13] DBI_1.1.2 colorspace_2.0-3 nnet_7.3-17 [16] withr_2.5.0 gridExtra_2.3 tidyselect_1.1.2 [19] prettyunits_1.1.1 compiler_4.1.2 cli_3.3.0 [22] rvest_1.0.2 htmlTable_2.4.0 xml2_1.3.3 [25] labeling_0.4.2 bookdown_0.26 sass_0.4.1 [28] checkmate_2.1.0 scales_1.2.0 digest_0.6.29 [31] foreign_0.8-82 rmarkdown_2.14 base64enc_0.1-3 [34] jpeg_0.1-9 pkgconfig_2.0.3 htmltools_0.5.2 [37] dbplyr_2.1.1 fastmap_1.1.0 highr_0.9 [40] htmlwidgets_1.5.4 rlang_1.0.2 readxl_1.4.0 [43] rstudioapi_0.13 jquerylib_0.1.4 farver_2.1.0 [46] generics_0.1.2 jsonlite_1.8.0 magrittr_2.0.3 [49] Formula_1.2-4 Matrix_1.4-1 Rcpp_1.0.8.3 [52] munsell_0.5.0 fansi_1.0.3 lifecycle_1.0.1 [55] stringi_1.7.6 yaml_2.3.5 plyr_1.8.7 [58] grid_4.1.2 crayon_1.5.1 lattice_0.20-45 [61] haven_2.5.0 splines_4.1.2 hms_1.1.1 [64] pillar_1.7.0 reprex_2.0.1 glue_1.6.2 [67] evaluate_0.15 latticeExtra_0.6-29 data.table_1.14.2 [70] gifski_1.6.6-1 modelr_0.1.8 vctrs_0.4.1 [73] png_0.1-7 tzdb_0.3.0 tweenr_1.0.2 [76] cellranger_1.1.0 gtable_0.3.0 assertthat_0.2.1 [79] xfun_0.30 broom_0.8.0 survival_3.3-1 [82] viridisLite_0.4.0 cluster_2.1.3 ellipsis_0.3.2 "],["data-wrangling-1.html", "Chapter 4 Data wrangling 1 4.1 Learning goals 4.2 Load packages 4.3 Some R basics 4.4 A quick note on naming things 4.5 Looking at data 4.6 Wrangling data 4.7 Additional resources 4.8 Session info", " Chapter 4 Data wrangling 1 In this lecture, we will take a look at how to wrangle data using the dplyr package. Again, getting our data into shape is something we’ll need to do throughout the course, so it’s worth spending some time getting a good sense for how this works. The nice thing about R is that (thanks to the tidyverse), both visualization and data wrangling are particularly powerful. 4.1 Learning goals Review R basics (incl. variable modes, data types, operators, control flow, and functions). Learn how the pipe operator %&gt;% works. See different ways for getting a sense of one’s data. Master key data manipulation verbs from the dplyr package (incl. filter(), arrange(), rename(), relocate(), select(), mutate()) as well as the helper functions across() and where(). 4.2 Load packages Let’s first load the packages that we need for this chapter. library(&quot;knitr&quot;) # for rendering the RMarkdown file library(&quot;skimr&quot;) # for visualizing data library(&quot;visdat&quot;) # for visualizing data library(&quot;DT&quot;) # for visualizing data library(&quot;tidyverse&quot;) # for data wrangling opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 4.3 Some R basics To test your knowledge of the R basics, I recommend taking the free interactive tutorial on datacamp: Introduction to R. Here, I will just give a very quick overview of some of the basics. 4.3.1 Modes Variables in R can have different modes. Table 4.1 shows the most common ones. Table 4.1: Most commonly used variable modes in R. name example numeric 1, 3, 48 character 'Steve', 'a', '78' logical TRUE, FALSE not available NA For characters you can either use \" or '. R has a number of functions to convert a variable from one mode to another. NA is used for missing values. tmp1 = &quot;1&quot; # we start with a character str(tmp1) chr &quot;1&quot; tmp2 = as.numeric(tmp1) # turn it into a numeric str(tmp2) num 1 tmp3 = as.factor(tmp2) # turn that into a factor str(tmp3) Factor w/ 1 level &quot;1&quot;: 1 tmp4 = as.character(tmp3) # and go full cycle by turning it back into a character str(tmp4) chr &quot;1&quot; identical(tmp1, tmp4) # checks whether tmp1 and tmp4 are the same [1] TRUE The str() function displays the structure of an R object. Here, it shows us what mode the variable is. 4.3.2 Data types R has a number of different data types. Table 4.2 shows the ones you’re most likely to come across (taken from this source): Table 4.2: Most commonly used data types in R. name description vector list of values with of the same variable mode factor for ordinal variables matrix 2D data structure array same as matrix for higher dimensional data data frame similar to matrix but with column names list flexible type that can contain different other variable types 4.3.2.1 Vectors We build vectors using the concatenate function c(), and we use [] to access one or more elements of a vector. numbers = c(1, 4, 5) # make a vector numbers[2] # access the second element [1] 4 numbers[1:2] # access the first two elements [1] 1 4 numbers[c(1, 3)] # access the first and last element [1] 1 5 In R (unlike in Python for example), 1 refers to the first element of a vector (or list). 4.3.2.2 Matrix We build a matrix using the matrix() function, and we use [] to access its elements. matrix = matrix(data = c(1, 2, 3, 4, 5, 6), nrow = 3, ncol = 2) matrix # the full matrix [,1] [,2] [1,] 1 4 [2,] 2 5 [3,] 3 6 matrix[1, 2] # element in row 1, column 2 [1] 4 matrix[1, ] # all elements in the first row [1] 1 4 matrix[ , 1] # all elements in the first column [1] 1 2 3 matrix[-1, ] # a matrix which excludes the first row [,1] [,2] [1,] 2 5 [2,] 3 6 Note how we use an empty placeholder to indicate that we want to select all the values in a row or column, and - to indicate that we want to remove something. 4.3.2.3 Array Arrays work the same was as matrices with data of more than two dimensions. 4.3.2.4 Data frame df = tibble(participant_id = c(1, 2, 3), participant_name = c(&quot;Leia&quot;, &quot;Luke&quot;, &quot;Darth&quot;)) # make the data frame df # the complete data frame # A tibble: 3 × 2 participant_id participant_name &lt;dbl&gt; &lt;chr&gt; 1 1 Leia 2 2 Luke 3 3 Darth df[1, 2] # a single element using numbers # A tibble: 1 × 1 participant_name &lt;chr&gt; 1 Leia df$participant_id # all participants [1] 1 2 3 df[[&quot;participant_id&quot;]] # same as before but using [[]] instead of $ [1] 1 2 3 df$participant_name[2] # name of the second participant [1] &quot;Luke&quot; df[[&quot;participant_name&quot;]][2] # same as above [1] &quot;Luke&quot; We’ll use data frames a lot. Data frames are like a matrix with column names. Data frames are also more general than matrices in that different columns can have different modes. For example, one column might be a character, another one numeric, and another one a factor. Here we used the tibble() function to create the data frame. A tibble is almost the same as a data frame but it has better defaults for formatting output in the console (more information on tibbles is here). 4.3.2.5 Lists l.mixed = list(number = 1, character = &quot;2&quot;, factor = factor(3), matrix = matrix(1:4, ncol = 2), df = tibble(x = c(1, 2), y = c(3, 4))) l.mixed $number [1] 1 $character [1] &quot;2&quot; $factor [1] 3 Levels: 3 $matrix [,1] [,2] [1,] 1 3 [2,] 2 4 $df # A tibble: 2 × 2 x y &lt;dbl&gt; &lt;dbl&gt; 1 1 3 2 2 4 # three different ways of accessing a list l.mixed$character [1] &quot;2&quot; l.mixed[[&quot;character&quot;]] [1] &quot;2&quot; l.mixed[[2]] [1] &quot;2&quot; Lists are a very flexible data format. You can put almost anything in a list. 4.3.3 Operators Table 4.3 shows the comparison operators that result in logical outputs. Table 4.3: Table of comparison operators that result in boolean (TRUE/FALSE) outputs. symbol name == equal to != not equal to &gt;, &lt; greater/less than &gt;=, &lt;= greater/less than or equal &amp;, &amp;#124;, ! logical operators: and, or, not %in% checks whether an element is in an object The %in% operator is very useful, and we can use it like so: x = c(1, 2, 3) 2 %in% x [1] TRUE c(3, 4) %in% x [1] TRUE FALSE It’s particularly useful for filtering data as we will see below. 4.3.4 Control flow 4.3.4.1 if-then number = 3 if (number == 1) { print(&quot;The number is 1.&quot;) } else if (number == 2) { print(&quot;The number is 2.&quot;) } else { print(&quot;The number is neither 1 nor 2.&quot;) } [1] &quot;The number is neither 1 nor 2.&quot; As a shorthand version, we can also use the ifelse() function like so: number = 3 ifelse(test = number == 1, yes = &quot;correct&quot;, no = &quot;false&quot;) [1] &quot;false&quot; 4.3.4.2 for loop sequence = 1:10 for(i in 1:length(sequence)){ print(i) } [1] 1 [1] 2 [1] 3 [1] 4 [1] 5 [1] 6 [1] 7 [1] 8 [1] 9 [1] 10 4.3.4.3 while loop number = 1 while(number &lt;= 10){ print(number) number = number + 1 } [1] 1 [1] 2 [1] 3 [1] 4 [1] 5 [1] 6 [1] 7 [1] 8 [1] 9 [1] 10 4.3.5 Functions fun.add_two_numbers = function(a, b){ x = a + b return(str_c(&quot;The result is &quot;, x)) } fun.add_two_numbers(1, 2) [1] &quot;The result is 3&quot; I’ve used the str_c() function here to concatenate the string with the number. (R converts the number x into a string for us.) Note, R functions can only return a single object. However, this object can be a list (which can contain anything). 4.3.5.1 Some often used functions Table 4.4: Some frequently used functions. name description length() length of an object dim() dimensions of an object (e.g. number of rows and columns) rm() remove an object seq() generate a sequence of numbers rep() repeat something n times max() maximum min() minimum which.max() index of the maximum which.min() index of the maximum mean() mean median() median sum() sum var() variance sd() standard deviation 4.3.6 The pipe operator %&gt;% Figure 4.1: Inspiration for the magrittr package name. Figure 4.2: The magrittr package logo. The pipe operator %&gt;% is a special operator introduced in the magrittr package. It is used heavily in the tidyverse. The basic idea is simple: this operator allows us to “pipe” several functions into one long chain that matches the order in which we want to do stuff. Let’s consider the following example of making and eating a cake (thanks to https://twitter.com/dmi3k/status/1191824875842879489?s=09). This would be the traditional way of writing some code: eat( slice( bake( put( pour( mix(ingredients), into = baking_form), into = oven), time = 30), pieces = 6), 1) To see what’s going on here, we need to read the code inside out. That is, we have to start in the innermost bracket, and then work our way outward. However, there is a natural causal ordering to these steps and wouldn’t it be nice if we could just write code in that order? Thanks to the pipe operator %&gt;% we can! Here is the same example using the pipe: ingredients %&gt;% mix %&gt;% pour(into = baking_form) %&gt;% put(into = oven) %&gt;% bake(time = 30) %&gt;% slice(pieces = 6) %&gt;% eat(1) This code is much easier to read and write, since it represents the order in which we want to do things! Abstractly, the pipe operator does the following: f(x) can be rewritten as x %&gt;% f() For example, in standard R, we would write: x = 1:3 # standard R sum(x) [1] 6 With the pipe, we can rewrite this as: x = 1:3 # with the pipe x %&gt;% sum() [1] 6 This doesn’t seem super useful yet, but just hold on a little longer. f(x, y) can be rewritten as x %&gt;% f(y) So, we could rewrite the following standard R code … # rounding pi to 6 digits, standard R round(pi, digits = 6) [1] 3.141593 … by using the pipe: # rounding pi to 6 digits, standard R pi %&gt;% round(digits = 6) [1] 3.141593 Here is another example: a = 3 b = 4 sum(a, b) # standard way [1] 7 a %&gt;% sum(b) # the pipe way [1] 7 The pipe operator inserts the result of the previous computation as a first element into the next computation. So, a %&gt;% sum(b) is equivalent to sum(a, b). We can also specify to insert the result at a different position via the . operator. For example: a = 1 b = 10 b %&gt;% seq(from = a, to = .) [1] 1 2 3 4 5 6 7 8 9 10 Here, I used the . operator to specify that I woud like to insert the result of b where I’ve put the . in the seq() function. f(x, y) can be rewritten as y %&gt;% f(x, .) Still not to thrilled about the pipe? We can keep going though (and I’m sure you’ll be convinced eventually.) h(g(f(x))) can be rewritten as x %&gt;% f() %&gt;% g() %&gt;% h() For example, consider that we want to calculate the root mean squared error (RMSE) between prediction and data. Here is how the RMSE is defined: \\[ \\text{RMSE} = \\sqrt\\frac{\\sum_{i=1}^n(\\hat{y}_i-y_i)^2}{n} \\] where \\(\\hat{y}_i\\) denotes the prediction, and \\(y_i\\) the actually observed value. In base R, we would do the following. data = c(1, 3, 4, 2, 5) prediction = c(1, 2, 2, 1, 4) # calculate root mean squared error rmse = sqrt(mean((prediction-data)^2)) print(rmse) [1] 1.183216 Using the pipe operator makes the operation more intuitive: data = c(1, 3, 4, 2, 5) prediction = c(1, 2, 2, 1, 4) # calculate root mean squared error the pipe way rmse = (prediction-data)^2 %&gt;% mean() %&gt;% sqrt() %&gt;% print() [1] 1.183216 First, we calculate the squared error, then we take the mean, then the square root, and then print the result. The pipe operator %&gt;% is similar to the + used in ggplot2. It allows us to take step-by-step actions in a way that fits the causal ordering of how we want to do things. Tip: The keyboard shortcut for the pipe operator is: cmd/ctrl + shift + m Definitely learn this one – we’ll use the pipe a lot!! Tip: Code is generally easier to read when the pipe %&gt;% is at the end of a line (just like the + in ggplot2). A key advantage of using the pipe is that you don’t have to save intermediate computations as new variables and this helps to keep your environment nice and clean! 4.3.6.1 Practice 1 Let’s practice the pipe operator. # here are some numbers x = seq(from = 1, to = 5, by = 1) # taking the log the standard way log(x) [1] 0.0000000 0.6931472 1.0986123 1.3862944 1.6094379 # now take the log the pipe way (write your code underneath) # some more numbers x = seq(from = 10, to = 5, by = -1) # the standard way mean(round(sqrt(x), digits = 2)) [1] 2.721667 # the pipe way (write your code underneath) 4.4 A quick note on naming things Personally, I like to name things in a (pretty) consistent way so that I have no trouble finding stuff even when I open up a project that I haven’t worked on for a while. I try to use the following naming conventions: Table 4.5: Some naming conventions I adopt to make my life easier. name use df.thing for data frames l.thing for lists fun.thing for functions tmp.thing for temporary variables 4.5 Looking at data The package dplyr which we loaded as part of the tidyverse, includes a data set with information about starwars characters. Let’s store this as df.starwars. df.starwars = starwars Note: Unlike in other languages (such as Python or Matlab), a . in a variable name has no special meaning and can just be used as part of the name. I’ve used df here to indicate for myself that this variable is a data frame. Before visualizing the data, it’s often useful to take a quick direct look at the data. There are several ways of taking a look at data in R. Personally, I like to look at the data within RStudio’s data viewer. To do so, you can: click on the df.starwars variable in the “Environment” tab type View(df.starwars) in the console move your mouse over (or select) the variable in the editor (or console) and hit F2 I like the F2 route the best as it’s fast and flexible. Sometimes it’s also helpful to look at data in the console instead of the data viewer. Particularly when the data is very large, the data viewer can be sluggish. Here are some useful functions: 4.5.1 head() Without any extra arguments specified, head() shows the top six rows of the data. head(df.starwars) # A tibble: 6 × 14 name height mass hair_color skin_color eye_color birth_year sex gender &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 Luke Sky… 172 77 blond fair blue 19 male mascu… 2 C-3PO 167 75 &lt;NA&gt; gold yellow 112 none mascu… 3 R2-D2 96 32 &lt;NA&gt; white, bl… red 33 none mascu… 4 Darth Va… 202 136 none white yellow 41.9 male mascu… 5 Leia Org… 150 49 brown light brown 19 fema… femin… 6 Owen Lars 178 120 brown, gr… light blue 52 male mascu… # … with 5 more variables: homeworld &lt;chr&gt;, species &lt;chr&gt;, films &lt;list&gt;, # vehicles &lt;list&gt;, starships &lt;list&gt; 4.5.2 glimpse() glimpse() is helpful when the data frame has many columns. The data is shown in a transposed way with columns as rows. glimpse(df.starwars) Rows: 87 Columns: 14 $ name &lt;chr&gt; &quot;Luke Skywalker&quot;, &quot;C-3PO&quot;, &quot;R2-D2&quot;, &quot;Darth Vader&quot;, &quot;Leia Or… $ height &lt;int&gt; 172, 167, 96, 202, 150, 178, 165, 97, 183, 182, 188, 180, 2… $ mass &lt;dbl&gt; 77.0, 75.0, 32.0, 136.0, 49.0, 120.0, 75.0, 32.0, 84.0, 77.… $ hair_color &lt;chr&gt; &quot;blond&quot;, NA, NA, &quot;none&quot;, &quot;brown&quot;, &quot;brown, grey&quot;, &quot;brown&quot;, N… $ skin_color &lt;chr&gt; &quot;fair&quot;, &quot;gold&quot;, &quot;white, blue&quot;, &quot;white&quot;, &quot;light&quot;, &quot;light&quot;, &quot;… $ eye_color &lt;chr&gt; &quot;blue&quot;, &quot;yellow&quot;, &quot;red&quot;, &quot;yellow&quot;, &quot;brown&quot;, &quot;blue&quot;, &quot;blue&quot;,… $ birth_year &lt;dbl&gt; 19.0, 112.0, 33.0, 41.9, 19.0, 52.0, 47.0, NA, 24.0, 57.0, … $ sex &lt;chr&gt; &quot;male&quot;, &quot;none&quot;, &quot;none&quot;, &quot;male&quot;, &quot;female&quot;, &quot;male&quot;, &quot;female&quot;,… $ gender &lt;chr&gt; &quot;masculine&quot;, &quot;masculine&quot;, &quot;masculine&quot;, &quot;masculine&quot;, &quot;femini… $ homeworld &lt;chr&gt; &quot;Tatooine&quot;, &quot;Tatooine&quot;, &quot;Naboo&quot;, &quot;Tatooine&quot;, &quot;Alderaan&quot;, &quot;T… $ species &lt;chr&gt; &quot;Human&quot;, &quot;Droid&quot;, &quot;Droid&quot;, &quot;Human&quot;, &quot;Human&quot;, &quot;Human&quot;, &quot;Huma… $ films &lt;list&gt; &lt;&quot;The Empire Strikes Back&quot;, &quot;Revenge of the Sith&quot;, &quot;Return… $ vehicles &lt;list&gt; &lt;&quot;Snowspeeder&quot;, &quot;Imperial Speeder Bike&quot;&gt;, &lt;&gt;, &lt;&gt;, &lt;&gt;, &quot;Imp… $ starships &lt;list&gt; &lt;&quot;X-wing&quot;, &quot;Imperial shuttle&quot;&gt;, &lt;&gt;, &lt;&gt;, &quot;TIE Advanced x1&quot;,… 4.5.3 distinct() distinct() shows all the distinct values for a character or factor column. df.starwars %&gt;% distinct(species) # A tibble: 38 × 1 species &lt;chr&gt; 1 Human 2 Droid 3 Wookiee 4 Rodian 5 Hutt 6 Yoda&#39;s species 7 Trandoshan 8 Mon Calamari 9 Ewok 10 Sullustan # … with 28 more rows 4.5.4 count() count() shows a count of all the different distinct values in a column. df.starwars %&gt;% count(eye_color) # A tibble: 15 × 2 eye_color n &lt;chr&gt; &lt;int&gt; 1 black 10 2 blue 19 3 blue-gray 1 4 brown 21 5 dark 1 6 gold 1 7 green, yellow 1 8 hazel 3 9 orange 8 10 pink 1 11 red 5 12 red, blue 1 13 unknown 3 14 white 1 15 yellow 11 It’s possible to do grouped counts by combining several variables. df.starwars %&gt;% count(eye_color, gender) %&gt;% head(n = 10) # A tibble: 10 × 3 eye_color gender n &lt;chr&gt; &lt;chr&gt; &lt;int&gt; 1 black feminine 2 2 black masculine 8 3 blue feminine 6 4 blue masculine 12 5 blue &lt;NA&gt; 1 6 blue-gray masculine 1 7 brown feminine 5 8 brown masculine 15 9 brown &lt;NA&gt; 1 10 dark masculine 1 4.5.5 datatable() For RMardkown files specifically, we can use the datatable() function from the DT package to get an interactive table widget. df.starwars %&gt;% DT::datatable() 4.5.6 Other tools for taking a quick look at data 4.5.6.1 vis_dat() The vis_dat() function from the visdat package, gives a visual summary that makes it easy to see the variable types and whether there are missing values in the data. visdat::vis_dat(df.starwars) Warning: `gather_()` was deprecated in tidyr 1.2.0. Please use `gather()` instead. This warning is displayed once every 8 hours. Call `lifecycle::last_lifecycle_warnings()` to see where this warning was generated. When R loads packages, functions loaded in earlier packages are overwritten by functions of the same name from later packages. This means that the order in which packages are loaded matters. To make sure that a function from the correct package is used, you can use the package_name::function_name() construction. This way, the function_name() from the package_name is used, rather than the same function from a different package. This is why, in general, I recommend to load the tidyverse package last (since it contains a large number of functions that we use a lot). 4.5.6.2 skim() The skim() function from the skimr package provides a nice overview of the data, separated by variable types. # install.packages(&quot;skimr&quot;) skimr::skim(df.starwars) Table 4.6: Data summary Name df.starwars Number of rows 87 Number of columns 14 _______________________ Column type frequency: character 8 list 3 numeric 3 ________________________ Group variables None Variable type: character skim_variable n_missing complete_rate min max empty n_unique whitespace name 0 1.00 3 21 0 87 0 hair_color 5 0.94 4 13 0 12 0 skin_color 0 1.00 3 19 0 31 0 eye_color 0 1.00 3 13 0 15 0 sex 4 0.95 4 14 0 4 0 gender 4 0.95 8 9 0 2 0 homeworld 10 0.89 4 14 0 48 0 species 4 0.95 3 14 0 37 0 Variable type: list skim_variable n_missing complete_rate n_unique min_length max_length films 0 1 24 1 7 vehicles 0 1 11 0 2 starships 0 1 17 0 5 Variable type: numeric skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100 hist height 6 0.93 174.36 34.77 66 167.0 180 191.0 264 ▁▁▇▅▁ mass 28 0.68 97.31 169.46 15 55.6 79 84.5 1358 ▇▁▁▁▁ birth_year 44 0.49 87.57 154.69 8 35.0 52 72.0 896 ▇▁▁▁▁ 4.5.6.3 dfSummary() The summarytools package is another great package for taking a look at the data. It renders a nice html output for the data frame including a lot of helpful information. You can find out more about this package here. df.starwars %&gt;% select(where(~ !is.list(.))) %&gt;% # this removes all list columns summarytools::dfSummary() %&gt;% summarytools::view() Note: The summarytools::view() function will not show up here in the html. It generates a summary of the data that is displayed in the Viewer in RStudio. Once we’ve taken a look at the data, the next step would be to visualize relationships between variables of interest. 4.6 Wrangling data We use the functions in the package dplyr to manipulate our data. 4.6.1 filter() filter() lets us apply logical (and other) operators (see Table 4.3) to subset the data. Here, I’ve filtered out the masculine characters. df.starwars %&gt;% filter(gender == &quot;masculine&quot;) # A tibble: 66 × 14 name height mass hair_color skin_color eye_color birth_year sex gender &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 Luke Sk… 172 77 blond fair blue 19 male mascu… 2 C-3PO 167 75 &lt;NA&gt; gold yellow 112 none mascu… 3 R2-D2 96 32 &lt;NA&gt; white, bl… red 33 none mascu… 4 Darth V… 202 136 none white yellow 41.9 male mascu… 5 Owen La… 178 120 brown, gr… light blue 52 male mascu… 6 R5-D4 97 32 &lt;NA&gt; white, red red NA none mascu… 7 Biggs D… 183 84 black light brown 24 male mascu… 8 Obi-Wan… 182 77 auburn, w… fair blue-gray 57 male mascu… 9 Anakin … 188 84 blond fair blue 41.9 male mascu… 10 Wilhuff… 180 NA auburn, g… fair blue 64 male mascu… # … with 56 more rows, and 5 more variables: homeworld &lt;chr&gt;, species &lt;chr&gt;, # films &lt;list&gt;, vehicles &lt;list&gt;, starships &lt;list&gt; We can combine multiple conditions in the same call. Here, I’ve filtered out masculine characters, whose height is greater than the median height (i.e. they are in the top 50 percentile), and whose mass was not NA. df.starwars %&gt;% filter(gender == &quot;masculine&quot;, height &gt; median(height, na.rm = T), !is.na(mass)) # A tibble: 27 × 14 name height mass hair_color skin_color eye_color birth_year sex gender &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 Darth V… 202 136 none white yellow 41.9 male mascu… 2 Biggs D… 183 84 black light brown 24 male mascu… 3 Obi-Wan… 182 77 auburn, w… fair blue-gray 57 male mascu… 4 Anakin … 188 84 blond fair blue 41.9 male mascu… 5 Chewbac… 228 112 brown unknown blue 200 male mascu… 6 Boba Fe… 183 78.2 black fair brown 31.5 male mascu… 7 IG-88 200 140 none metal red 15 none mascu… 8 Bossk 190 113 none green red 53 male mascu… 9 Qui-Gon… 193 89 brown fair blue 92 male mascu… 10 Nute Gu… 191 90 none mottled g… red NA male mascu… # … with 17 more rows, and 5 more variables: homeworld &lt;chr&gt;, species &lt;chr&gt;, # films &lt;list&gt;, vehicles &lt;list&gt;, starships &lt;list&gt; Many functions like mean(), median(), var(), sd(), sum() have the argument na.rm which is set to FALSE by default. I set the argument to TRUE here (or T for short), which means that the NA values are ignored, and the median() is calculated based on the remaining values. You can use , and &amp; interchangeably in filter(). Make sure to use parentheses when combining several logical operators to indicate which logical operation should be performed first: df.starwars %&gt;% filter((skin_color %in% c(&quot;dark&quot;, &quot;pale&quot;) | sex == &quot;hermaphroditic&quot;) &amp; height &gt; 170) # A tibble: 10 × 14 name height mass hair_color skin_color eye_color birth_year sex gender &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 Jabba D… 175 1358 &lt;NA&gt; green-tan… orange 600 herm… mascu… 2 Lando C… 177 79 black dark brown 31 male mascu… 3 Quarsh … 183 NA black dark brown 62 &lt;NA&gt; &lt;NA&gt; 4 Bib For… 180 NA none pale pink NA male mascu… 5 Mace Wi… 188 84 none dark brown 72 male mascu… 6 Ki-Adi-… 198 82 white pale yellow 92 male mascu… 7 Adi Gal… 184 50 none dark blue NA fema… femin… 8 Saesee … 188 NA none pale orange NA male mascu… 9 Gregar … 185 85 black dark brown NA male mascu… 10 Sly Moo… 178 48 none pale white NA &lt;NA&gt; &lt;NA&gt; # … with 5 more variables: homeworld &lt;chr&gt;, species &lt;chr&gt;, films &lt;list&gt;, # vehicles &lt;list&gt;, starships &lt;list&gt; The starwars characters that have either a \"dark\" or a \"pale\" skin tone, or whose sex is \"hermaphroditic\", and whose height is at least 170 cm. The %in% operator is useful when there are multiple options. Instead of skin_color %in% c(\"dark\", \"pale\"), I could have also written skin_color == \"dark\" | skin_color == \"pale\" but this gets cumbersome as the number of options increases. 4.6.2 arrange() arrange() allows us to sort the values in a data frame by one or more column entries. df.starwars %&gt;% arrange(hair_color, desc(height)) # A tibble: 87 × 14 name height mass hair_color skin_color eye_color birth_year sex gender &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 Mon Mot… 150 NA auburn fair blue 48 fema… femin… 2 Wilhuff… 180 NA auburn, g… fair blue 64 male mascu… 3 Obi-Wan… 182 77 auburn, w… fair blue-gray 57 male mascu… 4 Bail Pr… 191 NA black tan brown 67 male mascu… 5 Gregar … 185 85 black dark brown NA male mascu… 6 Biggs D… 183 84 black light brown 24 male mascu… 7 Boba Fe… 183 78.2 black fair brown 31.5 male mascu… 8 Quarsh … 183 NA black dark brown 62 &lt;NA&gt; &lt;NA&gt; 9 Jango F… 183 79 black tan brown 66 male mascu… 10 Lando C… 177 79 black dark brown 31 male mascu… # … with 77 more rows, and 5 more variables: homeworld &lt;chr&gt;, species &lt;chr&gt;, # films &lt;list&gt;, vehicles &lt;list&gt;, starships &lt;list&gt; Here, I’ve sorted the data frame first by hair_color, and then by height. I’ve used the desc() function to sort height in descending order. Bail Prestor Organa is the tallest black character in starwars. 4.6.3 rename() rename() renames column names. df.starwars %&gt;% rename(person = name, mass_kg = mass) # A tibble: 87 × 14 person height mass_kg hair_color skin_color eye_color birth_year sex gender &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 Luke … 172 77 blond fair blue 19 male mascu… 2 C-3PO 167 75 &lt;NA&gt; gold yellow 112 none mascu… 3 R2-D2 96 32 &lt;NA&gt; white, bl… red 33 none mascu… 4 Darth… 202 136 none white yellow 41.9 male mascu… 5 Leia … 150 49 brown light brown 19 fema… femin… 6 Owen … 178 120 brown, gr… light blue 52 male mascu… 7 Beru … 165 75 brown light blue 47 fema… femin… 8 R5-D4 97 32 &lt;NA&gt; white, red red NA none mascu… 9 Biggs… 183 84 black light brown 24 male mascu… 10 Obi-W… 182 77 auburn, w… fair blue-gray 57 male mascu… # … with 77 more rows, and 5 more variables: homeworld &lt;chr&gt;, species &lt;chr&gt;, # films &lt;list&gt;, vehicles &lt;list&gt;, starships &lt;list&gt; The new variable names goes on the LHS of the= sign, and the old name on the RHS. To rename all variables at the same time use rename_with(): df.starwars %&gt;% rename_with(.fn = ~ toupper(.)) # A tibble: 87 × 14 NAME HEIGHT MASS HAIR_COLOR SKIN_COLOR EYE_COLOR BIRTH_YEAR SEX GENDER &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 Luke Sk… 172 77 blond fair blue 19 male mascu… 2 C-3PO 167 75 &lt;NA&gt; gold yellow 112 none mascu… 3 R2-D2 96 32 &lt;NA&gt; white, bl… red 33 none mascu… 4 Darth V… 202 136 none white yellow 41.9 male mascu… 5 Leia Or… 150 49 brown light brown 19 fema… femin… 6 Owen La… 178 120 brown, gr… light blue 52 male mascu… 7 Beru Wh… 165 75 brown light blue 47 fema… femin… 8 R5-D4 97 32 &lt;NA&gt; white, red red NA none mascu… 9 Biggs D… 183 84 black light brown 24 male mascu… 10 Obi-Wan… 182 77 auburn, w… fair blue-gray 57 male mascu… # … with 77 more rows, and 5 more variables: HOMEWORLD &lt;chr&gt;, SPECIES &lt;chr&gt;, # FILMS &lt;list&gt;, VEHICLES &lt;list&gt;, STARSHIPS &lt;list&gt; Notice that I used the ~ here in the function call. I will explain what this does shortly. 4.6.4 relocate() relocate() moves columns. For example, the following piece of code moves the species column to the front of the data frame: df.starwars %&gt;% relocate(species) # A tibble: 87 × 14 species name height mass hair_color skin_color eye_color birth_year sex &lt;chr&gt; &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; 1 Human Luke S… 172 77 blond fair blue 19 male 2 Droid C-3PO 167 75 &lt;NA&gt; gold yellow 112 none 3 Droid R2-D2 96 32 &lt;NA&gt; white, bl… red 33 none 4 Human Darth … 202 136 none white yellow 41.9 male 5 Human Leia O… 150 49 brown light brown 19 fema… 6 Human Owen L… 178 120 brown, gr… light blue 52 male 7 Human Beru W… 165 75 brown light blue 47 fema… 8 Droid R5-D4 97 32 &lt;NA&gt; white, red red NA none 9 Human Biggs … 183 84 black light brown 24 male 10 Human Obi-Wa… 182 77 auburn, w… fair blue-gray 57 male # … with 77 more rows, and 5 more variables: gender &lt;chr&gt;, homeworld &lt;chr&gt;, # films &lt;list&gt;, vehicles &lt;list&gt;, starships &lt;list&gt; We could also move the species column after the name column like so: df.starwars %&gt;% relocate(species, .after = name) # A tibble: 87 × 14 name species height mass hair_color skin_color eye_color birth_year sex &lt;chr&gt; &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; 1 Luke S… Human 172 77 blond fair blue 19 male 2 C-3PO Droid 167 75 &lt;NA&gt; gold yellow 112 none 3 R2-D2 Droid 96 32 &lt;NA&gt; white, bl… red 33 none 4 Darth … Human 202 136 none white yellow 41.9 male 5 Leia O… Human 150 49 brown light brown 19 fema… 6 Owen L… Human 178 120 brown, gr… light blue 52 male 7 Beru W… Human 165 75 brown light blue 47 fema… 8 R5-D4 Droid 97 32 &lt;NA&gt; white, red red NA none 9 Biggs … Human 183 84 black light brown 24 male 10 Obi-Wa… Human 182 77 auburn, w… fair blue-gray 57 male # … with 77 more rows, and 5 more variables: gender &lt;chr&gt;, homeworld &lt;chr&gt;, # films &lt;list&gt;, vehicles &lt;list&gt;, starships &lt;list&gt; 4.6.5 select() select() allows us to select a subset of the columns in the data frame. df.starwars %&gt;% select(name, height, mass) # A tibble: 87 × 3 name height mass &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; 1 Luke Skywalker 172 77 2 C-3PO 167 75 3 R2-D2 96 32 4 Darth Vader 202 136 5 Leia Organa 150 49 6 Owen Lars 178 120 7 Beru Whitesun lars 165 75 8 R5-D4 97 32 9 Biggs Darklighter 183 84 10 Obi-Wan Kenobi 182 77 # … with 77 more rows We can select multiple columns using the (from:to) syntax: df.starwars %&gt;% select(name:birth_year) # from name to birth_year # A tibble: 87 × 7 name height mass hair_color skin_color eye_color birth_year &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; 1 Luke Skywalker 172 77 blond fair blue 19 2 C-3PO 167 75 &lt;NA&gt; gold yellow 112 3 R2-D2 96 32 &lt;NA&gt; white, bl… red 33 4 Darth Vader 202 136 none white yellow 41.9 5 Leia Organa 150 49 brown light brown 19 6 Owen Lars 178 120 brown, grey light blue 52 7 Beru Whitesun lars 165 75 brown light blue 47 8 R5-D4 97 32 &lt;NA&gt; white, red red NA 9 Biggs Darklighter 183 84 black light brown 24 10 Obi-Wan Kenobi 182 77 auburn, white fair blue-gray 57 # … with 77 more rows Or use a variable for column selection: columns = c(&quot;name&quot;, &quot;height&quot;, &quot;species&quot;) df.starwars %&gt;% select(one_of(columns)) # useful when using a variable for column selection # A tibble: 87 × 3 name height species &lt;chr&gt; &lt;int&gt; &lt;chr&gt; 1 Luke Skywalker 172 Human 2 C-3PO 167 Droid 3 R2-D2 96 Droid 4 Darth Vader 202 Human 5 Leia Organa 150 Human 6 Owen Lars 178 Human 7 Beru Whitesun lars 165 Human 8 R5-D4 97 Droid 9 Biggs Darklighter 183 Human 10 Obi-Wan Kenobi 182 Human # … with 77 more rows We can also deselect (multiple) columns: df.starwars %&gt;% select(-name, -(birth_year:vehicles)) # A tibble: 87 × 6 height mass hair_color skin_color eye_color starships &lt;int&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;list&gt; 1 172 77 blond fair blue &lt;chr [2]&gt; 2 167 75 &lt;NA&gt; gold yellow &lt;chr [0]&gt; 3 96 32 &lt;NA&gt; white, blue red &lt;chr [0]&gt; 4 202 136 none white yellow &lt;chr [1]&gt; 5 150 49 brown light brown &lt;chr [0]&gt; 6 178 120 brown, grey light blue &lt;chr [0]&gt; 7 165 75 brown light blue &lt;chr [0]&gt; 8 97 32 &lt;NA&gt; white, red red &lt;chr [0]&gt; 9 183 84 black light brown &lt;chr [1]&gt; 10 182 77 auburn, white fair blue-gray &lt;chr [5]&gt; # … with 77 more rows And select columns by partially matching the column name: df.starwars %&gt;% select(contains(&quot;_&quot;)) # every column that contains the character &quot;_&quot; # A tibble: 87 × 4 hair_color skin_color eye_color birth_year &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; 1 blond fair blue 19 2 &lt;NA&gt; gold yellow 112 3 &lt;NA&gt; white, blue red 33 4 none white yellow 41.9 5 brown light brown 19 6 brown, grey light blue 52 7 brown light blue 47 8 &lt;NA&gt; white, red red NA 9 black light brown 24 10 auburn, white fair blue-gray 57 # … with 77 more rows df.starwars %&gt;% select(starts_with(&quot;h&quot;)) # every column that starts with an &quot;h&quot; # A tibble: 87 × 3 height hair_color homeworld &lt;int&gt; &lt;chr&gt; &lt;chr&gt; 1 172 blond Tatooine 2 167 &lt;NA&gt; Tatooine 3 96 &lt;NA&gt; Naboo 4 202 none Tatooine 5 150 brown Alderaan 6 178 brown, grey Tatooine 7 165 brown Tatooine 8 97 &lt;NA&gt; Tatooine 9 183 black Tatooine 10 182 auburn, white Stewjon # … with 77 more rows We can rename some of the columns using select() like so: df.starwars %&gt;% select(person = name, height, mass_kg = mass) # A tibble: 87 × 3 person height mass_kg &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; 1 Luke Skywalker 172 77 2 C-3PO 167 75 3 R2-D2 96 32 4 Darth Vader 202 136 5 Leia Organa 150 49 6 Owen Lars 178 120 7 Beru Whitesun lars 165 75 8 R5-D4 97 32 9 Biggs Darklighter 183 84 10 Obi-Wan Kenobi 182 77 # … with 77 more rows 4.6.5.1 where() where() is a useful helper function that comes in handy, for example, when we want to select columns based on their data type. df.starwars %&gt;% select(where(fn = is.numeric)) # just select numeric columns # A tibble: 87 × 3 height mass birth_year &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; 1 172 77 19 2 167 75 112 3 96 32 33 4 202 136 41.9 5 150 49 19 6 178 120 52 7 165 75 47 8 97 32 NA 9 183 84 24 10 182 77 57 # … with 77 more rows The following selects all columns that are not numeric: df.starwars %&gt;% select(where(fn = ~ !is.numeric(.))) # selects all columns that are not numeric # A tibble: 87 × 11 name hair_color skin_color eye_color sex gender homeworld species films &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;lis&gt; 1 Luke Sk… blond fair blue male mascu… Tatooine Human &lt;chr&gt; 2 C-3PO &lt;NA&gt; gold yellow none mascu… Tatooine Droid &lt;chr&gt; 3 R2-D2 &lt;NA&gt; white, bl… red none mascu… Naboo Droid &lt;chr&gt; 4 Darth V… none white yellow male mascu… Tatooine Human &lt;chr&gt; 5 Leia Or… brown light brown fema… femin… Alderaan Human &lt;chr&gt; 6 Owen La… brown, gr… light blue male mascu… Tatooine Human &lt;chr&gt; 7 Beru Wh… brown light blue fema… femin… Tatooine Human &lt;chr&gt; 8 R5-D4 &lt;NA&gt; white, red red none mascu… Tatooine Droid &lt;chr&gt; 9 Biggs D… black light brown male mascu… Tatooine Human &lt;chr&gt; 10 Obi-Wan… auburn, w… fair blue-gray male mascu… Stewjon Human &lt;chr&gt; # … with 77 more rows, and 2 more variables: vehicles &lt;list&gt;, starships &lt;list&gt; Note that I used ~ here to indicate that I’m creating an anonymous function to check whether column type is numeric. A one-sided formula (expression beginning with ~) is interpreted as function(x), and wherever x would go in the function is represented by .. So, I could write the same code like so: df.starwars %&gt;% select(where(function(x) !is.numeric(x))) # selects all columns that are not numeric # A tibble: 87 × 11 name hair_color skin_color eye_color sex gender homeworld species films &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;lis&gt; 1 Luke Sk… blond fair blue male mascu… Tatooine Human &lt;chr&gt; 2 C-3PO &lt;NA&gt; gold yellow none mascu… Tatooine Droid &lt;chr&gt; 3 R2-D2 &lt;NA&gt; white, bl… red none mascu… Naboo Droid &lt;chr&gt; 4 Darth V… none white yellow male mascu… Tatooine Human &lt;chr&gt; 5 Leia Or… brown light brown fema… femin… Alderaan Human &lt;chr&gt; 6 Owen La… brown, gr… light blue male mascu… Tatooine Human &lt;chr&gt; 7 Beru Wh… brown light blue fema… femin… Tatooine Human &lt;chr&gt; 8 R5-D4 &lt;NA&gt; white, red red none mascu… Tatooine Droid &lt;chr&gt; 9 Biggs D… black light brown male mascu… Tatooine Human &lt;chr&gt; 10 Obi-Wan… auburn, w… fair blue-gray male mascu… Stewjon Human &lt;chr&gt; # … with 77 more rows, and 2 more variables: vehicles &lt;list&gt;, starships &lt;list&gt; For more details, take a look at the help file for select(), and this this great tutorial in which I learned about some of the more advanced ways of using select(). 4.6.6 Practice 2 Create a data frame that: - only has the species Human and Droid - with the following data columns (in this order): name, species, birth_year, homeworld - is arranged according to birth year (with the lowest entry at the top of the data frame) - and has the name column renamed to person # write your code here 4.6.7 mutate() mutate() is used to change existing columns or make new ones. df.starwars %&gt;% mutate(height = height / 100, # to get height in meters bmi = mass / (height^2)) %&gt;% # bmi = kg / (m^2) select(name, height, mass, bmi) # A tibble: 87 × 4 name height mass bmi &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 Luke Skywalker 1.72 77 26.0 2 C-3PO 1.67 75 26.9 3 R2-D2 0.96 32 34.7 4 Darth Vader 2.02 136 33.3 5 Leia Organa 1.5 49 21.8 6 Owen Lars 1.78 120 37.9 7 Beru Whitesun lars 1.65 75 27.5 8 R5-D4 0.97 32 34.0 9 Biggs Darklighter 1.83 84 25.1 10 Obi-Wan Kenobi 1.82 77 23.2 # … with 77 more rows Here, I’ve calculated the bmi for the different starwars characters. I first mutated the height variable by going from cm to m, and then created the new column “bmi”. A useful helper function for mutate() is ifelse() which is a shorthand for the if-else control flow (Section 4.3.4.1). Here is an example: df.starwars %&gt;% mutate(height_categorical = ifelse(height &gt; median(height, na.rm = T), &quot;tall&quot;, &quot;short&quot;)) %&gt;% select(name, contains(&quot;height&quot;)) # A tibble: 87 × 3 name height height_categorical &lt;chr&gt; &lt;int&gt; &lt;chr&gt; 1 Luke Skywalker 172 short 2 C-3PO 167 short 3 R2-D2 96 short 4 Darth Vader 202 tall 5 Leia Organa 150 short 6 Owen Lars 178 short 7 Beru Whitesun lars 165 short 8 R5-D4 97 short 9 Biggs Darklighter 183 tall 10 Obi-Wan Kenobi 182 tall # … with 77 more rows ifelse() works in the following way: we first specify the condition, then what should be returned if the condition is true, and finally what should be returned otherwise. The more verbose version of the statement above would be: ifelse(test = height &gt; median(height, na.rm = T), yes = \"tall\", no = \"short\") In previous versions of dplyr (the package we use for data wrangling), there were a variety of additional mutate functions such as mutate_at(), mutate_if(), and mutate_all(). In the most recent version of dplyr, these additional functions have been deprecated, and replaced with the flexible across() helper function. 4.6.7.1 across() across() allows us to use the syntax that we’ve learned for select() to select particular variables and apply a function to each of the selected variables. For example, let’s imagine that we want to z-score a number of variables in our data frame. We can do this like so: df.starwars %&gt;% mutate(across(.cols = c(height, mass, birth_year), .fns = scale)) # A tibble: 87 × 14 name height[,1] mass[,1] hair_color skin_color eye_color birth_year[,1] &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; 1 Luke Skyw… -0.0678 -0.120 blond fair blue -0.443 2 C-3PO -0.212 -0.132 &lt;NA&gt; gold yellow 0.158 3 R2-D2 -2.25 -0.385 &lt;NA&gt; white, bl… red -0.353 4 Darth Vad… 0.795 0.228 none white yellow -0.295 5 Leia Orga… -0.701 -0.285 brown light brown -0.443 6 Owen Lars 0.105 0.134 brown, gr… light blue -0.230 7 Beru Whit… -0.269 -0.132 brown light blue -0.262 8 R5-D4 -2.22 -0.385 &lt;NA&gt; white, red red NA 9 Biggs Dar… 0.249 -0.0786 black light brown -0.411 10 Obi-Wan K… 0.220 -0.120 auburn, w… fair blue-gray -0.198 # … with 77 more rows, and 7 more variables: sex &lt;chr&gt;, gender &lt;chr&gt;, # homeworld &lt;chr&gt;, species &lt;chr&gt;, films &lt;list&gt;, vehicles &lt;list&gt;, # starships &lt;list&gt; In the .cols = argument of across(), I’ve specified what variables to mutate. In the .fns = argument, I’ve specified that I want to use the function scale. Note that I wrote the function without (). The .fns argument expects allows these possible values: the function itself, e.g. mean a call to the function with . as a dummy argument, ~ mean(.) (note the ~ before the function call) a list of functions list(mean = mean, median = ~ median(.)) (where I’ve mixed both of the other ways) We can also use names to create new columns: df.starwars %&gt;% mutate(across(.cols = c(height, mass, birth_year), .fns = scale, .names = &quot;{.col}_z&quot;)) %&gt;% select(name, contains(&quot;height&quot;), contains(&quot;mass&quot;), contains(&quot;birth_year&quot;)) # A tibble: 87 × 7 name height height_z[,1] mass mass_z[,1] birth_year birth_year_z[,1] &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 Luke Skywal… 172 -0.0678 77 -0.120 19 -0.443 2 C-3PO 167 -0.212 75 -0.132 112 0.158 3 R2-D2 96 -2.25 32 -0.385 33 -0.353 4 Darth Vader 202 0.795 136 0.228 41.9 -0.295 5 Leia Organa 150 -0.701 49 -0.285 19 -0.443 6 Owen Lars 178 0.105 120 0.134 52 -0.230 7 Beru Whites… 165 -0.269 75 -0.132 47 -0.262 8 R5-D4 97 -2.22 32 -0.385 NA NA 9 Biggs Darkl… 183 0.249 84 -0.0786 24 -0.411 10 Obi-Wan Ken… 182 0.220 77 -0.120 57 -0.198 # … with 77 more rows I’ve specified how I’d like the new variables to be called by using the .names = argument of across(). {.col} stands of the name of the original column, and here I’ve just added _z to each column name for the scaled columns. We can also apply several functions at the same time. df.starwars %&gt;% mutate(across(.cols = c(height, mass, birth_year), .fns = list(z = scale, centered = ~ scale(., scale = FALSE)))) %&gt;% select(name, contains(&quot;height&quot;), contains(&quot;mass&quot;), contains(&quot;birth_year&quot;)) # A tibble: 87 × 10 name height height_z[,1] height_centered… mass mass_z[,1] mass_centered[,… &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 Luke … 172 -0.0678 -2.36 77 -0.120 -20.3 2 C-3PO 167 -0.212 -7.36 75 -0.132 -22.3 3 R2-D2 96 -2.25 -78.4 32 -0.385 -65.3 4 Darth… 202 0.795 27.6 136 0.228 38.7 5 Leia … 150 -0.701 -24.4 49 -0.285 -48.3 6 Owen … 178 0.105 3.64 120 0.134 22.7 7 Beru … 165 -0.269 -9.36 75 -0.132 -22.3 8 R5-D4 97 -2.22 -77.4 32 -0.385 -65.3 9 Biggs… 183 0.249 8.64 84 -0.0786 -13.3 10 Obi-W… 182 0.220 7.64 77 -0.120 -20.3 # … with 77 more rows, and 3 more variables: birth_year &lt;dbl&gt;, # birth_year_z &lt;dbl[,1]&gt;, birth_year_centered &lt;dbl[,1]&gt; Here, I’ve created z-scored and centered (i.e. only subtracted the mean but didn’t divide by the standard deviation) versions of the height, mass, and birth_year columns in one go. You can use the everything() helper function if you want to apply a function to all of the columns in your data frame. df.starwars %&gt;% select(height, mass) %&gt;% mutate(across(.cols = everything(), .fns = as.character)) # transform all columns to characters # A tibble: 87 × 2 height mass &lt;chr&gt; &lt;chr&gt; 1 172 77 2 167 75 3 96 32 4 202 136 5 150 49 6 178 120 7 165 75 8 97 32 9 183 84 10 182 77 # … with 77 more rows Here, I’ve selected some columns first, and then changed the mode to character in each of them. Sometimes, you want to apply a function only to those columns that have a particular data type. This is where where() comes in handy! For example, the following code changes all the numeric columns to character columns: df.starwars %&gt;% mutate(across(.cols = where(~ is.numeric(.)), .fns = ~ as.character(.))) # A tibble: 87 × 14 name height mass hair_color skin_color eye_color birth_year sex gender &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; 1 Luke Sk… 172 77 blond fair blue 19 male mascu… 2 C-3PO 167 75 &lt;NA&gt; gold yellow 112 none mascu… 3 R2-D2 96 32 &lt;NA&gt; white, bl… red 33 none mascu… 4 Darth V… 202 136 none white yellow 41.9 male mascu… 5 Leia Or… 150 49 brown light brown 19 fema… femin… 6 Owen La… 178 120 brown, gr… light blue 52 male mascu… 7 Beru Wh… 165 75 brown light blue 47 fema… femin… 8 R5-D4 97 32 &lt;NA&gt; white, red red &lt;NA&gt; none mascu… 9 Biggs D… 183 84 black light brown 24 male mascu… 10 Obi-Wan… 182 77 auburn, w… fair blue-gray 57 male mascu… # … with 77 more rows, and 5 more variables: homeworld &lt;chr&gt;, species &lt;chr&gt;, # films &lt;list&gt;, vehicles &lt;list&gt;, starships &lt;list&gt; Or we could round all the numeric columns to one digit: df.starwars %&gt;% mutate(across(.cols = where(~ is.numeric(.)), .fns = ~ round(., digits = 1))) # A tibble: 87 × 14 name height mass hair_color skin_color eye_color birth_year sex gender &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 Luke Sk… 172 77 blond fair blue 19 male mascu… 2 C-3PO 167 75 &lt;NA&gt; gold yellow 112 none mascu… 3 R2-D2 96 32 &lt;NA&gt; white, bl… red 33 none mascu… 4 Darth V… 202 136 none white yellow 41.9 male mascu… 5 Leia Or… 150 49 brown light brown 19 fema… femin… 6 Owen La… 178 120 brown, gr… light blue 52 male mascu… 7 Beru Wh… 165 75 brown light blue 47 fema… femin… 8 R5-D4 97 32 &lt;NA&gt; white, red red NA none mascu… 9 Biggs D… 183 84 black light brown 24 male mascu… 10 Obi-Wan… 182 77 auburn, w… fair blue-gray 57 male mascu… # … with 77 more rows, and 5 more variables: homeworld &lt;chr&gt;, species &lt;chr&gt;, # films &lt;list&gt;, vehicles &lt;list&gt;, starships &lt;list&gt; 4.6.8 Practice 3 Compute the body mass index for masculine characters who are human. select only the columns you need filter out only the rows you need make the new variable with the body mass index arrange the data frame starting with the highest body mass index # write your code here 4.7 Additional resources 4.7.1 Cheatsheets base R –&gt; summary of how to use base R (we will mostly use the tidyverse but it’s still important to know how to do things in base R) data transformation –&gt; transforming data using dplyr 4.7.2 Data camp courses dplyr tidyverse working with data in the tidyverse cleaning data cleaning data: case studies string manipulation in R Intermediate R Writing functions in R 4.7.3 Books and chapters Chapters 9-15 in “R for Data Science” Chapter 5 in “Data Visualization - A practical introduction” 4.8 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 DT_0.22 visdat_0.5.3 skimr_2.1.4 [13] knitr_1.39 loaded via a namespace (and not attached): [1] lubridate_1.8.0 assertthat_0.2.1 digest_0.6.29 utf8_1.2.2 [5] R6_2.5.1 cellranger_1.1.0 repr_1.1.4 backports_1.4.1 [9] reprex_2.0.1 evaluate_0.15 highr_0.9 httr_1.4.3 [13] pillar_1.7.0 rlang_1.0.2 readxl_1.4.0 rstudioapi_0.13 [17] jquerylib_0.1.4 rmarkdown_2.14 labeling_0.4.2 htmlwidgets_1.5.4 [21] munsell_0.5.0 broom_0.8.0 compiler_4.1.2 modelr_0.1.8 [25] xfun_0.30 pkgconfig_2.0.3 base64enc_0.1-3 htmltools_0.5.2 [29] tidyselect_1.1.2 bookdown_0.26 fansi_1.0.3 crayon_1.5.1 [33] tzdb_0.3.0 dbplyr_2.1.1 withr_2.5.0 grid_4.1.2 [37] jsonlite_1.8.0 gtable_0.3.0 lifecycle_1.0.1 DBI_1.1.2 [41] magrittr_2.0.3 scales_1.2.0 cli_3.3.0 stringi_1.7.6 [45] farver_2.1.0 fs_1.5.2 xml2_1.3.3 bslib_0.3.1 [49] ellipsis_0.3.2 generics_0.1.2 vctrs_0.4.1 tools_4.1.2 [53] glue_1.6.2 crosstalk_1.2.0 hms_1.1.1 fastmap_1.1.0 [57] yaml_2.3.5 colorspace_2.0-3 rvest_1.0.2 haven_2.5.0 [61] sass_0.4.1 "],["data-wrangling-2.html", "Chapter 5 Data wrangling 2 5.1 Learning goals 5.2 Load packages 5.3 Settings 5.4 Wrangling data (continued) 5.5 Reading in data 5.6 Saving data 5.7 Additional resources 5.8 Session info", " Chapter 5 Data wrangling 2 In this session, we will continue to learn about wrangling data. Some of the functions that I’ll introduce in this session are a little tricky to master. Like learning a new language, it takes some time to get fluent. However, it’s worth investing the time. 5.1 Learning goals Learn how to group and summarize data using group_by() and summarize(). Get familiar with how to reshape data using pivot_longer(), pivot_wider(), separate() and unite(). Learn the basics of how to join multiple data frames with a focus on left_join(). Learn how to deal with missing data entries NA. Master how to read and save data. 5.2 Load packages Let’s first load the packages that we need for this chapter. library(&quot;knitr&quot;) # for rendering the RMarkdown file library(&quot;tidyverse&quot;) # for data wrangling 5.3 Settings # sets how code looks in knitted document opts_chunk$set(comment = &quot;&quot;) # suppresses warning about grouping options(dplyr.summarise.inform = F) 5.4 Wrangling data (continued) 5.4.1 Summarizing data Let’s first load the starwars data set again: df.starwars = starwars A particularly powerful way of interacting with data is by grouping and summarizing it. summarize() returns a single value for each summary that we ask for: df.starwars %&gt;% summarize(height_mean = mean(height, na.rm = T), height_max = max(height, na.rm = T), n = n()) # A tibble: 1 × 3 height_mean height_max n &lt;dbl&gt; &lt;int&gt; &lt;int&gt; 1 174. 264 87 Here, I computed the mean height, the maximum height, and the total number of observations (using the function n()). Let’s say we wanted to get a quick sense for how tall starwars characters from different species are. To do that, we combine grouping with summarizing: df.starwars %&gt;% group_by(species) %&gt;% summarize(height_mean = mean(height, na.rm = T)) # A tibble: 38 × 2 species height_mean &lt;chr&gt; &lt;dbl&gt; 1 Aleena 79 2 Besalisk 198 3 Cerean 198 4 Chagrian 196 5 Clawdite 168 6 Droid 131. 7 Dug 112 8 Ewok 88 9 Geonosian 183 10 Gungan 209. # … with 28 more rows I’ve first used group_by() to group our data frame by the different species, and then used summarize() to calculate the mean height of each species. It would also be useful to know how many observations there are in each group. df.starwars %&gt;% group_by(species) %&gt;% summarize(height_mean = mean(height, na.rm = T), group_size = n()) %&gt;% arrange(desc(group_size)) # A tibble: 38 × 3 species height_mean group_size &lt;chr&gt; &lt;dbl&gt; &lt;int&gt; 1 Human 177. 35 2 Droid 131. 6 3 &lt;NA&gt; 181. 4 4 Gungan 209. 3 5 Kaminoan 221 2 6 Mirialan 168 2 7 Twi&#39;lek 179 2 8 Wookiee 231 2 9 Zabrak 173 2 10 Aleena 79 1 # … with 28 more rows Here, I’ve used the n() function to get the number of observations in each group, and then I’ve arranged the data frame according to group size in descending order. Note that n() always yields the number of observations in each group. If we don’t group the data, then we get the overall number of observations in our data frame (i.e. the number of rows). So, Humans are the largest group in our data frame, followed by Droids (who are considerably smaller) and Gungans (who would make for good Basketball players). Sometimes group_by() is also useful without summarizing the data. For example, we often want to z-score (i.e. normalize) data on the level of individual participants. To do so, we first group the data on the level of participants, and then use mutate() to scale the data. Here is an example: # first let&#39;s generate some random data set.seed(1) # to make this reproducible df.summarize = tibble(participant = rep(1:3, each = 5), judgment = sample(0:100, size = 15, replace = TRUE)) %&gt;% print() # A tibble: 15 × 2 participant judgment &lt;int&gt; &lt;int&gt; 1 1 67 2 1 38 3 1 0 4 1 33 5 1 86 6 2 42 7 2 13 8 2 81 9 2 58 10 2 50 11 3 96 12 3 84 13 3 20 14 3 53 15 3 73 df.summarize %&gt;% group_by(participant) %&gt;% # group by participants mutate(judgment_zscored = scale(judgment)) %&gt;% # z-score data of individual participants ungroup() %&gt;% # ungroup the data frame head(n = 10) # print the top 10 rows # A tibble: 10 × 3 participant judgment judgment_zscored[,1] &lt;int&gt; &lt;int&gt; &lt;dbl&gt; 1 1 67 0.671 2 1 38 -0.205 3 1 0 -1.35 4 1 33 -0.356 5 1 86 1.24 6 2 42 -0.275 7 2 13 -1.45 8 2 81 1.30 9 2 58 0.372 10 2 50 0.0485 First, I’ve generated some random data using the repeat function rep() for making a participant column, and the sample() function to randomly choose values from a range between 0 and 100 with replacement. (We will learn more about these functions later when we look into how to simulate data.) I’ve then grouped the data by participant, and used the scale function to z-score the data. TIP: Don’t forget to ungroup() your data frame. Otherwise, any subsequent operations are applied per group. Sometimes, I want to run operations on each row, rather than per column. For example, let’s say that I wanted each character’s average combined height and mass. Let’s see first what doesn’t work: df.starwars %&gt;% mutate(mean_height_mass = mean(c(height, mass), na.rm = T)) %&gt;% select(name, height, mass, mean_height_mass) # A tibble: 87 × 4 name height mass mean_height_mass &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; 1 Luke Skywalker 172 77 142. 2 C-3PO 167 75 142. 3 R2-D2 96 32 142. 4 Darth Vader 202 136 142. 5 Leia Organa 150 49 142. 6 Owen Lars 178 120 142. 7 Beru Whitesun lars 165 75 142. 8 R5-D4 97 32 142. 9 Biggs Darklighter 183 84 142. 10 Obi-Wan Kenobi 182 77 142. # … with 77 more rows Note that all the values are the same. The value shown here is just the mean of all the values in height and mass. df.starwars %&gt;% select(height, mass) %&gt;% unlist() %&gt;% # turns the data frame into a vector mean(na.rm = T) [1] 141.8886 To get the mean by row, we can either spell out the arithmetic df.starwars %&gt;% mutate(mean_height_mass = (height + mass) / 2) %&gt;% # here, I&#39;ve replaced the mean() function select(name, height, mass, mean_height_mass) # A tibble: 87 × 4 name height mass mean_height_mass &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; 1 Luke Skywalker 172 77 124. 2 C-3PO 167 75 121 3 R2-D2 96 32 64 4 Darth Vader 202 136 169 5 Leia Organa 150 49 99.5 6 Owen Lars 178 120 149 7 Beru Whitesun lars 165 75 120 8 R5-D4 97 32 64.5 9 Biggs Darklighter 183 84 134. 10 Obi-Wan Kenobi 182 77 130. # … with 77 more rows or use the rowwise() helper function which is like group_by() but treats each row like a group: df.starwars %&gt;% rowwise() %&gt;% # now, each row is treated like a separate group mutate(mean_height_mass = mean(c(height, mass), na.rm = T)) %&gt;% ungroup() %&gt;% select(name, height, mass, mean_height_mass) # A tibble: 87 × 4 name height mass mean_height_mass &lt;chr&gt; &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; 1 Luke Skywalker 172 77 124. 2 C-3PO 167 75 121 3 R2-D2 96 32 64 4 Darth Vader 202 136 169 5 Leia Organa 150 49 99.5 6 Owen Lars 178 120 149 7 Beru Whitesun lars 165 75 120 8 R5-D4 97 32 64.5 9 Biggs Darklighter 183 84 134. 10 Obi-Wan Kenobi 182 77 130. # … with 77 more rows 5.4.1.1 Practice 1 Find out what the average height and mass (as well as the standard deviation) is from different species in different homeworlds. Why is the standard deviation NA for many groups? # write your code here Who is the tallest member of each species? What eye color do they have? The top_n() function or the row_number() function (in combination with filter()) will be useful here. # write your code here 5.4.2 Reshaping data We want our data frames to be tidy. What’s tidy? Each variable must have its own column. Each observation must have its own row. Each value must have its own cell. For more information on tidy data frames see the Tidy data chapter in Hadley Wickham’s R for Data Science book. “Happy families are all alike; every unhappy family is unhappy in its own way.” –– Leo Tolstoy “Tidy datasets are all alike, but every messy dataset is messy in its own way.” –– Hadley Wickham 5.4.2.1 pivot_longer() and pivot_wider() Let’s first generate a data set that is not tidy. # construct data frame df.reshape = tibble(participant = c(1, 2), observation_1 = c(10, 25), observation_2 = c(100, 63), observation_3 = c(24, 45)) %&gt;% print() # A tibble: 2 × 4 participant observation_1 observation_2 observation_3 &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 1 10 100 24 2 2 25 63 45 Here, I’ve generated data from two participants with three observations. This data frame is not tidy since each row contains more than a single observation. Data frames that have one row per participant but many observations are called wide data frames. We can make it tidy using the pivot_longer() function. df.reshape.long = df.reshape %&gt;% pivot_longer(cols = -participant, names_to = &quot;index&quot;, values_to = &quot;rating&quot;) %&gt;% arrange(participant) %&gt;% print() # A tibble: 6 × 3 participant index rating &lt;dbl&gt; &lt;chr&gt; &lt;dbl&gt; 1 1 observation_1 10 2 1 observation_2 100 3 1 observation_3 24 4 2 observation_1 25 5 2 observation_2 63 6 2 observation_3 45 df.reshape.long now contains one observation in each row. Data frames with one row per observation are called long data frames. The pivot_longer() function takes at least four arguments: the data which I’ve passed to it via the pipe %&gt;% a specification for which columns we want to gather – here I’ve specified that we want to gather the values from all columns except the participant column a names_to argument which specifies the name of the column which will contain the column names of the original data frame a values_to argument which specifies the name of the column which will contain the values that were spread across different columns in the original data frame pivot_wider() is the counterpart of pivot_longer(). We can use it to go from a data frame that is in long format, to a data frame in wide format, like so: df.reshape.wide = df.reshape.long %&gt;% pivot_wider(names_from = index, values_from = rating) %&gt;% print() # A tibble: 2 × 4 participant observation_1 observation_2 observation_3 &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 1 10 100 24 2 2 25 63 45 For my data, I often have a wide data frame that contains demographic information about participants, and a long data frame that contains participants’ responses in the experiment. In Section 5.4.3, we will learn how to combine information from multiple data frames (with potentially different formats). Here is a more advanced example that involves reshaping a data frame. Let’s consider the following data frame to start with: # construct data frame df.reshape2 = tibble(participant = c(1, 2), stimulus_1 = c(&quot;flower&quot;, &quot;car&quot;), observation_1 = c(10, 25), stimulus_2 = c(&quot;house&quot;, &quot;flower&quot;), observation_2 = c(100, 63), stimulus_3 = c(&quot;car&quot;, &quot;house&quot;), observation_3 = c(24, 45)) %&gt;% print() # A tibble: 2 × 7 participant stimulus_1 observation_1 stimulus_2 observation_2 stimulus_3 &lt;dbl&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; &lt;dbl&gt; &lt;chr&gt; 1 1 flower 10 house 100 car 2 2 car 25 flower 63 house # … with 1 more variable: observation_3 &lt;dbl&gt; The data frame contains in each row: which stimuli a participant saw, and what rating she gave. The participants saw a picture of a flower, car, and house, and rated how much they liked the picture on a scale from 0 to 100. The order at which the pictures were presented was randomized between participants. I will use a combination of pivot_longer(), and pivot_wider() to turn this into a data frame in long format. df.reshape2 %&gt;% pivot_longer(cols = -participant, names_to = c(&quot;index&quot;, &quot;order&quot;), names_sep = &quot;_&quot;, values_to = &quot;rating&quot;, values_transform = list(rating = as.character)) %&gt;% pivot_wider(names_from = &quot;index&quot;, values_from = &quot;rating&quot;) %&gt;% mutate(across(.cols = c(order, observation), .fns = ~ as.numeric(.))) %&gt;% select(participant, order, stimulus, rating = observation) # A tibble: 6 × 4 participant order stimulus rating &lt;dbl&gt; &lt;dbl&gt; &lt;chr&gt; &lt;dbl&gt; 1 1 1 flower 10 2 1 2 house 100 3 1 3 car 24 4 2 1 car 25 5 2 2 flower 63 6 2 3 house 45 Voilà! Getting the desired data frame involved a few new tricks. Let’s take it step by step. First, I use pivot_longer() to make a long table. df.reshape2 %&gt;% pivot_longer(cols = -participant, names_to = c(&quot;index&quot;, &quot;order&quot;), names_sep = &quot;_&quot;, values_to = &quot;rating&quot;, values_transform = list(rating = as.character)) # A tibble: 12 × 4 participant index order rating &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; 1 1 stimulus 1 flower 2 1 observation 1 10 3 1 stimulus 2 house 4 1 observation 2 100 5 1 stimulus 3 car 6 1 observation 3 24 7 2 stimulus 1 car 8 2 observation 1 25 9 2 stimulus 2 flower 10 2 observation 2 63 11 2 stimulus 3 house 12 2 observation 3 45 Notice how I’ve used a combination of the names_to = and names_sep = arguments to create two columns. Because I’m combining data of two different types (“character” and “numeric”), I needed to specify what I want the resulting data type to be via the values_transform = argument. I would like to have the information about the stimulus and the observation in the same row. That is, I want to see what rating a participant gave to the flower stimulus, for example. To get there, I can use the pivot_wider() function to make a separate column for each entry in index that contains the values in rating. df.reshape2 %&gt;% pivot_longer(cols = -participant, names_to = c(&quot;index&quot;, &quot;order&quot;), names_sep = &quot;_&quot;, values_to = &quot;rating&quot;, values_transform = list(rating = as.character)) %&gt;% pivot_wider(names_from = &quot;index&quot;, values_from = &quot;rating&quot;) # A tibble: 6 × 4 participant order stimulus observation &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; 1 1 1 flower 10 2 1 2 house 100 3 1 3 car 24 4 2 1 car 25 5 2 2 flower 63 6 2 3 house 45 That’s pretty much it. Now, each row contains information about the order in which a stimulus was presented, what the stimulus was, and the judgment that a participant made in this trial. df.reshape2 %&gt;% pivot_longer(cols = -participant, names_to = c(&quot;index&quot;, &quot;order&quot;), names_sep = &quot;_&quot;, values_to = &quot;rating&quot;, values_transform = list(rating = as.character)) %&gt;% pivot_wider(names_from = &quot;index&quot;, values_from = &quot;rating&quot;) %&gt;% mutate(across(.cols = c(order, observation), .fns = ~ as.numeric(.))) %&gt;% select(participant, order, stimulus, rating = observation) # A tibble: 6 × 4 participant order stimulus rating &lt;dbl&gt; &lt;dbl&gt; &lt;chr&gt; &lt;dbl&gt; 1 1 1 flower 10 2 1 2 house 100 3 1 3 car 24 4 2 1 car 25 5 2 2 flower 63 6 2 3 house 45 The rest is familiar. I’ve used mutate() with across() to turn order and observation into numeric columns, select() to change the order of the columns (and renamed the observation column to rating along the way). Getting familiar with pivot_longer() and pivot_wider() takes some time plus trial and error. So don’t be discouraged if you don’t get what you want straight away. Once you’ve mastered these functions, they will make it much easier to beat your data frames into shape. After having done some transformations like this, it’s worth checking that nothing went wrong. I often compare a few values in the transformed and original data frame to make sure everything is legit. When reading older code, you will often see gather() (instead of pivot_longer()), and spread() (instead of pivot_wider()). gather and spread are not developed anymore now, and their newer counterparts have additional functionality that comes in handy. 5.4.2.2 separate() and unite() Sometimes, we want to separate one column into multiple columns. For example, we could have achieved the same result we did above slightly differently, like so: df.reshape2 %&gt;% pivot_longer(cols = -participant, names_to = &quot;index&quot;, values_to = &quot;rating&quot;, values_transform = list(rating = as.character)) %&gt;% separate(col = index, into = c(&quot;index&quot;, &quot;order&quot;), sep = &quot;_&quot;) # A tibble: 12 × 4 participant index order rating &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; 1 1 stimulus 1 flower 2 1 observation 1 10 3 1 stimulus 2 house 4 1 observation 2 100 5 1 stimulus 3 car 6 1 observation 3 24 7 2 stimulus 1 car 8 2 observation 1 25 9 2 stimulus 2 flower 10 2 observation 2 63 11 2 stimulus 3 house 12 2 observation 3 45 Here, I’ve used the separate() function to separate the original index column into two columns. The separate() function takes four arguments: the data which I’ve passed to it via the pipe %&gt;% the name of the column col which we want to separate the names of the columns into into which we want to separate the original column the separator sep that we want to use to split the columns. Note, like pivot_longer() and pivot_wider(), there is a partner for separate(), too. It’s called unite() and it allows you to combine several columns into one, like so: tibble(index = c(&quot;flower&quot;, &quot;observation&quot;), order = c(1, 2)) %&gt;% unite(&quot;combined&quot;, index, order) # A tibble: 2 × 1 combined &lt;chr&gt; 1 flower_1 2 observation_2 Sometimes, we may have a data frame where data is recorded in a long string. df.reshape3 = tibble(participant = 1:2, judgments = c(&quot;10, 4, 12, 15&quot;, &quot;3, 4&quot;)) %&gt;% print() # A tibble: 2 × 2 participant judgments &lt;int&gt; &lt;chr&gt; 1 1 10, 4, 12, 15 2 2 3, 4 Here, I’ve created a data frame with data from two participants. For whatever reason, we have four judgments from participant 1 and only two judgments from participant 2 (data is often messy in real life, too!). We can use the separate_rows() function to turn this into a tidy data frame in long format. df.reshape3 %&gt;% separate_rows(judgments) # A tibble: 6 × 2 participant judgments &lt;int&gt; &lt;chr&gt; 1 1 10 2 1 4 3 1 12 4 1 15 5 2 3 6 2 4 5.4.2.3 Practice 2 Load this data frame first. df.practice2 = tibble(participant = 1:10, initial = c(&quot;AR&quot;, &quot;FA&quot;, &quot;IR&quot;, &quot;NC&quot;, &quot;ER&quot;, &quot;PI&quot;, &quot;DH&quot;, &quot;CN&quot;, &quot;WT&quot;, &quot;JD&quot;), judgment_1 = c(12, 13, 1, 14, 5, 6, 12, 41, 100, 33), judgment_2 = c(2, 20, 10, 89, 94, 27, 29, 19, 57, 74), judgment_3 = c(2, 20, 10, 89, 94, 27, 29, 19, 57, 74)) Make the df.practice2 data frame tidy (by turning into a long format). Compute the z-score of each participants’ judgments (using the scale() function). Calculate the mean and standard deviation of each participants’ z-scored judgments. Notice anything interesting? Think about what z-scoring does … # write your code here 5.4.3 Joining multiple data frames It’s nice to have all the information we need in a single, tidy data frame. We have learned above how to go from a single untidy data frame to a tidy one. However, often our situation to start off with is even worse. The information we need sits in several, messy data frames. For example, we may have one data frame df.stimuli with information about each stimulus, and then have another data frame with participants’ responses df.responses that only contains a stimulus index but no other infromation about the stimuli. set.seed(1) # setting random seed to make this example reproducible # data frame with stimulus information df.stimuli = tibble(index = 1:5, height = c(2, 3, 1, 4, 5), width = c(4, 5, 2, 3, 1), n_dots = c(12, 15, 5, 13, 7), color = c(&quot;green&quot;, &quot;blue&quot;, &quot;white&quot;, &quot;red&quot;, &quot;black&quot;)) %&gt;% print() # A tibble: 5 × 5 index height width n_dots color &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;chr&gt; 1 1 2 4 12 green 2 2 3 5 15 blue 3 3 1 2 5 white 4 4 4 3 13 red 5 5 5 1 7 black # data frame with participants&#39; responses df.responses = tibble(participant = rep(1:3, each = 5), index = rep(1:5, 3), response = sample(0:100, size = 15, replace = TRUE)) %&gt;% # randomly sample 15 values from 0 to 100 print() # A tibble: 15 × 3 participant index response &lt;int&gt; &lt;int&gt; &lt;int&gt; 1 1 1 67 2 1 2 38 3 1 3 0 4 1 4 33 5 1 5 86 6 2 1 42 7 2 2 13 8 2 3 81 9 2 4 58 10 2 5 50 11 3 1 96 12 3 2 84 13 3 3 20 14 3 4 53 15 3 5 73 The df.stimuli data frame contains an index, information about the height, and width, as well as the number of dots, and their color. Let’s imagine that participants had to judge how much they liked each image from a scale of 0 (“not liking this dot pattern at all”) to 100 (“super thrilled about this dot pattern”). Let’s say that I now wanted to know what participants’ average response for the differently colored dot patterns are. Here is how I would do this: df.responses %&gt;% left_join(df.stimuli %&gt;% select(index, color), by = &quot;index&quot;) %&gt;% group_by(color) %&gt;% summarize(response_mean = mean(response)) # A tibble: 5 × 2 color response_mean &lt;chr&gt; &lt;dbl&gt; 1 black 69.7 2 blue 45 3 green 68.3 4 red 48 5 white 33.7 Let’s take it step by step. The key here is to add the information from the df.stimuli data frame to the df.responses data frame. df.responses %&gt;% left_join(df.stimuli %&gt;% select(index, color), by = &quot;index&quot;) # A tibble: 15 × 4 participant index response color &lt;int&gt; &lt;int&gt; &lt;int&gt; &lt;chr&gt; 1 1 1 67 green 2 1 2 38 blue 3 1 3 0 white 4 1 4 33 red 5 1 5 86 black 6 2 1 42 green 7 2 2 13 blue 8 2 3 81 white 9 2 4 58 red 10 2 5 50 black 11 3 1 96 green 12 3 2 84 blue 13 3 3 20 white 14 3 4 53 red 15 3 5 73 black I’ve joined the df.stimuli table in which I’ve only selected the index and color column, with the df.responses table, and specified the index column as the one by which the tables should be joined. This is the only column that both of the data frames have in common. To specify multiple columns by which we would like to join tables, we specify the by argument as follows: by = c(\"one_column\", \"another_column\"). Sometimes, the tables I want to join don’t have any column names in common. In that case, we can tell the left_join() function which column pair(s) should be used for joining. df.responses %&gt;% rename(stimuli = index) %&gt;% # I&#39;ve renamed the index column to stimuli left_join(df.stimuli %&gt;% select(index, color), by = c(&quot;stimuli&quot; = &quot;index&quot;)) # A tibble: 15 × 4 participant stimuli response color &lt;int&gt; &lt;int&gt; &lt;int&gt; &lt;chr&gt; 1 1 1 67 green 2 1 2 38 blue 3 1 3 0 white 4 1 4 33 red 5 1 5 86 black 6 2 1 42 green 7 2 2 13 blue 8 2 3 81 white 9 2 4 58 red 10 2 5 50 black 11 3 1 96 green 12 3 2 84 blue 13 3 3 20 white 14 3 4 53 red 15 3 5 73 black Here, I’ve first renamed the index column (to create the problem) and then used the by = c(\"stimuli\" = \"index\") construction (to solve the problem). In my experience, it often takes a little bit of playing around to make sure that the data frames were joined as intended. One very good indicator is the row number of the initial data frame, and the joined one. For a left_join(), most of the time, we want the row number of the original data frame (“the one on the left”) and the joined data frame to be the same. If the row number changed, something probably went wrong. Take a look at the join help file to see other operations for combining two or more data frames into one (make sure to look at the one from the dplyr package). 5.4.3.1 Practice 3 Load these three data frames first: set.seed(1) df.judgments = tibble(participant = rep(1:3, each = 5), stimulus = rep(c(&quot;red&quot;, &quot;green&quot;, &quot;blue&quot;), 5), judgment = sample(0:100, size = 15, replace = T)) df.information = tibble(number = seq(from = 0, to = 100, length.out = 5), color = c(&quot;red&quot;, &quot;green&quot;, &quot;blue&quot;, &quot;black&quot;, &quot;white&quot;)) Create a new data frame called df.join that combines the information from both df.judgments and df.information. Note that column with the colors is called stimulus in df.judgments and color in df.information. At the end, you want a data frame that contains the following columns: participant, stimulus, number, and judgment. # write your code here 5.4.4 Dealing with missing data There are two ways for data to be missing. implicit: data is not present in the table explicit: data is flagged with NA We can check for explicit missing values using the is.na() function like so: tmp.na = c(1, 2, NA, 3) is.na(tmp.na) [1] FALSE FALSE TRUE FALSE I’ve first created a vector tmp.na with a missing value at index 3. Calling the is.na() function on this vector yields a logical vector with FALSE for each value that is not missing, and TRUE for each missing value. Let’s say that we have a data frame with missing values and that we want to replace those missing values with something else. Let’s first create a data frame with missing values. df.missing = tibble(x = c(1, 2, NA), y = c(&quot;a&quot;, NA, &quot;b&quot;)) print(df.missing) # A tibble: 3 × 2 x y &lt;dbl&gt; &lt;chr&gt; 1 1 a 2 2 &lt;NA&gt; 3 NA b We can use the replace_na() function to replace the missing values with something else. df.missing %&gt;% mutate(x = replace_na(x, replace = 0), y = replace_na(y, replace = &quot;unknown&quot;)) # A tibble: 3 × 2 x y &lt;dbl&gt; &lt;chr&gt; 1 1 a 2 2 unknown 3 0 b We can also remove rows with missing values using the drop_na() function. df.missing %&gt;% drop_na() # A tibble: 1 × 2 x y &lt;dbl&gt; &lt;chr&gt; 1 1 a If we only want to drop values from specific columns, we can specify these columns within the drop_na() function call. So, if we only want to drop rows that have missing values in the x column, we can write: df.missing %&gt;% drop_na(x) # A tibble: 2 × 2 x y &lt;dbl&gt; &lt;chr&gt; 1 1 a 2 2 &lt;NA&gt; To make the distinction between implicit and explicit missing values more concrete, let’s consider the following example (taken from here): df.stocks = tibble(year = c(2015, 2015, 2015, 2015, 2016, 2016, 2016), qtr = c( 1, 2, 3, 4, 2, 3, 4), return = c(1.88, 0.59, 0.35, NA, 0.92, 0.17, 2.66)) There are two missing values in this dataset: The return for the fourth quarter of 2015 is explicitly missing, because the cell where its value should be instead contains NA. The return for the first quarter of 2016 is implicitly missing, because it simply does not appear in the dataset. We can use the complete() function to make implicit missing values explicit: df.stocks %&gt;% complete(year, qtr) # A tibble: 8 × 3 year qtr return &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 2015 1 1.88 2 2015 2 0.59 3 2015 3 0.35 4 2015 4 NA 5 2016 1 NA 6 2016 2 0.92 7 2016 3 0.17 8 2016 4 2.66 Note how now, the data frame contains an additional row in which year = 2016, qtr = 1 and return = NA even though we didn’t originally specify this. We can also directly tell the complete() function to replace the NA values via passing a list to its fill argument like so: df.stocks %&gt;% complete(year, qtr, fill = list(return = 0)) # A tibble: 8 × 3 year qtr return &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 2015 1 1.88 2 2015 2 0.59 3 2015 3 0.35 4 2015 4 0 5 2016 1 0 6 2016 2 0.92 7 2016 3 0.17 8 2016 4 2.66 This specifies that we would like to replace any NA in the return column with 0. Again, if we had multiple columns with NAs, we could speficy for each column separately how to replace it. 5.5 Reading in data So far, we’ve used data sets that already came with the packages we’ve loaded. In the visualization chapters, we used the diamonds data set from the ggplot2 package, and in the data wrangling chapters, we used the starwars data set from the dplyr package. file type platform description csv general medium-size data frames RData R saving the results of intensive computations xls excel people who use excel json general more complex data structures feather python &amp; R fast interaction between R and python The foreign package helps with importing data that was saved in SPSS, Stata, or Minitab. For data in a json format, I highly recommend the tidyjson package. 5.5.1 csv I’ve stored some data files in the data/ subfolder. Let’s first read a csv (= comma-separated-value) file. df.csv = read_csv(&quot;data/movies.csv&quot;) Rows: 2961 Columns: 11 ── Column specification ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Delimiter: &quot;,&quot; chr (3): title, genre, director dbl (8): year, duration, gross, budget, cast_facebook_likes, votes, reviews,... ℹ Use `spec()` to retrieve the full column specification for this data. ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message. The read_csv() function gives us information about how each column was parsed. Here, we have some columns that are characters (such as title and genre), and some columns that are numeric (such as year and duration). Note that it says double() in the specification but double and numeric are identical. And let’s take a quick peek at the data: df.csv %&gt;% glimpse() Rows: 2,961 Columns: 11 $ title &lt;chr&gt; &quot;Over the Hill to the Poorhouse&quot;, &quot;The Broadway Me… $ genre &lt;chr&gt; &quot;Crime&quot;, &quot;Musical&quot;, &quot;Comedy&quot;, &quot;Comedy&quot;, &quot;Comedy&quot;, … $ director &lt;chr&gt; &quot;Harry F. Millarde&quot;, &quot;Harry Beaumont&quot;, &quot;Lloyd Baco… $ year &lt;dbl&gt; 1920, 1929, 1933, 1935, 1936, 1937, 1939, 1939, 19… $ duration &lt;dbl&gt; 110, 100, 89, 81, 87, 83, 102, 226, 88, 144, 172, … $ gross &lt;dbl&gt; 3000000, 2808000, 2300000, 3000000, 163245, 184925… $ budget &lt;dbl&gt; 100000, 379000, 439000, 609000, 1500000, 2000000, … $ cast_facebook_likes &lt;dbl&gt; 4, 109, 995, 824, 352, 229, 2509, 1862, 1178, 2037… $ votes &lt;dbl&gt; 5, 4546, 7921, 13269, 143086, 133348, 291875, 2153… $ reviews &lt;dbl&gt; 2, 107, 162, 164, 331, 349, 746, 863, 252, 119, 33… $ rating &lt;dbl&gt; 4.8, 6.3, 7.7, 7.8, 8.6, 7.7, 8.1, 8.2, 7.5, 6.9, … The data frame contains a bunch of movies with information about their genre, director, rating, etc. The readr package (which contains the read_csv() function) has a number of other functions for reading data. Just type read_ in the console below and take a look at the suggestions that autocomplete offers. 5.5.2 RData RData is a data format native to R. Since this format can only be read by R, it’s not a good format for sharing data. However, it’s a useful format that allows us to flexibly save and load R objects. For example, consider that we always start our script by reading in and structuring data, and that this takes quite a while. One thing we can do is to save the output of intermediate steps as an RData object, and then simply load this object (instead of re-running the whole routine every time). We read (or load) an RData file in the following way: load(&quot;data/test.RData&quot;, verbose = TRUE) Loading objects: df.test I’ve set the verbose = argument to TRUE here so that the load() function tells me what objects it added to the environment. This is useful for checking whether existing objects were overwritten. 5.6 Saving data 5.6.1 csv To save a data frame as a csv file, we simply write: df.test = tibble(x = 1:3, y = c(&quot;test1&quot;, &quot;test2&quot;, &quot;test3&quot;)) write_csv(df.test, file = &quot;data/test.csv&quot;) Just like for reading in data, the readr package has a number of other functions for saving data. Just type write_ in the console below and take a look at the autocomplete suggestions. 5.6.2 RData To save objects as an RData file, we write: save(df.test, file = &quot;data/test.RData&quot;) We can add multiple objects simply by adding them at the beginning, like so: save(df.test, df.starwars, file = &quot;data/test_starwars.RData&quot;) 5.7 Additional resources 5.7.1 Cheatsheets wrangling data –&gt; wrangling data using dplyr and tidyr importing &amp; saving data –&gt; importing and saving data with readr 5.7.2 Data camp courses Joining tables writing functions importing data 1 importing data 2 categorical data dealing with missing data 5.7.3 Books and chapters Chapters 17-21 in R for Data Science Exploratory data analysis R programming for data science 5.7.4 Tutorials Joining data: Two-table verbs Tutorial by Jenny Bryan tidyexplain: Animations that illustrate how pivot_longer(), pivot_wider(), left_join(), etc. work 5.8 Session info R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 knitr_1.39 loaded via a namespace (and not attached): [1] tidyselect_1.1.2 xfun_0.30 bslib_0.3.1 haven_2.5.0 [5] colorspace_2.0-3 vctrs_0.4.1 generics_0.1.2 htmltools_0.5.2 [9] yaml_2.3.5 utf8_1.2.2 rlang_1.0.2 jquerylib_0.1.4 [13] pillar_1.7.0 withr_2.5.0 glue_1.6.2 DBI_1.1.2 [17] bit64_4.0.5 dbplyr_2.1.1 modelr_0.1.8 readxl_1.4.0 [21] lifecycle_1.0.1 munsell_0.5.0 gtable_0.3.0 cellranger_1.1.0 [25] rvest_1.0.2 evaluate_0.15 tzdb_0.3.0 fastmap_1.1.0 [29] parallel_4.1.2 fansi_1.0.3 highr_0.9 broom_0.8.0 [33] backports_1.4.1 scales_1.2.0 vroom_1.5.7 jsonlite_1.8.0 [37] bit_4.0.4 fs_1.5.2 hms_1.1.1 digest_0.6.29 [41] stringi_1.7.6 bookdown_0.26 grid_4.1.2 cli_3.3.0 [45] tools_4.1.2 magrittr_2.0.3 sass_0.4.1 crayon_1.5.1 [49] pkgconfig_2.0.3 ellipsis_0.3.2 xml2_1.3.3 reprex_2.0.1 [53] lubridate_1.8.0 rstudioapi_0.13 assertthat_0.2.1 rmarkdown_2.14 [57] httr_1.4.3 R6_2.5.1 compiler_4.1.2 "],["probability-and-causality.html", "Chapter 6 Probability and causality 6.1 Load packages, load data, set theme 6.2 Counting 6.3 The random secretary 6.4 Flipping a coin many times 6.5 Clue guide to probability 6.6 Probability operations 6.7 Bayesian reasoning explained 6.8 Getting Bayes right matters 6.9 Building a Bayesis 6.10 Additional resources 6.11 Session info", " Chapter 6 Probability and causality 6.1 Load packages, load data, set theme Let’s load the packages that we need for this chapter. library(&quot;knitr&quot;) # for rendering the RMarkdown file library(&quot;kableExtra&quot;) # for nicely formatted tables library(&quot;arrangements&quot;) # fast generators and iterators for creating combinations library(&quot;DiagrammeR&quot;) # for drawing diagrams library(&quot;tidyverse&quot;) # for data wrangling Set the plotting theme. theme_set(theme_classic() + theme(text = element_text(size = 20))) opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 6.2 Counting Imagine that there are three balls in an urn. The balls are labeled 1, 2, and 3. Let’s consider a few possible situations. balls = 1:3 # number of balls in urn ndraws = 2 # number of draws # order matters, without replacement permutations(balls, ndraws) [,1] [,2] [1,] 1 2 [2,] 1 3 [3,] 2 1 [4,] 2 3 [5,] 3 1 [6,] 3 2 # order matters, with replacement permutations(balls, ndraws, replace = T) [,1] [,2] [1,] 1 1 [2,] 1 2 [3,] 1 3 [4,] 2 1 [5,] 2 2 [6,] 2 3 [7,] 3 1 [8,] 3 2 [9,] 3 3 # order doesn&#39;t matter, with replacement combinations(balls, ndraws, replace = T) [,1] [,2] [1,] 1 1 [2,] 1 2 [3,] 1 3 [4,] 2 2 [5,] 2 3 [6,] 3 3 # order doesn&#39;t matter, without replacement combinations(balls, ndraws) [,1] [,2] [1,] 1 2 [2,] 1 3 [3,] 2 3 I’ve generated the figures below using the DiagrammeR package. It’s a powerful package for drawing diagrams in R. See information on how to use the DiagrammeR package here. Figure 6.1: Drawing two marbles out of an urn with replacement. Figure 6.2: Drawing two marbles out of an urn without replacement. 6.3 The random secretary A secretary types four letters to four people and addresses the four envelopes. If he inserts the letters at random, each in a different envelope, what is the probability that exactly three letters will go into the right envelope? df.letters = permutations(x = 1:4, k = 4) %&gt;% as_tibble(.name_repair = ~ str_c(&quot;person_&quot;, 1:4)) %&gt;% mutate(n_correct = (person_1 == 1) + (person_2 == 2) + (person_3 == 3) + (person_4 == 4)) df.letters %&gt;% summarize(prob_3_correct = sum(n_correct == 3) / n()) # A tibble: 1 × 1 prob_3_correct &lt;dbl&gt; 1 0 ggplot(data = df.letters, mapping = aes(x = n_correct)) + geom_bar(aes(y = stat(count)/sum(count)), color = &quot;black&quot;, fill = &quot;lightblue&quot;) + scale_y_continuous(labels = scales::percent, expand = c(0, 0)) + labs(x = &quot;number correct&quot;, y = &quot;probability&quot;) 6.4 Flipping a coin many times # Example taken from here: http://statsthinking21.org/probability.html#empirical-frequency set.seed(1) # set the seed so that the outcome is consistent nsamples = 50000 # how many flips do we want to make? # create some random coin flips using the rbinom() function with # a true probability of 0.5 df.samples = tibble(trial_number = seq(nsamples), outcomes = rbinom(nsamples, 1, 0.5)) %&gt;% mutate(mean_probability = cumsum(outcomes) / seq_along(outcomes)) %&gt;% filter(trial_number &gt;= 10) # start with a minimum sample of 10 flips ggplot(data = df.samples, mapping = aes(x = trial_number, y = mean_probability)) + geom_hline(yintercept = 0.5, color = &quot;gray&quot;, linetype = &quot;dashed&quot;) + geom_line() + labs(x = &quot;Number of trials&quot;, y = &quot;Estimated probability of heads&quot;) + theme_classic() + theme(text = element_text(size = 20)) Figure 2.7: A demonstration of the law of large numbers. 6.5 Clue guide to probability who = c(&quot;ms_scarlet&quot;, &quot;col_mustard&quot;, &quot;mrs_white&quot;, &quot;mr_green&quot;, &quot;mrs_peacock&quot;, &quot;prof_plum&quot;) what = c(&quot;candlestick&quot;, &quot;knife&quot;, &quot;lead_pipe&quot;, &quot;revolver&quot;, &quot;rope&quot;, &quot;wrench&quot;) where = c(&quot;study&quot;, &quot;kitchen&quot;, &quot;conservatory&quot;, &quot;lounge&quot;, &quot;billiard_room&quot;, &quot;hall&quot;, &quot;dining_room&quot;, &quot;ballroom&quot;, &quot;library&quot;) df.clue = expand_grid(who = who, what = what, where = where) df.suspects = df.clue %&gt;% distinct(who) %&gt;% mutate(gender = ifelse(test = who %in% c(&quot;ms_scarlet&quot;, &quot;mrs_white&quot;, &quot;mrs_peacock&quot;), yes = &quot;female&quot;, no = &quot;male&quot;)) df.suspects %&gt;% arrange(desc(gender)) %&gt;% kable() %&gt;% kable_styling(&quot;striped&quot;, full_width = F) who gender col_mustard male mr_green male prof_plum male ms_scarlet female mrs_white female mrs_peacock female 6.5.1 Conditional probability # conditional probability (via rules of probability) df.suspects %&gt;% summarize(p_prof_plum_given_male = sum(gender == &quot;male&quot; &amp; who == &quot;prof_plum&quot;) / sum(gender == &quot;male&quot;)) # A tibble: 1 × 1 p_prof_plum_given_male &lt;dbl&gt; 1 0.333 # conditional probability (via rejection) df.suspects %&gt;% filter(gender == &quot;male&quot;) %&gt;% summarize(p_prof_plum_given_male = sum(who == &quot;prof_plum&quot;) / n()) # A tibble: 1 × 1 p_prof_plum_given_male &lt;dbl&gt; 1 0.333 6.5.2 Law of total probability 6.6 Probability operations # Make a deck of cards df.cards = tibble(suit = rep(c(&quot;Clubs&quot;, &quot;Spades&quot;, &quot;Hearts&quot;, &quot;Diamonds&quot;), each = 8), value = rep(c(&quot;7&quot;, &quot;8&quot;, &quot;9&quot;, &quot;10&quot;, &quot;Jack&quot;, &quot;Queen&quot;, &quot;King&quot;, &quot;Ace&quot;), 4)) # conditional probability: p(Hearts | Queen) (via rules of probability) df.cards %&gt;% summarize(p_hearts_given_queen = sum(suit == &quot;Hearts&quot; &amp; value == &quot;Queen&quot;) / sum(value == &quot;Queen&quot;)) # A tibble: 1 × 1 p_hearts_given_queen &lt;dbl&gt; 1 0.25 # conditional probability: p(Hearts | Queen) (via rejection) df.cards %&gt;% filter(value == &quot;Queen&quot;) %&gt;% summarize(p_hearts_given_queen = sum(suit == &quot;Hearts&quot;)/n()) # A tibble: 1 × 1 p_hearts_given_queen &lt;dbl&gt; 1 0.25 6.7 Bayesian reasoning explained 6.8 Getting Bayes right matters 6.8.1 Bayesian reasoning example # prior probability of the disease p.D = 0.0001 # sensitivity of the test p.T_given_D = 0.999 # specificity of the test p.notT_given_notD = 0.999 p.T_given_notD = (1 - p.notT_given_notD) # posterior given a positive test result p.D_given_T = (p.T_given_D * p.D) / ((p.T_given_D * p.D) + (p.T_given_notD * (1-p.D))) p.D_given_T [1] 0.0908347 6.8.2 Bayesian reasoning example (COVID rapid test) https://pubmed.ncbi.nlm.nih.gov/34242764/#:~:text=The%20overall%20sensitivity%20of%20the,%25%20CI%2024.4%2D65.1). # prior probability of the disease p.D = 0.1 # sensitivity covid rapid test p.T_given_D = 0.653 # specificity of covid rapid test p.notT_given_notD = 0.999 p.T_given_notD = (1 - p.notT_given_notD) # posterior given a positive test result p.D_given_T = (p.T_given_D * p.D) / ((p.T_given_D * p.D) + (p.T_given_notD * (1-p.D))) # posterior given a negative test result p.D_given_notT = ((1-p.T_given_D) * p.D) / (((1-p.T_given_D) * p.D) + ((1-p.T_given_notD) * (1-p.D))) str_c(&quot;Probability of COVID given a positive test: &quot;, round(p.D_given_T * 100, 1), &quot;%&quot;) [1] &quot;Probability of COVID given a positive test: 98.6%&quot; str_c(&quot;Probability of COVID given a negative test: &quot;, round(p.D_given_notT * 100, 1), &quot;%&quot;) [1] &quot;Probability of COVID given a negative test: 3.7%&quot; 6.8.3 Most people in the hospital are vaccinated # probability of being vaccinated p.V = 0.8 # likelihood of hospital p.H_given_V = 0.2 p.H_given_notV = 0.5 # posterior probability p.V_given_H = (p.H_given_V * p.V) / ((p.H_given_V * p.V) + (p.H_given_notV * (1-p.V))) p.V_given_H [1] 0.6153846 6.9 Building a Bayesis 6.9.1 Dice example # prior p.four = 0.5 p.six = 0.5 # possibilities df.possibilities = tibble(observation = 1:6, p.four = c(rep(1/4, 4), rep(0, 2)), p.six = c(rep(1/6, 6))) # data # data = c(4) # data = c(4, 2, 1) data = c(4, 2, 1, 3, 1) # data = c(4, 2, 1, 3, 1, 5) # likelihood p.data_given_four = prod(df.possibilities$p.four[data]) p.data_given_six = prod(df.possibilities$p.six[data]) # posterior p.four_given_data = (p.data_given_four * p.four) / ((p.data_given_four * p.four) + (p.data_given_six * p.six)) p.four_given_data [1] 0.8836364 Given this data \\(d\\) = [4, 2, 1, 3, 1], there is a 88% chance that the four sided die was rolled rather than the six sided die. 6.10 Additional resources 6.10.1 Cheatsheets Probability cheatsheet 6.10.2 Books and chapters Probability and Statistics with examples using R Learning statistics with R: Chapter 9 Introduction to probability 6.10.3 Misc Bayes’ theorem in three panels Statistics 110: Probability; course at Harvard Bayes theorem and making probability intuitive 6.11 Session info Information about this R session including which version of R was used, and what packages were loaded. R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 DiagrammeR_1.0.9 arrangements_1.1.9 kableExtra_1.3.4 [13] knitr_1.39 loaded via a namespace (and not attached): [1] svglite_2.1.0 lubridate_1.8.0 visNetwork_2.1.0 assertthat_0.2.1 [5] digest_0.6.29 utf8_1.2.2 gmp_0.6-5 cellranger_1.1.0 [9] R6_2.5.1 backports_1.4.1 reprex_2.0.1 evaluate_0.15 [13] highr_0.9 httr_1.4.3 pillar_1.7.0 rlang_1.0.2 [17] readxl_1.4.0 rstudioapi_0.13 jquerylib_0.1.4 rmarkdown_2.14 [21] labeling_0.4.2 webshot_0.5.3 htmlwidgets_1.5.4 munsell_0.5.0 [25] broom_0.8.0 compiler_4.1.2 modelr_0.1.8 xfun_0.30 [29] pkgconfig_2.0.3 systemfonts_1.0.4 htmltools_0.5.2 tidyselect_1.1.2 [33] bookdown_0.26 fansi_1.0.3 viridisLite_0.4.0 withr_2.5.0 [37] tzdb_0.3.0 crayon_1.5.1 dbplyr_2.1.1 grid_4.1.2 [41] jsonlite_1.8.0 gtable_0.3.0 lifecycle_1.0.1 DBI_1.1.2 [45] magrittr_2.0.3 scales_1.2.0 cli_3.3.0 stringi_1.7.6 [49] farver_2.1.0 fs_1.5.2 xml2_1.3.3 bslib_0.3.1 [53] ellipsis_0.3.2 generics_0.1.2 vctrs_0.4.1 RColorBrewer_1.1-3 [57] tools_4.1.2 glue_1.6.2 hms_1.1.1 fastmap_1.1.0 [61] yaml_2.3.5 colorspace_2.0-3 rvest_1.0.2 haven_2.5.0 [65] sass_0.4.1 "],["simulation-1.html", "Chapter 7 Simulation 1 7.1 Load packages and set plotting theme 7.2 Sampling 7.3 Working with distributions 7.4 Bayesian inference with the normal distribution 7.5 Additional resources 7.6 Session info", " Chapter 7 Simulation 1 7.1 Load packages and set plotting theme library(&quot;knitr&quot;) library(&quot;kableExtra&quot;) library(&quot;MASS&quot;) library(&quot;patchwork&quot;) library(&quot;tidyverse&quot;) theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 7.2 Sampling 7.2.1 Drawing numbers from a vector numbers = 1:3 numbers %&gt;% sample(size = 10, replace = T) [1] 3 1 3 1 2 3 1 3 3 3 Use the prob = argument to change the probability with which each number should be drawn. numbers = 1:3 numbers %&gt;% sample(size = 10, replace = T, prob = c(0.8, 0.1, 0.1)) [1] 1 1 1 1 2 1 1 1 1 1 Make sure to set the seed in order to make your code reproducible. The code chunk below may give a different outcome each time is run. numbers = 1:5 numbers %&gt;% sample(5) [1] 3 1 4 5 2 The chunk below will produce the same outcome every time it’s run. set.seed(1) numbers = 1:5 numbers %&gt;% sample(5) [1] 1 4 3 5 2 7.2.2 Drawing rows from a data frame Generate a data frame. set.seed(1) n = 10 df.data = tibble(trial = 1:n, stimulus = sample(c(&quot;flower&quot;, &quot;pet&quot;), size = n, replace = T), rating = sample(1:10, size = n, replace = T)) Sample a given number of rows. set.seed(1) df.data %&gt;% slice_sample(n = 6, replace = T) # A tibble: 6 × 3 trial stimulus rating &lt;int&gt; &lt;chr&gt; &lt;int&gt; 1 9 pet 9 2 4 flower 5 3 7 flower 10 4 1 flower 3 5 2 pet 1 6 7 flower 10 set.seed(1) df.data %&gt;% slice_sample(prop = 0.5) # A tibble: 5 × 3 trial stimulus rating &lt;int&gt; &lt;chr&gt; &lt;int&gt; 1 9 pet 9 2 4 flower 5 3 7 flower 10 4 1 flower 3 5 2 pet 1 Note that there is a whole family of slice() functions in dplyr. Take a look at the help file here: help(slice) 7.3 Working with distributions Every distribution that R handles has four functions. There is a root name, for example, the root name for the normal distribution is norm. This root is prefixed by one of the letters here: letter description example d for “density”, the density function (probability function (for discrete variables) or probability density function (for continuous variables)) dnorm() p for “probability”, the cumulative distribution function pnorm() q for “quantile”, the inverse cumulative distribution function qnorm() r for “random”, a random variable having the specified distribution rnorm() For the normal distribution, these functions are dnorm, pnorm, qnorm, and rnorm. For the binomial distribution, these functions are dbinom, pbinom, qbinom, and rbinom. And so forth. You can get more info about the distributions that come with R via running help(Distributions) in your console. If you need a distribution that doesn’t already come with R, then take a look here for many more distributions that can be loaded with different R packages. 7.3.1 Plotting distributions Here’s an easy way to plot distributions in ggplot2 using the stat_function() function. We take a look at a normal distribution of height (in cm) with mean = 180 and sd = 10 (as this is the example we run with in class). ggplot(data = tibble(height = c(150, 210)), mapping = aes(x = height)) + stat_function(fun = ~ dnorm(., mean = 180, sd = 10)) Note that the data frame I created with tibble() only needs to have the minimum and the maximum value of the x-range that we are interested in. Here, I chose 150 and 210 as the minimum and maximum, respectively (which is the mean +/- 3 standard deviations). The stat_function() is very flexible. We can define our own functions and plot these like here: # define the breakpoint function fun.breakpoint = function(x, breakpoint){ x[x &lt; breakpoint] = breakpoint return(x) } # plot the function ggplot(data = tibble(x = c(-5, 5)), mapping = aes(x = x)) + stat_function(fun = ~ fun.breakpoint(., breakpoint = 2)) Here, I defined a breakpoint function. If the value of x is below the breakpoint, y equals the value of the breakpoint. If the value of x is greater than the breakpoint, then y equals x. 7.3.2 Sampling from distributions For each distribution, R provides a way of sampling random number from this distribution. For the normal distribution, we can use the rnorm() function to take random samples. So let’s take some random samples and plot a histogram. # make this example reproducible set.seed(1) # define how many samples to draw tmp.nsamples = 100 # make a data frame with the samples df.plot = tibble(height = rnorm(n = tmp.nsamples, mean = 180, sd = 10)) # plot the samples using a histogram ggplot(data = df.plot, mapping = aes(x = height)) + geom_histogram(binwidth = 1, color = &quot;black&quot;, fill = &quot;lightblue&quot;) + scale_x_continuous(breaks = c(160, 180, 200)) + coord_cartesian(xlim = c(150, 210), expand = F) # remove all variables with tmp in their name rm(list = ls() %&gt;% str_subset(pattern = &quot;tmp.&quot;)) Let’s see how many samples it takes to closely approximate the shape of the normal distribution with our histogram of samples. # make this example reproducible set.seed(1) # play around with this value # tmp.nsamples = 100 tmp.nsamples = 10000 tmp.binwidth = 1 # make a data frame with the samples df.plot = tibble(height = rnorm(n = tmp.nsamples, mean = 180, sd = 10)) # adjust the density of the normal distribution based on the samples and binwidth fun.dnorm = function(x, mean, sd, n, binwidth){ dnorm(x = x, mean = mean, sd = sd) * n * binwidth } # plot the samples using a histogram ggplot(data = df.plot, mapping = aes(x = height)) + geom_histogram(binwidth = tmp.binwidth, color = &quot;black&quot;, fill = &quot;lightblue&quot;) + stat_function(fun = ~ fun.dnorm(x = ., mean = 180, sd = 10, n = tmp.nsamples, binwidth = tmp.binwidth), xlim = c(min(df.plot$height), max(df.plot$height)), size = 2) + annotate(geom = &quot;text&quot;, label = str_c(&quot;n = &quot;, tmp.nsamples), x = -Inf, y = Inf, hjust = -0.1, vjust = 1.1, size = 10, family = &quot;Courier New&quot;) + scale_x_continuous(breaks = c(160, 180, 200)) + coord_cartesian(xlim = c(150, 210), expand = F) # remove all variables with tmp in their name rm(list = ls() %&gt;% str_subset(pattern = &quot;tmp.&quot;)) With 10,000 samples, our histogram of samples already closely resembles the theoretical shape of the normal distribution. To keep my environment clean, I’ve named the parameters tmp.nsamples and tmp.binwidth and then, at the end of the code chunk, I removed all variables from the environment that have “tmp.” in their name using the ls() function (which prints out all variables in the environment as a vector), and the str_subset() function which filters out only those variables that contain the specified pattern. 7.3.3 Understanding density() First, let’s calculate the density for a set of observations and store them in a data frame. # calculate density observations = c(1, 1.2, 1.5, 2, 3) bandwidth = 0.25 # bandwidth (= sd) of the Gaussian distribution tmp.density = density(observations, kernel = &quot;gaussian&quot;, bw = bandwidth, n = 512) # save density as data frame df.density = tibble(x = tmp.density$x, y = tmp.density$y) df.density %&gt;% head() %&gt;% kable(digits = 3) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) x y 0.250 0.004 0.257 0.004 0.264 0.005 0.271 0.005 0.277 0.005 0.284 0.006 Now, let’s plot the density. ggplot(data = df.density, mapping = aes(x = x, y = y)) + geom_line(size = 2) + geom_point(data = enframe(observations), mapping = aes(x = value, y = 0), size = 3) This density shows the sum of the densities of normal distributions that are centered at the observations with the specified bandwidth. # add densities for the individual normal distributions for (i in 1:length(observations)){ df.density[[str_c(&quot;observation_&quot;,i)]] = dnorm(df.density$x, mean = observations[i], sd = bandwidth) } # sum densities df.density = df.density %&gt;% mutate(sum_norm = rowSums(select(., contains(&quot;observation_&quot;))), y = y * length(observations)) df.density %&gt;% head() %&gt;% kable(digits = 3) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) x y observation_1 observation_2 observation_3 observation_4 observation_5 sum_norm 0.250 0.019 0.018 0.001 0 0 0 0.019 0.257 0.021 0.019 0.001 0 0 0 0.021 0.264 0.023 0.021 0.001 0 0 0 0.022 0.271 0.024 0.023 0.002 0 0 0 0.024 0.277 0.027 0.024 0.002 0 0 0 0.026 0.284 0.029 0.026 0.002 0 0 0 0.028 Now, let’s plot the individual densities as well as the overall density. # colors of individual Gaussian distributions colors = c(&quot;blue&quot;, &quot;green&quot;, &quot;red&quot;, &quot;purple&quot;, &quot;orange&quot;) # original density p = ggplot(data = df.density, aes(x = x, y = y)) + geom_line(size = 2) # individual densities for (i in 1:length(observations)){ p = p + stat_function(fun = &quot;dnorm&quot;, args = list(mean = observations[i], sd = bandwidth), color = colors[i]) } # individual observations p = p + geom_point(data = enframe(observations), mapping = aes(x = value, y = 0, color = factor(1:5)), size = 3, show.legend = F) + scale_color_manual(values = colors) # sum of the individual densities p = p + geom_line(data = df.density, aes(x = x, y = sum_norm), size = 1, color = &quot;red&quot;, linetype = 2) p # print the figure Here are the same results when specifying a different bandwidth: # colors of individual Gaussian distributions colors = c(&quot;blue&quot;, &quot;green&quot;, &quot;red&quot;, &quot;purple&quot;, &quot;orange&quot;) # calculate density observations = c(1, 1.2, 1.5, 2, 3) bandwidth = 0.5 # bandwidth (= sd) of the Gaussian distribution tmp.density = density(observations, kernel = &quot;gaussian&quot;, bw = bandwidth, n = 512) # save density as data frame df.density = tibble( x = tmp.density$x, y = tmp.density$y ) # add densities for the individual normal distributions for (i in 1:length(observations)){ df.density[[str_c(&quot;observation_&quot;,i)]] = dnorm(df.density$x, mean = observations[i], sd = bandwidth) } # sum densities df.density = df.density %&gt;% mutate(sum_norm = rowSums(select(., contains(&quot;observation_&quot;))), y = y * length(observations)) # original plot p = ggplot(data = df.density, aes(x = x, y = y)) + geom_line(size = 2) + geom_point(data = enframe(observations), mapping = aes(x = value, y = 0, color = factor(1:5)), size = 3, show.legend = F) + scale_color_manual(values = colors) # add individual Gaussians for (i in 1:length(observations)){ p = p + stat_function(fun = &quot;dnorm&quot;, args = list(mean = observations[i], sd = bandwidth), color = colors[i]) } # add the sum of Gaussians p = p + geom_line(data = df.density, aes(x = x, y = sum_norm), size = 1, color = &quot;red&quot;, linetype = 2) p 7.3.4 Cumulative probability distribution ggplot(data = tibble(height = c(150, 210)), mapping = aes(x = height)) + stat_function(fun = ~ pnorm(q = ., mean = 180, sd = 10)) + labs(x = &quot;height&quot;, y = &quot;cumulative probability&quot;) + scale_x_continuous(breaks = c(160, 180, 200)) + coord_cartesian(xlim = c(150, 210), ylim = c(0, 1.05), expand = F) Let’s find the cumulative probability of a particular value. tmp.x = 190 tmp.y = pnorm(tmp.x, mean = 180, sd = 10) print(tmp.y %&gt;% round(3)) [1] 0.841 # draw the cumulative probability distribution and show the value ggplot(data = tibble(height = c(150, 210)), mapping = aes(x = height)) + stat_function(fun = ~ pnorm(q = ., mean = 180, sd = 10 )) + annotate(geom = &quot;point&quot;, x = tmp.x, y = tmp.y, size = 4, color = &quot;blue&quot;) + geom_segment(mapping = aes(x = tmp.x, xend = tmp.x, y = 0, yend = tmp.y), size = 1, color = &quot;blue&quot;) + geom_segment(mapping = aes(x = -5, xend = tmp.x, y = tmp.y, yend = tmp.y), size = 1, color = &quot;blue&quot;) + labs(x = &quot;height&quot;, y = &quot;cumulative probability&quot;) + scale_x_continuous(breaks = c(160, 180, 200)) + coord_cartesian(xlim = c(150, 210), ylim = c(0, 1.05), expand = F) # remove all variables with tmp in their name rm(list = str_subset(string = ls(), pattern = &quot;tmp.&quot;)) Let’s illustrate what this would look like using a normal density plot. ggplot(data = tibble(height = c(150, 210)), mapping = aes(x = height)) + stat_function(fun = ~ dnorm(., mean = 180, sd = 10), geom = &quot;area&quot;, fill = &quot;lightblue&quot;, xlim = c(150, 190)) + stat_function(fun = ~ dnorm(., mean = 180, sd = 10), size = 1.5) + labs(x = &quot;height&quot;, y = &quot;density&quot;) + scale_y_continuous(expand = expansion(mult = c(0, 0.1))) + scale_x_continuous(breaks = c(160, 180, 200)) + coord_cartesian(xlim = c(150, 210)) 7.3.5 Inverse cumulative distribution ggplot(data = tibble(probability = c(0, 1)), mapping = aes(x = probability)) + stat_function(fun = ~ qnorm(p = ., mean = 180, sd = 10)) + labs(y = &quot;height&quot;, x = &quot;cumulative probability&quot;) + scale_x_continuous(breaks = seq(from = 0, to = 1, by = 0.1)) + scale_y_continuous(limits = c(160, 200)) + coord_cartesian(xlim = c(0, 1.05), expand = F) And let’s compute the inverse cumulative probability for a particular value. tmp.x = 0.3 tmp.y = qnorm(tmp.x, mean = 180, sd = 10) print(tmp.y %&gt;% round(3)) [1] 174.756 # draw the cumulative probability distribution and show the value ggplot(data = tibble(probability = c(0, 1)), mapping = aes(x = probability)) + stat_function(fun = ~ qnorm(., mean = 180, sd = 10)) + annotate(geom = &quot;point&quot;, x = tmp.x, y = tmp.y, size = 4, color = &quot;blue&quot;) + geom_segment(mapping = aes(x = tmp.x, xend = tmp.x, y = 160, yend = tmp.y), size = 1, color = &quot;blue&quot;) + geom_segment(mapping = aes(x = 0, xend = tmp.x, y = tmp.y, yend = tmp.y), size = 1, color = &quot;blue&quot;) + labs(x = &quot;cumulative probability&quot;, y = &quot;height&quot;) + scale_x_continuous(breaks = seq(from = 0, to = 1, by = 0.1)) + scale_y_continuous(limits = c(160, 200)) + coord_cartesian(xlim = c(0, 1.05), expand = F) # remove all variables with tmp in their name rm(list = str_subset(string = ls(), pattern = &quot;tmp.&quot;)) 7.3.6 Computing probabilities 7.3.6.1 Via probability distributions Let’s compute the probability of observing a particular value \\(x\\) in a given range. tmp.lower = 170 tmp.upper = 180 tmp.prob = pnorm(tmp.upper, mean = 180, sd = 10) - pnorm(tmp.lower, mean = 180, sd = 10) tmp.prob [1] 0.3413447 ggplot(data = tibble(x = c(150, 210)), mapping = aes(x = x)) + stat_function(fun = ~ dnorm(., mean = 180, sd = 10), geom = &quot;area&quot;, fill = &quot;lightblue&quot;, xlim = c(tmp.lower, tmp.upper), color = &quot;black&quot;, linetype = 2) + stat_function(fun = ~ dnorm(., mean = 180, sd = 10), size = 1.5) + labs(x = &quot;height&quot;, y = &quot;density&quot;) + scale_y_continuous(expand = expansion(mult = c(0, 0.1))) + scale_x_continuous(breaks = c(160, 180, 200)) + coord_cartesian(xlim = c(150, 210)) # remove all variables with tmp in their name rm(list = str_subset(string = ls(), pattern = &quot;tmp.&quot;)) We find that ~34% of the heights are between 170 and 180 cm. 7.3.6.2 Via sampling We can also compute the probability of observing certain events using sampling. We first generate samples from the desired probability distribution, and then use these samples to compute our statistic of interest. # let&#39;s compute the probability of observing a value within a certain range tmp.lower = 170 tmp.upper = 180 # make example reproducible set.seed(1) # generate some samples and store them in a data frame tmp.nsamples = 10000 df.samples = tibble(height = rnorm(n = tmp.nsamples, mean = 180, sd = 10)) # compute the probability that sample lies within the range of interest tmp.prob = df.samples %&gt;% filter(height &gt;= tmp.lower, height &lt;= tmp.upper) %&gt;% summarize(prob = n()/tmp.nsamples) # illustrate the result using a histogram ggplot(data = df.samples, mapping = aes(x = height)) + geom_histogram(binwidth = 1, color = &quot;black&quot;, fill = &quot;lightblue&quot;) + geom_vline(xintercept = tmp.lower, size = 1, color = &quot;red&quot;, linetype = 2) + geom_vline(xintercept = tmp.upper, size = 1, color = &quot;red&quot;, linetype = 2) + annotate(geom = &quot;label&quot;, label = str_c(tmp.prob %&gt;% round(3) * 100, &quot;%&quot;), x = 175, y = 200, hjust = 0.5, size = 10) + scale_y_continuous(expand = expansion(mult = c(0, 0.1))) + labs(x = &quot;height&quot;) # remove all variables with tmp in their name rm(list = str_subset(string = ls(), pattern = &quot;tmp.&quot;)) ## Breakout room exercise 7.3.7 Make the plot ggplot(data = tibble(height = c(30, 70)), mapping = aes(x = height)) + stat_function(fun = ~ dgamma(., shape = 50, rate = 1)) 7.3.8 Analytic solutions 7.3.8.1 Question: A 60cm tall Penguin claims that no more than 10% are taller than her. Is she correct? 1 - pgamma(60, shape = 50, rate = 1) [1] 0.08440668 Answer: Yes, she is correct. Only ~ 8.4% of Penguins are taller than her. 7.3.8.2 Question: Are there more penguins between 50 and 55cm or between 55 and 65cm? first_range = pgamma(55, shape = 50, rate = 1) - pgamma(50, shape = 50, rate = 1) second_range = pgamma(65, shape = 50, rate = 1) - pgamma(55, shape = 50, rate = 1) first_range - second_range [1] 0.04029452 Answer: There are 4% more Penguins between 50 and 55cm than between 55 and 65 cm. 7.3.8.3 Question: What size is a Penguin who is taller than 75% of the rest? qgamma(0.75, shape = 50, rate = 1) [1] 54.57062 Answer: A Penguin who is ~54.6cm tall is taller than 75% of the rest. 7.3.9 Sampling solution Let’s just simulate a bunch of Penguins, yay! set.seed(1) df.penguins = tibble(height = rgamma(n = 100000, shape = 50, rate = 1)) 7.3.9.1 Question: A 60cm tall Penguin claims that no more than 10% are taller than her. Is she correct? df.penguins %&gt;% summarize(probability = sum(height &gt; 60) / n()) # A tibble: 1 × 1 probability &lt;dbl&gt; 1 0.0835 Answer: Yes, she is correct. Only ~ 8.3% of Penguins are taller than her. 7.3.9.2 Question: Are there more penguins between 50 and 55cm or between 55 and 65cm? df.penguins %&gt;% summarize(probability = (sum(between(height, 50, 55)) - sum(between(height, 55, 65)))/n()) # A tibble: 1 × 1 probability &lt;dbl&gt; 1 0.0387 Answer: There are 3.9% more Penguins between 50 and 55cm than between 55 and 65 cm. 7.3.9.3 Question: What size is a Penguin who is taller than 75% of the rest? df.penguins %&gt;% arrange(height) %&gt;% slice_head(prop = 0.75) %&gt;% summarize(height = max(height)) # A tibble: 1 × 1 height &lt;dbl&gt; 1 54.6 Answer: A Penguin who is ~54.6cm tall is taller than 75% of the rest. 7.4 Bayesian inference with the normal distribution Let’s consider the following scenario. You are helping out at a summer camp. This summer, two different groups of kids go to the same summer camp. The chess kids, and the basketball kids. The chess summer camp is not quite as popular as the basketball summer camp (shocking, I know!). In fact, twice as many children have signed up for the basketball camp. When signing up for the camp, the children were asked for some demographic information including their height in cm. Unsurprisingly, the basketball players tend to be taller on average than the chess players. In fact, the basketball players’ height is approximately normally distributed with a mean of 180cm and a standard deviation of 10cm. For the chess players, the mean height is 170cm with a standard deviation of 8cm. At the camp site, a child walks over to you and asks you where their gym is. You gage that the child is around 175cm tall. Where should you direct the child to? To the basketball gym, or to the chess gym? 7.4.1 Analytic solution height = 175 # priors prior_basketball = 2/3 prior_chess = 1/3 # likelihood mean_basketball = 180 sd_basketball = 10 mean_chess = 170 sd_chess = 8 likelihood_basketball = dnorm(height, mean = mean_basketball, sd = sd_basketball) likelihood_chess = dnorm(height, mean = mean_chess, sd = sd_chess) # posterior posterior_basketball = (likelihood_basketball * prior_basketball) / ((likelihood_basketball * prior_basketball) + (likelihood_chess * prior_chess)) posterior_basketball %&gt;% print() [1] 0.631886 7.4.2 Solution via sampling Let’s do the same thing via sampling. # number of kids tmp.nkids = 10000 # make reproducible set.seed(1) # priors prior_basketball = 2/3 prior_chess = 1/3 # likelihood functions mean_basketball = 180 sd_basketball = 10 mean_chess = 170 sd_chess = 8 # data frame with the kids df.camp = tibble(kid = 1:tmp.nkids, sport = sample(c(&quot;chess&quot;, &quot;basketball&quot;), size = tmp.nkids, replace = T, prob = c(prior_chess, prior_basketball))) %&gt;% rowwise() %&gt;% mutate(height = ifelse(test = sport == &quot;chess&quot;, yes = rnorm(., mean = mean_chess, sd = sd_chess), no = rnorm(., mean = mean_basketball, sd = sd_basketball))) %&gt;% ungroup df.camp %&gt;% print() # A tibble: 10,000 × 3 kid sport height &lt;int&gt; &lt;chr&gt; &lt;dbl&gt; 1 1 basketball 165. 2 2 basketball 163. 3 3 basketball 191. 4 4 chess 160. 5 5 basketball 183. 6 6 chess 164. 7 7 chess 169. 8 8 basketball 193. 9 9 basketball 172. 10 10 basketball 177. # … with 9,990 more rows Now we have a data frame with kids whose height was randomly sampled depending on which sport they do. I’ve used the sample() function to assign a sport to each kid first using the prob = argument to make sure that a kid is more likely to be assigned the sport “basketball” than “chess”. Note that the solution above is not particularly efficient since it uses the rowwise() function to make sure that a different random value for height is drawn for each row. Running this code will get slow for large samples. A more efficient solution would be the following: # number of kids tmp.nkids = 100000 # make reproducible set.seed(3) df.camp2 = tibble( kid = 1:tmp.nkids, sport = sample(c(&quot;chess&quot;, &quot;basketball&quot;), size = tmp.nkids, replace = T, prob = c(prior_chess, prior_basketball))) %&gt;% arrange(sport) %&gt;% mutate(height = c(rnorm(sum(sport == &quot;basketball&quot;), mean = mean_basketball, sd = sd_basketball), rnorm(sum(sport == &quot;chess&quot;), mean = mean_chess, sd = sd_chess))) In this solution, I take advantage of the fact that rnorm() is vectorized. That is, it can produce many random draws in one call. To make this work, I first arrange the data frame, and then draw the correct number of samples from each of the two distributions. This works fast, even if I’m drawing a large number of samples. How can we now use these samples to answer our question of interest? Let’s see what doesn’t work first: tmp.height = 175 df.camp %&gt;% filter(height == tmp.height) %&gt;% count(sport) %&gt;% pivot_wider(names_from = sport, values_from = n) %&gt;% summarize(prob_basketball = basketball/(basketball + chess)) The reason this doesn’t work is because none of our kids is exactly 175cm tall. Instead, we need to filter kids that are within a certain height range. tmp.height = 175 tmp.margin = 1 df.camp %&gt;% filter(between(height, left = tmp.height - tmp.margin, right = tmp.height + tmp.margin)) %&gt;% count(sport) %&gt;% pivot_wider(names_from = sport, values_from = n) %&gt;% summarize(prob_basketball = basketball/(basketball + chess)) # A tibble: 1 × 1 prob_basketball &lt;dbl&gt; 1 0.632 Here, I’ve used the between() function which is a shortcut for otherwise writing x &gt;= left &amp; x &lt;= right. You can play around with the margin to see how the result changes. 7.5 Additional resources 7.5.1 Datacamp Foundations of probability in R 7.6 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 patchwork_1.1.1 MASS_7.3-57 kableExtra_1.3.4 [13] knitr_1.39 loaded via a namespace (and not attached): [1] svglite_2.1.0 lubridate_1.8.0 assertthat_0.2.1 digest_0.6.29 [5] utf8_1.2.2 R6_2.5.1 cellranger_1.1.0 backports_1.4.1 [9] reprex_2.0.1 evaluate_0.15 httr_1.4.3 highr_0.9 [13] pillar_1.7.0 rlang_1.0.2 readxl_1.4.0 rstudioapi_0.13 [17] jquerylib_0.1.4 rmarkdown_2.14 labeling_0.4.2 webshot_0.5.3 [21] munsell_0.5.0 broom_0.8.0 compiler_4.1.2 modelr_0.1.8 [25] xfun_0.30 pkgconfig_2.0.3 systemfonts_1.0.4 htmltools_0.5.2 [29] tidyselect_1.1.2 bookdown_0.26 fansi_1.0.3 viridisLite_0.4.0 [33] crayon_1.5.1 tzdb_0.3.0 dbplyr_2.1.1 withr_2.5.0 [37] grid_4.1.2 jsonlite_1.8.0 gtable_0.3.0 lifecycle_1.0.1 [41] DBI_1.1.2 magrittr_2.0.3 scales_1.2.0 cli_3.3.0 [45] stringi_1.7.6 farver_2.1.0 fs_1.5.2 xml2_1.3.3 [49] bslib_0.3.1 ellipsis_0.3.2 generics_0.1.2 vctrs_0.4.1 [53] tools_4.1.2 glue_1.6.2 hms_1.1.1 fastmap_1.1.0 [57] yaml_2.3.5 colorspace_2.0-3 rvest_1.0.2 haven_2.5.0 [61] sass_0.4.1 "],["simulation-2.html", "Chapter 8 Simulation 2 8.1 Load packages and set plotting theme 8.2 Making statistical inferences (frequentist style) 8.3 Understanding p-values 8.4 Confidence intervals 8.5 Additional resources 8.6 Session info", " Chapter 8 Simulation 2 In which we figure out some key statistical concepts through simulation and plotting. On the menu we have: Sampling distributions p-value Confidence interval Bootstrapping 8.1 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 8.2 Making statistical inferences (frequentist style) 8.2.1 Population distribution Let’s first put the information we need for our population distribution in a data frame. # the distribution from which we want to sample (aka the heavy metal distribution) df.population = tibble(numbers = 1:6, probability = c(1/3, 0, 1/6, 1/6, 0, 1/3)) And then let’s plot it: # plot the distribution ggplot(data = df.population, mapping = aes(x = numbers, y = probability)) + geom_bar(stat = &quot;identity&quot;, fill = &quot;lightblue&quot;, color = &quot;black&quot;) + scale_x_continuous(breaks = df.population$numbers, labels = df.population$numbers, limits = c(0.1, 6.9)) + coord_cartesian(expand = F) Here are the true mean and standard deviation of our population distribution: # mean and standard deviation (see: https://nzmaths.co.nz/category/glossary/standard-deviation-discrete-random-variable) df.population %&gt;% summarize(population_mean = sum(numbers * probability), population_sd = sqrt(sum(numbers^2 * probability) - population_mean^2)) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) population_mean population_sd 3.5 2.06 8.2.2 Distribution of a single sample Let’s draw a single sample of size \\(n = 40\\) from the population distribution and plot it: # make example reproducible set.seed(1) # set the sample size sample_size = 40 # create data frame df.sample = sample(df.population$numbers, size = sample_size, replace = T, prob = df.population$probability) %&gt;% enframe(name = &quot;draw&quot;, value = &quot;number&quot;) # draw a plot of the sample ggplot(data = df.sample, mapping = aes(x = number, y = stat(density))) + geom_histogram(binwidth = 0.5, fill = &quot;lightblue&quot;, color = &quot;black&quot;) + scale_x_continuous(breaks = min(df.sample$number):max(df.sample$number)) + scale_y_continuous(expand = expansion(mult = c(0, 0.01))) Here are the sample mean and standard deviation: # print out sample mean and standard deviation df.sample %&gt;% summarize(sample_mean = mean(number), sample_sd = sd(number)) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) sample_mean sample_sd 3.72 2.05 8.2.3 The sampling distribution And let’s now create the sampling distribution (making the unrealistic assumption that we know the population distribution). # make example reproducible set.seed(1) # parameters sample_size = 40 # size of each sample sample_n = 10000 # number of samples # define a function that draws samples from a discrete distribution fun.draw_sample = function(sample_size, distribution){ x = sample(distribution$numbers, size = sample_size, replace = T, prob = distribution$probability) return(x) } # generate many samples samples = replicate(n = sample_n, fun.draw_sample(sample_size, df.population)) # set up a data frame with samples df.sampling_distribution = matrix(samples, ncol = sample_n) %&gt;% as_tibble(.name_repair = ~ str_c(1:sample_n)) %&gt;% pivot_longer(cols = everything(), names_to = &quot;sample&quot;, values_to = &quot;number&quot;) %&gt;% mutate(sample = as.numeric(sample)) %&gt;% group_by(sample) %&gt;% mutate(draw = 1:n()) %&gt;% select(sample, draw, number) %&gt;% ungroup() # turn the data frame into long format and calculate the means of each sample df.sampling_distribution_means = df.sampling_distribution %&gt;% group_by(sample) %&gt;% summarize(mean = mean(number)) %&gt;% ungroup() And plot it: set.seed(1) # plot a histogram of the means with density overlaid df.plot = df.sampling_distribution_means ggplot(data = df.plot, mapping = aes(x = mean)) + geom_histogram(aes(y = stat(density)), binwidth = 0.05, fill = &quot;lightblue&quot;, color = &quot;black&quot;) + stat_density(bw = 0.1, size = 2, geom = &quot;line&quot;) + scale_y_continuous(expand = expansion(mult = c(0, 0.01))) Even though our population distribution was far from normal (and much more heavy-metal like), the means of the sampling distribution are normally distributed. And here are the mean and standard deviation of the sampling distribution: # print out sampling distribution mean and standard deviation df.sampling_distribution_means %&gt;% summarize(sampling_distribution_mean = mean(mean), sampling_distribution_sd = sd(mean)) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) sampling_distribution_mean sampling_distribution_sd 3.5 0.33 Here is a data frame that I’ve used for illustrating the idea behind how a sampling distribution is constructed from the population distribution. # data frame for illustration in class df.sampling_distribution %&gt;% filter(sample &lt;= 10, draw &lt;= 4) %&gt;% pivot_wider(names_from = draw, values_from = number) %&gt;% set_names(c(&quot;sample&quot;, str_c(&quot;draw_&quot;, 1:(ncol(.) - 1)))) %&gt;% mutate(sample_mean = rowMeans(.[, -1])) %&gt;% head(10) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) sample draw_1 draw_2 draw_3 draw_4 sample_mean 1 1 6 6 4 4.25 2 3 6 3 6 4.50 3 6 3 6 1 4.00 4 4 6 6 1 4.25 5 1 4 6 3 3.50 6 1 1 6 1 2.25 7 1 6 4 1 3.00 8 1 6 4 6 4.25 9 6 1 6 4 4.25 10 1 1 4 1 1.75 8.2.3.1 Bootstrapping a sampling distribution Of course, in actuality, we never have access to the population distribution. We try to infer characteristics of that distribution (e.g. its mean) from our sample. So using the population distribution to create a sampling distribution is sort of cheating – helpful cheating though since it gives us a sense for the relationship between population, sample, and sampling distribution. It urns out that we can approximate the sampling distribution only using our actual sample. The idea is to take the sample that we drew, and generate new samples from it by drawing with replacement. Essentially, we are treating our original sample like the population from which we are generating random samples to derive the sampling distribution. # make example reproducible set.seed(1) # how many bootstrapped samples shall we draw? n_samples = 1000 # generate a new sample from the original one by sampling with replacement func.bootstrap = function(df){ df %&gt;% sample_frac(size = 1, replace = T) %&gt;% summarize(mean = mean(number)) %&gt;% pull(mean) } # data frame with bootstrapped results df.bootstrap = tibble(bootstrap = 1:n_samples, average = replicate(n = n_samples, func.bootstrap(df.sample))) Let’s plot our sample first: # plot the distribution ggplot(data = df.sample, mapping = aes(x = number)) + geom_bar(stat = &quot;count&quot;, fill = &quot;lightblue&quot;, color = &quot;black&quot;) + scale_x_continuous(breaks = 1:6, labels = 1:6, limits = c(0.1, 6.9)) + coord_cartesian(expand = F) Let’s plot the bootstrapped sampling distribution: # plot the bootstrapped sampling distribution ggplot(data = df.bootstrap, mapping = aes(x = average)) + geom_histogram(aes(y = stat(density)), color = &quot;black&quot;, fill = &quot;lightblue&quot;, binwidth = 0.05) + # stat_density(geom = &quot;line&quot;, # size = 1.5, # bw = 0.1, # color = &quot;blue&quot;, # linetype = 2) + stat_function(fun = ~ dnorm(., mean = mean(df.sample$number), sd = sd(df.sample$number / sqrt(nrow(df.sample)))), size = 2) + labs(x = &quot;mean&quot;) + scale_y_continuous(expand = expansion(mult = c(0, 0.01))) And let’s calculate the mean and standard deviation: # print out sampling distribution mean and standard deviation df.bootstrap %&gt;% summarize(bootstrapped_distribution_mean = mean(average), bootstrapped_distribution_sd = sd(average)) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) bootstrapped_distribution_mean bootstrapped_distribution_sd 3.74 0.33 Neat, as we can see, the mean and standard deviation of the bootstrapped sampling distribution are very close to the sampling distribution that we generated from the population distribution. 8.3 Understanding p-values The p-value is the probability of finding the observed, or more extreme, results when the null hypothesis (\\(H_0\\)) is true. \\[ \\text{p-value = p(observed or more extreme test statistic} | H_{0}=\\text{true}) \\] What we are really interested in is the probability of a hypothesis given the data. However, frequentist statistics doesn’t give us this probability – we’ll get to Bayesian statistics later in the course. Instead, we define a null hypothesis, construct a sampling distribution that tells us what we would expect the test statistic of interest to look like if the null hypothesis were true. We reject the null hypothesis in case our observed result would be unlikely if the null hypothesis were true. An intutive way for illustrating (this rather unintuitive procedure) is the permutation test. 8.3.1 Permutation test Let’s start by generating some random data from two different normal distributions (simulating a possible experiment). # make example reproducible set.seed(1) # generate data from two conditions df.permutation = tibble(control = rnorm(25, mean = 5.5, sd = 2), experimental = rnorm(25, mean = 4.5, sd = 1.5)) %&gt;% pivot_longer(cols = everything(), names_to = &quot;condition&quot;, values_to = &quot;performance&quot;) Here is a summary of how each group performed: df.permutation %&gt;% group_by(condition) %&gt;% summarize(mean = mean(performance), sd = sd(performance)) %&gt;% pivot_longer(cols = - condition, names_to = &quot;statistic&quot;, values_to = &quot;value&quot;) %&gt;% pivot_wider(names_from = condition, values_from = value) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) statistic control experimental mean 5.84 4.55 sd 1.90 1.06 Let’s plot the results: ggplot(data = df.permutation, mapping = aes(x = condition, y = performance)) + geom_point(position = position_jitter(height = 0, width = 0.1), alpha = 0.5) + stat_summary(fun.data = mean_cl_boot, geom = &quot;linerange&quot;, size = 1) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;, shape = 21, color = &quot;black&quot;, fill = &quot;white&quot;, size = 4) + scale_y_continuous(breaks = 0:10, labels = 0:10, limits = c(0, 10)) We are interested in the difference in the mean performance between the two groups: # calculate the difference between conditions difference_actual = df.permutation %&gt;% group_by(condition) %&gt;% summarize(mean = mean(performance)) %&gt;% pull(mean) %&gt;% diff() The difference in the mean rating between the control and experimental condition is -1.2889834. Is this difference between conditions statistically significant? What we are asking is: what are the chances that a result like this (or more extreme) could have come about due to chance? Let’s answer the question using simulation. Here is the main idea: imagine that we were very sloppy in how we recorded the data, and now we don’t remember anymore which participants were in the controld condition and which ones were in experimental condition (we still remember though, that we tested 25 participants in each condition). set.seed(0) df.permutation = df.permutation %&gt;% mutate(permutation = sample(condition)) #randomly assign labels df.permutation %&gt;% group_by(permutation) %&gt;% summarize(mean = mean(performance), sd = sd(performance)) %&gt;% ungroup() %&gt;% summarize(diff = diff(mean)) # A tibble: 1 × 1 diff &lt;dbl&gt; 1 -0.0105 Here, the difference between the two conditions is 0.0105496. After randomly shuffling the condition labels, this is how the results would look like: ggplot(data = df.permutation, mapping = aes(x = permutation, y = performance))+ geom_point(mapping = aes(color = condition), position = position_jitter(height = 0, width = 0.1)) + stat_summary(fun.data = mean_cl_boot, geom = &quot;linerange&quot;, size = 1) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;, shape = 21, color = &quot;black&quot;, fill = &quot;white&quot;, size = 4) + scale_y_continuous(breaks = 0:10, labels = 0:10, limits = c(0, 10)) The idea is now that, similar to bootstrapping above, we can get a sampling distribution of the difference in the means between the two conditions (assuming that the null hypothesis were true), by randomly shuffling the labels and calculating the difference in means (and doing this many times). What we get is a distribution of the differences we would expect, if there was no effect of condition. set.seed(1) n_permutations = 500 # permutation function fun.permutations = function(df){ df %&gt;% mutate(condition = sample(condition)) %&gt;% #we randomly shuffle the condition labels group_by(condition) %&gt;% summarize(mean = mean(performance)) %&gt;% pull(mean) %&gt;% diff() } # data frame with permutation results df.permutations = tibble(permutation = 1:n_permutations, mean_difference = replicate(n = n_permutations, fun.permutations(df.permutation))) #plot the distribution of the differences ggplot(data = df.permutations, aes(x = mean_difference)) + geom_histogram(aes(y = stat(density)), color = &quot;black&quot;, fill = &quot;lightblue&quot;, binwidth = 0.05) + stat_density(geom = &quot;line&quot;, size = 1.5, bw = 0.2) + geom_vline(xintercept = difference_actual, color = &quot;red&quot;, size = 2) + labs(x = &quot;difference between means&quot;) + scale_x_continuous(breaks = seq(-1.5, 1.5, 0.5), labels = seq(-1.5, 1.5, 0.5), limits = c(-2, 2)) + coord_cartesian(expand = F, clip = &quot;off&quot;) Warning: Removed 2 rows containing missing values (geom_bar). And we can then simply calculate the p-value by using some basic data wrangling (i.e. finding the proportion of differences that were as or more extreme than the one we observed). #calculate p-value of our observed result df.permutations %&gt;% summarize(p_value = sum(mean_difference &lt;= difference_actual)/n()) # A tibble: 1 × 1 p_value &lt;dbl&gt; 1 0.002 8.3.2 t-test by hand Examining the t-distribution. set.seed(1) n_simulations = 1000 sample_size = 100 mean = 5 sd = 2 fun.normal_sample_mean = function(sample_size, mean, sd){ rnorm(n = sample_size, mean = mean, sd = sd) %&gt;% mean() } df.ttest = tibble(simulation = 1:n_simulations) %&gt;% mutate(sample1 = replicate(n = n_simulations, expr = fun.normal_sample_mean(sample_size, mean, sd)), sample2 = replicate(n = n_simulations, expr = fun.normal_sample_mean(sample_size, mean, sd))) %&gt;% mutate(difference = sample1 - sample2, # assuming the same standard deviation in each sample tstatistic = difference / sqrt(sd^2 * (1/sample_size + 1/sample_size))) df.ttest # A tibble: 1,000 × 5 simulation sample1 sample2 difference tstatistic &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 1 5.22 4.99 0.225 0.796 2 2 4.92 5.06 -0.132 -0.467 3 3 5.06 5.40 -0.340 -1.20 4 4 5.10 4.97 0.132 0.468 5 5 4.92 4.75 0.173 0.613 6 6 4.91 4.99 -0.0787 -0.278 7 7 4.60 5.11 -0.513 -1.82 8 8 5.00 5.01 -0.0113 -0.0401 9 9 5.02 5.10 -0.0773 -0.273 10 10 5.01 4.98 0.0267 0.0945 # … with 990 more rows Population distribution mean = 0 sd = 1 ggplot(data = tibble(x = c(mean - 3 * sd, mean + 3 * sd)), mapping = aes(x = x)) + stat_function(fun = ~ dnorm(.,mean = mean, sd = sd), color = &quot;black&quot;, size = 2) + geom_vline(xintercept = qnorm(c(0.025, 0.975), mean = mean, sd = sd), linetype = 2) # labs(x = &quot;performance&quot;) Distribution of differences in means ggplot(data = df.ttest, mapping = aes(x = difference)) + geom_density(size = 1) + geom_vline(xintercept = quantile(df.ttest$difference, probs = c(0.025, 0.975)), linetype = 2) t-distribution ggplot(data = df.ttest, mapping = aes(x = tstatistic)) + stat_function(fun = ~ dt(., df = sample_size * 2 - 2), color = &quot;red&quot;, size = 2) + geom_density(size = 1) + geom_vline(xintercept = qt(c(0.025, 0.975), df = sample_size * 2 - 2), linetype = 2) + scale_x_continuous(limits = c(-4, 4), breaks = seq(-4, 4, 1)) 8.4 Confidence intervals The definition of the confidence interval is the following: “If we were to repeat the experiment over and over, then 95% of the time the confidence intervals contain the true mean.” If we assume normally distributed data (and a large enough sample size), then we can calculate the confidence interval on the estimate of the mean in the following way: \\(\\overline X \\pm Z \\frac{s}{\\sqrt{n}}\\), where \\(Z\\) equals the value of the standard normal distribution for the desired level of confidence. For smaller sample sizes, we can use the \\(t\\)-distribution instead with \\(n-1\\) degrees of freedom. For larger \\(n\\) the \\(t\\)-distribution closely approximates the normal distribution. So let’s run a a simulation to check whether the definition of the confidence interval seems right. We will use our heavy metal distribution from above, take samples from the distribution, calculate the mean and confidence interval, and check how often the true mean of the population (\\(M = 3.5\\)) is contained within the confidence interval. # make example reproducible set.seed(1) # parameters sample_size = 25 # size of each sample sample_n = 20 # number of samples confidence_level = 0.95 # desired level of confidence # define a function that draws samples and calculates means and CIs fun.confidence = function(sample_size, distribution){ df = tibble(values = sample(distribution$numbers, size = sample_size, replace = T, prob = distribution$probability)) %&gt;% summarize(mean = mean(values), sd = sd(values), n = n(), # confidence interval assuming a normal distribution # error = qnorm(1 - (1 - confidence_level)/2) * sd / sqrt(n), # assuming a t-distribution (more conservative, appropriate for smaller # sample sizes) error = qt(1 - (1 - confidence_level)/2, df = n - 1) * sd / sqrt(n), conf_low = mean - error, conf_high = mean + error) return(df) } # build data frame of confidence intervals df.confidence = tibble() for(i in 1:sample_n){ df.tmp = fun.confidence(sample_size, df.population) df.confidence = df.confidence %&gt;% bind_rows(df.tmp) } # code which CIs contain the true value, and which ones don&#39;t population_mean = 3.5 df.confidence = df.confidence %&gt;% mutate(sample = 1:n(), conf_index = ifelse(conf_low &gt; population_mean | conf_high &lt; population_mean, &#39;outside&#39;, &#39;inside&#39;)) # plot the result ggplot(data = df.confidence, aes(x = sample, y = mean, color = conf_index)) + geom_hline(yintercept = 3.5, color = &quot;red&quot;) + geom_point() + geom_linerange(aes(ymin = conf_low, ymax = conf_high)) + coord_flip() + scale_color_manual(values = c(&quot;black&quot;, &quot;red&quot;), labels = c(&quot;inside&quot;, &quot;outside&quot;)) + theme(axis.text.y = element_text(size = 12), legend.position = &quot;none&quot;) So, out of the 20 samples that we drew the 95% confidence interval of 1 sample did not contain the true mean. That makes sense! Feel free to play around with the code above. For example, change the sample size, the number of samples, the confidence level. 8.4.1 mean_cl_boot() explained set.seed(1) n = 10 # sample size per group k = 3 # number of groups df.data = tibble(participant = 1:(n*k), condition = as.factor(rep(1:k, each = n)), rating = rnorm(n*k, mean = 7, sd = 1)) p = ggplot(data = df.data, mapping = aes(x = condition, y = rating)) + geom_point(alpha = 0.1, position = position_jitter(width = 0.1, height = 0)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, shape = 21, size = 1, fill = &quot;lightblue&quot;) print(p) Peeking behind the scenes build = ggplot_build(p) build$data[[2]] %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) x group y ymin ymax PANEL flipped_aes colour size linetype shape fill alpha stroke 1 1 7.13 6.69 7.62 1 FALSE black 1 1 21 lightblue NA 1 2 2 7.25 6.56 7.82 1 FALSE black 1 1 21 lightblue NA 1 3 3 6.87 6.28 7.41 1 FALSE black 1 1 21 lightblue NA 1 Let’s focus on condition 1 set.seed(1) df.condition1 = df.data %&gt;% filter(condition == 1) fun.sample_with_replacement = function(df){ df %&gt;% slice_sample(n = nrow(df), replace = T) %&gt;% summarize(mean = mean(rating)) %&gt;% pull(mean) } bootstraps = replicate(n = 100, fun.sample_with_replacement(df.condition1)) quantile(bootstraps, prob = c(0.025, 0.975)) 2.5% 97.5% 6.671962 7.583905 ggplot(data = as_tibble(bootstraps), mapping = aes(x = value)) + geom_density(size = 1) + geom_vline(xintercept = quantile(bootstraps, probs = c(0.025, 0.975)), linetype = 2) 8.5 Additional resources 8.5.1 Datacamp Foundations of Inference 8.6 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 janitor_2.1.0 kableExtra_1.3.4 knitr_1.39 loaded via a namespace (and not attached): [1] fs_1.5.2 lubridate_1.8.0 webshot_0.5.3 [4] RColorBrewer_1.1-3 httr_1.4.3 tools_4.1.2 [7] backports_1.4.1 bslib_0.3.1 utf8_1.2.2 [10] R6_2.5.1 rpart_4.1.16 Hmisc_4.7-0 [13] DBI_1.1.2 colorspace_2.0-3 nnet_7.3-17 [16] withr_2.5.0 tidyselect_1.1.2 gridExtra_2.3 [19] compiler_4.1.2 cli_3.3.0 rvest_1.0.2 [22] htmlTable_2.4.0 xml2_1.3.3 labeling_0.4.2 [25] bookdown_0.26 sass_0.4.1 scales_1.2.0 [28] checkmate_2.1.0 systemfonts_1.0.4 digest_0.6.29 [31] foreign_0.8-82 rmarkdown_2.14 svglite_2.1.0 [34] base64enc_0.1-3 jpeg_0.1-9 pkgconfig_2.0.3 [37] htmltools_0.5.2 dbplyr_2.1.1 fastmap_1.1.0 [40] highr_0.9 htmlwidgets_1.5.4 rlang_1.0.2 [43] readxl_1.4.0 rstudioapi_0.13 jquerylib_0.1.4 [46] farver_2.1.0 generics_0.1.2 jsonlite_1.8.0 [49] magrittr_2.0.3 Formula_1.2-4 Matrix_1.4-1 [52] munsell_0.5.0 fansi_1.0.3 lifecycle_1.0.1 [55] stringi_1.7.6 yaml_2.3.5 snakecase_0.11.0 [58] grid_4.1.2 crayon_1.5.1 lattice_0.20-45 [61] haven_2.5.0 splines_4.1.2 hms_1.1.1 [64] pillar_1.7.0 reprex_2.0.1 glue_1.6.2 [67] evaluate_0.15 latticeExtra_0.6-29 data.table_1.14.2 [70] modelr_0.1.8 vctrs_0.4.1 png_0.1-7 [73] tzdb_0.3.0 cellranger_1.1.0 gtable_0.3.0 [76] assertthat_0.2.1 xfun_0.30 broom_0.8.0 [79] survival_3.3-1 viridisLite_0.4.0 cluster_2.1.3 [82] ellipsis_0.3.2 "],["modeling-data.html", "Chapter 9 Modeling data 9.1 Load packages and set plotting theme 9.2 Modeling data 9.3 Hypothesis testing: “One-sample t-test” 9.4 Building a sampling distribution of PRE 9.5 Misc 9.6 Additional resources 9.7 Session info", " Chapter 9 Modeling data 9.1 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 9.2 Modeling data 9.2.1 Simplicity vs. accuracy trade-off # make example reproducible set.seed(1) n_samples = 20 # sample size n_parameters = 2 # number of parameters in the polynomial regression # generate data df.data = tibble(x = runif(n_samples, min = 0, max = 10), y = 10 + 3 * x + 3 * x^2 + rnorm(n_samples, sd = 20)) # plot a fit to the data ggplot(data = df.data, mapping = aes(x = x, y = y)) + geom_point(size = 3) + # geom_hline(yintercept = mean(df.data$y), color = &quot;blue&quot;) + geom_smooth(method = &quot;lm&quot;, se = F, formula = y ~ poly(x, degree = n_parameters, raw = TRUE)) + theme(axis.ticks = element_blank(), axis.title = element_blank(), axis.text = element_blank()) Figure 2.5: Tradeoff between fit and model simplicity. # make example reproducible set.seed(1) # n_samples = 20 n_samples = 3 df.pre = tibble(x = runif(n_samples, min = 0, max = 10), y = 2 * x + rnorm(n_samples, sd = 1)) # plot a fit to the data ggplot(data = df.pre, mapping = aes(x = x, y = y)) + geom_point(size = 3) + # geom_hline(yintercept = mean(df.pre$y), color = &quot;blue&quot;) + geom_smooth(method = &quot;lm&quot;, se = F, formula = y ~ poly(x, 1, raw = TRUE)) + theme(axis.ticks = element_blank(), axis.title = element_blank(), axis.text = element_blank()) Figure 6.1: Figure that I used to illustrate that fitting more data points with fewer parameter is more impressive. 9.2.2 Sampling distributions for median and mean # make example reproducible set.seed(1) sample_size = 40 # size of each sample sample_n = 1000 # number of samples # draw sample fun.draw_sample = function(sample_size, distribution){ x = 50 + rnorm(sample_size) return(x) } # generate many samples samples = replicate(n = sample_n, fun.draw_sample(sample_size, df.population)) # set up a data frame with samples df.sampling_distribution = matrix(samples, ncol = sample_n) %&gt;% as_tibble(.name_repair = ~ str_c(1:sample_n)) %&gt;% pivot_longer(cols = everything(), names_to = &quot;sample&quot;, values_to = &quot;number&quot;) %&gt;% mutate(sample = as.numeric(sample)) %&gt;% group_by(sample) %&gt;% mutate(draw = 1:n()) %&gt;% select(sample, draw, number) %&gt;% ungroup() # turn the data frame into long format and calculate the mean and median of each sample df.sampling_distribution_summaries = df.sampling_distribution %&gt;% group_by(sample) %&gt;% summarize(mean = mean(number), median = median(number)) %&gt;% ungroup() %&gt;% pivot_longer(cols = -sample, names_to = &quot;index&quot;, values_to = &quot;value&quot;) And plot it: # plot a histogram of the means with density overlaid ggplot(data = df.sampling_distribution_summaries, mapping = aes(x = value, color = index)) + stat_density(bw = 0.1, size = 2, geom = &quot;line&quot;) + scale_y_continuous(expand = expansion(mult = c(0, 0.01))) 9.2.3 Residuals need to be normally distributed, not the data itself set.seed(1) n_participants = 1000 df.normal = tibble(participant = 1:n_participants, condition = rep(c(&quot;control&quot;, &quot;experimental&quot;), each = n_participants/2)) %&gt;% mutate(score = ifelse(condition == &quot;control&quot;, rnorm(n = n_participants, mean = 5, sd = 2), rnorm(n = n_participants, mean = 15, sd = 3))) # distribution of the data ggplot(data = df.normal, mapping = aes(x = score)) + geom_density() + geom_density(mapping = aes(group = condition, color = condition)) # distribution of the residuals after having fitted a linear model # we&#39;ll learn how to do this later fit = lm(formula = score ~ 1 + condition, data = df.normal) ggplot(data = tibble(residuals = fit$residuals), mapping = aes(x = residuals)) + geom_density() 9.3 Hypothesis testing: “One-sample t-test” df.internet = read_table2(file = &quot;data/internet_access.txt&quot;) %&gt;% clean_names() ── Column specification ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── cols( State = col_character(), Internet = col_double(), College = col_double(), Auto = col_double(), Density = col_double() ) df.internet %&gt;% mutate(i = 1:n()) %&gt;% select(i, internet, everything()) %&gt;% head(10) %&gt;% kable(digits = 1) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) i internet state college auto density 1 79.0 AK 28.0 1.2 1.2 2 63.5 AL 23.5 1.3 94.4 3 60.9 AR 20.6 1.7 56.0 4 73.9 AZ 27.4 1.3 56.3 5 77.9 CA 31.0 0.8 239.1 6 79.4 CO 37.8 1.0 48.5 7 77.5 CT 37.2 1.0 738.1 8 74.5 DE 29.8 1.1 460.8 9 74.3 FL 27.2 1.2 350.6 10 72.2 GA 28.3 1.1 168.4 # parameters per model pa = 1 pc = 0 df.model = df.internet %&gt;% select(internet, state) %&gt;% mutate(i = 1:n(), compact_b = 75, augmented_b = mean(internet), compact_se = (internet-compact_b)^2, augmented_se = (internet-augmented_b)^2) %&gt;% select(i, state, internet, contains(&quot;compact&quot;), contains(&quot;augmented&quot;)) df.model %&gt;% summarize(augmented_sse = sum(augmented_se), compact_sse = sum(compact_se), pre = 1 - augmented_sse/compact_sse, f = (pre/(pa-pc))/((1-pre)/(nrow(df.model)-pa)), p_value = 1-pf(f, pa-pc, nrow(df.model)-1), mean = mean(internet), sd = sd(internet)) %&gt;% kable() %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) augmented_sse compact_sse pre f p_value mean sd 1355.028 1595.71 0.1508305 8.703441 0.0048592 72.806 5.258673 df1 = 1 df2 = 49 ggplot(data = tibble(x = c(0, 10)), mapping = aes(x = x)) + stat_function(fun = &quot;df&quot;, geom = &quot;area&quot;, fill = &quot;red&quot;, alpha = 0.5, args = list(df1 = df1, df2 = df2), size = 1, xlim = c(qf(0.95, df1 = df1, df2 = df2), 10)) + stat_function(fun = ~ df(x = ., df1 = df1, df2 = df2), size = 0.5) + scale_y_continuous(expand = expansion(add = c(0.001, 0.1))) + labs(y = &quot;Density&quot;, x = &quot;Proportional reduction in error&quot;) Figure 9.1: F-distribution We’ve implemented a one sample t-test (compare the p-value here to the one I computed above using PRE and the F statistic). t.test(df.internet$internet, mu = 75) One Sample t-test data: df.internet$internet t = -2.9502, df = 49, p-value = 0.004859 alternative hypothesis: true mean is not equal to 75 95 percent confidence interval: 71.3115 74.3005 sample estimates: mean of x 72.806 9.4 Building a sampling distribution of PRE Here is the general procedure for building a sampling distribution of the proportional reduction in error (PRE). In this instance, I compare the following two models Model C (compact): \\(Y_i = 75 + \\epsilon_i\\) Model A (augmented): \\(Y_i = \\overline Y + \\epsilon_i\\) whereby I assume that \\(\\epsilon_i \\sim \\mathcal{N}(0, \\sigma)\\). For this example, I assume that I know the population distribution. I first draw a sample from that distribution, and then calculate PRE. # make example reproducible set.seed(1) # set the sample size sample_size = 50 # draw sample from the population distribution (I&#39;ve fixed sigma -- the standard deviation # of the population distribution to be 5) df.sample = tibble(observation = 1:sample_size, value = 75 + rnorm(sample_size, mean = 0, sd = 5)) # calculate SSE for each model, and then PRE based on that df.summary = df.sample %&gt;% mutate(compact = 75, augmented = mean(value)) %&gt;% summarize(sse_compact = sum((value - compact)^2), sse_augmented = sum((value - augmented)^2), pre = 1 - (sse_augmented/sse_compact)) To generate the sampling distribution, I assume that the null hypothesis is true, and then take a look at what values for PRE we could expect by chance for our given sample size. # simulation parameters n_samples = 1000 sample_size = 50 mu = 75 # true mean of the distribution sigma = 5 # true standard deviation of the errors # function to draw samples from the population distribution fun.draw_sample = function(sample_size, mu, sigma){ sample = mu + rnorm(sample_size, mean = 0, sd = sigma) return(sample) } # draw samples samples = n_samples %&gt;% replicate(fun.draw_sample(sample_size, mu, sigma)) %&gt;% t() # transpose the resulting matrix (i.e. flip rows and columns) # put samples in data frame and compute PRE df.samples = samples %&gt;% as_tibble(.name_repair = ~ str_c(1:ncol(samples))) %&gt;% mutate(sample = 1:n()) %&gt;% pivot_longer(cols = -sample, names_to = &quot;index&quot;, values_to = &quot;value&quot;) %&gt;% mutate(compact = mu) %&gt;% group_by(sample) %&gt;% mutate(augmented = mean(value)) %&gt;% summarize(sse_compact = sum((value - compact)^2), sse_augmented = sum((value - augmented)^2), pre = 1 - sse_augmented/sse_compact) # plot the sampling distribution for PRE ggplot(data = df.samples, mapping = aes(x = pre)) + stat_density(geom = &quot;line&quot;) + labs(x = &quot;Proportional reduction in error&quot;) # calculate the p-value for our sample df.samples %&gt;% summarize(p_value = sum(pre &gt;= df.summary$pre)/n()) # A tibble: 1 × 1 p_value &lt;dbl&gt; 1 0.394 Some code I wrote to show a subset of the samples. samples %&gt;% as_tibble(.name_repair = &quot;unique&quot;) %&gt;% mutate(sample = 1:n()) %&gt;% pivot_longer(cols = -sample, names_to = &quot;index&quot;, values_to = &quot;value&quot;) %&gt;% mutate(compact = mu) %&gt;% group_by(sample) %&gt;% mutate(augmented = mean(value)) %&gt;% ungroup() %&gt;% mutate(index = str_extract(index, pattern = &quot;\\\\-*\\\\d+\\\\.*\\\\d*&quot;), index = as.numeric(index)) %&gt;% filter(index &lt; 6) %&gt;% arrange(sample, index) %&gt;% head(15) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) sample index value compact augmented 1 1 76.99 75 75.59 1 2 71.94 75 75.59 1 3 76.71 75 75.59 1 4 69.35 75 75.59 1 5 82.17 75 75.59 2 1 71.90 75 74.24 2 2 75.21 75 74.24 2 3 70.45 75 74.24 2 4 75.79 75 74.24 2 5 71.73 75 74.24 3 1 77.25 75 75.38 3 2 74.91 75 75.38 3 3 73.41 75 75.38 3 4 70.35 75 75.38 3 5 67.56 75 75.38 9.5 Misc Some code to plot probability distributions together with values of interest highlighted. value_mean = 3.73 value_sd = 2.05/sqrt(40) q_low = qnorm(0.025, mean = value_mean, sd = value_sd) q_high = qnorm(0.975, mean = value_mean, sd = value_sd) qnorm(0.975) * value_sd [1] 0.6352899 # density function ggplot(data = tibble(x = c(2.73, 4.73)), mapping = aes(x = x)) + stat_function(fun = ~ dnorm(., mean = value_mean, sd = value_sd), size = 2) + geom_vline(xintercept = c(q_low, q_high), linetype = 2) # quantile function df.paths = tibble(x = c(rep(c(0.025, 0.975), each = 2), -Inf, 0.025, -Inf, 0.975), y = c(2.9, q_low, 2.9, q_high, q_low, q_low, q_high, q_high), group = rep(1:4, each = 2)) ggplot(data = tibble(x = c(0, 1)), mapping = aes(x = x)) + stat_function(fun = ~ qnorm(., mean = value_mean, sd = value_sd)) + geom_path(data = df.paths, mapping = aes(x = x, y = y, group = group), color = &quot;blue&quot;, size = 2, lineend = &quot;round&quot;) + coord_cartesian(xlim = c(-0.05, 1.05), ylim = c(2.9, 4.5), expand = F) 9.6 Additional resources 9.6.1 Reading Judd, C. M., McClelland, G. H., &amp; Ryan, C. S. (2011). Data analysis: A model comparison approach. Routledge. –&gt; Chapters 1–4 9.6.2 Datacamp Foundations of Inference 9.7 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 janitor_2.1.0 kableExtra_1.3.4 knitr_1.39 loaded via a namespace (and not attached): [1] lattice_0.20-45 svglite_2.1.0 lubridate_1.8.0 assertthat_0.2.1 [5] digest_0.6.29 utf8_1.2.2 R6_2.5.1 cellranger_1.1.0 [9] backports_1.4.1 reprex_2.0.1 evaluate_0.15 highr_0.9 [13] httr_1.4.3 pillar_1.7.0 rlang_1.0.2 readxl_1.4.0 [17] rstudioapi_0.13 jquerylib_0.1.4 Matrix_1.4-1 rmarkdown_2.14 [21] labeling_0.4.2 splines_4.1.2 webshot_0.5.3 munsell_0.5.0 [25] broom_0.8.0 compiler_4.1.2 modelr_0.1.8 xfun_0.30 [29] pkgconfig_2.0.3 systemfonts_1.0.4 mgcv_1.8-40 htmltools_0.5.2 [33] tidyselect_1.1.2 bookdown_0.26 fansi_1.0.3 viridisLite_0.4.0 [37] withr_2.5.0 crayon_1.5.1 tzdb_0.3.0 dbplyr_2.1.1 [41] grid_4.1.2 nlme_3.1-157 jsonlite_1.8.0 gtable_0.3.0 [45] lifecycle_1.0.1 DBI_1.1.2 magrittr_2.0.3 scales_1.2.0 [49] cli_3.3.0 stringi_1.7.6 farver_2.1.0 fs_1.5.2 [53] snakecase_0.11.0 xml2_1.3.3 bslib_0.3.1 ellipsis_0.3.2 [57] generics_0.1.2 vctrs_0.4.1 tools_4.1.2 glue_1.6.2 [61] hms_1.1.1 fastmap_1.1.0 yaml_2.3.5 colorspace_2.0-3 [65] rvest_1.0.2 haven_2.5.0 sass_0.4.1 "],["linear-model-1.html", "Chapter 10 Linear model 1 10.1 Load packages and set plotting theme 10.2 Correlation 10.3 Regression 10.4 Credit example 10.5 Additional resources 10.6 Session info", " Chapter 10 Linear model 1 10.1 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;broom&quot;) # for tidying up linear models library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 10.2 Correlation # make example reproducible set.seed(1) n_samples = 20 # create correlated data df.correlation = tibble(x = runif(n_samples, min = 0, max = 100), y = x + rnorm(n_samples, sd = 15)) # plot the data ggplot(data = df.correlation, mapping = aes(x = x, y = y)) + geom_point(size = 2) + labs(x = &quot;chocolate&quot;, y = &quot;happiness&quot;) 10.2.0.1 Variance Variance is the average squared difference between each data point and the mean: \\(Var(Y) = \\frac{\\sum_{i = 1}^n(Y_i - \\overline Y)^2}{n-1}\\) # make example reproducible set.seed(1) # generate random data df.variance = tibble(x = 1:10, y = runif(10, min = 0, max = 1)) # plot the data ggplot(data = df.variance, mapping = aes(x = x, y = y)) + geom_segment(aes(x = x, xend = x, y = y, yend = mean(df.variance$y))) + geom_point(size = 3) + geom_hline(yintercept = mean(df.variance$y), color = &quot;blue&quot;) + theme(axis.text.x = element_blank(), axis.title.x = element_blank(), axis.ticks.x = element_blank()) Warning: Use of `df.variance$y` is discouraged. Use `y` instead. 10.2.0.2 Covariance Covariance is defined in the following way: \\(Cov(X,Y) = \\sum_{i=1}^n\\frac{(X_i-\\overline X)(Y_i-\\overline Y)}{n-1}\\) # make example reproducible set.seed(1) # generate random data df.covariance = tibble(x = runif(20, min = 0, max = 1), y = x + rnorm(x, mean = 0.5, sd = 0.25)) # plot the data ggplot(df.covariance, aes(x = x, y = y)) + geom_point(size = 3) + theme(axis.text = element_blank(), axis.title = element_blank(), axis.ticks = element_blank()) Add lines for \\(\\overline X\\) and \\(\\overline Y\\) to the data: ggplot(df.covariance, aes(x = x, y = y)) + geom_hline(yintercept = mean(df.covariance$y), color = &quot;red&quot;, size = 1) + geom_vline(xintercept = mean(df.covariance$x), color = &quot;red&quot;, size = 1) + geom_point(size = 3) + theme(axis.text = element_blank(), axis.title = element_blank(), axis.ticks = element_blank()) Illustrate how covariance is computed by drawing the distance to \\(\\overline X\\) and \\(\\overline Y\\) for three data points: df.plot = df.covariance %&gt;% mutate(covariance = (x-mean(x)) *( y-mean(y))) %&gt;% arrange(abs(covariance)) %&gt;% mutate(color = NA) mean_xy = c(mean(df.covariance$x), mean(df.covariance$y)) df.plot$color[1] = 1 df.plot$color[10] = 2 df.plot$color[19] = 3 ggplot(df.plot, aes(x = x, y = y, color = as.factor(color))) + geom_segment(data = df.plot %&gt;% filter(color == 1), mapping = aes(x = x, xend = mean_xy[1], y = y, yend = y), size = 1) + geom_segment(data = df.plot %&gt;% filter(color == 1), mapping = aes(x = x, xend = x, y = y, yend = mean_xy[2]), size = 1) + geom_segment(data = df.plot %&gt;% filter(color == 2), mapping = aes(x = x, xend = mean_xy[1], y = y, yend = y), size = 1) + geom_segment(data = df.plot %&gt;% filter(color == 2), mapping = aes(x = x, xend = x, y = y, yend = mean_xy[2]), size = 1) + geom_segment(data = df.plot %&gt;% filter(color == 3), mapping = aes(x = x, xend = mean_xy[1], y = y, yend = y), size = 1) + geom_segment(data = df.plot %&gt;% filter(color == 3), mapping = aes(x = x, xend = x, y = y, yend = mean_xy[2]), size = 1) + geom_hline(yintercept = mean_xy[2], color = &quot;red&quot;, size = 1) + geom_vline(xintercept = mean_xy[1], color = &quot;red&quot;, size = 1) + geom_point(size = 3) + theme(axis.text = element_blank(), axis.title = element_blank(), axis.ticks = element_blank(), legend.position = &quot;none&quot;) 10.2.0.3 Spearman’s rank order correlation Spearman’s \\(\\rho\\) captures the extent to which the relationship between two variables is monotonic. # create data frame with data points and ranks df.ranking = tibble(x = c(1.2, 2.5, 4.5), y = c(2.2, 1, 3.3), label = str_c(&quot;(&quot;, x, &quot;, &quot;, y, &quot;)&quot;), x_rank = dense_rank(x), y_rank = dense_rank(y), label_rank = str_c(&quot;(&quot;, x_rank, &quot;, &quot;, y_rank, &quot;)&quot;)) # plot the data (and show their ranks) ggplot(df.ranking, aes(x = x, y = y)) + geom_point(size = 3) + geom_text(aes(label = label), hjust = -0.2, vjust = 0, size = 6) + geom_text(aes(label = label_rank), hjust = -0.4, vjust = 2, size = 6, color = &quot;red&quot;) + coord_cartesian(xlim = c(1, 6), ylim = c(0, 4)) Show that Spearman’s \\(\\rho\\) is equivalent to Pearson’s \\(r\\) applied to ranked data. # data set df.spearman = df.correlation %&gt;% mutate(x_rank = dense_rank(x), y_rank = dense_rank(y)) # correlation df.spearman %&gt;% summarize(r = cor(x, y, method = &quot;pearson&quot;), spearman = cor(x, y, method = &quot;spearman&quot;), r_ranks = cor(x_rank, y_rank)) # A tibble: 1 × 3 r spearman r_ranks &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 0.851 0.836 0.836 # plot ggplot(df.spearman, aes(x = x_rank, y = y_rank)) + geom_point(size = 3) + scale_x_continuous(breaks = 1:20) + scale_y_continuous(breaks = 1:20) + theme(axis.text = element_text(size = 10)) # show some of the data and ranks df.spearman %&gt;% head(10) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) x y x_rank y_rank 26.55 49.23 5 10 37.21 43.06 6 7 57.29 47.97 10 8 90.82 57.60 18 11 20.17 37.04 3 6 89.84 89.16 17 19 94.47 94.22 19 20 66.08 80.24 12 16 62.91 75.23 11 14 6.18 15.09 1 2 Comparison between \\(r\\) and \\(\\rho\\) for a given data set: # data set df.example = tibble(x = 1:10, y = c(-10, 2:9, 20)) %&gt;% mutate(x_rank = dense_rank(x), y_rank = dense_rank(y)) # correlation df.example %&gt;% summarize(r = cor(x, y, method = &quot;pearson&quot;), spearman = cor(x, y, method = &quot;spearman&quot;), r_ranks = cor(x_rank, y_rank)) # A tibble: 1 × 3 r spearman r_ranks &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 0.878 1 1 # plot ggplot(df.example, # aes(x = x_rank, y = y_rank)) + # see the ranked data aes(x = x, y = y)) + # see the original data geom_point(size = 3) + theme(axis.text = element_text(size = 10)) Another example # make example reproducible set.seed(1) # data set df.example2 = tibble(x = c(1, rnorm(8, mean = 5, sd = 1), 10), y = c(-10, rnorm(8, sd = 1), 20)) %&gt;% mutate(x_rank = dense_rank(x), y_rank = dense_rank(y)) # correlation df.example2 %&gt;% summarize(r = cor(x, y, method = &quot;pearson&quot;), spearman = cor(x, y, method = &quot;spearman&quot;), r_ranks = cor(x_rank, y_rank)) # A tibble: 1 × 3 r spearman r_ranks &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 0.919 0.467 0.467 # plot ggplot(df.example2, # aes(x = x_rank, y = y_rank)) + # see the ranked data aes(x = x, y = y)) + # see the original data geom_point(size = 3) + theme(axis.text = element_text(size = 10)) 10.3 Regression # make example reproducible set.seed(1) # set the sample size n_samples = 10 # generate correlated data df.regression = tibble(chocolate = runif(n_samples, min = 0, max = 100), happiness = chocolate * 0.5 + rnorm(n_samples, sd = 15)) # plot the data ggplot(data = df.regression, aes(x = chocolate, y = happiness)) + geom_point(size = 3) 10.3.1 Define and fit the models Define and fit the compact model (Model C): \\(Y_i = \\beta_0 + \\epsilon_i\\) # fit the compact model lm.compact = lm(happiness ~ 1, data = df.regression) # store the results of the model fit in a data frame df.compact = tidy(lm.compact) # plot the data with model prediction ggplot(data = df.regression, aes(x = chocolate, y = happiness)) + geom_hline(yintercept = df.compact$estimate, color = &quot;blue&quot;, size = 1) + geom_point(size = 3) Define and fit the augmented model (Model A): \\(Y_i = \\beta_0 + \\beta_1 X_{1i} + \\epsilon_i\\) # fit the augmented model lm.augmented = lm(happiness ~ chocolate, data = df.regression) # store the results of the model fit in a data frame df.augmented = tidy(lm.augmented) # plot the data with model prediction ggplot(data = df.regression, aes(x = chocolate, y = happiness)) + geom_abline(intercept = df.augmented$estimate[1], slope = df.augmented$estimate[2], color = &quot;red&quot;, size = 1) + geom_point(size = 3) 10.3.2 Calculate the sum of squared errors of each model Illustration of the residuals for the compact model: # fit the model lm.compact = lm(happiness ~ 1, data = df.regression) # store the model information df.compact_summary = tidy(lm.compact) # create a data frame that contains the residuals df.compact_model = augment(lm.compact) %&gt;% clean_names() %&gt;% left_join(df.regression, by = &quot;happiness&quot;) # plot model prediction with residuals ggplot(data = df.compact_model, aes(x = chocolate, y = happiness)) + geom_hline(yintercept = df.compact_summary$estimate, color = &quot;blue&quot;, size = 1) + geom_segment(aes(xend = chocolate, yend = df.compact_summary$estimate), color = &quot;blue&quot;) + geom_point(size = 3) # calculate the sum of squared errors df.compact_model %&gt;% summarize(SSE = sum(resid^2)) # A tibble: 1 × 1 SSE &lt;dbl&gt; 1 5215. Illustration of the residuals for the augmented model: # fit the model lm.augmented = lm(happiness ~ chocolate, data = df.regression) # store the model information df.augmented_summary = tidy(lm.augmented) # create a data frame that contains the residuals df.augmented_model = augment(lm.augmented) %&gt;% clean_names() %&gt;% left_join(df.regression, by = c(&quot;happiness&quot;, &quot;chocolate&quot;)) # plot model prediction with residuals ggplot(data = df.augmented_model, aes(x = chocolate, y = happiness)) + geom_abline(intercept = df.augmented_summary$estimate[1], slope = df.augmented_summary$estimate[2], color = &quot;red&quot;, size = 1) + geom_segment(aes(xend = chocolate, yend = fitted), color = &quot;red&quot;) + geom_point(size = 3) # calculate the sum of squared errors df.augmented_model %&gt;% summarize(SSE = sum(resid^2)) # A tibble: 1 × 1 SSE &lt;dbl&gt; 1 2397. Calculate the F-test to determine whether PRE is significant. pc = 1 # number of parameters in the compact model pa = 2 # number of parameters in the augmented model n = 10 # number of observations # SSE of the compact model sse_compact = df.compact_model %&gt;% summarize(SSE = sum(resid^2)) # SSE of the augmented model sse_augmented = df.augmented_model %&gt;% summarize(SSE = sum(resid^2)) # Proportional reduction of error pre = as.numeric(1 - (sse_augmented/sse_compact)) # F-statistic f = (pre/(pa-pc))/((1-pre)/(n-pa)) # p-value p_value = 1-pf(f, df1 = pa-pc, df2 = n-pa) print(p_value) [1] 0.01542156 F-distribution with a red line indicating the calculated F-statistic. ggplot(data = tibble(x = c(0, 10)), mapping = aes(x = x)) + stat_function(fun = &quot;df&quot;, args = list(df1 = pa-pc, df2 = n-pa), size = 1) + geom_vline(xintercept = f, color = &quot;red&quot;, size = 1) The short version of doing what we did above :) anova(lm.compact, lm.augmented) Analysis of Variance Table Model 1: happiness ~ 1 Model 2: happiness ~ chocolate Res.Df RSS Df Sum of Sq F Pr(&gt;F) 1 9 5215.0 2 8 2396.9 1 2818.1 9.4055 0.01542 * --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 10.4 Credit example Let’s load the credit card data: df.credit = read_csv(&quot;data/credit.csv&quot;) %&gt;% clean_names() Here is a short description of the variables: variable description income in thousand dollars limit credit limit rating credit rating cards number of credit cards age in years education years of education gender male or female student student or not married married or not ethnicity African American, Asian, Caucasian balance average credit card debt Scatterplot of the relationship between income and balance. ggplot(data = df.credit, mapping = aes(x = income, y = balance)) + geom_point(alpha = 0.3) + coord_cartesian(xlim = c(0, max(df.credit$income))) To make the model intercept interpretable, we can center the predictor variable by subtracting the mean from each value. df.plot = df.credit %&gt;% mutate(income_centered = income - mean(income)) %&gt;% select(balance, income, income_centered) fit = lm(balance ~ 1 + income_centered, data = df.plot) ggplot(data = df.plot, mapping = aes(x = income_centered, y = balance)) + geom_vline(xintercept = 0, linetype = 2, color = &quot;black&quot;) + geom_hline(yintercept = mean(df.plot$balance), color = &quot;red&quot;) + geom_point(alpha = 0.3) + geom_smooth(method = &quot;lm&quot;, se = F) + scale_color_manual(values = c(&quot;black&quot;, &quot;red&quot;)) `geom_smooth()` using formula &#39;y ~ x&#39; # coord_cartesian(xlim = c(0, max(df.plot$income_centered))) Let’s fit the model and take a look at the model summary: fit = lm(balance ~ 1 + income, data = df.credit) fit %&gt;% summary() Call: lm(formula = balance ~ 1 + income, data = df.credit) Residuals: Min 1Q Median 3Q Max -803.64 -348.99 -54.42 331.75 1100.25 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 246.5148 33.1993 7.425 6.9e-13 *** income 6.0484 0.5794 10.440 &lt; 2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 407.9 on 398 degrees of freedom Multiple R-squared: 0.215, Adjusted R-squared: 0.213 F-statistic: 109 on 1 and 398 DF, p-value: &lt; 2.2e-16 Here, I double check that I understand how the statistics about the residuals are calculated that the model summary gives me. fit %&gt;% augment() %&gt;% clean_names() %&gt;% summarize(min = min(resid), first_quantile = quantile(resid, 0.25), median = median(resid), third_quantile = quantile(resid, 0.75), max = max(resid), rmse = sqrt(mean(resid^2))) # A tibble: 1 × 6 min first_quantile median third_quantile max rmse &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 -804. -349. -54.4 332. 1100. 407. Here is a plot of the residuals. Residual plots are important for checking whether any of the linear model assumptions have been violated. fit %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(aes(x = fitted, y = resid)) + geom_hline(yintercept = 0, color = &quot;blue&quot;) + geom_point(alpha = 0.3) We can use the glance() function from the broom package to print out model statistics. fit %&gt;% glance() %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) r.squared adj.r.squared sigma statistic p.value df logLik AIC BIC deviance df.residual nobs 0.21 0.21 407.86 108.99 0 1 -2970.95 5947.89 5959.87 66208745 398 400 Let’s test whether income is a significant predictor of balance in the credit data set. # fitting the compact model fit_c = lm(formula = balance ~ 1, data = df.credit) # fitting the augmented model fit_a = lm(formula = balance ~ 1 + income, data = df.credit) # run the F test anova(fit_c, fit_a) Analysis of Variance Table Model 1: balance ~ 1 Model 2: balance ~ 1 + income Res.Df RSS Df Sum of Sq F Pr(&gt;F) 1 399 84339912 2 398 66208745 1 18131167 108.99 &lt; 2.2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Let’s print out the parameters of the augmented model with confidence intervals: fit_a %&gt;% tidy(conf.int = T) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) term estimate std.error statistic p.value conf.low conf.high (Intercept) 246.51 33.20 7.43 0 181.25 311.78 income 6.05 0.58 10.44 0 4.91 7.19 We can use augment() with the newdata = argument to get predictions about new data from our fitted model: fit %&gt;% augment(newdata = tibble(income = 130)) # A tibble: 1 × 2 income .fitted &lt;dbl&gt; &lt;dbl&gt; 1 130 1033. Here is a plot of the model with confidence interval (that captures our uncertainty in the intercept and slope of the model) and the predicted balance value for an income of 130: ggplot(data = df.credit, mapping = aes(x = income, y = balance)) + geom_point(alpha = 0.3) + geom_smooth(method = &quot;lm&quot;) + annotate(geom = &quot;point&quot;, color = &quot;red&quot;, size = 5, x = 130, y = predict(fit, newdata = tibble(income = 130))) + coord_cartesian(xlim = c(0, max(df.credit$income))) `geom_smooth()` using formula &#39;y ~ x&#39; Finally, let’s take a look at how the residuals are distributed. # get the residuals df.plot = fit_a %&gt;% augment() %&gt;% clean_names() # and a density of the residuals ggplot(df.plot, aes(x = resid)) + stat_density(geom = &quot;line&quot;) Not quite as normally distributed as we would hope. We learn what to do if some of the assumptions of the linear model are violated later in class. In general, we’d like the residuals to have the following shape: The model assumptions are: independent observations Y is continuous errors are normally distributed errors have constant variance error terms are uncorrelated Here are some examples of what the residuals could look like when things go wrong: 10.5 Additional resources 10.5.1 Datacamp Statistical modeling 1 Statistical modeling 2 Correlation and regression 10.5.2 Misc Spurious correlations 10.6 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 broom_0.8.0 janitor_2.1.0 kableExtra_1.3.4 [13] knitr_1.39 loaded via a namespace (and not attached): [1] httr_1.4.3 sass_0.4.1 bit64_4.0.5 vroom_1.5.7 [5] jsonlite_1.8.0 viridisLite_0.4.0 splines_4.1.2 modelr_0.1.8 [9] bslib_0.3.1 assertthat_0.2.1 highr_0.9 cellranger_1.1.0 [13] yaml_2.3.5 pillar_1.7.0 backports_1.4.1 lattice_0.20-45 [17] glue_1.6.2 digest_0.6.29 rvest_1.0.2 snakecase_0.11.0 [21] colorspace_2.0-3 htmltools_0.5.2 Matrix_1.4-1 pkgconfig_2.0.3 [25] haven_2.5.0 bookdown_0.26 scales_1.2.0 webshot_0.5.3 [29] svglite_2.1.0 tzdb_0.3.0 mgcv_1.8-40 generics_0.1.2 [33] farver_2.1.0 ellipsis_0.3.2 withr_2.5.0 cli_3.3.0 [37] magrittr_2.0.3 crayon_1.5.1 readxl_1.4.0 evaluate_0.15 [41] fs_1.5.2 fansi_1.0.3 nlme_3.1-157 xml2_1.3.3 [45] tools_4.1.2 hms_1.1.1 lifecycle_1.0.1 munsell_0.5.0 [49] reprex_2.0.1 compiler_4.1.2 jquerylib_0.1.4 systemfonts_1.0.4 [53] rlang_1.0.2 grid_4.1.2 rstudioapi_0.13 labeling_0.4.2 [57] rmarkdown_2.14 gtable_0.3.0 DBI_1.1.2 R6_2.5.1 [61] lubridate_1.8.0 fastmap_1.1.0 bit_4.0.4 utf8_1.2.2 [65] stringi_1.7.6 parallel_4.1.2 png_0.1-7 vctrs_0.4.1 [69] dbplyr_2.1.1 tidyselect_1.1.2 xfun_0.30 "],["linear-model-2.html", "Chapter 11 Linear model 2 11.1 Learning goals 11.2 Load packages and set plotting theme 11.3 Load data sets 11.4 Multiple continuous variables 11.5 One categorical variable 11.6 One continuous and one categorical variable 11.7 Interactions 11.8 Additional resources 11.9 Session info 11.10 References", " Chapter 11 Linear model 2 11.1 Learning goals Multiple regression. Appreciate model assumptions. Several continuous predictors. Hypothesis tests. Interpreting parameters. Reporting results. One categorical predictor. Both continuous and categorical predictors. Interpreting interactions. 11.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;broom&quot;) # for tidying up linear models library(&quot;corrr&quot;) # for calculating correlations between many variables library(&quot;corrplot&quot;) # for plotting correlations library(&quot;GGally&quot;) # for running ggpairs() function library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. # include references for used packages knitr::write_bib(.packages(), &quot;packages.bib&quot;) theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 11.3 Load data sets Let’s load the data sets that we’ll explore in this class: # credit data set df.credit = read_csv(&quot;data/credit.csv&quot;) %&gt;% rename(index = `...1`) %&gt;% clean_names() # advertising data set df.ads = read_csv(&quot;data/advertising.csv&quot;) %&gt;% rename(index = `...1`) %&gt;% clean_names() variable description income in thousand dollars limit credit limit rating credit rating cards number of credit cards age in years education years of education gender male or female student student or not married married or not ethnicity African American, Asian, Caucasian balance average credit card debt 11.4 Multiple continuous variables Let’s take a look at a case where we have multiple continuous predictor variables. In this case, we want to make sure that our predictors are not too highly correlated with each other (as this makes the interpration of how much each variable explains the outcome difficult). So we first need to explore the pairwise correlations between variables. 11.4.1 Explore correlations The corrr package is great for exploring correlations between variables. To find out more how corrr works, take a look at this vignette: vignette(topic = &quot;using-corrr&quot;, package = &quot;corrr&quot;) Here is an example that illustrates some of the key functions in the corrr package (using the advertisement data): df.ads %&gt;% select(where(is.numeric)) %&gt;% correlate(quiet = T) %&gt;% shave() %&gt;% fashion() term index tv radio newspaper sales 1 index 2 tv .02 3 radio -.11 .05 4 newspaper -.15 .06 .35 5 sales -.05 .78 .58 .23 11.4.1.1 Visualize correlations 11.4.1.1.1 Correlations with the dependent variable df.credit %&gt;% select(where(is.numeric)) %&gt;% correlate(quiet = T) %&gt;% select(term, income) %&gt;% mutate(term = reorder(term, income)) %&gt;% drop_na() %&gt;% ggplot(aes(x = term, y = income, fill = income)) + geom_hline(yintercept = 0) + geom_col(color = &quot;black&quot;, show.legend = F) + scale_fill_gradient2(low = &quot;indianred2&quot;, mid = &quot;white&quot;, high = &quot;skyblue1&quot;, limits = c(-1, 1)) + coord_flip() + theme(axis.title.y = element_blank()) Figure 2.6: Bar plot illustrating how strongly different variables correlate with income. 11.4.1.1.2 All pairwise correlations tmp = df.credit %&gt;% select(where(is.numeric), -index) %&gt;% correlate(diagonal = 0, quiet = T) %&gt;% rearrange() %&gt;% select(-term) %&gt;% as.matrix() %&gt;% corrplot() df.ads %&gt;% select(-index) %&gt;% ggpairs() Figure 11.1: Pairwise correlations with scatter plots, correlation values, and densities on the diagonal. With some customization: df.ads %&gt;% select(-index) %&gt;% ggpairs(lower = list(continuous = wrap(&quot;points&quot;, alpha = 0.3)), upper = list(continuous = wrap(&quot;cor&quot;, size = 8))) + theme(panel.grid.major = element_blank()) Figure 11.2: Pairwise correlations with scatter plots, correlation values, and densities on the diagonal (customized). 11.4.2 Multipe regression Now that we’ve explored the correlations, let’s have a go at the multiple regression. 11.4.2.1 Visualization We’ll first take another look at the pairwise relationships: tmp.x = &quot;tv&quot; # tmp.x = &quot;radio&quot; # tmp.x = &quot;newspaper&quot; # tmp.y = &quot;radio&quot; tmp.y = &quot;radio&quot; # tmp.y = &quot;tv&quot; ggplot(df.ads, aes_string(x = tmp.x, y = tmp.y)) + stat_smooth(method = &quot;lm&quot;, color = &quot;black&quot;, fullrange = T) + geom_point(alpha = 0.3) + annotate(geom = &quot;text&quot;, x = -Inf, y = Inf, hjust = -0.5, vjust = 1.5, label = str_c(&quot;r = &quot;, cor(df.ads[[tmp.x]], df.ads[[tmp.y]]) %&gt;% round(2) %&gt;% # round str_remove(&quot;^0+&quot;) # remove 0 ), size = 8) + theme(text = element_text(size = 30)) `geom_smooth()` using formula &#39;y ~ x&#39; TV ads and radio ads aren’t correlated. Yay! 11.4.2.2 Fitting, hypothesis testing, evaluation Let’s see whether adding radio ads is worth it (over and above having TV ads). # fit the models fit_c = lm(sales ~ 1 + tv, data = df.ads) fit_a = lm(sales ~ 1 + tv + radio, data = df.ads) # do the F test anova(fit_c, fit_a) Analysis of Variance Table Model 1: sales ~ 1 + tv Model 2: sales ~ 1 + tv + radio Res.Df RSS Df Sum of Sq F Pr(&gt;F) 1 198 2102.53 2 197 556.91 1 1545.6 546.74 &lt; 2.2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 It’s worth it! Let’s evaluate how well the model actually does. We do this by taking a look at the residual plot, and check whether the residuals are normally distributed. tmp.fit = lm(sales ~ 1 + tv + radio, data = df.ads) df.plot = tmp.fit %&gt;% augment() %&gt;% clean_names() # residual plot ggplot(df.plot, aes(x = fitted, y = resid)) + geom_point() # density of residuals ggplot(df.plot, aes(x = resid)) + stat_density(geom = &quot;line&quot;) # QQ plot ggplot(df.plot, aes(sample = resid)) + geom_qq() + geom_qq_line() There is a slight non-linear trend in the residuals. We can also see that the residuals aren’t perfectly normally distributed. We’ll see later what we can do about this … Let’s see how well the model does overall: fit_a %&gt;% glance() %&gt;% kable(digits = 3) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) r.squared adj.r.squared sigma statistic p.value df logLik AIC BIC deviance df.residual nobs 0.897 0.896 1.681 859.618 0 2 -386.197 780.394 793.587 556.914 197 200 As we can see, the model almost explains 90% of the variance. That’s very decent! 11.4.2.3 Visualizing the model fits Here is a way of visualizing how both tv ads and radio ads affect sales: df.plot = lm(sales ~ 1 + tv + radio, data = df.ads) %&gt;% augment() %&gt;% clean_names() df.tidy = lm(sales ~ 1 + tv + radio, data = df.ads) %&gt;% tidy() ggplot(df.plot, aes(x = radio, y = sales, color = tv)) + geom_point() + scale_color_gradient(low = &quot;gray80&quot;, high = &quot;black&quot;) + theme(legend.position = c(0.1, 0.8)) We used color here to encode TV ads (and the x-axis for the radio ads). In addition, we might want to illustrate what relationship between radio ads and sales the model predicts for three distinct values for TV ads. Like so: df.plot = lm(sales ~ 1 + tv + radio, data = df.ads) %&gt;% augment() %&gt;% clean_names() df.tidy = lm(sales ~ 1 + tv + radio, data = df.ads) %&gt;% tidy() ggplot(df.plot, aes(x = radio, y = sales, color = tv)) + geom_point() + geom_abline(intercept = df.tidy$estimate[1] + df.tidy$estimate[2] * 200, slope = df.tidy$estimate[3]) + geom_abline(intercept = df.tidy$estimate[1] + df.tidy$estimate[2] * 100, slope = df.tidy$estimate[3]) + geom_abline(intercept = df.tidy$estimate[1] + df.tidy$estimate[2] * 0, slope = df.tidy$estimate[3]) + scale_color_gradient(low = &quot;gray80&quot;, high = &quot;black&quot;) + theme(legend.position = c(0.1, 0.8)) 11.4.2.4 Interpreting the model fits Fitting the augmented model yields the following estimates for the coefficients in the model: fit_a %&gt;% tidy(conf.int = T) %&gt;% head(10) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) term estimate std.error statistic p.value conf.low conf.high (Intercept) 2.92 0.29 9.92 0 2.34 3.50 tv 0.05 0.00 32.91 0 0.04 0.05 radio 0.19 0.01 23.38 0 0.17 0.20 11.4.2.5 Standardizing the predictors One thing we can do to make different predictors more comparable is to standardize them. df.ads = df.ads %&gt;% mutate(across(.cols = c(tv, radio), .fns = ~ scale(.), .names = &quot;{.col}_scaled&quot;)) df.ads %&gt;% select(-newspaper) %&gt;% head(10) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) index tv radio sales tv_scaled radio_scaled 1 230.1 37.8 22.1 0.96742460 0.9790656 2 44.5 39.3 10.4 -1.19437904 1.0800974 3 17.2 45.9 9.3 -1.51235985 1.5246374 4 151.5 41.3 18.5 0.05191939 1.2148065 5 180.8 10.8 12.9 0.39319551 -0.8395070 6 8.7 48.9 7.2 -1.61136487 1.7267010 7 57.5 32.8 11.8 -1.04295960 0.6422929 8 120.2 19.6 13.2 -0.31265202 -0.2467870 9 8.6 2.1 4.8 -1.61252963 -1.4254915 10 199.8 2.6 10.6 0.61450084 -1.3918142 We can standardize (z-score) variables using the scale() function. # tmp.variable = &quot;tv&quot; tmp.variable = &quot;tv_scaled&quot; ggplot(df.ads, aes(x = .data[[tmp.variable]])) + stat_density(geom = &quot;line&quot;, size = 1) + annotate(geom = &quot;text&quot;, x = median(df.ads[[tmp.variable]]), y = -Inf, label = str_c(&quot;sd = &quot;, sd(df.ads[[tmp.variable]]) %&gt;% round(2)), size = 10, vjust = -1, hjust = 0.5) + annotate(geom = &quot;text&quot;, x = median(df.ads[[tmp.variable]]), y = -Inf, label = str_c(&quot;mean = &quot;, mean(df.ads[[tmp.variable]]) %&gt;% round(2)), size = 10, vjust = -3, hjust = 0.5) Scaling a variable leaves the distribution intact, but changes the mean to 0 and the SD to 1. 11.5 One categorical variable Let’s compare a compact model that only predicts the mean, with a model that uses the student variable as an additional predictor. # fit the models fit_c = lm(balance ~ 1, data = df.credit) fit_a = lm(balance ~ 1 + student, data = df.credit) # run the F test anova(fit_c, fit_a) Analysis of Variance Table Model 1: balance ~ 1 Model 2: balance ~ 1 + student Res.Df RSS Df Sum of Sq F Pr(&gt;F) 1 399 84339912 2 398 78681540 1 5658372 28.622 1.488e-07 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 fit_a %&gt;% summary() Call: lm(formula = balance ~ 1 + student, data = df.credit) Residuals: Min 1Q Median 3Q Max -876.82 -458.82 -40.87 341.88 1518.63 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 480.37 23.43 20.50 &lt; 2e-16 *** studentYes 396.46 74.10 5.35 1.49e-07 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 444.6 on 398 degrees of freedom Multiple R-squared: 0.06709, Adjusted R-squared: 0.06475 F-statistic: 28.62 on 1 and 398 DF, p-value: 1.488e-07 The summary() shows that it’s worth it: the augmented model explains a signifcant amount of the variance (i.e. it significantly reduces the proportion in error PRE). 11.5.1 Visualization of the model predictions Let’s visualize the model predictions. Here is the compact model: ggplot(df.credit, aes(x = index, y = balance)) + geom_hline(yintercept = mean(df.credit$balance), size = 1) + geom_segment(aes(xend = index, yend = mean(df.credit$balance)), alpha = 0.1) + geom_point(alpha = 0.5) It just predicts the mean (the horizontal black line). The vertical lines from each data point to the mean illustrate the residuals. And here is the augmented model: df.fit = fit_a %&gt;% tidy() %&gt;% mutate(estimate = round(estimate,2)) ggplot(df.credit, aes(x = index, y = balance, color = student)) + geom_hline(yintercept = df.fit$estimate[1], size = 1, color = &quot;#E41A1C&quot;) + geom_hline(yintercept = df.fit$estimate[1] + df.fit$estimate[2], size = 1, color = &quot;#377EB8&quot;) + geom_segment(data = df.credit %&gt;% filter(student == &quot;No&quot;), aes(xend = index, yend = df.fit$estimate[1]), alpha = 0.1, color = &quot;#E41A1C&quot;) + geom_segment(data = df.credit %&gt;% filter(student == &quot;Yes&quot;), aes(xend = index, yend = df.fit$estimate[1] + df.fit$estimate[2]), alpha = 0.1, color = &quot;#377EB8&quot;) + geom_point(alpha = 0.5) + scale_color_brewer(palette = &quot;Set1&quot;) + guides(color = guide_legend(reverse = T)) Note that this model predicts two horizontal lines. One for students, and one for non-students. Let’s make simple plot that shows the means of both groups with bootstrapped confidence intervals. ggplot(data = df.credit, mapping = aes(x = student, y = balance, fill = student)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;, show.legend = F) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1) + scale_fill_brewer(palette = &quot;Set1&quot;) And let’s double check that we also get a signifcant result when we run a t-test instead of our model comparison procedure: t.test(x = df.credit$balance[df.credit$student == &quot;No&quot;], y = df.credit$balance[df.credit$student == &quot;Yes&quot;]) Welch Two Sample t-test data: df.credit$balance[df.credit$student == &quot;No&quot;] and df.credit$balance[df.credit$student == &quot;Yes&quot;] t = -4.9028, df = 46.241, p-value = 1.205e-05 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -559.2023 -233.7088 sample estimates: mean of x mean of y 480.3694 876.8250 11.5.2 Dummy coding When we put a variable in a linear model that is coded as a character or as a factor, R automatically recodes this variable using dummy coding. It uses level 1 as the reference category for factors, or the value that comes first in the alphabet for characters. df.credit %&gt;% select(income, student) %&gt;% mutate(student_dummy = ifelse(student == &quot;No&quot;, 0, 1))%&gt;% head(10) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) income student student_dummy 14.89 No 0 106.03 Yes 1 104.59 No 0 148.92 No 0 55.88 No 0 80.18 No 0 21.00 No 0 71.41 No 0 15.12 No 0 71.06 Yes 1 11.5.3 Reporting the results To report the results, we could show a plot like this: df.plot = df.credit ggplot(df.plot, aes(x = student, y = balance)) + geom_point(alpha = 0.1, position = position_jitter(height = 0, width = 0.1)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, size = 1) And then report the means and standard deviations together with the result of our signifance test: df.credit %&gt;% group_by(student) %&gt;% summarize(mean = mean(balance), sd = sd(balance)) %&gt;% mutate(across(where(is.numeric), ~ round(., 2))) # A tibble: 2 × 3 student mean sd &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; 1 No 480. 439. 2 Yes 877. 490 11.6 One continuous and one categorical variable Now let’s take a look at a case where we have one continuous and one categorical predictor variable. Let’s first formulate and fit our models: # fit the models fit_c = lm(balance ~ 1 + income, df.credit) fit_a = lm(balance ~ 1 + income + student, df.credit) # run the F test anova(fit_c, fit_a) Analysis of Variance Table Model 1: balance ~ 1 + income Model 2: balance ~ 1 + income + student Res.Df RSS Df Sum of Sq F Pr(&gt;F) 1 398 66208745 2 397 60939054 1 5269691 34.331 9.776e-09 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 We see again that it’s worth it. The augmented model explains significantly more variance than the compact model. 11.6.1 Visualization of the model predictions Let’s visualize the model predictions again. Let’s start with the compact model: df.augment = fit_c %&gt;% augment() %&gt;% clean_names() ggplot(df.augment, aes(x = income, y = balance)) + geom_smooth(method = &quot;lm&quot;, se = F, color = &quot;black&quot;) + geom_segment(aes(xend = income, yend = fitted), alpha = 0.3) + geom_point(alpha = 0.3) `geom_smooth()` using formula &#39;y ~ x&#39; This time, the compact model still predicts just one line (like above) but note that this line is not horizontal anymore. df.tidy = fit_a %&gt;% tidy() %&gt;% mutate(estimate = round(estimate,2)) df.augment = fit_a %&gt;% augment() %&gt;% clean_names() ggplot(df.augment, aes(x = income, y = balance, group = student, color = student)) + geom_segment(data = df.augment %&gt;% filter(student == &quot;No&quot;), aes(xend = income, yend = fitted), color = &quot;#E41A1C&quot;, alpha = 0.3) + geom_segment(data = df.augment %&gt;% filter(student == &quot;Yes&quot;), aes(xend = income, yend = fitted), color = &quot;#377EB8&quot;, alpha = 0.3) + geom_abline(intercept = df.tidy$estimate[1], slope = df.tidy$estimate[2], color = &quot;#E41A1C&quot;, size = 1) + geom_abline(intercept = df.tidy$estimate[1] + df.tidy$estimate[3], slope = df.tidy$estimate[2], color = &quot;#377EB8&quot;, size = 1) + geom_point(alpha = 0.3) + scale_color_brewer(palette = &quot;Set1&quot;) + theme(legend.position = c(0.1, 0.9)) + guides(color = guide_legend(reverse = T)) The augmented model predicts two lines again, each with the same slope (but the intercept differs). 11.7 Interactions Let’s check whether there is an interaction between how income affects balance for students vs. non-students. 11.7.1 Visualization Let’s take a look at the data first. ggplot(data = df.credit, mapping = aes(x = income, y = balance, group = student, color = student)) + geom_smooth(method = &quot;lm&quot;, se = F) + geom_point(alpha = 0.3) + scale_color_brewer(palette = &quot;Set1&quot;) + theme(legend.position = c(0.1, 0.9)) + guides(color = guide_legend(reverse = T)) `geom_smooth()` using formula &#39;y ~ x&#39; Note that we just specified here that we want to have a linear model (via geom_smooth(method = \"lm\")). By default, ggplot2 assumes that we want a model that includes interactions. We can see this by the fact that two fitted lines are not parallel. But is the interaction in the model worth it? That is, does a model that includes an interaction explain significantly more variance in the data, than a model that does not have an interaction. 11.7.2 Hypothesis test Let’s check: # fit models fit_c = lm(formula = balance ~ income + student, data = df.credit) fit_a = lm(formula = balance ~ income * student, data = df.credit) # F-test anova(fit_c, fit_a) Analysis of Variance Table Model 1: balance ~ income + student Model 2: balance ~ income * student Res.Df RSS Df Sum of Sq F Pr(&gt;F) 1 397 60939054 2 396 60734545 1 204509 1.3334 0.2489 Nope, not worth it! The F-test comes out non-significant. 11.8 Additional resources 11.8.1 Datacamp Statistical modeling 1 Statistical modeling 2 Correlation and regression 11.8.2 Misc Nice review of multiple regression in R 11.9 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 tidyverse_1.3.1 [9] GGally_2.1.2 ggplot2_3.3.6 corrplot_0.92 corrr_0.4.3 [13] broom_0.8.0 janitor_2.1.0 kableExtra_1.3.4 knitr_1.39 loaded via a namespace (and not attached): [1] nlme_3.1-157 fs_1.5.2 lubridate_1.8.0 [4] bit64_4.0.5 webshot_0.5.3 RColorBrewer_1.1-3 [7] httr_1.4.3 tools_4.1.2 backports_1.4.1 [10] bslib_0.3.1 utf8_1.2.2 R6_2.5.1 [13] rpart_4.1.16 Hmisc_4.7-0 DBI_1.1.2 [16] mgcv_1.8-40 colorspace_2.0-3 nnet_7.3-17 [19] withr_2.5.0 gridExtra_2.3 tidyselect_1.1.2 [22] bit_4.0.4 compiler_4.1.2 cli_3.3.0 [25] rvest_1.0.2 htmlTable_2.4.0 TSP_1.2-0 [28] xml2_1.3.3 labeling_0.4.2 bookdown_0.26 [31] sass_0.4.1 checkmate_2.1.0 scales_1.2.0 [34] systemfonts_1.0.4 digest_0.6.29 foreign_0.8-82 [37] rmarkdown_2.14 svglite_2.1.0 jpeg_0.1-9 [40] base64enc_0.1-3 pkgconfig_2.0.3 htmltools_0.5.2 [43] dbplyr_2.1.1 fastmap_1.1.0 highr_0.9 [46] htmlwidgets_1.5.4 rlang_1.0.2 readxl_1.4.0 [49] rstudioapi_0.13 jquerylib_0.1.4 farver_2.1.0 [52] generics_0.1.2 jsonlite_1.8.0 vroom_1.5.7 [55] magrittr_2.0.3 Formula_1.2-4 Matrix_1.4-1 [58] Rcpp_1.0.8.3 munsell_0.5.0 fansi_1.0.3 [61] lifecycle_1.0.1 stringi_1.7.6 yaml_2.3.5 [64] snakecase_0.11.0 plyr_1.8.7 grid_4.1.2 [67] parallel_4.1.2 crayon_1.5.1 lattice_0.20-45 [70] haven_2.5.0 splines_4.1.2 hms_1.1.1 [73] pillar_1.7.0 codetools_0.2-18 reprex_2.0.1 [76] glue_1.6.2 evaluate_0.15 latticeExtra_0.6-29 [79] data.table_1.14.2 modelr_0.1.8 png_0.1-7 [82] vctrs_0.4.1 tzdb_0.3.0 foreach_1.5.2 [85] cellranger_1.1.0 gtable_0.3.0 reshape_0.8.9 [88] assertthat_0.2.1 xfun_0.30 survival_3.3-1 [91] viridisLite_0.4.0 seriation_1.3.5 iterators_1.0.14 [94] registry_0.5-1 cluster_2.1.3 ellipsis_0.3.2 11.10 References "],["linear-model-3.html", "Chapter 12 Linear model 3 12.1 Learning goals 12.2 Load packages and set plotting theme 12.3 Load data sets 12.4 One-way ANOVA 12.5 Two-way ANOVA 12.6 Two-way ANOVA (with interaction) 12.7 Additional resources 12.8 Session info", " Chapter 12 Linear model 3 12.1 Learning goals Linear model with one multi-level categorical predictor (One-way ANOVA). Linear model with multiple categorical predictors (N-way ANOVA). 12.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;broom&quot;) # for tidying up linear models library(&quot;car&quot;) # for running ANOVAs library(&quot;afex&quot;) # also for running ANOVAs library(&quot;emmeans&quot;) # for calculating constrasts library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size # these options here change the formatting of how comments are rendered opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) # suppress grouping warnings options(dplyr.summarise.inform = F) 12.3 Load data sets df.poker = read_csv(&quot;data/poker.csv&quot;) %&gt;% mutate(skill = factor(skill, levels = 1:2, labels = c(&quot;expert&quot;, &quot;average&quot;)), skill = fct_relevel(skill, &quot;average&quot;, &quot;expert&quot;), hand = factor(hand, levels = 1:3, labels = c(&quot;bad&quot;, &quot;neutral&quot;, &quot;good&quot;)), limit = factor(limit, levels = 1:2, labels = c(&quot;fixed&quot;, &quot;none&quot;)), participant = 1:n()) %&gt;% select(participant, everything()) Selection of the data: df.poker %&gt;% group_by(skill, hand, limit) %&gt;% filter(row_number() &lt; 3) %&gt;% head(10) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) participant skill hand limit balance 1 expert bad fixed 4.00 2 expert bad fixed 5.55 26 expert bad none 5.52 27 expert bad none 8.28 51 expert neutral fixed 11.74 52 expert neutral fixed 10.04 76 expert neutral none 21.55 77 expert neutral none 3.12 101 expert good fixed 10.86 102 expert good fixed 8.68 12.4 One-way ANOVA 12.4.1 Visualization df.poker %&gt;% ggplot(mapping = aes(x = hand, y = balance, fill = hand)) + geom_point(alpha = 0.2, position = position_jitter(height = 0, width = 0.1)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;, shape = 21, size = 4) + labs(y = &quot;final balance (in Euros)&quot;) + scale_fill_manual(values = c(&quot;red&quot;, &quot;orange&quot;, &quot;green&quot;)) + theme(legend.position = &quot;none&quot;) 12.4.2 Model fitting We pass the result of the lm() function to anova() to calculate an analysis of variance like so: lm(formula = balance ~ hand, data = df.poker) %&gt;% anova() Analysis of Variance Table Response: balance Df Sum Sq Mean Sq F value Pr(&gt;F) hand 2 2559.4 1279.7 75.703 &lt; 2.2e-16 *** Residuals 297 5020.6 16.9 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 12.4.3 Hypothesis test The F-test reported by the ANOVA compares the fitted model with a compact model that only predicts the grand mean: # fit the models fit_c = lm(formula = balance ~ 1, data = df.poker) fit_a = lm(formula = balance ~ hand, data = df.poker) # compare via F-test anova(fit_c, fit_a) Analysis of Variance Table Model 1: balance ~ 1 Model 2: balance ~ hand Res.Df RSS Df Sum of Sq F Pr(&gt;F) 1 299 7580.0 2 297 5020.6 2 2559.4 75.703 &lt; 2.2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 12.4.4 Visualize the model’s predictions Here is the model prediction of the compact model: set.seed(1) df.plot = df.poker %&gt;% mutate(hand_jitter = 1 + runif(n(), min = -0.25, max = 0.25)) df.augment = fit_c %&gt;% augment() %&gt;% clean_names() %&gt;% bind_cols(df.plot %&gt;% select(hand, hand_jitter)) ggplot(data = df.plot, mapping = aes(x = hand_jitter, y = balance, fill = hand)) + geom_hline(yintercept = mean(df.poker$balance)) + geom_point(alpha = 0.5) + geom_segment(data = df.augment, mapping = aes(xend = hand_jitter, yend = fitted), alpha = 0.2) + labs(y = &quot;balance&quot;) + theme(legend.position = &quot;none&quot;, axis.text.x = element_blank(), axis.title.x = element_blank()) Note that since we have a categorical variable here, we don’t really have a continuous x-axis. I’ve just jittered the values so it’s easier to show the residuals. And here is the prediction of the augmented model (which predicts different means for each group). set.seed(1) df.plot = df.poker %&gt;% mutate(hand_jitter = hand %&gt;% as.numeric(), hand_jitter = hand_jitter + runif(n(), min = -0.4, max = 0.4)) df.tidy = fit_a %&gt;% tidy() %&gt;% select(where(is.numeric)) %&gt;% mutate(across(.fns = ~ round(., digits = 2))) df.augment = fit_a %&gt;% augment() %&gt;% clean_names() %&gt;% bind_cols(df.plot %&gt;% select(hand_jitter)) ggplot(data = df.plot, mapping = aes(x = hand_jitter, y = balance, color = hand)) + geom_point(alpha = 0.8) + geom_segment(data = NULL, mapping = aes(x = 0.6, xend = 1.4, y = df.tidy$estimate[1], yend = df.tidy$estimate[1]), color = &quot;red&quot;, size = 1) + geom_segment(data = NULL, aes(x = 1.6, xend = 2.4, y = df.tidy$estimate[1] + df.tidy$estimate[2], yend = df.tidy$estimate[1] + df.tidy$estimate[2]), color = &quot;orange&quot;, size = 1) + geom_segment(data = NULL, aes(x = 2.6, xend = 3.4, y = df.tidy$estimate[1] + df.tidy$estimate[3], yend = df.tidy$estimate[1] + df.tidy$estimate[3]), color = &quot;green&quot;, size = 1) + geom_segment(data = df.augment, aes(xend = hand_jitter, y = balance, yend = fitted), alpha = 0.3) + labs(y = &quot;balance&quot;) + scale_color_manual(values = c(&quot;red&quot;, &quot;orange&quot;, &quot;green&quot;)) + scale_x_continuous(breaks = 1:3, labels = c(&quot;bad&quot;, &quot;neutral&quot;, &quot;good&quot;)) + theme(legend.position = &quot;none&quot;, axis.title.x = element_blank()) The vertical lines illustrate the residual sum of squares. We can illustrate the model sum of squares like so: set.seed(1) df.plot = df.poker %&gt;% mutate(hand_jitter = hand %&gt;% as.numeric(), hand_jitter = hand_jitter + runif(n(), min = -0.4, max = 0.4)) %&gt;% group_by(hand) %&gt;% mutate(mean_group = mean(balance)) %&gt;% ungroup() %&gt;% mutate(mean_grand = mean(balance)) df.means = df.poker %&gt;% group_by(hand) %&gt;% summarize(mean = mean(balance)) %&gt;% pivot_wider(names_from = hand, values_from = mean) ggplot(data = df.plot, mapping = aes(x = hand_jitter, y = mean_group, color = hand)) + geom_point(alpha = 0.8) + geom_segment(data = NULL, mapping = aes(x = 0.6, xend = 1.4, y = df.means$bad, yend = df.means$bad), color = &quot;red&quot;, size = 1) + geom_segment(data = NULL, mapping = aes(x = 1.6, xend = 2.4, y = df.means$neutral, yend = df.means$neutral), color = &quot;orange&quot;, size = 1) + geom_segment(data = NULL, mapping = aes(x = 2.6, xend = 3.4, y = df.means$good, yend = df.means$good), color = &quot;green&quot;, size = 1) + geom_segment(mapping = aes(xend = hand_jitter, y = mean_group, yend = mean_grand), alpha = 0.3) + geom_hline(yintercept = mean(df.poker$balance), size = 1) + labs(y = &quot;balance&quot;) + scale_color_manual(values = c(&quot;red&quot;, &quot;orange&quot;, &quot;green&quot;)) + scale_x_continuous(breaks = 1:3, labels = c(&quot;bad&quot;, &quot;neutral&quot;, &quot;good&quot;)) + scale_y_continuous(breaks = c(0, 10, 20), labels = c(0, 10, 20), limits = c(0, 25)) + theme(legend.position = &quot;none&quot;, axis.title.x = element_blank()) This captures the variance in the data that is accounted for by the hand variable. Just for kicks, let’s calculate our cherished proportion of reduction in error PRE: df.c = fit_c %&gt;% augment() %&gt;% clean_names() %&gt;% summarize(sse = sum(resid^2) %&gt;% round) df.a = fit_a %&gt;% augment() %&gt;% clean_names() %&gt;% summarize(sse = sum(resid^2) %&gt;% round) pre = 1 - df.a$sse/df.c$sse print(pre %&gt;% round(2)) [1] 0.34 Note that this is the same as the \\(R^2\\) for the augmented model: fit_a %&gt;% summary() Call: lm(formula = balance ~ hand, data = df.poker) Residuals: Min 1Q Median 3Q Max -12.9264 -2.5902 -0.0115 2.6573 15.2834 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 5.9415 0.4111 14.451 &lt; 2e-16 *** handneutral 4.4051 0.5815 7.576 4.55e-13 *** handgood 7.0849 0.5815 12.185 &lt; 2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 4.111 on 297 degrees of freedom Multiple R-squared: 0.3377, Adjusted R-squared: 0.3332 F-statistic: 75.7 on 2 and 297 DF, p-value: &lt; 2.2e-16 12.4.5 Dummy coding Let’s check that we understand how dummy-coding works for a variable with more than 2 levels: # dummy code the hand variable df.poker = df.poker %&gt;% mutate(hand_neutral = ifelse(hand == &quot;neutral&quot;, 1, 0), hand_good = ifelse(hand == &quot;good&quot;, 1, 0)) # show the dummy coded variables df.poker %&gt;% select(participant, contains(&quot;hand&quot;), balance) %&gt;% group_by(hand) %&gt;% top_n(3) %&gt;% head(10) %&gt;% kable(digits = 3) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) Selecting by balance participant hand hand_neutral hand_good balance 31 bad 0 0 12.22 46 bad 0 0 12.06 50 bad 0 0 16.68 76 neutral 1 0 21.55 87 neutral 1 0 20.89 89 neutral 1 0 25.63 127 good 0 1 26.99 129 good 0 1 21.36 283 good 0 1 22.48 # fit the model fit.tmp = lm(balance ~ 1 + hand_neutral + hand_good, df.poker) # show the model summary fit.tmp %&gt;% summary() Call: lm(formula = balance ~ 1 + hand_neutral + hand_good, data = df.poker) Residuals: Min 1Q Median 3Q Max -12.9264 -2.5902 -0.0115 2.6573 15.2834 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 5.9415 0.4111 14.451 &lt; 2e-16 *** hand_neutral 4.4051 0.5815 7.576 4.55e-13 *** hand_good 7.0849 0.5815 12.185 &lt; 2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 4.111 on 297 degrees of freedom Multiple R-squared: 0.3377, Adjusted R-squared: 0.3332 F-statistic: 75.7 on 2 and 297 DF, p-value: &lt; 2.2e-16 Here, I’ve directly put the dummy-coded variables as predictors into the lm(). We get the same model as if we used the hand variable instead. 12.4.6 Follow up questions Here are some follow up questions we may ask about the data. Are bad hands different from neutral hands? df.poker %&gt;% filter(hand %in% c(&quot;bad&quot;, &quot;neutral&quot;)) %&gt;% lm(formula = balance ~ hand, data = .) %&gt;% summary() Call: lm(formula = balance ~ hand, data = .) Residuals: Min 1Q Median 3Q Max -9.9566 -2.5078 -0.2365 2.4410 15.2834 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 5.9415 0.3816 15.570 &lt; 2e-16 *** handneutral 4.4051 0.5397 8.163 3.76e-14 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 3.816 on 198 degrees of freedom Multiple R-squared: 0.2518, Adjusted R-squared: 0.248 F-statistic: 66.63 on 1 and 198 DF, p-value: 3.758e-14 Are neutral hands different from good hands? df.poker %&gt;% filter(hand %in% c(&quot;neutral&quot;, &quot;good&quot;)) %&gt;% lm(formula = balance ~ hand, data = .) %&gt;% summary() Call: lm(formula = balance ~ hand, data = .) Residuals: Min 1Q Median 3Q Max -12.9264 -2.7141 0.2585 2.7184 15.2834 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 10.3466 0.4448 23.26 &lt; 2e-16 *** handgood 2.6798 0.6291 4.26 3.16e-05 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 4.448 on 198 degrees of freedom Multiple R-squared: 0.08396, Adjusted R-squared: 0.07933 F-statistic: 18.15 on 1 and 198 DF, p-value: 3.158e-05 Doing the same thing by recoding our hand factor and taking “neutral” to be the reference category: df.poker %&gt;% mutate(hand = fct_relevel(hand, &quot;neutral&quot;)) %&gt;% lm(formula = balance ~ hand, data = .) %&gt;% summary() Call: lm(formula = balance ~ hand, data = .) Residuals: Min 1Q Median 3Q Max -12.9264 -2.5902 -0.0115 2.6573 15.2834 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 10.3466 0.4111 25.165 &lt; 2e-16 *** handbad -4.4051 0.5815 -7.576 4.55e-13 *** handgood 2.6798 0.5815 4.609 6.02e-06 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 4.111 on 297 degrees of freedom Multiple R-squared: 0.3377, Adjusted R-squared: 0.3332 F-statistic: 75.7 on 2 and 297 DF, p-value: &lt; 2.2e-16 12.4.7 Variance decomposition Let’s first run the model fit = lm(formula = balance ~ hand, data = df.poker) fit %&gt;% anova() Analysis of Variance Table Response: balance Df Sum Sq Mean Sq F value Pr(&gt;F) hand 2 2559.4 1279.7 75.703 &lt; 2.2e-16 *** Residuals 297 5020.6 16.9 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 12.4.7.1 Calculate sums of squares And then let’s make sure that we understand how the variance is broken down: df.poker %&gt;% mutate(mean_grand = mean(balance)) %&gt;% group_by(hand) %&gt;% mutate(mean_group = mean(balance)) %&gt;% ungroup() %&gt;% summarize(variance_total = sum((balance - mean_grand)^2), variance_model = sum((mean_group - mean_grand)^2), variance_residual = variance_total - variance_model) # A tibble: 1 × 3 variance_total variance_model variance_residual &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 7580. 2559. 5021. 12.4.7.2 Visualize model predictions 12.4.7.2.1 Total variance set.seed(1) fit_c = lm(formula = balance ~ 1, data = df.poker) df.plot = df.poker %&gt;% mutate(hand_jitter = 1 + runif(n(), min = -0.25, max = 0.25)) df.augment = fit_c %&gt;% augment() %&gt;% clean_names() %&gt;% bind_cols(df.plot %&gt;% select(hand, hand_jitter)) ggplot(data = df.plot, mapping = aes(x = hand_jitter, y = balance, fill = hand)) + geom_hline(yintercept = mean(df.poker$balance)) + geom_point(alpha = 0.5) + geom_segment(data = df.augment, aes(xend = hand_jitter, yend = fitted), alpha = 0.2) + labs(y = &quot;balance&quot;) + theme(legend.position = &quot;none&quot;, axis.text.x = element_blank(), axis.title.x = element_blank()) 12.4.7.2.2 Model variance set.seed(1) df.plot = df.poker %&gt;% mutate(hand_jitter = hand %&gt;% as.numeric(), hand_jitter = hand_jitter + runif(n(), min = -0.4, max = 0.4)) %&gt;% group_by(hand) %&gt;% mutate(mean_group = mean(balance)) %&gt;% ungroup() %&gt;% mutate(mean_grand = mean(balance)) df.means = df.poker %&gt;% group_by(hand) %&gt;% summarize(mean = mean(balance)) %&gt;% pivot_wider(names_from = hand, values_from = mean) ggplot(data = df.plot, mapping = aes(x = hand_jitter, y = mean_group, color = hand)) + geom_point(alpha = 0.8) + geom_segment(data = NULL, aes(x = 0.6, xend = 1.4, y = df.means$bad, yend = df.means$bad), color = &quot;red&quot;, size = 1) + geom_segment(data = NULL, aes(x = 1.6, xend = 2.4, y = df.means$neutral, yend = df.means$neutral), color = &quot;orange&quot;, size = 1) + geom_segment(data = NULL, aes(x = 2.6, xend = 3.4, y = df.means$good, yend = df.means$good), color = &quot;green&quot;, size = 1) + geom_segment(aes(xend = hand_jitter, y = mean_group, yend = mean_grand), alpha = 0.3) + geom_hline(yintercept = mean(df.poker$balance), size = 1) + labs(y = &quot;balance&quot;) + scale_color_manual(values = c(&quot;red&quot;, &quot;orange&quot;, &quot;green&quot;)) + scale_x_continuous(breaks = 1:3, labels = c(&quot;bad&quot;, &quot;neutral&quot;, &quot;good&quot;)) + scale_y_continuous(breaks = c(0, 10, 20), labels = c(0, 10, 20), limits = c(0, 25)) + theme(legend.position = &quot;none&quot;, axis.title.x = element_blank()) 12.4.7.2.3 Residual variance set.seed(1) fit_a = lm(formula = balance ~ hand, data = df.poker) df.plot = df.poker %&gt;% mutate(hand_jitter = hand %&gt;% as.numeric(), hand_jitter = hand_jitter + runif(n(), min = -0.4, max = 0.4)) df.tidy = fit_a %&gt;% tidy() %&gt;% select(where(is.numeric)) %&gt;% mutate(across(.fns = ~ round(., digits = 2))) df.augment = fit_a %&gt;% augment() %&gt;% clean_names() %&gt;% bind_cols(df.plot %&gt;% select(hand_jitter)) ggplot(data = df.plot, mapping = aes(x = hand_jitter, y = balance, color = hand)) + geom_point(alpha = 0.8) + geom_segment(data = NULL, aes(x = 0.6, xend = 1.4, y = df.tidy$estimate[1], yend = df.tidy$estimate[1]), color = &quot;red&quot;, size = 1) + geom_segment(data = NULL, aes(x = 1.6, xend = 2.4, y = df.tidy$estimate[1] + df.tidy$estimate[2], yend = df.tidy$estimate[1] + df.tidy$estimate[2]), color = &quot;orange&quot;, size = 1) + geom_segment(data = NULL, aes(x = 2.6, xend = 3.4, y = df.tidy$estimate[1] + df.tidy$estimate[3], yend = df.tidy$estimate[1] + df.tidy$estimate[3]), color = &quot;green&quot;, size = 1) + geom_segment(data = df.augment, aes(xend = hand_jitter, y = balance, yend = fitted), alpha = 0.3) + labs(y = &quot;balance&quot;) + scale_color_manual(values = c(&quot;red&quot;, &quot;orange&quot;, &quot;green&quot;)) + scale_x_continuous(breaks = 1:3, labels = c(&quot;bad&quot;, &quot;neutral&quot;, &quot;good&quot;)) + theme(legend.position = &quot;none&quot;, axis.title.x = element_blank()) 12.5 Two-way ANOVA Now let’s take a look at a case where we have multiple categorical predictors. 12.5.1 Visualization Let’s look at the overall effect of skill: ggplot(data = df.poker, mapping = aes(x = skill, y = balance)) + geom_point(position = position_jitter(width = 0.2, height = 0), alpha = 0.2) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, color = &quot;black&quot;, position = position_dodge(0.9)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;, color = &quot;black&quot;, position = position_dodge(0.9), aes(shape = skill), size = 3, fill = &quot;black&quot;) + scale_shape_manual(values = c(21, 22)) + guides(shape = F) Warning: `guides(&lt;scale&gt; = FALSE)` is deprecated. Please use `guides(&lt;scale&gt; = &quot;none&quot;)` instead. And now let’s take a look at the means for the full the 3 (hand) x 2 (skill) design: ggplot(data = df.poker, mapping = aes(x = hand, y = balance, group = skill, fill = hand)) + geom_point(position = position_jitterdodge(jitter.width = 0.3, jitter.height = 0, dodge.width = 0.9), alpha = 0.2) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, color = &quot;black&quot;, position = position_dodge(0.9)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;, aes(shape = skill), color = &quot;black&quot;, position = position_dodge(0.9), size = 3) + scale_fill_manual(values = c(&quot;red&quot;, &quot;orange&quot;, &quot;green&quot;)) + scale_shape_manual(values = c(21, 22)) + guides(fill = F) Warning: `guides(&lt;scale&gt; = FALSE)` is deprecated. Please use `guides(&lt;scale&gt; = &quot;none&quot;)` instead. 12.5.2 Model fitting For N-way ANOVAs, we need to be careful about what sums of squares we are using. The standard (based on the SPSS output) is to use type III sums of squares. We set this up in the following way: lm(formula = balance ~ hand * skill, data = df.poker, contrasts = list(hand = &quot;contr.sum&quot;, skill = &quot;contr.sum&quot;)) %&gt;% Anova(type = 3) Anova Table (Type III tests) Response: balance Sum Sq Df F value Pr(&gt;F) (Intercept) 28644.7 1 1772.1137 &lt; 2.2e-16 *** hand 2559.4 2 79.1692 &lt; 2.2e-16 *** skill 39.3 1 2.4344 0.1197776 hand:skill 229.0 2 7.0830 0.0009901 *** Residuals 4752.3 294 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 So, we fit our linear model, but set the contrasts to “contr.sum” (which yields effect coding instead of dummy coding), and then specify the desired type of sums of squares in the Anova() function call. Alternatively, we could use the afex package and specify the ANOVA like so: aov_ez(id = &quot;participant&quot;, dv = &quot;balance&quot;, data = df.poker, between = c(&quot;hand&quot;, &quot;skill&quot;) ) Contrasts set to contr.sum for the following variables: hand, skill Anova Table (Type 3 tests) Response: balance Effect df MSE F ges p.value 1 hand 2, 294 16.16 79.17 *** .350 &lt;.001 2 skill 1, 294 16.16 2.43 .008 .120 3 hand:skill 2, 294 16.16 7.08 *** .046 &lt;.001 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;+&#39; 0.1 &#39; &#39; 1 The afex package uses effect coding and type 3 sums of squares by default. 12.5.3 Interpreting interactions Code I’ve used to generate the different plots in the competition: set.seed(1) b0 = 15 nsamples = 30 sd = 5 # simple effect of condition b1 = 10 b2 = 1 b1_2 = 1 # two simple effects # b1 = 5 # b2 = -5 # b1_2 = 0 # interaction effect # b1 = 10 # b2 = 10 # b1_2 = -20 # interaction and simple effect # b1 = 10 # b2 = 0 # b1_2 = -20 # all three # b1 = 2 # b2 = 2 # b1_2 = 10 df.data = tibble( condition = rep(c(0, 1), each = nsamples), treatment = rep(c(0, 1), nsamples), rating = b0 + b1 * condition + b2 * treatment + (b1_2 * condition * treatment) + rnorm(nsamples, sd = sd)) %&gt;% mutate(condition = factor(condition, labels = c(&quot;A&quot;, &quot;B&quot;)), treatment = factor(treatment, labels = c(&quot;1&quot;, &quot;2&quot;))) ggplot(df.data, aes(x = condition, y = rating, group = treatment, fill = treatment)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;, position = position_dodge(0.9)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1, position = position_dodge(0.9)) + scale_fill_brewer(palette = &quot;Set1&quot;) And here is one specific example. Let’s generate the data first: # make example reproducible set.seed(1) # set parameters nsamples = 30 b0 = 15 b1 = 10 # simple effect of condition b2 = 0 # simple effect of treatment b1_2 = -20 # interaction effect sd = 5 # generate data df.data = tibble( condition = rep(c(0, 1), each = nsamples), treatment = rep(c(0, 1), nsamples), rating = b0 + b1 * condition + b2 * treatment + (b1_2 * condition * treatment) + rnorm(nsamples, sd = sd)) %&gt;% mutate(condition = factor(condition, labels = c(&quot;A&quot;, &quot;B&quot;)), treatment = factor(treatment, labels = c(&quot;1&quot;, &quot;2&quot;))) Show part of the generated data frame: # show data frame df.data %&gt;% group_by(condition, treatment) %&gt;% filter(row_number() &lt; 3) %&gt;% ungroup() %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) condition treatment rating A 1 11.87 A 2 15.92 A 1 10.82 A 2 22.98 B 1 21.87 B 2 5.92 B 1 20.82 B 2 12.98 Plot the data: # plot data ggplot(df.data, aes(x = condition, y = rating, group = treatment, fill = treatment)) + stat_summary(fun = &quot;mean&quot;, geom = &quot;bar&quot;, color = &quot;black&quot;, position = position_dodge(0.9)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, geom = &quot;linerange&quot;, size = 1, position = position_dodge(0.9)) + scale_fill_brewer(palette = &quot;Set1&quot;) And check whether we can successfully infer the parameters that we used to generate the data: # infer parameters lm(formula = rating ~ 1 + condition + treatment + condition:treatment, data = df.data) %&gt;% summary() Call: lm(formula = rating ~ 1 + condition + treatment + condition:treatment, data = df.data) Residuals: Min 1Q Median 3Q Max -10.6546 -3.6343 0.7988 3.3514 8.3953 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 16.244 1.194 13.608 &lt; 2e-16 *** conditionB 10.000 1.688 5.924 2.02e-07 *** treatment2 -1.662 1.688 -0.985 0.329 conditionB:treatment2 -20.000 2.387 -8.378 1.86e-11 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 4.623 on 56 degrees of freedom Multiple R-squared: 0.7473, Adjusted R-squared: 0.7338 F-statistic: 55.21 on 3 and 56 DF, p-value: &lt; 2.2e-16 12.5.4 Variance decomposition Let’s fit the model first: fit = lm(formula = balance ~ hand * skill, data = df.poker) fit %&gt;% anova() Analysis of Variance Table Response: balance Df Sum Sq Mean Sq F value Pr(&gt;F) hand 2 2559.4 1279.70 79.1692 &lt; 2.2e-16 *** skill 1 39.3 39.35 2.4344 0.1197776 hand:skill 2 229.0 114.49 7.0830 0.0009901 *** Residuals 294 4752.3 16.16 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 12.5.4.1 Calculate sums of squares df.poker %&gt;% mutate(mean_grand = mean(balance)) %&gt;% group_by(skill) %&gt;% mutate(mean_skill = mean(balance)) %&gt;% group_by(hand) %&gt;% mutate(mean_hand = mean(balance)) %&gt;% ungroup() %&gt;% summarize(variance_total = sum((balance - mean_grand)^2), variance_skill = sum((mean_skill - mean_grand)^2), variance_hand = sum((mean_hand - mean_grand)^2), variance_residual = variance_total - variance_skill - variance_hand) # A tibble: 1 × 4 variance_total variance_skill variance_hand variance_residual &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 7580. 39.3 2559. 4981. 12.5.4.2 Visualize model predictions 12.5.4.2.1 Skill factor set.seed(1) df.plot = df.poker %&gt;% mutate(skill_jitter = skill %&gt;% as.numeric(), skill_jitter = skill_jitter + runif(n(), min = -0.4, max = 0.4)) %&gt;% group_by(skill) %&gt;% mutate(mean_group = mean(balance)) %&gt;% ungroup() %&gt;% mutate(mean_grand = mean(balance)) df.means = df.poker %&gt;% group_by(skill) %&gt;% summarize(mean = mean(balance)) %&gt;% pivot_wider(names_from = skill, values_from = mean) ggplot(data = df.plot, mapping = aes(x = skill_jitter, y = mean_group, color = skill)) + geom_point(alpha = 0.8) + geom_segment(data = NULL, aes(x = 0.6, xend = 1.4, y = df.means$average, yend = df.means$average), color = &quot;black&quot;, size = 1) + geom_segment(data = NULL, aes(x = 1.6, xend = 2.4, y = df.means$expert, yend = df.means$expert), color = &quot;gray50&quot;, size = 1) + geom_segment(aes(xend = skill_jitter, y = mean_group, yend = mean_grand), alpha = 0.3) + geom_hline(yintercept = mean(df.poker$balance), size = 1) + labs(y = &quot;balance&quot;) + scale_color_manual(values = c(&quot;black&quot;, &quot;gray50&quot;)) + scale_x_continuous(breaks = 1:2, labels = c(&quot;average&quot;, &quot;expert&quot;)) + scale_y_continuous(breaks = c(0, 10, 20), labels = c(0, 10, 20), limits = c(0, 25)) + theme(legend.position = &quot;none&quot;, axis.title.x = element_blank()) 12.6 Two-way ANOVA (with interaction) Let’s fit a two-way ANOVA with the interaction term. fit = lm(formula = balance ~ hand * skill, data = df.poker) fit %&gt;% anova() Analysis of Variance Table Response: balance Df Sum Sq Mean Sq F value Pr(&gt;F) hand 2 2559.4 1279.70 79.1692 &lt; 2.2e-16 *** skill 1 39.3 39.35 2.4344 0.1197776 hand:skill 2 229.0 114.49 7.0830 0.0009901 *** Residuals 294 4752.3 16.16 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 And let’s compute how the the sums of squares are decomposed: df.poker %&gt;% mutate(mean_grand = mean(balance)) %&gt;% group_by(skill) %&gt;% mutate(mean_skill = mean(balance)) %&gt;% group_by(hand) %&gt;% mutate(mean_hand = mean(balance)) %&gt;% group_by(hand, skill) %&gt;% mutate(mean_hand_skill = mean(balance)) %&gt;% ungroup() %&gt;% summarize(variance_total = sum((balance - mean_grand)^2), variance_skill = sum((mean_skill - mean_grand)^2), variance_hand = sum((mean_hand - mean_grand)^2), variance_hand_skill = sum((mean_hand_skill - mean_skill - mean_hand + mean_grand)^2), variance_residual = variance_total - variance_skill - variance_hand - variance_hand_skill) # A tibble: 1 × 5 variance_total variance_skill variance_hand variance_hand_sk… variance_residu… &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 7580. 39.3 2559. 229. 4752. 12.7 Additional resources 12.7.1 Datacamp Statistical modeling 1 Statistical modeling 2 Correlation and regression 12.7.2 Misc Explanation of different types of sums of squares 12.8 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 emmeans_1.7.3 afex_1.1-1 lme4_1.1-29 [13] Matrix_1.4-1 car_3.0-13 carData_3.0-5 broom_0.8.0 [17] janitor_2.1.0 kableExtra_1.3.4 knitr_1.39 loaded via a namespace (and not attached): [1] TH.data_1.1-1 minqa_1.2.4 colorspace_2.0-3 [4] ellipsis_0.3.2 htmlTable_2.4.0 estimability_1.3 [7] snakecase_0.11.0 base64enc_0.1-3 fs_1.5.2 [10] rstudioapi_0.13 farver_2.1.0 bit64_4.0.5 [13] fansi_1.0.3 mvtnorm_1.1-3 lubridate_1.8.0 [16] xml2_1.3.3 codetools_0.2-18 splines_4.1.2 [19] Formula_1.2-4 jsonlite_1.8.0 nloptr_2.0.0 [22] cluster_2.1.3 dbplyr_2.1.1 png_0.1-7 [25] compiler_4.1.2 httr_1.4.3 backports_1.4.1 [28] assertthat_0.2.1 fastmap_1.1.0 cli_3.3.0 [31] htmltools_0.5.2 tools_4.1.2 lmerTest_3.1-3 [34] coda_0.19-4 gtable_0.3.0 glue_1.6.2 [37] reshape2_1.4.4 Rcpp_1.0.8.3 cellranger_1.1.0 [40] jquerylib_0.1.4 vctrs_0.4.1 svglite_2.1.0 [43] nlme_3.1-157 xfun_0.30 rvest_1.0.2 [46] lifecycle_1.0.1 MASS_7.3-57 zoo_1.8-10 [49] scales_1.2.0 vroom_1.5.7 hms_1.1.1 [52] parallel_4.1.2 sandwich_3.0-1 RColorBrewer_1.1-3 [55] yaml_2.3.5 gridExtra_2.3 sass_0.4.1 [58] rpart_4.1.16 latticeExtra_0.6-29 stringi_1.7.6 [61] highr_0.9 checkmate_2.1.0 boot_1.3-28 [64] rlang_1.0.2 pkgconfig_2.0.3 systemfonts_1.0.4 [67] evaluate_0.15 lattice_0.20-45 labeling_0.4.2 [70] htmlwidgets_1.5.4 bit_4.0.4 tidyselect_1.1.2 [73] plyr_1.8.7 magrittr_2.0.3 bookdown_0.26 [76] R6_2.5.1 generics_0.1.2 Hmisc_4.7-0 [79] multcomp_1.4-19 DBI_1.1.2 foreign_0.8-82 [82] pillar_1.7.0 haven_2.5.0 withr_2.5.0 [85] nnet_7.3-17 survival_3.3-1 abind_1.4-5 [88] modelr_0.1.8 crayon_1.5.1 utf8_1.2.2 [91] tzdb_0.3.0 rmarkdown_2.14 jpeg_0.1-9 [94] grid_4.1.2 readxl_1.4.0 data.table_1.14.2 [97] reprex_2.0.1 digest_0.6.29 webshot_0.5.3 [100] xtable_1.8-4 numDeriv_2016.8-1.1 munsell_0.5.0 [103] viridisLite_0.4.0 bslib_0.3.1 "],["linear-model-4.html", "Chapter 13 Linear model 4 13.1 Load packages and set plotting theme 13.2 Load data sets 13.3 ANOVA with unbalanced design 13.4 Interpreting parameters (very important!) 13.5 Linear contrasts 13.6 Session info", " Chapter 13 Linear model 4 13.1 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;broom&quot;) # for tidying up linear models library(&quot;afex&quot;) # for running ANOVAs library(&quot;emmeans&quot;) # for calculating contrasts library(&quot;car&quot;) # for calculating ANOVAs library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set( theme_classic() + #set the theme theme(text = element_text(size = 20)) #set the default text size ) # these options here change the formatting of how comments are rendered opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 13.2 Load data sets Read in the data: df.poker = read_csv(&quot;data/poker.csv&quot;) %&gt;% mutate(skill = factor(skill, levels = 1:2, labels = c(&quot;expert&quot;, &quot;average&quot;)), skill = fct_relevel(skill, &quot;average&quot;, &quot;expert&quot;), hand = factor(hand, levels = 1:3, labels = c(&quot;bad&quot;, &quot;neutral&quot;, &quot;good&quot;)), limit = factor(limit, levels = 1:2, labels = c(&quot;fixed&quot;, &quot;none&quot;)), participant = 1:n()) %&gt;% select(participant, everything()) # creating an unbalanced data set by removing the first 10 participants df.poker.unbalanced = df.poker %&gt;% filter(!participant %in% 1:10) 13.3 ANOVA with unbalanced design For the standard anova() function, the order of the independent predictors matters when the design is unbalanced. There are two reasons for why this happens. In an unbalanced design, the predictors in the model aren’t uncorrelated anymore. The standard anova() function computes Type I (sequential) sums of squares. Sequential sums of squares means that the predictors are added to the model in the order in which the are specified. # one order lm(formula = balance ~ skill + hand, data = df.poker.unbalanced) %&gt;% anova() Analysis of Variance Table Response: balance Df Sum Sq Mean Sq F value Pr(&gt;F) skill 1 74.3 74.28 4.2904 0.03922 * hand 2 2385.1 1192.57 68.8827 &lt; 2e-16 *** Residuals 286 4951.5 17.31 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 # another order lm(formula = balance ~ hand + skill, data = df.poker.unbalanced) %&gt;% anova() Analysis of Variance Table Response: balance Df Sum Sq Mean Sq F value Pr(&gt;F) hand 2 2419.8 1209.92 69.8845 &lt;2e-16 *** skill 1 39.6 39.59 2.2867 0.1316 Residuals 286 4951.5 17.31 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 We should compute an ANOVA with type 3 sums of squares, and set the contrast to sum contrasts. I like to use the joint_tests() function from the “emmeans” package for doing so. It does both of these things for us. # one order lm(formula = balance ~ hand * skill, data = df.poker.unbalanced) %&gt;% joint_tests() model term df1 df2 F.ratio p.value hand 2 284 68.973 &lt;.0001 skill 1 284 2.954 0.0868 hand:skill 2 284 7.440 0.0007 # another order lm(formula = balance ~ skill + hand, data = df.poker.unbalanced) %&gt;% joint_tests() model term df1 df2 F.ratio p.value skill 1 286 2.287 0.1316 hand 2 286 68.883 &lt;.0001 Now, the order of the independent variables doesn’t matter anymore. Alternatively,we can also use the aov_ez() function from the afex package. lm(formula = balance ~ skill * hand, data = df.poker.unbalanced) %&gt;% joint_tests() model term df1 df2 F.ratio p.value skill 1 284 2.954 0.0868 hand 2 284 68.973 &lt;.0001 skill:hand 2 284 7.440 0.0007 fit = aov_ez(id = &quot;participant&quot;, dv = &quot;balance&quot;, data = df.poker.unbalanced, between = c(&quot;hand&quot;, &quot;skill&quot;)) Contrasts set to contr.sum for the following variables: hand, skill fit$Anova Anova Table (Type III tests) Response: dv Sum Sq Df F value Pr(&gt;F) (Intercept) 27781.3 1 1676.9096 &lt; 2.2e-16 *** hand 2285.3 2 68.9729 &lt; 2.2e-16 *** skill 48.9 1 2.9540 0.0867525 . hand:skill 246.5 2 7.4401 0.0007089 *** Residuals 4705.0 284 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 13.4 Interpreting parameters (very important!) fit = lm(formula = balance ~ skill * hand, data = df.poker) fit %&gt;% summary() Call: lm(formula = balance ~ skill * hand, data = df.poker) Residuals: Min 1Q Median 3Q Max -13.6976 -2.4739 0.0348 2.4644 14.7806 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 4.5866 0.5686 8.067 1.85e-14 *** skillexpert 2.7098 0.8041 3.370 0.000852 *** handneutral 5.2572 0.8041 6.538 2.75e-10 *** handgood 9.2110 0.8041 11.455 &lt; 2e-16 *** skillexpert:handneutral -1.7042 1.1372 -1.499 0.135038 skillexpert:handgood -4.2522 1.1372 -3.739 0.000222 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 4.02 on 294 degrees of freedom Multiple R-squared: 0.3731, Adjusted R-squared: 0.3624 F-statistic: 34.99 on 5 and 294 DF, p-value: &lt; 2.2e-16 Important: The t-statistic for skillexpert is not telling us that there is a main effect of skill. Instead, it shows the difference between skill = average and skill = expert when all other predictors in the model are 0!! Here, this parameter just captures whether there is a significant difference between average and skilled players when they have a bad hand (because that’s the reference category here). Let’s check that this is true. df.poker %&gt;% group_by(skill, hand) %&gt;% summarize(mean = mean(balance)) %&gt;% filter(hand == &quot;bad&quot;) %&gt;% pivot_wider(names_from = skill, values_from = mean) %&gt;% mutate(difference = expert - average) `summarise()` has grouped output by &#39;skill&#39;. You can override using the `.groups` argument. # A tibble: 1 × 4 hand average expert difference &lt;fct&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 bad 4.59 7.30 2.71 We see here that the difference in balance between the average and expert players when they have a bad hand is 2.7098. This is the same value as the skillexpert parameter in the summary() table above, and the corresponding significance test captures whether this difference is significantly different from 0. It doesn’t capture, whether there is an effect of skill overall! To test this, we need to do an analysis of variance (using the Anova(type = 3) function). 13.5 Linear contrasts Here is a linear contrast that assumes that there is a linear relationship between the quality of one’s hand, and the final balance. df.poker = df.poker %&gt;% mutate(hand_contrast = factor(hand, levels = c(&quot;bad&quot;, &quot;neutral&quot;, &quot;good&quot;), labels = c(-1, 0, 1)), hand_contrast = hand_contrast %&gt;% as.character() %&gt;% as.numeric()) fit.contrast = lm(formula = balance ~ hand_contrast, data = df.poker) Here is a visualization of the model prediction together with the residuals. df.plot = df.poker %&gt;% mutate(hand_jitter = hand %&gt;% as.numeric(), hand_jitter = hand_jitter + runif(n(), min = -0.4, max = 0.4)) df.tidy = fit.contrast %&gt;% tidy() %&gt;% select_if(is.numeric) %&gt;% mutate_all(~ round(., 2)) df.augment = fit.contrast %&gt;% augment() %&gt;% clean_names() %&gt;% bind_cols(df.plot %&gt;% select(hand_jitter)) ggplot(data = df.plot, mapping = aes(x = hand_jitter, y = balance, color = as.factor(hand_contrast))) + geom_point(alpha = 0.8) + geom_segment(data = NULL, aes(x = 0.6, xend = 1.4, y = df.tidy$estimate[1]-df.tidy$estimate[2], yend = df.tidy$estimate[1]-df.tidy$estimate[2]), color = &quot;red&quot;, size = 1) + geom_segment(data = NULL, aes(x = 1.6, xend = 2.4, y = df.tidy$estimate[1], yend = df.tidy$estimate[1]), color = &quot;orange&quot;, size = 1) + geom_segment(data = NULL, aes(x = 2.6, xend = 3.4, y = df.tidy$estimate[1] + df.tidy$estimate[2], yend = df.tidy$estimate[1] + df.tidy$estimate[2]), color = &quot;green&quot;, size = 1) + geom_segment(data = df.augment, aes(xend = hand_jitter, y = balance, yend = fitted), alpha = 0.3) + labs(y = &quot;balance&quot;) + scale_color_manual(values = c(&quot;red&quot;, &quot;orange&quot;, &quot;green&quot;)) + scale_x_continuous(breaks = 1:3, labels = c(&quot;bad&quot;, &quot;neutral&quot;, &quot;good&quot;)) + theme(legend.position = &quot;none&quot;, axis.title.x = element_blank()) 13.5.1 Hypothetical data Here is some code to generate a hypothetical developmental data set. # make example reproducible set.seed(1) # means = c(5, 10, 5) means = c(3, 5, 20) # means = c(3, 5, 7) # means = c(3, 7, 12) sd = 2 sample_size = 20 # generate data df.development = tibble( group = rep(c(&quot;3-4&quot;, &quot;5-6&quot;, &quot;7-8&quot;), each = sample_size), performance = NA) %&gt;% mutate(performance = ifelse(group == &quot;3-4&quot;, rnorm(sample_size, mean = means[1], sd = sd), performance), performance = ifelse(group == &quot;5-6&quot;, rnorm(sample_size, mean = means[2], sd = sd), performance), performance = ifelse(group == &quot;7-8&quot;, rnorm(sample_size, mean = means[3], sd = sd), performance), group = factor(group, levels = c(&quot;3-4&quot;, &quot;5-6&quot;, &quot;7-8&quot;)), group_contrast = group %&gt;% fct_recode(`-1` = &quot;3-4&quot;, `0` = &quot;5-6&quot;, `1` = &quot;7-8&quot;) %&gt;% as.character() %&gt;% as.numeric()) Let’s define a linear contrast using the emmeans package, and test whether it’s significant. fit = lm(formula = performance ~ group, data = df.development) fit %&gt;% emmeans(&quot;group&quot;, contr = list(linear = c(-0.5, 0, 0.5)), adjust = &quot;bonferroni&quot;) %&gt;% pluck(&quot;contrasts&quot;) contrast estimate SE df t.ratio p.value linear 8.45 0.274 57 30.856 &lt;.0001 Yes, we see that there is a significant positive linear contrast with an estimate of 8.45. This means, it predicts a difference of 8.45 in performance between each of the consecutive age groups. For a visualization of the predictions of this model, see Figure @ref{fig:linear-contrast-model}. 13.5.2 Visualization Total variance: set.seed(1) fit_c = lm(formula = performance ~ 1, data = df.development) df.plot = df.development %&gt;% mutate(group_jitter = 1 + runif(n(), min = -0.25, max = 0.25)) df.augment = fit_c %&gt;% augment() %&gt;% clean_names() %&gt;% bind_cols(df.plot %&gt;% select(group, group_jitter)) ggplot(data = df.plot, mapping = aes(x = group_jitter, y = performance, fill = group)) + geom_hline(yintercept = mean(df.development$performance)) + geom_point(alpha = 0.5) + geom_segment(data = df.augment, aes(xend = group_jitter, yend = fitted), alpha = 0.2) + labs(y = &quot;performance&quot;) + theme(legend.position = &quot;none&quot;, axis.text.x = element_blank(), axis.title.x = element_blank()) With contrast # make example reproducible set.seed(1) fit = lm(formula = performance ~ group_contrast, data = df.development) df.plot = df.development %&gt;% mutate(group_jitter = group %&gt;% as.numeric(), group_jitter = group_jitter + runif(n(), min = -0.4, max = 0.4)) df.tidy = fit %&gt;% tidy() %&gt;% select(where(is.numeric)) %&gt;% mutate(across(.fns = ~ round(. , 2))) df.augment = fit %&gt;% augment() %&gt;% clean_names() %&gt;% bind_cols(df.plot %&gt;% select(group_jitter)) ggplot(data = df.plot, mapping = aes(x = group_jitter, y = performance, color = as.factor(group_contrast))) + geom_point(alpha = 0.8) + geom_segment(data = NULL, aes(x = 0.6, xend = 1.4, y = df.tidy$estimate[1]-df.tidy$estimate[2], yend = df.tidy$estimate[1]-df.tidy$estimate[2]), color = &quot;red&quot;, size = 1) + geom_segment(data = NULL, aes(x = 1.6, xend = 2.4, y = df.tidy$estimate[1], yend = df.tidy$estimate[1]), color = &quot;orange&quot;, size = 1) + geom_segment(data = NULL, aes(x = 2.6, xend = 3.4, y = df.tidy$estimate[1] + df.tidy$estimate[2], yend = df.tidy$estimate[1] + df.tidy$estimate[2]), color = &quot;green&quot;, size = 1) + geom_segment(data = df.augment, aes(xend = group_jitter, y = performance, yend = fitted), alpha = 0.3) + labs(y = &quot;performance&quot;) + scale_color_manual(values = c(&quot;red&quot;, &quot;orange&quot;, &quot;green&quot;)) + scale_x_continuous(breaks = 1:3, labels = levels(df.development$group)) + theme(legend.position = &quot;none&quot;, axis.title.x = element_blank()) Figure 13.1: Predictions of the linear contrast model Results figure df.development %&gt;% ggplot(mapping = aes(x = group, y = performance)) + geom_point(alpha = 0.3, position = position_jitter(width = 0.1, height = 0)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, shape = 21, fill = &quot;white&quot;, size = 0.75) Here we test some more specific hypotheses: the the two youngest groups of children are different from the oldest group, and that the 3 year olds are different from the 5 year olds. # fit the linear model fit = lm(formula = performance ~ group, data = df.development) # check factor levels levels(df.development$group) [1] &quot;3-4&quot; &quot;5-6&quot; &quot;7-8&quot; # define the contrasts of interest contrasts = list(young_vs_old = c(-0.5, -0.5, 1), three_vs_five = c(-0.5, 0.5, 0)) # compute significance test on contrasts fit %&gt;% emmeans(&quot;group&quot;, contr = contrasts, adjust = &quot;bonferroni&quot;) %&gt;% pluck(&quot;contrasts&quot;) contrast estimate SE df t.ratio p.value young_vs_old 16.094 0.474 57 33.936 &lt;.0001 three_vs_five 0.803 0.274 57 2.933 0.0097 P value adjustment: bonferroni method for 2 tests 13.5.3 Post-hoc tests Post-hoc tests for a single predictor (using the poker data set). fit = lm(formula = balance ~ hand, data = df.poker) # post hoc tests fit %&gt;% emmeans(pairwise ~ hand, adjust = &quot;bonferroni&quot;) %&gt;% pluck(&quot;contrasts&quot;) contrast estimate SE df t.ratio p.value bad - neutral -4.41 0.581 297 -7.576 &lt;.0001 bad - good -7.08 0.581 297 -12.185 &lt;.0001 neutral - good -2.68 0.581 297 -4.609 &lt;.0001 P value adjustment: bonferroni method for 3 tests Post-hoc tests for two predictors (: # fit the model fit = lm(formula = balance ~ hand + skill, data = df.poker) # post hoc tests fit %&gt;% emmeans(pairwise ~ hand + skill, adjust = &quot;bonferroni&quot;) %&gt;% pluck(&quot;contrasts&quot;) contrast estimate SE df t.ratio p.value bad average - neutral average -4.405 0.580 296 -7.593 &lt;.0001 bad average - good average -7.085 0.580 296 -12.212 &lt;.0001 bad average - bad expert -0.724 0.474 296 -1.529 1.0000 bad average - neutral expert -5.129 0.749 296 -6.849 &lt;.0001 bad average - good expert -7.809 0.749 296 -10.427 &lt;.0001 neutral average - good average -2.680 0.580 296 -4.619 0.0001 neutral average - bad expert 3.681 0.749 296 4.914 &lt;.0001 neutral average - neutral expert -0.724 0.474 296 -1.529 1.0000 neutral average - good expert -3.404 0.749 296 -4.545 0.0001 good average - bad expert 6.361 0.749 296 8.492 &lt;.0001 good average - neutral expert 1.955 0.749 296 2.611 0.1424 good average - good expert -0.724 0.474 296 -1.529 1.0000 bad expert - neutral expert -4.405 0.580 296 -7.593 &lt;.0001 bad expert - good expert -7.085 0.580 296 -12.212 &lt;.0001 neutral expert - good expert -2.680 0.580 296 -4.619 0.0001 P value adjustment: bonferroni method for 15 tests fit = lm(formula = balance ~ hand, data = df.poker) # comparing each to the mean fit %&gt;% emmeans(eff ~ hand) %&gt;% pluck(&quot;contrasts&quot;) contrast estimate SE df t.ratio p.value bad effect -3.830 0.336 297 -11.409 &lt;.0001 neutral effect 0.575 0.336 297 1.713 0.0877 good effect 3.255 0.336 297 9.696 &lt;.0001 P value adjustment: fdr method for 3 tests # one vs. all others fit %&gt;% emmeans(del.eff ~ hand) %&gt;% pluck(&quot;contrasts&quot;) contrast estimate SE df t.ratio p.value bad effect -5.745 0.504 297 -11.409 &lt;.0001 neutral effect 0.863 0.504 297 1.713 0.0877 good effect 4.882 0.504 297 9.696 &lt;.0001 P value adjustment: fdr method for 3 tests 13.5.4 Understanding dummy coding fit = lm(formula = balance ~ 1 + hand, data = df.poker) fit %&gt;% summary() Call: lm(formula = balance ~ 1 + hand, data = df.poker) Residuals: Min 1Q Median 3Q Max -12.9264 -2.5902 -0.0115 2.6573 15.2834 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 5.9415 0.4111 14.451 &lt; 2e-16 *** handneutral 4.4051 0.5815 7.576 4.55e-13 *** handgood 7.0849 0.5815 12.185 &lt; 2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 4.111 on 297 degrees of freedom Multiple R-squared: 0.3377, Adjusted R-squared: 0.3332 F-statistic: 75.7 on 2 and 297 DF, p-value: &lt; 2.2e-16 model.matrix(fit) %&gt;% as_tibble() %&gt;% distinct() # A tibble: 3 × 3 `(Intercept)` handneutral handgood &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 1 0 0 2 1 1 0 3 1 0 1 df.poker %&gt;% select(participant, hand, balance) %&gt;% group_by(hand) %&gt;% top_n(3, wt = -participant) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) participant hand balance 1 bad 4.00 2 bad 5.55 3 bad 9.45 51 neutral 11.74 52 neutral 10.04 53 neutral 9.49 101 good 10.86 102 good 8.68 103 good 14.36 13.5.5 Understanding sum coding fit = lm(formula = balance ~ 1 + hand, contrasts = list(hand = &quot;contr.sum&quot;), data = df.poker) fit %&gt;% summary() Call: lm(formula = balance ~ 1 + hand, data = df.poker, contrasts = list(hand = &quot;contr.sum&quot;)) Residuals: Min 1Q Median 3Q Max -12.9264 -2.5902 -0.0115 2.6573 15.2834 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 9.7715 0.2374 41.165 &lt;2e-16 *** hand1 -3.8300 0.3357 -11.409 &lt;2e-16 *** hand2 0.5751 0.3357 1.713 0.0877 . --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 4.111 on 297 degrees of freedom Multiple R-squared: 0.3377, Adjusted R-squared: 0.3332 F-statistic: 75.7 on 2 and 297 DF, p-value: &lt; 2.2e-16 model.matrix(fit) %&gt;% as_tibble() %&gt;% distinct() %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) (Intercept) hand1 hand2 1 1 0 1 0 1 1 -1 -1 13.6 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 car_3.0-13 carData_3.0-5 emmeans_1.7.3 [13] afex_1.1-1 lme4_1.1-29 Matrix_1.4-1 broom_0.8.0 [17] janitor_2.1.0 kableExtra_1.3.4 knitr_1.39 loaded via a namespace (and not attached): [1] TH.data_1.1-1 minqa_1.2.4 colorspace_2.0-3 [4] ellipsis_0.3.2 htmlTable_2.4.0 estimability_1.3 [7] snakecase_0.11.0 base64enc_0.1-3 fs_1.5.2 [10] rstudioapi_0.13 farver_2.1.0 bit64_4.0.5 [13] fansi_1.0.3 mvtnorm_1.1-3 lubridate_1.8.0 [16] xml2_1.3.3 codetools_0.2-18 splines_4.1.2 [19] Formula_1.2-4 jsonlite_1.8.0 nloptr_2.0.0 [22] cluster_2.1.3 dbplyr_2.1.1 png_0.1-7 [25] compiler_4.1.2 httr_1.4.3 backports_1.4.1 [28] assertthat_0.2.1 fastmap_1.1.0 cli_3.3.0 [31] htmltools_0.5.2 tools_4.1.2 lmerTest_3.1-3 [34] coda_0.19-4 gtable_0.3.0 glue_1.6.2 [37] reshape2_1.4.4 Rcpp_1.0.8.3 cellranger_1.1.0 [40] jquerylib_0.1.4 vctrs_0.4.1 svglite_2.1.0 [43] nlme_3.1-157 xfun_0.30 rvest_1.0.2 [46] lifecycle_1.0.1 MASS_7.3-57 zoo_1.8-10 [49] scales_1.2.0 vroom_1.5.7 hms_1.1.1 [52] parallel_4.1.2 sandwich_3.0-1 RColorBrewer_1.1-3 [55] yaml_2.3.5 gridExtra_2.3 sass_0.4.1 [58] rpart_4.1.16 latticeExtra_0.6-29 stringi_1.7.6 [61] highr_0.9 checkmate_2.1.0 boot_1.3-28 [64] rlang_1.0.2 pkgconfig_2.0.3 systemfonts_1.0.4 [67] evaluate_0.15 lattice_0.20-45 htmlwidgets_1.5.4 [70] labeling_0.4.2 bit_4.0.4 tidyselect_1.1.2 [73] plyr_1.8.7 magrittr_2.0.3 bookdown_0.26 [76] R6_2.5.1 generics_0.1.2 Hmisc_4.7-0 [79] multcomp_1.4-19 DBI_1.1.2 foreign_0.8-82 [82] pillar_1.7.0 haven_2.5.0 withr_2.5.0 [85] nnet_7.3-17 survival_3.3-1 abind_1.4-5 [88] modelr_0.1.8 crayon_1.5.1 utf8_1.2.2 [91] tzdb_0.3.0 rmarkdown_2.14 jpeg_0.1-9 [94] grid_4.1.2 readxl_1.4.0 data.table_1.14.2 [97] reprex_2.0.1 digest_0.6.29 webshot_0.5.3 [100] xtable_1.8-4 numDeriv_2016.8-1.1 munsell_0.5.0 [103] viridisLite_0.4.0 bslib_0.3.1 "],["power-analysis.html", "Chapter 14 Power analysis 14.1 Learning goals 14.2 Load packages and set plotting theme 14.3 Load data sets 14.4 Decision-making 14.5 Effect sizes 14.6 Determining sample size 14.7 Additional resources 14.8 Session info", " Chapter 14 Power analysis 14.1 Learning goals Making decisions based on statistical inference. The concept of statistical power. Calculating power. Common effect size measures. Determining sample size via simulation. Understanding map() and its children. Understanding nest() and unnest(). 14.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;broom&quot;) # for tidying up model fits library(&quot;magrittr&quot;) # for going all in with the pipe library(&quot;effectsize&quot;) # for computing effect size measures library(&quot;pwr&quot;) # for power calculations library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size # markdown settings for rendered code chunks opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) # suppress summary warnings options(dplyr.summarise.inform = F) 14.3 Load data sets df.poker = read_csv(&quot;data/poker.csv&quot;) 14.4 Decision-making Figures to illustrate power: mu0 = 10 mu1 = 18 # mu0 = 8 # mu1 = 20 # sd0 = 3 # sd1 = 3 sd0 = 2 sd1 = 2 alpha = 0.05 # alpha = 0.01 ggplot(data = tibble(x = c(0, 30)), mapping = aes(x = x)) + stat_function(fun = &quot;dnorm&quot;, size = 1, color = &quot;blue&quot;, args = list(mean = mu0, sd = sd0)) + stat_function(fun = &quot;dnorm&quot;, size = 1, color = &quot;red&quot;, args = list(mean = mu1, sd = sd1)) + stat_function(fun = &quot;dnorm&quot;, geom = &quot;area&quot;, size = 1, fill = &quot;blue&quot;, alpha = 0.5, args = list(mean = mu0, sd = sd0), xlim = c(qnorm(1-alpha, mean = mu0, sd = sd0), 20)) + stat_function(fun = &quot;dnorm&quot;, geom = &quot;area&quot;, size = 1, fill = &quot;red&quot;, alpha = 0.5, args = list(mean = mu1, sd = sd1), xlim = c(0, c(qnorm(1-alpha, mean = mu0, sd = sd0)))) + geom_vline(xintercept = qnorm(1-alpha, mean = mu0, sd = sd0), size = 1) + coord_cartesian(expand = F) 14.5 Effect sizes 14.5.1 Cohen’s d Cohen’s \\(d\\) is defined as: \\[ d = \\frac{\\overline y_1 - \\overline y_2}{s_p} \\] where \\[ s_p = \\sqrt\\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2} \\] # using the effectsize package cohens_d(x = balance ~ skill, data = df.poker) Cohen&#39;s d | 95% CI ------------------------- 0.14 | [-0.08, 0.37] - Estimated using pooled SD. # compute by hand df.cohen = df.poker %&gt;% group_by(skill) %&gt;% summarize(mean = mean(balance), var = var(balance), n = n()) %&gt;% ungroup() %&gt;% pivot_wider(names_from = skill, values_from = c(mean, var, n), names_sep = &quot;&quot;) %&gt;% mutate(sp = sqrt(((n1 - 1) * var1 + (n2 - 1) * var2) / (n1 + n2 - 2)), d = abs(mean1 - mean2) / sp) print(df.cohen) # A tibble: 1 × 8 mean1 mean2 var1 var2 n1 n2 sp d &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;int&gt; &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; 1 10.1 9.41 20.3 30.3 150 150 5.03 0.144 14.6 Determining sample size One way to determine sample size is by using the pwr package. While this packages is very convenient, we cannot compute power for all the hypotheses that we might be interested in testing. 14.6.1 pwr package The pwr package has a number of functions that we can use do determine the desired sample size for different experimental designs. Check out this vignette here for more information. 14.6.1.1 Binomial test We can determine what sample size we need for a Binomial test that compares two different proportions like so: pwr.p.test(h = ES.h(p1 = 0.75, p2 = 0.50), sig.level = 0.05, power = 0.80, alternative = &quot;greater&quot;) proportion power calculation for binomial distribution (arcsine transformation) h = 0.5235988 n = 22.55126 sig.level = 0.05 power = 0.8 alternative = greater The pwr.p.test() function wants the effect size h as an argument which we can compute via the ES.h() function that takes two proportions as arguments. I then further defined the desired significance level, power, and whether the test is one-sided or two-sided. To have an 80% chance of detecting a difference between a proportion of p1 = 0.75 and p2 = 0.50, we would need to run a study with 23 participants. We can use the plot() function to illustrate what power we would get for different sample sizes. pwr.p.test(h = ES.h(p1 = 0.75, p2 = 0.50), sig.level = 0.05, power = 0.80, alternative = &quot;greater&quot;) %&gt;% plot() Notice that this is a ggplot object, so we could tweak it further if we’d like to, like so: pwr.p.test(h = ES.h(p1 = 0.75, p2 = 0.50), sig.level = 0.05, power = 0.80, alternative = &quot;greater&quot;) %&gt;% plot() + geom_hline(yintercept = 0.5, linetype = 2, color = &quot;blue&quot;) + theme(plot.title = element_text(size = 20)) 14.6.1.2 t-test (two independent samples) Here is how we could calculate the desired sample size for a t-test with two independent samples. pwr.t.test(d = 0.3, power = 0.8, type = &quot;two.sample&quot;, alternative = &quot;two.sided&quot;) Two-sample t test power calculation n = 175.3847 d = 0.3 sig.level = 0.05 power = 0.8 alternative = two.sided NOTE: n is number in *each* group So, to achieve a power of 0.8 for an effect size of d = 0.3, we would need to have 176 participants in each condition! 14.6.2 Power analysis While the pwr package works great for relatively simple designs, it quickly reaches its limits. For example, you may be interested in the sample size required to achieve adequate power for detecting an interaction, or for a specific linear contrast. Luckily, there is a very flexible approach to determining sample size: via simulation! 14.6.2.1 The general recipe assume: significance level, n, effect size simulate a large number of data sets of size n with the specified effect size for each data set, run a statistical test to calculate the p-value determine the probability of rejecting the H0 (given that H1 is true) 14.6.2.2 Using map() and list columns 14.6.2.2.1 Understanding map() map() is a powerful family of functions that’s part of the purrr package (which is included in the tidyverse, so we don’t have to load it separately). Using map() allows us to avoid nasty for-loops! Let’s take a look at the help function. help(map) As the help function says, map() allows us to apply a function to each element of a vector. Here is a simple example: map(.x = 1:3, .f = ~ .x^2) [[1]] [1] 1 [[2]] [1] 4 [[3]] [1] 9 I’ve passed the vector of numbers .x = 1:3 to map(), and then applied the anonymous function .f = ~ .x^2 to each of the elements in that vector. Always make sure to use the ~ for defining anonymous functions! As with many R functions, you can achieve the same outcome in many different ways. Here are a number of ways to do the same thing: [1] &quot;Is tmp1 identical to tmp2? TRUE&quot; [1] &quot;Is tmp2 identical to tmp3? TRUE&quot; 14.6.2.2.1.1 Understanding map()’s children Notice that the output of map() is a list. map(.x = 1:3, .f = ~ .x^2) [[1]] [1] 1 [[2]] [1] 4 [[3]] [1] 9 The map() function has many children that differ in what they output. For example, map_dbl() outputs a numeric vector instead of a list. map_dbl(.x = 1:3, .f = ~ .x^2) [1] 1 4 9 And map_lgl() returns logical values. map_lgl(.x = 1:3, .f = ~ .x == 1) [1] TRUE FALSE FALSE We can also return data frames by using either map_dfr() which binds data frames by row, or map_dfc() which binds data frames by column. Here is an example: set.seed(1) # function to create a data frame fun_make_df = function(x){ tibble(number = x, group = sample(c(&quot;A&quot;, &quot;B&quot;), size = 3, replace = T), value = rnorm(n = 3)) } # bind data frames by row map_dfr(.x = 1:3, .f = ~ fun_make_df(.)) # A tibble: 9 × 3 number group value &lt;int&gt; &lt;chr&gt; &lt;dbl&gt; 1 1 A 1.33 2 1 B 1.27 3 1 A 0.415 4 2 B 0.487 5 2 A 0.738 6 2 A 0.576 7 3 B -0.799 8 3 A -1.15 9 3 A -0.289 A nice use-case of the map_dfr() function can be to read in a number of csv files from individual participants into one larger data frame. Let’s simulate some data first, and save the data of each participant as a separate csv file (using map() of course). set.seed(1) fun_simulate_csv = function(x){ n_observations = sample(3:6, size = 1) df = tibble(age = sample(18:99, size = n_observations), responses = rnorm(n = n_observations, mean = 100, sd = 10)) write_csv(df, file = str_c(&quot;data/participant&quot;, x, &quot;.csv&quot;)) } map(.x = 1:3, .f = ~ fun_simulate_csv(.)) [[1]] # A tibble: 3 × 2 age responses &lt;int&gt; &lt;dbl&gt; 1 85 91.6 2 56 116. 3 18 103. [[2]] # A tibble: 5 × 2 age responses &lt;int&gt; &lt;dbl&gt; 1 68 115. 2 38 104. 3 71 93.8 4 91 77.9 5 24 111. [[3]] # A tibble: 4 × 2 age responses &lt;int&gt; &lt;dbl&gt; 1 61 106. 2 96 109. 3 50 108. 4 52 101. Now, let’s read in the data from the three participants and combine it into a single data frame. map_dfr(.x = 1:3, .f = ~ read_csv(str_c(&quot;data/participant&quot;, .x, &quot;.csv&quot;)), .id = &quot;participant&quot;) # A tibble: 12 × 3 participant age responses &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; 1 1 85 91.6 2 1 56 116. 3 1 18 103. 4 2 68 115. 5 2 38 104. 6 2 71 93.8 7 2 91 77.9 8 2 24 111. 9 3 61 106. 10 3 96 109. 11 3 50 108. 12 3 52 101. Notice how I used the .id = argument of the function to add a participant column to my data frame in this case. 14.6.2.2.1.2 Use map2() for functions with more than one input If you have a function with more than one input, map2() is your friend. Here is a silly example: map2_dbl(.x = c(1.23, 2.13, 5.32), .y = c(0, 1, 2), .f = ~ round(.x, digits = .y)) [1] 1.00 2.10 5.32 Here, I took the vector of numbers .x and rounded it to a different number of digits according to what I’ve specified it .y. The same works with data frames, too, like so: tibble(x = c(1.23, 2.13, 5.32), n = c(0, 1, 2)) %&gt;% mutate(rounded = map2_dbl(.x = x, .y = n, .f = ~ round(.x, digits = .y))) # A tibble: 3 × 3 x n rounded &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 1.23 0 1 2 2.13 1 2.1 3 5.32 2 5.32 14.6.2.2.1.3 Use pmap() to go all out (i.e. for functions with more than two arguments) pmap() is your friend for functions that have more than two arguments. Here is an example: tibble(x = c(1, 2, 3), y = c(23, 12, 1), z = c(4, 5, 4)) %&gt;% mutate(result = pmap_dbl(.l = ., .f = ~ ..1 * ..2 + ..3)) # A tibble: 3 × 4 x y z result &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 1 23 4 27 2 2 12 5 29 3 3 1 4 7 Notice than when using more than two arguments, we refer to each function argument with ..1, ..2, ..3, etc. 14.6.2.2.1.4 Practice 1 – Having fun with map() Use the map2_dbl() function to create a new variable in this data frame that’s the maximum of each row across columns a and b. df.practice = tibble(a = c(12, 14, 52, 23, 23), b = c(29, 12, 4, 48, 37)) # write your code here For the fast ones: For each row in the data frame, write a function that calculates the mean of columns a and b, and the rounds to the number of digits specified in column d. df.practice = tibble(a = c(12.123, 53.234, 23.324, 54.232, 12.454), b = c(12.456, 23.234, 6.736, 3.346, 7.232), d = c(1, 2, 2, 3, 1)) # write your code here 14.6.2.2.2 List columns map() becomes particularly powerful when combined with list columns. List columns allow you to put data frames into a column of your data frame. For example, you can do something like this: df.data = tibble(participant = 1, age = 23, data = list(tibble(trial = c(1, 2, 3), response = c(23, 95, 37)))) print(df.data) # A tibble: 1 × 3 participant age data &lt;dbl&gt; &lt;dbl&gt; &lt;list&gt; 1 1 23 &lt;tibble [3 × 2]&gt; We could access what’s in this data column like so df.data %&gt;% pluck(&quot;data&quot;, 1) # A tibble: 3 × 2 trial response &lt;dbl&gt; &lt;dbl&gt; 1 1 23 2 2 95 3 3 37 14.6.2.2.2.1 unnest() For getting data frames out of list columns, we can use the unnest() function. df.data %&gt;% unnest(cols = data) # A tibble: 3 × 4 participant age trial response &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 1 23 1 23 2 1 23 2 95 3 1 23 3 37 14.6.2.2.2.2 nest() We can use the nest() function to create list columns. nest() works particularly well in combination with group_by(). For example, here I’m created three separate data sets where the size of each data set is determined by the x column. # original data frame df.data = tibble(participant = c(1, 1, 1, 2, 2, 3), response1 = 1:6, response2 = 6:1) print(df.data) # A tibble: 6 × 3 participant response1 response2 &lt;dbl&gt; &lt;int&gt; &lt;int&gt; 1 1 1 6 2 1 2 5 3 1 3 4 4 2 4 3 5 2 5 2 6 3 6 1 # nested data frame df.data = df.data %&gt;% group_by(participant) %&gt;% nest() %&gt;% ungroup() print(df.data) # A tibble: 3 × 2 participant data &lt;dbl&gt; &lt;list&gt; 1 1 &lt;tibble [3 × 2]&gt; 2 2 &lt;tibble [2 × 2]&gt; 3 3 &lt;tibble [1 × 2]&gt; # and back to the original data frame df.data = df.data %&gt;% unnest(cols = data) print(df.data) # A tibble: 6 × 3 participant response1 response2 &lt;dbl&gt; &lt;int&gt; &lt;int&gt; 1 1 1 6 2 1 2 5 3 1 3 4 4 2 4 3 5 2 5 2 6 3 6 1 And, of course, I can use unnest() to get back to my original data frame. 14.6.2.2.3 Combining nest() and map() nest() and map() unfold their power together when it comes to fitting models. For example, consider that you want to fit a separate linear to subests of your data. Here is how you can do that using nest() and map(). I’ll demonstrate via the infamous mtcars data set that comes with R. mtcars %&gt;% head(10) mpg cyl disp hp drat wt qsec vs am gear carb Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1 Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2 Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4 What I want to do is to fit separate regression models predicting mpg (miles per gallon) as a function of wt (the car’s weight) for cars with different numbers of cylinders. Here is how that works. df.data = mtcars %&gt;% group_by(cyl) %&gt;% nest() %&gt;% mutate(fit = map(.x = data, .f = ~ lm(formula = mpg ~ 1 + wt, data = .))) I first grouped by the cyl (the number of cylinders), used nest() to put the rest of the data into a list column, and then used mutate() to run a separate linear model on each data set and saved the modle result into the fit column. With some more wrangling, I could, for example, extract the coefficients of each model like so: mtcars %&gt;% group_by(cyl) %&gt;% nest() %&gt;% mutate(fit = map(.x = data, .f = ~ lm(mpg ~ 1 + wt, data = .)), coef = map(.x = fit, .f = ~ tidy(.))) %&gt;% unnest(cols = coef) %&gt;% select(-c(data, fit)) # A tibble: 6 × 6 # Groups: cyl [3] cyl term estimate std.error statistic p.value &lt;dbl&gt; &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 6 (Intercept) 28.4 4.18 6.79 0.00105 2 6 wt -2.78 1.33 -2.08 0.0918 3 4 (Intercept) 39.6 4.35 9.10 0.00000777 4 4 wt -5.65 1.85 -3.05 0.0137 5 8 (Intercept) 23.9 3.01 7.94 0.00000405 6 8 wt -2.19 0.739 -2.97 0.0118 Voila! A data frame that contains the coefficients for each of the three models. Base R has a number of functions like apply(), sapply(), lapply(), etc. that do similar things to map(). However, the map() family of functions works very well with the rest of the tidyverse, that’s why we’ll use it. 14.6.2.3 Power analysis via simulation So, after this long detour via map(), list columns, nest(), and unnest(), we can finally start doing some power analysis via simulation, yay! 14.6.2.3.1 Simulating the Binomial test Let’s start with the Binomial test that we played around with above. We want to use simulation to determine the sample size we need to have an 80% of detecting a difference between two proportions p = 0.75 and p = 0.5. I’ll first do it step by step, and then afterwards put all the code in one place together 14.6.2.3.1.1 Step by step Here is how we would go about this. First, I’ll set up a simulation grid. # number of simulations n_simulations = 10 # set up simulation grid df.power = expand_grid(n = seq(10, 40, 2), simulation = 1:n_simulations, p = 0.75) %&gt;% mutate(index = 1:n(), .before = n) # add an index column df.power # A tibble: 160 × 4 index n simulation p &lt;int&gt; &lt;dbl&gt; &lt;int&gt; &lt;dbl&gt; 1 1 10 1 0.75 2 2 10 2 0.75 3 3 10 3 0.75 4 4 10 4 0.75 5 5 10 5 0.75 6 6 10 6 0.75 7 7 10 7 0.75 8 8 10 8 0.75 9 9 10 9 0.75 10 10 10 10 0.75 # … with 150 more rows The expand_grid() function creates a data frame that contains all the combinations of the variables. Now, let’s generate data according to our hypothesis. set.seed(1) df.power = df.power %&gt;% group_by(index, n, simulation) %&gt;% mutate(response = rbinom(n = 1, size = n, prob = p)) df.power # A tibble: 160 × 5 # Groups: index, n, simulation [160] index n simulation p response &lt;int&gt; &lt;dbl&gt; &lt;int&gt; &lt;dbl&gt; &lt;int&gt; 1 1 10 1 0.75 8 2 2 10 2 0.75 8 3 3 10 3 0.75 7 4 4 10 4 0.75 6 5 5 10 5 0.75 9 6 6 10 6 0.75 6 7 7 10 7 0.75 5 8 8 10 8 0.75 7 9 9 10 9 0.75 7 10 10 10 10 0.75 9 # … with 150 more rows The response variable now contains samples from the sample size n according to the probability specified in p. Now it’s time for group_by() and nest() because we want to calculate the p-value for observing this response if in fact the null hypothesis was true (i.e. p = 0.5). df.power = df.power %&gt;% # generate random data mutate(fit = map2(.x = response, .y = n, .f = ~ binom.test(x = .x, # define formula n = .y, p = 0.5, alternative = &quot;two.sided&quot;))) df.power # A tibble: 160 × 6 # Groups: index, n, simulation [160] index n simulation p response fit &lt;int&gt; &lt;dbl&gt; &lt;int&gt; &lt;dbl&gt; &lt;int&gt; &lt;list&gt; 1 1 10 1 0.75 8 &lt;htest&gt; 2 2 10 2 0.75 8 &lt;htest&gt; 3 3 10 3 0.75 7 &lt;htest&gt; 4 4 10 4 0.75 6 &lt;htest&gt; 5 5 10 5 0.75 9 &lt;htest&gt; 6 6 10 6 0.75 6 &lt;htest&gt; 7 7 10 7 0.75 5 &lt;htest&gt; 8 8 10 8 0.75 7 &lt;htest&gt; 9 9 10 9 0.75 7 &lt;htest&gt; 10 10 10 10 0.75 9 &lt;htest&gt; # … with 150 more rows Now that we’ve fitted a bunch of binomial models, we only need to get at the p-values. Again, we can use the tidy() function from the broom package for help, like so: df.power = df.power %&gt;% mutate(coef = map(.x = fit, .f = ~ tidy(.))) %&gt;% select(simulation, p, index, coef) %&gt;% unnest(cols = coef) %&gt;% select(index, n, simulation, p.value) Adding missing grouping variables: `n` df.power # A tibble: 160 × 4 # Groups: index, n, simulation [160] index n simulation p.value &lt;int&gt; &lt;dbl&gt; &lt;int&gt; &lt;dbl&gt; 1 1 10 1 0.109 2 2 10 2 0.109 3 3 10 3 0.344 4 4 10 4 0.754 5 5 10 5 0.0215 6 6 10 6 0.754 7 7 10 7 1 8 8 10 8 0.344 9 9 10 9 0.344 10 10 10 10 0.0215 # … with 150 more rows Finally, all that’s left is to calculate power by looking at the proportion of times in which we rejected the null hypothesis. df.power %&gt;% group_by(n) %&gt;% summarize(power = sum(p.value &lt; 0.05) / n()) # A tibble: 16 × 2 n power &lt;dbl&gt; &lt;dbl&gt; 1 10 0.2 2 12 0.4 3 14 0.4 4 16 0.2 5 18 0.2 6 20 0.8 7 22 0.6 8 24 0.5 9 26 0.7 10 28 0.6 11 30 0.8 12 32 0.9 13 34 0.9 14 36 0.8 15 38 1 16 40 1 Notice here that the power values fluctuate quite a bit. This is because we only ran 10 simulations for each sample size. To have more robust results, we need to increase the number of simulations. But first, let’s make a plot that visualizes what we found: df.plot = df.power %&gt;% group_by(n) %&gt;% summarize(power = sum(p.value &lt; 0.05) / n()) ggplot(data = df.plot, mapping = aes(x = n, y = power)) + geom_hline(yintercept = seq(0, 1, 0.1), linetype = 2, color = &quot;gray50&quot;, size = 0.1) + geom_smooth(method = &quot;loess&quot;, formula = &quot;y ~ x&quot;) + geom_point() As expected, the power increases with the sample size n.  14.6.2.3.1.2 All in one Here is a slightly different way to run the same simulation we just did before in one go: # make reproducible set.seed(1) # number of simulations n_simulations = 5 # run simulation expand_grid(n = seq(10, 40, 2), simulation = 1:n_simulations, p = 0.75) %&gt;% mutate(index = 1:n(), .before = n) %&gt;% group_by(index, n, p, simulation) %&gt;% mutate(response = rbinom(n = 1, size = n, prob = p), p.value = binom.test(x = response, n = n, p = 0.5, alternative = &quot;two.sided&quot;)$p.value) %&gt;% group_by(n, p) %&gt;% summarize(power = sum(p.value &lt; 0.05) / n()) %&gt;% ungroup() # A tibble: 16 × 3 n p power &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 10 0.75 0.2 2 12 0.75 0.2 3 14 0.75 0.4 4 16 0.75 0.2 5 18 0.75 0.6 6 20 0.75 0.8 7 22 0.75 0.6 8 24 0.75 0.4 9 26 0.75 0.6 10 28 0.75 0.8 11 30 0.75 0.8 12 32 0.75 1 13 34 0.75 0.8 14 36 0.75 0.8 15 38 0.75 1 16 40 0.75 0.8 And another time with a larger sample size, and also for two different alternative hypotheses: p1 = 0.75, and p2 = 0.9. # make reproducible set.seed(1) # number of simulations n_simulations = 100 # run simulation df.power = expand_grid(n = seq(10, 40, 2), simulation = 1:n_simulations, p = c(0.75, 0.9)) %&gt;% # added another hypothesis here mutate(index = 1:n(), .before = n) %&gt;% group_by(index, n, simulation) %&gt;% mutate(response = rbinom(n = 1, size = n, prob = p)) %&gt;% ungroup() %&gt;% mutate(fit = map2(.x = response, .y = n, .f = ~ binom.test(x = .x, n = .y, p = 0.5, alternative = &quot;two.sided&quot;))) %&gt;% mutate(coef = map(.x = fit, .f = ~ tidy(.))) %&gt;% unnest(cols = coef) %&gt;% select(index, n, p, p.value) %&gt;% group_by(n, p) %&gt;% summarize(power = sum(p.value &lt; 0.05) / n()) %&gt;% ungroup() # visualize results ggplot(data = df.power, mapping = aes(x = n, y = power, fill = as.factor(p), group = p)) + geom_hline(yintercept = seq(0, 1, 0.1), linetype = 2, color = &quot;gray50&quot;, size = 0.1) + geom_smooth(method = &quot;loess&quot;, formula = &quot;y ~ x&quot;, color = &quot;black&quot;) + geom_point(shape = 21) + labs(fill = &quot;alternative&quot;) + guides(fill = guide_legend(reverse = T)) 14.6.2.3.2 Simulating an independent samples t-test Let’s simulate data for an independent samples t-test. To do so, we need to make some assumptions about what we expect the distribution of the data to look like. Here, I assume that we get normally distributed data with some mean and standard deviation. The procedure will be very similar to the Binomial test above. The only thing that changes really is how we generate the data (and then some small wrangling differences). Let’s say that we collected the following pilot data set: set.seed(1) # parameters n = 10 mean1 = 10 sd1 = 2 mean2 = 11 sd2 = 3 df.ttest = tibble(group1 = rnorm(n = n, mean = mean1, sd = sd1), group2 = rnorm(n = n, mean = mean2, sd = sd2)) %&gt;% pivot_longer(cols = everything()) %&gt;% arrange(name) The two groups in our sample don’t differ significantly from each other. # visualize the data ggplot(data = df.ttest, mapping = aes(x = name, y = value)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;) # compute a t-test t.test(formula = value ~ name, data = df.ttest) Welch Two Sample t-test data: value by name t = -1.3135, df = 13.035, p-value = 0.2117 alternative hypothesis: true difference in means between group group1 and group group2 is not equal to 0 95 percent confidence interval: -3.9191375 0.9548788 sample estimates: mean in group group1 mean in group group2 10.26441 11.74653 Let’s calculate what the effect size was in our sample. Remember that Cohen’s d is defined as \\[d = \\frac{\\lvert\\overline y_1 - \\overline y_2\\rvert}{s_p}\\] where \\[s_p = \\sqrt\\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}\\] In our sample, the effect size Cohen’s d was: df.sample = df.ttest %&gt;% group_by(name) %&gt;% summarize(mean = mean(value), sd = sd(value), n = n()) %&gt;% ungroup() # compute the pooled standard deviation sp = sqrt(((df.sample$n[1] - 1) * df.sample$sd[1]^2 + (df.sample$n[2] - 1) * df.sample$sd[2]^2) / (df.sample$n[1] + df.sample$n[2] - 2)) d = abs(df.sample$mean[1] - df.sample$mean[2]) / sp d [1] 0.5874251 Let’s double check that we got it right: cohens_d(x = value ~ name, data = df.ttest) Cohen&#39;s d | 95% CI ------------------------- -0.59 | [-1.48, 0.32] - Estimated using pooled SD. We did! So let’s now calculate the means and standard deviations based on our pilot data and run a power analysis to determine how many participants we would need, to have an 80% chance of rejecting the null hypothesis for the estimated effect size. # make reproducible set.seed(1) # parameters mean1 = df.sample$mean[1] mean2 = df.sample$mean[2] sd1 = df.sample$sd[1] sd2 = df.sample$sd[2] # number of simulations n_simulations = 10 # n_simulations = 100 # run simulation df.power2 = expand_grid(n = seq(from = 10, to = 60, by = 5), simulation = 1:n_simulations) %&gt;% mutate(index = 1:n(), .before = n) %&gt;% group_by(index, n, simulation) %&gt;% mutate(data = list(tibble(group1 = rnorm(n = n, mean = mean1, sd = sd1), group2 = rnorm(n = n, mean = mean2, sd = sd2)) %&gt;% pivot_longer(cols = everything()))) %&gt;% mutate(fit = map(.x = data, .f = ~ lm(formula = value ~ 1 + name, data = .)), parameters = map(.x = fit, .f = ~ tidy(.))) %&gt;% select(index, n, simulation, parameters) %&gt;% unnest(cols = parameters) %&gt;% filter(term != &quot;(Intercept)&quot;) %&gt;% select(index, n, simulation, p.value) %&gt;% group_by(n) %&gt;% summarize(power = sum(p.value &lt; 0.05) / n()) %&gt;% ungroup() # visualize results ggplot(data = df.power2, mapping = aes(x = n, y = power)) + geom_hline(yintercept = seq(0, 1, 0.1), linetype = 2, color = &quot;gray50&quot;, size = 0.1) + geom_smooth(method = &quot;loess&quot;, color = &quot;black&quot;, formula = &quot;y ~ x&quot;) + geom_point(shape = 21) + scale_x_continuous(breaks = seq(10, 60, 10), labels = seq(10, 60, 10)) + scale_y_continuous(breaks = seq(0, 1, 0.1), labels = seq(0, 1, 0.1)) Let’s compare to what we’d get from the pwr package. pwr.t.test(d = 0.5874251, power = 0.8, type = &quot;two.sample&quot;, alternative = &quot;two.sided&quot;) Two-sample t test power calculation n = 46.4718 d = 0.5874251 sig.level = 0.05 power = 0.8 alternative = two.sided NOTE: n is number in *each* group Looks pretty similar! 14.6.2.3.3 Practice 2 – Simulation of an interaction effect Try to run a simulation to determine how many participants you would need to have an 80% chance of rejecting the null hypothesis that there is no interaction based on the following pilot data set: set.seed(1) # population parameters b0 = 1 b1 = 2 b2 = 3 b1_2 = -2 sd = 2 n = 10 df.linear = tibble(x = runif(n = n), y = rep(c(0, 1), each = n/2), z = b0 + b1 * x + b2 * y + b1_2 * x * y + rnorm(n = n, sd = sd)) Let’s visualize the pilot data first: ggplot(data = df.linear, mapping = aes(x = x, y = z, group = y, fill = as.factor(y), color = as.factor(y))) + geom_smooth(method = &quot;lm&quot;, se = F, show.legend = F) + geom_point(shape = 21, color = &quot;black&quot;, show.legend = F) Let’s estimate the parameters based on our sample: # parameter estimates for the coefficients based on the sample b = lm(formula = z ~ x * y, data = df.linear) %&gt;% tidy() %&gt;% select(term, estimate, p.value) # parameter estimate of the residual standard deviation sigma = lm(formula = z ~ x * y, data = df.linear) %&gt;% glance() %&gt;% pull(sigma) Run a power analysis to see how many participants you would need to have an 80% of rejecting the null hypothesis that there is no interaction. Use the parameter estimates (the beta coefficients and the standard deviation of the residuals sigma) based on your pilot data to simulate new data. Here is the strategy: Try to closely emulate what we’ve been doing for the independent samples t-test above. However, this time, we have a different way of generating the data (namely by using the regression equation: \\(z \\sim b_0 + b_1 \\cdot x + b_2 \\cdot y + b_{1\\_2} \\cdot x \\cdot y + e)\\), where \\(e \\sim N(0, \\sigma)\\). Fit the model first to extract the estimates for the beta coefficients, and the standard deviation of the residuals sigma. Then use these parameters to generate new data assuming that x is a continuous predictor between 0 and 1 (x = runif(n = n)) and y is a binary, dummy-coded variable (y = rep(c(0, 1), each = n/2)). Extract the coefficients of each model fit, and check whether the interaction is significant. Make a plot that shows how power changes with the sample size n.  set.seed(1) # write your code here Run the same power analysis this time assuming the ground truth parameters from the population (rather than the parameters that we’ve estimated from the sample). set.seed(1) # write your code here 14.7 Additional resources 14.7.1 Datacamp Functional programming with purrr 14.7.2 Cheatsheets purrr 14.7.3 Misc Getting started with pwr Visualize power Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs purrr tutorial simr: R Package for running power analysis for generalized linear mixed effects models. simglm: Alternative R package for running power analysis via simulation. cautionary tale about using pilot studies for power calculations 14.8 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 pwr_1.3-0 effectsize_0.6.0.1 magrittr_2.0.3 [13] broom_0.8.0 kableExtra_1.3.4 knitr_1.39 loaded via a namespace (and not attached): [1] nlme_3.1-157 fs_1.5.2 lubridate_1.8.0 [4] bit64_4.0.5 RColorBrewer_1.1-3 insight_0.17.0 [7] webshot_0.5.3 httr_1.4.3 tools_4.1.2 [10] backports_1.4.1 bslib_0.3.1 utf8_1.2.2 [13] R6_2.5.1 rpart_4.1.16 Hmisc_4.7-0 [16] mgcv_1.8-40 DBI_1.1.2 colorspace_2.0-3 [19] nnet_7.3-17 withr_2.5.0 gridExtra_2.3 [22] tidyselect_1.1.2 emmeans_1.7.3 bit_4.0.4 [25] compiler_4.1.2 performance_0.9.0 cli_3.3.0 [28] rvest_1.0.2 htmlTable_2.4.0 xml2_1.3.3 [31] sandwich_3.0-1 labeling_0.4.2 bookdown_0.26 [34] bayestestR_0.12.1 sass_0.4.1 checkmate_2.1.0 [37] scales_1.2.0 mvtnorm_1.1-3 systemfonts_1.0.4 [40] digest_0.6.29 foreign_0.8-82 rmarkdown_2.14 [43] svglite_2.1.0 jpeg_0.1-9 base64enc_0.1-3 [46] pkgconfig_2.0.3 htmltools_0.5.2 highr_0.9 [49] dbplyr_2.1.1 fastmap_1.1.0 htmlwidgets_1.5.4 [52] rlang_1.0.2 readxl_1.4.0 rstudioapi_0.13 [55] jquerylib_0.1.4 generics_0.1.2 farver_2.1.0 [58] zoo_1.8-10 jsonlite_1.8.0 vroom_1.5.7 [61] Formula_1.2-4 parameters_0.17.0 Matrix_1.4-1 [64] munsell_0.5.0 fansi_1.0.3 lifecycle_1.0.1 [67] stringi_1.7.6 multcomp_1.4-19 yaml_2.3.5 [70] MASS_7.3-57 grid_4.1.2 parallel_4.1.2 [73] crayon_1.5.1 lattice_0.20-45 haven_2.5.0 [76] splines_4.1.2 hms_1.1.1 pillar_1.7.0 [79] estimability_1.3 codetools_0.2-18 reprex_2.0.1 [82] glue_1.6.2 evaluate_0.15 latticeExtra_0.6-29 [85] data.table_1.14.2 modelr_0.1.8 png_0.1-7 [88] vctrs_0.4.1 tzdb_0.3.0 cellranger_1.1.0 [91] gtable_0.3.0 assertthat_0.2.1 datawizard_0.4.0 [94] xfun_0.30 xtable_1.8-4 coda_0.19-4 [97] survival_3.3-1 viridisLite_0.4.0 cluster_2.1.3 [100] TH.data_1.1-1 ellipsis_0.3.2 "],["model-comparison.html", "Chapter 15 Model comparison 15.1 Learning goals 15.2 Load packages and set plotting theme 15.3 Model comparison 15.4 Additional resources 15.5 Session info", " Chapter 15 Model comparison 15.1 Learning goals Model comparison. Underfitting vs. overfitting. Cross-validation. Leave-one-out cross-validation. k-fold cross-validation. Monte Carlo cross-validation. Information criteria: AIC and BIC. 15.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;broom&quot;) # for tidying up linear models library(&quot;patchwork&quot;) # for figure panels library(&quot;modelr&quot;) # for cross-validation library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 15.3 Model comparison In general, we want our models to explain the data we observed, and correctly predict future data. Often, there is a trade-off between how well the model fits the data we have (e.g. how much of the variance it explains), and how well the model will predict future data. If our model is too complex, then it will not only capture the systematicity in the data but also fit to the noise in the data. If our mdoel is too simple, however, it will not capture some of the systematicity that’s actually present in the data. The goal, as always in statistical modeling, is to find a model that finds the sweet spot between simplicity and complexity. 15.3.1 Fitting vs. predicting Let’s illustrate the trad-off between complexity and simplicty for fitting vs. prediction. We generate data from a model of the following form: \\[ Y_i = \\beta_0 + \\beta_1 \\cdot X_i + \\beta_2 + X_i^2 + \\epsilon_i \\] where \\[ \\epsilon_i \\sim \\mathcal{N}(\\text{mean} = 0, ~\\text{sd} = 20) \\] Here, I’ll use the following parameters: \\(\\beta_0 = 10\\), \\(\\beta_1 = 3\\), and \\(\\beta_2 = 2\\) to generate the data: set.seed(1) n_plots = 3 # sample size n_samples = 20 # number of parameters in the polynomial regression n_parameters = c(1:4, seq(7, 19, length.out = 5)) # generate data df.data = tibble(x = runif(n_samples, min = 0, max = 10), y = 10 + 3 * x + 3 * x^2 + rnorm(n_samples, sd = 20)) # plotting function plot_fit = function(i){ # calculate RMSE rmse = lm(formula = y ~ poly(x, degree = i, raw = TRUE), data = df.data) %&gt;% rmse(data = df.data) # make a plot ggplot(data = df.data, mapping = aes(x = x, y = y)) + geom_point(size = 2) + geom_smooth(method = &quot;lm&quot;, se = F, formula = y ~ poly(x, degree = i, raw = TRUE)) + annotate(geom = &quot;text&quot;, x = Inf, y = -Inf, label = str_c(&quot;RMSE = &quot;, round(rmse, 2)), hjust = 1.1, vjust = -0.3) + theme(axis.ticks = element_blank(), axis.title = element_blank(), axis.text = element_blank()) } # save plots in a list l.p = map(.x = n_parameters, .f = ~ plot_fit(.)) # make figure panel wrap_plots(plotlist = l.p, ncol = 3) As we can see, RMSE becomes smaller and smaller the more parameters the model has to fit the data. But how does the RMSE look like for new data that is generated from the same underlying ground truth? set.seed(1) n_plots = 3 # sample size n_samples = 20 # number of parameters in the polynomial regression n_parameters = c(1:4, seq(7, 19, length.out = 5)) # generate data df.data = tibble( x = runif(n_samples, min = 0, max = 10), y = 10 + 3 * x + 3 * x^2 + rnorm(n_samples, sd = 20) ) # generate some more data df.more_data = tibble(x = runif(50, min = 0, max = 10), y = 10 + 3 * x + 3 * x^2 + rnorm(50, sd = 20)) # list for plots l.p = list() # plotting function plot_fit = function(i){ # calculate RMSE for fitted data fit = lm(formula = y ~ poly(x, degree = i, raw = TRUE), data = df.data) # calculate RMSE for training data rmse = fit %&gt;% rmse(data = df.data) # calculate RMSE for new data rmse_new = fit %&gt;% rmse(data = df.more_data) # make a plot ggplot(data = df.data, mapping = aes(x = x, y = y)) + geom_point(size = 2) + geom_point(data = df.more_data, size = 2, color = &quot;red&quot;) + geom_smooth(method = &quot;lm&quot;, se = F, formula = y ~ poly(x, degree = i, raw = TRUE)) + annotate(geom = &quot;text&quot;, x = Inf, y = -Inf, label = str_c(&quot;RMSE = &quot;, round(rmse, 2)), hjust = 1.1, vjust = -0.3) + annotate(geom = &quot;text&quot;, x = Inf, y = -Inf, label = str_c(&quot;RMSE = &quot;, round(rmse_new, 2)), hjust = 1.1, vjust = -2, color = &quot;red&quot;) + theme(axis.ticks = element_blank(), axis.title = element_blank(), axis.text = element_blank()) } # map over the parameters l.p = map(.x = n_parameters, .f = ~ plot_fit(.)) # make figure panel wrap_plots(plotlist = l.p, ncol = 3) The RMSE in black shows the root mean squared error for the data that the model was fit on. The RMSE in red shows the RMSE on the new data. As you can see, the complex models do really poorly. They overfit the noise in the original data which leads to make poor predictions for new data. The simplest model (with two parameters) doesn’t do particularly well either since it misses out on the quadratic trend in the data. Both the model with the quadratic term (top middle) and a model that includes a cubic term (top right) provide a good balance – their RMSE on the new data is lowest. Let’s generate another data set: # make example reproducible set.seed(1) # parameters sample_size = 100 b0 = 1 b1 = 2 b2 = 3 sd = 0.5 # sample df.data = tibble(participant = 1:sample_size, x = runif(sample_size, min = 0, max = 1), y = b0 + b1*x + b2*x^2 + rnorm(sample_size, sd = sd)) And plot it: ggplot(data = df.data, mapping = aes(x = x, y = y)) + geom_smooth(method = &quot;lm&quot;, formula = y ~ x + I(x^2)) + geom_point() 15.3.2 F-test Let’s fit three models of increasing complexity to the data. The model which fits the way in which the data were generated has the following form: \\[ \\widehat Y_i = b_0 + b_1 \\cdot X_i + b_2 \\cdot X_i^2 \\] # fit models to the data fit_simple = lm(y ~ 1 + x, data = df.data) fit_correct = lm(y ~ 1 + x + I(x^2), data = df.data) fit_complex = lm(y ~ 1 + x + I(x^2) + I(x^3), data = df.data) # compare the models using an F-test anova(fit_simple, fit_correct) Analysis of Variance Table Model 1: y ~ 1 + x Model 2: y ~ 1 + x + I(x^2) Res.Df RSS Df Sum of Sq F Pr(&gt;F) 1 98 25.297 2 97 21.693 1 3.6039 16.115 0.0001175 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 anova(fit_correct, fit_complex) Analysis of Variance Table Model 1: y ~ 1 + x + I(x^2) Model 2: y ~ 1 + x + I(x^2) + I(x^3) Res.Df RSS Df Sum of Sq F Pr(&gt;F) 1 97 21.693 2 96 21.643 1 0.050399 0.2236 0.6374 The F-test tells us that fit_correct explains significantly more variance than fit_simple, whereas fit_complex doesn’t explain significantly more variance than fit_correct. But, as discussed in class, there are many situations in which we cannot use the F-test to compare models. Namely, whenever we want to compare unnested models where one models does not include all the predictors of the other model. But, we can still use cross-validation in this case. Let’s take a look. 15.3.3 Cross-validation Cross-validation is a powerful technique for finding the sweet spot between simplicity and complexity. Moreover, we can use cross-validation to compare models that we cannot compare using the F-test approach that we’ve been using up until now. There are many different kinds of cross-validation. All have the same idea in common though: we first fit the model to a subset of the data, often called training data and then check how well the model captures the held-out data, often called test data Different versions of cross-validation differ in how the training and test data sets are defined. We’ll look at three different cross-validation techniques: Leave-on-out cross-validation k-fold cross-validation Monte Carlo cross-validation 15.3.3.1 Leave-one-out cross-validation I’ve used code similar to this one to illustrate how LOO works in class. Here is a simple data set with 9 data points. We fit 9 models, where for each model, the training set includes one of the data points, and then we look at how well the model captures the held-out data point. We can then characterize the model’s performance by calculating the mean squared error across the 9 runs. # make example reproducible set.seed(1) # sample df.loo = tibble(x = 1:9, y = c(5, 2, 4, 10, 3, 4, 10, 2, 8)) df.loo_cross = df.loo %&gt;% crossv_loo() %&gt;% mutate(fit = map(.x = train, .f = ~ lm(y ~ x, data = .)), tidy = map(.x = fit, .f = ~ tidy(.))) %&gt;% unnest(tidy) # original plot df.plot = df.loo %&gt;% mutate(color = 1) # fit to all data except one fun.cv_plot = function(data_point){ # determine which point to leave out df.plot = df.plot %&gt;% mutate(color = ifelse(row_number() == data_point, 2, color)) # fit df.fit = df.plot %&gt;% filter(color != 2) %&gt;% lm(formula = y ~ x, data = .) %&gt;% augment(newdata = df.plot %&gt;% filter(color == 2)) %&gt;% clean_names() p = ggplot(df.plot, aes(x, y, color = as.factor(color))) + geom_segment(aes(xend = x, yend = fitted), data = df.fit, color = &quot;red&quot;, size = 1) + geom_point(size = 2) + geom_smooth(method = &quot;lm&quot;, formula = &quot;y ~ x&quot;, se = F, color = &quot;black&quot;, fullrange = T, data = df.plot %&gt;% filter(color != 2)) + scale_color_manual(values = c(&quot;black&quot;, &quot;red&quot;)) + theme(legend.position = &quot;none&quot;, axis.title = element_blank(), axis.ticks = element_blank(), axis.text = element_blank()) return(p) } # save plots in list l.plots = map(.x = 1:9, .f = ~ fun.cv_plot(.)) # make figure panel wrap_plots(plotlist = l.plots, ncol = 3) As you can see, the regression line changes quite a bit depending on which data point is in the test set. Now, let’s use LOO to evaluate the models on the data set I’ve created above: # fit the models and calculate the RMSE for each model on the test set df.cross = df.data %&gt;% crossv_loo() %&gt;% # function which generates training and test data sets mutate(model_simple = map(.x = train, .f = ~ lm(y ~ 1 + x, data = .)), model_correct = map(.x = train, .f = ~ lm(y ~ 1 + x + I(x^2), data = .)), model_complex = map(.x = train, .f = ~ lm(y ~ 1 + x + I(x^2) + I(x^3), data = .))) %&gt;% pivot_longer(cols = contains(&quot;model&quot;), names_to = &quot;model&quot;, values_to = &quot;fit&quot;) %&gt;% mutate(rmse = map2_dbl(.x = fit, .y = test, .f = ~ rmse(.x, .y))) # show the average RMSE for each model df.cross %&gt;% group_by(model) %&gt;% summarize(mean_rmse = mean(rmse) %&gt;% round(3)) # A tibble: 3 × 2 model mean_rmse &lt;chr&gt; &lt;dbl&gt; 1 model_complex 0.382 2 model_correct 0.378 3 model_simple 0.401 As we can see, the model_correct has the lowest average RMSE on the test data. One downside with LOO is that it becomes unfeasible when the number of data points is very large, as the number of cross validation runs equals the number of data points. The next cross-validation procedures help in this case. 15.3.3.2 k-fold cross-validation For k-fold cross-validation, we split the data set in k folds, and then use k-1 folds as the training set, and the remaining fold as the test set. The code is almost identical as before. Instead of crossv_loo(), we use the crossv_kfold() function instead and say how many times we want to “fold” the data. # crossvalidation scheme df.cross = df.data %&gt;% crossv_kfold(k = 10) %&gt;% mutate(model_simple = map(.x = train, .f = ~ lm(y ~ 1 + x, data = .)), model_correct = map(.x = train, .f = ~ lm(y ~ 1 + x + I(x^2), data = .)), model_complex = map(.x = train, .f = ~ lm(y ~ 1 + x + I(x^2) + I(x^3), data = .))) %&gt;% pivot_longer(cols = contains(&quot;model&quot;), names_to = &quot;model&quot;, values_to = &quot;fit&quot;) %&gt;% mutate(rsquare = map2_dbl(.x = fit, .y = test, .f = ~ rsquare(.x, .y))) df.cross %&gt;% group_by(model) %&gt;% summarize(median_rsquare = median(rsquare)) # A tibble: 3 × 2 model median_rsquare &lt;chr&gt; &lt;dbl&gt; 1 model_complex 0.884 2 model_correct 0.889 3 model_simple 0.880 Note, for this example, I’ve calculated \\(R^2\\) (the variance explained by each model) instead of RMSE – just to show you that you can do this, too. Often it’s useful to do both: show how well the model correlates, but also show the error. 15.3.3.3 Monte Carlo cross-validation Finally, let’s consider another very flexible version of cross-validation. For this version of cross-validation, we determine how many random splits into training set and test set we would like to do, and what proportion of the data should be in the test set. # crossvalidation scheme df.cross = df.data %&gt;% crossv_mc(n = 50, test = 0.5) %&gt;% # number of samples, and percentage of test mutate(model_simple = map(.x = train, .f = ~ lm(y ~ 1 + x, data = .x)), model_correct = map(.x = train, .f = ~ lm(y ~ 1 + x + I(x^2), data = .x)), model_complex = map(.x = train, .f = ~ lm(y ~ 1 + x + I(x^2) + I(x^3), data = .))) %&gt;% pivot_longer(cols = contains(&quot;model&quot;), names_to = &quot;model&quot;, values_to = &quot;fit&quot;) %&gt;% mutate(rmse = map2_dbl(.x = fit, .y = test, .f = ~ rmse(.x, .y))) df.cross %&gt;% group_by(model) %&gt;% summarize(mean_rmse = mean(rmse)) # A tibble: 3 × 2 model mean_rmse &lt;chr&gt; &lt;dbl&gt; 1 model_complex 0.492 2 model_correct 0.484 3 model_simple 0.515 In this example, I’ve asked for \\(n = 50\\) splits and for each split, half of the data was in the training set, and half of the data in the test set. 15.3.4 Bootstrap We can also use the modelr package for bootstrapping. The idea is the same as when we did cross-validation. We create a number of data sets from our original data set. Instead of splitting the data set in a training and test data set, for bootstrapping, we sample values from the original data set with replacement. Doing so, we can, for example, calculate the confidence interval of different statistics of interest. Here is an example for how to boostrap confidence intervals for a mean. # make example reproducible set.seed(1) sample_size = 10 # sample df.data = tibble(participant = 1:sample_size, x = runif(sample_size, min = 0, max = 1)) # mean of the actual sample mean(df.data$x) [1] 0.5515139 # bootstrap to get confidence intervals around the mean df.data %&gt;% bootstrap(n = 1000) %&gt;% # create 1000 bootstrapped samples mutate(estimate = map_dbl(.x = strap, .f = ~ .x %&gt;% as_tibble() %&gt;% pull(x) %&gt;% mean())) %&gt;% summarize(mean = mean(estimate), low = quantile(estimate, 0.025), # calculate the 2.5 / 97.5 percentiles high = quantile(estimate, 0.975)) # A tibble: 1 × 3 mean low high &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 0.556 0.378 0.732 15.3.5 AIC and BIC The Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) are defined as follows: \\[ \\text{AIC} = 2k-2\\ln(\\hat L) \\] \\[ \\text{BIC} = \\ln(n)k-2\\ln(\\hat L) \\] where \\(k\\) is the number of parameters in the model, \\(n\\) is the number of observations, and \\(\\hat L\\) is the maximized value of the likelihood function of the model. Both AIC and BIC trade off model fit (as measured by the maximum likelihood of the data \\(\\hat L\\)) and the number of parameters in the model. Calculating AIC and BIC in R is straightforward. We simply need to fit a linear model, and then call the AIC() or BIC() functions on the fitted model like so: set.seed(0) # let&#39;s generate some data df.example = tibble(x = runif(20, min = 0, max = 1), y = 1 + 3 * x + rnorm(20, sd = 2)) # fit a linear model fit = lm(formula = y ~ 1 + x, data = df.example) # get AIC AIC(fit) [1] 75.47296 # get BIC BIC(fit) [1] 78.46016 We can also just use the broom package to get that information: fit %&gt;% glance() # A tibble: 1 × 12 r.squared adj.r.squared sigma statistic p.value df logLik AIC BIC &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 0.255 0.214 1.45 6.16 0.0232 1 -34.7 75.5 78.5 # … with 3 more variables: deviance &lt;dbl&gt;, df.residual &lt;int&gt;, nobs &lt;int&gt; Both AIC and BIC take the number of parameters and the model’s likelihood into account. BIC additionally considers the number of observations. But how is the likelihood of a linear model determined? Let’s visualize the data first: # plot the data with a linear model fit ggplot(data = df.example, mapping = aes(x = x, y = y)) + geom_point(size = 2) + geom_smooth(method = &quot;lm&quot;, color = &quot;black&quot;) Now, let’s take a look at the residuals by plotting the fitted values on the x axis, and the residuals on the y axis. # residual plot df.plot = df.example %&gt;% lm(formula = y ~ x, data = .) %&gt;% augment() %&gt;% clean_names() ggplot(data = df.plot, mapping = aes(x = fitted, y = resid)) + geom_point(size = 2) Remember that the linear model makes the assumption that the residuals are normally distributed with mean 0 (which is always the case if we fit a linear model) and some fitted standard deviation. In fact, the standard deviation of the normal distribution is fitted such that the overall likelihood of the data is maximized. Let’s make a plot that shows a normal distribution alongside the residuals: # define a normal distribution df.normal = tibble(y = seq(-5, 5, 0.1), x = dnorm(y, sd = 2) + 3.75) # show the residual plot together with the normal distribution ggplot(data = df.plot , mapping = aes(x = fitted, y = resid)) + geom_point() + geom_path(data = df.normal, aes(x = x, y = y), size = 2) To determine the likelihood of the data given the model \\(\\hat L\\), we now calculate the likelihood of each point (with the dnorm() function), and then multiply the likelihood of each data point to get the overall likelihood. We can simply multiply the data points since we also assume that the data points are independent. Instead of multiplying likelihoods, we often sum the log likelihoods instead. This is because if we multiply many small values, the overall value gets to close to 0 so that computers get confused. By taking logs instead, we avoid these nasty precision errors. To better understand AIC and BIC, let’s calculate them by hand: # we first get the estimate of the standard deviation of the residuals sigma = fit %&gt;% glance() %&gt;% pull(sigma) # then we calculate the log likelihood of the model log_likelihood = fit %&gt;% augment() %&gt;% mutate(likelihood = dnorm(.resid, sd = sigma)) %&gt;% summarize(logLik = sum(log(likelihood))) %&gt;% as.numeric() # then we calculate AIC and BIC using the formulas introduced above aic = 2*3 - 2 * log_likelihood bic = log(nrow(df.example)) * 3 - 2 * log_likelihood print(aic) [1] 75.58017 print(bic) [1] 78.56737 Cool! The values are the same as when we use the glance() function like so (except for a small difference due to rounding): fit %&gt;% glance() %&gt;% select(AIC, BIC) # A tibble: 1 × 2 AIC BIC &lt;dbl&gt; &lt;dbl&gt; 1 75.5 78.5 15.3.5.1 log() is your friend ggplot(data = tibble(x = c(0, 1)), mapping = aes(x = x)) + stat_function(fun = &quot;log&quot;, size = 1) + labs(x = &quot;probability&quot;, y = &quot;log(probability)&quot;) + theme(axis.text = element_text(size = 24), axis.title = element_text(size = 26)) 15.4 Additional resources 15.4.1 Datacamp course Foundations of Functional Programming with purrr Intermediate functional programming with purrr 15.4.2 Reading R for Data Science: Chapter 25 15.5 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 modelr_0.1.8 patchwork_1.1.1 broom_0.8.0 [13] janitor_2.1.0 kableExtra_1.3.4 knitr_1.39 loaded via a namespace (and not attached): [1] lattice_0.20-45 svglite_2.1.0 lubridate_1.8.0 assertthat_0.2.1 [5] digest_0.6.29 utf8_1.2.2 R6_2.5.1 cellranger_1.1.0 [9] backports_1.4.1 reprex_2.0.1 evaluate_0.15 highr_0.9 [13] httr_1.4.3 pillar_1.7.0 rlang_1.0.2 readxl_1.4.0 [17] rstudioapi_0.13 jquerylib_0.1.4 Matrix_1.4-1 rmarkdown_2.14 [21] labeling_0.4.2 splines_4.1.2 webshot_0.5.3 munsell_0.5.0 [25] compiler_4.1.2 xfun_0.30 pkgconfig_2.0.3 systemfonts_1.0.4 [29] mgcv_1.8-40 htmltools_0.5.2 tidyselect_1.1.2 bookdown_0.26 [33] fansi_1.0.3 viridisLite_0.4.0 withr_2.5.0 crayon_1.5.1 [37] tzdb_0.3.0 dbplyr_2.1.1 grid_4.1.2 nlme_3.1-157 [41] jsonlite_1.8.0 gtable_0.3.0 lifecycle_1.0.1 DBI_1.1.2 [45] magrittr_2.0.3 scales_1.2.0 cli_3.3.0 stringi_1.7.6 [49] farver_2.1.0 fs_1.5.2 snakecase_0.11.0 xml2_1.3.3 [53] bslib_0.3.1 ellipsis_0.3.2 generics_0.1.2 vctrs_0.4.1 [57] tools_4.1.2 glue_1.6.2 hms_1.1.1 fastmap_1.1.0 [61] yaml_2.3.5 colorspace_2.0-3 rvest_1.0.2 haven_2.5.0 [65] sass_0.4.1 "],["causation.html", "Chapter 16 Causation 16.1 Learning goals 16.2 Recommended reading 16.3 Load packages and set plotting theme 16.4 Bayesian networks 16.5 Controlling for variables 16.6 Mediation 16.7 Moderation 16.8 Additional resources 16.9 Session info", " Chapter 16 Causation Some of these notes are adapted from this tutorial: Mediation and moderation 16.1 Learning goals Understanding what controlling for variables means. Learning a graphical procedure that helps identify when it’s good vs. bad to control for variables. Simulating a mediation analysis. Baron and Kenny’s (1986) steps for mediation. Testing the significance of a mediation. Sobel test. Bootstrapping. Bayesian approach. Limitations of mediation analysis. Simulating a moderator effect. 16.2 Recommended reading Fiedler, Schott, and Meiser (2011) MacKinnon, Fairchild, and Fritz (2007) 16.3 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;mediation&quot;) # for mediation and moderation analysis library(&quot;multilevel&quot;) # Sobel test library(&quot;broom&quot;) # tidying up regression results library(&quot;DiagrammeR&quot;) # for drawing diagrams library(&quot;DiagrammeRsvg&quot;) # for exporting pdfs of graphs library(&quot;rsvg&quot;) # for exporting pdfs of graphs library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) options(dplyr.summarise.inform = FALSE) # Disable summarize ungroup messages 16.4 Bayesian networks 16.4.1 Sprinkler example # cloudy df.cloudy = tibble(`p(C)` = 0.5) df.cloudy %&gt;% kable() %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F, font_size = 20) p(C) 0.5 # sprinkler given cloudy df.sprinkler_given_cloudy = tibble(C = c(&quot;F&quot;, &quot;T&quot;), `p(S)`= c(0.5, 0.1)) df.sprinkler_given_cloudy %&gt;% kable() %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F, font_size = 20) C p(S) F 0.5 T 0.1 # rain given cloudy df.rain_given_cloudy = tibble(C = c(&quot;F&quot;, &quot;T&quot;), `p(R)`= c(0.2, 0.8)) df.rain_given_cloudy %&gt;% kable() %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F, font_size = 20) C p(R) F 0.2 T 0.8 # wet given sprinkler and rain df.rain_given_sprinkler_and_rain = tibble( S = rep(c(&quot;F&quot;, &quot;T&quot;), 2), R = rep(c(&quot;F&quot;, &quot;T&quot;), each = 2), `p(W)`= c(0, 0.9, 0.9, 0.99) ) df.rain_given_sprinkler_and_rain %&gt;% kable() %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F, font_size = 20) S R p(W) F F 0.00 T F 0.90 F T 0.90 T T 0.99 16.5 Controlling for variables 16.5.1 Illustration of the d-separation algorithm Question: Are D and E independent? 16.5.1.1 Full DAG g = grViz(&quot; digraph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node a [label = &#39;A&#39; pos = &#39;0,0!&#39;] b [label = &#39;B&#39; pos = &#39;2,0!&#39;] c [label = &#39;C&#39; pos = &#39;1,-1!&#39;] d [label = &#39;D&#39; pos = &#39;0,-2!&#39;] e [label = &#39;E&#39; pos = &#39;2,-2!&#39;] f [label = &#39;F&#39; pos = &#39;1,-3!&#39;] g [label = &#39;G&#39; pos = &#39;0,-4!&#39;] # edges between nodes edge [color = black] a -&gt; c b -&gt; c c -&gt; {d e} d -&gt; f f -&gt; g # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/dag.pdf&quot;) # show plot g 16.5.1.2 Draw the ancestral graph g = grViz(&quot; digraph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node a [label = &#39;A&#39; pos = &#39;0,0!&#39;] b [label = &#39;B&#39; pos = &#39;2,0!&#39;] c [label = &#39;C&#39; pos = &#39;1,-1!&#39;] d [label = &#39;D&#39; pos = &#39;0,-2!&#39;] e [label = &#39;E&#39; pos = &#39;2,-2!&#39;] # edges between nodes edge [color = black] a -&gt; c b -&gt; c c -&gt; {d e} # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/ancestral_graph.pdf&quot;) # show plot g 16.5.1.3 “Moralize” the ancestral graph by “marrying” any parents, and disorient by replacing arrows with edges g = grViz(&quot; graph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node a [label = &#39;A&#39; pos = &#39;0,0!&#39;] b [label = &#39;B&#39; pos = &#39;2,0!&#39;] c [label = &#39;C&#39; pos = &#39;1,-1!&#39;] d [label = &#39;D&#39; pos = &#39;0,-2!&#39;] e [label = &#39;E&#39; pos = &#39;2,-2!&#39;] # edges between nodes edge [color = black] a -- c b -- c c -- {d e} edge [color = black] a -- b # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/moralize_and_disorient.pdf&quot;) # show plot g For the case in which we check whether D and E are independent conditioned on C g = grViz(&quot; graph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node a [label = &#39;A&#39; pos = &#39;0,0!&#39;] b [label = &#39;B&#39; pos = &#39;2,0!&#39;] d [label = &#39;D&#39; pos = &#39;0,-2!&#39;] e [label = &#39;E&#39; pos = &#39;2,-2!&#39;] # edges between nodes edge [color = black] edge [color = black] a -- b # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) ## export as pdf #g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/moralize_and_disorient2.pdf&quot;) # show plot g 16.5.2 Good controls 16.5.2.1 Common cause (with direct link between X and Y) 16.5.2.1.1 DAG g = grViz(&quot; digraph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node x [label = &#39;X&#39; pos = &#39;0,0!&#39;] y [label = &#39;Y&#39; pos = &#39;2,0!&#39;] z [label = &#39;Z&#39; pos = &#39;1,1!&#39;, fontcolor = &#39;red&#39;] # edges between nodes edge [color = black] x -&gt; y z -&gt; {x y} # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/common_cause1.pdf&quot;) # show plot g 16.5.2.1.2 Regression set.seed(1) n = 1000 b_zx = 2 b_xy = 2 b_zy = 2 sd = 1 df = tibble(z = rnorm(n = n, sd = sd), x = b_zx * z + rnorm(n = n, sd = sd), y = b_zy * z + b_xy * x + rnorm(n = n, sd = sd)) # without control lm(formula = y ~ x, data = df) %&gt;% summary() Call: lm(formula = y ~ x, data = df) Residuals: Min 1Q Median 3Q Max -4.6011 -0.9270 -0.0506 0.9711 4.0454 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.02449 0.04389 0.558 0.577 x 2.82092 0.01890 149.225 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.388 on 998 degrees of freedom Multiple R-squared: 0.9571, Adjusted R-squared: 0.9571 F-statistic: 2.227e+04 on 1 and 998 DF, p-value: &lt; 2.2e-16 # with control lm(formula = y ~ x + z, data = df) %&gt;% summary() Call: lm(formula = y ~ x + z, data = df) Residuals: Min 1Q Median 3Q Max -3.6151 -0.6564 -0.0223 0.6815 2.8132 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.01624 0.03260 0.498 0.618 x 2.02202 0.03135 64.489 &lt;2e-16 *** z 2.00501 0.07036 28.497 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.031 on 997 degrees of freedom Multiple R-squared: 0.9764, Adjusted R-squared: 0.9763 F-statistic: 2.059e+04 on 2 and 997 DF, p-value: &lt; 2.2e-16 16.5.2.1.3 Moralize and disorient the ancestral graph g = grViz(&quot; graph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node x [label = &#39;X&#39; pos = &#39;0,0!&#39;] y [label = &#39;Y&#39; pos = &#39;2,0!&#39;] z [label = &#39;Z&#39; pos = &#39;1,1!&#39;, fontcolor = &#39;red&#39;] # edges between nodes edge [color = black] x -- y z -- {x y} # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/common_cause1_undirected.pdf&quot;) # # rsvg_pdf(&quot;figures/common_cause1_undirected2.pdf&quot;) # show plot g 16.5.2.2 Common cause (without direct link between X and Y) 16.5.2.2.1 DAG g = grViz(&quot; digraph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node x [label = &#39;X&#39; pos = &#39;0,0!&#39;] y [label = &#39;Y&#39; pos = &#39;2,0!&#39;] z [label = &#39;Z&#39; pos = &#39;1,1!&#39;, fontcolor = &#39;red&#39;] # edges between nodes edge [color = black] z -&gt; {x y} # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/common_cause2.pdf&quot;) # show plot g 16.5.2.2.2 Regression set.seed(1) n = 1000 b_zx = 2 b_zy = 2 sd = 1 df = tibble(z = rnorm(n = n, sd = sd), x = b_zx * z + rnorm(n = n, sd = sd), y = b_zy * z + rnorm(n = n, sd = sd)) # without control lm(formula = y ~ x, data = df) %&gt;% summary() Call: lm(formula = y ~ x, data = df) Residuals: Min 1Q Median 3Q Max -4.6011 -0.9270 -0.0506 0.9711 4.0454 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.02449 0.04389 0.558 0.577 x 0.82092 0.01890 43.426 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.388 on 998 degrees of freedom Multiple R-squared: 0.6539, Adjusted R-squared: 0.6536 F-statistic: 1886 on 1 and 998 DF, p-value: &lt; 2.2e-16 # with control lm(formula = y ~ x + z, data = df) %&gt;% summary() Call: lm(formula = y ~ x + z, data = df) Residuals: Min 1Q Median 3Q Max -3.6151 -0.6564 -0.0223 0.6815 2.8132 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.01624 0.03260 0.498 0.618 x 0.02202 0.03135 0.702 0.483 z 2.00501 0.07036 28.497 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.031 on 997 degrees of freedom Multiple R-squared: 0.8093, Adjusted R-squared: 0.8089 F-statistic: 2115 on 2 and 997 DF, p-value: &lt; 2.2e-16 16.5.3 Bad controls 16.5.3.1 Common effect 16.5.3.1.1 DAG g = grViz(&quot; digraph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node x [label = &#39;X&#39; pos = &#39;0,0!&#39;] y [label = &#39;Y&#39; pos = &#39;2,0!&#39;] z [label = &#39;Z&#39; pos = &#39;1,-1!&#39;, fontcolor = &#39;red&#39;] # edges between nodes edge [color = black] x -&gt; z y -&gt; z # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/common_effect.pdf&quot;) # show plot g 16.5.3.1.2 Regression set.seed(1) n = 1000 b_xz = 2 b_yz = 2 sd = 1 df = tibble(x = rnorm(n = n, sd = sd), y = rnorm(n = n, sd = sd), z = x * b_xz + y * b_yz + rnorm(n = n, sd = sd)) # without control lm(formula = y ~ x, data = df) %&gt;% summary() Call: lm(formula = y ~ x, data = df) Residuals: Min 1Q Median 3Q Max -3.2484 -0.6720 -0.0138 0.7554 3.6443 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) -0.016187 0.032905 -0.492 0.623 x 0.006433 0.031809 0.202 0.840 Residual standard error: 1.04 on 998 degrees of freedom Multiple R-squared: 4.098e-05, Adjusted R-squared: -0.000961 F-statistic: 0.0409 on 1 and 998 DF, p-value: 0.8398 # with control lm(formula = y ~ x + z, data = df) %&gt;% summary() Call: lm(formula = y ~ x + z, data = df) Residuals: Min 1Q Median 3Q Max -1.35547 -0.30016 0.00298 0.31119 1.73408 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) -0.009608 0.014477 -0.664 0.507 x -0.816164 0.018936 -43.102 &lt;2e-16 *** z 0.398921 0.006186 64.489 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 0.4578 on 997 degrees of freedom Multiple R-squared: 0.8066, Adjusted R-squared: 0.8062 F-statistic: 2079 on 2 and 997 DF, p-value: &lt; 2.2e-16 16.5.3.1.3 Moralize and disorient the ancestral graph g = grViz(&quot; graph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node x [label = &#39;X&#39; pos = &#39;0,0!&#39;] y [label = &#39;Y&#39; pos = &#39;2,0!&#39;] z [label = &#39;Z&#39; pos = &#39;1,-1!&#39;, fontcolor = &#39;red&#39;] # edges between nodes edge [color = black] x -- y x -- z y -- z # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/common_effect_undirected1.pdf&quot;) # rsvg_pdf(&quot;figures/common_effect_undirected2.pdf&quot;) # show plot g 16.5.3.2 Causal chain 1 16.5.3.2.1 DAG g = grViz(&quot; digraph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node x [label = &#39;X&#39; pos = &#39;0,0!&#39;] y [label = &#39;Y&#39; pos = &#39;2,0!&#39;] z [label = &#39;Z&#39; pos = &#39;1, 0!&#39;, fontcolor = &#39;red&#39;] # edges between nodes edge [color = black] x -&gt; z z -&gt; y # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/causal_chain.pdf&quot;) # show plot g 16.5.3.2.2 Regression set.seed(1) n = 20 b_xz = 2 b_zy = 2 sd = 1 df = tibble(x = rnorm(n = n, sd = sd), z = x * b_xz + rnorm(n = n, sd = sd), y = z * b_zy + rnorm(n = n, sd = sd)) # without control lm(formula = y ~ x, data = df) %&gt;% summary() Call: lm(formula = y ~ x, data = df) Residuals: Min 1Q Median 3Q Max -3.336 -1.208 0.209 1.220 3.189 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.1547 0.3982 0.388 0.702 x 3.8488 0.4374 8.799 6.15e-08 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.741 on 18 degrees of freedom Multiple R-squared: 0.8114, Adjusted R-squared: 0.8009 F-statistic: 77.43 on 1 and 18 DF, p-value: 6.154e-08 # with control lm(formula = y ~ x + z, data = df) %&gt;% summary() Call: lm(formula = y ~ x + z, data = df) Residuals: Min 1Q Median 3Q Max -1.35959 -0.56643 -0.06193 0.48088 1.80592 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.09559 0.18177 0.526 0.606 x 0.64724 0.43278 1.496 0.153 z 1.78614 0.21425 8.337 2.07e-07 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 0.7943 on 17 degrees of freedom Multiple R-squared: 0.9629, Adjusted R-squared: 0.9586 F-statistic: 220.8 on 2 and 17 DF, p-value: 6.868e-13 16.5.3.3 Causal chain 2 16.5.3.3.1 DAG g = grViz(&quot; digraph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node x [label = &#39;X&#39; pos = &#39;0,0!&#39;] y [label = &#39;Y&#39; pos = &#39;1,0!&#39;] z [label = &#39;Z&#39; pos = &#39;2, 0!&#39;, fontcolor = &#39;red&#39;] # edges between nodes edge [color = black] x -&gt; y y -&gt; z # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/causal_chain2.pdf&quot;) # show plot g 16.5.3.3.2 Regression set.seed(1) n = 20 b_xy = 2 b_yz = 2 sd = 1 df = tibble(x = rnorm(n = n, sd = sd), y = x * b_xy + rnorm(n = n, sd = sd), z = y * b_yz + rnorm(n = n, sd = sd),) # without control lm(formula = y ~ x, data = df) %&gt;% summary() Call: lm(formula = y ~ x, data = df) Residuals: Min 1Q Median 3Q Max -1.69133 -0.43739 -0.07132 0.68033 1.63937 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.03307 0.19981 0.166 0.87 x 1.79245 0.21951 8.166 1.83e-07 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 0.8738 on 18 degrees of freedom Multiple R-squared: 0.7874, Adjusted R-squared: 0.7756 F-statistic: 66.68 on 1 and 18 DF, p-value: 1.827e-07 # with control lm(formula = y ~ x + z, data = df) %&gt;% summary() Call: lm(formula = y ~ x + z, data = df) Residuals: Min 1Q Median 3Q Max -0.9023 -0.2316 0.1173 0.2396 0.6319 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) -0.03650 0.09153 -0.399 0.695 x 0.06113 0.23056 0.265 0.794 z 0.44983 0.05396 8.337 2.07e-07 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 0.3986 on 17 degrees of freedom Multiple R-squared: 0.9582, Adjusted R-squared: 0.9533 F-statistic: 195 on 2 and 17 DF, p-value: 1.896e-12 16.5.3.4 Bias amplification 16.5.3.4.1 DAG g = grViz(&quot; digraph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node x [label = &#39;X&#39; pos = &#39;0,0!&#39;] y [label = &#39;Y&#39; pos = &#39;2,0!&#39;] z [label = &#39;Z&#39; pos = &#39;-1, 1!&#39;, fontcolor = &#39;red&#39;] u [label = &#39;U&#39; pos = &#39;1, 1!&#39;, fillcolor = &#39;white&#39;] # edges between nodes edge [color = black] x -&gt; y z -&gt; x u -&gt; {x y} # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) # # export as pdf # g %&gt;% # export_svg %&gt;% # charToRaw %&gt;% # rsvg_pdf(&quot;figures/bias_amplification.pdf&quot;) # show plot g 16.5.3.5 Regression set.seed(1) n = 20 b_xy = 2 b_ux = 2 b_uy = 2 b_zx = 2 sd = 1 df = tibble(u = rnorm(n = n, sd = sd), z = rnorm(n = n, sd = sd), x = u * b_ux + z * b_zx + rnorm(n = n, sd = sd), y = u * b_uy + x * b_xy + rnorm(n = n, sd = sd)) # without control lm(formula = y ~ x, data = df) %&gt;% summary() Call: lm(formula = y ~ x, data = df) Residuals: Min 1Q Median 3Q Max -2.2838 -0.8662 -0.2281 0.7201 3.1619 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.1903 0.3282 0.58 0.569 x 2.5771 0.1375 18.74 2.96e-13 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.434 on 18 degrees of freedom Multiple R-squared: 0.9512, Adjusted R-squared: 0.9485 F-statistic: 351.1 on 1 and 18 DF, p-value: 2.961e-13 # with control lm(formula = y ~ x + z, data = df) %&gt;% summary() Call: lm(formula = y ~ x + z, data = df) Residuals: Min 1Q Median 3Q Max -1.9114 -0.4876 -0.1044 0.6333 1.8935 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.07431 0.24564 0.303 0.76594 x 2.78984 0.11553 24.147 1.35e-14 *** z -1.25270 0.31719 -3.949 0.00103 ** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.066 on 17 degrees of freedom Multiple R-squared: 0.9746, Adjusted R-squared: 0.9716 F-statistic: 325.7 on 2 and 17 DF, p-value: 2.793e-14 16.6 Mediation Figure 16.1: Basic mediation model. c = the total effect of X on Y; c = c’ + ab; c’ = the direct effect of X on Y after controlling for M; c’ = c - ab; ab = indirect effect of X on Y. Mediation tests whether the effects of X (the independent variable) on Y (the dependent variable) operate through a third variable, M (the mediator). In this way, mediators explain the causal relationship between two variables or “how” the relationship works, making it a very popular method in psychological research. Figure 16.1 shows the standard mediation model. Perfect mediation occurs when the effect of X on Y decreases to 0 with M in the model. Partial mediation occurs when the effect of X on Y decreases by a nontrivial amount (the actual amount is up for debate) with M in the model. Important: Both mediation and moderation assume that the DV did not CAUSE the mediator/moderator. 16.6.1 Generate data # make example reproducible set.seed(123) # number of participants n = 100 # generate data df.mediation = tibble(x = rnorm(n, 75, 7), # grades m = 0.7 * x + rnorm(n, 0, 5), # self-esteem y = 0.4 * m + rnorm(n, 0, 5)) # happiness 16.6.2 Method 1: Baron &amp; Kenny’s (1986) indirect effect method The Baron and Kenny (1986) method is among the original methods for testing for mediation but tends to have low statistical power. It is covered in this chapter because it provides a very clear approach to establishing relationships between variables and is still occassionally requested by reviewers. The three steps: Estimate the relationship between \\(X\\) and \\(Y\\) (hours since dawn on degree of wakefulness). Path “c” must be significantly different from 0; must have a total effect between the IV &amp; DV. Estimate the relationship between \\(X\\) and \\(M\\) (hours since dawn on coffee consumption). Path “a” must be significantly different from 0; IV and mediator must be related. Estimate the relationship between \\(M\\) and \\(Y\\) controlling for \\(X\\) (coffee consumption on wakefulness, controlling for hours since dawn). Path “b” must be significantly different from 0; mediator and DV must be related. The effect of \\(X\\) on \\(Y\\) decreases with the inclusion of \\(M\\) in the model. 16.6.2.1 Total effect Total effect of X on Y (not controlling for M). # fit the model fit.y_x = lm(formula = y ~ 1 + x, data = df.mediation) # summarize the results fit.y_x %&gt;% summary() Call: lm(formula = y ~ 1 + x, data = df.mediation) Residuals: Min 1Q Median 3Q Max -10.917 -3.738 -0.259 2.910 12.540 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 8.78300 6.16002 1.426 0.1571 x 0.16899 0.08116 2.082 0.0399 * --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 5.16 on 98 degrees of freedom Multiple R-squared: 0.04237, Adjusted R-squared: 0.0326 F-statistic: 4.336 on 1 and 98 DF, p-value: 0.03993 16.6.2.2 Path a fit.m_x = lm(formula = m ~ 1 + x, data = df.mediation) fit.m_x %&gt;% summary() Call: lm(formula = m ~ 1 + x, data = df.mediation) Residuals: Min 1Q Median 3Q Max -9.5367 -3.4175 -0.4375 2.9032 16.4520 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 2.29696 5.79432 0.396 0.693 x 0.66252 0.07634 8.678 8.87e-14 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 4.854 on 98 degrees of freedom Multiple R-squared: 0.4346, Adjusted R-squared: 0.4288 F-statistic: 75.31 on 1 and 98 DF, p-value: 8.872e-14 16.6.2.3 Path b and c’ Effect of M on Y controlling for X. fit.y_mx = lm(formula = y ~ 1 + m + x, data = df.mediation) fit.y_mx %&gt;% summary() Call: lm(formula = y ~ 1 + m + x, data = df.mediation) Residuals: Min 1Q Median 3Q Max -9.3651 -3.3037 -0.6222 3.1068 10.3991 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 7.80952 5.68297 1.374 0.173 m 0.42381 0.09899 4.281 4.37e-05 *** x -0.11179 0.09949 -1.124 0.264 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 4.756 on 97 degrees of freedom Multiple R-squared: 0.1946, Adjusted R-squared: 0.1779 F-statistic: 11.72 on 2 and 97 DF, p-value: 2.771e-05 16.6.2.4 Interpretation fit.y_x %&gt;% tidy() %&gt;% mutate(path = &quot;c&quot;) %&gt;% bind_rows(fit.m_x %&gt;% tidy() %&gt;% mutate(path = &quot;a&quot;), fit.y_mx %&gt;% tidy() %&gt;% mutate(path = c(&quot;(Intercept)&quot;, &quot;b&quot;, &quot;c&#39;&quot;))) %&gt;% filter(term != &quot;(Intercept)&quot;) %&gt;% mutate(significance = p.value &lt; .05, dv = ifelse(path %in% c(&quot;c&#39;&quot;, &quot;b&quot;), &quot;y&quot;, &quot;m&quot;)) %&gt;% select(path, iv = term, dv, estimate, p.value, significance) # A tibble: 4 × 6 path iv dv estimate p.value significance &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;lgl&gt; 1 c x m 0.169 3.99e- 2 TRUE 2 a x m 0.663 8.87e-14 TRUE 3 b m y 0.424 4.37e- 5 TRUE 4 c&#39; x y -0.112 2.64e- 1 FALSE Here we find that our total effect model shows a significant positive relationship between hours since dawn (X) and wakefulness (Y). Our Path A model shows that hours since down (X) is also positively related to coffee consumption (M). Our Path B model then shows that coffee consumption (M) positively predicts wakefulness (Y) when controlling for hours since dawn (X). Since the relationship between hours since dawn and wakefulness is no longer significant when controlling for coffee consumption, this suggests that coffee consumption does in fact mediate this relationship. However, this method alone does not allow for a formal test of the indirect effect so we don’t know if the change in this relationship is truly meaningful. 16.6.3 Method 2: Sobel Test The Sobel Test tests whether the indirect effect from X via M to Y is significant. # run the sobel test fit.sobel = sobel(pred = df.mediation$x, med = df.mediation$m, out = df.mediation$y) # calculate the p-value (1 - pnorm(fit.sobel$z.value))*2 [1] 0.0001233403 The relationship between “hours since dawn” and “wakefulness” is significantly mediated by “coffee consumption”. The Sobel Test is largely considered an outdated method since it assumes that the indirect effect (ab) is normally distributed and tends to only have adequate power with large sample sizes. Thus, again, it is highly recommended to use the mediation bootstrapping method instead. 16.6.4 Method 3: Bootstrapping The “mediation” packages uses the more recent bootstrapping method of Preacher and Hayes (2004) to address the power limitations of the Sobel Test. This method does not require that the data are normally distributed, and is particularly suitable for small sample sizes. # bootstrapped mediation fit.mediation = mediate(model.m = fit.m_x, model.y = fit.y_mx, treat = &quot;x&quot;, mediator = &quot;m&quot;, boot = T) Running nonparametric bootstrap # summarize results summary(fit.mediation) Causal Mediation Analysis Nonparametric Bootstrap Confidence Intervals with the Percentile Method Estimate 95% CI Lower 95% CI Upper p-value ACME 0.28078 0.14133 0.43 &lt;2e-16 *** ADE -0.11179 -0.30293 0.11 0.274 Total Effect 0.16899 -0.00539 0.35 0.056 . Prop. Mediated 1.66151 -1.91644 10.16 0.056 . --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Sample Size Used: 100 Simulations: 1000 ACME = Average causal mediation effect ADE = Average direct effect Total effect = ACME + ADE Plot the results: plot(fit.mediation) 16.6.4.1 Interpretation The mediate() function gives us our Average Causal Mediation Effects (ACME), our Average Direct Effects (ADE), our combined indirect and direct effects (Total Effect), and the ratio of these estimates (Prop. Mediated). The ACME here is the indirect effect of M (total effect - direct effect) and thus this value tells us if our mediation effect is significant. 16.6.5 Be careful about mediation! Different causal structures can lead to the same “mediation” effects. So it’s difficult to tell from a pattern of regression results what the relationships between variables is. Here are three different causal structures that lead to similar effects set.seed(1) n = 100 # number of observations # causal chain df.causal_chain = tibble(x = rnorm(n, 0, 1), z = 2 * x + rnorm(n, 0, 1), y = 2 * z + rnorm(n, 0, 1)) # visualize the graph g = grViz(&quot; digraph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node x [label = &#39;X&#39; pos = &#39;0,0!&#39;] z [label = &#39;Z&#39; pos = &#39;1,1!&#39;] y [label = &#39;Y&#39; pos = &#39;2,0!&#39;] # edges between nodes edge [color = black] x -&gt; z z -&gt; y # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) g # fit models fit.yx = lm(formula = y ~ 1 + x, data = df.causal_chain) fit.zx = lm(formula = z ~ 1 + x, data = df.causal_chain) fit.yxz = lm(formula = y ~ 1 + x + z, data = df.causal_chain) # print models summary(fit.yx) Call: lm(formula = y ~ 1 + x, data = df.causal_chain) Residuals: Min 1Q Median 3Q Max -4.0180 -1.2143 -0.4065 1.1288 6.0959 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) -0.04802 0.21582 -0.222 0.824 x 4.01905 0.23972 16.766 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 2.142 on 98 degrees of freedom Multiple R-squared: 0.7415, Adjusted R-squared: 0.7389 F-statistic: 281.1 on 1 and 98 DF, p-value: &lt; 2.2e-16 summary(fit.zx) Call: lm(formula = z ~ 1 + x, data = df.causal_chain) Residuals: Min 1Q Median 3Q Max -1.8768 -0.6138 -0.1395 0.5394 2.3462 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) -0.03769 0.09699 -0.389 0.698 x 1.99894 0.10773 18.556 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 0.9628 on 98 degrees of freedom Multiple R-squared: 0.7784, Adjusted R-squared: 0.7762 F-statistic: 344.3 on 1 and 98 DF, p-value: &lt; 2.2e-16 summary(fit.yxz) Call: lm(formula = y ~ 1 + x + z, data = df.causal_chain) Residuals: Min 1Q Median 3Q Max -2.94359 -0.43645 0.00202 0.63692 2.63941 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.02535 0.10519 0.241 0.810 x 0.12804 0.24804 0.516 0.607 z 1.94653 0.10948 17.780 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.043 on 97 degrees of freedom Multiple R-squared: 0.9393, Adjusted R-squared: 0.9381 F-statistic: 750.6 on 2 and 97 DF, p-value: &lt; 2.2e-16 # mediation analysis fit.m = mediate(model.m = fit.zx, model.y = fit.yxz, treat = &quot;x&quot;, mediator = &quot;z&quot;, boot = T) Running nonparametric bootstrap summary(fit.m) Causal Mediation Analysis Nonparametric Bootstrap Confidence Intervals with the Percentile Method Estimate 95% CI Lower 95% CI Upper p-value ACME 3.891 3.390 4.51 &lt;2e-16 *** ADE 0.128 -0.395 0.62 0.64 Total Effect 4.019 3.585 4.44 &lt;2e-16 *** Prop. Mediated 0.968 0.849 1.11 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Sample Size Used: 100 Simulations: 1000 set.seed(1) n = 100 # number of observations # common cause df.common_cause = tibble(z = rnorm(n, 0, 1), x = 2 * z + rnorm(n, 0, 1), y = 2 * z + rnorm(n, 0, 1)) # visualize the graph g = grViz(&quot; digraph neato { graph[layout = neato] # general settings for all nodes node [ shape = circle, style = filled, color = black, label = &#39;&#39; fontname = &#39;Helvetica&#39;, fontsize = 16, fillcolor = lightblue ] # labels for each node x [label = &#39;X&#39; pos = &#39;0,0!&#39;] z [label = &#39;Z&#39; pos = &#39;1,1!&#39;] y [label = &#39;Y&#39; pos = &#39;2,0!&#39;] # edges between nodes edge [color = black] z -&gt; x z -&gt; y # direction in which arrows are drawn (from left to right) rankdir = LR } &quot;) g # fit models fit.yx = lm(formula = y ~ 1 + x, data = df.common_cause) fit.zx = lm(formula = z ~ 1 + x, data = df.common_cause) fit.yxz = lm(formula = y ~ 1 + x + z, data = df.common_cause) # print models summary(fit.yx) Call: lm(formula = y ~ 1 + x, data = df.common_cause) Residuals: Min 1Q Median 3Q Max -3.4514 -0.9780 -0.1073 1.0144 3.3204 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.10793 0.13821 0.781 0.437 x 0.77525 0.06799 11.402 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.377 on 98 degrees of freedom Multiple R-squared: 0.5702, Adjusted R-squared: 0.5658 F-statistic: 130 on 1 and 98 DF, p-value: &lt; 2.2e-16 summary(fit.zx) Call: lm(formula = z ~ 1 + x, data = df.common_cause) Residuals: Min 1Q Median 3Q Max -0.90848 -0.28101 0.06274 0.24570 0.85736 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.03880 0.04266 0.91 0.365 x 0.38942 0.02099 18.56 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 0.4249 on 98 degrees of freedom Multiple R-squared: 0.7784, Adjusted R-squared: 0.7762 F-statistic: 344.3 on 1 and 98 DF, p-value: &lt; 2.2e-16 summary(fit.yxz) Call: lm(formula = y ~ 1 + x + z, data = df.common_cause) Residuals: Min 1Q Median 3Q Max -2.94359 -0.43645 0.00202 0.63692 2.63941 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.02535 0.10519 0.241 0.810 x -0.05347 0.10948 -0.488 0.626 z 2.12804 0.24804 8.580 1.55e-13 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.043 on 97 degrees of freedom Multiple R-squared: 0.7556, Adjusted R-squared: 0.7506 F-statistic: 150 on 2 and 97 DF, p-value: &lt; 2.2e-16 # mediation analysis fit.m = mediate(model.m = fit.zx, model.y = fit.yxz, treat = &quot;x&quot;, mediator = &quot;z&quot;, boot = T) Running nonparametric bootstrap summary(fit.m) Causal Mediation Analysis Nonparametric Bootstrap Confidence Intervals with the Percentile Method Estimate 95% CI Lower 95% CI Upper p-value ACME 0.8287 0.6065 1.04 &lt;2e-16 *** ADE -0.0535 -0.2675 0.16 0.56 Total Effect 0.7752 0.6353 0.90 &lt;2e-16 *** Prop. Mediated 1.0690 0.8134 1.37 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Sample Size Used: 100 Simulations: 1000 Both models here lead to the same pattern of “mediation”. However, only the first one is a chain: X–&gt;Z–&gt;Y. The second one is a common cause X&lt;–Z–&gt;Y (where X does not cause Y). 16.7 Moderation Figure 16.2: Basic moderation model. Moderation can be tested by looking for significant interactions between the moderating variable (Z) and the IV (X). Notably, it is important to mean center both your moderator and your IV to reduce multicolinearity and make interpretation easier. 16.7.1 Generate data # make example reproducible set.seed(123) # number of participants n = 100 df.moderation = tibble(x = abs(rnorm(n, 6, 4)), # hours of sleep x1 = abs(rnorm(n, 60, 30)), # adding some systematic variance to our DV z = rnorm(n, 30, 8), # ounces of coffee consumed y = abs((-0.8 * x) * (0.2 * z) - 0.5 * x - 0.4 * x1 + 10 + rnorm(n, 0, 3))) # attention Paid 16.7.2 Moderation analysis # scale the predictors df.moderation = df.moderation %&gt;% mutate_at(vars(x, z), ~ scale(.)[,]) # run regression model with interaction fit.moderation = lm(formula = y ~ 1 + x * z, data = df.moderation) # summarize result fit.moderation %&gt;% summary() Call: lm(formula = y ~ 1 + x * z, data = df.moderation) Residuals: Min 1Q Median 3Q Max -21.466 -8.972 -0.233 6.180 38.051 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 48.544 1.173 41.390 &lt; 2e-16 *** x 17.863 1.196 14.936 &lt; 2e-16 *** z 8.393 1.181 7.108 2.08e-10 *** x:z 6.094 1.077 5.656 1.59e-07 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 11.65 on 96 degrees of freedom Multiple R-squared: 0.7661, Adjusted R-squared: 0.7587 F-statistic: 104.8 on 3 and 96 DF, p-value: &lt; 2.2e-16 16.7.2.1 Visualize result # generate data grid with three levels of the moderator df.newdata = df.moderation %&gt;% expand(x = c(min(x), max(x)), z = c(mean(z) - sd(z), mean(z), mean(z) + sd(z))) %&gt;% mutate(moderator = rep(c(&quot;low&quot;, &quot;average&quot;, &quot;high&quot;), nrow(.)/3)) # predictions for the three levels of the moderator df.prediction = fit.moderation %&gt;% augment(newdata = df.newdata) %&gt;% mutate(moderator = factor(moderator, levels = c(&quot;high&quot;, &quot;average&quot;, &quot;low&quot;))) # visualize the result df.moderation %&gt;% ggplot(aes(x = x, y = y)) + geom_point() + geom_line(data = df.prediction, mapping = aes(y = .fitted, group = moderator, color = moderator), size = 1) + labs(x = &quot;hours of sleep (z-scored)&quot;, y = &quot;attention paid&quot;, color = &quot;coffee consumed&quot;) + scale_color_brewer(palette = &quot;Set1&quot;) df.prediction %&gt;% head(9) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) x z moderator .fitted -1.83 -1 low 18.58 -1.83 0 average 15.80 -1.83 1 high 13.02 2.41 -1 low 68.52 2.41 0 average 91.60 2.41 1 high 114.68 16.8 Additional resources 16.8.1 Books Introduction to Mediation, Moderation, and Conditional Process Analysis (Second Edition): A Regression-Based Approach Recoded with BRMS and Tidyverse 16.8.2 Tutorials R tutorial on mediation and moderation R tutorial on moderated mediation Path analysis with brms Understanding d-separation 16.8.3 Misc Judea Pearl on good, bad, and neutral controls Mike Frank on covariates vs. confounds 16.9 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 rsvg_2.3.1 DiagrammeRsvg_0.1 DiagrammeR_1.0.9 [13] broom_0.8.0 multilevel_2.7 nlme_3.1-157 mediation_4.5.0 [17] sandwich_3.0-1 mvtnorm_1.1-3 Matrix_1.4-1 MASS_7.3-57 [21] janitor_2.1.0 kableExtra_1.3.4 knitr_1.39 loaded via a namespace (and not attached): [1] fs_1.5.2 lubridate_1.8.0 webshot_0.5.3 [4] RColorBrewer_1.1-3 httr_1.4.3 tools_4.1.2 [7] backports_1.4.1 bslib_0.3.1 utf8_1.2.2 [10] R6_2.5.1 rpart_4.1.16 Hmisc_4.7-0 [13] DBI_1.1.2 colorspace_2.0-3 nnet_7.3-17 [16] withr_2.5.0 tidyselect_1.1.2 gridExtra_2.3 [19] curl_4.3.2 compiler_4.1.2 cli_3.3.0 [22] rvest_1.0.2 htmlTable_2.4.0 xml2_1.3.3 [25] labeling_0.4.2 bookdown_0.26 sass_0.4.1 [28] scales_1.2.0 checkmate_2.1.0 systemfonts_1.0.4 [31] digest_0.6.29 foreign_0.8-82 minqa_1.2.4 [34] rmarkdown_2.14 svglite_2.1.0 base64enc_0.1-3 [37] jpeg_0.1-9 pkgconfig_2.0.3 htmltools_0.5.2 [40] lme4_1.1-29 highr_0.9 dbplyr_2.1.1 [43] fastmap_1.1.0 readxl_1.4.0 htmlwidgets_1.5.4 [46] rlang_1.0.2 rstudioapi_0.13 farver_2.1.0 [49] visNetwork_2.1.0 jquerylib_0.1.4 generics_0.1.2 [52] zoo_1.8-10 jsonlite_1.8.0 magrittr_2.0.3 [55] Formula_1.2-4 Rcpp_1.0.8.3 munsell_0.5.0 [58] fansi_1.0.3 lifecycle_1.0.1 stringi_1.7.6 [61] yaml_2.3.5 snakecase_0.11.0 grid_4.1.2 [64] crayon_1.5.1 lattice_0.20-45 haven_2.5.0 [67] splines_4.1.2 hms_1.1.1 pillar_1.7.0 [70] boot_1.3-28 lpSolve_5.6.15 reprex_2.0.1 [73] glue_1.6.2 evaluate_0.15 V8_4.1.0 [76] latticeExtra_0.6-29 modelr_0.1.8 data.table_1.14.2 [79] tzdb_0.3.0 png_0.1-7 vctrs_0.4.1 [82] nloptr_2.0.0 cellranger_1.1.0 gtable_0.3.0 [85] assertthat_0.2.1 xfun_0.30 survival_3.3-1 [88] viridisLite_0.4.0 cluster_2.1.3 ellipsis_0.3.2 References "],["linear-mixed-effects-models-1.html", "Chapter 17 Linear mixed effects models 1 17.1 Learning goals 17.2 Load packages and set plotting theme 17.3 Dependence 17.4 Additional resources 17.5 Session info", " Chapter 17 Linear mixed effects models 1 17.1 Learning goals Understanding sources of dependence in data. fixed effects vs. random effects. lmer() syntax in R. Understanding the lmer() summary. Simulating data from an lmer(). 17.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;broom.mixed&quot;) # for tidying up linear models library(&quot;patchwork&quot;) # for making figure panels library(&quot;lme4&quot;) # for linear mixed effects models library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 17.3 Dependence Let’s generate a data set in which two observations from the same participants are dependent, and then let’s also shuffle this data set to see whether taking into account the dependence in the data matters. # make example reproducible set.seed(1) df.dependence = tibble(participant = 1:20, condition1 = rnorm(20), condition2 = condition1 + rnorm(20, mean = 0.2, sd = 0.1)) %&gt;% mutate(condition2shuffled = sample(condition2)) # shuffles the condition label Let’s visualize the original and shuffled data set: df.plot = df.dependence %&gt;% pivot_longer(cols = -participant, names_to = &quot;condition&quot;, values_to = &quot;value&quot;) %&gt;% mutate(condition = str_replace(condition, &quot;condition&quot;, &quot;&quot;)) p1 = ggplot(data = df.plot %&gt;% filter(condition != &quot;2shuffled&quot;), mapping = aes(x = condition, y = value)) + geom_line(aes(group = participant), alpha = 0.3) + geom_point() + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;, shape = 21, fill = &quot;red&quot;, size = 4) + labs(title = &quot;original&quot;, tag = &quot;a)&quot;) p2 = ggplot(data = df.plot %&gt;% filter(condition != &quot;2&quot;), mapping = aes(x = condition, y = value)) + geom_line(aes(group = participant), alpha = 0.3) + geom_point() + stat_summary(fun = &quot;mean&quot;, geom = &quot;point&quot;, shape = 21, fill = &quot;red&quot;, size = 4) + labs(title = &quot;shuffled&quot;, tag = &quot;b)&quot;) p1 + p2 Let’s save the two original and shuffled data set as two separate data sets. # separate the data sets df.original = df.dependence %&gt;% pivot_longer(cols = -participant, names_to = &quot;condition&quot;, values_to = &quot;value&quot;) %&gt;% mutate(condition = str_replace(condition, &quot;condition&quot;, &quot;&quot;)) %&gt;% filter(condition != &quot;2shuffled&quot;) df.shuffled = df.dependence %&gt;% pivot_longer(cols = -participant, names_to = &quot;condition&quot;, values_to = &quot;value&quot;) %&gt;% mutate(condition = str_replace(condition, &quot;condition&quot;, &quot;&quot;)) %&gt;% filter(condition != &quot;2&quot;) Let’s run a linear model, and independent samples t-test on the original data set. # linear model (assuming independent samples) lm(formula = value ~ condition, data = df.original) %&gt;% summary() Call: lm(formula = value ~ condition, data = df.original) Residuals: Min 1Q Median 3Q Max -2.4100 -0.5530 0.1945 0.5685 1.4578 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.1905 0.2025 0.941 0.353 condition2 0.1994 0.2864 0.696 0.491 Residual standard error: 0.9058 on 38 degrees of freedom Multiple R-squared: 0.01259, Adjusted R-squared: -0.0134 F-statistic: 0.4843 on 1 and 38 DF, p-value: 0.4907 t.test(df.original$value[df.original$condition == &quot;1&quot;], df.original$value[df.original$condition == &quot;2&quot;], alternative = &quot;two.sided&quot;, paired = F) Welch Two Sample t-test data: df.original$value[df.original$condition == &quot;1&quot;] and df.original$value[df.original$condition == &quot;2&quot;] t = -0.69595, df = 37.99, p-value = 0.4907 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -0.7792396 0.3805339 sample estimates: mean of x mean of y 0.1905239 0.3898767 The mean difference between the conditions is extremely small, and non-significant (if we ignore the dependence in the data). Let’s fit a linear mixed effects model with a random intercept for each participant: # fit a linear mixed effects model lmer(formula = value ~ condition + (1 | participant), data = df.original) %&gt;% summary() Linear mixed model fit by REML [&#39;lmerMod&#39;] Formula: value ~ condition + (1 | participant) Data: df.original REML criterion at convergence: 17.3 Scaled residuals: Min 1Q Median 3Q Max -1.55996 -0.36399 -0.03341 0.34400 1.65823 Random effects: Groups Name Variance Std.Dev. participant (Intercept) 0.816722 0.90373 Residual 0.003796 0.06161 Number of obs: 40, groups: participant, 20 Fixed effects: Estimate Std. Error t value (Intercept) 0.19052 0.20255 0.941 condition2 0.19935 0.01948 10.231 Correlation of Fixed Effects: (Intr) condition2 -0.048 To test for whether condition is a significant predictor, we need to use our model comparison approach: # fit models fit.compact = lmer(formula = value ~ 1 + (1 | participant), data = df.original) fit.augmented = lmer(formula = value ~ condition + (1 | participant), data = df.original) # compare via Chisq-test anova(fit.compact, fit.augmented) refitting model(s) with ML (instead of REML) Data: df.original Models: fit.compact: value ~ 1 + (1 | participant) fit.augmented: value ~ condition + (1 | participant) npar AIC BIC logLik deviance Chisq Df Pr(&gt;Chisq) fit.compact 3 53.315 58.382 -23.6575 47.315 fit.augmented 4 17.849 24.605 -4.9247 9.849 37.466 1 9.304e-10 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 This result is identical to running a paired samples t-test: t.test(df.original$value[df.original$condition == &quot;1&quot;], df.original$value[df.original$condition == &quot;2&quot;], alternative = &quot;two.sided&quot;, paired = T) Paired t-test data: df.original$value[df.original$condition == &quot;1&quot;] and df.original$value[df.original$condition == &quot;2&quot;] t = -10.231, df = 19, p-value = 3.636e-09 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -0.2401340 -0.1585717 sample estimates: mean of the differences -0.1993528 But, unlike in the paired samples t-test, the linear mixed effects model explicitly models the variation between participants, and it’s a much more flexible approach for modeling dependence in data. Let’s fit a linear model and a linear mixed effects model to the original (non-shuffled) data. # model assuming independence fit.independent = lm(formula = value ~ 1 + condition, data = df.original) # model assuming dependence fit.dependent = lmer(formula = value ~ 1 + condition + (1 | participant), data = df.original) Let’s visualize the linear model’s predictions: # plot with predictions by fit.independent fit.independent %&gt;% augment() %&gt;% bind_cols(df.original %&gt;% select(participant)) %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = condition, y = value, group = participant)) + geom_point(alpha = 0.5) + geom_line(alpha = 0.5) + geom_point(aes(y = fitted), color = &quot;red&quot;) + geom_line(aes(y = fitted), color = &quot;red&quot;) And this is what the residuals look like: # make example reproducible set.seed(1) fit.independent %&gt;% augment() %&gt;% bind_cols(df.original %&gt;% select(participant)) %&gt;% clean_names() %&gt;% mutate(index = as.numeric(condition), index = index + runif(n(), min = -0.3, max = 0.3)) %&gt;% ggplot(data = ., mapping = aes(x = index, y = value, group = participant, color = condition)) + geom_point() + geom_smooth(method = &quot;lm&quot;, se = F, formula = &quot;y ~ 1&quot;, aes(group = condition)) + geom_segment(aes(xend = index, yend = fitted), alpha = 0.5) + scale_color_brewer(palette = &quot;Set1&quot;) + scale_x_continuous(breaks = 1:2, labels = 1:2) + labs(x = &quot;condition&quot;) + theme(legend.position = &quot;none&quot;) It’s clear from this residual plot, that fitting two separate lines (or points) is not much better than just fitting one line (or point). Let’s visualize the predictions of the linear mixed effects model: # plot with predictions by fit.independent fit.dependent %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = condition, y = value, group = participant)) + geom_point(alpha = 0.5) + geom_line(alpha = 0.5) + geom_point(aes(y = fitted), color = &quot;red&quot;) + geom_line(aes(y = fitted), color = &quot;red&quot;) Let’s compare the residuals of the linear model with that of the linear mixed effects model: # linear model p1 = fit.independent %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = fitted, y = resid)) + geom_point() + coord_cartesian(ylim = c(-2.5, 2.5)) # linear mixed effects model p2 = fit.dependent %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = fitted, y = resid)) + geom_point() + coord_cartesian(ylim = c(-2.5, 2.5)) p1 + p2 The residuals of the linear mixed effects model are much smaller. Let’s test whether taking the individual variation into account is worth it (statistically speaking). # fit models (without and with dependence) fit.compact = lm(formula = value ~ 1 + condition, data = df.original) fit.augmented = lmer(formula = value ~ 1 + condition + (1 | participant), data = df.original) # compare models # note: the lmer model has to be supplied first anova(fit.augmented, fit.compact) refitting model(s) with ML (instead of REML) Data: df.original Models: fit.compact: value ~ 1 + condition fit.augmented: value ~ 1 + condition + (1 | participant) npar AIC BIC logLik deviance Chisq Df Pr(&gt;Chisq) fit.compact 3 109.551 114.617 -51.775 103.551 fit.augmented 4 17.849 24.605 -4.925 9.849 93.701 1 &lt; 2.2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Yes, the likelihood of the data given the linear mixed effects model is significantly higher compared to its likelihood given the linear model. 17.4 Additional resources 17.4.1 Readings Linear mixed effects models tutorial by Bodo Winter 17.5 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 [4] purrr_0.3.4 readr_2.1.2 tidyr_1.2.0 [7] tibble_3.1.7 ggplot2_3.3.6 tidyverse_1.3.1 [10] lme4_1.1-29 Matrix_1.4-1 patchwork_1.1.1 [13] broom.mixed_0.2.9.4 janitor_2.1.0 kableExtra_1.3.4 [16] knitr_1.39 loaded via a namespace (and not attached): [1] nlme_3.1-157 fs_1.5.2 lubridate_1.8.0 webshot_0.5.3 [5] RColorBrewer_1.1-3 httr_1.4.3 tools_4.1.2 backports_1.4.1 [9] bslib_0.3.1 utf8_1.2.2 R6_2.5.1 DBI_1.1.2 [13] mgcv_1.8-40 colorspace_2.0-3 withr_2.5.0 tidyselect_1.1.2 [17] compiler_4.1.2 cli_3.3.0 rvest_1.0.2 xml2_1.3.3 [21] labeling_0.4.2 bookdown_0.26 sass_0.4.1 scales_1.2.0 [25] systemfonts_1.0.4 digest_0.6.29 minqa_1.2.4 rmarkdown_2.14 [29] svglite_2.1.0 pkgconfig_2.0.3 htmltools_0.5.2 parallelly_1.31.1 [33] dbplyr_2.1.1 fastmap_1.1.0 highr_0.9 rlang_1.0.2 [37] readxl_1.4.0 rstudioapi_0.13 jquerylib_0.1.4 generics_0.1.2 [41] farver_2.1.0 jsonlite_1.8.0 magrittr_2.0.3 Rcpp_1.0.8.3 [45] munsell_0.5.0 fansi_1.0.3 lifecycle_1.0.1 furrr_0.3.0 [49] stringi_1.7.6 yaml_2.3.5 snakecase_0.11.0 MASS_7.3-57 [53] grid_4.1.2 parallel_4.1.2 listenv_0.8.0 crayon_1.5.1 [57] lattice_0.20-45 haven_2.5.0 splines_4.1.2 hms_1.1.1 [61] pillar_1.7.0 boot_1.3-28 codetools_0.2-18 reprex_2.0.1 [65] glue_1.6.2 evaluate_0.15 modelr_0.1.8 vctrs_0.4.1 [69] nloptr_2.0.0 tzdb_0.3.0 cellranger_1.1.0 gtable_0.3.0 [73] future_1.25.0 assertthat_0.2.1 xfun_0.30 broom_0.8.0 [77] viridisLite_0.4.0 globals_0.14.0 ellipsis_0.3.2 "],["linear-mixed-effects-models-2.html", "Chapter 18 Linear mixed effects models 2 18.1 Learning goals 18.2 Load packages and set plotting theme 18.3 A worked example 18.4 Simulating a linear mixed effects model 18.5 Additional resources 18.6 Session info", " Chapter 18 Linear mixed effects models 2 18.1 Learning goals An lmer() worked example complete pooling vs. no pooling vs. partial pooling getting p-values checking model assumptions Simulating mixed effects models effect of outliers non-homogeneity of variance Simpson’s paradox 18.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;broom.mixed&quot;) # for tidying up linear models library(&quot;ggeffects&quot;) # for plotting marginal effects library(&quot;emmeans&quot;) # for the joint_tests() function library(&quot;lme4&quot;) # for linear mixed effects models library(&quot;performance&quot;) # for assessing model performance library(&quot;see&quot;) # for assessing model performance library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 18.3 A worked example Let’s illustrate the concept of pooling and shrinkage via the sleep data set that comes with the lmer package. # load sleepstudy data set df.sleep = sleepstudy %&gt;% as_tibble() %&gt;% clean_names() %&gt;% mutate(subject = as.character(subject)) %&gt;% select(subject, days, reaction) # add two fake participants (with missing data) df.sleep = df.sleep %&gt;% bind_rows(tibble(subject = &quot;374&quot;, days = 0:1, reaction = c(286, 288)), tibble(subject = &quot;373&quot;, days = 0, reaction = 245)) Let’s start by visualizing the data # visualize the data ggplot(data = df.sleep, mapping = aes(x = days, y = reaction)) + geom_point() + facet_wrap(~subject, ncol = 5) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) The plot shows the effect of the number of days of sleep deprivation on the average reaction time (presumably in an experiment). Note that for participant 373 and 374 we only have one and two data points respectively. 18.3.1 Complete pooling Let’s first fit a model the simply combines all the data points. This model ignores the dependence structure in the data (i.e. the fact that we have repeated observations from the same participants). fit.complete = lm(formula = reaction ~ days, data = df.sleep) fit.params = tidy(fit.complete) summary(fit.complete) Call: lm(formula = reaction ~ days, data = df.sleep) Residuals: Min 1Q Median 3Q Max -110.646 -27.951 1.829 26.388 139.875 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 252.321 6.406 39.389 &lt; 2e-16 *** days 10.328 1.210 8.537 5.48e-15 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 47.43 on 181 degrees of freedom Multiple R-squared: 0.2871, Adjusted R-squared: 0.2831 F-statistic: 72.88 on 1 and 181 DF, p-value: 5.484e-15 And let’s visualize the predictions of this model. # visualization (aggregate) ggplot(data = df.sleep, mapping = aes(x = days, y = reaction)) + geom_abline(intercept = fit.params$estimate[1], slope = fit.params$estimate[2], color = &quot;blue&quot;) + geom_point() + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) And here is what the model’s predictions look like separated by participant. # visualization (separate participants) ggplot(data = df.sleep, mapping = aes(x = days, y = reaction)) + geom_abline(intercept = fit.params$estimate[1], slope = fit.params$estimate[2], color = &quot;blue&quot;) + geom_point() + facet_wrap(~subject, ncol = 5) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) The model predicts the same relationship between sleep deprivation and reaction time for each participant (not surprising since we didn’t even tell the model that this data is based on different participants). 18.3.2 No pooling We could also fit separate regressions for each participant. Let’s do that. # fit regressions and extract parameter estimates df.no_pooling = df.sleep %&gt;% group_by(subject) %&gt;% nest(data = c(days, reaction)) %&gt;% mutate(fit = map(data, ~ lm(reaction ~ days, data = .)), params = map(fit, tidy)) %&gt;% ungroup() %&gt;% unnest(c(params)) %&gt;% select(subject, term, estimate) %&gt;% complete(subject, term, fill = list(estimate = 0)) %&gt;% pivot_wider(names_from = term, values_from = estimate) %&gt;% clean_names() And let’s visualize what the predictions of these separate regressions would look like: ggplot(data = df.sleep, mapping = aes(x = days, y = reaction)) + geom_abline(data = df.no_pooling %&gt;% filter(subject != 373), aes(intercept = intercept, slope = days), color = &quot;blue&quot;) + geom_point() + facet_wrap(~subject, ncol = 5) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) When we fit separate regression, no information is shared between participants. 18.3.3 Partial pooling By usign linear mixed effects models, we are partially pooling information. That is, the estimates for one participant are influenced by the rest of the participants. We’ll fit a number of mixed effects models that differ in their random effects structure. 18.3.3.1 Random intercept and random slope This model allows for random differences in the intercepts and slopes between subjects (and also models the correlation between intercepts and slopes). Let’s fit the model fit.random_intercept_slope = lmer(formula = reaction ~ 1 + days + (1 + days | subject), data = df.sleep) and take a look at the model’s predictions: fit.random_intercept_slope %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = days, y = reaction)) + geom_line(aes(y = fitted), color = &quot;blue&quot;) + geom_point() + facet_wrap(~subject, ncol = 5) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) As we can see, the lines for each participant are different. We’ve allowed for the intercept as well as the relationship between sleep deprivation and reaction time to be different between participants. 18.3.3.2 Only random intercepts Let’s fit a model that only allows for the intercepts to vary between participants. fit.random_intercept = lmer(formula = reaction ~ 1 + days + (1 | subject), data = df.sleep) And let’s visualize what these predictions look like: fit.random_intercept %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = days, y = reaction)) + geom_line(aes(y = fitted), color = &quot;blue&quot;) + geom_point() + facet_wrap(~subject, ncol = 5) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) Now, all the lines are parallel but the intercept differs between participants. 18.3.3.3 Only random slopes Finally, let’s compare a model that only allows for the slopes to differ but not the intercepts. fit.random_slope = lmer(formula = reaction ~ 1 + days + (0 + days | subject), data = df.sleep) And let’s visualize the model fit: fit.random_slope %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = days, y = reaction)) + geom_line(aes(y = fitted), color = &quot;blue&quot;) + geom_point() + facet_wrap(vars(subject), ncol = 5) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) Here, all the lines have the same starting point (i.e. the same intercept) but the slopes are different. 18.3.4 Compare results Let’s compare the results of the different methods – complete pooling, no pooling, and partial pooling (with random intercepts and slopes). # complete pooling fit.complete_pooling = lm(formula = reaction ~ days, data = df.sleep) df.complete_pooling = fit.complete_pooling %&gt;% augment() %&gt;% bind_rows(fit.complete_pooling %&gt;% augment(newdata = tibble(subject = c(&quot;373&quot;, &quot;374&quot;), days = rep(10, 2)))) %&gt;% clean_names() %&gt;% select(reaction, days, complete_pooling = fitted) # no pooling df.no_pooling = df.sleep %&gt;% group_by(subject) %&gt;% nest(data = c(days, reaction)) %&gt;% mutate(fit = map(data, ~ lm(reaction ~ days, data = .)), augment = map(fit, augment)) %&gt;% unnest(c(augment)) %&gt;% ungroup() %&gt;% clean_names() %&gt;% select(subject, reaction, days, no_pooling = fitted) # partial pooling fit.lmer = lmer(formula = reaction ~ 1 + days + (1 + days | subject), data = df.sleep) df.partial_pooling = fit.lmer %&gt;% augment() %&gt;% bind_rows(fit.lmer %&gt;% augment(newdata = tibble(subject = c(&quot;373&quot;, &quot;374&quot;), days = rep(10, 2)))) %&gt;% clean_names() %&gt;% select(subject, reaction, days, partial_pooling = fitted) # combine results df.pooling = df.partial_pooling %&gt;% left_join(df.complete_pooling, by = c(&quot;reaction&quot;, &quot;days&quot;)) %&gt;% left_join(df.no_pooling, by = c(&quot;subject&quot;, &quot;reaction&quot;, &quot;days&quot;)) Let’s compare the predictions of the different models visually: ggplot(data = df.pooling, mapping = aes(x = days, y = reaction)) + geom_smooth(method = &quot;lm&quot;, se = F, color = &quot;orange&quot;, fullrange = T) + geom_line(aes(y = complete_pooling), color = &quot;green&quot;) + geom_line(aes(y = partial_pooling), color = &quot;blue&quot;) + geom_point() + facet_wrap(~subject, ncol = 5) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) To better see the differences between the approaches, let’s focus on the predictions for the participants with incomplete data: # subselection ggplot(data = df.pooling %&gt;% filter(subject %in% c(&quot;373&quot;, &quot;374&quot;)), mapping = aes(x = days, y = reaction)) + geom_smooth(method = &quot;lm&quot;, se = F, color = &quot;orange&quot;, fullrange = T) + geom_line(aes(y = complete_pooling), color = &quot;green&quot;) + geom_line(aes(y = partial_pooling), color = &quot;blue&quot;) + geom_point() + facet_wrap(vars(subject)) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) 18.3.4.1 Coefficients One good way to get a sense for what the different models are doing is by taking a look at the coefficients: coef(fit.complete_pooling) (Intercept) days 252.32070 10.32766 coef(fit.random_intercept) $subject (Intercept) days 308 292.2749 10.43191 309 174.0559 10.43191 310 188.7454 10.43191 330 256.0247 10.43191 331 261.8141 10.43191 332 259.8262 10.43191 333 268.0765 10.43191 334 248.6471 10.43191 335 206.5096 10.43191 337 323.5643 10.43191 349 230.5114 10.43191 350 265.6957 10.43191 351 243.7988 10.43191 352 287.8850 10.43191 369 258.6454 10.43191 370 245.2931 10.43191 371 248.3508 10.43191 372 269.6861 10.43191 373 248.2086 10.43191 374 273.9400 10.43191 attr(,&quot;class&quot;) [1] &quot;coef.mer&quot; coef(fit.random_slope) $subject (Intercept) days 308 252.2965 19.9526801 309 252.2965 -4.3719650 310 252.2965 -0.9574726 330 252.2965 8.9909957 331 252.2965 10.5394285 332 252.2965 11.3994289 333 252.2965 12.6074020 334 252.2965 10.3413879 335 252.2965 -0.5722073 337 252.2965 24.2246485 349 252.2965 7.7702676 350 252.2965 15.0661415 351 252.2965 7.9675415 352 252.2965 17.0002999 369 252.2965 11.6982767 370 252.2965 11.3939807 371 252.2965 9.4535879 372 252.2965 13.4569059 373 252.2965 10.4142695 374 252.2965 11.9097917 attr(,&quot;class&quot;) [1] &quot;coef.mer&quot; coef(fit.random_intercept_slope) $subject (Intercept) days 308 253.9478 19.6264337 309 211.7331 1.7319161 310 213.1582 4.9061511 330 275.1425 5.6436007 331 273.7286 7.3862730 332 260.6504 10.1632571 333 268.3683 10.2246059 334 244.5524 11.4837802 335 251.3702 -0.3355788 337 286.2319 19.1090424 349 226.7663 11.5531844 350 238.7807 17.0156827 351 256.2344 7.4119456 352 272.3511 13.9920878 369 254.9484 11.2985770 370 226.3701 15.2027877 371 252.5051 9.4335409 372 263.8916 11.7253429 373 248.9753 10.3915288 374 271.1450 11.0782516 attr(,&quot;class&quot;) [1] &quot;coef.mer&quot; 18.3.4.2 Shrinkage In mixed effects models, the variance of parameter estimates across participants shrinks compared to a no pooling model (where we fit a different regression to each participant). Expressed differently, individual parameter estimates are borrowing strength from the overall data set in mixed effects models. # get estimates from partial pooling model df.partial_pooling = fit.random_intercept_slope %&gt;% coef() %&gt;% .$subject %&gt;% rownames_to_column(&quot;subject&quot;) %&gt;% clean_names() # combine estimates from no pooling with partial pooling model df.plot = df.sleep %&gt;% group_by(subject) %&gt;% nest(data = c(days, reaction)) %&gt;% mutate(fit = map(data, ~ lm(reaction ~ days, data = .)), tidy = map(fit, tidy)) %&gt;% unnest(c(tidy)) %&gt;% select(subject, term, estimate) %&gt;% pivot_wider(names_from = term, values_from = estimate) %&gt;% clean_names() %&gt;% mutate(method = &quot;no pooling&quot;) %&gt;% bind_rows(df.partial_pooling %&gt;% mutate(method = &quot;partial pooling&quot;)) %&gt;% pivot_longer(cols = -c(subject, method), names_to = &quot;index&quot;, values_to = &quot;value&quot;) %&gt;% mutate(index = factor(index, levels = c(&quot;intercept&quot;, &quot;days&quot;))) # visualize the results ggplot(data = df.plot, mapping = aes(x = value, group = method, fill = method)) + stat_density(position = &quot;identity&quot;, geom = &quot;area&quot;, color = &quot;black&quot;, alpha = 0.3) + facet_grid(cols = vars(index), scales = &quot;free&quot;) Warning: Removed 1 rows containing non-finite values (stat_density). 18.3.5 Getting p-values To get p-values for mixed effects models, I recommend using the joint_tests() function from the emmeans package. lmer(formula = reaction ~ 1 + days + (1 + days | subject), data = df.sleep) %&gt;% joint_tests() model term df1 df2 F.ratio p.value days 1 17.08 45.531 &lt;.0001 Our good ol’ model comparison approach produces a Likelihood ratio test in this case: fit1 = lmer(formula = reaction ~ 1 + days + (1 + days | subject), data = df.sleep) fit2 = lmer(formula = reaction ~ 1 + (1 + days | subject), data = df.sleep) anova(fit1, fit2) refitting model(s) with ML (instead of REML) Data: df.sleep Models: fit2: reaction ~ 1 + (1 + days | subject) fit1: reaction ~ 1 + days + (1 + days | subject) npar AIC BIC logLik deviance Chisq Df Pr(&gt;Chisq) fit2 5 1813.2 1829.3 -901.62 1803.2 fit1 6 1791.6 1810.9 -889.82 1779.6 23.602 1 1.184e-06 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 18.3.6 Reporting results 18.3.6.1 Plotting marginal effects # library(&quot;ggeffects&quot;) # using the plot() function ggpredict(model = fit.random_intercept_slope, terms = &quot;days&quot;, type = &quot;fe&quot;) %&gt;% plot() # using our own ggplot magic df.plot = ggpredict(model = fit.random_intercept_slope, terms = &quot;days&quot;, type = &quot;fe&quot;) ggplot(data = df.plot, mapping = aes(x = x, y = predicted, ymin = conf.low, ymax = conf.high)) + geom_ribbon(fill = &quot;lightblue&quot;) + geom_line(size = 1) 18.3.6.2 Checking model performance lmer(formula = reaction ~ 1 + days + (1 + days | subject), data = df.sleep) %&gt;% check_model() 18.4 Simulating a linear mixed effects model To generate some data for a linear mixed effects model with random intercepts, we do pretty much what we are used to doing when we generated data for a linear model. However, this time, we have an additional parameter that captures the variance in the intercepts between participants. So, we draw a separate (offset from the global) intercept for each participant from this distribution. # make example reproducible set.seed(1) # parameters sample_size = 100 b0 = 1 b1 = 2 sd_residual = 1 sd_participant = 0.5 # generate the data df.mixed = tibble(participant = rep(1:sample_size, 2), condition = rep(0:1, each = sample_size)) %&gt;% group_by(participant) %&gt;% mutate(intercepts = rnorm(n = 1, sd = sd_participant)) %&gt;% ungroup() %&gt;% mutate(value = b0 + b1 * condition + intercepts + rnorm(n(), sd = sd_residual)) %&gt;% arrange(participant, condition) df.mixed # A tibble: 200 × 4 participant condition intercepts value &lt;int&gt; &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; 1 1 0 -0.313 0.0664 2 1 1 -0.313 3.10 3 2 0 0.0918 1.13 4 2 1 0.0918 4.78 5 3 0 -0.418 -0.329 6 3 1 -0.418 4.17 7 4 0 0.798 1.96 8 4 1 0.798 3.47 9 5 0 0.165 0.510 10 5 1 0.165 0.880 # … with 190 more rows Let’s fit a model to this data now and take a look at the summary output: # fit model fit.mixed = lmer(formula = value ~ 1 + condition + (1 | participant), data = df.mixed) summary(fit.mixed) Linear mixed model fit by REML [&#39;lmerMod&#39;] Formula: value ~ 1 + condition + (1 | participant) Data: df.mixed REML criterion at convergence: 606 Scaled residuals: Min 1Q Median 3Q Max -2.53710 -0.62295 -0.04364 0.67035 2.19899 Random effects: Groups Name Variance Std.Dev. participant (Intercept) 0.1607 0.4009 Residual 1.0427 1.0211 Number of obs: 200, groups: participant, 100 Fixed effects: Estimate Std. Error t value (Intercept) 1.0166 0.1097 9.267 condition 2.0675 0.1444 14.317 Correlation of Fixed Effects: (Intr) condition -0.658 Let’s visualize the model’s predictions: fit.mixed %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = condition, y = value, group = participant)) + geom_point(alpha = 0.5) + geom_line(alpha = 0.5) + geom_point(aes(y = fitted), color = &quot;red&quot;) + geom_line(aes(y = fitted), color = &quot;red&quot;) Let’s simulate some data from this fitted model: # simulated data fit.mixed %&gt;% simulate() %&gt;% bind_cols(df.mixed) %&gt;% ggplot(data = ., mapping = aes(x = condition, y = sim_1, group = participant)) + geom_line(alpha = 0.5) + geom_point(alpha = 0.5) Even though we only fitted random intercepts in this model, when we simulate from the model, we get different slopes since, when simulating new data, the model takes our uncertainty in the residuals into account as well. Let’s see whether fitting random intercepts was worth it in this case: # using chisq test fit.compact = lm(formula = value ~ 1 + condition, data = df.mixed) fit.augmented = lmer(formula = value ~ 1 + condition + (1 | participant), data = df.mixed) anova(fit.augmented, fit.compact) refitting model(s) with ML (instead of REML) Data: df.mixed Models: fit.compact: value ~ 1 + condition fit.augmented: value ~ 1 + condition + (1 | participant) npar AIC BIC logLik deviance Chisq Df Pr(&gt;Chisq) fit.compact 3 608.6 618.49 -301.3 602.6 fit.augmented 4 608.8 621.99 -300.4 600.8 1.7999 1 0.1797 Nope, it’s not worth it in this case. That said, even though having random intercepts does not increase the likelihood of the data given the model significantly, we should still include random intercepts to capture the dependence in the data. 18.4.1 The effect of outliers Let’s take 20 participants from our df.mixed data set, and make one of the participants be an outlier: # let&#39;s make one outlier df.outlier = df.mixed %&gt;% mutate(participant = participant %&gt;% as.character() %&gt;% as.numeric()) %&gt;% filter(participant &lt;= 20) %&gt;% mutate(value = ifelse(participant == 20, value + 30, value), participant = as.factor(participant)) Let’s fit the model and look at the summary: # fit model fit.outlier = lmer(formula = value ~ 1 + condition + (1 | participant), data = df.outlier) summary(fit.outlier) Linear mixed model fit by REML [&#39;lmerMod&#39;] Formula: value ~ 1 + condition + (1 | participant) Data: df.outlier REML criterion at convergence: 192 Scaled residuals: Min 1Q Median 3Q Max -1.44598 -0.48367 0.03043 0.44689 1.41232 Random effects: Groups Name Variance Std.Dev. participant (Intercept) 45.1359 6.7183 Residual 0.6738 0.8209 Number of obs: 40, groups: participant, 20 Fixed effects: Estimate Std. Error t value (Intercept) 2.7091 1.5134 1.790 condition 2.1512 0.2596 8.287 Correlation of Fixed Effects: (Intr) condition -0.086 The variance of the participants’ intercepts has increased dramatically! Let’s visualize the data together with the model’s predictions: fit.outlier %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = condition, y = value, group = participant)) + geom_point(alpha = 0.5) + geom_line(alpha = 0.5) + geom_point(aes(y = fitted), color = &quot;red&quot;) + geom_line(aes(y = fitted), color = &quot;red&quot;) The model is still able to capture the participants quite well. But note what its simulated data looks like now: # simulated data from lmer with outlier fit.outlier %&gt;% simulate() %&gt;% bind_cols(df.outlier) %&gt;% ggplot(data = ., mapping = aes(x = condition, y = sim_1, group = participant)) + geom_line(alpha = 0.5) + geom_point(alpha = 0.5) The simulated data doesn’t look like our original data. This is because one normal distribution is used to model the variance in the intercepts between participants. 18.4.2 Different slopes Let’s generate data where the effect of condition is different for participants: # make example reproducible set.seed(1) tmp = rnorm(n = 20) df.slopes = tibble( condition = rep(1:2, each = 20), participant = rep(1:20, 2), value = ifelse(condition == 1, tmp, mean(tmp) + rnorm(n = 20, sd = 0.3)) # regression to the mean ) %&gt;% mutate(condition = as.factor(condition), participant = as.factor(participant)) Let’s fit a model with random intercepts. fit.slopes = lmer(formula = value ~ 1 + condition + (1 | participant), data = df.slopes) boundary (singular) fit: see help(&#39;isSingular&#39;) summary(fit.slopes) Linear mixed model fit by REML [&#39;lmerMod&#39;] Formula: value ~ 1 + condition + (1 | participant) Data: df.slopes REML criterion at convergence: 83.6 Scaled residuals: Min 1Q Median 3Q Max -3.5808 -0.3184 0.0130 0.4551 2.0913 Random effects: Groups Name Variance Std.Dev. participant (Intercept) 0.0000 0.0000 Residual 0.4512 0.6717 Number of obs: 40, groups: participant, 20 Fixed effects: Estimate Std. Error t value (Intercept) 0.190524 0.150197 1.268 condition2 -0.001941 0.212411 -0.009 Correlation of Fixed Effects: (Intr) condition2 -0.707 optimizer (nloptwrap) convergence code: 0 (OK) boundary (singular) fit: see help(&#39;isSingular&#39;) Note how the summary says “singular fit”, and how the variance for random intercepts is 0. Here, fitting random intercepts did not help the model fit at all, so the lmer gave up … How about fitting random slopes? # fit model lmer(formula = value ~ 1 + condition + (1 + condition | participant), data = df.slopes) This won’t work because the model has more parameters than there are data points. To fit random slopes, we need more than 2 observations per participants. 18.4.3 Simpson’s paradox Taking dependence in the data into account is extremely important. The Simpson’s paradox is an instructive example for what can go wrong when we ignore the dependence in the data. Let’s start by simulating some data to demonstrate the paradox. # make example reproducible set.seed(2) n_participants = 20 n_observations = 10 slope = -10 sd_error = 0.4 sd_participant = 5 intercept = rnorm(n_participants, sd = sd_participant) %&gt;% sort() df.simpson = tibble(x = runif(n_participants * n_observations, min = 0, max = 1)) %&gt;% arrange(x) %&gt;% mutate(intercept = rep(intercept, each = n_observations), y = intercept + x * slope + rnorm(n(), sd = sd_error), participant = factor(intercept, labels = 1:n_participants)) Let’s visualize the overall relationship between x and y with a simple linear model. # overall effect ggplot(data = df.simpson, mapping = aes(x = x, y = y)) + geom_point() + geom_smooth(method = &quot;lm&quot;, color = &quot;black&quot;) As we see, overall, there is a positive relationship between x and y. lm(formula = y ~ x, data = df.simpson) %&gt;% summary() Call: lm(formula = y ~ x, data = df.simpson) Residuals: Min 1Q Median 3Q Max -5.8731 -0.6362 0.2272 1.0051 2.6410 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) -7.1151 0.2107 -33.76 &lt;2e-16 *** x 6.3671 0.3631 17.54 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.55 on 198 degrees of freedom Multiple R-squared: 0.6083, Adjusted R-squared: 0.6064 F-statistic: 307.5 on 1 and 198 DF, p-value: &lt; 2.2e-16 And this relationship is significant. Let’s take another look at the data use different colors for the different participants. # effect by participant ggplot(data = df.simpson, mapping = aes(x = x, y = y, color = participant)) + geom_point() + geom_smooth(method = &quot;lm&quot;, color = &quot;black&quot;) + theme(legend.position = &quot;none&quot;) And let’s fit a different regression for each participant: # effect by participant ggplot(data = df.simpson, mapping = aes(x = x, y = y, color = participant, group = participant)) + geom_point() + geom_smooth(method = &quot;lm&quot;, color = &quot;black&quot;) + theme(legend.position = &quot;none&quot;) What this plot shows, is that for almost all individual participants, the relationship between x and y is negative. The different participants where along the x spectrum they are. Let’s fit a linear mixed effects model with random intercepts: fit.lmer = lmer(formula = y ~ 1 + x + (1 | participant), data = df.simpson) fit.lmer %&gt;% summary() Linear mixed model fit by REML [&#39;lmerMod&#39;] Formula: y ~ 1 + x + (1 | participant) Data: df.simpson REML criterion at convergence: 345.1 Scaled residuals: Min 1Q Median 3Q Max -2.43394 -0.59687 0.04493 0.62694 2.68828 Random effects: Groups Name Variance Std.Dev. participant (Intercept) 21.4898 4.6357 Residual 0.1661 0.4075 Number of obs: 200, groups: participant, 20 Fixed effects: Estimate Std. Error t value (Intercept) -0.1577 1.3230 -0.119 x -7.6678 1.6572 -4.627 Correlation of Fixed Effects: (Intr) x -0.621 As we can see, the fixed effect for x is now negative! fit.lmer %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., aes(x = x, y = y, group = participant, color = participant)) + geom_point() + geom_line(aes(y = fitted), size = 1, color = &quot;black&quot;) + theme(legend.position = &quot;none&quot;) Lesson learned: taking dependence into account is critical for drawing correct inferences! 18.5 Additional resources 18.5.1 Readings Linear mixed effects models tutorial by Bodo Winter Simpson’s paradox Tutorial on pooling 18.6 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 [4] purrr_0.3.4 readr_2.1.2 tidyr_1.2.0 [7] tibble_3.1.7 ggplot2_3.3.6 tidyverse_1.3.1 [10] see_0.7.0 performance_0.9.0 lme4_1.1-29 [13] Matrix_1.4-1 emmeans_1.7.3 ggeffects_1.1.2 [16] broom.mixed_0.2.9.4 janitor_2.1.0 kableExtra_1.3.4 [19] knitr_1.39 loaded via a namespace (and not attached): [1] nlme_3.1-157 pbkrtest_0.5.1 fs_1.5.2 lubridate_1.8.0 [5] insight_0.17.0 webshot_0.5.3 httr_1.4.3 tools_4.1.2 [9] backports_1.4.1 bslib_0.3.1 sjlabelled_1.2.0 utf8_1.2.2 [13] R6_2.5.1 mgcv_1.8-40 DBI_1.1.2 colorspace_2.0-3 [17] withr_2.5.0 tidyselect_1.1.2 compiler_4.1.2 cli_3.3.0 [21] rvest_1.0.2 xml2_1.3.3 sandwich_3.0-1 bayestestR_0.12.1 [25] labeling_0.4.2 bookdown_0.26 sass_0.4.1 scales_1.2.0 [29] mvtnorm_1.1-3 systemfonts_1.0.4 digest_0.6.29 minqa_1.2.4 [33] rmarkdown_2.14 svglite_2.1.0 pkgconfig_2.0.3 htmltools_0.5.2 [37] parallelly_1.31.1 highr_0.9 dbplyr_2.1.1 fastmap_1.1.0 [41] rlang_1.0.2 readxl_1.4.0 rstudioapi_0.13 farver_2.1.0 [45] jquerylib_0.1.4 generics_0.1.2 zoo_1.8-10 jsonlite_1.8.0 [49] magrittr_2.0.3 patchwork_1.1.1 Rcpp_1.0.8.3 munsell_0.5.0 [53] fansi_1.0.3 lifecycle_1.0.1 furrr_0.3.0 stringi_1.7.6 [57] multcomp_1.4-19 yaml_2.3.5 snakecase_0.11.0 MASS_7.3-57 [61] grid_4.1.2 ggrepel_0.9.1 parallel_4.1.2 listenv_0.8.0 [65] crayon_1.5.1 lattice_0.20-45 haven_2.5.0 splines_4.1.2 [69] hms_1.1.1 pillar_1.7.0 boot_1.3-28 estimability_1.3 [73] codetools_0.2-18 reprex_2.0.1 glue_1.6.2 evaluate_0.15 [77] modelr_0.1.8 vctrs_0.4.1 nloptr_2.0.0 tzdb_0.3.0 [81] cellranger_1.1.0 gtable_0.3.0 datawizard_0.4.0 future_1.25.0 [85] assertthat_0.2.1 xfun_0.30 xtable_1.8-4 broom_0.8.0 [89] coda_0.19-4 survival_3.3-1 viridisLite_0.4.0 globals_0.14.0 [93] TH.data_1.1-1 ellipsis_0.3.2 "],["linear-mixed-effects-models-3.html", "Chapter 19 Linear mixed effects models 3 19.1 Learning goals 19.2 Load packages and set plotting theme 19.3 Load data sets 19.4 Understanding the lmer() syntax 19.5 ANOVA vs. Lmer 19.6 Session info", " Chapter 19 Linear mixed effects models 3 19.1 Learning goals Pitfalls in fitting lmers()s (and what to do about it). Understanding lmer() syntax even better. ANOVA vs. Lmer 19.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;broom.mixed&quot;) # for tidying up linear mixed effects models library(&quot;patchwork&quot;) # for making figure panels library(&quot;lme4&quot;) # for linear mixed effects models library(&quot;afex&quot;) # for ANOVAs library(&quot;car&quot;) # for ANOVAs library(&quot;datarium&quot;) # for ANOVA dataset library(&quot;modelr&quot;) # for bootstrapping library(&quot;boot&quot;) # also for bootstrapping library(&quot;ggeffects&quot;) # for plotting marginal effects library(&quot;emmeans&quot;) # for marginal effects library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size # knitr display options opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) # # set contrasts to using sum contrasts # options(contrasts = c(&quot;contr.sum&quot;, &quot;contr.poly&quot;)) # suppress grouping warning messages options(dplyr.summarise.inform = F) 19.3 Load data sets 19.3.1 Sleep data # load sleepstudy data set df.sleep = sleepstudy %&gt;% as_tibble() %&gt;% clean_names() %&gt;% mutate(subject = as.character(subject)) %&gt;% select(subject, days, reaction) # add two fake participants (with missing data) df.sleep = df.sleep %&gt;% bind_rows(tibble(subject = &quot;374&quot;, days = 0:1, reaction = c(286, 288)), tibble(subject = &quot;373&quot;, days = 0, reaction = 245)) 19.3.2 Reasoning data df.reasoning = sk2011.1 19.3.3 Weight loss data data(&quot;weightloss&quot;, package = &quot;datarium&quot;) # Modify it to have three-way mixed design df.weightloss = weightloss %&gt;% mutate(id = rep(1:24, 2)) %&gt;% pivot_longer(cols = t1:t3, names_to = &quot;timepoint&quot;, values_to = &quot;score&quot;) %&gt;% arrange(id) 19.3.4 Politness data df.politeness = read_csv(&quot;data/politeness_data.csv&quot;) %&gt;% mutate(scenario = as.factor(scenario)) Rows: 84 Columns: 5 ── Column specification ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Delimiter: &quot;,&quot; chr (3): subject, gender, attitude dbl (2): scenario, frequency ℹ Use `spec()` to retrieve the full column specification for this data. ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message. 19.4 Understanding the lmer() syntax Here is an overview of how to specify different kinds of linear mixed effects models. formula description dv ~ x1 + (1 | g) Random intercept for each level of g dv ~ x1 + (0 + x1 | g) Random slope for each level of g dv ~ x1 + (x1 | g) Correlated random slope and intercept for each level of g dv ~ x1 + (x1 || g) Uncorrelated random slope and intercept for each level of g dv ~ x1 + (1 | school) + (1 | teacher) Random intercept for each level of school and for each level of teacher (crossed) dv ~ x1 + (1 | school/teacher) Random intercept for each level of school and for each level of teacher in school (nested) Note that this (1 | school/teacher) is equivalent to (1 | school) + (1 | teacher:school) (see here). 19.5 ANOVA vs. Lmer 19.5.1 Between subjects ANOVA Let’s start with a between subjects ANOVA (which means we are in lm() world). We’ll take a look whether what type of instruction participants received made a difference to their response. First, we use the aov_ez() function from the “afex” package to do so. aov_ez(id = &quot;id&quot;, dv = &quot;response&quot;, between = &quot;instruction&quot;, data = df.reasoning) Warning: More than one observation per design cell, aggregating data using `fun_aggregate = mean`. To turn off this warning, pass `fun_aggregate = mean` explicitly. Contrasts set to contr.sum for the following variables: instruction Anova Table (Type 3 tests) Response: response Effect df MSE F ges p.value 1 instruction 1, 38 253.43 0.31 .008 .583 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;+&#39; 0.1 &#39; &#39; 1 Looks like there was no main effect of instruction on participants’ responses. An alternative route for getting at the same test, would be via combining lm() with Anova() (as we’ve done before in class). lm(formula = response ~ instruction, data = df.reasoning %&gt;% group_by(id, instruction) %&gt;% summarize(response = mean(response)) %&gt;% ungroup()) %&gt;% joint_tests() model term df1 df2 F.ratio p.value instruction 1 38 0.307 0.5830 The two routes yield the same result. Notice that for the lm() approach, I calculated the means for each participant in each condition first (using group_by() and summarize()). 19.5.2 Repeated-measures ANOVA Now let’s take a look whether validity and plausibility affected participants’ responses in the reasoning task. These two factors were varied within participants. Again, we’ll use the aov_ez() function like so: aov_ez(id = &quot;id&quot;, dv = &quot;response&quot;, within = c(&quot;validity&quot;, &quot;plausibility&quot;), data = df.reasoning %&gt;% filter(instruction == &quot;probabilistic&quot;)) Warning: More than one observation per design cell, aggregating data using `fun_aggregate = mean`. To turn off this warning, pass `fun_aggregate = mean` explicitly. Anova Table (Type 3 tests) Response: response Effect df MSE F ges p.value 1 validity 1, 19 183.01 0.01 &lt;.001 .904 2 plausibility 1, 19 321.44 30.30 *** .366 &lt;.001 3 validity:plausibility 1, 19 65.83 9.21 ** .035 .007 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;+&#39; 0.1 &#39; &#39; 1 For the linear model route, given that we have repeated observations from the same participants, we need to use lmer(). The repeated measures anova has the random effect structure as shown below: lmer(formula = response ~ validity * plausibility + (1 | id) + (1 | validity:id) + (1 | plausibility:id), data = df.reasoning %&gt;% filter(instruction == &quot;probabilistic&quot;) %&gt;% group_by(id, validity, plausibility) %&gt;% summarize(response = mean(response))) %&gt;% joint_tests() boundary (singular) fit: see help(&#39;isSingular&#39;) model term df1 df2 F.ratio p.value validity 1 19 0.016 0.9007 plausibility 1 19 34.210 &lt;.0001 validity:plausibility 1 19 8.927 0.0076 Again, we get a similar result using the joint_tests() function. Note though that the results of the ANOVA route and the lmer() route weren’t identical here (although they were very close). For more information as to why this happens, see this post. 19.5.3 Mixed ANOVA Now let’s take a look at both between- as well as within-subjects factors. Let’s compare the aov_ez() route aov_ez(id = &quot;id&quot;, dv = &quot;response&quot;, between = &quot;instruction&quot;, within = c(&quot;validity&quot;, &quot;plausibility&quot;), data = df.reasoning) Warning: More than one observation per design cell, aggregating data using `fun_aggregate = mean`. To turn off this warning, pass `fun_aggregate = mean` explicitly. Contrasts set to contr.sum for the following variables: instruction Anova Table (Type 3 tests) Response: response Effect df MSE F ges p.value 1 instruction 1, 38 1013.71 0.31 .005 .583 2 validity 1, 38 339.32 4.12 * .020 .049 3 instruction:validity 1, 38 339.32 4.65 * .023 .037 4 plausibility 1, 38 234.41 34.23 *** .106 &lt;.001 5 instruction:plausibility 1, 38 234.41 10.67 ** .036 .002 6 validity:plausibility 1, 38 185.94 0.14 &lt;.001 .715 7 instruction:validity:plausibility 1, 38 185.94 4.78 * .013 .035 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;+&#39; 0.1 &#39; &#39; 1 with the lmer() route: lmer(formula = response ~ instruction * validity * plausibility + (1 | id) + (1 | validity:id) + (1 | plausibility:id), data = df.reasoning %&gt;% group_by(id, validity, plausibility, instruction) %&gt;% summarize(response = mean(response))) %&gt;% joint_tests() model term df1 df2 F.ratio p.value instruction 1 38 0.307 0.5830 validity 1 38 4.121 0.0494 plausibility 1 38 34.227 &lt;.0001 instruction:validity 1 38 4.651 0.0374 instruction:plausibility 1 38 10.667 0.0023 validity:plausibility 1 38 0.136 0.7148 instruction:validity:plausibility 1 38 4.777 0.0351 Here, both routes yield the same results. 19.6 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 [4] purrr_0.3.4 readr_2.1.2 tidyr_1.2.0 [7] tibble_3.1.7 ggplot2_3.3.6 tidyverse_1.3.1 [10] emmeans_1.7.3 ggeffects_1.1.2 boot_1.3-28 [13] modelr_0.1.8 datarium_0.1.0 car_3.0-13 [16] carData_3.0-5 afex_1.1-1 lme4_1.1-29 [19] Matrix_1.4-1 patchwork_1.1.1 broom.mixed_0.2.9.4 [22] janitor_2.1.0 kableExtra_1.3.4 knitr_1.39 loaded via a namespace (and not attached): [1] TH.data_1.1-1 minqa_1.2.4 colorspace_2.0-3 [4] ellipsis_0.3.2 estimability_1.3 snakecase_0.11.0 [7] fs_1.5.2 rstudioapi_0.13 listenv_0.8.0 [10] furrr_0.3.0 bit64_4.0.5 fansi_1.0.3 [13] mvtnorm_1.1-3 lubridate_1.8.0 xml2_1.3.3 [16] codetools_0.2-18 splines_4.1.2 jsonlite_1.8.0 [19] nloptr_2.0.0 pbkrtest_0.5.1 broom_0.8.0 [22] dbplyr_2.1.1 compiler_4.1.2 httr_1.4.3 [25] backports_1.4.1 assertthat_0.2.1 fastmap_1.1.0 [28] cli_3.3.0 htmltools_0.5.2 tools_4.1.2 [31] lmerTest_3.1-3 coda_0.19-4 gtable_0.3.0 [34] glue_1.6.2 reshape2_1.4.4 Rcpp_1.0.8.3 [37] cellranger_1.1.0 jquerylib_0.1.4 vctrs_0.4.1 [40] svglite_2.1.0 nlme_3.1-157 xfun_0.30 [43] globals_0.14.0 rvest_1.0.2 lifecycle_1.0.1 [46] future_1.25.0 MASS_7.3-57 zoo_1.8-10 [49] scales_1.2.0 vroom_1.5.7 hms_1.1.1 [52] parallel_4.1.2 sandwich_3.0-1 yaml_2.3.5 [55] sass_0.4.1 stringi_1.7.6 highr_0.9 [58] rlang_1.0.2 pkgconfig_2.0.3 systemfonts_1.0.4 [61] evaluate_0.15 lattice_0.20-45 bit_4.0.4 [64] tidyselect_1.1.2 parallelly_1.31.1 plyr_1.8.7 [67] magrittr_2.0.3 bookdown_0.26 R6_2.5.1 [70] generics_0.1.2 multcomp_1.4-19 DBI_1.1.2 [73] pillar_1.7.0 haven_2.5.0 withr_2.5.0 [76] survival_3.3-1 abind_1.4-5 crayon_1.5.1 [79] utf8_1.2.2 tzdb_0.3.0 rmarkdown_2.14 [82] grid_4.1.2 readxl_1.4.0 reprex_2.0.1 [85] digest_0.6.29 webshot_0.5.3 xtable_1.8-4 [88] numDeriv_2016.8-1.1 munsell_0.5.0 viridisLite_0.4.0 [91] bslib_0.3.1 "],["linear-mixed-effects-models-4.html", "Chapter 20 Linear mixed effects models 4 20.1 Learning goals 20.2 Load packages and set plotting theme 20.3 Load data sets 20.4 Follow-up tests with emmeans 20.5 Mixtures of participants 20.6 Bootstrapping 20.7 Session info", " Chapter 20 Linear mixed effects models 4 20.1 Learning goals Some worked examples. Doing follow-up tests with the emmeans package 20.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;broom.mixed&quot;) # for tidying up linear mixed effects models library(&quot;lme4&quot;) # for linear mixed effects models library(&quot;afex&quot;) # for ANOVAs library(&quot;car&quot;) # for ANOVAs library(&quot;datarium&quot;) # for ANOVA dataset library(&quot;modelr&quot;) # for bootstrapping library(&quot;boot&quot;) # also for bootstrapping library(&quot;ggeffects&quot;) # for plotting marginal effects library(&quot;emmeans&quot;) # for marginal effects library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size # knitr display options opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) # suppress grouping warning messages options(dplyr.summarise.inform = F) 20.3 Load data sets 20.3.1 Sleep data # load sleepstudy data set df.sleep = sleepstudy %&gt;% as_tibble() %&gt;% clean_names() %&gt;% mutate(subject = as.character(subject)) %&gt;% select(subject, days, reaction) # add two fake participants (with missing data) df.sleep = df.sleep %&gt;% bind_rows(tibble(subject = &quot;374&quot;, days = 0:1, reaction = c(286, 288)), tibble(subject = &quot;373&quot;, days = 0, reaction = 245)) 20.3.2 Reasoning data df.reasoning = sk2011.1 20.3.3 Weight loss data data(&quot;weightloss&quot;, package = &quot;datarium&quot;) # Modify it to have three-way mixed design df.weightloss = weightloss %&gt;% mutate(id = rep(1:24, 2)) %&gt;% pivot_longer(cols = t1:t3, names_to = &quot;timepoint&quot;, values_to = &quot;score&quot;) %&gt;% arrange(id) 20.3.4 Politness data df.politeness = read_csv(&quot;data/politeness_data.csv&quot;) %&gt;% mutate(scenario = as.factor(scenario)) Rows: 84 Columns: 5 ── Column specification ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Delimiter: &quot;,&quot; chr (3): subject, gender, attitude dbl (2): scenario, frequency ℹ Use `spec()` to retrieve the full column specification for this data. ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message. 20.4 Follow-up tests with emmeans Just like with the linear model lm(), we can use linear contrasts to test more specific hypotheses with lmer(). The emmeans() function from the emmeans package will be our friend. 20.4.1 Sleep study Let’s ask some more specific question aboust the sleep study. Do reaction times differ between day 0 and the first day of sleep deprivation? Do reaction times differ between the first and the second half of the study? Let’s visualize the data first: ggplot(data = df.sleep %&gt;% mutate(days = as.factor(days)), mapping = aes(x = days, y = reaction)) + geom_point(position = position_jitter(width = 0.1), alpha = 0.1) + stat_summary(fun.data = &quot;mean_cl_boot&quot;) And now let’s fit the model, and compute the contrasts: fit = lmer(formula = reaction ~ 1 + days + (1 | subject), data = df.sleep %&gt;% mutate(days = as.factor(days))) contrast = list(first_vs_second = c(-1, 1, rep(0, 8)), early_vs_late = c(rep(-1, 5)/5, rep(1, 5)/5)) fit %&gt;% emmeans(specs = &quot;days&quot;, contr = contrast) %&gt;% pluck(&quot;contrasts&quot;) contrast estimate SE df t.ratio p.value first_vs_second 7.82 10.10 156 0.775 0.4398 early_vs_late 53.66 4.65 155 11.534 &lt;.0001 Degrees-of-freedom method: kenward-roger df.sleep %&gt;% # filter(days %in% c(0, 1)) %&gt;% group_by(days) %&gt;% summarize(reaction = mean(reaction)) # A tibble: 10 × 2 days reaction &lt;dbl&gt; &lt;dbl&gt; 1 0 258. 2 1 266. 3 2 265. 4 3 283. 5 4 289. 6 5 309. 7 6 312. 8 7 319. 9 8 337. 10 9 351. df.sleep %&gt;% mutate(index = ifelse(days %in% 0:4, &quot;early&quot;, &quot;late&quot;)) %&gt;% group_by(index) %&gt;% summarize(reaction = mean(reaction)) # A tibble: 2 × 2 index reaction &lt;chr&gt; &lt;dbl&gt; 1 early 272. 2 late 325. 20.4.2 Weight loss study For the weight loss data set, we want to check: Whether there was a difference between the first two vs. the last time point. Whether there was a linear trend across the time points. Let’s first visualize again: ggplot(data = df.weightloss, mapping = aes(x = timepoint, y = score, group = diet, color = diet)) + geom_point(position = position_jitterdodge(dodge.width = 0.5, jitter.width = 0.1, jitter.height = 0), alpha = 0.1) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, position = position_dodge(width = 0.5)) + facet_wrap(~ exercises) + scale_color_brewer(palette = &quot;Set1&quot;) ggplot(data = df.weightloss, mapping = aes(x = timepoint, y = score)) + geom_point(position = position_jitter(width = 0.1), alpha = 0.1) + stat_summary(fun.data = &quot;mean_cl_boot&quot;) + scale_color_brewer(palette = &quot;Set1&quot;) And then fit the model, and compute the contrasts: fit = aov_ez(id = &quot;id&quot;, dv = &quot;score&quot;, between = &quot;exercises&quot;, within = c(&quot;diet&quot;, &quot;timepoint&quot;), data = df.weightloss) Contrasts set to contr.sum for the following variables: exercises contrasts = list(first_two_vs_last = c(-0.5, -0.5, 1), linear_increase = c(-1, 0, 1)) fit %&gt;% emmeans(spec = &quot;timepoint&quot;, contr = contrasts) $emmeans timepoint emmean SE df lower.CL upper.CL t1 11.2 0.169 22 10.9 11.6 t2 12.7 0.174 22 12.3 13.0 t3 14.2 0.182 22 13.8 14.6 Results are averaged over the levels of: exercises, diet Confidence level used: 0.95 $contrasts contrast estimate SE df t.ratio p.value first_two_vs_last 2.24 0.204 22 11.016 &lt;.0001 linear_increase 2.97 0.191 22 15.524 &lt;.0001 Results are averaged over the levels of: exercises, diet Because we only had one observation in each cell of our design, the ANOVA was appropriate here (no data points needed to be aggregated). Both contrasts are significant. 20.4.3 Politeness study For the politeness study, we’ll be interested in one particular contrast: Was there an effect of attitude on frequency for female participants? Let’s visualize first: # overview of the data ggplot(data = df.politeness, mapping = aes(x = attitude, y = frequency, group = gender, color = gender)) + geom_point(position = position_jitter(width = 0.1), alpha = 0.1) + stat_summary(fun.data = &quot;mean_cl_boot&quot;) + scale_color_brewer(palette = &quot;Set1&quot;) Warning: Removed 1 rows containing non-finite values (stat_summary). Warning: Removed 1 rows containing missing values (geom_point). # variation across scenarios ggplot(data = df.politeness, mapping = aes(x = scenario, y = frequency)) + geom_point(position = position_jitter(width = 0.1), alpha = 0.1) + stat_summary(fun.data = &quot;mean_cl_boot&quot;) + scale_color_brewer(palette = &quot;Set1&quot;) Warning: Removed 1 rows containing non-finite values (stat_summary). Removed 1 rows containing missing values (geom_point). # variation across participants ggplot(data = df.politeness, mapping = aes(x = subject, y = frequency)) + geom_point(position = position_jitter(width = 0.1), alpha = 0.1) + stat_summary(fun.data = &quot;mean_cl_boot&quot;) + scale_color_brewer(palette = &quot;Set1&quot;) Warning: Removed 1 rows containing non-finite values (stat_summary). Removed 1 rows containing missing values (geom_point). We fit the model and compute the contrasts. fit = lmer(formula = frequency ~ 1 + attitude * gender + (1 | subject) + (1 | scenario), data = df.politeness) fit %&gt;% emmeans(specs = pairwise ~ attitude + gender, adjust = &quot;none&quot;) $emmeans attitude gender emmean SE df lower.CL upper.CL inf F 261 16.3 5.73 220.2 301 pol F 233 16.3 5.73 192.8 274 inf M 144 16.3 5.73 104.0 185 pol M 133 16.4 5.80 92.2 173 Degrees-of-freedom method: kenward-roger Confidence level used: 0.95 $contrasts contrast estimate SE df t.ratio p.value inf F - pol F 27.4 7.79 69.00 3.517 0.0008 inf F - inf M 116.2 21.73 4.56 5.348 0.0040 inf F - pol M 128.0 21.77 4.59 5.881 0.0027 pol F - inf M 88.8 21.73 4.56 4.087 0.0115 pol F - pol M 100.6 21.77 4.59 4.623 0.0071 inf M - pol M 11.8 7.90 69.08 1.497 0.1390 Degrees-of-freedom method: kenward-roger Here, I’ve computed all pairwise contrasts. We were only interested in one: inf F - pol F and that one is significant. So the frequency of female participants’ pitch differed between the informal and polite condition. If we had used an ANOVA approach for this data set, we could have done it like so: aov_ez(id = &quot;subject&quot;, dv = &quot;frequency&quot;, between = &quot;gender&quot;, within = &quot;attitude&quot;, data = df.politeness) Converting to factor: gender Warning: More than one observation per design cell, aggregating data using `fun_aggregate = mean`. To turn off this warning, pass `fun_aggregate = mean` explicitly. Warning: Missing values for following ID(s): M4 Removing those cases from the analysis. Contrasts set to contr.sum for the following variables: gender Anova Table (Type 3 tests) Response: frequency Effect df MSE F ges p.value 1 gender 1, 3 1729.42 17.22 * .851 .025 2 attitude 1, 3 3.65 309.71 *** .179 &lt;.001 3 gender:attitude 1, 3 3.65 21.30 * .015 .019 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;+&#39; 0.1 &#39; &#39; 1 This approach ignores the variation across scenarios (and just computed the mean instead). Arguably, the lmer() approach is better here as it takes all of the data into account. 20.5 Mixtures of participants What if we have groups of participants who differ from each other? Let’s generate data for which this is the case. # make example reproducible set.seed(1) sample_size = 20 b0 = 1 b1 = 2 sd_residual = 0.5 sd_participant = 0.5 mean_group1 = 1 mean_group2 = 10 df.mixed = tibble( condition = rep(0:1, each = sample_size), participant = rep(1:sample_size, 2)) %&gt;% group_by(participant) %&gt;% mutate(group = sample(1:2, size = 1), intercept = ifelse(group == 1, rnorm(n(), mean = mean_group1, sd = sd_participant), rnorm(n(), mean = mean_group2, sd = sd_participant))) %&gt;% group_by(condition) %&gt;% mutate(value = b0 + b1 * condition + intercept + rnorm(n(), sd = sd_residual)) %&gt;% ungroup %&gt;% mutate(condition = as.factor(condition), participant = as.factor(participant)) 20.5.0.1 Ignoring mixture Let’ first fit a model that ignores the fact that there are two different groups of participants. # fit model fit.mixed = lmer(formula = value ~ 1 + condition + (1 | participant), data = df.mixed) summary(fit.mixed) Linear mixed model fit by REML. t-tests use Satterthwaite&#39;s method [ lmerModLmerTest] Formula: value ~ 1 + condition + (1 | participant) Data: df.mixed REML criterion at convergence: 163.5 Scaled residuals: Min 1Q Median 3Q Max -1.62997 -0.41663 -0.05607 0.54750 1.54023 Random effects: Groups Name Variance Std.Dev. participant (Intercept) 19.2206 4.3841 Residual 0.3521 0.5934 Number of obs: 40, groups: participant, 20 Fixed effects: Estimate Std. Error df t value Pr(&gt;|t|) (Intercept) 5.8729 0.9893 19.3449 5.937 9.54e-06 *** condition1 1.6652 0.1876 19.0000 8.875 3.47e-08 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Correlation of Fixed Effects: (Intr) condition1 -0.095 Let’s look at the model’s predictions: fit.mixed %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = condition, y = value, group = participant)) + geom_point(alpha = 0.5) + geom_line(alpha = 0.5) + geom_point(aes(y = fitted), color = &quot;red&quot;) + geom_line(aes(y = fitted), color = &quot;red&quot;) And let’s simulate some data from the fitted model: # simulated data fit.mixed %&gt;% simulate() %&gt;% bind_cols(df.mixed) %&gt;% ggplot(data = ., mapping = aes(x = condition, y = sim_1, group = participant)) + geom_line(alpha = 0.5) + geom_point(alpha = 0.5) As we can see, the simulated data doesn’t look like the data that was used to fit the model. 20.5.0.2 Modeling mixture Now, let’s fit a model that takes the differences between groups into account by adding a fixed effect for group. # fit model fit.grouped = lmer(formula = value ~ 1 + group + condition + (1 | participant), data = df.mixed) summary(fit.grouped) Linear mixed model fit by REML. t-tests use Satterthwaite&#39;s method [ lmerModLmerTest] Formula: value ~ 1 + group + condition + (1 | participant) Data: df.mixed REML criterion at convergence: 82.2 Scaled residuals: Min 1Q Median 3Q Max -1.61879 -0.61378 0.02557 0.49842 2.19076 Random effects: Groups Name Variance Std.Dev. participant (Intercept) 0.09265 0.3044 Residual 0.35208 0.5934 Number of obs: 40, groups: participant, 20 Fixed effects: Estimate Std. Error df t value Pr(&gt;|t|) (Intercept) -6.3136 0.3633 20.5655 -17.381 9.10e-14 *** group 8.7046 0.2366 18.0000 36.791 &lt; 2e-16 *** condition1 1.6652 0.1876 19.0000 8.875 3.47e-08 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Correlation of Fixed Effects: (Intr) group group -0.912 condition1 -0.258 0.000 Note how the variance of the random intercepts is much smaller now that we’ve taken the group structure in the data into account. Let’s visualize the model’s predictions: fit.grouped %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = condition, y = value, group = participant)) + geom_point(alpha = 0.5) + geom_line(alpha = 0.5) + geom_point(aes(y = fitted), color = &quot;red&quot;) + geom_line(aes(y = fitted), color = &quot;red&quot;) And simulate some data from the model: # simulated data fit.grouped %&gt;% simulate() %&gt;% bind_cols(df.mixed) %&gt;% ggplot(data = ., mapping = aes(x = condition, y = sim_1, group = participant)) + geom_line(alpha = 0.5) + geom_point(alpha = 0.5) This time, the simulated data looks much more like the data that was used to fit the model. Yay! ggpredict(model = fit.grouped, terms = &quot;condition&quot;) %&gt;% plot() ggpredict(model = fit.mixed, terms = &quot;condition&quot;) %&gt;% plot() 20.5.0.3 Heterogeneity in variance The example above has shown that we can take overall differences between groups into account by adding a fixed effect. Can we also deal with heterogeneity in variance between groups? For example, what if the responses of one group exhibit much more variance than the responses of another group? Let’s first generate some data with heterogeneous variance: # make example reproducible set.seed(1) sample_size = 20 b0 = 1 b1 = 2 sd_residual = 0.5 mean_group1 = 1 sd_group1 = 1 mean_group2 = 30 sd_group2 = 10 df.variance = tibble( condition = rep(0:1, each = sample_size), participant = rep(1:sample_size, 2)) %&gt;% group_by(participant) %&gt;% mutate(group = sample(1:2, size = 1), intercept = ifelse(group == 1, rnorm(n(), mean = mean_group1, sd = sd_group1), rnorm(n(), mean = mean_group2, sd = sd_group2))) %&gt;% group_by(condition) %&gt;% mutate(value = b0 + b1 * condition + intercept + rnorm(n(), sd = sd_residual)) %&gt;% ungroup %&gt;% mutate(condition = as.factor(condition), participant = as.factor(participant)) Let’s fit the model: # fit model fit.variance = lmer(formula = value ~ 1 + group + condition + (1 | participant), data = df.variance) summary(fit.variance) Linear mixed model fit by REML. t-tests use Satterthwaite&#39;s method [ lmerModLmerTest] Formula: value ~ 1 + group + condition + (1 | participant) Data: df.variance REML criterion at convergence: 232.7 Scaled residuals: Min 1Q Median 3Q Max -2.96291 -0.19619 0.03751 0.28317 1.45552 Random effects: Groups Name Variance Std.Dev. participant (Intercept) 17.12 4.137 Residual 13.74 3.706 Number of obs: 40, groups: participant, 20 Fixed effects: Estimate Std. Error df t value Pr(&gt;|t|) (Intercept) -24.0018 3.3669 19.1245 -7.129 8.56e-07 *** group 27.0696 2.2353 18.0000 12.110 4.36e-10 *** condition1 0.5716 1.1720 19.0000 0.488 0.631 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Correlation of Fixed Effects: (Intr) group group -0.929 condition1 -0.174 0.000 Look at the data and model predictions: fit.variance %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = condition, y = value, group = participant)) + geom_point(alpha = 0.5) + geom_line(alpha = 0.5) + geom_point(aes(y = fitted), color = &quot;red&quot;) + geom_line(aes(y = fitted), color = &quot;red&quot;) And the simulated data: # simulated data fit.variance %&gt;% simulate() %&gt;% bind_cols(df.mixed) %&gt;% ggplot(data = ., mapping = aes(x = condition, y = sim_1, group = participant)) + geom_line(alpha = 0.5) + geom_point(alpha = 0.5) The lmer() fails here. It uses one normal distribution to model the variance between participants. It cannot account for the fact that the answers of one group of participants vary more than the answers from another groups of participants. Again, the simulated data doesn’t look like the original data, even though we did take the grouping into account. We will later see that it’s straightforward in Bayesian models to explicitly model heterogeneity in variance. 20.6 Bootstrapping Bootstrapping is a good way to estimate our uncertainty on the parameter estimates in the model. 20.6.1 Linear model Let’s briefly review how to do bootstrapping in a simple linear model. # fit model fit.lm = lm(formula = reaction ~ 1 + days, data = df.sleep) # coefficients coef(fit.lm) (Intercept) days 252.32070 10.32766 # bootstrapping df.boot = df.sleep %&gt;% bootstrap(n = 100, id = &quot;id&quot;) %&gt;% mutate(fit = map(.x = strap, .f = ~ lm(formula = reaction ~ 1 + days, data = .)), tidy = map(.x = fit, .f = tidy)) %&gt;% unnest(tidy) %&gt;% select(id, term, estimate) %&gt;% spread(term, estimate) %&gt;% clean_names() Let’s illustrate the linear model with a confidence interval (making parametric assumptions using the t-distribution). ggplot(data = df.sleep, mapping = aes(x = days, y = reaction)) + geom_smooth(method = &quot;lm&quot;) + geom_point(alpha = 0.3) And let’s compare this with the different regression lines that we get out of our bootstrapped samples: ggplot(data = df.sleep, mapping = aes(x = days, y = reaction)) + geom_abline(data = df.boot, aes(intercept = intercept, slope = days, group = id), alpha = 0.1) + geom_point(alpha = 0.3) 20.6.1.1 bootmer() function For the linear mixed effects model, we can use the bootmer() function to do bootstrapping. set.seed(1) # fit the model fit.lmer = lmer(formula = reaction ~ 1 + days + (1 + days | subject), data = df.sleep) # bootstrap parameter estimates boot.lmer = bootMer(fit.lmer, FUN = fixef, nsim = 100) # compute confidence interval boot.ci(boot.lmer, index = 2, type = &quot;perc&quot;) BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS Based on 100 bootstrap replicates CALL : boot.ci(boot.out = boot.lmer, type = &quot;perc&quot;, index = 2) Intervals : Level Percentile 95% ( 7.26, 13.79 ) Calculations and Intervals on Original Scale Some percentile intervals may be unstable # plot estimates boot.lmer$t %&gt;% as_tibble() %&gt;% clean_names() %&gt;% mutate(id = 1:n()) %&gt;% pivot_longer(cols = -id, names_to = &quot;index&quot;, values_to = &quot;value&quot;) %&gt;% ggplot(data = ., mapping = aes(x = value)) + geom_density() + facet_grid(cols = vars(index), scales = &quot;free&quot;) + coord_cartesian(expand = F) 20.7 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 [4] purrr_0.3.4 readr_2.1.2 tidyr_1.2.0 [7] tibble_3.1.7 ggplot2_3.3.6 tidyverse_1.3.1 [10] emmeans_1.7.3 ggeffects_1.1.2 boot_1.3-28 [13] modelr_0.1.8 datarium_0.1.0 car_3.0-13 [16] carData_3.0-5 afex_1.1-1 lme4_1.1-29 [19] Matrix_1.4-1 broom.mixed_0.2.9.4 janitor_2.1.0 [22] kableExtra_1.3.4 knitr_1.39 loaded via a namespace (and not attached): [1] TH.data_1.1-1 minqa_1.2.4 colorspace_2.0-3 [4] ellipsis_0.3.2 sjlabelled_1.2.0 htmlTable_2.4.0 [7] estimability_1.3 snakecase_0.11.0 base64enc_0.1-3 [10] fs_1.5.2 rstudioapi_0.13 farver_2.1.0 [13] listenv_0.8.0 furrr_0.3.0 bit64_4.0.5 [16] fansi_1.0.3 mvtnorm_1.1-3 lubridate_1.8.0 [19] xml2_1.3.3 codetools_0.2-18 splines_4.1.2 [22] Formula_1.2-4 jsonlite_1.8.0 nloptr_2.0.0 [25] pbkrtest_0.5.1 broom_0.8.0 cluster_2.1.3 [28] dbplyr_2.1.1 png_0.1-7 compiler_4.1.2 [31] httr_1.4.3 backports_1.4.1 assertthat_0.2.1 [34] fastmap_1.1.0 cli_3.3.0 htmltools_0.5.2 [37] tools_4.1.2 lmerTest_3.1-3 coda_0.19-4 [40] gtable_0.3.0 glue_1.6.2 reshape2_1.4.4 [43] Rcpp_1.0.8.3 cellranger_1.1.0 jquerylib_0.1.4 [46] vctrs_0.4.1 svglite_2.1.0 nlme_3.1-157 [49] insight_0.17.0 xfun_0.30 globals_0.14.0 [52] rvest_1.0.2 lifecycle_1.0.1 future_1.25.0 [55] MASS_7.3-57 zoo_1.8-10 scales_1.2.0 [58] vroom_1.5.7 hms_1.1.1 parallel_4.1.2 [61] sandwich_3.0-1 RColorBrewer_1.1-3 yaml_2.3.5 [64] gridExtra_2.3 sass_0.4.1 rpart_4.1.16 [67] latticeExtra_0.6-29 stringi_1.7.6 highr_0.9 [70] checkmate_2.1.0 rlang_1.0.2 pkgconfig_2.0.3 [73] systemfonts_1.0.4 evaluate_0.15 lattice_0.20-45 [76] labeling_0.4.2 htmlwidgets_1.5.4 bit_4.0.4 [79] tidyselect_1.1.2 parallelly_1.31.1 plyr_1.8.7 [82] magrittr_2.0.3 bookdown_0.26 R6_2.5.1 [85] generics_0.1.2 Hmisc_4.7-0 multcomp_1.4-19 [88] DBI_1.1.2 mgcv_1.8-40 foreign_0.8-82 [91] pillar_1.7.0 haven_2.5.0 withr_2.5.0 [94] nnet_7.3-17 survival_3.3-1 abind_1.4-5 [97] crayon_1.5.1 utf8_1.2.2 tzdb_0.3.0 [100] rmarkdown_2.14 jpeg_0.1-9 grid_4.1.2 [103] readxl_1.4.0 data.table_1.14.2 reprex_2.0.1 [106] digest_0.6.29 webshot_0.5.3 xtable_1.8-4 [109] numDeriv_2016.8-1.1 munsell_0.5.0 viridisLite_0.4.0 [112] bslib_0.3.1 "],["generalized-linear-model.html", "Chapter 21 Generalized linear model 21.1 Learning goals 21.2 Load packages and set plotting theme 21.3 Load data set 21.4 Logistic regression 21.5 Simulate a logistic regression 21.6 Testing hypotheses 21.7 Logistic mixed effects model 21.8 Additional information 21.9 Session info", " Chapter 21 Generalized linear model 21.1 Learning goals Logistic regression. Logit transform. Fitting a logistic regression in R. Visualizing and interpreting model predictions. Simulating data from a logistic regression. Assessing model fit. Testing hypotheses. Reporting results. Mixed effects logistic regression. 21.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;titanic&quot;) # titanic dataset library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;broom&quot;) # for tidying up linear models library(&quot;lme4&quot;) # for linear mixed effects models library(&quot;boot&quot;) # for bootstrapping (also has an inverse logit function) library(&quot;ggeffects&quot;) # for showing effects in linear, generalized linear, and other models library(&quot;afex&quot;) # for significance testing of mixed effects models library(&quot;emmeans&quot;) # for the joint_tests() function library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 21.3 Load data set df.titanic = titanic_train %&gt;% clean_names() %&gt;% mutate(sex = as.factor(sex)) Let’s take a quick look at the data: df.titanic %&gt;% glimpse() Rows: 891 Columns: 12 $ passenger_id &lt;int&gt; 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17… $ survived &lt;int&gt; 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, … $ pclass &lt;int&gt; 3, 1, 3, 1, 3, 3, 1, 3, 3, 2, 3, 1, 3, 3, 3, 2, 3, 2, 3, … $ name &lt;chr&gt; &quot;Braund, Mr. Owen Harris&quot;, &quot;Cumings, Mrs. John Bradley (F… $ sex &lt;fct&gt; male, female, female, female, male, male, male, male, fem… $ age &lt;dbl&gt; 22, 38, 26, 35, 35, NA, 54, 2, 27, 14, 4, 58, 20, 39, 14,… $ sib_sp &lt;int&gt; 1, 1, 0, 1, 0, 0, 0, 3, 0, 1, 1, 0, 0, 1, 0, 0, 4, 0, 1, … $ parch &lt;int&gt; 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 1, 0, 0, 5, 0, 0, 1, 0, 0, … $ ticket &lt;chr&gt; &quot;A/5 21171&quot;, &quot;PC 17599&quot;, &quot;STON/O2. 3101282&quot;, &quot;113803&quot;, &quot;3… $ fare &lt;dbl&gt; 7.2500, 71.2833, 7.9250, 53.1000, 8.0500, 8.4583, 51.8625… $ cabin &lt;chr&gt; &quot;&quot;, &quot;C85&quot;, &quot;&quot;, &quot;C123&quot;, &quot;&quot;, &quot;&quot;, &quot;E46&quot;, &quot;&quot;, &quot;&quot;, &quot;&quot;, &quot;G6&quot;, &quot;… $ embarked &lt;chr&gt; &quot;S&quot;, &quot;C&quot;, &quot;S&quot;, &quot;S&quot;, &quot;S&quot;, &quot;Q&quot;, &quot;S&quot;, &quot;S&quot;, &quot;S&quot;, &quot;C&quot;, &quot;S&quot;, &quot;S… # Table of the first 10 entries df.titanic %&gt;% head(10) %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) passenger_id survived pclass name sex age sib_sp parch ticket fare cabin embarked 1 0 3 Braund, Mr. Owen Harris male 22 1 0 A/5 21171 7.25 S 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Thayer) female 38 1 0 PC 17599 71.28 C85 C 3 1 3 Heikkinen, Miss. Laina female 26 0 0 STON/O2. 3101282 7.92 S 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35 1 0 113803 53.10 C123 S 5 0 3 Allen, Mr. William Henry male 35 0 0 373450 8.05 S 6 0 3 Moran, Mr. James male NA 0 0 330877 8.46 Q 7 0 1 McCarthy, Mr. Timothy J male 54 0 0 17463 51.86 E46 S 8 0 3 Palsson, Master. Gosta Leonard male 2 3 1 349909 21.08 S 9 1 3 Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg) female 27 0 2 347742 11.13 S 10 1 2 Nasser, Mrs. Nicholas (Adele Achem) female 14 1 0 237736 30.07 C 21.4 Logistic regression Let’s see if we can predict whether or not a passenger survived based on the price of their ticket. Let’s run a simple regression first: # fit a linear model fit.lm = lm(formula = survived ~ 1 + fare, data = df.titanic) # summarize the results fit.lm %&gt;% summary() Call: lm(formula = survived ~ 1 + fare, data = df.titanic) Residuals: Min 1Q Median 3Q Max -0.9653 -0.3391 -0.3222 0.6044 0.6973 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.3026994 0.0187849 16.114 &lt; 2e-16 *** fare 0.0025195 0.0003174 7.939 6.12e-15 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 0.4705 on 889 degrees of freedom Multiple R-squared: 0.06621, Adjusted R-squared: 0.06516 F-statistic: 63.03 on 1 and 889 DF, p-value: 6.12e-15 Look’s like fare is a significant predictor of whether or not a person survived. Let’s visualize the model’s predictions: ggplot(data = df.titanic, mapping = aes(x = fare, y = survived)) + geom_smooth(method = &quot;lm&quot;) + geom_point() + labs(y = &quot;survived&quot;) This doesn’t look good! The model predicts intermediate values of survived (which doesn’t make sense given that a person either survived or didn’t survive). Furthermore, the model predicts values greater than 1 for fares greather than ~ 300. Let’s run a logistic regression instead. # fit a logistic regression fit.glm = glm(formula = survived ~ 1 + fare, family = &quot;binomial&quot;, data = df.titanic) fit.glm %&gt;% summary() Call: glm(formula = survived ~ 1 + fare, family = &quot;binomial&quot;, data = df.titanic) Deviance Residuals: Min 1Q Median 3Q Max -2.4906 -0.8878 -0.8531 1.3429 1.5942 Coefficients: Estimate Std. Error z value Pr(&gt;|z|) (Intercept) -0.941330 0.095129 -9.895 &lt; 2e-16 *** fare 0.015197 0.002232 6.810 9.79e-12 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 1186.7 on 890 degrees of freedom Residual deviance: 1117.6 on 889 degrees of freedom AIC: 1121.6 Number of Fisher Scoring iterations: 4 And let’s visualize the predictions of the logistic regression: ggplot(data = df.titanic, mapping = aes(x = fare, y = survived)) + geom_smooth(method = &quot;glm&quot;, method.args = list(family = &quot;binomial&quot;)) + geom_point() + labs(y = &quot;p(survived)&quot;) Much better! Note that we’ve changed the interpretation of our dependent variable. We are now predicting the probability that a person survived based on their fare. The model now only predicts values between 0 and 1. To achieve this, we apply a logit transform to the outcome variable like so: \\[ \\ln(\\frac{\\pi_i}{1-\\pi_i}) = b_0 + b_1 \\cdot X_i + e_i \\] where \\(\\pi_i\\) is the probability of passenger \\(i\\) having survived. Importantly, this affects our interpretation of the model parameters. They are now defined in log-odds, and can apply an inverse logit transformation to turn this back into a probability: With \\[ \\pi = P(Y = 1) \\] and the logit transformation \\[ \\ln(\\frac{\\pi}{1-\\pi}) = V, \\] where \\(V\\) is just a placeholder for our linear model formula, we can go back to \\(\\pi\\) through the inverse logit transformation like so: \\[ \\pi = \\frac{e^V}{1 + e^V} \\] In R, we can use log(x) to calculate the natural logarithm \\(\\ln(x)\\), and exp(x) to calculate e^x. 21.4.1 Interpreting the parameters fit.glm %&gt;% summary() Call: glm(formula = survived ~ 1 + fare, family = &quot;binomial&quot;, data = df.titanic) Deviance Residuals: Min 1Q Median 3Q Max -2.4906 -0.8878 -0.8531 1.3429 1.5942 Coefficients: Estimate Std. Error z value Pr(&gt;|z|) (Intercept) -0.941330 0.095129 -9.895 &lt; 2e-16 *** fare 0.015197 0.002232 6.810 9.79e-12 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 1186.7 on 890 degrees of freedom Residual deviance: 1117.6 on 889 degrees of freedom AIC: 1121.6 Number of Fisher Scoring iterations: 4 The estimate for the intercept and fare are in log-odds. Let’s take a look at the linear model’s predictions in log-odds space. df.plot = fit.glm %&gt;% augment() %&gt;% clean_names() ggplot(data = df.plot, mapping = aes(x = fare, y = fitted)) + geom_line() Nice, looks like a good old linear model. But what’s the y-axis here? It’s in log-odds (buh!). This is difficult to interpret. Let’s transform the y-axis back to probabilities to make it easier to see what’s going on. ggplot(data = df.plot, mapping = aes(x = fare, y = inv.logit(fitted))) + geom_line() Great! Now the y-axis is back in probability space. We used the inverse logit function inv.logit() to transfer to log-odds back into probabilities. Let’s check what the intercept of our model is in probability space: fit.glm %&gt;% pluck(coefficients, 1) %&gt;% inv.logit() [1] 0.2806318 Here, we see that the intercept is \\(p = 0.28\\). That is, the predicted chance of survival for someone who didn’t pay any fare at all is 28% according to the model. Interpreting the slope is a little more tricky. Let’s look at a situation first where we have a binary predictor. 21.4.1.1 Logit transform Here is a visualization of what the odds and log odds transformation look like. # going from probabilities to odds (ranges from 0 to +infinity) ggplot(data = tibble(x = seq(0, 1, 0.1)), mapping = aes(x = x)) + stat_function(fun = ~ ./(1 - .), size = 1) # going from probabilities to log odds (ranges from -infinity to +infinity) ggplot(data = tibble(x = seq(0, 1, 0.1)), mapping = aes(x = x)) + stat_function(fun = ~ log(./(1 - .)), size = 1) 21.4.2 Binary predictor Let’s see whether the probability of survival differed between male and female passengers. fit.glm2 = glm(formula = survived ~ 1 + sex, family = &quot;binomial&quot;, data = df.titanic) fit.glm2 %&gt;% summary() Call: glm(formula = survived ~ 1 + sex, family = &quot;binomial&quot;, data = df.titanic) Deviance Residuals: Min 1Q Median 3Q Max -1.6462 -0.6471 -0.6471 0.7725 1.8256 Coefficients: Estimate Std. Error z value Pr(&gt;|z|) (Intercept) 1.0566 0.1290 8.191 2.58e-16 *** sexmale -2.5137 0.1672 -15.036 &lt; 2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 1186.7 on 890 degrees of freedom Residual deviance: 917.8 on 889 degrees of freedom AIC: 921.8 Number of Fisher Scoring iterations: 4 It looks like it did! Let’s visualize: df.titanic %&gt;% mutate(survived = factor(survived, labels = c(&quot;died&quot;, &quot;survived&quot;))) %&gt;% ggplot(data = ., mapping = aes(x = sex, fill = survived)) + geom_bar(position = &quot;fill&quot;, color = &quot;black&quot;) + scale_fill_brewer(palette = &quot;Set1&quot;) + labs(x = &quot;&quot;, fill = &quot;&quot;, y = &quot;probability&quot;) And let’s interpret the parameters by applying the inverse logit transform. To get the prediction for female passengers we get \\[ \\widehat{\\ln(\\frac{\\pi_i}{1-\\pi_i})} = b_0 + b_1 \\cdot \\text{sex}_i = b_0 + b_1 \\cdot 0 = b_0 \\] since we dummy coded the predictor and female is our reference category. To get the predicted probability of survival for women we do the logit transform: \\[ \\pi = \\frac{e^{b_0}}{1 + e^{b_0}} \\] The predicted probability is: fit.glm2 %&gt;% pluck(coefficients, 1) %&gt;% inv.logit() [1] 0.7420382 To get the prediction for male passengers we have: \\[ \\widehat{\\ln(\\frac{\\pi_i}{1-\\pi_i})} = b_0 + b_1 \\cdot \\text{sex}_i = b_0 + b_1 \\cdot 1 = b_0 + b_1 \\] Applying the logit transform like so \\[ \\pi = \\frac{e^{b_0 + b_1}}{1 + e^{b_0 + b_1}} \\] The predicted probability of male passengers surviving is: fit.glm2 %&gt;% pluck(coefficients) %&gt;% sum() %&gt;% inv.logit() [1] 0.1889081 Here is the same information in a table: df.titanic %&gt;% count(sex, survived) %&gt;% mutate(p = n / sum(n)) %&gt;% group_by(sex) %&gt;% mutate(`p(survived|sex)` = p / sum(p)) # A tibble: 4 × 5 # Groups: sex [2] sex survived n p `p(survived|sex)` &lt;fct&gt; &lt;int&gt; &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; 1 female 0 81 0.0909 0.258 2 female 1 233 0.262 0.742 3 male 0 468 0.525 0.811 4 male 1 109 0.122 0.189 21.4.3 Continuous predictor To interpret the predictions when a continuous predictor is involved, it’s easiest to consider a few concrete cases. Here, I use the augment() function from the “broom” package to get the model’s predictions for some values of interest: fit.glm %&gt;% augment(newdata = tibble(fare = c(0, 10, 50, 100, 500))) %&gt;% clean_names() %&gt;% select(fare, prediction = fitted) %&gt;% mutate(`p(survival)` = inv.logit(prediction)) # A tibble: 5 × 3 fare prediction `p(survival)` &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 0 -0.941 0.281 2 10 -0.789 0.312 3 50 -0.181 0.455 4 100 0.578 0.641 5 500 6.66 0.999 21.4.4 Several predictors Let’s fit a logistic regression that predicts the probability of survival based both on the passenger’s sex and what fare they paid (allowing for an interaction of the two predictors): fit.glm3 = glm(formula = survived ~ 1 + sex * fare, family = &quot;binomial&quot;, data = df.titanic) fit.glm3 %&gt;% summary() Call: glm(formula = survived ~ 1 + sex * fare, family = &quot;binomial&quot;, data = df.titanic) Deviance Residuals: Min 1Q Median 3Q Max -2.6280 -0.6279 -0.5991 0.8172 1.9288 Coefficients: Estimate Std. Error z value Pr(&gt;|z|) (Intercept) 0.408428 0.189999 2.150 0.031584 * sexmale -2.099345 0.230291 -9.116 &lt; 2e-16 *** fare 0.019878 0.005372 3.701 0.000215 *** sexmale:fare -0.011617 0.005934 -1.958 0.050252 . --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 1186.66 on 890 degrees of freedom Residual deviance: 879.85 on 887 degrees of freedom AIC: 887.85 Number of Fisher Scoring iterations: 5 Make sure not to interpret the significance test on the coefficients here as main effects. Based on this summary table, you cannot say whether there is a significant difference between male vs. female passenger in their probability of survival. What coefficient for sexmale captures is whether there is a significant difference between male and female passengers who paid a fare of 0. That is, it’s the predicted difference between the reference category (female) and the other category (male) when all other predictors are 0. Let’s visualize the model predictions: ggplot(data = df.titanic, mapping = aes(x = fare, y = survived, color = sex)) + geom_point(alpha = 0.1, size = 2) + geom_smooth(method = &quot;glm&quot;, method.args = list(family = &quot;binomial&quot;), alpha = 0.2, aes(fill = sex)) + scale_color_brewer(palette = &quot;Set1&quot;) Just for kicks, to get a better sense for what the interaction looks like, here is the visualization in log-odds space: fit.glm3 %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = fare, color = sex, y = fitted)) + geom_line() + scale_color_brewer(palette = &quot;Set1&quot;) Let’s see how large the difference between genders is once we take into account how much each person paid for the fair: ggpredict(fit.glm3, terms = c(&quot;sex&quot;)) # Predicted probabilities of survived sex | Predicted | 95% CI --------------------------------- female | 0.74 | [0.69, 0.79] male | 0.19 | [0.16, 0.23] Adjusted for: * fare = 32.20 We notice that there is one outlier who was male and paid a $500 fare (or maybe this is a mistake in the data entry?!). Let’s remove this outlier and see what happens: fit.glm3_no_outlier = glm(formula = survived ~ 1 + sex * fare, family = &quot;binomial&quot;, data = df.titanic %&gt;% filter(fare &lt; 500)) fit.glm3_no_outlier %&gt;% summary() Call: glm(formula = survived ~ 1 + sex * fare, family = &quot;binomial&quot;, data = df.titanic %&gt;% filter(fare &lt; 500)) Deviance Residuals: Min 1Q Median 3Q Max -2.6280 -0.6288 -0.6018 0.8172 1.9226 Coefficients: Estimate Std. Error z value Pr(&gt;|z|) (Intercept) 0.408436 0.190019 2.149 0.031598 * sexmale -2.085344 0.232260 -8.979 &lt; 2e-16 *** fare 0.019878 0.005373 3.699 0.000216 *** sexmale:fare -0.012178 0.006066 -2.008 0.044688 * --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 1180.89 on 887 degrees of freedom Residual deviance: 879.51 on 884 degrees of freedom AIC: 887.51 Number of Fisher Scoring iterations: 5 df.titanic %&gt;% filter(fare &lt; 500) %&gt;% mutate(sex = as.factor(sex)) %&gt;% ggplot(data = ., mapping = aes(x = fare, y = survived, color = sex)) + geom_point(alpha = 0.1, size = 2) + stat_smooth(method = &quot;glm&quot;, method.args = list(family = &quot;binomial&quot;), alpha = 0.2, fullrange = T, aes(fill = sex)) + scale_color_brewer(palette = &quot;Set1&quot;) + scale_x_continuous(limits = c(0, 500)) There is still a clear difference between female and male passengers, but the prediction for male passengers has changed a bit. Let’s look at a concrete example: # with the outlier: # predicted probability of survival for a male passenger who paid $200 for their fare inv.logit(fit.glm3$coefficients[1] + fit.glm3$coefficients[2] + fit.glm3$coefficients[3] * 200 + fit.glm3$coefficients[4] * 200) (Intercept) 0.4903402 # without the outlier: # predicted probability of survival for a male passenger who paid $200 for their fare inv.logit(fit.glm3_no_outlier$coefficients[1] + fit.glm3_no_outlier$coefficients[2] + fit.glm3_no_outlier$coefficients[3] * 200 + fit.glm3_no_outlier$coefficients[4] * 200) (Intercept) 0.4658284 With the outlier removed, the predicted probability of survival for a male passenger who paid $200 decreases from 49% to 47%. 21.4.5 Using the “ggeffects” package The “ggeffects” package helps with the interpretation of the results. It applies the inverse logit transform for us, and shows the predictions for a range of cases. # show effects ggeffect(model = fit.glm, terms = &quot;fare [0, 100, 200, 300, 400, 500]&quot;) # Predicted probabilities of survived fare | Predicted | 95% CI ------------------------------- 0 | 0.28 | [0.24, 0.32] 100 | 0.64 | [0.56, 0.72] 200 | 0.89 | [0.79, 0.95] 300 | 0.97 | [0.92, 0.99] 400 | 0.99 | [0.97, 1.00] 500 | 1.00 | [0.99, 1.00] I’ve used the [] construction to specify for what values of the predictor fare, I’d like get the predicted values. Here, the prediction is marginalized across both women and men. We can also get a plot of the model predictions like so: ggeffect(model = fit.glm, terms = &quot;fare [0, 100, 200, 300, 400, 500]&quot;) %&gt;% plot() And, we can also get the predicted probability of survival for sex marginalized over the fare, using the model which included both sex and fare, as well as its interaction as predictors. ggeffect(model = fit.glm3, terms = &quot;sex&quot;) # Predicted probabilities of survived sex | Predicted | 95% CI --------------------------------- female | 0.74 | [0.69, 0.79] male | 0.19 | [0.16, 0.23] Finally, we can ask for predictions for specific combinations of our predictor variables, by using the ggpredict() function. ggpredict(model = fit.glm3, terms = c(&quot;sex&quot;, &quot;fare [200]&quot;)) # Predicted probabilities of survived sex | Predicted | 95% CI --------------------------------- female | 0.99 | [0.93, 1.00] male | 0.49 | [0.29, 0.70] The example above, shows the predicted probability of survival for male vs. female passengers, assuming that they paid 200 for the fare. 21.5 Simulate a logistic regression As always, to better understand a statistical modeling procedure, it’s helpful to simulate data from the assumed data-generating process, fit the model, and see whether we can reconstruct the parameters. # make example reproducible set.seed(1) # set parameters sample_size = 1000 b0 = 0 b1 = 1 # b1 = 8 # generate data df.data = tibble(x = rnorm(n = sample_size), y = b0 + b1 * x, p = inv.logit(y)) %&gt;% mutate(response = rbinom(n(), size = 1, p = p)) # fit model fit = glm(formula = response ~ 1 + x, family = &quot;binomial&quot;, data = df.data) # model summary fit %&gt;% summary() Call: glm(formula = response ~ 1 + x, family = &quot;binomial&quot;, data = df.data) Deviance Residuals: Min 1Q Median 3Q Max -2.1137 -1.0118 -0.4591 1.0287 2.2591 Coefficients: Estimate Std. Error z value Pr(&gt;|z|) (Intercept) -0.06214 0.06918 -0.898 0.369 x 0.92905 0.07937 11.705 &lt;2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 1385.4 on 999 degrees of freedom Residual deviance: 1209.6 on 998 degrees of freedom AIC: 1213.6 Number of Fisher Scoring iterations: 3 df.data %&gt;% kable(digits = 2) %&gt;% kable_styling(bootstrap_options = &quot;striped&quot;, full_width = F) x y p response -0.63 -0.63 0.35 1 0.18 0.18 0.55 0 -0.84 -0.84 0.30 1 1.60 1.60 0.83 1 0.33 0.33 0.58 1 -0.82 -0.82 0.31 0 0.49 0.49 0.62 1 0.74 0.74 0.68 1 0.58 0.58 0.64 1 -0.31 -0.31 0.42 0 1.51 1.51 0.82 1 0.39 0.39 0.60 1 -0.62 -0.62 0.35 1 -2.21 -2.21 0.10 0 1.12 1.12 0.75 1 -0.04 -0.04 0.49 0 -0.02 -0.02 0.50 0 0.94 0.94 0.72 1 0.82 0.82 0.69 0 0.59 0.59 0.64 1 0.92 0.92 0.71 1 0.78 0.78 0.69 0 0.07 0.07 0.52 1 -1.99 -1.99 0.12 0 0.62 0.62 0.65 1 -0.06 -0.06 0.49 1 -0.16 -0.16 0.46 0 -1.47 -1.47 0.19 0 -0.48 -0.48 0.38 1 0.42 0.42 0.60 1 1.36 1.36 0.80 1 -0.10 -0.10 0.47 0 0.39 0.39 0.60 1 -0.05 -0.05 0.49 1 -1.38 -1.38 0.20 0 -0.41 -0.41 0.40 0 -0.39 -0.39 0.40 0 -0.06 -0.06 0.49 1 1.10 1.10 0.75 0 0.76 0.76 0.68 1 -0.16 -0.16 0.46 0 -0.25 -0.25 0.44 0 0.70 0.70 0.67 0 0.56 0.56 0.64 1 -0.69 -0.69 0.33 1 -0.71 -0.71 0.33 0 0.36 0.36 0.59 0 0.77 0.77 0.68 1 -0.11 -0.11 0.47 1 0.88 0.88 0.71 0 0.40 0.40 0.60 1 -0.61 -0.61 0.35 1 0.34 0.34 0.58 1 -1.13 -1.13 0.24 0 1.43 1.43 0.81 1 1.98 1.98 0.88 1 -0.37 -0.37 0.41 0 -1.04 -1.04 0.26 0 0.57 0.57 0.64 0 -0.14 -0.14 0.47 0 2.40 2.40 0.92 1 -0.04 -0.04 0.49 1 0.69 0.69 0.67 1 0.03 0.03 0.51 0 -0.74 -0.74 0.32 0 0.19 0.19 0.55 0 -1.80 -1.80 0.14 0 1.47 1.47 0.81 1 0.15 0.15 0.54 0 2.17 2.17 0.90 1 0.48 0.48 0.62 0 -0.71 -0.71 0.33 1 0.61 0.61 0.65 1 -0.93 -0.93 0.28 1 -1.25 -1.25 0.22 1 0.29 0.29 0.57 1 -0.44 -0.44 0.39 1 0.00 0.00 0.50 1 0.07 0.07 0.52 1 -0.59 -0.59 0.36 0 -0.57 -0.57 0.36 1 -0.14 -0.14 0.47 1 1.18 1.18 0.76 1 -1.52 -1.52 0.18 0 0.59 0.59 0.64 1 0.33 0.33 0.58 0 1.06 1.06 0.74 1 -0.30 -0.30 0.42 1 0.37 0.37 0.59 0 0.27 0.27 0.57 1 -0.54 -0.54 0.37 1 1.21 1.21 0.77 1 1.16 1.16 0.76 0 0.70 0.70 0.67 1 1.59 1.59 0.83 1 0.56 0.56 0.64 1 -1.28 -1.28 0.22 0 -0.57 -0.57 0.36 1 -1.22 -1.22 0.23 0 -0.47 -0.47 0.38 0 -0.62 -0.62 0.35 0 0.04 0.04 0.51 0 -0.91 -0.91 0.29 0 0.16 0.16 0.54 1 -0.65 -0.65 0.34 0 1.77 1.77 0.85 0 0.72 0.72 0.67 0 0.91 0.91 0.71 1 0.38 0.38 0.59 0 1.68 1.68 0.84 1 -0.64 -0.64 0.35 0 -0.46 -0.46 0.39 0 1.43 1.43 0.81 0 -0.65 -0.65 0.34 1 -0.21 -0.21 0.45 0 -0.39 -0.39 0.40 0 -0.32 -0.32 0.42 1 -0.28 -0.28 0.43 1 0.49 0.49 0.62 1 -0.18 -0.18 0.46 1 -0.51 -0.51 0.38 0 1.34 1.34 0.79 1 -0.21 -0.21 0.45 1 -0.18 -0.18 0.46 0 -0.10 -0.10 0.47 1 0.71 0.71 0.67 1 -0.07 -0.07 0.48 0 -0.04 -0.04 0.49 0 -0.68 -0.68 0.34 0 -0.32 -0.32 0.42 1 0.06 0.06 0.52 1 -0.59 -0.59 0.36 1 0.53 0.53 0.63 1 -1.52 -1.52 0.18 0 0.31 0.31 0.58 1 -1.54 -1.54 0.18 0 -0.30 -0.30 0.43 0 -0.53 -0.53 0.37 1 -0.65 -0.65 0.34 0 -0.06 -0.06 0.49 0 -1.91 -1.91 0.13 0 1.18 1.18 0.76 1 -1.66 -1.66 0.16 0 -0.46 -0.46 0.39 0 -1.12 -1.12 0.25 0 -0.75 -0.75 0.32 0 2.09 2.09 0.89 1 0.02 0.02 0.50 1 -1.29 -1.29 0.22 1 -1.64 -1.64 0.16 0 0.45 0.45 0.61 1 -0.02 -0.02 0.50 1 -0.32 -0.32 0.42 0 -0.93 -0.93 0.28 0 -1.49 -1.49 0.18 0 -1.08 -1.08 0.25 0 1.00 1.00 0.73 1 -0.62 -0.62 0.35 0 -1.38 -1.38 0.20 0 1.87 1.87 0.87 1 0.43 0.43 0.60 0 -0.24 -0.24 0.44 1 1.06 1.06 0.74 1 0.89 0.89 0.71 1 -0.62 -0.62 0.35 1 2.21 2.21 0.90 1 -0.26 -0.26 0.44 1 -1.42 -1.42 0.19 0 -0.14 -0.14 0.46 0 0.21 0.21 0.55 1 2.31 2.31 0.91 1 0.11 0.11 0.53 1 0.46 0.46 0.61 0 -0.08 -0.08 0.48 1 -0.33 -0.33 0.42 0 -0.03 -0.03 0.49 0 0.79 0.79 0.69 0 2.08 2.08 0.89 1 1.03 1.03 0.74 1 1.21 1.21 0.77 1 -1.23 -1.23 0.23 1 0.98 0.98 0.73 0 0.22 0.22 0.55 0 -1.47 -1.47 0.19 0 0.52 0.52 0.63 1 -0.16 -0.16 0.46 1 1.46 1.46 0.81 1 -0.77 -0.77 0.32 0 -0.43 -0.43 0.39 0 -0.93 -0.93 0.28 0 -0.18 -0.18 0.46 0 0.40 0.40 0.60 0 -0.73 -0.73 0.32 1 0.83 0.83 0.70 0 -1.21 -1.21 0.23 1 -1.05 -1.05 0.26 0 1.44 1.44 0.81 1 -1.02 -1.02 0.27 0 0.41 0.41 0.60 0 -0.38 -0.38 0.41 0 0.41 0.41 0.60 1 1.69 1.69 0.84 0 1.59 1.59 0.83 1 -0.33 -0.33 0.42 0 -2.29 -2.29 0.09 1 2.50 2.50 0.92 1 0.67 0.67 0.66 1 0.54 0.54 0.63 1 -0.01 -0.01 0.50 1 0.51 0.51 0.62 1 -0.16 -0.16 0.46 1 0.42 0.42 0.60 0 -0.40 -0.40 0.40 0 -1.37 -1.37 0.20 0 0.99 0.99 0.73 1 1.52 1.52 0.82 1 -0.31 -0.31 0.42 0 -1.25 -1.25 0.22 0 0.64 0.64 0.66 0 -0.04 -0.04 0.49 0 -1.73 -1.73 0.15 0 0.00 0.00 0.50 0 -0.63 -0.63 0.35 1 -0.34 -0.34 0.42 0 -1.16 -1.16 0.24 0 1.80 1.80 0.86 1 -0.33 -0.33 0.42 1 -1.61 -1.61 0.17 0 0.20 0.20 0.55 1 0.26 0.26 0.57 0 -0.99 -0.99 0.27 1 -2.89 -2.89 0.05 0 -0.64 -0.64 0.35 0 0.57 0.57 0.64 1 -0.06 -0.06 0.49 0 -0.10 -0.10 0.48 1 0.56 0.56 0.64 1 -1.19 -1.19 0.23 0 1.10 1.10 0.75 1 -0.01 -0.01 0.50 1 0.71 0.71 0.67 1 1.03 1.03 0.74 1 0.22 0.22 0.56 1 -0.88 -0.88 0.29 0 1.16 1.16 0.76 1 -2.00 -2.00 0.12 0 -0.54 -0.54 0.37 0 -0.26 -0.26 0.44 0 -0.17 -0.17 0.46 1 1.02 1.02 0.74 0 0.14 0.14 0.53 1 0.41 0.41 0.60 0 -0.07 -0.07 0.48 0 -0.25 -0.25 0.44 0 0.70 0.70 0.67 1 1.15 1.15 0.76 1 -2.40 -2.40 0.08 0 0.57 0.57 0.64 1 0.37 0.37 0.59 1 -0.43 -0.43 0.40 1 0.95 0.95 0.72 0 -0.39 -0.39 0.40 0 -0.28 -0.28 0.43 0 0.86 0.86 0.70 1 1.72 1.72 0.85 1 0.27 0.27 0.57 1 -0.42 -0.42 0.40 0 -1.19 -1.19 0.23 1 -0.33 -0.33 0.42 0 -0.94 -0.94 0.28 1 -0.26 -0.26 0.44 1 0.39 0.39 0.60 0 -0.85 -0.85 0.30 0 2.65 2.65 0.93 1 0.16 0.16 0.54 1 1.13 1.13 0.76 0 -2.29 -2.29 0.09 1 0.74 0.74 0.68 1 -1.32 -1.32 0.21 0 0.92 0.92 0.72 0 0.40 0.40 0.60 0 -0.41 -0.41 0.40 1 1.32 1.32 0.79 1 -0.70 -0.70 0.33 0 -0.58 -0.58 0.36 0 -1.00 -1.00 0.27 1 -0.67 -0.67 0.34 1 0.95 0.95 0.72 1 0.43 0.43 0.61 0 1.01 1.01 0.73 1 -0.39 -0.39 0.40 0 0.38 0.38 0.59 1 0.24 0.24 0.56 1 -1.43 -1.43 0.19 0 1.78 1.78 0.86 1 0.13 0.13 0.53 1 0.77 0.77 0.68 0 0.96 0.96 0.72 1 -0.05 -0.05 0.49 0 -0.31 -0.31 0.42 1 0.89 0.89 0.71 1 -1.05 -1.05 0.26 1 1.97 1.97 0.88 1 -0.38 -0.38 0.41 1 1.65 1.65 0.84 1 1.51 1.51 0.82 1 0.08 0.08 0.52 1 0.57 0.57 0.64 0 -1.02 -1.02 0.26 0 0.32 0.32 0.58 0 1.04 1.04 0.74 1 0.10 0.10 0.52 1 -0.45 -0.45 0.39 0 -0.66 -0.66 0.34 0 -0.04 -0.04 0.49 0 1.07 1.07 0.74 1 -0.48 -0.48 0.38 0 -0.12 -0.12 0.47 0 -1.29 -1.29 0.22 0 0.49 0.49 0.62 0 1.31 1.31 0.79 1 1.50 1.50 0.82 1 0.81 0.81 0.69 0 -1.87 -1.87 0.13 0 0.48 0.48 0.62 1 0.46 0.46 0.61 1 -0.35 -0.35 0.41 0 0.17 0.17 0.54 0 -0.86 -0.86 0.30 0 0.68 0.68 0.66 0 -0.33 -0.33 0.42 0 -1.57 -1.57 0.17 0 -0.37 -0.37 0.41 1 1.36 1.36 0.80 1 -0.33 -0.33 0.42 0 0.73 0.73 0.68 0 0.95 0.95 0.72 1 0.00 0.00 0.50 1 -0.35 -0.35 0.41 0 -0.53 -0.53 0.37 1 0.74 0.74 0.68 1 -1.06 -1.06 0.26 0 0.25 0.25 0.56 1 -0.29 -0.29 0.43 1 -2.26 -2.26 0.09 0 -1.41 -1.41 0.20 1 0.92 0.92 0.71 0 -0.19 -0.19 0.45 1 0.80 0.80 0.69 0 1.89 1.89 0.87 1 1.47 1.47 0.81 0 0.68 0.68 0.66 0 0.38 0.38 0.59 1 -0.19 -0.19 0.45 1 1.58 1.58 0.83 1 0.60 0.60 0.64 1 -1.17 -1.17 0.24 0 -0.16 -0.16 0.46 1 -1.92 -1.92 0.13 0 -0.20 -0.20 0.45 0 -2.59 -2.59 0.07 1 1.31 1.31 0.79 1 -0.64 -0.64 0.35 0 -0.43 -0.43 0.39 0 -0.17 -0.17 0.46 0 0.61 0.61 0.65 0 0.68 0.68 0.66 1 0.57 0.57 0.64 0 -0.57 -0.57 0.36 1 -1.36 -1.36 0.20 0 -0.39 -0.39 0.40 1 0.28 0.28 0.57 1 -0.82 -0.82 0.31 0 -0.07 -0.07 0.48 0 -1.17 -1.17 0.24 0 -0.01 -0.01 0.50 1 0.13 0.13 0.53 0 -0.15 -0.15 0.46 1 -0.16 -0.16 0.46 0 1.76 1.76 0.85 1 0.76 0.76 0.68 0 1.11 1.11 0.75 1 -0.92 -0.92 0.28 0 0.16 0.16 0.54 0 1.15 1.15 0.76 1 -0.06 -0.06 0.49 1 -2.13 -2.13 0.11 0 0.34 0.34 0.59 1 -1.90 -1.90 0.13 0 -0.81 -0.81 0.31 0 1.32 1.32 0.79 1 0.62 0.62 0.65 1 1.09 1.09 0.75 1 0.31 0.31 0.58 1 -0.11 -0.11 0.47 0 -0.92 -0.92 0.28 0 1.59 1.59 0.83 1 0.05 0.05 0.51 1 -0.72 -0.72 0.33 0 0.87 0.87 0.70 1 1.07 1.07 0.75 1 1.90 1.90 0.87 1 -0.60 -0.60 0.35 1 -0.39 -0.39 0.40 1 -0.42 -0.42 0.40 0 -0.38 -0.38 0.41 0 -0.37 -0.37 0.41 0 -0.30 -0.30 0.43 1 1.44 1.44 0.81 1 -0.70 -0.70 0.33 0 -0.39 -0.39 0.40 1 0.65 0.65 0.66 0 1.12 1.12 0.75 1 -0.77 -0.77 0.32 0 -0.51 -0.51 0.38 0 0.52 0.52 0.63 1 1.02 1.02 0.73 0 -0.25 -0.25 0.44 1 -1.43 -1.43 0.19 1 1.71 1.71 0.85 1 1.44 1.44 0.81 1 -0.71 -0.71 0.33 1 -0.07 -0.07 0.48 1 -1.76 -1.76 0.15 0 0.57 0.57 0.64 1 1.61 1.61 0.83 1 -1.64 -1.64 0.16 0 -0.78 -0.78 0.31 0 -0.64 -0.64 0.34 1 -0.68 -0.68 0.34 0 -2.03 -2.03 0.12 0 0.50 0.50 0.62 0 -1.53 -1.53 0.18 1 -0.02 -0.02 0.49 0 0.59 0.59 0.64 1 -0.20 -0.20 0.45 1 0.89 0.89 0.71 0 -0.03 -0.03 0.49 1 -0.65 -0.65 0.34 0 0.65 0.65 0.66 0 -0.43 -0.43 0.39 0 1.77 1.77 0.85 1 -0.02 -0.02 0.50 1 0.85 0.85 0.70 1 0.21 0.21 0.55 0 -3.01 -3.01 0.05 0 -1.37 -1.37 0.20 0 -0.42 -0.42 0.40 1 0.24 0.24 0.56 0 -2.34 -2.34 0.09 0 0.96 0.96 0.72 1 -0.60 -0.60 0.35 1 -0.75 -0.75 0.32 0 -1.56 -1.56 0.17 1 -1.45 -1.45 0.19 0 0.06 0.06 0.51 1 0.51 0.51 0.62 0 -2.10 -2.10 0.11 0 -1.00 -1.00 0.27 0 0.54 0.54 0.63 1 -0.45 -0.45 0.39 0 2.17 2.17 0.90 1 1.25 1.25 0.78 1 0.60 0.60 0.64 1 0.00 0.00 0.50 0 0.28 0.28 0.57 1 -0.71 -0.71 0.33 0 0.63 0.63 0.65 0 1.48 1.48 0.81 1 1.08 1.08 0.75 1 -0.81 -0.81 0.31 1 -1.62 -1.62 0.17 0 -0.11 -0.11 0.47 1 0.44 0.44 0.61 1 1.35 1.35 0.79 1 -1.32 -1.32 0.21 0 0.36 0.36 0.59 1 0.23 0.23 0.56 1 1.19 1.19 0.77 1 -0.03 -0.03 0.49 0 -0.36 -0.36 0.41 0 -1.15 -1.15 0.24 0 -0.52 -0.52 0.37 1 -0.36 -0.36 0.41 0 2.35 2.35 0.91 1 2.45 2.45 0.92 1 -0.17 -0.17 0.46 0 -1.04 -1.04 0.26 0 -1.97 -1.97 0.12 0 0.51 0.51 0.63 0 -1.09 -1.09 0.25 0 2.28 2.28 0.91 1 -0.89 -0.89 0.29 1 0.11 0.11 0.53 1 3.81 3.81 0.98 1 -1.11 -1.11 0.25 1 0.31 0.31 0.58 1 -1.11 -1.11 0.25 1 0.35 0.35 0.59 1 -0.87 -0.87 0.29 0 0.08 0.08 0.52 0 -0.30 -0.30 0.43 0 -1.18 -1.18 0.23 1 0.01 0.01 0.50 1 0.99 0.99 0.73 0 1.59 1.59 0.83 1 -1.37 -1.37 0.20 1 -0.25 -0.25 0.44 0 1.16 1.16 0.76 1 -1.11 -1.11 0.25 0 -2.53 -2.53 0.07 0 -0.94 -0.94 0.28 0 -0.97 -0.97 0.28 1 0.05 0.05 0.51 0 -0.40 -0.40 0.40 0 0.23 0.23 0.56 1 -0.42 -0.42 0.40 1 0.37 0.37 0.59 0 -0.37 -0.37 0.41 0 1.19 1.19 0.77 0 -0.74 -0.74 0.32 0 0.29 0.29 0.57 1 -0.88 -0.88 0.29 0 0.21 0.21 0.55 1 -0.05 -0.05 0.49 0 -1.68 -1.68 0.16 0 -0.14 -0.14 0.46 0 1.18 1.18 0.76 0 0.68 0.68 0.66 1 0.14 0.14 0.54 0 -1.19 -1.19 0.23 0 1.17 1.17 0.76 1 0.08 0.08 0.52 0 -0.45 -0.45 0.39 0 1.64 1.64 0.84 1 -0.77 -0.77 0.32 1 0.30 0.30 0.58 1 1.28 1.28 0.78 0 0.60 0.60 0.65 0 -0.31 -0.31 0.42 0 -0.42 -0.42 0.40 1 0.36 0.36 0.59 1 0.51 0.51 0.63 0 0.02 0.02 0.50 1 1.32 1.32 0.79 0 -0.07 -0.07 0.48 1 -0.70 -0.70 0.33 0 0.54 0.54 0.63 1 -2.20 -2.20 0.10 1 0.39 0.39 0.60 1 0.50 0.50 0.62 0 -0.22 -0.22 0.44 0 -1.12 -1.12 0.25 0 -0.39 -0.39 0.40 1 1.55 1.55 0.82 1 -0.74 -0.74 0.32 1 -2.33 -2.33 0.09 0 0.81 0.81 0.69 1 -0.50 -0.50 0.38 0 -0.51 -0.51 0.37 0 -1.22 -1.22 0.23 0 -0.02 -0.02 0.49 0 0.70 0.70 0.67 0 -0.59 -0.59 0.36 0 -0.61 -0.61 0.35 1 1.10 1.10 0.75 0 -0.25 -0.25 0.44 1 -0.16 -0.16 0.46 0 -0.63 -0.63 0.35 0 0.90 0.90 0.71 1 -0.99 -0.99 0.27 0 0.85 0.85 0.70 1 0.81 0.81 0.69 1 -0.47 -0.47 0.39 0 0.85 0.85 0.70 0 0.99 0.99 0.73 0 0.58 0.58 0.64 0 2.02 2.02 0.88 0 -1.96 -1.96 0.12 0 -1.16 -1.16 0.24 0 -1.38 -1.38 0.20 1 0.17 0.17 0.54 0 1.58 1.58 0.83 1 1.68 1.68 0.84 1 0.49 0.49 0.62 0 0.88 0.88 0.71 1 -0.14 -0.14 0.46 1 0.47 0.47 0.62 0 0.38 0.38 0.59 0 -0.76 -0.76 0.32 0 -0.29 -0.29 0.43 0 -0.13 -0.13 0.47 0 1.39 1.39 0.80 1 -1.04 -1.04 0.26 0 -2.11 -2.11 0.11 0 0.77 0.77 0.68 0 -0.82 -0.82 0.31 0 -0.44 -0.44 0.39 0 0.90 0.90 0.71 0 -0.76 -0.76 0.32 0 -0.34 -0.34 0.42 1 1.50 1.50 0.82 1 0.53 0.53 0.63 0 0.54 0.54 0.63 1 -0.14 -0.14 0.47 0 -1.14 -1.14 0.24 0 -1.50 -1.50 0.18 0 -0.22 -0.22 0.44 0 2.00 2.00 0.88 0 0.22 0.22 0.56 1 0.16 0.16 0.54 0 0.33 0.33 0.58 0 -0.39 -0.39 0.40 0 -1.40 -1.40 0.20 0 2.68 2.68 0.94 1 -0.42 -0.42 0.40 0 -0.30 -0.30 0.43 0 -1.79 -1.79 0.14 0 -0.25 -0.25 0.44 1 -0.25 -0.25 0.44 0 -0.26 -0.26 0.44 0 -1.79 -1.79 0.14 0 1.78 1.78 0.86 0 1.76 1.76 0.85 1 0.69 0.69 0.67 1 -1.10 -1.10 0.25 0 0.71 0.71 0.67 1 -0.25 -0.25 0.44 0 -0.32 -0.32 0.42 0 1.36 1.36 0.80 1 -1.23 -1.23 0.23 0 -0.51 -0.51 0.37 1 -0.73 -0.73 0.32 0 0.02 0.02 0.50 1 -1.57 -1.57 0.17 1 -0.70 -0.70 0.33 0 0.72 0.72 0.67 1 0.47 0.47 0.61 1 -0.97 -0.97 0.27 1 0.56 0.56 0.64 0 -2.43 -2.43 0.08 0 -0.34 -0.34 0.42 0 0.71 0.71 0.67 0 -0.66 -0.66 0.34 0 -0.04 -0.04 0.49 0 -1.59 -1.59 0.17 1 0.85 0.85 0.70 1 -1.85 -1.85 0.14 0 -0.32 -0.32 0.42 1 -0.26 -0.26 0.44 0 0.06 0.06 0.52 1 -0.82 -0.82 0.31 0 1.83 1.83 0.86 0 -1.43 -1.43 0.19 0 0.25 0.25 0.56 1 -2.94 -2.94 0.05 0 0.00 0.00 0.50 1 0.51 0.51 0.62 1 -1.08 -1.08 0.25 0 0.70 0.70 0.67 1 0.33 0.33 0.58 1 0.98 0.98 0.73 1 -0.84 -0.84 0.30 0 -0.97 -0.97 0.27 0 -1.77 -1.77 0.15 0 -0.32 -0.32 0.42 1 -1.34 -1.34 0.21 0 0.69 0.69 0.67 0 0.07 0.07 0.52 1 2.19 2.19 0.90 0 -1.16 -1.16 0.24 0 1.18 1.18 0.77 0 -0.53 -0.53 0.37 0 -1.46 -1.46 0.19 0 0.57 0.57 0.64 0 -1.43 -1.43 0.19 0 -1.06 -1.06 0.26 0 -0.73 -0.73 0.32 0 0.21 0.21 0.55 1 -1.00 -1.00 0.27 0 1.08 1.08 0.75 1 -1.20 -1.20 0.23 0 0.22 0.22 0.55 1 0.14 0.14 0.54 1 -1.07 -1.07 0.26 0 -0.43 -0.43 0.39 1 -0.66 -0.66 0.34 1 0.96 0.96 0.72 1 1.56 1.56 0.83 0 -1.04 -1.04 0.26 0 0.93 0.93 0.72 0 -0.08 -0.08 0.48 0 -1.97 -1.97 0.12 0 -0.76 -0.76 0.32 0 0.46 0.46 0.61 1 0.15 0.15 0.54 0 -2.44 -2.44 0.08 0 0.58 0.58 0.64 1 0.66 0.66 0.66 1 -0.30 -0.30 0.42 1 -0.71 -0.71 0.33 0 1.97 1.97 0.88 1 -0.09 -0.09 0.48 1 -0.01 -0.01 0.50 1 -1.12 -1.12 0.25 0 -1.34 -1.34 0.21 0 -1.52 -1.52 0.18 0 -0.42 -0.42 0.40 1 1.36 1.36 0.80 1 1.75 1.75 0.85 1 1.57 1.57 0.83 1 1.30 1.30 0.79 1 -0.24 -0.24 0.44 0 -1.22 -1.22 0.23 0 -0.33 -0.33 0.42 0 -2.41 -2.41 0.08 0 -0.31 -0.31 0.42 1 1.66 1.66 0.84 0 0.13 0.13 0.53 1 1.10 1.10 0.75 1 0.49 0.49 0.62 1 -0.78 -0.78 0.31 1 1.74 1.74 0.85 1 -0.08 -0.08 0.48 0 -0.98 -0.98 0.27 0 0.07 0.07 0.52 0 -1.52 -1.52 0.18 0 0.86 0.86 0.70 0 0.50 0.50 0.62 0 -0.35 -0.35 0.41 1 -0.49 -0.49 0.38 0 0.94 0.94 0.72 1 -1.06 -1.06 0.26 0 -0.98 -0.98 0.27 0 0.42 0.42 0.60 1 -0.45 -0.45 0.39 0 0.93 0.93 0.72 1 -0.20 -0.20 0.45 1 1.19 1.19 0.77 1 0.50 0.50 0.62 1 -2.25 -2.25 0.10 0 -1.34 -1.34 0.21 1 1.28 1.28 0.78 1 0.69 0.69 0.67 1 -0.97 -0.97 0.28 1 -1.35 -1.35 0.21 0 1.03 1.03 0.74 0 -0.81 -0.81 0.31 0 1.80 1.80 0.86 1 1.77 1.77 0.85 1 -1.45 -1.45 0.19 0 -0.85 -0.85 0.30 1 -1.25 -1.25 0.22 1 0.67 0.67 0.66 1 -1.29 -1.29 0.22 0 -2.04 -2.04 0.12 0 2.02 2.02 0.88 1 1.01 1.01 0.73 1 0.82 0.82 0.69 1 -0.66 -0.66 0.34 0 -0.01 -0.01 0.50 0 0.62 0.62 0.65 1 -1.28 -1.28 0.22 0 -0.12 -0.12 0.47 0 0.18 0.18 0.54 1 1.69 1.69 0.84 0 0.64 0.64 0.66 1 1.28 1.28 0.78 1 0.14 0.14 0.54 0 -1.11 -1.11 0.25 0 -0.34 -0.34 0.42 0 -1.66 -1.66 0.16 0 0.93 0.93 0.72 1 1.42 1.42 0.80 1 -0.06 -0.06 0.48 0 -0.98 -0.98 0.27 0 1.09 1.09 0.75 0 0.14 0.14 0.53 1 -0.39 -0.39 0.40 0 1.12 1.12 0.75 0 -0.76 -0.76 0.32 0 1.15 1.15 0.76 1 -0.84 -0.84 0.30 0 0.39 0.39 0.60 0 0.89 0.89 0.71 1 -1.34 -1.34 0.21 0 0.40 0.40 0.60 0 -0.11 -0.11 0.47 0 0.68 0.68 0.66 1 -0.79 -0.79 0.31 0 -0.09 -0.09 0.48 1 1.38 1.38 0.80 0 0.17 0.17 0.54 1 0.82 0.82 0.69 1 -0.22 -0.22 0.44 1 -1.03 -1.03 0.26 0 -0.01 -0.01 0.50 1 -1.22 -1.22 0.23 1 -2.60 -2.60 0.07 0 1.17 1.17 0.76 1 -1.09 -1.09 0.25 1 -1.83 -1.83 0.14 0 1.00 1.00 0.73 1 -0.01 -0.01 0.50 0 -0.60 -0.60 0.35 0 -0.18 -0.18 0.46 1 -0.43 -0.43 0.40 0 1.00 1.00 0.73 1 0.73 0.73 0.67 1 -1.73 -1.73 0.15 0 0.35 0.35 0.59 1 0.73 0.73 0.67 1 0.67 0.67 0.66 1 -2.42 -2.42 0.08 1 -0.24 -0.24 0.44 0 1.98 1.98 0.88 1 0.80 0.80 0.69 1 -1.71 -1.71 0.15 0 -1.66 -1.66 0.16 0 0.49 0.49 0.62 1 -0.17 -0.17 0.46 1 0.96 0.96 0.72 1 0.29 0.29 0.57 1 0.08 0.08 0.52 1 0.18 0.18 0.55 1 0.17 0.17 0.54 0 -1.27 -1.27 0.22 0 2.35 2.35 0.91 0 -1.41 -1.41 0.20 0 -0.02 -0.02 0.50 0 -0.54 -0.54 0.37 0 1.80 1.80 0.86 1 1.01 1.01 0.73 1 -0.56 -0.56 0.36 0 0.21 0.21 0.55 0 1.17 1.17 0.76 1 2.24 2.24 0.90 1 0.30 0.30 0.57 0 -1.04 -1.04 0.26 1 -0.98 -0.98 0.27 0 2.01 2.01 0.88 1 -2.07 -2.07 0.11 0 3.06 3.06 0.96 1 -0.26 -0.26 0.44 0 -0.45 -0.45 0.39 1 0.16 0.16 0.54 1 0.93 0.93 0.72 0 0.30 0.30 0.58 0 -1.96 -1.96 0.12 1 0.35 0.35 0.59 0 0.45 0.45 0.61 1 0.66 0.66 0.66 0 -1.03 -1.03 0.26 0 -2.37 -2.37 0.09 0 -0.32 -0.32 0.42 0 -0.94 -0.94 0.28 0 -0.77 -0.77 0.32 0 -0.95 -0.95 0.28 0 -0.40 -0.40 0.40 1 -0.31 -0.31 0.42 0 0.80 0.80 0.69 1 0.99 0.99 0.73 1 -0.79 -0.79 0.31 1 -0.31 -0.31 0.42 0 0.36 0.36 0.59 0 1.40 1.40 0.80 1 -0.06 -0.06 0.49 1 -1.70 -1.70 0.15 0 0.23 0.23 0.56 0 -0.12 -0.12 0.47 0 1.77 1.77 0.85 1 0.34 0.34 0.59 0 -0.62 -0.62 0.35 1 -0.44 -0.44 0.39 0 -0.51 -0.51 0.38 0 0.19 0.19 0.55 1 0.18 0.18 0.54 0 0.92 0.92 0.71 1 0.32 0.32 0.58 0 -0.37 -0.37 0.41 0 -0.94 -0.94 0.28 0 0.63 0.63 0.65 0 -0.06 -0.06 0.48 0 0.18 0.18 0.55 0 1.10 1.10 0.75 1 1.75 1.75 0.85 1 -0.95 -0.95 0.28 0 1.64 1.64 0.84 1 -0.87 -0.87 0.30 0 0.27 0.27 0.57 1 0.22 0.22 0.56 1 -0.28 -0.28 0.43 1 1.39 1.39 0.80 0 -0.66 -0.66 0.34 0 0.66 0.66 0.66 1 -0.01 -0.01 0.50 1 -0.93 -0.93 0.28 0 1.21 1.21 0.77 0 -2.09 -2.09 0.11 0 -0.53 -0.53 0.37 1 -1.54 -1.54 0.18 0 0.19 0.19 0.55 1 0.26 0.26 0.57 1 -1.12 -1.12 0.25 0 0.65 0.65 0.66 1 -1.03 -1.03 0.26 0 0.66 0.66 0.66 1 0.24 0.24 0.56 0 0.72 0.72 0.67 1 -0.94 -0.94 0.28 0 0.10 0.10 0.52 1 -0.46 -0.46 0.39 0 -1.47 -1.47 0.19 0 0.15 0.15 0.54 1 1.77 1.77 0.85 0 -0.65 -0.65 0.34 1 -0.20 -0.20 0.45 1 0.69 0.69 0.67 1 0.04 0.04 0.51 0 1.94 1.94 0.87 1 0.74 0.74 0.68 1 2.32 2.32 0.91 1 0.35 0.35 0.59 0 -1.13 -1.13 0.24 1 0.42 0.42 0.60 0 -0.92 -0.92 0.28 1 -1.01 -1.01 0.27 0 -0.19 -0.19 0.45 0 0.93 0.93 0.72 1 0.34 0.34 0.59 1 0.81 0.81 0.69 0 0.92 0.92 0.71 0 -0.17 -0.17 0.46 0 -2.40 -2.40 0.08 0 0.80 0.80 0.69 0 2.17 2.17 0.90 1 0.06 0.06 0.51 1 -1.35 -1.35 0.21 0 -0.37 -0.37 0.41 0 -0.93 -0.93 0.28 0 -0.04 -0.04 0.49 0 0.68 0.68 0.66 0 0.87 0.87 0.70 1 0.24 0.24 0.56 0 -0.93 -0.93 0.28 0 0.81 0.81 0.69 1 1.35 1.35 0.79 1 2.25 2.25 0.90 1 -0.49 -0.49 0.38 1 0.47 0.47 0.62 0 1.19 1.19 0.77 1 -0.12 -0.12 0.47 1 0.52 0.52 0.63 0 0.21 0.21 0.55 1 -0.13 -0.13 0.47 0 0.17 0.17 0.54 0 0.96 0.96 0.72 1 0.41 0.41 0.60 1 -0.47 -0.47 0.39 0 -2.24 -2.24 0.10 0 -0.80 -0.80 0.31 0 -0.02 -0.02 0.50 1 -2.51 -2.51 0.07 0 2.21 2.21 0.90 1 -1.49 -1.49 0.18 0 -1.16 -1.16 0.24 1 1.46 1.46 0.81 1 -2.19 -2.19 0.10 0 0.74 0.74 0.68 0 -0.34 -0.34 0.41 0 0.46 0.46 0.61 0 0.88 0.88 0.71 1 -0.96 -0.96 0.28 1 -0.71 -0.71 0.33 1 -3.00 -3.00 0.05 0 -0.96 -0.96 0.28 1 0.38 0.38 0.59 1 0.51 0.51 0.62 1 2.03 2.03 0.88 1 0.06 0.06 0.52 1 0.46 0.46 0.61 0 0.07 0.07 0.52 1 -0.49 -0.49 0.38 0 0.75 0.75 0.68 0 0.46 0.46 0.61 1 0.13 0.13 0.53 1 -0.82 -0.82 0.31 1 -0.04 -0.04 0.49 1 0.78 0.78 0.69 0 0.68 0.68 0.66 0 -0.48 -0.48 0.38 0 -0.67 -0.67 0.34 0 0.51 0.51 0.63 1 1.05 1.05 0.74 1 0.12 0.12 0.53 0 -0.31 -0.31 0.42 0 -0.88 -0.88 0.29 0 -0.42 -0.42 0.40 0 -1.48 -1.48 0.19 0 -0.70 -0.70 0.33 0 Nice! The inferred estimates are very close to the parameter values we used to simulate the data. Let’s visualize the result: ggplot(data = df.data, mapping = aes(x = x, y = response)) + geom_smooth(method = &quot;glm&quot;, method.args = list(family = &quot;binomial&quot;)) + geom_point(alpha = 0.1) + labs(y = &quot;p(response)&quot;) 21.5.0.1 Calculate the model’s likelihood To calculate the likelihood of the data for a given logistic model, we look at the actual response, and the probability of the predicted response, and then determine the likelihood of the observation assuming a Bernoulli process. To get the overall likelihood of the data, we then multiply the likelihood of each data point (or take the logs first and then the sum to get the log-likelihood). This table illustrate the steps involved: fit %&gt;% augment() %&gt;% clean_names() %&gt;% mutate(p = inv.logit(fitted)) %&gt;% select(response, p) %&gt;% mutate(p_response = ifelse(response == 1, p, 1-p), log_p = log(p_response)) %&gt;% rename(`p(Y = 1)` = p, `p(Y = response)` = p_response, `log(p(Y = response))` = log_p) # A tibble: 1,000 × 4 response `p(Y = 1)` `p(Y = response)` `log(p(Y = response))` &lt;int&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 1 0.344 0.344 -1.07 2 0 0.527 0.473 -0.749 3 1 0.302 0.302 -1.20 4 1 0.805 0.805 -0.217 5 1 0.561 0.561 -0.579 6 0 0.305 0.695 -0.364 7 1 0.596 0.596 -0.517 8 1 0.651 0.651 -0.429 9 1 0.616 0.616 -0.484 10 0 0.414 0.586 -0.535 # … with 990 more rows Let’s calculate the log-likelihood by hand: fit %&gt;% augment() %&gt;% clean_names() %&gt;% mutate(p = inv.logit(fitted), log_likelihood = response * log(p) + (1 - response) * log(1 - p)) %&gt;% summarize(log_likelihood = sum(log_likelihood)) # A tibble: 1 × 1 log_likelihood &lt;dbl&gt; 1 -605. And compare it with the model summary fit %&gt;% glance() %&gt;% select(logLik, AIC, BIC) # A tibble: 1 × 3 logLik AIC BIC &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 -605. 1214. 1223. We’re getting the same result – neat! Now we know how the likelihood of the data is calculated for a logistic regression model. 21.6 Testing hypotheses To test hypotheses, we can use our good old model comparison approach: # fit compact model fit.compact = glm(formula = survived ~ 1 + fare, family = &quot;binomial&quot;, data = df.titanic) # fit augmented model fit.augmented = glm(formula = survived ~ 1 + sex + fare, family = &quot;binomial&quot;, data = df.titanic) # likelihood ratio test anova(fit.compact, fit.augmented, test = &quot;LRT&quot;) Analysis of Deviance Table Model 1: survived ~ 1 + fare Model 2: survived ~ 1 + sex + fare Resid. Df Resid. Dev Df Deviance Pr(&gt;Chi) 1 889 1117.57 2 888 884.31 1 233.26 &lt; 2.2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Note that in order to get a p-value out of this, we need to specify what statistical test we’d like to run. In this case, we use the likelihood ratio test (“LRT”). We can also test for both effects of survived and fare in one go using the joint_tests() function from the “emmeans” package like so: glm(formula = survived ~ 1 + sex + fare, family = &quot;binomial&quot;, data = df.titanic) %&gt;% joint_tests() model term df1 df2 F.ratio p.value sex 1 Inf 201.881 &lt;.0001 fare 1 Inf 23.869 &lt;.0001 Notice that the F.ratio reported using joint_tests() (201.881) is not quite the same as the deviance value that we get through the likelihood ratio test (233.26). You can read more about why these two can come apart here. 21.7 Logistic mixed effects model Just like we can build linear mixed effects models using lmer() instead of lm(), we can also build a logistic mixed effects regression using glmer() instead of glm(). Let’s read in some data: # load bdf data set from nlme package data(bdf, package = &quot;nlme&quot;) df.language = bdf %&gt;% clean_names() %&gt;% filter(repeatgr != 2) %&gt;% mutate(repeatgr = repeatgr %&gt;% as.character() %&gt;% as.numeric()) rm(bdf) Fit the model, and print out the results: fit = glmer(repeatgr ~ 1 + ses + minority + (1 | school_nr), data = df.language, family = &quot;binomial&quot;) fit %&gt;% summary() Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) [glmerMod] Family: binomial ( logit ) Formula: repeatgr ~ 1 + ses + minority + (1 | school_nr) Data: df.language AIC BIC logLik deviance df.resid 1659.1 1682.1 -825.6 1651.1 2279 Scaled residuals: Min 1Q Median 3Q Max -0.9235 -0.4045 -0.3150 -0.2249 5.8372 Random effects: Groups Name Variance Std.Dev. school_nr (Intercept) 0.2489 0.4989 Number of obs: 2283, groups: school_nr, 131 Fixed effects: Estimate Std. Error z value Pr(&gt;|z|) (Intercept) -0.506291 0.197566 -2.563 0.01039 * ses -0.060086 0.007523 -7.986 1.39e-15 *** minorityY 0.673612 0.238657 2.823 0.00476 ** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Correlation of Fixed Effects: (Intr) ses ses -0.898 minorityY -0.308 0.208 To visualize the results, we can use the ggeffects package. ggpredict(model = fit, terms = c(&quot;ses [all]&quot;, &quot;minority&quot;)) %&gt;% plot() And for significance testing, we can use the the joint_tests() function from the “emmeans” package glmer(formula = repeatgr ~ 1 + ses + minority + (1 | school_nr), data = df.language, family = &quot;binomial&quot;) %&gt;% joint_tests() model term df1 df2 F.ratio p.value ses 1 Inf 63.784 &lt;.0001 minority 1 Inf 7.967 0.0048 The results show that there was both a significant effect of ses and of minority. Note: This post here says a little more about the relationship of the F.ratio in the joint_tests() function, and what a likelihood ratio test yields. In short, it’s roughly the same thing. If you’d like to compute the likelihood ratio test, a convenient way of doing so is by using the mixed() function from the “afex” package. mixed(formula = repeatgr ~ 1 + ses + minority + (1 | school_nr), family = &quot;binomial&quot;, data = df.language, method = &quot;LRT&quot;) Contrasts set to contr.sum for the following variables: minority, school_nr Numerical variables NOT centered on 0: ses If in interactions, interpretation of lower order (e.g., main) effects difficult. Mixed Model Anova Table (Type 3 tests, LRT-method) Model: repeatgr ~ 1 + ses + minority + (1 | school_nr) Data: df.language Df full model: 4 Effect df Chisq p.value 1 ses 1 75.39 *** &lt;.001 2 minority 1 7.53 ** .006 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;+&#39; 0.1 &#39; &#39; 1 And we can compare that the model comparison approach gives us the same result: fit_a = glmer(repeatgr ~ 1 + ses + minority + (1 | school_nr), data = df.language, family = &quot;binomial&quot;) # dropping ses as a predictor fit_c = glmer(repeatgr ~ 1 + minority + (1 | school_nr), data = df.language, family = &quot;binomial&quot;) anova(fit_a, fit_c, test = &quot;LRT&quot;) Data: df.language Models: fit_c: repeatgr ~ 1 + minority + (1 | school_nr) fit_a: repeatgr ~ 1 + ses + minority + (1 | school_nr) npar AIC BIC logLik deviance Chisq Df Pr(&gt;Chisq) fit_c 3 1732.5 1749.7 -863.27 1726.5 fit_a 4 1659.1 1682.1 -825.57 1651.1 75.395 1 &lt; 2.2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 21.8 Additional information 21.8.1 Datacamp Multiple and logistic regression Generalized linear models in R Categorical data in the tidyverse 21.9 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 ggplot2_3.3.6 [9] tidyverse_1.3.1 emmeans_1.7.3 afex_1.1-1 ggeffects_1.1.2 [13] boot_1.3-28 lme4_1.1-29 Matrix_1.4-1 broom_0.8.0 [17] janitor_2.1.0 kableExtra_1.3.4 titanic_0.1.0 knitr_1.39 loaded via a namespace (and not attached): [1] TH.data_1.1-1 minqa_1.2.4 colorspace_2.0-3 [4] ellipsis_0.3.2 sjlabelled_1.2.0 estimability_1.3 [7] snakecase_0.11.0 fs_1.5.2 rstudioapi_0.13 [10] farver_2.1.0 fansi_1.0.3 mvtnorm_1.1-3 [13] lubridate_1.8.0 xml2_1.3.3 codetools_0.2-18 [16] splines_4.1.2 effects_4.2-1 jsonlite_1.8.0 [19] nloptr_2.0.0 dbplyr_2.1.1 compiler_4.1.2 [22] httr_1.4.3 backports_1.4.1 assertthat_0.2.1 [25] fastmap_1.1.0 survey_4.1-1 cli_3.3.0 [28] htmltools_0.5.2 tools_4.1.2 lmerTest_3.1-3 [31] coda_0.19-4 gtable_0.3.0 glue_1.6.2 [34] reshape2_1.4.4 Rcpp_1.0.8.3 carData_3.0-5 [37] cellranger_1.1.0 jquerylib_0.1.4 vctrs_0.4.1 [40] svglite_2.1.0 nlme_3.1-157 insight_0.17.0 [43] xfun_0.30 rvest_1.0.2 lifecycle_1.0.1 [46] MASS_7.3-57 zoo_1.8-10 scales_1.2.0 [49] hms_1.1.1 parallel_4.1.2 sandwich_3.0-1 [52] RColorBrewer_1.1-3 yaml_2.3.5 sass_0.4.1 [55] stringi_1.7.6 highr_0.9 rlang_1.0.2 [58] pkgconfig_2.0.3 systemfonts_1.0.4 evaluate_0.15 [61] lattice_0.20-45 labeling_0.4.2 tidyselect_1.1.2 [64] plyr_1.8.7 magrittr_2.0.3 bookdown_0.26 [67] R6_2.5.1 generics_0.1.2 multcomp_1.4-19 [70] DBI_1.1.2 pillar_1.7.0 haven_2.5.0 [73] withr_2.5.0 mgcv_1.8-40 survival_3.3-1 [76] abind_1.4-5 nnet_7.3-17 modelr_0.1.8 [79] crayon_1.5.1 car_3.0-13 utf8_1.2.2 [82] tzdb_0.3.0 rmarkdown_2.14 grid_4.1.2 [85] readxl_1.4.0 reprex_2.0.1 digest_0.6.29 [88] webshot_0.5.3 xtable_1.8-4 numDeriv_2016.8-1.1 [91] munsell_0.5.0 viridisLite_0.4.0 bslib_0.3.1 [94] mitools_2.4 "],["bayesian-data-analysis-1.html", "Chapter 22 Bayesian data analysis 1 22.1 Learning goals 22.2 Load packages and set plotting theme 22.3 Doing Bayesian inference “by hand” 22.4 Doing Bayesian inference with Greta 22.5 Session info", " Chapter 22 Bayesian data analysis 1 22.1 Learning goals Doing Bayesian inference “by hand” Understanding the effect that prior, likelihood, and sample size have on the posterior. Doing Bayesian data analysis with greta A simple linear regression. 22.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;janitor&quot;) # for cleaning column names library(&quot;patchwork&quot;) # for figure panels library(&quot;tidybayes&quot;) # tidying up results from Bayesian models library(&quot;greta&quot;) # for writing Bayesian models library(&quot;gganimate&quot;) # for animations library(&quot;extraDistr&quot;) # additional probability distributions library(&quot;broom&quot;) # for tidy regression results library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 22.3 Doing Bayesian inference “by hand” 22.3.1 Sequential updating based on the Beta distribution # data data = c(0, 1, 1, 0, 1, 1, 1, 1) # whether observation is a success or failure success = c(0, cumsum(data)) failure = c(0, cumsum(1 - data)) # I&#39;ve added 0 at the beginning to show the prior # plotting function fun.plot_beta = function(success, failure){ ggplot(data = tibble(x = c(0, 1)), mapping = aes(x = x)) + stat_function(fun = &quot;dbeta&quot;, args = list(shape1 = success + 1, shape2 = failure + 1), geom = &quot;area&quot;, color = &quot;black&quot;, fill = &quot;lightblue&quot;) + coord_cartesian(expand = F) + scale_x_continuous(breaks = seq(0.25, 0.75, 0.25)) + theme(axis.title = element_blank(), axis.text.y = element_blank(), axis.ticks.y = element_blank(), plot.margin = margin(r = 1, t = 0.5, unit = &quot;cm&quot;)) } # generate the plots plots = map2(success, failure, ~ fun.plot_beta(.x, .y)) # make a grid of plots wrap_plots(plots, ncol = 3) 22.3.2 Coin flip example Is the coin biased? # data data = rep(0:1, c(8, 2)) # parameters theta = c(0.1, 0.5, 0.9) # prior prior = c(0.25, 0.5, 0.25) # prior = c(0.1, 0.1, 0.8) # alternative setting of the prior # prior = c(0.000001, 0.000001, 0.999998) # another prior setting # likelihood likelihood = dbinom(sum(data == 1), size = length(data), prob = theta) # posterior posterior = likelihood * prior / sum(likelihood * prior) # store in data frame df.coins = tibble(theta = theta, prior = prior, likelihood = likelihood, posterior = posterior) Visualize the results: df.coins %&gt;% pivot_longer(cols = -theta, names_to = &quot;index&quot;, values_to = &quot;value&quot;) %&gt;% mutate(index = factor(index, levels = c(&quot;prior&quot;, &quot;likelihood&quot;, &quot;posterior&quot;)), theta = factor(theta, labels = c(&quot;p = 0.1&quot;, &quot;p = 0.5&quot;, &quot;p = 0.9&quot;))) %&gt;% ggplot(data = ., mapping = aes(x = theta, y = value, fill = index)) + geom_bar(stat = &quot;identity&quot;, color = &quot;black&quot;) + facet_grid(rows = vars(index), switch = &quot;y&quot;, scales = &quot;free&quot;) + annotate(&quot;segment&quot;, x = -Inf, xend = Inf, y = -Inf, yend = -Inf) + annotate(&quot;segment&quot;, x = -Inf, xend = -Inf, y = -Inf, yend = Inf) + theme(legend.position = &quot;none&quot;, strip.background = element_blank(), axis.title.y = element_blank(), axis.text.y = element_blank(), axis.ticks.y = element_blank(), axis.title.x = element_blank(), axis.line = element_blank()) 22.3.3 Bayesian inference by discretization 22.3.3.1 Effect of the prior # grid theta = seq(0, 1, 0.01) # data data = rep(0:1, c(8, 2)) # calculate posterior df.prior_effect = tibble(theta = theta, prior_uniform = dbeta(theta, shape1 = 1, shape2 = 1), prior_normal = dbeta(theta, shape1 = 5, shape2 = 5), prior_biased = dbeta(theta, shape1 = 8, shape2 = 2)) %&gt;% pivot_longer(cols = -theta, names_to = &quot;prior_index&quot;, values_to = &quot;prior&quot;) %&gt;% mutate(likelihood = dbinom(sum(data == 1), size = length(data), prob = theta)) %&gt;% group_by(prior_index) %&gt;% mutate(posterior = likelihood * prior / sum(likelihood * prior)) %&gt;% ungroup() %&gt;% pivot_longer(cols = -c(theta, prior_index), names_to = &quot;index&quot;, values_to = &quot;value&quot;) # make the plot df.prior_effect %&gt;% mutate(index = factor(index, levels = c(&quot;prior&quot;, &quot;likelihood&quot;, &quot;posterior&quot;)), prior_index = factor(prior_index, levels = c(&quot;prior_uniform&quot;, &quot;prior_normal&quot;, &quot;prior_biased&quot;), labels = c(&quot;uniform&quot;, &quot;symmetric&quot;, &quot;asymmetric&quot;))) %&gt;% ggplot(data = ., mapping = aes(x = theta, y = value, color = index)) + geom_line(size = 1) + facet_grid(cols = vars(prior_index), rows = vars(index), scales = &quot;free&quot;, switch = &quot;y&quot;) + scale_x_continuous(breaks = seq(0, 1, 0.2)) + annotate(&quot;segment&quot;, x = -Inf, xend = Inf, y = -Inf, yend = -Inf) + annotate(&quot;segment&quot;, x = -Inf, xend = -Inf, y = -Inf, yend = Inf) + theme(legend.position = &quot;none&quot;, strip.background = element_blank(), axis.title.y = element_blank(), axis.text.y = element_blank(), axis.ticks.y = element_blank(), axis.text.x = element_text(size = 10), axis.line = element_blank()) Figure 2.6: Illustration of how the prior affects the posterior. 22.3.3.2 Effect of the likelihood # grid theta = seq(0, 1, 0.01) df.likelihood_effect = tibble(theta = theta, prior = dbeta(theta, shape1 = 2, shape2 = 8), likelihood_left = dbeta(theta, shape1 = 1, shape2 = 9), likelihood_center = dbeta(theta, shape1 = 5, shape2 = 5), likelihood_right = dbeta(theta, shape1 = 9, shape2 = 1)) %&gt;% pivot_longer(cols = -c(theta, prior), names_to = &quot;likelihood_index&quot;, values_to = &quot;likelihood&quot;) %&gt;% group_by(likelihood_index) %&gt;% mutate(posterior = likelihood * prior / sum(likelihood * prior)) %&gt;% ungroup() %&gt;% pivot_longer(cols = -c(theta, likelihood_index), names_to = &quot;index&quot;, values_to = &quot;value&quot;) df.likelihood_effect %&gt;% mutate(index = factor(index, levels = c(&quot;prior&quot;, &quot;likelihood&quot;, &quot;posterior&quot;)), likelihood_index = factor(likelihood_index, levels = c(&quot;likelihood_left&quot;, &quot;likelihood_center&quot;, &quot;likelihood_right&quot;), labels = c(&quot;left&quot;, &quot;center&quot;, &quot;right&quot;))) %&gt;% ggplot(data = ., mapping = aes(x = theta, y = value, color = index)) + geom_line(size = 1) + facet_grid(cols = vars(likelihood_index), rows = vars(index), scales = &quot;free&quot;, switch = &quot;y&quot;) + scale_x_continuous(breaks = seq(0, 1, 0.2)) + annotate(&quot;segment&quot;, x = -Inf, xend = Inf, y = -Inf, yend = -Inf) + annotate(&quot;segment&quot;, x = -Inf, xend = -Inf, y = -Inf, yend = Inf) + theme(legend.position = &quot;none&quot;, strip.background = element_blank(), axis.title.y = element_blank(), axis.text.y = element_blank(), axis.ticks.y = element_blank(), axis.text.x = element_text(size = 10), axis.line = element_blank(), strip.text.x = element_blank()) Figure 2.7: Illustration of how the likelihood of the data affects the posterior. 22.3.3.3 Effect of the sample size # grid theta = seq(0, 1, 0.01) df.sample_size_effect = tibble(theta = theta, prior = dbeta(theta, shape1 = 5, shape2 = 5), likelihood_low = dbeta(theta, shape1 = 2, shape2 = 8), likelihood_medium = dbeta(theta, shape1 = 10, shape2 = 40), likelihood_high = dbeta(theta, shape1 = 20, shape2 = 80)) %&gt;% pivot_longer(cols = -c(theta, prior), names_to = &quot;likelihood_index&quot;, values_to = &quot;likelihood&quot;) %&gt;% group_by(likelihood_index) %&gt;% mutate(posterior = likelihood * prior / sum(likelihood * prior)) %&gt;% ungroup() %&gt;% pivot_longer(cols = -c(theta, likelihood_index), names_to = &quot;index&quot;, values_to = &quot;value&quot;) df.sample_size_effect %&gt;% mutate(index = factor(index, levels = c(&quot;prior&quot;, &quot;likelihood&quot;, &quot;posterior&quot;)), likelihood_index = factor(likelihood_index, levels = c(&quot;likelihood_low&quot;, &quot;likelihood_medium&quot;, &quot;likelihood_high&quot;), labels = c(&quot;n = low&quot;, &quot;n = medium&quot;, &quot;n = high&quot;))) %&gt;% ggplot(data = ., mapping = aes(x = theta, y = value, color = index)) + geom_line(size = 1) + facet_grid(cols = vars(likelihood_index), rows = vars(index), scales = &quot;free&quot;, switch = &quot;y&quot;) + scale_x_continuous(breaks = seq(0, 1, 0.2)) + annotate(&quot;segment&quot;, x = -Inf, xend = Inf, y = -Inf, yend = -Inf) + annotate(&quot;segment&quot;, x = -Inf, xend = -Inf, y = -Inf, yend = Inf) + theme(legend.position = &quot;none&quot;, strip.background = element_blank(), axis.title.y = element_blank(), axis.text.y = element_blank(), axis.ticks.y = element_blank(), axis.text.x = element_text(size = 10), axis.line = element_blank()) 22.4 Doing Bayesian inference with Greta You can find out more about how get started with “greta” here: https://greta-stats.org/articles/get_started.html. Make sure to install the development version of “greta” (as shown in the “install-packages” code chunk above: devtools::install_github(\"greta-dev/greta\")). 22.4.1 Attitude data set # load the attitude data set df.attitude = attitude Visualize relationship between how well complaints are handled and the overall rating of an employee ggplot(data = df.attitude, mapping = aes(x = complaints, y = rating)) + geom_point() 22.4.2 Frequentist analysis # fit model fit.lm = lm(formula = rating ~ 1 + complaints, data = df.attitude) # print summary fit.lm %&gt;% summary() Call: lm(formula = rating ~ 1 + complaints, data = df.attitude) Residuals: Min 1Q Median 3Q Max -12.8799 -5.9905 0.1783 6.2978 9.6294 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 14.37632 6.61999 2.172 0.0385 * complaints 0.75461 0.09753 7.737 1.99e-08 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 6.993 on 28 degrees of freedom Multiple R-squared: 0.6813, Adjusted R-squared: 0.6699 F-statistic: 59.86 on 1 and 28 DF, p-value: 1.988e-08 Visualize the model’s predictions ggplot(data = df.attitude, mapping = aes(x = complaints, y = rating)) + geom_smooth(method = &quot;lm&quot;, formula = &quot;y ~ x&quot;, color = &quot;black&quot;) + geom_point() 22.4.3 Bayesian regression 22.4.3.1 Fit the model set.seed(1) # variables &amp; priors b0 = normal(0, 10) ℹ Initialising python and checking dependencies, this may take a moment.[K ✔ Initialising python and checking dependencies ... done![K b1 = normal(0, 10) sd = cauchy(0, 3, truncation = c(0, Inf)) # linear predictor mu = b0 + b1 * df.attitude$complaints # observation model (likelihood) distribution(df.attitude$rating) = normal(mu, sd) # define the model m = model(b0, b1, sd) Visualize the model as graph: # plotting plot(m) Draw samples from the posterior distribution: set.seed(1) # sampling draws = mcmc(m, n_samples = 1000) # tidy up the draws df.draws = tidy_draws(draws) %&gt;% clean_names() 22.4.3.2 Visualize the priors These are the priors I used for the intercept, regression weights, and the standard deviation of the Gaussian likelihood function: # Gaussian ggplot(tibble(x = c(-30, 30)), aes(x = x)) + stat_function(fun = &quot;dnorm&quot;, size = 2, args = list(sd = 10)) # Cauchy ggplot(tibble(x = c(0, 30)), aes(x = x)) + stat_function(fun = &quot;dcauchy&quot;, size = 2, args = list(location = 0, scale = 3)) 22.4.3.3 Visualize the posteriors This is what the posterior looks like for the three parameters in the model: df.draws %&gt;% select(draw:sd) %&gt;% pivot_longer(cols = -draw, names_to = &quot;index&quot;, values_to = &quot;value&quot;) %&gt;% ggplot(data = ., mapping = aes(x = value)) + stat_density(geom = &quot;line&quot;) + facet_grid(rows = vars(index), scales = &quot;free_y&quot;, switch = &quot;y&quot;) + annotate(&quot;segment&quot;, x = -Inf, xend = Inf, y = -Inf, yend = -Inf) + annotate(&quot;segment&quot;, x = -Inf, xend = -Inf, y = -Inf, yend = Inf) + theme(legend.position = &quot;none&quot;, strip.background = element_blank(), axis.title.y = element_blank(), axis.text.y = element_blank(), axis.ticks.y = element_blank(), axis.text.x = element_text(size = 10), axis.line = element_blank(), strip.text.x = element_blank()) #### Credible interval vs. confidence interval fit.lm %&gt;% tidy(conf.int = T) %&gt;% ggplot(mapping = aes(y = term, x = estimate, xmin = conf.low, xmax = conf.high)) + geom_pointrange() 22.4.3.4 Visualize model predictions Let’s take some samples from the posterior to visualize the model predictions: ggplot(data = df.attitude, mapping = aes(x = complaints, y = rating)) + geom_abline(data = df.draws %&gt;% slice_sample(n = 50), mapping = aes(intercept = b0, slope = b1), alpha = 0.3, color = &quot;lightblue&quot;) + geom_point() 22.4.3.5 Posterior predictive check Let’s make an animation that illustrates what predicted data sets (based on samples from the posterior) would look like: p = df.draws %&gt;% slice_sample(n = 10) %&gt;% mutate(complaints = list(seq(min(df.attitude$complaints), max(df.attitude$complaints), length.out = nrow(df.attitude)))) %&gt;% unnest(c(complaints)) %&gt;% mutate(prediction = b0 + b1 * complaints + rnorm(n(), sd = sd)) %&gt;% ggplot(aes(x = complaints, y = prediction)) + geom_point(alpha = 0.8, color = &quot;lightblue&quot;) + geom_point(data = df.attitude, aes(y = rating, x = complaints)) + coord_cartesian(xlim = c(20, 100), ylim = c(20, 100)) + transition_manual(draw) animate(p, nframes = 60, width = 800, height = 600, res = 96, type = &quot;cairo&quot;) # anim_save(&quot;posterior_predictive.gif&quot;) 22.4.3.6 Prior predictive check And let’s illustrate what data we would have expected to see just based on the information that we encoded in our priors. sample_size = 10 p = tibble(b0 = rnorm(sample_size, mean = 0, sd = 10), b1 = rnorm(sample_size, mean = 0, sd = 10), sd = rhcauchy(sample_size, sigma = 3), draw = 1:sample_size) %&gt;% mutate(complaints = list(runif(nrow(df.attitude), min = min(df.attitude$complaints), max = max(df.attitude$complaints)))) %&gt;% unnest(c(complaints)) %&gt;% mutate(prediction = b0 + b1 * complaints + rnorm(n(), sd = sd)) %&gt;% ggplot(aes(x = complaints, y = prediction)) + geom_point(alpha = 0.8, color = &quot;lightblue&quot;) + geom_point(data = df.attitude, aes(y = rating, x = complaints)) + transition_manual(draw) animate(p, nframes = 60, width = 800, height = 600, res = 96, type = &quot;cairo&quot;) # anim_save(&quot;prior_predictive.gif&quot;) 22.5 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 tidyverse_1.3.1 [9] broom_0.8.0 extraDistr_1.9.1 gganimate_1.0.7 ggplot2_3.3.6 [13] greta_0.4.2 tidybayes_3.0.2 patchwork_1.1.1 janitor_2.1.0 [17] knitr_1.39 loaded via a namespace (and not attached): [1] nlme_3.1-157 fs_1.5.2 lubridate_1.8.0 [4] RColorBrewer_1.1-3 progress_1.2.2 httr_1.4.3 [7] rprojroot_2.0.3 tensorA_0.36.2 DiagrammeRsvg_0.1 [10] tools_4.1.2 backports_1.4.1 bslib_0.3.1 [13] utf8_1.2.2 R6_2.5.1 mgcv_1.8-40 [16] DBI_1.1.2 colorspace_2.0-3 ggdist_3.1.1 [19] withr_2.5.0 tidyselect_1.1.2 prettyunits_1.1.1 [22] processx_3.5.3 curl_4.3.2 compiler_4.1.2 [25] rvest_1.0.2 cli_3.3.0 arrayhelpers_1.1-0 [28] xml2_1.3.3 labeling_0.4.2 bookdown_0.26 [31] posterior_1.2.1 sass_0.4.1 scales_1.2.0 [34] checkmate_2.1.0 callr_3.7.0 tfruns_1.5.0 [37] digest_0.6.29 rmarkdown_2.14 base64enc_0.1-3 [40] pkgconfig_2.0.3 htmltools_0.5.2 parallelly_1.31.1 [43] highr_0.9 dbplyr_2.1.1 fastmap_1.1.0 [46] htmlwidgets_1.5.4 rlang_1.0.2 readxl_1.4.0 [49] rstudioapi_0.13 visNetwork_2.1.0 jquerylib_0.1.4 [52] farver_2.1.0 generics_0.1.2 svUnit_1.0.6 [55] jsonlite_1.8.0 tensorflow_2.8.0 distributional_0.3.0 [58] magrittr_2.0.3 Matrix_1.4-1 Rcpp_1.0.8.3 [61] munsell_0.5.0 fansi_1.0.3 abind_1.4-5 [64] reticulate_1.24 lifecycle_1.0.1 stringi_1.7.6 [67] whisker_0.4 yaml_2.3.5 snakecase_0.11.0 [70] plyr_1.8.7 grid_4.1.2 parallel_4.1.2 [73] listenv_0.8.0 crayon_1.5.1 lattice_0.20-45 [76] splines_4.1.2 haven_2.5.0 hms_1.1.1 [79] ps_1.7.0 pillar_1.7.0 igraph_1.3.1 [82] codetools_0.2-18 reprex_2.0.1 glue_1.6.2 [85] evaluate_0.15 V8_4.1.0 gifski_1.6.6-1 [88] modelr_0.1.8 png_0.1-7 vctrs_0.4.1 [91] tzdb_0.3.0 tweenr_1.0.2 cellranger_1.1.0 [94] gtable_0.3.0 future_1.25.0 assertthat_0.2.1 [97] xfun_0.30 coda_0.19-4 DiagrammeR_1.0.9 [100] globals_0.14.0 ellipsis_0.3.2 here_1.0.1 "],["bayesian-data-analysis-2.html", "Chapter 23 Bayesian data analysis 2 23.1 Learning goals 23.2 Load packages and set plotting theme 23.3 Load data sets 23.4 Poker 23.5 Sleep study 23.6 Titanic study 23.7 Politeness data 23.8 Additional resources 23.9 Session info", " Chapter 23 Bayesian data analysis 2 23.1 Learning goals Building Bayesian models with brms. Model evaluation: Visualizing and interpreting results. Testing hypotheses. Inference evaluation: Did things work out? 23.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;tidybayes&quot;) # tidying up results from Bayesian models library(&quot;brms&quot;) # Bayesian regression models with Stan library(&quot;patchwork&quot;) # for making figure panels library(&quot;GGally&quot;) # for pairs plot library(&quot;broom.mixed&quot;) # for tidy lmer results library(&quot;bayesplot&quot;) # for visualization of Bayesian model fits library(&quot;modelr&quot;) # for modeling functions library(&quot;lme4&quot;) # for linear mixed effects models library(&quot;afex&quot;) # for ANOVAs library(&quot;car&quot;) # for ANOVAs library(&quot;emmeans&quot;) # for linear contrasts library(&quot;ggeffects&quot;) # for help with logistic regressions library(&quot;titanic&quot;) # titanic dataset library(&quot;gganimate&quot;) # for animations library(&quot;parameters&quot;) # for getting parameters library(&quot;transformr&quot;) # for gganimate # install via: devtools::install_github(&quot;thomasp85/transformr&quot;) library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + # set the theme theme(text = element_text(size = 20))) # set the default text size opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) options(dplyr.summarise.inform = F) # set default color scheme in ggplot options(ggplot2.discrete.color = RColorBrewer::brewer.pal(9,&quot;Set1&quot;)) 23.3 Load data sets # poker df.poker = read_csv(&quot;data/poker.csv&quot;) %&gt;% mutate(skill = factor(skill, levels = 1:2, labels = c(&quot;expert&quot;, &quot;average&quot;)), skill = fct_relevel(skill, &quot;average&quot;, &quot;expert&quot;), hand = factor(hand, levels = 1:3, labels = c(&quot;bad&quot;, &quot;neutral&quot;, &quot;good&quot;)), limit = factor(limit, levels = 1:2, labels = c(&quot;fixed&quot;, &quot;none&quot;)), participant = 1:n()) %&gt;% select(participant, everything()) # sleep df.sleep = sleepstudy %&gt;% as_tibble() %&gt;% clean_names() %&gt;% mutate(subject = as.character(subject)) %&gt;% select(subject, days, reaction) %&gt;% bind_rows(tibble(subject = &quot;374&quot;, days = 0:1, reaction = c(286, 288)), tibble(subject = &quot;373&quot;, days = 0, reaction = 245)) # titanic df.titanic = titanic_train %&gt;% clean_names() %&gt;% mutate(sex = as.factor(sex)) # politeness df.politeness = read_csv(&quot;data/politeness_data.csv&quot;) %&gt;% rename(pitch = frequency) 23.4 Poker 23.4.1 1. Visualize the data Let’s visualize the data first. set.seed(1) df.poker %&gt;% ggplot(mapping = aes(x = hand, y = balance, fill = hand, group = skill, shape = skill)) + geom_point(alpha = 0.2, position = position_jitterdodge(dodge.width = 0.5, jitter.height = 0, jitter.width = 0.2)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, position = position_dodge(width = 0.5), size = 1) + labs(y = &quot;final balance (in Euros)&quot;) + scale_shape_manual(values = c(21, 22)) + guides(fill = guide_legend(override.aes = list(shape = 21, fill = RColorBrewer::brewer.pal(3, &quot;Set1&quot;))), shape = guide_legend(override.aes = list(alpha = 1, fill = &quot;black&quot;))) 23.4.2 2. Specify and fit the model 23.4.2.1 Frequentist model And let’s now fit a simple (frequentist) ANOVA model. You have multiple options to do so: # Option 1: Using the &quot;afex&quot; package aov_ez(id = &quot;participant&quot;, dv = &quot;balance&quot;, between = c(&quot;hand&quot;, &quot;skill&quot;), data = df.poker) Contrasts set to contr.sum for the following variables: hand, skill Anova Table (Type 3 tests) Response: balance Effect df MSE F ges p.value 1 hand 2, 294 16.16 79.17 *** .350 &lt;.001 2 skill 1, 294 16.16 2.43 .008 .120 3 hand:skill 2, 294 16.16 7.08 *** .046 &lt;.001 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;+&#39; 0.1 &#39; &#39; 1 # Option 2: Using the car package (here we have to remember to set the contrasts to sum # contrasts!) lm(balance ~ hand * skill, contrasts = list(hand = &quot;contr.sum&quot;, skill = &quot;contr.sum&quot;), data = df.poker) %&gt;% car::Anova(type = 3) Anova Table (Type III tests) Response: balance Sum Sq Df F value Pr(&gt;F) (Intercept) 28644.7 1 1772.1137 &lt; 2.2e-16 *** hand 2559.4 2 79.1692 &lt; 2.2e-16 *** skill 39.3 1 2.4344 0.1197776 hand:skill 229.0 2 7.0830 0.0009901 *** Residuals 4752.3 294 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 # Option 3: Using the emmeans package (I like this one the best! It let&#39;s us use the # general lm() syntax and we don&#39;t have to remember to set the contrast) fit.lm_poker = lm(balance ~ hand * skill, data = df.poker) fit.lm_poker %&gt;% joint_tests() model term df1 df2 F.ratio p.value hand 2 294 79.169 &lt;.0001 skill 1 294 2.434 0.1198 hand:skill 2 294 7.083 0.0010 All three options give the same result. Personally, I like Option 3 the best. 23.4.2.2 Bayesian model Now, let’s fit a Bayesian regression model using the brm() function (starting with a simple model that only considers hand as a predictor): fit.brm_poker = brm(formula = balance ~ 1 + hand, data = df.poker, seed = 1, file = &quot;cache/brm_poker&quot;) # we&#39;ll use this model here later fit.brm_poker2 = brm(formula = balance ~ 1 + hand * skill, data = df.poker, seed = 1, file = &quot;cache/brm_poker2&quot;) fit.brm_poker %&gt;% summary() Family: gaussian Links: mu = identity; sigma = identity Formula: balance ~ 1 + hand Data: df.poker (Number of observations: 300) Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1; total post-warmup draws = 4000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept 5.94 0.41 5.10 6.73 1.00 3385 2517 handneutral 4.40 0.58 3.28 5.59 1.00 3607 2612 handgood 7.08 0.59 5.94 8.23 1.00 3719 2923 Family Specific Parameters: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS sigma 4.12 0.17 3.80 4.48 1.00 3893 3019 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). I use the file = argument to save the model’s results so that when I run this code chunk again, the model doesn’t need to be fit again (fitting Bayesian models takes a while …). And I used the seed = argument to make this example reproducible. 23.4.2.2.1 Full specification So far, we have used the defaults that brm() comes with and not bothered about specifiying the priors, etc. Notice that we didn’t specify any priors in the model. By default, “brms” assigns weakly informative priors to the parameters in the model. We can see what these are by running the following command: fit.brm_poker %&gt;% prior_summary() prior class coef group resp dpar nlpar lb ub (flat) b (flat) b handgood (flat) b handneutral student_t(3, 9.5, 5.6) Intercept student_t(3, 0, 5.6) sigma 0 source default (vectorized) (vectorized) default default We can also get information about which priors need to be specified before fitting a model: get_prior(formula = balance ~ 1 + hand, family = &quot;gaussian&quot;, data = df.poker) prior class coef group resp dpar nlpar lb ub (flat) b (flat) b handgood (flat) b handneutral student_t(3, 9.5, 5.6) Intercept student_t(3, 0, 5.6) sigma 0 source default (vectorized) (vectorized) default default Here is an example for what a more complete model specification could look like: fit.brm_poker_full = brm(formula = balance ~ 1 + hand, family = &quot;gaussian&quot;, data = df.poker, prior = c(prior(normal(0, 10), class = &quot;b&quot;, coef = &quot;handgood&quot;), prior(normal(0, 10), class = &quot;b&quot;, coef = &quot;handneutral&quot;), prior(student_t(3, 3, 10), class = &quot;Intercept&quot;), prior(student_t(3, 0, 10), class = &quot;sigma&quot;)), inits = list(list(Intercept = 0, sigma = 1, handgood = 5, handneutral = 5), list(Intercept = -5, sigma = 3, handgood = 2, handneutral = 2), list(Intercept = 2, sigma = 1, handgood = -1, handneutral = 1), list(Intercept = 1, sigma = 2, handgood = 2, handneutral = -2)), iter = 4000, warmup = 1000, chains = 4, file = &quot;cache/brm_poker_full&quot;, seed = 1) fit.brm_poker_full %&gt;% summary() Family: gaussian Links: mu = identity; sigma = identity Formula: balance ~ 1 + hand Data: df.poker (Number of observations: 300) Draws: 4 chains, each with iter = 4000; warmup = 1000; thin = 1; total post-warmup draws = 12000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept 5.96 0.41 5.17 6.78 1.00 9472 7929 handneutral 4.37 0.59 3.21 5.49 1.00 10221 9046 handgood 7.05 0.59 5.89 8.20 1.00 10521 8934 Family Specific Parameters: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS sigma 4.13 0.17 3.81 4.47 1.00 10744 8408 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). We can also take a look at the Stan code that the brm() function creates: fit.brm_poker_full %&gt;% stancode() // generated with brms 2.14.4 functions { } data { int&lt;lower=1&gt; N; // total number of observations vector[N] Y; // response variable int&lt;lower=1&gt; K; // number of population-level effects matrix[N, K] X; // population-level design matrix int prior_only; // should the likelihood be ignored? } transformed data { int Kc = K - 1; matrix[N, Kc] Xc; // centered version of X without an intercept vector[Kc] means_X; // column means of X before centering for (i in 2:K) { means_X[i - 1] = mean(X[, i]); Xc[, i - 1] = X[, i] - means_X[i - 1]; } } parameters { vector[Kc] b; // population-level effects real Intercept; // temporary intercept for centered predictors real&lt;lower=0&gt; sigma; // residual SD } transformed parameters { } model { // likelihood including all constants if (!prior_only) { target += normal_id_glm_lpdf(Y | Xc, Intercept, b, sigma); } // priors including all constants target += normal_lpdf(b[1] | 0, 10); target += normal_lpdf(b[2] | 0, 10); target += student_t_lpdf(Intercept | 3, 3, 10); target += student_t_lpdf(sigma | 3, 0, 10) - 1 * student_t_lccdf(0 | 3, 0, 10); } generated quantities { // actual population-level intercept real b_Intercept = Intercept - dot_product(means_X, b); } One thing worth noticing: by default, “brms” centers the predictors which makes it easier to assign a default prior over the intercept. 23.4.3 3. Model evaluation 23.4.3.1 a) Did the inference work? So far, we’ve assumed that the inference has worked out. We can check this by running plot() on our brm object: plot(fit.brm_poker, N = 7, ask = F) The posterior distributions (left hand side), and the trace plots of the samples from the posterior (right hand side) look good. Let’s make our own version of a trace plot for one parameter in the model: fit.brm_poker %&gt;% spread_draws(b_Intercept) %&gt;% clean_names() %&gt;% mutate(chain = as.factor(chain)) %&gt;% ggplot(aes(x = iteration, y = b_intercept, group = chain, color = chain)) + geom_line() + scale_color_brewer(type = &quot;seq&quot;, direction = -1) We can also take a look at the auto-correlation plot. Ideally, we want to generate independent samples from the posterior. So we don’t want subsequent samples to be strongly correlated with each other. Let’s take a look: variables = fit.brm_poker %&gt;% get_variables() %&gt;% .[1:4] fit.brm_poker %&gt;% posterior_samples() %&gt;% mcmc_acf(pars = variables, lags = 4) Warning: Method &#39;posterior_samples&#39; is deprecated. Please see ?as_draws for recommended alternatives. Looking good! The autocorrelation should become very small as the lag increases (indicating that we are getting independent samples from the posterior). 23.4.3.1.0.1 When things go wrong Let’s try to fit a model to very little data (just two observations) with extremely uninformative priors: df.data = tibble(y = c(-1, 1)) fit.brm_wrong = brm(data = df.data, family = gaussian, formula = y ~ 1, prior = c(prior(uniform(-1e10, 1e10), class = Intercept), prior(uniform(0, 1e10), class = sigma)), inits = list(list(Intercept = 0, sigma = 1), list(Intercept = 0, sigma = 1)), iter = 4000, warmup = 1000, chains = 2, file = &quot;cache/brm_wrong&quot;) Let’s take a look at the posterior distributions of the model parameters: summary(fit.brm_wrong) Warning: Parts of the model have not converged (some Rhats are &gt; 1.05). Be careful when analysing the results! We recommend running more iterations and/or setting stronger priors. Warning: There were 1348 divergent transitions after warmup. Increasing adapt_delta above 0.8 may help. See http://mc-stan.org/misc/ warnings.html#divergent-transitions-after-warmup Family: gaussian Links: mu = identity; sigma = identity Formula: y ~ 1 Data: df.data (Number of observations: 2) Draws: 2 chains, each with iter = 4000; warmup = 1000; thin = 1; total post-warmup draws = 6000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Intercept -44780254.01 1085496757.06 -3091150221.00 2498148317.09 1.73 218 Tail_ESS Intercept 113 Family Specific Parameters: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS sigma 924899544.31 1859482970.46 22790.00 6962912540.71 1.69 3 64 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). Not looking good – The estimates and credible intervals are off the charts. And the effective samples sizes in the chains are very small. Let’s visualize the trace plots: plot(fit.brm_wrong, N = 2, ask = F) fit.brm_wrong %&gt;% spread_draws(b_Intercept) %&gt;% clean_names() %&gt;% mutate(chain = as.factor(chain)) %&gt;% ggplot(aes(x = iteration, y = b_intercept, group = chain, color = chain)) + geom_line() + scale_color_brewer(direction = -1) Given that we have so little data in this case, we need to help the model a little bit by providing some slighlty more specific priors. fit.brm_right = brm(data = df.data, family = gaussian, formula = y ~ 1, prior = c(prior(normal(0, 10), class = Intercept), # more reasonable priors prior(cauchy(0, 1), class = sigma)), iter = 4000, warmup = 1000, chains = 2, seed = 1, file = &quot;cache/brm_right&quot;) Let’s take a look at the posterior distributions of the model parameters: summary(fit.brm_right) Family: gaussian Links: mu = identity; sigma = identity Formula: y ~ 1 Data: df.data (Number of observations: 2) Draws: 2 chains, each with iter = 4000; warmup = 1000; thin = 1; total post-warmup draws = 6000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept -0.03 1.72 -3.62 3.50 1.00 1240 1015 Family Specific Parameters: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS sigma 2.07 2.03 0.59 6.69 1.00 1180 1608 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). This looks much better. There is still quite a bit of uncertainty in our paremeter estimates, but it has reduced dramatically. Let’s visualize the trace plots: plot(fit.brm_right, N = 2, ask = F) fit.brm_right %&gt;% spread_draws(b_Intercept, sigma) %&gt;% clean_names() %&gt;% mutate(chain = as.factor(chain)) %&gt;% pivot_longer(cols = c(b_intercept, sigma)) %&gt;% ggplot(aes(x = iteration, y = value, group = chain, color = chain)) + geom_line() + facet_wrap(vars(name), ncol = 1) + scale_color_brewer(direction = -1) Looking mostly good! 23.4.3.2 b) Visualize model predictions 23.4.3.2.1 Posterior predictive check To check whether the model did a good job capturing the data, we can simulate what future data the Bayesian model predicts, now that it has learned from the data we feed into it. pp_check(fit.brm_poker, nsamples = 100) Warning: Argument &#39;nsamples&#39; is deprecated. Please use argument &#39;ndraws&#39; instead. This looks good! The predicted shaped of the data based on samples from the posterior distribution looks very similar to the shape of the actual data. Let’s make a hypothetical outcome plot that shows what concrete data sets the model would predict. The add_predicted_draws() function from the “tidybayes” package is helpful for generating predictions from the posterior. df.predictive_samples = df.poker %&gt;% add_predicted_draws(newdata = ., object = fit.brm_poker2, ndraws = 10) p = ggplot(data = df.predictive_samples, mapping = aes(x = hand, y = .prediction, fill = hand, group = skill, shape = skill)) + geom_point(alpha = 0.2, position = position_jitterdodge(dodge.width = 0.5, jitter.height = 0, jitter.width = 0.2)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, position = position_dodge(width = 0.5), size = 1) + labs(y = &quot;final balance (in Euros)&quot;) + scale_shape_manual(values = c(21, 22)) + guides(fill = guide_legend(override.aes = list(shape = 21)), shape = guide_legend(override.aes = list(alpha = 1, fill = &quot;black&quot;))) + transition_manual(.draw) animate(p, nframes = 120, width = 800, height = 600, res = 96, type = &quot;cairo&quot;) nframes and fps adjusted to match transition Rendering [========&gt;-----------------------------------] at 7.6 fps ~ eta: 1s Rendering [============&gt;-------------------------------] at 7.6 fps ~ eta: 1s Rendering [=================&gt;--------------------------] at 7.7 fps ~ eta: 1s Rendering [=====================&gt;----------------------] at 7.8 fps ~ eta: 1s Rendering [=========================&gt;------------------] at 7.9 fps ~ eta: 1s Rendering [==============================&gt;-------------] at 7.9 fps ~ eta: 0s Rendering [==================================&gt;---------] at 7.9 fps ~ eta: 0s Rendering [========================================&gt;-----] at 8 fps ~ eta: 0s Rendering [==============================================] at 8 fps ~ eta: 0s 23.4.3.2.2 Prior predictive check fit.brm_poker_prior = brm(formula = balance ~ 0 + Intercept + hand * skill, family = &quot;gaussian&quot;, data = df.poker, prior = c(prior(normal(0, 10), class = &quot;b&quot;), prior(student_t(3, 0, 10), class = &quot;sigma&quot;)), iter = 4000, warmup = 1000, chains = 4, file = &quot;cache/brm_poker_prior&quot;, sample_prior = &quot;only&quot;, seed = 1) # generate prior samples df.prior_samples = df.poker %&gt;% add_predicted_draws(newdata = ., object = fit.brm_poker_prior, ndraws = 10) # plot the results as an animation p = ggplot(data = df.prior_samples, mapping = aes(x = hand, y = .prediction, fill = hand, group = skill, shape = skill)) + geom_point(alpha = 0.2, position = position_jitterdodge(dodge.width = 0.5, jitter.height = 0, jitter.width = 0.2)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, position = position_dodge(width = 0.5), size = 1) + labs(y = &quot;final balance (in Euros)&quot;) + scale_shape_manual(values = c(21, 22)) + guides(fill = guide_legend(override.aes = list(shape = 21, fill = RColorBrewer::brewer.pal(3, &quot;Set1&quot;))), shape = guide_legend(override.aes = list(alpha = 1, fill = &quot;black&quot;))) + transition_manual(.draw) animate(p, nframes = 120, width = 800, height = 600, res = 96, type = &quot;cairo&quot;) nframes and fps adjusted to match transition Rendering [========&gt;-----------------------------------] at 8.5 fps ~ eta: 1s Rendering [============&gt;-------------------------------] at 8.4 fps ~ eta: 1s Rendering [=================&gt;--------------------------] at 8.3 fps ~ eta: 1s Rendering [=====================&gt;----------------------] at 8.3 fps ~ eta: 1s Rendering [=========================&gt;------------------] at 8.2 fps ~ eta: 0s Rendering [==============================&gt;-------------] at 8.2 fps ~ eta: 0s Rendering [==================================&gt;---------] at 8.2 fps ~ eta: 0s Rendering [=======================================&gt;----] at 8.2 fps ~ eta: 0s Rendering [============================================] at 8.2 fps ~ eta: 0s # anim_save(&quot;poker_prior_predictive.gif&quot;) 23.4.4 4. Interpret the model parameters 23.4.4.1 Visualize the posteriors Let’s visualize what the posterior for the different parameters looks like. We use the stat_halfeye() function from the “tidybayes” package to do so: fit.brm_poker %&gt;% posterior_samples() %&gt;% clean_names() %&gt;% select(starts_with(&quot;b_&quot;), sigma) %&gt;% pivot_longer(cols = everything(), names_to = &quot;variable&quot;, values_to = &quot;value&quot;) %&gt;% ggplot(data = ., mapping = aes(y = fct_rev(variable), x = value)) + stat_halfeye(fill = &quot;lightblue&quot;) + theme(axis.title.y = element_blank()) Warning: Method &#39;posterior_samples&#39; is deprecated. Please see ?as_draws for recommended alternatives. 23.4.4.2 Compute highest density intervals To compute the MAP (maximum a posteriori probability) estimate and highest density interval, we use the mean_hdi() function that comes with the “tidybayes” package. fit.brm_poker %&gt;% posterior_samples() %&gt;% clean_names() %&gt;% select(starts_with(&quot;b_&quot;), sigma) %&gt;% mean_hdi() %&gt;% pivot_longer(cols = -c(.width:.interval), names_to = &quot;index&quot;, values_to = &quot;value&quot;) %&gt;% select(index, value) %&gt;% mutate(index = ifelse(str_detect(index, fixed(&quot;.&quot;)), index, str_c(index, &quot;.mean&quot;))) %&gt;% separate(index, into = c(&quot;parameter&quot;, &quot;type&quot;), sep = &quot;\\\\.&quot;) %&gt;% pivot_wider(names_from = type, values_from = value) Warning: Method &#39;posterior_samples&#39; is deprecated. Please see ?as_draws for recommended alternatives. # A tibble: 4 × 4 parameter mean lower upper &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 b_intercept 5.94 5.16 6.78 2 b_handneutral 4.40 3.24 5.52 3 b_handgood 7.08 5.92 8.20 4 sigma 4.12 3.79 4.45 23.4.5 5. Test specific hypotheses 23.4.5.1 with hypothesis() One key advantage of Bayesian over frequentist analysis is that we can test hypothesis in a very flexible manner by directly probing our posterior samples in different ways. We may ask, for example, what the probability is that the parameter for the difference between a bad hand and a neutral hand (b_handneutral) is greater than 0. Let’s plot the posterior distribution together with the criterion: fit.brm_poker %&gt;% posterior_samples() %&gt;% select(b_handneutral) %&gt;% pivot_longer(cols = everything(), names_to = &quot;variable&quot;, values_to = &quot;value&quot;) %&gt;% ggplot(data = ., mapping = aes(y = variable, x = value)) + stat_halfeye(fill = &quot;lightblue&quot;) + geom_vline(xintercept = 0, color = &quot;red&quot;) Warning: Method &#39;posterior_samples&#39; is deprecated. Please see ?as_draws for recommended alternatives. We see that the posterior is definitely greater than 0. We can ask many different kinds of questions about the data by doing basic arithmetic on our posterior samples. The hypothesis() function makes this even easier. Here are some examples: # the probability that the posterior for handneutral is less than 0 hypothesis(fit.brm_poker, hypothesis = &quot;handneutral &lt; 0&quot;) Hypothesis Tests for class b: Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio Post.Prob 1 (handneutral) &lt; 0 4.4 0.58 3.47 5.38 0 0 Star 1 --- &#39;CI&#39;: 90%-CI for one-sided and 95%-CI for two-sided hypotheses. &#39;*&#39;: For one-sided hypotheses, the posterior probability exceeds 95%; for two-sided hypotheses, the value tested against lies outside the 95%-CI. Posterior probabilities of point hypotheses assume equal prior probabilities. # the probability that the posterior for handneutral is greater than 4 hypothesis(fit.brm_poker, hypothesis = &quot;handneutral &gt; 4&quot;) %&gt;% plot() # the probability that good hands make twice as much as bad hands hypothesis(fit.brm_poker, hypothesis = &quot;Intercept + handgood &gt; 2 * Intercept&quot;) Hypothesis Tests for class b: Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio 1 (Intercept+handgo... &gt; 0 1.14 0.92 -0.37 2.68 8.55 Post.Prob Star 1 0.9 --- &#39;CI&#39;: 90%-CI for one-sided and 95%-CI for two-sided hypotheses. &#39;*&#39;: For one-sided hypotheses, the posterior probability exceeds 95%; for two-sided hypotheses, the value tested against lies outside the 95%-CI. Posterior probabilities of point hypotheses assume equal prior probabilities. We can also make a plot of what the posterior distribution of the hypothesis looks like: hypothesis(fit.brm_poker, hypothesis = &quot;Intercept + handgood &gt; 2 * Intercept&quot;) %&gt;% plot() # the probability that neutral hands make less than the average of bad and good hands hypothesis(fit.brm_poker, hypothesis = &quot;Intercept + handneutral &lt; (Intercept + Intercept + handgood) / 2&quot;) Hypothesis Tests for class b: Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio 1 (Intercept+handne... &lt; 0 0.86 0.51 0.02 1.7 0.05 Post.Prob Star 1 0.04 --- &#39;CI&#39;: 90%-CI for one-sided and 95%-CI for two-sided hypotheses. &#39;*&#39;: For one-sided hypotheses, the posterior probability exceeds 95%; for two-sided hypotheses, the value tested against lies outside the 95%-CI. Posterior probabilities of point hypotheses assume equal prior probabilities. Let’s double check one example, and calculate the result directly based on the posterior samples: df.hypothesis = fit.brm_poker %&gt;% posterior_samples() %&gt;% clean_names() %&gt;% select(starts_with(&quot;b_&quot;)) %&gt;% mutate(neutral = b_intercept + b_handneutral, bad_good_average = (b_intercept + b_intercept + b_handgood)/2, hypothesis = neutral &lt; bad_good_average) Warning: Method &#39;posterior_samples&#39; is deprecated. Please see ?as_draws for recommended alternatives. df.hypothesis %&gt;% summarize(p = sum(hypothesis)/n()) p 1 0.045 23.4.5.2 with emmeans() We can also use the emmeans() function to compute contrasts. fit.brm_poker %&gt;% emmeans(specs = consec ~ hand) Loading required namespace: rstanarm $emmeans hand emmean lower.HPD upper.HPD bad 5.94 5.16 6.78 neutral 10.34 9.55 11.15 good 13.02 12.22 13.82 Point estimate displayed: median HPD interval probability: 0.95 $contrasts contrast estimate lower.HPD upper.HPD neutral - bad 4.38 3.24 5.52 good - neutral 2.69 1.51 3.78 Point estimate displayed: median HPD interval probability: 0.95 Here, it computed the estimated means for each group for us, as well as the consecutive contrasts between each group. Let’s visualize the contrasts. First, let’s just use the plot() function as it’s been adapted by the emmeans package: fit.brm_poker %&gt;% emmeans(specs = consec ~ hand) %&gt;% pluck(&quot;contrasts&quot;) %&gt;% plot() To get full posterior distributions instead of summaries, we can use the “tidybayes” package like so: fit.brm_poker %&gt;% emmeans(specs = consec ~ hand) %&gt;% pluck(&quot;contrasts&quot;) %&gt;% gather_emmeans_draws() %&gt;% ggplot(mapping = aes(y = contrast, x = .value)) + stat_halfeye(fill = &quot;lightblue&quot;, point_interval = mean_hdi, .width = c(0.5, 0.75, 0.95)) To see whether neutral hands did differently from bad and good hands (combined), we can define the following contrast. contrasts = list(neutral_vs_rest = c(-1, 2, -1)) fit.brm_poker %&gt;% emmeans(specs = &quot;hand&quot;, contr = contrasts) %&gt;% pluck(&quot;contrasts&quot;) %&gt;% gather_emmeans_draws() %&gt;% mean_hdi() # A tibble: 1 × 7 contrast .value .lower .upper .width .point .interval &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 neutral_vs_rest 1.72 -0.298 3.63 0.95 mean hdi Here, the HDP does not exclude 0. Let’s double check that we get the same result using the hypothesis() function, or by directly computing from the posterior samples. # using hypothesis() fit.brm_poker %&gt;% hypothesis(&quot;(Intercept + handneutral)*2 &lt; (Intercept + Intercept + handgood)&quot;) Hypothesis Tests for class b: Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio 1 ((Intercept+handn... &lt; 0 1.72 1.02 0.05 3.41 0.05 Post.Prob Star 1 0.04 --- &#39;CI&#39;: 90%-CI for one-sided and 95%-CI for two-sided hypotheses. &#39;*&#39;: For one-sided hypotheses, the posterior probability exceeds 95%; for two-sided hypotheses, the value tested against lies outside the 95%-CI. Posterior probabilities of point hypotheses assume equal prior probabilities. # directly computing from the posterior fit.brm_poker %&gt;% posterior_samples() %&gt;% clean_names() %&gt;% mutate(contrast = (b_intercept + b_handneutral) * 2 - (b_intercept + b_intercept + b_handgood)) %&gt;% summarize(contrast = mean(contrast)) Warning: Method &#39;posterior_samples&#39; is deprecated. Please see ?as_draws for recommended alternatives. contrast 1 1.724884 The emmeans() function becomes particularly useful when our model has several categorical predictors, and we are interested in comparing differences along one predictor while marginalizing over the values of the other predictor. Let’s take a look for a model that considers both skill and hand as predictors (as well as the interaction). fit.brm_poker2 = brm(formula = balance ~ hand * skill, data = df.poker, seed = 1, file = &quot;cache/brm_poker2&quot;) fit.brm_poker2 %&gt;% summary() Family: gaussian Links: mu = identity; sigma = identity Formula: balance ~ hand * skill Data: df.poker (Number of observations: 300) Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1; total post-warmup draws = 4000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Intercept 4.59 0.57 3.46 5.73 1.00 1870 handneutral 5.24 0.81 3.66 6.84 1.00 2163 handgood 9.20 0.81 7.61 10.76 1.00 2027 skillexpert 2.71 0.81 1.09 4.26 1.00 1785 handneutral:skillexpert -1.69 1.16 -3.96 0.57 1.00 1826 handgood:skillexpert -4.26 1.14 -6.52 -2.05 1.00 1822 Tail_ESS Intercept 2429 handneutral 2747 handgood 2805 skillexpert 2506 handneutral:skillexpert 2488 handgood:skillexpert 2495 Family Specific Parameters: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS sigma 4.03 0.17 3.72 4.37 1.00 3665 2627 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). In the summary table above, skillexpert captures the difference between an expert and an average player when they have a bad hand. To see whether there was a difference in expertise overall (i.e. across all three kinds of hands), we can calculate a linear contrast. fit.brm_poker2 %&gt;% emmeans(pairwise ~ skill) NOTE: Results may be misleading due to involvement in interactions $emmeans skill emmean lower.HPD upper.HPD average 9.41 8.74 10.0 expert 10.14 9.45 10.7 Point estimate displayed: median HPD interval probability: 0.95 $contrasts contrast estimate lower.HPD upper.HPD average - expert -0.73 -1.6 0.207 Point estimate displayed: median HPD interval probability: 0.95 It looks like overall, skilled players weren’t doing much better than average players. We can even do something like an equivalent of an ANOVA using emmeans(), like so: joint_tests(fit.brm_poker2) model term df1 df2 F.ratio p.value hand 2 Inf 78.554 &lt;.0001 skill 1 Inf 2.450 0.1176 hand:skill 2 Inf 7.040 0.0009 The values we get here are very similar to what we would get from a frequentist ANOVA: aov_ez(id = &quot;participant&quot;, dv = &quot;balance&quot;, between = c(&quot;hand&quot;, &quot;skill&quot;), data = df.poker) Contrasts set to contr.sum for the following variables: hand, skill Anova Table (Type 3 tests) Response: balance Effect df MSE F ges p.value 1 hand 2, 294 16.16 79.17 *** .350 &lt;.001 2 skill 1, 294 16.16 2.43 .008 .120 3 hand:skill 2, 294 16.16 7.08 *** .046 &lt;.001 --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;+&#39; 0.1 &#39; &#39; 1 23.4.5.3 Bayes factor Another way of testing hypothesis is via the Bayes factor. Let’s fit the two models we are interested in comparing with each other: fit.brm_poker_bf1 = brm(formula = balance ~ 1 + hand, data = df.poker, save_pars = save_pars(all = T), file = &quot;cache/brm_poker_bf1&quot;) fit.brm_poker_bf2 = brm(formula = balance ~ 1 + hand + skill, data = df.poker, save_pars = save_pars(all = T), file = &quot;cache/brm_poker_bf2&quot;) And then compare the models using the bayes_factor() function: bayes_factor(fit.brm_poker_bf2, fit.brm_poker_bf1) Iteration: 1 Iteration: 2 Iteration: 3 Iteration: 4 Iteration: 5 Iteration: 1 Iteration: 2 Iteration: 3 Iteration: 4 Estimated Bayes factor in favor of fit.brm_poker_bf2 over fit.brm_poker_bf1: 3.80995 Bayes factors don’t have a very good reputation (see here and here). Instead, the way to go these days appears to be via approximate leave one out cross-validation. 23.4.5.4 Approximate leave one out cross-validation fit.brm_poker_bf1 = add_criterion(fit.brm_poker_bf1, criterion = &quot;loo&quot;, reloo = T, file = &quot;cache/brm_poker_bf1&quot;) fit.brm_poker_bf2 = add_criterion(fit.brm_poker_bf2, criterion = &quot;loo&quot;, reloo = T, file = &quot;cache/brm_poker_bf2&quot;) loo_compare(fit.brm_poker_bf1, fit.brm_poker_bf2) elpd_diff se_diff fit.brm_poker_bf2 0.0 0.0 fit.brm_poker_bf1 -0.2 1.5 23.5 Sleep study 23.5.1 1. Visualize the data set.seed(1) ggplot(data = df.sleep %&gt;% mutate(days = as.factor(days)), mapping = aes(x = days, y = reaction)) + geom_point(alpha = 0.2, position = position_jitter(width = 0.1)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;) 23.5.2 2. Specify and fit the model 23.5.2.1 Frequentist analysis fit.lmer_sleep = lmer(formula = reaction ~ 1 + days + (1 + days | subject), data = df.sleep) fit.lmer_sleep %&gt;% summary() Linear mixed model fit by REML. t-tests use Satterthwaite&#39;s method [ lmerModLmerTest] Formula: reaction ~ 1 + days + (1 + days | subject) Data: df.sleep REML criterion at convergence: 1771.4 Scaled residuals: Min 1Q Median 3Q Max -3.9707 -0.4703 0.0276 0.4594 5.2009 Random effects: Groups Name Variance Std.Dev. Corr subject (Intercept) 582.72 24.140 days 35.03 5.919 0.07 Residual 649.36 25.483 Number of obs: 183, groups: subject, 20 Fixed effects: Estimate Std. Error df t value Pr(&gt;|t|) (Intercept) 252.543 6.433 19.295 39.257 &lt; 2e-16 *** days 10.452 1.542 17.163 6.778 3.06e-06 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Correlation of Fixed Effects: (Intr) days -0.137 23.5.2.2 Bayesian analysis fit.brm_sleep = brm(formula = reaction ~ 1 + days + (1 + days | subject), data = df.sleep, seed = 1, file = &quot;cache/brm_sleep&quot;) 23.5.3 3. Model evaluation 23.5.3.1 a) Did the inference work? fit.brm_sleep %&gt;% summary() Family: gaussian Links: mu = identity; sigma = identity Formula: reaction ~ 1 + days + (1 + days | subject) Data: df.sleep (Number of observations: 183) Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1; total post-warmup draws = 4000 Group-Level Effects: ~subject (Number of levels: 20) Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS sd(Intercept) 26.14 6.37 15.50 40.39 1.00 1988 2693 sd(days) 6.55 1.54 4.12 10.02 1.00 1341 1613 cor(Intercept,days) 0.09 0.30 -0.48 0.67 1.00 943 1403 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept 252.39 7.00 238.22 266.40 1.00 1631 2124 days 10.34 1.72 6.88 13.70 1.00 1240 1911 Family Specific Parameters: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS sigma 25.80 1.54 23.01 29.10 1.00 3592 2762 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). fit.brm_sleep %&gt;% plot(N = 6) 23.5.3.2 b) Visualize model predictions pp_check(fit.brm_sleep, nsamples = 100) Warning: Argument &#39;nsamples&#39; is deprecated. Please use argument &#39;ndraws&#39; instead. 23.5.4 4. Interpret the parameters fit.brm_sleep %&gt;% tidy(conf.method = &quot;HPDinterval&quot;) # A tibble: 6 × 8 effect component group term estimate std.error conf.low conf.high &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 fixed cond &lt;NA&gt; (Intercept) 252. 7.00 239. 267. 2 fixed cond &lt;NA&gt; days 10.3 1.72 7.05 13.8 3 ran_pars cond subject sd__(Interc… 26.1 6.37 15.0 39.3 4 ran_pars cond subject sd__days 6.55 1.54 3.92 9.50 5 ran_pars cond subject cor__(Inter… 0.0909 0.298 -0.467 0.677 6 ran_pars cond Residual sd__Observa… 25.8 1.54 22.9 28.9 23.5.4.1 Summary of posterior distributions # all parameters fit.brm_sleep %&gt;% posterior_samples() %&gt;% select(-c(lp__, contains(&quot;[&quot;))) %&gt;% pivot_longer(cols = everything(), names_to = &quot;variable&quot;, values_to = &quot;value&quot;) %&gt;% ggplot(data = ., mapping = aes(x = value)) + stat_halfeye(point_interval = mode_hdi, fill = &quot;lightblue&quot;) + facet_wrap(~ variable, ncol = 2, scales = &quot;free&quot;) + theme(text = element_text(size = 12)) Warning: Method &#39;posterior_samples&#39; is deprecated. Please see ?as_draws for recommended alternatives. # just the parameter of interest fit.brm_sleep %&gt;% posterior_samples() %&gt;% select(b_days) %&gt;% ggplot(data = ., mapping = aes(x = b_days)) + stat_halfeye(point_interval = mode_hdi, fill = &quot;lightblue&quot;) + theme(text = element_text(size = 12)) Warning: Method &#39;posterior_samples&#39; is deprecated. Please see ?as_draws for recommended alternatives. 23.5.5 5. Test specific hypotheses Here, we were just interested in how the number of days of sleep deprivation affected reaction time (and we can see that by inspecting the posterior for the days predictor in the model). 23.5.6 6. Report results 23.5.6.1 Model prediction with posterior draws (aggregate) df.model = tibble(days = 0:9) %&gt;% add_linpred_draws(newdata = ., object = fit.brm_sleep, ndraws = 10, seed = 1, re_formula = NA) ggplot(data = df.sleep, mapping = aes(x = days, y = reaction)) + geom_point(alpha = 0.2, position = position_jitter(width = 0.1)) + geom_line(data = df.model, mapping = aes(y = .linpred, group = .draw), color = &quot;lightblue&quot;) + stat_summary(fun.data = &quot;mean_cl_boot&quot;) + scale_x_continuous(breaks = 0:9) 23.5.6.2 Model prediction with credible intervals (aggregate) df.model = fit.brm_sleep %&gt;% fitted(re_formula = NA, newdata = tibble(days = 0:9)) %&gt;% as_tibble() %&gt;% mutate(days = 0:9) %&gt;% clean_names() ggplot(data = df.sleep, mapping = aes(x = days, y = reaction)) + geom_point(alpha = 0.2, position = position_jitter(width = 0.1)) + geom_ribbon(data = df.model, mapping = aes(y = estimate, ymin = q2_5, ymax = q97_5), fill = &quot;lightblue&quot;, alpha = 0.5) + geom_line(data = df.model, mapping = aes(y = estimate), color = &quot;lightblue&quot;, size = 1) + stat_summary(fun.data = &quot;mean_cl_boot&quot;) + scale_x_continuous(breaks = 0:9) 23.5.6.3 Model prediction with credible intervals (individual participants) fit.brm_sleep %&gt;% fitted() %&gt;% as_tibble() %&gt;% clean_names() %&gt;% bind_cols(df.sleep) %&gt;% ggplot(data = ., mapping = aes(x = days, y = reaction)) + geom_ribbon(aes(ymin = q2_5, ymax = q97_5), fill = &quot;lightblue&quot;) + geom_line(aes(y = estimate), color = &quot;blue&quot;) + geom_point() + facet_wrap(~subject, ncol = 5) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) 23.5.6.4 Model prediction for random samples df.model = df.sleep %&gt;% complete(subject, days) %&gt;% add_linpred_draws(newdata = ., object = fit.brm_sleep, ndraws = 10, seed = 1) df.sleep %&gt;% ggplot(data = ., mapping = aes(x = days, y = reaction)) + geom_line(data = df.model, aes(y = .linpred, group = .draw), color = &quot;lightblue&quot;, alpha = 0.5) + geom_point() + facet_wrap(~subject, ncol = 5) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) 23.5.6.5 Animated model prediction for random samples df.model = df.sleep %&gt;% complete(subject, days) %&gt;% add_linpred_draws(newdata = ., object = fit.brm_sleep, ndraws = 10, seed = 1) p = df.sleep %&gt;% ggplot(data = ., mapping = aes(x = days, y = reaction)) + geom_line(data = df.model, aes(y = .linpred, group = .draw), color = &quot;black&quot;) + geom_point() + facet_wrap(~subject, ncol = 5) + labs(x = &quot;Days of sleep deprivation&quot;, y = &quot;Average reaction time (ms)&quot;) + scale_x_continuous(breaks = 0:4 * 2) + theme(strip.text = element_text(size = 12), axis.text.y = element_text(size = 12)) + transition_states(.draw, 0, 1) + shadow_mark(past = TRUE, alpha = 1/5, color = &quot;gray50&quot;) animate(p, nframes = 10, fps = 1, width = 800, height = 600, res = 96, type = &quot;cairo&quot;) Rendering [===&gt;----------------------------------------] at 2.4 fps ~ eta: 4s Rendering [========&gt;-----------------------------------] at 2.4 fps ~ eta: 3s Rendering [============&gt;-------------------------------] at 2.3 fps ~ eta: 3s Rendering [=================&gt;--------------------------] at 2.3 fps ~ eta: 3s Rendering [=====================&gt;----------------------] at 2.3 fps ~ eta: 2s Rendering [=========================&gt;------------------] at 2.3 fps ~ eta: 2s Rendering [==============================&gt;-------------] at 2.3 fps ~ eta: 1s Rendering [==================================&gt;---------] at 2.2 fps ~ eta: 1s Rendering [=======================================&gt;----] at 2.2 fps ~ eta: 0s Rendering [============================================] at 2.2 fps ~ eta: 0s # anim_save(&quot;sleep_posterior_predictive.gif&quot;) 23.6 Titanic study 23.6.1 1. Visualize that data df.titanic %&gt;% mutate(sex = as.factor(sex)) %&gt;% ggplot(data = ., mapping = aes(x = fare, y = survived, color = sex)) + geom_point(alpha = 0.1, size = 2) + geom_smooth(method = &quot;glm&quot;, method.args = list(family = &quot;binomial&quot;), alpha = 0.2, aes(fill = sex)) + scale_color_brewer(palette = &quot;Set1&quot;) 23.6.2 2. Specify and fit the model 23.6.2.1 Frequentist analysis fit.glm_titanic = glm(formula = survived ~ 1 + fare * sex, family = &quot;binomial&quot;, data = df.titanic) fit.glm_titanic %&gt;% summary() Call: glm(formula = survived ~ 1 + fare * sex, family = &quot;binomial&quot;, data = df.titanic) Deviance Residuals: Min 1Q Median 3Q Max -2.6280 -0.6279 -0.5991 0.8172 1.9288 Coefficients: Estimate Std. Error z value Pr(&gt;|z|) (Intercept) 0.408428 0.189999 2.150 0.031584 * fare 0.019878 0.005372 3.701 0.000215 *** sexmale -2.099345 0.230291 -9.116 &lt; 2e-16 *** fare:sexmale -0.011617 0.005934 -1.958 0.050252 . --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 1186.66 on 890 degrees of freedom Residual deviance: 879.85 on 887 degrees of freedom AIC: 887.85 Number of Fisher Scoring iterations: 5 23.6.2.2 Bayesian analysis fit.brm_titanic = brm(formula = survived ~ 1 + fare * sex, family = &quot;bernoulli&quot;, data = df.titanic, file = &quot;cache/brm_titanic&quot;, seed = 1) 23.6.3 3. Model evaluation 23.6.3.1 a) Did the inference work? fit.brm_titanic %&gt;% summary() Family: bernoulli Links: mu = logit Formula: survived ~ 1 + fare * sex Data: df.titanic (Number of observations: 891) Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1; total post-warmup draws = 4000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept 0.39 0.19 0.03 0.76 1.00 2010 2625 fare 0.02 0.01 0.01 0.03 1.00 1545 2124 sexmale -2.09 0.23 -2.54 -1.65 1.00 1754 1984 fare:sexmale -0.01 0.01 -0.02 -0.00 1.00 1479 2041 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). fit.brm_titanic %&gt;% plot() 23.6.3.2 b) Visualize model predictions pp_check(fit.brm_titanic, nsamples = 100) Warning: Argument &#39;nsamples&#39; is deprecated. Please use argument &#39;ndraws&#39; instead. Let’s visualize what the posterior predictive would have looked like for a linear model (instead of a logistic model). fit.brm_titanic_linear = brm(formula = survived ~ 1 + fare * sex, data = df.titanic, file = &quot;cache/brm_titanic_linear&quot;, seed = 1) pp_check(fit.brm_titanic_linear, nsamples = 100) Warning: Argument &#39;nsamples&#39; is deprecated. Please use argument &#39;ndraws&#39; instead. 23.6.4 4. Interpret the parameters fit.brm_titanic %&gt;% posterior_samples() %&gt;% select(-lp__) %&gt;% pivot_longer(cols = everything(), names_to = &quot;variable&quot;, values_to = &quot;value&quot;) %&gt;% ggplot(data = ., mapping = aes(y = variable, x = value)) + stat_intervalh() + scale_color_brewer() Warning: Method &#39;posterior_samples&#39; is deprecated. Please see ?as_draws for recommended alternatives. Warning: &#39;stat_intervalh&#39; is deprecated. Use &#39;stat_interval&#39; instead. See help(&quot;Deprecated&quot;) and help(&quot;tidybayes-deprecated&quot;). fit.brm_titanic %&gt;% parameters(centrality = &quot;mean&quot;, ci = 0.95) fit.brm_titanic %&gt;% ggpredict(terms = c(&quot;fare [0:500]&quot;, &quot;sex&quot;)) %&gt;% plot() 23.6.5 5. Test specific hypotheses Difference between men and women in survival? fit.brm_titanic %&gt;% emmeans(specs = pairwise ~ sex, type = &quot;response&quot;) NOTE: Results may be misleading due to involvement in interactions $emmeans sex response lower.HPD upper.HPD female 0.743 0.69 0.795 male 0.194 0.16 0.225 Point estimate displayed: median Results are back-transformed from the logit scale HPD interval probability: 0.95 $contrasts contrast odds.ratio lower.HPD upper.HPD female / male 12.1 8.39 16.6 Point estimate displayed: median Results are back-transformed from the log odds ratio scale HPD interval probability: 0.95 Difference in how fare affected the chances of survival for men and women? fit.brm_titanic %&gt;% emtrends(specs = pairwise ~ sex, var = &quot;fare&quot;) $emtrends sex fare.trend lower.HPD upper.HPD female 0.02083 0.01129 0.0316 male 0.00845 0.00385 0.0135 Point estimate displayed: median HPD interval probability: 0.95 $contrasts contrast estimate lower.HPD upper.HPD female - male 0.0124 0.000884 0.0232 Point estimate displayed: median HPD interval probability: 0.95 23.6.6 6. Report results df.model = add_linpred_draws(newdata = expand_grid(sex = c(&quot;female&quot;, &quot;male&quot;), fare = 0:500) %&gt;% mutate(sex = factor(sex, levels = c(&quot;female&quot;, &quot;male&quot;))), object = fit.brm_titanic, ndraws = 10) ggplot(data = df.titanic, mapping = aes(x = fare, y = survived, color = sex)) + geom_point(alpha = 0.1, size = 2) + geom_line(data = df.model %&gt;% filter(sex == &quot;male&quot;), aes(y = .linpred, group = .draw, color = sex)) + geom_line(data = df.model %&gt;% filter(sex == &quot;female&quot;), aes(y = .linpred, group = .draw, color = sex)) + scale_color_brewer(palette = &quot;Set1&quot;) 23.7 Politeness data (I’m skipping some of the steps of our recipe for Bayesian data analysis here.) 23.7.1 1. Visualize the data ggplot(data = df.politeness, mapping = aes(x = attitude, y = pitch, fill = gender, color = gender)) + geom_point(alpha = 0.2, position = position_jitter(width = 0.1, height = 0)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;, shape = 21, size = 1, color = &quot;black&quot;) Warning: Removed 1 rows containing non-finite values (stat_summary). Warning: Removed 1 rows containing missing values (geom_point). 23.7.2 2. Specify and fit the model 23.7.2.1 Frequentist analysis fit.lm_polite = lm(formula = pitch ~ gender * attitude, data = df.politeness) 23.7.2.2 Bayesian analysis fit.brm_polite = brm(formula = pitch ~ gender * attitude, data = df.politeness, file = &quot;cache/brm_polite&quot;, seed = 1) 23.7.3 5. Test specific hypotheses 23.7.3.1 Frequentist fit.lm_polite %&gt;% joint_tests() model term df1 df2 F.ratio p.value gender 1 79 191.028 &lt;.0001 attitude 1 79 6.170 0.0151 gender:attitude 1 79 1.029 0.3135 It looks like there are significant main effects of gender and attitude, but no interaction effect. Let’s check whether there is a difference in attitude separately for each gender: fit.lm_polite %&gt;% emmeans(specs = pairwise ~ attitude | gender) %&gt;% pluck(&quot;contrasts&quot;) gender = F: contrast estimate SE df t.ratio p.value inf - pol 27.4 11.0 79 2.489 0.0149 gender = M: contrast estimate SE df t.ratio p.value inf - pol 11.5 11.1 79 1.033 0.3048 There was a significant difference of attitude for female participants but not for male participants. 23.7.3.2 Bayesian Let’s whether there was a main effect of gender. # main effect of gender fit.brm_polite %&gt;% emmeans(specs = pairwise ~ gender) %&gt;% pluck(&quot;contrasts&quot;) NOTE: Results may be misleading due to involvement in interactions contrast estimate lower.HPD upper.HPD F - M 108 94.1 124 Point estimate displayed: median HPD interval probability: 0.95 Let’s take a look what the full posterior distribution over this contrast looks like: fit.brm_polite %&gt;% emmeans(specs = pairwise ~ gender) %&gt;% pluck(&quot;contrasts&quot;) %&gt;% gather_emmeans_draws() %&gt;% ggplot(mapping = aes(x = .value)) + stat_halfeye() NOTE: Results may be misleading due to involvement in interactions Looks neat! And let’s confirm that we really estimated the main effect here. Let’s fit a model that only has gender as a predictor, and then compare: fit.brm_polite_gender = brm(formula = pitch ~ 1 + gender, data = df.politeness, file = &quot;cache/brm_polite_gender&quot;, seed = 1) # using the gather_emmeans_draws to get means rather than medians fit.brm_polite %&gt;% emmeans(spec = pairwise ~ gender) %&gt;% pluck(&quot;contrasts&quot;) %&gt;% gather_emmeans_draws() %&gt;% mean_hdi() NOTE: Results may be misleading due to involvement in interactions # A tibble: 1 × 7 contrast .value .lower .upper .width .point .interval &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 F - M 108. 94.1 124. 0.95 mean hdi fit.brm_polite_gender %&gt;% fixef() %&gt;% as_tibble(rownames = &quot;term&quot;) # A tibble: 2 × 5 term Estimate Est.Error Q2.5 Q97.5 &lt;chr&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 Intercept 247. 5.68 236. 258. 2 genderM -108. 8.08 -124. -92.1 Yip, both of these methods give us the same result (the sign is flipped but that’s just because emmeans computed F-M, whereas the other method computed M-F)! Again, the emmeans() route is more convenient because we can more easily check for several main effects (and take a look at specific contrast, too). # main effect attitude fit.brm_polite %&gt;% emmeans(specs = pairwise ~ attitude) %&gt;% pluck(&quot;contrasts&quot;) NOTE: Results may be misleading due to involvement in interactions contrast estimate lower.HPD upper.HPD inf - pol 19.7 4.49 35.4 Point estimate displayed: median HPD interval probability: 0.95 # effect of attitude separately for each gender fit.brm_polite %&gt;% emmeans(specs = pairwise ~ attitude | gender) %&gt;% pluck(&quot;contrasts&quot;) gender = F: contrast estimate lower.HPD upper.HPD inf - pol 27.4 4.69 49.3 gender = M: contrast estimate lower.HPD upper.HPD inf - pol 11.6 -10.06 33.9 Point estimate displayed: median HPD interval probability: 0.95 # in case you want the means instead of medians fit.brm_polite %&gt;% emmeans(specs = pairwise ~ attitude | gender) %&gt;% pluck(&quot;contrasts&quot;) %&gt;% gather_emmeans_draws() %&gt;% mean_hdi() # A tibble: 2 × 8 contrast gender .value .lower .upper .width .point .interval &lt;fct&gt; &lt;fct&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;chr&gt; &lt;chr&gt; 1 inf - pol F 27.5 4.69 49.3 0.95 mean hdi 2 inf - pol M 11.6 -10.1 33.9 0.95 mean hdi Here is a way to visualize the contrasts: fit.brm_polite %&gt;% emmeans(specs = pairwise ~ attitude | gender) %&gt;% pluck(&quot;contrasts&quot;) %&gt;% gather_emmeans_draws() %&gt;% ggplot(aes(x = .value, y = gender, fill = stat(x &gt; 0))) + facet_wrap(~ contrast) + stat_halfeye(show.legend = F) + geom_vline(xintercept = 0, linetype = 2) + scale_fill_manual(values = c(&quot;gray80&quot;, &quot;skyblue&quot;)) Here is one way to check whether there was an interaction between attitude and gender (see this vignette for more info). fit.brm_polite %&gt;% emmeans(pairwise ~ attitude | gender) %&gt;% pluck(&quot;emmeans&quot;) %&gt;% contrast(interaction = c(&quot;consec&quot;), by = NULL) attitude_consec gender_consec estimate lower.HPD upper.HPD pol - inf M - F 16.2 -17.2 46.9 Point estimate displayed: median HPD interval probability: 0.95 23.8 Additional resources Tutorial on visualizing brms posteriors with tidybayes Hypothetical outcome plots Visual MCMC diagnostics Visualization of different MCMC algorithms Article describing the different inference algorithms 23.9 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 [4] purrr_0.3.4 readr_2.1.2 tidyr_1.2.0 [7] tibble_3.1.7 tidyverse_1.3.1 transformr_0.1.3 [10] parameters_0.17.0 gganimate_1.0.7 titanic_0.1.0 [13] ggeffects_1.1.2 emmeans_1.7.3 car_3.0-13 [16] carData_3.0-5 afex_1.1-1 lme4_1.1-29 [19] Matrix_1.4-1 modelr_0.1.8 bayesplot_1.9.0 [22] broom.mixed_0.2.9.4 GGally_2.1.2 ggplot2_3.3.6 [25] patchwork_1.1.1 brms_2.17.0 Rcpp_1.0.8.3 [28] tidybayes_3.0.2 janitor_2.1.0 kableExtra_1.3.4 [31] knitr_1.39 loaded via a namespace (and not attached): [1] utf8_1.2.2 tidyselect_1.1.2 htmlwidgets_1.5.4 [4] grid_4.1.2 lpSolve_5.6.15 munsell_0.5.0 [7] codetools_0.2-18 units_0.8-0 DT_0.22 [10] future_1.25.0 gifski_1.6.6-1 miniUI_0.1.1.1 [13] withr_2.5.0 Brobdingnag_1.2-7 colorspace_2.0-3 [16] highr_0.9 rstudioapi_0.13 stats4_4.1.2 [19] listenv_0.8.0 labeling_0.4.2 rstan_2.21.5 [22] bit64_4.0.5 farver_2.1.0 datawizard_0.4.0 [25] bridgesampling_1.1-2 coda_0.19-4 parallelly_1.31.1 [28] vctrs_0.4.1 generics_0.1.2 TH.data_1.1-1 [31] xfun_0.30 R6_2.5.1 markdown_1.1 [34] HDInterval_0.2.2 rstanarm_2.21.3 reshape_0.8.9 [37] assertthat_0.2.1 vroom_1.5.7 promises_1.2.0.1 [40] scales_1.2.0 nnet_7.3-17 multcomp_1.4-19 [43] gtable_0.3.0 globals_0.14.0 processx_3.5.3 [46] sandwich_3.0-1 rlang_1.0.2 systemfonts_1.0.4 [49] splines_4.1.2 broom_0.8.0 checkmate_2.1.0 [52] inline_0.3.19 yaml_2.3.5 reshape2_1.4.4 [55] abind_1.4-5 threejs_0.3.3 crosstalk_1.2.0 [58] backports_1.4.1 httpuv_1.6.5 Hmisc_4.7-0 [61] tensorA_0.36.2 tools_4.1.2 bookdown_0.26 [64] ellipsis_0.3.2 jquerylib_0.1.4 posterior_1.2.1 [67] RColorBrewer_1.1-3 proxy_0.4-26 ggridges_0.5.3 [70] plyr_1.8.7 base64enc_0.1-3 progress_1.2.2 [73] classInt_0.4-3 ps_1.7.0 prettyunits_1.1.1 [76] rpart_4.1.16 zoo_1.8-10 cluster_2.1.3 [79] haven_2.5.0 fs_1.5.2 furrr_0.3.0 [82] magrittr_2.0.3 data.table_1.14.2 ggdist_3.1.1 [85] lmerTest_3.1-3 reprex_2.0.1 colourpicker_1.1.1 [88] mvtnorm_1.1-3 matrixStats_0.62.0 hms_1.1.1 [91] shinyjs_2.1.0 mime_0.12 evaluate_0.15 [94] arrayhelpers_1.1-0 xtable_1.8-4 shinystan_2.6.0 [97] jpeg_0.1-9 readxl_1.4.0 gridExtra_2.3 [100] rstantools_2.2.0 compiler_4.1.2 KernSmooth_2.23-20 [103] crayon_1.5.1 minqa_1.2.4 StanHeaders_2.21.0-7 [106] htmltools_0.5.2 mgcv_1.8-40 tzdb_0.3.0 [109] later_1.3.0 Formula_1.2-4 RcppParallel_5.1.5 [112] lubridate_1.8.0 DBI_1.1.2 sjlabelled_1.2.0 [115] tweenr_1.0.2 dbplyr_2.1.1 MASS_7.3-57 [118] sf_1.0-7 boot_1.3-28 cli_3.3.0 [121] parallel_4.1.2 insight_0.17.0 igraph_1.3.1 [124] pkgconfig_2.0.3 foreign_0.8-82 numDeriv_2016.8-1.1 [127] xml2_1.3.3 svUnit_1.0.6 dygraphs_1.1.1.6 [130] svglite_2.1.0 bslib_0.3.1 webshot_0.5.3 [133] estimability_1.3 rvest_1.0.2 snakecase_0.11.0 [136] distributional_0.3.0 callr_3.7.0 digest_0.6.29 [139] cellranger_1.1.0 rmarkdown_2.14 htmlTable_2.4.0 [142] shiny_1.7.1 gtools_3.9.2 nloptr_2.0.0 [145] lifecycle_1.0.1 nlme_3.1-157 jsonlite_1.8.0 [148] viridisLite_0.4.0 fansi_1.0.3 pillar_1.7.0 [151] lattice_0.20-45 loo_2.5.1 fastmap_1.1.0 [154] httr_1.4.3 pkgbuild_1.3.1 survival_3.3-1 [157] glue_1.6.2 xts_0.12.1 bayestestR_0.12.1 [160] png_0.1-7 shinythemes_1.2.0 bit_4.0.4 [163] class_7.3-20 stringi_1.7.6 sass_0.4.1 [166] latticeExtra_0.6-29 e1071_1.7-9 "],["bayesian-data-analysis-3.html", "Chapter 24 Bayesian data analysis 3 24.1 Learning goals 24.2 Load packages and set plotting theme 24.3 Evidence for the null hypothesis 24.4 Dealing with heteroscedasticity 24.5 Zero-one inflated beta binomial model 24.6 Ordinal regression 24.7 Additional resources 24.8 Session info", " Chapter 24 Bayesian data analysis 3 24.1 Learning goals Evidence for null results. Only positive predictors. Dealing with unequal variance. Modeling slider data: Zero-one inflated beta binomial model. Modeling Likert scale data: Ordinal logistic regression. 24.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for making nice tables library(&quot;janitor&quot;) # for cleaning column names library(&quot;tidybayes&quot;) # tidying up results from Bayesian models library(&quot;brms&quot;) # Bayesian regression models with Stan library(&quot;patchwork&quot;) # for making figure panels library(&quot;GGally&quot;) # for pairs plot library(&quot;broom.mixed&quot;) # for tidy lmer results library(&quot;bayesplot&quot;) # for visualization of Bayesian model fits library(&quot;modelr&quot;) # for modeling functions library(&quot;lme4&quot;) # for linear mixed effects models library(&quot;afex&quot;) # for ANOVAs library(&quot;car&quot;) # for ANOVAs library(&quot;emmeans&quot;) # for linear contrasts library(&quot;ggeffects&quot;) # for help with logistic regressions library(&quot;titanic&quot;) # titanic dataset library(&quot;gganimate&quot;) # for animations library(&quot;parameters&quot;) # for getting parameters library(&quot;transformr&quot;) # for gganimate library(&quot;rstanarm&quot;) # for Bayesian models library(&quot;ggrepel&quot;) # for labels in ggplots library(&quot;scales&quot;) # for percent y-axis library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size # set rstan options rstan::rstan_options(auto_write = TRUE) options(mc.cores = parallel::detectCores()) opts_chunk$set(comment = &quot;&quot;, fig.show = &quot;hold&quot;) 24.3 Evidence for the null hypothesis See this tutorial and this paper (Wagenmakers et al. 2010) for more information. 24.3.1 Bayes factor 24.3.1.1 Fit the model Define a binomial model Give a uniform prior beta(1, 1) Get samples from the prior df.null = tibble(s = 6, k = 10) fit.brm_bayes = brm(s | trials(k) ~ 0 + Intercept, family = binomial(link = &quot;identity&quot;), prior = set_prior(&quot;beta(1, 1)&quot;, class = &quot;b&quot;, lb = 0, ub = 1), data = df.null, sample_prior = TRUE, cores = 4, file = &quot;cache/brm_bayes&quot;) 24.3.1.2 Visualize the results Visualize the prior and posterior samples: fit.brm_bayes %&gt;% posterior_samples(pars = &quot;b&quot;) %&gt;% pivot_longer(cols = everything()) %&gt;% ggplot(mapping = aes(x = value, fill = name)) + geom_density(alpha = 0.5) + scale_fill_brewer(palette = &quot;Set1&quot;) Warning: Method &#39;posterior_samples&#39; is deprecated. Please see ?as_draws for recommended alternatives. We test the H0: \\(\\theta = 0.5\\) versus the H1: \\(\\theta \\neq 0.5\\) using the Savage-Dickey Method, according to which we can compute the Bayes factor like so: \\(BF_{01} = \\frac{p(D|H_0)}{p(D|H_1)} = \\frac{p(\\theta = 0.5|D, H_1)}{p(\\theta = 0.5|H_1)}\\) fit.brm_bayes %&gt;% hypothesis(hypothesis = &quot;Intercept = 0.5&quot;) Hypothesis Tests for class b: Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio 1 (Intercept)-(0.5) = 0 0.08 0.14 -0.21 0.34 2.13 Post.Prob Star 1 0.68 --- &#39;CI&#39;: 90%-CI for one-sided and 95%-CI for two-sided hypotheses. &#39;*&#39;: For one-sided hypotheses, the posterior probability exceeds 95%; for two-sided hypotheses, the value tested against lies outside the 95%-CI. Posterior probabilities of point hypotheses assume equal prior probabilities. The result shows that the evidence ratio is in favor of the H0 with \\(BF_{01} = 2.22\\). This means that H0 is 2.2 more likely than H1 given the data. 24.3.2 LOO Another way to test different models is to compare them via approximate leave-one-out cross-validation. set.seed(1) df.loo = tibble(x = rnorm(n = 50), y = rnorm(n = 50)) # visualize ggplot(data = df.loo, mapping = aes(x = x, y = y)) + geom_point() # fit the frequentist model fit.lm_loo = lm(formula = y ~ 1 + x, data = df.loo) fit.lm_loo %&gt;% summary() Call: lm(formula = y ~ 1 + x, data = df.loo) Residuals: Min 1Q Median 3Q Max -1.92760 -0.66898 -0.00225 0.48768 2.34858 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.12190 0.13935 0.875 0.386 x -0.04555 0.16807 -0.271 0.788 Residual standard error: 0.9781 on 48 degrees of freedom Multiple R-squared: 0.001528, Adjusted R-squared: -0.01927 F-statistic: 0.07345 on 1 and 48 DF, p-value: 0.7875 # fit and compare bayesian models fit.brm_loo1 = brm(formula = y ~ 1, data = df.loo, seed = 1, file = &quot;cache/brm_loo1&quot;) fit.brm_loo2 = brm(formula = y ~ 1 + x, data = df.loo, seed = 1, file = &quot;cache/brm_loo2&quot;) fit.brm_loo1 = add_criterion(fit.brm_loo1, criterion = &quot;loo&quot;, file = &quot;cache/brm_loo1&quot;) fit.brm_loo2 = add_criterion(fit.brm_loo2, criterion = &quot;loo&quot;, file = &quot;cache/brm_loo2&quot;) loo_compare(fit.brm_loo1, fit.brm_loo2) elpd_diff se_diff fit.brm_loo1 0.0 0.0 fit.brm_loo2 -1.1 0.5 model_weights(fit.brm_loo1, fit.brm_loo2) fit.brm_loo1 fit.brm_loo2 9.999986e-01 1.351561e-06 24.4 Dealing with heteroscedasticity Let’s generate some fake developmental data where the variance in the data is greatest for young children, smaller for older children, and even smaller for adults: # make example reproducible set.seed(1) df.variance = tibble(group = rep(c(&quot;3yo&quot;, &quot;5yo&quot;, &quot;adults&quot;), each = 20), response = rnorm(n = 60, mean = rep(c(0, 5, 8), each = 20), sd = rep(c(3, 1.5, 0.3), each = 20))) 24.4.1 Visualize the data df.variance %&gt;% ggplot(aes(x = group, y = response)) + geom_jitter(height = 0, width = 0.1, alpha = 0.7) 24.4.2 Frequentist analysis 24.4.2.1 Fit the model fit.lm_variance = lm(formula = response ~ 1 + group, data = df.variance) fit.lm_variance %&gt;% summary() Call: lm(formula = response ~ 1 + group, data = df.variance) Residuals: Min 1Q Median 3Q Max -7.2157 -0.3613 0.0200 0.7001 4.2143 Coefficients: Estimate Std. Error t value Pr(&gt;|t|) (Intercept) 0.5716 0.3931 1.454 0.151 group5yo 4.4187 0.5560 7.948 8.4e-11 *** groupadults 7.4701 0.5560 13.436 &lt; 2e-16 *** --- Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 Residual standard error: 1.758 on 57 degrees of freedom Multiple R-squared: 0.762, Adjusted R-squared: 0.7537 F-statistic: 91.27 on 2 and 57 DF, p-value: &lt; 2.2e-16 fit.lm_variance %&gt;% glance() # A tibble: 1 × 12 r.squared adj.r.squared sigma statistic p.value df logLik AIC BIC &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 0.762 0.754 1.76 91.3 1.70e-18 2 -117. 243. 251. # … with 3 more variables: deviance &lt;dbl&gt;, df.residual &lt;int&gt;, nobs &lt;int&gt; 24.4.2.2 Visualize the model predictions set.seed(1) fit.lm_variance %&gt;% simulate() %&gt;% bind_cols(df.variance) %&gt;% ggplot(aes(x = group, y = sim_1)) + geom_jitter(height = 0, width = 0.1, alpha = 0.7) Notice how the model predicts that the variance is equal for each group. 24.4.3 Bayesian analysis While frequentist models (such as a linear regression) assume equality of variance, Bayesian models afford us with the flexibility of inferring both the parameter estimates of the groups (i.e. the means and differences between the means), as well as the variances. 24.4.3.1 Fit the model We define a multivariate model which tries to fit both the response as well as the variance sigma: fit.brm_variance = brm(formula = bf(response ~ group, sigma ~ group), data = df.variance, file = &quot;cache/brm_variance&quot;, seed = 1) summary(fit.brm_variance) Family: gaussian Links: mu = identity; sigma = log Formula: response ~ group sigma ~ group Data: df.variance (Number of observations: 60) Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1; total post-warmup draws = 4000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept -0.01 0.73 -1.41 1.51 1.01 1107 1072 sigma_Intercept 1.15 0.17 0.85 1.51 1.00 1991 1922 group5yo 5.18 0.77 3.60 6.65 1.00 1253 1327 groupadults 7.98 0.74 6.47 9.37 1.01 1110 1079 sigma_group5yo -1.05 0.24 -1.51 -0.57 1.00 2249 2420 sigma_groupadults -2.19 0.24 -2.66 -1.74 1.00 2171 2427 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). Notice that sigma is on the log scale. To get the standard deviations, we have to exponentiate the predictors, like so: fit.brm_variance %&gt;% tidy(parameters = &quot;^b_&quot;) %&gt;% filter(str_detect(term, &quot;sigma&quot;)) %&gt;% select(term, estimate) %&gt;% mutate(term = str_remove(term, &quot;b_sigma_&quot;)) %&gt;% pivot_wider(names_from = term, values_from = estimate) %&gt;% clean_names() %&gt;% mutate(across(-intercept, ~ exp(. + intercept))) %&gt;% mutate(intercept = exp(intercept)) Warning in tidy.brmsfit(., parameters = &quot;^b_&quot;): some parameter names contain underscores: term naming may be unreliable! # A tibble: 1 × 3 intercept group5yo groupadults &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt; 1 3.16 1.10 0.352 24.4.3.2 Visualize the model predictions df.variance %&gt;% expand(group) %&gt;% add_fitted_draws(fit.brm_variance, dpar = TRUE) %&gt;% select(group, .row, .draw, posterior = .value, mu, sigma) %&gt;% pivot_longer(cols = c(mu, sigma), names_to = &quot;index&quot;, values_to = &quot;value&quot;) %&gt;% ggplot(aes(x = value, y = group)) + geom_halfeyeh() + geom_vline(xintercept = 0, linetype = &quot;dashed&quot;) + facet_grid(cols = vars(index)) Warning: `fitted_draws` and `add_fitted_draws` are deprecated as their names were confusing. Use [add_]epred_draws() to get the expectation of the posterior predictive. Use [add_]linpred_draws() to get the distribution of the linear predictor. For example, you used [add_]fitted_draws(..., scale = &quot;response&quot;), which means you most likely want [add_]epred_draws(...). Warning: &#39;geom_halfeyeh&#39; is deprecated. Use &#39;stat_halfeye&#39; instead. See help(&quot;Deprecated&quot;) and help(&quot;tidybayes-deprecated&quot;). This plot shows what the posterior looks like for both mu (the inferred means), and for sigma (the inferred variances) for the different groups. set.seed(1) df.variance %&gt;% add_predicted_draws(object = fit.brm_variance, ndraws = 1) %&gt;% ggplot(aes(x = group, y = .prediction)) + geom_jitter(height = 0, width = 0.1, alpha = 0.7) 24.5 Zero-one inflated beta binomial model See this blog post. 24.6 Ordinal regression Check out the following two papers: Liddell and Kruschke (2018) Bürkner and Vuorre (2019) Let’s read in some movie ratings: df.movies = read_csv(file = &quot;data/MoviesData.csv&quot;) df.movies = df.movies %&gt;% pivot_longer(cols = n1:n5, names_to = &quot;stars&quot;, values_to = &quot;rating&quot;) %&gt;% mutate(stars = str_remove(stars,&quot;n&quot;), stars = as.numeric(stars)) df.movies = df.movies %&gt;% uncount(weights = rating) %&gt;% mutate(id = as.factor(ID)) %&gt;% filter(ID &lt;= 6) 24.6.1 Ordinal regression (assuming equal variance) 24.6.1.1 Fit the model fit.brm_ordinal = brm(formula = stars ~ 1 + id, family = cumulative(link = &quot;probit&quot;), data = df.movies, file = &quot;cache/brm_ordinal&quot;, seed = 1) summary(fit.brm_ordinal) Family: cumulative Links: mu = probit; disc = identity Formula: stars ~ 1 + id Data: df.movies (Number of observations: 21708) Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1; total post-warmup draws = 4000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept[1] -1.22 0.04 -1.30 -1.14 1.00 1483 2215 Intercept[2] -0.90 0.04 -0.98 -0.82 1.00 1448 1966 Intercept[3] -0.44 0.04 -0.52 -0.36 1.00 1419 1853 Intercept[4] 0.33 0.04 0.25 0.41 1.00 1380 1861 id2 0.84 0.06 0.72 0.96 1.00 2045 2455 id3 0.22 0.06 0.11 0.33 1.00 1973 2400 id4 0.33 0.04 0.25 0.42 1.00 1491 1937 id5 0.45 0.05 0.34 0.55 1.00 1734 2681 id6 0.76 0.04 0.68 0.84 1.00 1367 1851 Family Specific Parameters: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS disc 1.00 0.00 1.00 1.00 NA NA NA Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). 24.6.1.2 Visualizations 24.6.1.2.1 Model parameters The model infers the thresholds and the means of the Gaussian distributions in latent space. df.params = fit.brm_ordinal %&gt;% parameters(centrality = &quot;mean&quot;) %&gt;% as_tibble() %&gt;% clean_names() %&gt;% select(term = parameter, estimate = mean) ggplot(data = tibble(x = c(-3, 3)), mapping = aes(x = x)) + stat_function(fun = ~ dnorm(.), size = 1, color = &quot;black&quot;) + stat_function(fun = ~ dnorm(., mean = df.params %&gt;% filter(str_detect(term, &quot;id2&quot;)) %&gt;% pull(estimate)), size = 1, color = &quot;blue&quot;) + geom_vline(xintercept = df.params %&gt;% filter(str_detect(term, &quot;Intercept&quot;)) %&gt;% pull(estimate)) 24.6.1.2.2 MCMC inference fit.brm_ordinal %&gt;% plot(N = 9) fit.brm_ordinal %&gt;% pp_check(nsamples = 20) Warning: Argument &#39;nsamples&#39; is deprecated. Please use argument &#39;ndraws&#39; instead. 24.6.1.2.3 Model predictions conditional_effects(fit.brm_ordinal, effects = &quot;id&quot;, categorical = T) df.model = add_fitted_draws(newdata = expand_grid(id = 1:6), model = fit.brm_ordinal, n = 10) Warning: `fitted_draws` and `add_fitted_draws` are deprecated as their names were confusing. Use [add_]epred_draws() to get the expectation of the posterior predictive. Use [add_]linpred_draws() to get the distribution of the linear predictor. For example, you used [add_]fitted_draws(..., scale = &quot;response&quot;), which means you most likely want [add_]epred_draws(...). df.plot = df.movies %&gt;% count(id, stars) %&gt;% group_by(id) %&gt;% mutate(p = n / sum(n)) %&gt;% mutate(stars = as.factor(stars)) ggplot(data = df.plot, mapping = aes(x = stars, y = p)) + geom_col(color = &quot;black&quot;, fill = &quot;lightblue&quot;) + geom_point(data = df.model, mapping = aes(x = .category, y = .value), alpha = 0.3, position = position_jitter(width = 0.3)) + facet_wrap(~id, ncol = 6) 24.6.2 Gaussian regression (assuming equal variance) 24.6.2.1 Fit the model fit.brm_metric = brm(formula = stars ~ 1 + id, data = df.movies, file = &quot;cache/brm_metric&quot;, seed = 1) summary(fit.brm_metric) Family: gaussian Links: mu = identity; sigma = identity Formula: stars ~ 1 + id Data: df.movies (Number of observations: 21708) Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1; total post-warmup draws = 4000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept 3.77 0.04 3.70 3.85 1.00 1100 1471 id2 0.64 0.05 0.54 0.75 1.00 1532 2032 id3 0.20 0.05 0.10 0.30 1.00 1435 2045 id4 0.37 0.04 0.29 0.45 1.00 1146 1448 id5 0.30 0.05 0.20 0.39 1.00 1416 1784 id6 0.72 0.04 0.64 0.80 1.00 1130 1506 Family Specific Parameters: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS sigma 1.00 0.00 0.99 1.01 1.00 3221 2577 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). 24.6.2.2 Visualizations 24.6.2.2.1 Model predictions # get the predictions for each value of the Likert scale df.model = fit.brm_metric %&gt;% parameters(centrality = &quot;mean&quot;) %&gt;% as_tibble() %&gt;% select(term = Parameter, estimate = Mean) %&gt;% mutate(term = str_remove(term, &quot;b_&quot;)) %&gt;% pivot_wider(names_from = term, values_from = estimate) %&gt;% clean_names() %&gt;% mutate(across(.cols = id2:id6, .fns = ~ . + intercept)) %&gt;% rename_with(.fn = ~ c(str_c(&quot;mu_&quot;, 1:6), &quot;sigma&quot;)) %&gt;% pivot_longer(cols = everything(), names_to = c(&quot;parameter&quot;, &quot;movie&quot;), names_sep = &quot;_&quot;, values_to = &quot;value&quot;) %&gt;% pivot_wider(names_from = parameter, values_from = value) %&gt;% mutate(data = map(.x = mu, .f = ~ tibble(x = 1:5, y = dnorm(x, mean = .x)))) %&gt;% select(movie, data) %&gt;% unnest(c(data)) %&gt;% group_by(movie) %&gt;% mutate(y = y/sum(y)) %&gt;% ungroup() %&gt;% rename(id = movie) Warning: Expected 2 pieces. Missing pieces filled with `NA` in 1 rows [7]. # visualize the predictions df.plot = df.movies %&gt;% count(id, stars) %&gt;% group_by(id) %&gt;% mutate(p = n / sum(n)) %&gt;% mutate(stars = as.factor(stars)) ggplot(data = df.plot, mapping = aes(x = stars, y = p)) + geom_col(color = &quot;black&quot;, fill = &quot;lightblue&quot;) + geom_point(data = df.model, mapping = aes(x = x, y = y)) + facet_wrap(~id, ncol = 6) Warning: Removed 5 rows containing missing values (geom_point). 24.6.3 Oridnal regression (unequal variance) 24.6.3.1 Fit the model fit.brm_ordinal_variance = brm(formula = bf(stars ~ 1 + id) + lf(disc ~ 0 + id, cmc = FALSE), family = cumulative(link = &quot;probit&quot;), data = df.movies, file = &quot;cache/brm_ordinal_variance&quot;, seed = 1) summary(fit.brm_ordinal_variance) Family: cumulative Links: mu = probit; disc = log Formula: stars ~ 1 + id disc ~ 0 + id Data: df.movies (Number of observations: 21708) Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1; total post-warmup draws = 4000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept[1] -1.41 0.06 -1.53 -1.29 1.00 1395 2292 Intercept[2] -1.00 0.05 -1.10 -0.91 1.00 1906 2567 Intercept[3] -0.46 0.04 -0.54 -0.38 1.00 2645 2897 Intercept[4] 0.41 0.05 0.32 0.50 1.00 1358 1824 id2 2.70 0.32 2.13 3.38 1.00 1577 2220 id3 0.33 0.07 0.20 0.48 1.00 1795 2246 id4 0.36 0.05 0.26 0.46 1.00 1506 1917 id5 1.64 0.17 1.32 1.98 1.00 1568 2099 id6 0.86 0.06 0.74 0.97 1.00 1065 1919 disc_id2 -1.12 0.10 -1.32 -0.93 1.00 1557 2377 disc_id3 -0.23 0.06 -0.34 -0.12 1.00 1336 2071 disc_id4 -0.01 0.04 -0.10 0.07 1.00 927 1572 disc_id5 -1.09 0.07 -1.23 -0.95 1.00 1249 1966 disc_id6 -0.07 0.04 -0.15 0.01 1.00 856 1528 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). 24.6.3.2 Visualizations 24.6.3.2.1 Model parameters df.params = fit.brm_ordinal_variance %&gt;% tidy(parameters = &quot;^b_&quot;) %&gt;% select(term, estimate) %&gt;% mutate(term = str_remove(term, &quot;b_&quot;)) Warning in tidy.brmsfit(., parameters = &quot;^b_&quot;): some parameter names contain underscores: term naming may be unreliable! ggplot(data = tibble(x = c(-3, 3)), mapping = aes(x = x)) + stat_function(fun = ~ dnorm(.), size = 1, color = &quot;black&quot;) + stat_function(fun = ~ dnorm(., mean = 1, sd = 2), size = 1, color = &quot;blue&quot;) + geom_vline(xintercept = df.params %&gt;% filter(str_detect(term, &quot;Intercept&quot;)) %&gt;% pull(estimate)) 24.6.3.2.2 Model predictions df.model = add_fitted_draws(newdata = expand_grid(id = 1:6), model = fit.brm_ordinal_variance, n = 10) Warning: `fitted_draws` and `add_fitted_draws` are deprecated as their names were confusing. Use [add_]epred_draws() to get the expectation of the posterior predictive. Use [add_]linpred_draws() to get the distribution of the linear predictor. For example, you used [add_]fitted_draws(..., scale = &quot;response&quot;), which means you most likely want [add_]epred_draws(...). df.plot = df.movies %&gt;% count(id, stars) %&gt;% group_by(id) %&gt;% mutate(p = n / sum(n)) %&gt;% mutate(stars = as.factor(stars)) ggplot(data = df.plot, mapping = aes(x = stars, y = p)) + geom_col(color = &quot;black&quot;, fill = &quot;lightblue&quot;) + geom_point(data = df.model, mapping = aes(x = .category, y = .value), alpha = 0.3, position = position_jitter(width = 0.3)) + facet_wrap(~id, ncol = 6) 24.6.4 Gaussian regression (unequal variance) 24.6.4.1 Fit the model fit.brm_metric_variance = brm(formula = bf(stars ~ 1 + id, sigma ~ 1 + id), data = df.movies, file = &quot;cache/brm_metric_variance&quot;, seed = 1) summary(fit.brm_metric_variance) Family: gaussian Links: mu = identity; sigma = log Formula: stars ~ 1 + id sigma ~ 1 + id Data: df.movies (Number of observations: 21708) Draws: 4 chains, each with iter = 2000; warmup = 1000; thin = 1; total post-warmup draws = 4000 Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept 3.77 0.05 3.68 3.86 1.00 1562 1853 sigma_Intercept 0.20 0.03 0.15 0.26 1.00 1194 1613 id2 0.64 0.07 0.51 0.77 1.00 2058 2521 id3 0.20 0.06 0.08 0.33 1.00 2029 2337 id4 0.37 0.05 0.27 0.47 1.00 1660 2146 id5 0.30 0.06 0.17 0.42 1.00 2033 2370 id6 0.72 0.05 0.63 0.81 1.00 1601 1961 sigma_id2 0.02 0.04 -0.05 0.09 1.00 1727 2141 sigma_id3 0.03 0.04 -0.04 0.10 1.00 1436 1982 sigma_id4 -0.14 0.03 -0.19 -0.08 1.00 1266 1581 sigma_id5 0.20 0.03 0.14 0.27 1.00 1505 2072 sigma_id6 -0.35 0.03 -0.40 -0.29 1.00 1210 1576 Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS and Tail_ESS are effective sample size measures, and Rhat is the potential scale reduction factor on split chains (at convergence, Rhat = 1). 24.6.4.2 Visualizations 24.6.4.2.1 Model predictions df.model = fit.brm_metric_variance %&gt;% tidy(parameters = &quot;^b_&quot;) %&gt;% select(term, estimate) %&gt;% mutate(term = str_remove(term, &quot;b_&quot;)) %&gt;% pivot_wider(names_from = term, values_from = estimate) %&gt;% clean_names() %&gt;% mutate(across(.cols = c(id2:id6), .fns = ~ . + intercept)) %&gt;% mutate(across(.cols = contains(&quot;sigma&quot;), .fns = ~ 1/exp(.))) %&gt;% mutate(across(.cols = c(sigma_id2:sigma_id5), .fns = ~ . + sigma_intercept)) %&gt;% set_names(c(&quot;mu_1&quot;, &quot;sigma_1&quot;, str_c(&quot;mu_&quot;, 2:6), str_c(&quot;sigma_&quot;, 2:6))) %&gt;% pivot_longer(cols = everything(), names_to = c(&quot;parameter&quot;, &quot;movie&quot;), names_sep = &quot;_&quot;, values_to = &quot;value&quot;) %&gt;% pivot_wider(names_from = parameter, values_from = value) %&gt;% mutate(data = map2(.x = mu, .y = sigma, .f = ~ tibble(x = 1:5, y = dnorm(x, mean = .x, sd = .y)))) %&gt;% select(movie, data) %&gt;% unnest(c(data)) %&gt;% group_by(movie) %&gt;% mutate(y = y/sum(y)) %&gt;% ungroup() %&gt;% rename(id = movie) Warning in tidy.brmsfit(., parameters = &quot;^b_&quot;): some parameter names contain underscores: term naming may be unreliable! df.plot = df.movies %&gt;% count(id, stars) %&gt;% group_by(id) %&gt;% mutate(p = n / sum(n)) %&gt;% mutate(stars = as.factor(stars)) ggplot(data = df.plot, mapping = aes(x = stars, y = p)) + geom_col(color = &quot;black&quot;, fill = &quot;lightblue&quot;) + geom_point(data = df.model, mapping = aes(x = x, y = y)) + facet_wrap(~id, ncol = 6) 24.6.5 Model comparison # ordinal regression with equal variance fit.brm_ordinal = add_criterion(fit.brm_ordinal, criterion = &quot;loo&quot;, file = &quot;cache/brm_ordinal&quot;) # Gaussian regression with equal variance fit.brm_ordinal_variance = add_criterion(fit.brm_ordinal_variance, criterion = &quot;loo&quot;, file = &quot;cache/brm_ordinal_variance&quot;) loo_compare(fit.brm_ordinal, fit.brm_ordinal_variance) elpd_diff se_diff fit.brm_ordinal_variance 0.0 0.0 fit.brm_ordinal -340.2 27.0 24.7 Additional resources Tutorial on visualizing brms posteriors with tidybayes Hypothetical outcome plots Visual MCMC diagnostics Visualiztion of different MCMC algorithms For additional resources, I highly recommend the brms and tidyverse implementations of the Statistical rethinking book (McElreath 2020), as well as of the Doing Bayesian Data analysis book (Kruschke 2014), by Solomon Kurz (Kurz 2020, 2022). 24.8 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Big Sur 10.16 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 [4] purrr_0.3.4 readr_2.1.2 tidyr_1.2.0 [7] tibble_3.1.7 tidyverse_1.3.1 scales_1.2.0 [10] ggrepel_0.9.1 rstanarm_2.21.3 transformr_0.1.3 [13] parameters_0.17.0 gganimate_1.0.7 titanic_0.1.0 [16] ggeffects_1.1.2 emmeans_1.7.3 car_3.0-13 [19] carData_3.0-5 afex_1.1-1 lme4_1.1-29 [22] Matrix_1.4-1 modelr_0.1.8 bayesplot_1.9.0 [25] broom.mixed_0.2.9.4 GGally_2.1.2 ggplot2_3.3.6 [28] patchwork_1.1.1 brms_2.17.0 Rcpp_1.0.8.3 [31] tidybayes_3.0.2 janitor_2.1.0 kableExtra_1.3.4 [34] knitr_1.39 loaded via a namespace (and not attached): [1] utf8_1.2.2 tidyselect_1.1.2 htmlwidgets_1.5.4 [4] grid_4.1.2 lpSolve_5.6.15 munsell_0.5.0 [7] codetools_0.2-18 units_0.8-0 DT_0.22 [10] future_1.25.0 gifski_1.6.6-1 miniUI_0.1.1.1 [13] withr_2.5.0 Brobdingnag_1.2-7 colorspace_2.0-3 [16] highr_0.9 rstudioapi_0.13 stats4_4.1.2 [19] listenv_0.8.0 labeling_0.4.2 rstan_2.21.5 [22] bit64_4.0.5 farver_2.1.0 datawizard_0.4.0 [25] bridgesampling_1.1-2 coda_0.19-4 parallelly_1.31.1 [28] vctrs_0.4.1 generics_0.1.2 TH.data_1.1-1 [31] xfun_0.30 R6_2.5.1 markdown_1.1 [34] reshape_0.8.9 assertthat_0.2.1 vroom_1.5.7 [37] promises_1.2.0.1 multcomp_1.4-19 gtable_0.3.0 [40] globals_0.14.0 processx_3.5.3 sandwich_3.0-1 [43] rlang_1.0.2 systemfonts_1.0.4 splines_4.1.2 [46] broom_0.8.0 checkmate_2.1.0 inline_0.3.19 [49] yaml_2.3.5 reshape2_1.4.4 abind_1.4-5 [52] threejs_0.3.3 crosstalk_1.2.0 backports_1.4.1 [55] httpuv_1.6.5 tensorA_0.36.2 tools_4.1.2 [58] bookdown_0.26 ellipsis_0.3.2 jquerylib_0.1.4 [61] posterior_1.2.1 RColorBrewer_1.1-3 proxy_0.4-26 [64] ggridges_0.5.3 plyr_1.8.7 base64enc_0.1-3 [67] progress_1.2.2 classInt_0.4-3 ps_1.7.0 [70] prettyunits_1.1.1 zoo_1.8-10 haven_2.5.0 [73] fs_1.5.2 furrr_0.3.0 magrittr_2.0.3 [76] ggdist_3.1.1 lmerTest_3.1-3 reprex_2.0.1 [79] colourpicker_1.1.1 mvtnorm_1.1-3 matrixStats_0.62.0 [82] hms_1.1.1 shinyjs_2.1.0 mime_0.12 [85] evaluate_0.15 arrayhelpers_1.1-0 xtable_1.8-4 [88] shinystan_2.6.0 readxl_1.4.0 gridExtra_2.3 [91] rstantools_2.2.0 compiler_4.1.2 KernSmooth_2.23-20 [94] crayon_1.5.1 minqa_1.2.4 StanHeaders_2.21.0-7 [97] htmltools_0.5.2 tzdb_0.3.0 later_1.3.0 [100] RcppParallel_5.1.5 lubridate_1.8.0 DBI_1.1.2 [103] tweenr_1.0.2 dbplyr_2.1.1 MASS_7.3-57 [106] sf_1.0-7 boot_1.3-28 cli_3.3.0 [109] parallel_4.1.2 insight_0.17.0 igraph_1.3.1 [112] pkgconfig_2.0.3 numDeriv_2016.8-1.1 xml2_1.3.3 [115] svUnit_1.0.6 dygraphs_1.1.1.6 svglite_2.1.0 [118] bslib_0.3.1 webshot_0.5.3 estimability_1.3 [121] rvest_1.0.2 snakecase_0.11.0 distributional_0.3.0 [124] callr_3.7.0 digest_0.6.29 cellranger_1.1.0 [127] rmarkdown_2.14 shiny_1.7.1 gtools_3.9.2 [130] nloptr_2.0.0 lifecycle_1.0.1 nlme_3.1-157 [133] jsonlite_1.8.0 viridisLite_0.4.0 fansi_1.0.3 [136] pillar_1.7.0 lattice_0.20-45 loo_2.5.1 [139] fastmap_1.1.0 httr_1.4.3 pkgbuild_1.3.1 [142] survival_3.3-1 glue_1.6.2 xts_0.12.1 [145] bayestestR_0.12.1 shinythemes_1.2.0 bit_4.0.4 [148] class_7.3-20 stringi_1.7.6 sass_0.4.1 [151] e1071_1.7-9 References "],["model-assumptions.html", "Chapter 25 Model assumptions 25.1 Learning goals 25.2 Load packages and set plotting theme 25.3 Model assumptions and what to do if they are violated 25.4 Additional resources 25.5 Session info", " Chapter 25 Model assumptions 25.1 Learning goals Review model assumptions. Explore how to test for model assumptions. What to do if model assumptions aren’t met. 25.2 Load packages and set plotting theme library(&quot;knitr&quot;) # for knitting RMarkdown library(&quot;kableExtra&quot;) # for nice RMarkdown tables library(&quot;tidybayes&quot;) # tidying up results from Bayesian models library(&quot;lme4&quot;) # for linear mixed effects models library(&quot;brms&quot;) # Bayesian regression models with Stan library(&quot;car&quot;) # for bootstrapping regression models library(&quot;broom&quot;) # for tidy regression results library(&quot;janitor&quot;) # for cleaning up variable names library(&quot;patchwork&quot;) # for figure panels library(&quot;ggeffects&quot;) # for visualizing estimated marginal means library(&quot;stargazer&quot;) # for latex regression tables library(&quot;sjPlot&quot;) # for nice RMarkdown regression tables library(&quot;xtable&quot;) # for latex tables library(&quot;ggrepel&quot;) # for smart text annotation in ggplot library(&quot;tidyverse&quot;) # for wrangling, plotting, etc. theme_set(theme_classic() + #set the theme theme(text = element_text(size = 20))) #set the default text size # set rstan options rstan::rstan_options(auto_write = TRUE) options(mc.cores = parallel::detectCores()) 25.3 Model assumptions and what to do if they are violated “Regression diagnostics are methods for determining whether a fitted regression model adequately represents the data.” (p. 385) (Fox and Weisberg 2018) 25.3.1 Influential data points Because linear regression models are fitted by minimizing the squared error between prediction and data, the results can be strongly influenced by outliers. There are a number of ways of checking for outliers. 25.3.1.1 Leverage: Hat-values Data points that are far from the center of the predictor space have potentially greater influence on the results – these points have high leverage. hat-values are a way of characterizing how much influence individual data points have. df.credit = read_csv(&quot;data/credit.csv&quot;) %&gt;% clean_names() fit.credit = lm(formula = balance ~ income, data = df.credit) # fit model without the data point of interest fit.credit2 = update(fit.credit, data = df.credit %&gt;% filter(x1 != 324)) res_with_outlier = fit.credit %&gt;% augment() %&gt;% filter(row_number() == 324) %&gt;% pull(.resid) res_without_outlier = fit.credit2 %&gt;% augment(newdata = df.credit) %&gt;% mutate(.resid = balance - .fitted) %&gt;% filter(row_number() == 324) %&gt;% pull(.resid) hat1 = 1 - (res_with_outlier/res_without_outlier) %&gt;% round(3) hat2 = fit.credit %&gt;% augment() %&gt;% filter(row_number() == 324) %&gt;% pull(.hat) %&gt;% round(3) print(str_c(&quot;hat1: &quot;, hat1)) ## [1] &quot;hat1: 0.041&quot; print(str_c(&quot;hat2: &quot;, hat2)) ## [1] &quot;hat2: 0.041&quot; Cook’s distance is defined as \\[D_i = \\frac{e^2_{Si}}{k + 1} \\times \\frac{h_i}{1-h_1}\\], where \\(e^2_{Si}\\) is the squared standardized residual, \\(k\\) is the number of coefficients in the model (excluding the intercept), and \\(h_i\\) is the hat-value for case \\(i\\). Let’s double check here: fit.credit %&gt;% augment() %&gt;% mutate(cook = ((.std.resid^2)/(2 + 1)) * (.hat/(1 - .hat))) %&gt;% select(contains(&quot;cook&quot;)) %&gt;% head(10) ## # A tibble: 10 × 2 ## .cooksd cook ## &lt;dbl&gt; &lt;dbl&gt; ## 1 0.000000169 0.000000113 ## 2 0.00000706 0.00000471 ## 3 0.00264 0.00176 ## 4 0.00257 0.00171 ## 5 0.000530 0.000353 ## 6 0.00265 0.00177 ## 7 0.000324 0.000216 ## 8 0.000441 0.000294 ## 9 0.0000457 0.0000304 ## 10 0.00529 0.00353 Looking good! fit.credit %&gt;% augment() %&gt;% ggplot(aes(x = .hat, y = .std.resid)) + geom_point() + geom_line(aes(y = .cooksd), color = &quot;red&quot;) 25.3.1.1.1 Toy example Generate some data with an outlier. set.seed(1) df.hat = tibble(x = runif(n = 5), y = 10 * x + rnorm(n = 5, sd = 2)) %&gt;% bind_rows(tibble(x = 0.7, y = 15)) %&gt;% mutate(index = 1:n()) Illustrate the hat-values and cook’s distance. fit.hat = lm(formula = y ~ x, data = df.hat) fit.hat %&gt;% augment() %&gt;% mutate(index = 1:n()) %&gt;% ggplot(aes(x = .hat, y = .std.resid)) + geom_point() + geom_line(aes(y = .cooksd), color = &quot;red&quot;) + geom_text(aes(label = index), nudge_y = -0.2) Illustrate what the regression line looks like when all points are fit vs. one of the points is excluded. ggplot(data = df.hat, mapping = aes(x = x, y = y)) + geom_point(size = 2) + geom_smooth(method = &quot;lm&quot;, se = F, color = &quot;blue&quot;) + geom_smooth(data = df.hat %&gt;% filter(index != 6), method = &quot;lm&quot;, se = F, color = &quot;red&quot;) ## `geom_smooth()` using formula &#39;y ~ x&#39; ## `geom_smooth()` using formula &#39;y ~ x&#39; Summary of each observation. fit.hat %&gt;% augment() %&gt;% clean_names() %&gt;% kable(digits = 2) %&gt;% kable_styling() y x fitted resid hat sigma cooksd std_resid 5.20 0.27 3.58 1.62 0.32 5.00 0.05 0.44 4.55 0.37 4.67 -0.12 0.21 5.12 0.00 -0.03 2.65 0.57 6.72 -4.07 0.18 4.42 0.11 -1.01 7.22 0.91 10.13 -2.91 0.61 4.37 0.84 -1.05 1.43 0.20 2.93 -1.51 0.41 5.00 0.07 -0.44 15.00 0.70 8.01 6.99 0.27 1.98 0.63 1.84 Compute cook’s distance fit.hat_with = lm(formula = y ~ x, data = df.hat) fit.hat_without = lm(formula = y ~ x, data = df.hat %&gt;% filter(index != 6)) residual_without = fit.hat_without %&gt;% augment(newdata = df.hat) %&gt;% clean_names() %&gt;% mutate(resid = y - fitted) %&gt;% filter(row_number() == 6) %&gt;% pull(resid) residual_with = fit.hat %&gt;% augment() %&gt;% clean_names() %&gt;% filter(row_number() == 6) %&gt;% pull(resid) hat = 1 - (residual_with/residual_without) hat ## [1] 0.270516 25.3.2 Linear and additive df.car = mtcars df.car %&gt;% head(6) %&gt;% kable(digits = 2) %&gt;% kable_styling() mpg cyl disp hp drat wt qsec vs am gear carb Mazda RX4 21.0 6 160 110 3.90 2.62 16.46 0 1 4 4 Mazda RX4 Wag 21.0 6 160 110 3.90 2.88 17.02 0 1 4 4 Datsun 710 22.8 4 108 93 3.85 2.32 18.61 1 1 4 1 Hornet 4 Drive 21.4 6 258 110 3.08 3.21 19.44 1 0 3 1 Hornet Sportabout 18.7 8 360 175 3.15 3.44 17.02 0 0 3 2 Valiant 18.1 6 225 105 2.76 3.46 20.22 1 0 3 1 fit.car = lm(formula = mpg ~ 1 + hp, data = df.car) ggplot(data = df.car, mapping = aes(x = hp, y = mpg)) + geom_smooth(method = &quot;lm&quot;) + geom_smooth(color = &quot;red&quot;, se = F) + geom_point() ## `geom_smooth()` using formula &#39;y ~ x&#39; ## `geom_smooth()` using method = &#39;loess&#39; and formula &#39;y ~ x&#39; Residual plot fit.car %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = fitted, y = resid)) + geom_hline(yintercept = 0, linetype = 2) + geom_point() + geom_smooth(color = &quot;red&quot;, se = F) Include a squared predictor ggplot(data = df.car, mapping = aes(x = hp, y = mpg)) + geom_smooth(method = &quot;lm&quot;, formula = y ~ 1 + x + I(x^2)) + geom_point() fit.car2 = lm(formula = mpg ~ 1 + hp + I(hp^2), data = df.car) fit.car2 %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = fitted, y = resid)) + geom_hline(yintercept = 0, linetype = 2) + geom_point() + geom_smooth(color = &quot;red&quot;, se = F) ## `geom_smooth()` using method = &#39;loess&#39; and formula &#39;y ~ x&#39; 25.3.3 Normally distributed residuals Let’s look at the residuals for the credit card model. fit.credit %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = fitted, y = resid)) + geom_point() This plot helps assess whether there is homogeneity of variance. Overall, the residual plot looks pretty ok. The diagonal points in the bottom left of th plot arise because credit card balance is not an unbounded variable, and some of the people have a credit card balance of 0. We can also check whether the residuals are normally distributed by plotting a density of the residuals, and a quantile quantile plot. df.plot = fit.credit %&gt;% augment() %&gt;% clean_names() p1 = ggplot(data = df.plot, mapping = aes(x = resid)) + geom_density() + labs(title = &quot;Density plot&quot;) p2 = ggplot(data = df.plot, mapping = aes(sample = scale(resid))) + geom_qq_line() + geom_qq() + labs(title = &quot;QQ plot&quot;, x = &quot;theoretical&quot;, y = &quot;standardized residuals&quot;) p1 + p2 The residuals aren’t really normally distributed. As both the density and the QQ plot show, residuals with low/negative values are more frequent than residuals with high/positive values. 25.3.3.1 Transforming the outcome variable When the residuals aren’t normally distributed and/or when the variance is not homogeneous, one option is to transform some of the variables. 25.3.3.1.1 Logarithmic transform df.un = UN %&gt;% clean_names() %&gt;% drop_na(infant_mortality, ppgdp) df.un %&gt;% head(5) %&gt;% kable(digits = 2) %&gt;% kable_styling() region group fertility ppgdp life_exp_f pct_urban infant_mortality Afghanistan Asia other 5.97 499.0 49.49 23 124.53 Albania Europe other 1.52 3677.2 80.40 53 16.56 Algeria Africa africa 2.14 4473.0 75.00 67 21.46 Angola Africa africa 5.14 4321.9 53.17 59 96.19 Argentina Latin Amer other 2.17 9162.1 79.89 93 12.34 The linear model fit (blue) versus the “loess” (local regression) fit (red). ggplot(data = df.un, mapping = aes(x = ppgdp, y = infant_mortality)) + geom_point() + geom_smooth(method = &quot;lm&quot;, aes(color = &quot;lm&quot;), fill = &quot;blue&quot;, alpha = 0.1) + geom_smooth(aes(color = &quot;loess&quot;), fill = &quot;red&quot;, alpha = 0.1) + scale_color_manual(values = c(&quot;blue&quot;, &quot;red&quot;)) + theme(legend.title = element_blank(), legend.position = c(1, 1), legend.justification = c(1, 1)) + guides(color = guide_legend(override.aes = list(fill = c(&quot;red&quot;, &quot;blue&quot;)), reverse = T)) Densities of the untransformed and log-transformed variables. p1 = ggplot(data = df.un, mapping = aes(x = infant_mortality)) + geom_density() # log transformed p2 = ggplot(data = df.un, mapping = aes(x = log(infant_mortality))) + geom_density() p3 = ggplot(data = df.un, mapping = aes(x = ppgdp)) + geom_density() # log transformed p4 = ggplot(data = df.un, mapping = aes(x = log(ppgdp))) + geom_density() p1 + p2 + p3 + p4 + plot_layout(nrow = 2) Fitting different models with / without transformations. fit.mortality1 = lm(formula = infant_mortality ~ ppgdp, data = df.un) fit.mortality2 = lm(formula = log(infant_mortality) ~ log(ppgdp), data = df.un) fit.mortality3 = lm(formula = log(infant_mortality) ~ ppgdp, data = df.un) fit.mortality4 = lm(formula = infant_mortality ~ log(ppgdp), data = df.un) summary(fit.mortality1) ## ## Call: ## lm(formula = infant_mortality ~ ppgdp, data = df.un) ## ## Residuals: ## Min 1Q Median 3Q Max ## -31.48 -18.65 -8.59 10.86 83.59 ## ## Coefficients: ## Estimate Std. Error t value Pr(&gt;|t|) ## (Intercept) 41.3780016 2.2157454 18.675 &lt; 2e-16 *** ## ppgdp -0.0008656 0.0001041 -8.312 1.73e-14 *** ## --- ## Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 ## ## Residual standard error: 25.13 on 191 degrees of freedom ## Multiple R-squared: 0.2656, Adjusted R-squared: 0.2618 ## F-statistic: 69.08 on 1 and 191 DF, p-value: 1.73e-14 summary(fit.mortality2) ## ## Call: ## lm(formula = log(infant_mortality) ~ log(ppgdp), data = df.un) ## ## Residuals: ## Min 1Q Median 3Q Max ## -1.16789 -0.36738 -0.02351 0.24544 2.43503 ## ## Coefficients: ## Estimate Std. Error t value Pr(&gt;|t|) ## (Intercept) 8.10377 0.21087 38.43 &lt;2e-16 *** ## log(ppgdp) -0.61680 0.02465 -25.02 &lt;2e-16 *** ## --- ## Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 ## ## Residual standard error: 0.5281 on 191 degrees of freedom ## Multiple R-squared: 0.7662, Adjusted R-squared: 0.765 ## F-statistic: 625.9 on 1 and 191 DF, p-value: &lt; 2.2e-16 summary(fit.mortality3) ## ## Call: ## lm(formula = log(infant_mortality) ~ ppgdp, data = df.un) ## ## Residuals: ## Min 1Q Median 3Q Max ## -1.61611 -0.48094 -0.07858 0.53930 2.17745 ## ## Coefficients: ## Estimate Std. Error t value Pr(&gt;|t|) ## (Intercept) 3.479e+00 6.537e-02 53.23 &lt;2e-16 *** ## ppgdp -4.595e-05 3.072e-06 -14.96 &lt;2e-16 *** ## --- ## Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 ## ## Residual standard error: 0.7413 on 191 degrees of freedom ## Multiple R-squared: 0.5394, Adjusted R-squared: 0.537 ## F-statistic: 223.7 on 1 and 191 DF, p-value: &lt; 2.2e-16 summary(fit.mortality4) ## ## Call: ## lm(formula = infant_mortality ~ log(ppgdp), data = df.un) ## ## Residuals: ## Min 1Q Median 3Q Max ## -38.239 -11.609 -2.829 8.122 82.183 ## ## Coefficients: ## Estimate Std. Error t value Pr(&gt;|t|) ## (Intercept) 155.7698 7.2431 21.51 &lt;2e-16 *** ## log(ppgdp) -14.8617 0.8468 -17.55 &lt;2e-16 *** ## --- ## Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 ## ## Residual standard error: 18.14 on 191 degrees of freedom ## Multiple R-squared: 0.6172, Adjusted R-squared: 0.6152 ## F-statistic: 308 on 1 and 191 DF, p-value: &lt; 2.2e-16 Diagnostics plots for the model without transformed variables. fit.mortality1 %&gt;% plot() Residual plot using ggplot. fit.mortality1 %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = fitted, y = resid)) + geom_hline(yintercept = 0, linetype = 2) + geom_point() + geom_smooth(color = &quot;red&quot;, se = F) ## `geom_smooth()` using method = &#39;loess&#39; and formula &#39;y ~ x&#39; Diagnostic plots for the log-log transformed model. fit.mortality2 %&gt;% plot() Model fit. ggplot(data = df.un, mapping = aes(x = log(ppgdp), y = log(infant_mortality))) + geom_point() + geom_smooth(method = &quot;lm&quot;, color = &quot;blue&quot;, fill = &quot;blue&quot;, alpha = 0.1) Illustration of the model predictions in the original scale. fit.mortality2 %&gt;% ggpredict(terms = &quot;ppgdp&quot;) ## Model has log-transformed response. Back-transforming predictions to original response scale. Standard errors are still on the log-scale. ## # Predicted values of infant_mortality ## ## ppgdp | Predicted | 95% CI ## --------------------------------- ## 0 | Inf | ## 15000 | 8.78 | [7.99, 9.65] ## 25000 | 6.41 | [5.73, 7.16] ## 40000 | 4.80 | [4.21, 5.46] ## 55000 | 3.94 | [3.42, 4.54] ## 70000 | 3.40 | [2.92, 3.95] ## 85000 | 3.01 | [2.57, 3.54] ## 110000 | 2.57 | [2.16, 3.05] fit.mortality2 %&gt;% ggpredict(terms = &quot;ppgdp [exp]&quot;) %&gt;% plot() ## Model has log-transformed response. Back-transforming predictions to original response scale. Standard errors are still on the log-scale. Model predictions for models with multiple predictors. # with log transforms fit.mortality5 = lm(formula = log(infant_mortality) ~ log(ppgdp) + group, data = df.un) # without log transforms fit.mortality6 = lm(formula = infant_mortality ~ ppgdp + group, data = df.un) p1 = ggpredict(fit.mortality5, terms = c(&quot;ppgdp [exp]&quot;, &quot;group&quot;)) %&gt;% plot() + labs(title = &quot;Prediction with log transform&quot;) + coord_cartesian(xlim = c(0, 20000)) p2 = ggpredict(fit.mortality6, terms = c(&quot;ppgdp&quot;, &quot;group&quot;)) %&gt;% plot() + labs(title = &quot;Prediction without log transform&quot;) + coord_cartesian(xlim = c(0, 20000)) p1 + p2 25.3.4 Non-parametric tests 25.3.4.1 Mann-Whitney df.ttest = tibble(group1 = rnorm(n = 20, mean = 10, sd = 1), group2 = rnorm(n = 20, mean = 8, sd = 3)) %&gt;% pivot_longer(cols = everything()) %&gt;% mutate(participant = 1:n()) ggplot(data = df.ttest, mapping = aes(x = name, y = value)) + geom_point(alpha = 0.3, position = position_jitter(width = 0.1)) + stat_summary(fun.data = &quot;mean_cl_boot&quot;) t.test(formula = value ~ name, data = df.ttest) ## ## Welch Two Sample t-test ## ## data: value by name ## t = 3.3, df = 22.304, p-value = 0.003221 ## alternative hypothesis: true difference in means between group group1 and group group2 is not equal to 0 ## 95 percent confidence interval: ## 0.8300203 3.6319540 ## sample estimates: ## mean in group group1 mean in group group2 ## 10.035302 7.804315 wilcox.test(formula = value ~ name, data = df.ttest) ## ## Wilcoxon rank sum exact test ## ## data: value by name ## W = 287, p-value = 0.01809 ## alternative hypothesis: true location shift is not equal to 0 25.3.5 Bootstrapping regressions This section is based on this post here. # make reproducible set.seed(1) n = 250 df.turkey = tibble(turkey_time = runif(n = n, min = 0, max = 50), nap_time = 500 + turkey_time ^ 2 + rnorm(n, sd = 16)) Visualize the data ggplot(data = df.turkey, mapping = aes(x = turkey_time, y = nap_time)) + geom_smooth(method = &quot;lm&quot;) + geom_point() A simple linear regression doesn’t fit the data well (not suprising since we included a squared predictor). Let’s fit a simple linear model and print out the model summary. fit.turkey = lm(formula = nap_time ~ 1 + turkey_time, data = df.turkey) summary(fit.turkey) ## ## Call: ## lm(formula = nap_time ~ 1 + turkey_time, data = df.turkey) ## ## Residuals: ## Min 1Q Median 3Q Max ## -212.82 -146.78 -55.17 125.74 462.52 ## ## Coefficients: ## Estimate Std. Error t value Pr(&gt;|t|) ## (Intercept) 15.4974 23.3827 0.663 0.508 ## turkey_time 51.5746 0.8115 63.557 &lt;2e-16 *** ## --- ## Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 ## ## Residual standard error: 172.4 on 248 degrees of freedom ## Multiple R-squared: 0.9422, Adjusted R-squared: 0.9419 ## F-statistic: 4039 on 1 and 248 DF, p-value: &lt; 2.2e-16 A regression with a squared predictor would fit well. fit.turkey2 = lm(formula = nap_time ~ 1 + I(turkey_time ^ 2), data = df.turkey) summary(fit.turkey2) ## ## Call: ## lm(formula = nap_time ~ 1 + I(turkey_time^2), data = df.turkey) ## ## Residuals: ## Min 1Q Median 3Q Max ## -45.611 -9.911 -0.652 11.137 43.008 ## ## Coefficients: ## Estimate Std. Error t value Pr(&gt;|t|) ## (Intercept) 4.994e+02 1.575e+00 317.0 &lt;2e-16 *** ## I(turkey_time^2) 1.001e+00 1.439e-03 695.3 &lt;2e-16 *** ## --- ## Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 ## ## Residual standard error: 16.23 on 248 degrees of freedom ## Multiple R-squared: 0.9995, Adjusted R-squared: 0.9995 ## F-statistic: 4.835e+05 on 1 and 248 DF, p-value: &lt; 2.2e-16 fit.turkey2 %&gt;% augment() %&gt;% clean_names() %&gt;% ggplot(data = ., mapping = aes(x = i_turkey_time_2, y = nap_time)) + geom_line(mapping = aes(y = fitted), color = &quot;blue&quot;) + geom_point() Let’s fit a bootstrap regression. boot.turkey = Boot(fit.turkey) summary(boot.turkey) ## ## Number of bootstrap replications R = 999 ## original bootBias bootSE bootMed ## (Intercept) 15.497 -1.130589 29.330 15.080 ## turkey_time 51.575 0.023717 1.058 51.625 fit.turkey %&gt;% tidy(conf.int = T) %&gt;% kable(digits = 2) %&gt;% kable_styling() term estimate std.error statistic p.value conf.low conf.high (Intercept) 15.50 23.38 0.66 0.51 -30.56 61.55 turkey_time 51.57 0.81 63.56 0.00 49.98 53.17 boot.turkey %&gt;% tidy(conf.int = T) %&gt;% kable(digits = 2) %&gt;% kable_styling() term statistic bias std.error conf.low conf.high (Intercept) 15.50 -1.13 29.33 -47.64 71.64 turkey_time 51.57 0.02 1.06 49.35 53.61 We see that the confidence intervals using the bootstrap method are wider than the ones that use the linear regression model (particularly for the intercept). 25.4 Additional resources Assumptions of a linear regression 25.5 Session info Information about this R session including which version of R was used, and what packages were loaded. sessionInfo() ## R version 4.1.2 (2021-11-01) ## Platform: x86_64-apple-darwin17.0 (64-bit) ## Running under: macOS Big Sur 10.16 ## ## Matrix products: default ## BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib ## LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib ## ## locale: ## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 ## ## attached base packages: ## [1] stats graphics grDevices utils datasets methods base ## ## other attached packages: ## [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 purrr_0.3.4 ## [5] readr_2.1.2 tidyr_1.2.0 tibble_3.1.7 tidyverse_1.3.1 ## [9] ggrepel_0.9.1 ggplot2_3.3.6 xtable_1.8-4 sjPlot_2.8.10 ## [13] stargazer_5.2.3 ggeffects_1.1.2 patchwork_1.1.1 janitor_2.1.0 ## [17] broom_0.8.0 car_3.0-13 carData_3.0-5 brms_2.17.0 ## [21] Rcpp_1.0.8.3 lme4_1.1-29 Matrix_1.4-1 tidybayes_3.0.2 ## [25] kableExtra_1.3.4 knitr_1.39 ## ## loaded via a namespace (and not attached): ## [1] utf8_1.2.2 tidyselect_1.1.2 htmlwidgets_1.5.4 ## [4] grid_4.1.2 munsell_0.5.0 codetools_0.2-18 ## [7] effectsize_0.6.0.1 DT_0.22 miniUI_0.1.1.1 ## [10] withr_2.5.0 Brobdingnag_1.2-7 colorspace_2.0-3 ## [13] highr_0.9 rstudioapi_0.13 stats4_4.1.2 ## [16] bayesplot_1.9.0 labeling_0.4.2 emmeans_1.7.3 ## [19] rstan_2.21.5 bit64_4.0.5 farver_2.1.0 ## [22] datawizard_0.4.0 bridgesampling_1.1-2 coda_0.19-4 ## [25] vctrs_0.4.1 generics_0.1.2 TH.data_1.1-1 ## [28] xfun_0.30 R6_2.5.1 markdown_1.1 ## [31] assertthat_0.2.1 promises_1.2.0.1 scales_1.2.0 ## [34] vroom_1.5.7 nnet_7.3-17 multcomp_1.4-19 ## [37] gtable_0.3.0 processx_3.5.3 sandwich_3.0-1 ## [40] rlang_1.0.2 systemfonts_1.0.4 splines_4.1.2 ## [43] checkmate_2.1.0 inline_0.3.19 yaml_2.3.5 ## [46] reshape2_1.4.4 abind_1.4-5 modelr_0.1.8 ## [49] threejs_0.3.3 crosstalk_1.2.0 backports_1.4.1 ## [52] httpuv_1.6.5 Hmisc_4.7-0 tensorA_0.36.2 ## [55] tools_4.1.2 bookdown_0.26 ellipsis_0.3.2 ## [58] RColorBrewer_1.1-3 jquerylib_0.1.4 posterior_1.2.1 ## [61] ggridges_0.5.3 plyr_1.8.7 base64enc_0.1-3 ## [64] ps_1.7.0 prettyunits_1.1.1 rpart_4.1.16 ## [67] zoo_1.8-10 cluster_2.1.3 haven_2.5.0 ## [70] fs_1.5.2 magrittr_2.0.3 data.table_1.14.2 ## [73] ggdist_3.1.1 colourpicker_1.1.1 reprex_2.0.1 ## [76] mvtnorm_1.1-3 sjmisc_2.8.9 matrixStats_0.62.0 ## [79] hms_1.1.1 shinyjs_2.1.0 mime_0.12 ## [82] evaluate_0.15 arrayhelpers_1.1-0 shinystan_2.6.0 ## [85] jpeg_0.1-9 sjstats_0.18.1 readxl_1.4.0 ## [88] gridExtra_2.3 rstantools_2.2.0 compiler_4.1.2 ## [91] crayon_1.5.1 minqa_1.2.4 StanHeaders_2.21.0-7 ## [94] htmltools_0.5.2 mgcv_1.8-40 later_1.3.0 ## [97] tzdb_0.3.0 Formula_1.2-4 RcppParallel_5.1.5 ## [100] lubridate_1.8.0 DBI_1.1.2 sjlabelled_1.2.0 ## [103] dbplyr_2.1.1 MASS_7.3-57 boot_1.3-28 ## [106] cli_3.3.0 parallel_4.1.2 insight_0.17.0 ## [109] igraph_1.3.1 pkgconfig_2.0.3 foreign_0.8-82 ## [112] xml2_1.3.3 svUnit_1.0.6 dygraphs_1.1.1.6 ## [115] svglite_2.1.0 bslib_0.3.1 webshot_0.5.3 ## [118] estimability_1.3 rvest_1.0.2 snakecase_0.11.0 ## [121] distributional_0.3.0 callr_3.7.0 digest_0.6.29 ## [124] parameters_0.17.0 rmarkdown_2.14 cellranger_1.1.0 ## [127] htmlTable_2.4.0 shiny_1.7.1 gtools_3.9.2 ## [130] nloptr_2.0.0 lifecycle_1.0.1 nlme_3.1-157 ## [133] jsonlite_1.8.0 viridisLite_0.4.0 fansi_1.0.3 ## [136] pillar_1.7.0 lattice_0.20-45 loo_2.5.1 ## [139] fastmap_1.1.0 httr_1.4.3 pkgbuild_1.3.1 ## [142] survival_3.3-1 glue_1.6.2 xts_0.12.1 ## [145] bayestestR_0.12.1 png_0.1-7 shinythemes_1.2.0 ## [148] bit_4.0.4 stringi_1.7.6 sass_0.4.1 ## [151] performance_0.9.0 latticeExtra_0.6-29 References "],["reporting-statistics.html", "Chapter 26 Reporting statistics 26.1 General advice 26.2 Some concrete example 26.3 Session info", " Chapter 26 Reporting statistics In this chapter, I’ll give a few examples for how to report statistical analysis. 26.1 General advice Here is some general advice first: Make good figures! Use statistical models to answer concrete research questions. Illustrate the uncertainty in your statistical inferences. Report effect sizes. 26.1.1 Make good figures! Chapters 2 and 3 go into how to make figures and also talk a little bit about what makes for a good figure. Personally, I like it when the figures give me a good sense for the actual data. For example, for an experimental study, I would like to get a good sense for the responses that participants gave in the different experimental conditions. Sometimes, papers just report the results of statistical tests, or only visually display estimates of the parameters in the model. I’m not a fan of that since, as we’ve learned, the parameters of the model are only useful in so far the model captures the data-generating process reasonably well. 26.1.2 Use statistical models to answer concrete research questions. Ideally, we formulate our research questions as statistical models upfront and pre-register our planned analyses (e.g. as an RMarkdown script with a complete analysis based on simulated data). We can then organize the results section by going through the sequence of research questions. Each statistical analysis then provides an answer to a specific research question. 26.1.3 Illustrate the uncertainty in your statistical inferences. For frequentist statistics, we can calculate confidence intervals (e.g. using bootstrapping) and we should provide these intervals together with the point estimates of the model’s predictors. For Bayesian statistics, we can calculate credible intervals based on the posterior over the model parameters. Our figures should also indicate the uncertainty that we have in our statistical inferences (e.g. by adding confidence bands, or by showing some samples from the posterior). 26.1.4 Report effect sizes. Rather than just saying whether the results of a statistical test was significant or not, you should, where possible, provide a measure of the effect size. Chapter 14 gives an overview of commonly used measures of effect size. 26.1.5 Reporting statistical results using RMarkdown For reporting statistical results in RMarkdown, I recommend the papaja package (see this chapter in the online book). 26.2 Some concrete example In this section, I’ll give a few concrete examples for how to report the results of statistical tests. Each example tries to implement the general advice mentioned above. I will discuss frequentist and Bayesian statistics separately. 26.2.1 Frequentist statistics 26.2.1.1 Simple regression df.credit = read_csv(&quot;data/credit.csv&quot;) %&gt;% rename(index = `...1`) %&gt;% clean_names() Research question: Do people with more income have a higher credit card balance? ggplot(data = df.credit, mapping = aes(x = income, y = balance)) + geom_smooth(method = &quot;lm&quot;, color = &quot;black&quot;) + geom_point(alpha = 0.2) + coord_cartesian(xlim = c(0, max(df.credit$income))) + labs(x = &quot;Income in $1K per year&quot;, y = &quot;Credit card balance in $&quot;) ## `geom_smooth()` using formula &#39;y ~ x&#39; Figure 26.1: Relationship between income level and credit card balance. The error band indicates a 95% confidence interval. # fit a model fit = lm(formula = balance ~ income, data = df.credit) summary(fit) ## ## Call: ## lm(formula = balance ~ income, data = df.credit) ## ## Residuals: ## Min 1Q Median 3Q Max ## -803.64 -348.99 -54.42 331.75 1100.25 ## ## Coefficients: ## Estimate Std. Error t value Pr(&gt;|t|) ## (Intercept) 246.5148 33.1993 7.425 6.9e-13 *** ## income 6.0484 0.5794 10.440 &lt; 2e-16 *** ## --- ## Signif. codes: 0 &#39;***&#39; 0.001 &#39;**&#39; 0.01 &#39;*&#39; 0.05 &#39;.&#39; 0.1 &#39; &#39; 1 ## ## Residual standard error: 407.9 on 398 degrees of freedom ## Multiple R-squared: 0.215, Adjusted R-squared: 0.213 ## F-statistic: 109 on 1 and 398 DF, p-value: &lt; 2.2e-16 # summarize the model results results_regression = fit %&gt;% apa_print() results_prediction = fit %&gt;% ggpredict(terms = &quot;income [20, 100]&quot;) %&gt;% mutate(across(where(is.numeric), ~ round(., 2))) Possible text: People with a higher income have a greater credit card balance \\(R^2 = .21\\), \\(F(1, 398) = 108.99\\), \\(p &lt; .001\\) (see Table 26.1). For each increase in income of $1K per year, the credit card balance is predicted to increase by \\(b = 6.05\\), 95% CI \\([4.91, 7.19]\\). For example, the predicted credit card balance of a person with an income of $20K per year is $367.48, 95% CI [318.31, 416.65], whereas for a person with an income of $100K per year, it is $851.35, 95% CI [777.41, 925.29] (see Figure 26.1). apa_table(results_regression$table, caption = &quot;A full regression table.&quot;, escape = FALSE) Table 26.1: A full regression table. Predictor \\(b\\) 95% CI \\(t\\) \\(\\mathit{df}\\) \\(p\\) Intercept 246.51 [181.25, 311.78] 7.43 398 &lt; .001 Income 6.05 [4.91, 7.19] 10.44 398 &lt; .001 26.3 Session info ## R version 4.1.2 (2021-11-01) ## Platform: x86_64-apple-darwin17.0 (64-bit) ## Running under: macOS Big Sur 10.16 ## ## Matrix products: default ## BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib ## LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib ## ## locale: ## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 ## ## attached base packages: ## [1] stats graphics grDevices utils datasets methods base ## ## other attached packages: ## [1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.9 ## [4] purrr_0.3.4 readr_2.1.2 tidyr_1.2.0 ## [7] tibble_3.1.7 ggplot2_3.3.6 tidyverse_1.3.1 ## [10] statsExpressions_1.3.1 ggeffects_1.1.2 tidybayes_3.0.2 ## [13] modelr_0.1.8 brms_2.17.0 Rcpp_1.0.8.3 ## [16] lme4_1.1-29 Matrix_1.4-1 broom_0.8.0 ## [19] papaja_0.1.0.9999 tinylabels_0.2.3 janitor_2.1.0 ## [22] kableExtra_1.3.4 knitr_1.39 ## ## loaded via a namespace (and not attached): ## [1] readxl_1.4.0 backports_1.4.1 systemfonts_1.0.4 ## [4] plyr_1.8.7 igraph_1.3.1 splines_4.1.2 ## [7] svUnit_1.0.6 crosstalk_1.2.0 TH.data_1.1-1 ## [10] rstantools_2.2.0 inline_0.3.19 digest_0.6.29 ## [13] htmltools_0.5.2 fansi_1.0.3 magrittr_2.0.3 ## [16] checkmate_2.1.0 tzdb_0.3.0 RcppParallel_5.1.5 ## [19] matrixStats_0.62.0 vroom_1.5.7 xts_0.12.1 ## [22] sandwich_3.0-1 svglite_2.1.0 prettyunits_1.1.1 ## [25] colorspace_2.0-3 rvest_1.0.2 ggdist_3.1.1 ## [28] haven_2.5.0 xfun_0.30 callr_3.7.0 ## [31] crayon_1.5.1 jsonlite_1.8.0 zeallot_0.1.0 ## [34] survival_3.3-1 zoo_1.8-10 glue_1.6.2 ## [37] gtable_0.3.0 emmeans_1.7.3 webshot_0.5.3 ## [40] distributional_0.3.0 pkgbuild_1.3.1 rstan_2.21.5 ## [43] abind_1.4-5 scales_1.2.0 mvtnorm_1.1-3 ## [46] DBI_1.1.2 miniUI_0.1.1.1 viridisLite_0.4.0 ## [49] xtable_1.8-4 performance_0.9.0 bit_4.0.4 ## [52] stats4_4.1.2 StanHeaders_2.21.0-7 DT_0.22 ## [55] datawizard_0.4.0 htmlwidgets_1.5.4 httr_1.4.3 ## [58] threejs_0.3.3 arrayhelpers_1.1-0 posterior_1.2.1 ## [61] ellipsis_0.3.2 pkgconfig_2.0.3 loo_2.5.1 ## [64] farver_2.1.0 dbplyr_2.1.1 sass_0.4.1 ## [67] utf8_1.2.2 labeling_0.4.2 tidyselect_1.1.2 ## [70] rlang_1.0.2 reshape2_1.4.4 later_1.3.0 ## [73] effectsize_0.6.0.1 cellranger_1.1.0 munsell_0.5.0 ## [76] tools_4.1.2 cli_3.3.0 generics_0.1.2 ## [79] sjlabelled_1.2.0 ggridges_0.5.3 evaluate_0.15 ## [82] fastmap_1.1.0 yaml_2.3.5 bit64_4.0.5 ## [85] fs_1.5.2 processx_3.5.3 nlme_3.1-157 ## [88] mime_0.12 xml2_1.3.3 compiler_4.1.2 ## [91] bayesplot_1.9.0 shinythemes_1.2.0 rstudioapi_0.13 ## [94] reprex_2.0.1 bslib_0.3.1 stringi_1.7.6 ## [97] highr_0.9 ps_1.7.0 parameters_0.17.0 ## [100] Brobdingnag_1.2-7 lattice_0.20-45 nloptr_2.0.0 ## [103] markdown_1.1 shinyjs_2.1.0 tensorA_0.36.2 ## [106] vctrs_0.4.1 pillar_1.7.0 lifecycle_1.0.1 ## [109] jquerylib_0.1.4 bridgesampling_1.1-2 estimability_1.3 ## [112] insight_0.17.0 httpuv_1.6.5 R6_2.5.1 ## [115] bookdown_0.26 promises_1.2.0.1 gridExtra_2.3 ## [118] codetools_0.2-18 boot_1.3-28 colourpicker_1.1.1 ## [121] MASS_7.3-57 gtools_3.9.2 assertthat_0.2.1 ## [124] withr_2.5.0 shinystan_2.6.0 multcomp_1.4-19 ## [127] mgcv_1.8-40 hms_1.1.1 bayestestR_0.12.1 ## [130] parallel_4.1.2 grid_4.1.2 coda_0.19-4 ## [133] minqa_1.2.4 rmarkdown_2.14 snakecase_0.11.0 ## [136] shiny_1.7.1 lubridate_1.8.0 base64enc_0.1-3 ## [139] dygraphs_1.1.1.6 "],["cheatsheets-6.html", "Chapter 27 Cheatsheets 27.1 Statistics 27.2 R", " Chapter 27 Cheatsheets This chapter contains a selection of useful cheatsheets. For updates check here: https://www.rstudio.com/resources/ To download the pdf of a cheatsheet, just click on the link in the figure caption. 27.1 Statistics Figure 27.1: Stats cheatsheet 27.2 R Figure 27.2: Data wrangling in the tidyverse Figure 27.3: advancedr Figure 27.4: base-r Figure 27.5: data-import Figure 27.6: data-transformation Figure 27.7: data-visualization Figure 27.8: how-big-is-your-graph Figure 27.9: latexsheet Figure 27.10: leaflet Figure 27.11: lubridate Figure 27.12: mosaic Figure 27.13: purrr Figure 27.14: regexcheatsheet Figure 27.15: rmarkdown-reference Figure 27.16: rmarkdown Figure 27.17: rstudio-ide Figure 27.18: shiny Figure 27.19: strings Figure 27.20: syntax Figure 27.21: tidyeval Figure 27.22: visualization principles "],["references-1.html", "References", " References "],["404.html", "Page not found", " Page not found The page you requested cannot be found (perhaps it was moved or renamed). You may want to try searching to find the page's new location, or use the table of contents to find the page you are looking for. "]]
+[["introduction.html", "Chapter 1 Introduction 1.1 Thanks 1.2 List of R packages used in this book 1.3 Session info", " Chapter 1 Introduction 1.1 Thanks Various people have helped in the process of putting together these materials (either knowingly, or unknowingly). Big thanks go to: Alexandra Chouldechova Allison Horst Andrew Heiss Ben Baumer Benoit Monin David Lagnado Ewart Thomas Henrik Singmann Julian Jara-Ettinger Justin Gardner Kevin Smith Lisa DeBruine Maarten Speekenbrink Matthew Kay Matthew Salganik Mika Braginsky Mike Frank Mine Çetinkaya-Rundel Nick C. Huntington-Klein Patrick Mair Paul-Christian Bürkner Peter Cushner Mohanty Richard McElreath Russ Poldrack Stephen Dewitt Solomon Kurz Tom Hardwicke Tristan Mahr Special thanks go to my teaching teams: 2022: Ari Beller Sarah Wu Chengxu Zhuang 2021: Andrew Nam Catherine Thomas Jon Walters Dan Yamins 2020: Tyler Bonnen Andrew Nam Jinxiao Zhang 2019: Andrew Lampinen Mona Rosenke Shao-Fang (Pam) Wang 1.2 List of R packages used in this book # RMarkdown library(&quot;knitr&quot;) # markdown things library(&quot;kableExtra&quot;) # for nicely formatted tables # Datasets library(&quot;gapminder&quot;) # data available from Gapminder.org library(&quot;NHANES&quot;) # data set library(&quot;titanic&quot;) # titanic dataset # Data manipulation library(&quot;arrangements&quot;) # fast generators and iterators for permutations, combinations and partitions library(&quot;magrittr&quot;) # for wrangling library(&quot;tidyverse&quot;) # everything else # Visualization library(&quot;patchwork&quot;) # making figure panels library(&quot;ggpol&quot;) # for making fancy boxplots library(&quot;ggridges&quot;) # for making joyplots library(&quot;gganimate&quot;) # for making animations library(&quot;GGally&quot;) # for pairs plot library(&quot;ggrepel&quot;) # for labels in ggplots library(&quot;corrr&quot;) # for calculating correlations between many variables library(&quot;corrplot&quot;) # for plotting correlations library(&quot;DiagrammeR&quot;) # for drawing diagrams library(&quot;ggeffects&quot;) # for visualizing effects library(&quot;bayesplot&quot;) # for visualization of Bayesian model fits # Modeling library(&quot;afex&quot;) # also for running ANOVAs library(&quot;lme4&quot;) # mixed effects models library(&quot;emmeans&quot;) # comparing estimated marginal means library(&quot;broom.mixed&quot;) # getting tidy mixed model summaries library(&quot;janitor&quot;) # cleaning variable names library(&quot;car&quot;) # for running ANOVAs library(&quot;rstanarm&quot;) # for Bayesian models library(&quot;greta&quot;) # Bayesian models library(&quot;tidybayes&quot;) # tidying up results from Bayesian models library(&quot;boot&quot;) # bootstrapping library(&quot;modelr&quot;) # cross-validation and bootstrapping library(&quot;mediation&quot;) # for mediation and moderation analysis library(&quot;multilevel&quot;) # Sobel test library(&quot;extraDistr&quot;) # additional probability distributions library(&quot;effects&quot;) # for showing effects in linear, generalized linear, and other models library(&quot;brms&quot;) # Bayesian regression library(&quot;parameters&quot;) # For extracting parameters # Misc library(&quot;tictoc&quot;) # timing things library(&quot;MASS&quot;) # various useful functions (e.g. bootstrapped confidence intervals) library(&quot;lsr&quot;) # for computing effect size measures library(&quot;extrafont&quot;) # additional fonts library(&quot;pwr&quot;) # for power calculations library(&quot;arrangements&quot;) # fast generators and iterators for permutations, combinations and partitions library(&quot;stargazer&quot;) # for regression tables library(&quot;sjPlot&quot;) # for regression tables library(&quot;xtable&quot;) # for tables 1.3 Session info ## R version 4.1.2 (2021-11-01) ## Platform: x86_64-apple-darwin17.0 (64-bit) ## Running under: macOS Big Sur 10.16 ## ## Matrix products: default ## BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib ## LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib ## ## locale: ## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 ## ## attached base packages: ## [1] stats graphics grDevices utils datasets methods base ## ## loaded via a namespace (and not attached): ## [1] bookdown_0.26 digest_0.6.29 R6_2.5.1 jsonlite_1.8.0 ## [5] magrittr_2.0.3 evaluate_0.15 stringi_1.7.6 rlang_1.0.2 ## [9] cli_3.3.0 rstudioapi_0.13 jquerylib_0.1.4 bslib_0.3.1 ## [13] rmarkdown_2.14 tools_4.1.2 stringr_1.4.0 xfun_0.30 ## [17] yaml_2.3.5 fastmap_1.1.0 compiler_4.1.2 htmltools_0.5.2 ## [21] knitr_1.39 sass_0.4.1 "],["404.html", "Page not found", " Page not found The page you requested cannot be found (perhaps it was moved or renamed). You may want to try searching to find the page's new location, or use the table of contents to find the page you are looking for. "]]
diff --git a/docs/visualization-1.html b/docs/visualization-1.html
index 11d1bd7..3e130e0 100644
--- a/docs/visualization-1.html
+++ b/docs/visualization-1.html
@@ -51,19 +51,6 @@
 <link href="libs/anchor-sections-1.1.0/anchor-sections.css" rel="stylesheet" />
 <link href="libs/anchor-sections-1.1.0/anchor-sections-hash.css" rel="stylesheet" />
 <script src="libs/anchor-sections-1.1.0/anchor-sections.js"></script>
-<script src="libs/htmlwidgets-1.5.4/htmlwidgets.js"></script>
-<link href="libs/datatables-css-0.0.0/datatables-crosstalk.css" rel="stylesheet" />
-<script src="libs/datatables-binding-0.22/datatables.js"></script>
-<link href="libs/dt-core-1.11.3/css/jquery.dataTables.min.css" rel="stylesheet" />
-<link href="libs/dt-core-1.11.3/css/jquery.dataTables.extra.css" rel="stylesheet" />
-<script src="libs/dt-core-1.11.3/js/jquery.dataTables.min.js"></script>
-<link href="libs/crosstalk-1.2.0/css/crosstalk.min.css" rel="stylesheet" />
-<script src="libs/crosstalk-1.2.0/js/crosstalk.min.js"></script>
-<script src="libs/viz-1.8.2/viz.js"></script>
-<link href="libs/DiagrammeR-styles-0.2/styles.css" rel="stylesheet" />
-<script src="libs/grViz-binding-1.0.9/grViz.js"></script>
-<script src="libs/kePrint-0.0.1/kePrint.js"></script>
-<link href="libs/lightable-0.0.1/lightable.css" rel="stylesheet" />
 
 
 <style type="text/css">
@@ -369,20 +356,23 @@
 <li class="chapter" data-level="7.3.4" data-path="simulation-1.html"><a href="simulation-1.html#cumulative-probability-distribution"><i class="fa fa-check"></i><b>7.3.4</b> Cumulative probability distribution</a></li>
 <li class="chapter" data-level="7.3.5" data-path="simulation-1.html"><a href="simulation-1.html#inverse-cumulative-distribution"><i class="fa fa-check"></i><b>7.3.5</b> Inverse cumulative distribution</a></li>
 <li class="chapter" data-level="7.3.6" data-path="simulation-1.html"><a href="simulation-1.html#computing-probabilities"><i class="fa fa-check"></i><b>7.3.6</b> Computing probabilities</a></li>
-<li class="chapter" data-level="7.3.7" data-path="simulation-1.html"><a href="simulation-1.html#make-the-plot"><i class="fa fa-check"></i><b>7.3.7</b> Make the plot</a></li>
-<li class="chapter" data-level="7.3.8" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solutions"><i class="fa fa-check"></i><b>7.3.8</b> Analytic solutions</a></li>
-<li class="chapter" data-level="7.3.9" data-path="simulation-1.html"><a href="simulation-1.html#sampling-solution"><i class="fa fa-check"></i><b>7.3.9</b> Sampling solution</a></li>
 </ul></li>
-<li class="chapter" data-level="7.4" data-path="simulation-1.html"><a href="simulation-1.html#bayesian-inference-with-the-normal-distribution"><i class="fa fa-check"></i><b>7.4</b> Bayesian inference with the normal distribution</a>
+<li class="chapter" data-level="7.4" data-path="simulation-1.html"><a href="simulation-1.html#breakout-room-exercise"><i class="fa fa-check"></i><b>7.4</b> Breakout room exercise</a>
 <ul>
-<li class="chapter" data-level="7.4.1" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solution"><i class="fa fa-check"></i><b>7.4.1</b> Analytic solution</a></li>
-<li class="chapter" data-level="7.4.2" data-path="simulation-1.html"><a href="simulation-1.html#solution-via-sampling"><i class="fa fa-check"></i><b>7.4.2</b> Solution via sampling</a></li>
+<li class="chapter" data-level="7.4.1" data-path="simulation-1.html"><a href="simulation-1.html#make-the-plot"><i class="fa fa-check"></i><b>7.4.1</b> Make the plot</a></li>
+<li class="chapter" data-level="7.4.2" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solutions"><i class="fa fa-check"></i><b>7.4.2</b> Analytic solutions</a></li>
+<li class="chapter" data-level="7.4.3" data-path="simulation-1.html"><a href="simulation-1.html#sampling-solution"><i class="fa fa-check"></i><b>7.4.3</b> Sampling solution</a></li>
 </ul></li>
-<li class="chapter" data-level="7.5" data-path="simulation-1.html"><a href="simulation-1.html#additional-resources-5"><i class="fa fa-check"></i><b>7.5</b> Additional resources</a>
+<li class="chapter" data-level="7.5" data-path="simulation-1.html"><a href="simulation-1.html#bayesian-inference-with-the-normal-distribution"><i class="fa fa-check"></i><b>7.5</b> Bayesian inference with the normal distribution</a>
 <ul>
-<li class="chapter" data-level="7.5.1" data-path="simulation-1.html"><a href="simulation-1.html#datacamp"><i class="fa fa-check"></i><b>7.5.1</b> Datacamp</a></li>
+<li class="chapter" data-level="7.5.1" data-path="simulation-1.html"><a href="simulation-1.html#analytic-solution"><i class="fa fa-check"></i><b>7.5.1</b> Analytic solution</a></li>
+<li class="chapter" data-level="7.5.2" data-path="simulation-1.html"><a href="simulation-1.html#solution-via-sampling"><i class="fa fa-check"></i><b>7.5.2</b> Solution via sampling</a></li>
 </ul></li>
-<li class="chapter" data-level="7.6" data-path="simulation-1.html"><a href="simulation-1.html#session-info-6"><i class="fa fa-check"></i><b>7.6</b> Session info</a></li>
+<li class="chapter" data-level="7.6" data-path="simulation-1.html"><a href="simulation-1.html#additional-resources-5"><i class="fa fa-check"></i><b>7.6</b> Additional resources</a>
+<ul>
+<li class="chapter" data-level="7.6.1" data-path="simulation-1.html"><a href="simulation-1.html#datacamp"><i class="fa fa-check"></i><b>7.6.1</b> Datacamp</a></li>
+</ul></li>
+<li class="chapter" data-level="7.7" data-path="simulation-1.html"><a href="simulation-1.html#session-info-6"><i class="fa fa-check"></i><b>7.7</b> Session info</a></li>
 </ul></li>
 <li class="chapter" data-level="8" data-path="simulation-2.html"><a href="simulation-2.html"><i class="fa fa-check"></i><b>8</b> Simulation 2</a>
 <ul>
@@ -861,13 +851,13 @@ <h2><span class="header-section-number">2.1</span> Learning goals<a href="visual
 <div id="load-packages" class="section level2 hasAnchor" number="2.2">
 <h2><span class="header-section-number">2.2</span> Load packages<a href="visualization-1.html#load-packages" class="anchor-section" aria-label="Anchor link to header"></a></h2>
 <p>Let’s first load the packages that we need for this chapter. You can click on the green arrow to execute the code chunk below.</p>
-<div class="sourceCode" id="cb3"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb3-1"><a href="visualization-1.html#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;knitr&quot;</span>)     <span class="co"># for rendering the RMarkdown file</span></span>
-<span id="cb3-2"><a href="visualization-1.html#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;tidyverse&quot;</span>) <span class="co"># for plotting (and many more cool things we&#39;ll discover later)</span></span>
-<span id="cb3-3"><a href="visualization-1.html#cb3-3" aria-hidden="true" tabindex="-1"></a></span>
-<span id="cb3-4"><a href="visualization-1.html#cb3-4" aria-hidden="true" tabindex="-1"></a><span class="co"># these options here change the formatting of how comments are rendered</span></span>
-<span id="cb3-5"><a href="visualization-1.html#cb3-5" aria-hidden="true" tabindex="-1"></a><span class="co"># in RMarkdown </span></span>
-<span id="cb3-6"><a href="visualization-1.html#cb3-6" aria-hidden="true" tabindex="-1"></a>opts_chunk<span class="sc">$</span><span class="fu">set</span>(<span class="at">comment =</span> <span class="st">&quot;&quot;</span>,</span>
-<span id="cb3-7"><a href="visualization-1.html#cb3-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">fig.show =</span> <span class="st">&quot;hold&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb1"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb1-1"><a href="visualization-1.html#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;knitr&quot;</span>)     <span class="co"># for rendering the RMarkdown file</span></span>
+<span id="cb1-2"><a href="visualization-1.html#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="fu">library</span>(<span class="st">&quot;tidyverse&quot;</span>) <span class="co"># for plotting (and many more cool things we&#39;ll discover later)</span></span>
+<span id="cb1-3"><a href="visualization-1.html#cb1-3" aria-hidden="true" tabindex="-1"></a></span>
+<span id="cb1-4"><a href="visualization-1.html#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="co"># these options here change the formatting of how comments are rendered</span></span>
+<span id="cb1-5"><a href="visualization-1.html#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="co"># in RMarkdown </span></span>
+<span id="cb1-6"><a href="visualization-1.html#cb1-6" aria-hidden="true" tabindex="-1"></a>opts_chunk<span class="sc">$</span><span class="fu">set</span>(<span class="at">comment =</span> <span class="st">&quot;&quot;</span>,</span>
+<span id="cb1-7"><a href="visualization-1.html#cb1-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">fig.show =</span> <span class="st">&quot;hold&quot;</span>)</span></code></pre></div>
 <p>The <code>tidyverse</code> is a collection of packages that includes <code>ggplot2</code>.</p>
 </div>
 <div id="why-visualize-data" class="section level2 hasAnchor" number="2.3">
@@ -934,30 +924,30 @@ <h3><span class="header-section-number">2.4.1</span> Setting up RStudio<a href="
 <li>Soft-wrap R source files is <em>selected</em></li>
 </ul>
 <p>This way you don’t have to scroll horizontally. At the same time, avoid writing long single lines of code. For example, instead of writing code like so:</p>
-<div class="sourceCode" id="cb4"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb4-1"><a href="visualization-1.html#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> diamonds, <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> cut, <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb4-2"><a href="visualization-1.html#cb4-2" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>, <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>, <span class="at">color =</span> <span class="st">&quot;black&quot;</span>, <span class="at">fill =</span> <span class="st">&quot;lightblue&quot;</span>, <span class="at">width =</span> <span class="fl">0.85</span>) <span class="sc">+</span></span>
-<span id="cb4-3"><a href="visualization-1.html#cb4-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>, <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>, <span class="at">size =</span> <span class="fl">1.5</span>) <span class="sc">+</span></span>
-<span id="cb4-4"><a href="visualization-1.html#cb4-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">labs</span>(<span class="at">title =</span> <span class="st">&quot;Price as a function of quality of cut&quot;</span>, <span class="at">subtitle =</span> <span class="st">&quot;Note: The price is in US dollars&quot;</span>, <span class="at">tag =</span> <span class="st">&quot;A&quot;</span>, <span class="at">x =</span> <span class="st">&quot;Quality of the cut&quot;</span>, <span class="at">y =</span> <span class="st">&quot;Price&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb2"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb2-1"><a href="visualization-1.html#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> diamonds, <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> cut, <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb2-2"><a href="visualization-1.html#cb2-2" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>, <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>, <span class="at">color =</span> <span class="st">&quot;black&quot;</span>, <span class="at">fill =</span> <span class="st">&quot;lightblue&quot;</span>, <span class="at">width =</span> <span class="fl">0.85</span>) <span class="sc">+</span></span>
+<span id="cb2-3"><a href="visualization-1.html#cb2-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>, <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>, <span class="at">size =</span> <span class="fl">1.5</span>) <span class="sc">+</span></span>
+<span id="cb2-4"><a href="visualization-1.html#cb2-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">labs</span>(<span class="at">title =</span> <span class="st">&quot;Price as a function of quality of cut&quot;</span>, <span class="at">subtitle =</span> <span class="st">&quot;Note: The price is in US dollars&quot;</span>, <span class="at">tag =</span> <span class="st">&quot;A&quot;</span>, <span class="at">x =</span> <span class="st">&quot;Quality of the cut&quot;</span>, <span class="at">y =</span> <span class="st">&quot;Price&quot;</span>)</span></code></pre></div>
 <p>You may want to write it this way instead:</p>
-<div class="sourceCode" id="cb5"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb5-1"><a href="visualization-1.html#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> diamonds, </span>
-<span id="cb5-2"><a href="visualization-1.html#cb5-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> cut,</span>
-<span id="cb5-3"><a href="visualization-1.html#cb5-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb5-4"><a href="visualization-1.html#cb5-4" aria-hidden="true" tabindex="-1"></a>  <span class="co"># display the means</span></span>
-<span id="cb5-5"><a href="visualization-1.html#cb5-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb5-6"><a href="visualization-1.html#cb5-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>,</span>
-<span id="cb5-7"><a href="visualization-1.html#cb5-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;black&quot;</span>,</span>
-<span id="cb5-8"><a href="visualization-1.html#cb5-8" aria-hidden="true" tabindex="-1"></a>               <span class="at">fill =</span> <span class="st">&quot;lightblue&quot;</span>,</span>
-<span id="cb5-9"><a href="visualization-1.html#cb5-9" aria-hidden="true" tabindex="-1"></a>               <span class="at">width =</span> <span class="fl">0.85</span>) <span class="sc">+</span></span>
-<span id="cb5-10"><a href="visualization-1.html#cb5-10" aria-hidden="true" tabindex="-1"></a>  <span class="co"># display the error bars</span></span>
-<span id="cb5-11"><a href="visualization-1.html#cb5-11" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
-<span id="cb5-12"><a href="visualization-1.html#cb5-12" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>,</span>
-<span id="cb5-13"><a href="visualization-1.html#cb5-13" aria-hidden="true" tabindex="-1"></a>               <span class="at">size =</span> <span class="fl">1.5</span>) <span class="sc">+</span></span>
-<span id="cb5-14"><a href="visualization-1.html#cb5-14" aria-hidden="true" tabindex="-1"></a>  <span class="co"># change labels</span></span>
-<span id="cb5-15"><a href="visualization-1.html#cb5-15" aria-hidden="true" tabindex="-1"></a>  <span class="fu">labs</span>(<span class="at">title =</span> <span class="st">&quot;Price as a function of quality of cut&quot;</span>,</span>
-<span id="cb5-16"><a href="visualization-1.html#cb5-16" aria-hidden="true" tabindex="-1"></a>       <span class="at">subtitle =</span> <span class="st">&quot;Note: The price is in US dollars&quot;</span>, <span class="co"># we might want to change this later</span></span>
-<span id="cb5-17"><a href="visualization-1.html#cb5-17" aria-hidden="true" tabindex="-1"></a>       <span class="at">tag =</span> <span class="st">&quot;A&quot;</span>,</span>
-<span id="cb5-18"><a href="visualization-1.html#cb5-18" aria-hidden="true" tabindex="-1"></a>       <span class="at">x =</span> <span class="st">&quot;Quality of the cut&quot;</span>,</span>
-<span id="cb5-19"><a href="visualization-1.html#cb5-19" aria-hidden="true" tabindex="-1"></a>       <span class="at">y =</span> <span class="st">&quot;Price&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb3"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb3-1"><a href="visualization-1.html#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> diamonds, </span>
+<span id="cb3-2"><a href="visualization-1.html#cb3-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> cut,</span>
+<span id="cb3-3"><a href="visualization-1.html#cb3-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb3-4"><a href="visualization-1.html#cb3-4" aria-hidden="true" tabindex="-1"></a>  <span class="co"># display the means</span></span>
+<span id="cb3-5"><a href="visualization-1.html#cb3-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb3-6"><a href="visualization-1.html#cb3-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>,</span>
+<span id="cb3-7"><a href="visualization-1.html#cb3-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;black&quot;</span>,</span>
+<span id="cb3-8"><a href="visualization-1.html#cb3-8" aria-hidden="true" tabindex="-1"></a>               <span class="at">fill =</span> <span class="st">&quot;lightblue&quot;</span>,</span>
+<span id="cb3-9"><a href="visualization-1.html#cb3-9" aria-hidden="true" tabindex="-1"></a>               <span class="at">width =</span> <span class="fl">0.85</span>) <span class="sc">+</span></span>
+<span id="cb3-10"><a href="visualization-1.html#cb3-10" aria-hidden="true" tabindex="-1"></a>  <span class="co"># display the error bars</span></span>
+<span id="cb3-11"><a href="visualization-1.html#cb3-11" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
+<span id="cb3-12"><a href="visualization-1.html#cb3-12" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>,</span>
+<span id="cb3-13"><a href="visualization-1.html#cb3-13" aria-hidden="true" tabindex="-1"></a>               <span class="at">size =</span> <span class="fl">1.5</span>) <span class="sc">+</span></span>
+<span id="cb3-14"><a href="visualization-1.html#cb3-14" aria-hidden="true" tabindex="-1"></a>  <span class="co"># change labels</span></span>
+<span id="cb3-15"><a href="visualization-1.html#cb3-15" aria-hidden="true" tabindex="-1"></a>  <span class="fu">labs</span>(<span class="at">title =</span> <span class="st">&quot;Price as a function of quality of cut&quot;</span>,</span>
+<span id="cb3-16"><a href="visualization-1.html#cb3-16" aria-hidden="true" tabindex="-1"></a>       <span class="at">subtitle =</span> <span class="st">&quot;Note: The price is in US dollars&quot;</span>, <span class="co"># we might want to change this later</span></span>
+<span id="cb3-17"><a href="visualization-1.html#cb3-17" aria-hidden="true" tabindex="-1"></a>       <span class="at">tag =</span> <span class="st">&quot;A&quot;</span>,</span>
+<span id="cb3-18"><a href="visualization-1.html#cb3-18" aria-hidden="true" tabindex="-1"></a>       <span class="at">x =</span> <span class="st">&quot;Quality of the cut&quot;</span>,</span>
+<span id="cb3-19"><a href="visualization-1.html#cb3-19" aria-hidden="true" tabindex="-1"></a>       <span class="at">y =</span> <span class="st">&quot;Price&quot;</span>)</span></code></pre></div>
 <p>This makes it much easier to see what’s going on, and you can easily add comments to individual lines of code.</p>
 <blockquote>
 <p><strong>Tip</strong>: If a function has more than two arguments put each argument on a new line.</p>
@@ -974,8 +964,8 @@ <h3><span class="header-section-number">2.4.1</span> Setting up RStudio<a href="
 <div id="getting-help" class="section level3 hasAnchor" number="2.4.2">
 <h3><span class="header-section-number">2.4.2</span> Getting help<a href="visualization-1.html#getting-help" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>There are three simple ways to get help in R. You can either put a <code>?</code> in front of the function you’d like to learn more about, or use the <code>help()</code> function.</p>
-<div class="sourceCode" id="cb6"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb6-1"><a href="visualization-1.html#cb6-1" aria-hidden="true" tabindex="-1"></a>?print</span>
-<span id="cb6-2"><a href="visualization-1.html#cb6-2" aria-hidden="true" tabindex="-1"></a><span class="fu">help</span>(<span class="st">&quot;print&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb4"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb4-1"><a href="visualization-1.html#cb4-1" aria-hidden="true" tabindex="-1"></a>?print</span>
+<span id="cb4-2"><a href="visualization-1.html#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="fu">help</span>(<span class="st">&quot;print&quot;</span>)</span></code></pre></div>
 <blockquote>
 <p><strong>Tip</strong>: To see the help file, hover over a function (or dataset) with the mouse (or select the text) and then press <code>F1</code>.</p>
 </blockquote>
@@ -1044,9 +1034,9 @@ <h2><span class="header-section-number">2.5</span> Data visualization<a href="vi
 <div id="setting-up-a-plot" class="section level3 hasAnchor" number="2.5.1">
 <h3><span class="header-section-number">2.5.1</span> Setting up a plot<a href="visualization-1.html#setting-up-a-plot" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>Let’s first get some data.</p>
-<div class="sourceCode" id="cb7"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb7-1"><a href="visualization-1.html#cb7-1" aria-hidden="true" tabindex="-1"></a>df.diamonds <span class="ot">=</span> diamonds</span></code></pre></div>
+<div class="sourceCode" id="cb5"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb5-1"><a href="visualization-1.html#cb5-1" aria-hidden="true" tabindex="-1"></a>df.diamonds <span class="ot">=</span> diamonds</span></code></pre></div>
 <p>The <code>diamonds</code> dataset comes with the <code>ggplot2</code> package. We can get a description of the dataset by running the following command:</p>
-<div class="sourceCode" id="cb8"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb8-1"><a href="visualization-1.html#cb8-1" aria-hidden="true" tabindex="-1"></a>?diamonds</span></code></pre></div>
+<div class="sourceCode" id="cb6"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb6-1"><a href="visualization-1.html#cb6-1" aria-hidden="true" tabindex="-1"></a>?diamonds</span></code></pre></div>
 <p>Above, we assigned the <code>diamonds</code> dataset to the variable <code>df.diamonds</code> so that we can see it in the data explorer.</p>
 <p>Let’s take a look at the full dataset by clicking on it in the explorer.</p>
 <blockquote>
@@ -1054,48 +1044,48 @@ <h3><span class="header-section-number">2.5.1</span> Setting up a plot<a href="v
 </blockquote>
 <p>The <code>df.diamonds</code> data frame contains information about almost 60,000 diamonds, including their <code>price</code>, <code>carat</code> value, size, etc. Let’s use visualization to get a better sense for this dataset.</p>
 <p>We start by setting up the plot. To do so, we pass a data frame to the function <code>ggplot()</code> in the following way.</p>
-<div class="sourceCode" id="cb9"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb9-1"><a href="visualization-1.html#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds)</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-11-1.png" width="672" /></p>
+<div class="sourceCode" id="cb7"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb7-1"><a href="visualization-1.html#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-11-1.png" width="672" /></p>
 <p>This, by itself, won’t do anything yet. We also need to specify what to plot.</p>
 <p>Let’s take a look at how much diamonds of different color cost. The help file says that diamonds labeled D have the best color, and diamonds labeled J the worst color. Let’s make a bar plot that shows the average price of diamonds for different colors.</p>
 <p>We do so via specifying a mapping from the data to the plot aesthetics with the function <code>aes()</code>. We need to tell <code>aes()</code> what we would like to display on the x-axis, and the y-axis of the plot.</p>
-<div class="sourceCode" id="cb10"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb10-1"><a href="visualization-1.html#cb10-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb10-2"><a href="visualization-1.html#cb10-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
-<span id="cb10-3"><a href="visualization-1.html#cb10-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price))</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-12-1.png" width="672" /></p>
+<div class="sourceCode" id="cb8"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb8-1"><a href="visualization-1.html#cb8-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb8-2"><a href="visualization-1.html#cb8-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
+<span id="cb8-3"><a href="visualization-1.html#cb8-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price))</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-12-1.png" width="672" /></p>
 <p>Here, we specified that we want to plot <code>color</code> on the x-axis, and <code>price</code> on the y-axis. As you can see, <code>ggplot2</code> has already figured out how to label the axes. However, we still need to specify <em>how</em> to plot it.</p>
 </div>
 <div id="bar-plot" class="section level3 hasAnchor" number="2.5.2">
 <h3><span class="header-section-number">2.5.2</span> Bar plot<a href="visualization-1.html#bar-plot" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>Let’s make a <strong>bar graph</strong>:</p>
-<div class="sourceCode" id="cb11"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb11-1"><a href="visualization-1.html#cb11-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb11-2"><a href="visualization-1.html#cb11-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
-<span id="cb11-3"><a href="visualization-1.html#cb11-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb11-4"><a href="visualization-1.html#cb11-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb11-5"><a href="visualization-1.html#cb11-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>)</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-13-1.png" width="672" /></p>
+<div class="sourceCode" id="cb9"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb9-1"><a href="visualization-1.html#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb9-2"><a href="visualization-1.html#cb9-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
+<span id="cb9-3"><a href="visualization-1.html#cb9-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb9-4"><a href="visualization-1.html#cb9-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb9-5"><a href="visualization-1.html#cb9-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-13-1.png" width="672" /></p>
 <p>Neat! Three lines of code produce an almost-publication-ready plot (to be published in the <em>Proceedings of Unnecessary Diamonds</em>)! Note how we used a <code>+</code> at the end of the first line of code to specify that there will be more. This is a very powerful idea underlying <code>ggplot2</code>. We can start simple and keep adding things to the plot step by step.</p>
 <p>We used the <code>stat_summary()</code> function to define <em>what</em> we want to plot (the “mean”), and <em>how</em> (as a “bar” chart). Let’s take a closer look at that function.</p>
-<div class="sourceCode" id="cb12"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb12-1"><a href="visualization-1.html#cb12-1" aria-hidden="true" tabindex="-1"></a><span class="fu">help</span>(stat_summary)</span></code></pre></div>
+<div class="sourceCode" id="cb10"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb10-1"><a href="visualization-1.html#cb10-1" aria-hidden="true" tabindex="-1"></a><span class="fu">help</span>(stat_summary)</span></code></pre></div>
 <p>Not the the easiest help file … We supplied two arguments to the function, <code>fun =</code> and <code>geom =</code>.</p>
 <ol style="list-style-type: decimal">
 <li>The <code>fun</code> argument specifies <em>what</em> function we’d like to apply to the data for each value of <code>x</code>. Here, we said that we would like to take the <code>mean</code> and we specified that as a string.</li>
 <li>The <code>geom</code> (= geometric object) argument specifies <em>how</em> we would like to plot the result, namely as a “bar” plot.</li>
 </ol>
 <p>Instead of showing the “mean”, we could also show the “median” instead.</p>
-<div class="sourceCode" id="cb13"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb13-1"><a href="visualization-1.html#cb13-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb13-2"><a href="visualization-1.html#cb13-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
-<span id="cb13-3"><a href="visualization-1.html#cb13-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb13-4"><a href="visualization-1.html#cb13-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;median&quot;</span>,</span>
-<span id="cb13-5"><a href="visualization-1.html#cb13-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>)</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-15-1.png" width="672" /></p>
+<div class="sourceCode" id="cb11"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb11-1"><a href="visualization-1.html#cb11-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb11-2"><a href="visualization-1.html#cb11-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
+<span id="cb11-3"><a href="visualization-1.html#cb11-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb11-4"><a href="visualization-1.html#cb11-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;median&quot;</span>,</span>
+<span id="cb11-5"><a href="visualization-1.html#cb11-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-15-1.png" width="672" /></p>
 <p>And instead of making a bar plot, we could plot some points.</p>
-<div class="sourceCode" id="cb14"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb14-1"><a href="visualization-1.html#cb14-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(df.diamonds,</span>
-<span id="cb14-2"><a href="visualization-1.html#cb14-2" aria-hidden="true" tabindex="-1"></a>       <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
-<span id="cb14-3"><a href="visualization-1.html#cb14-3" aria-hidden="true" tabindex="-1"></a>           <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb14-4"><a href="visualization-1.html#cb14-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb14-5"><a href="visualization-1.html#cb14-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;point&quot;</span>)</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-16-1.png" width="672" /></p>
+<div class="sourceCode" id="cb12"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb12-1"><a href="visualization-1.html#cb12-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(df.diamonds,</span>
+<span id="cb12-2"><a href="visualization-1.html#cb12-2" aria-hidden="true" tabindex="-1"></a>       <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
+<span id="cb12-3"><a href="visualization-1.html#cb12-3" aria-hidden="true" tabindex="-1"></a>           <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb12-4"><a href="visualization-1.html#cb12-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb12-5"><a href="visualization-1.html#cb12-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;point&quot;</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-16-1.png" width="672" /></p>
 <blockquote>
 <p><strong>Tip</strong>: Take a look <a href="https://ggplot2.tidyverse.org/reference/#section-layer-geoms">here</a> to see what other geoms ggplot2 supports.</p>
 </blockquote>
@@ -1104,32 +1094,32 @@ <h3><span class="header-section-number">2.5.2</span> Bar plot<a href="visualizat
 <div id="setting-the-default-plot-theme" class="section level3 hasAnchor" number="2.5.3">
 <h3><span class="header-section-number">2.5.3</span> Setting the default plot theme<a href="visualization-1.html#setting-the-default-plot-theme" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>Before moving on, let’s set a different default theme for our plots. Personally, I’m not a big fan of the gray background and the white grid lines. Also, the default size of the text should be bigger. We can change the default theme using the <code>theme_set()</code> function like so:</p>
-<div class="sourceCode" id="cb15"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb15-1"><a href="visualization-1.html#cb15-1" aria-hidden="true" tabindex="-1"></a><span class="fu">theme_set</span>(<span class="fu">theme_classic</span>() <span class="sc">+</span> <span class="co"># set the theme</span></span>
-<span id="cb15-2"><a href="visualization-1.html#cb15-2" aria-hidden="true" tabindex="-1"></a>            <span class="fu">theme</span>(<span class="at">text =</span> <span class="fu">element_text</span>(<span class="at">size =</span> <span class="dv">20</span>))) <span class="co"># set the default text size</span></span></code></pre></div>
+<div class="sourceCode" id="cb13"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb13-1"><a href="visualization-1.html#cb13-1" aria-hidden="true" tabindex="-1"></a><span class="fu">theme_set</span>(<span class="fu">theme_classic</span>() <span class="sc">+</span> <span class="co"># set the theme</span></span>
+<span id="cb13-2"><a href="visualization-1.html#cb13-2" aria-hidden="true" tabindex="-1"></a>            <span class="fu">theme</span>(<span class="at">text =</span> <span class="fu">element_text</span>(<span class="at">size =</span> <span class="dv">20</span>))) <span class="co"># set the default text size</span></span></code></pre></div>
 <p>From now on, all our plots will use what’s specified in <code>theme_classic()</code>, and the default text size will be larger, too. For any individual plot, we can still override these settings.</p>
 </div>
 <div id="scatter-plot" class="section level3 hasAnchor" number="2.5.4">
 <h3><span class="header-section-number">2.5.4</span> Scatter plot<a href="visualization-1.html#scatter-plot" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>I don’t know much about diamonds, but I do know that diamonds with a higher <code>carat</code> value tend to be more expensive. <code>color</code> was a discrete variable with seven different values. <code>carat</code>, however, is a continuous variable. We want to see how the price of diamonds differs as a function of the <code>carat</code> value. Since we are interested in the relationship between two continuous variables, plotting a bar graph won’t work. Instead, let’s make a <strong>scatter plot</strong>. Let’s put the <code>carat</code> value on the x-axis, and the <code>price</code> on the y-axis.</p>
-<div class="sourceCode" id="cb16"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb16-1"><a href="visualization-1.html#cb16-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb16-2"><a href="visualization-1.html#cb16-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> carat,</span>
-<span id="cb16-3"><a href="visualization-1.html#cb16-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb16-4"><a href="visualization-1.html#cb16-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>()</span></code></pre></div>
+<div class="sourceCode" id="cb14"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb14-1"><a href="visualization-1.html#cb14-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb14-2"><a href="visualization-1.html#cb14-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> carat,</span>
+<span id="cb14-3"><a href="visualization-1.html#cb14-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb14-4"><a href="visualization-1.html#cb14-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>()</span></code></pre></div>
 <div class="figure"><span style="display:block;" id="fig:scatter"></span>
-<img src="02-visualization1_files/figure-html/scatter-1.png" alt="Scatterplot." width="672" />
+<img src="psych252_files/figure-html/scatter-1.png" alt="Scatterplot." width="672" />
 <p class="caption">
 Figure 2.8: Scatterplot.
 </p>
 </div>
 <p>Cool! That looks sensible. Diamonds with a higher <code>carat</code> value tend to have a higher <code>price</code>. Our dataset has 53940 rows. So the plot actually shows 53940 circles even though we can’t see all of them since they overlap.</p>
 <p>Let’s make some progress on trying to figure out why the diamonds with the better color weren’t the most expensive ones on average. We’ll add some color to the scatter plot in Figure <a href="visualization-1.html#fig:scatter">2.8</a>. We color each of the points based on the diamond’s color. To do so, we pass another argument to the aesthetics of the plot via <code>aes()</code>.</p>
-<div class="sourceCode" id="cb17"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb17-1"><a href="visualization-1.html#cb17-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb17-2"><a href="visualization-1.html#cb17-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> carat,</span>
-<span id="cb17-3"><a href="visualization-1.html#cb17-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
-<span id="cb17-4"><a href="visualization-1.html#cb17-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">color =</span> color)) <span class="sc">+</span></span>
-<span id="cb17-5"><a href="visualization-1.html#cb17-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>()</span></code></pre></div>
+<div class="sourceCode" id="cb15"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb15-1"><a href="visualization-1.html#cb15-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb15-2"><a href="visualization-1.html#cb15-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> carat,</span>
+<span id="cb15-3"><a href="visualization-1.html#cb15-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
+<span id="cb15-4"><a href="visualization-1.html#cb15-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">color =</span> color)) <span class="sc">+</span></span>
+<span id="cb15-5"><a href="visualization-1.html#cb15-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>()</span></code></pre></div>
 <div class="figure"><span style="display:block;" id="fig:scatter-color"></span>
-<img src="02-visualization1_files/figure-html/scatter-color-1.png" alt="Scatterplot with color." width="672" />
+<img src="psych252_files/figure-html/scatter-color-1.png" alt="Scatterplot with color." width="672" />
 <p class="caption">
 Figure 2.9: Scatterplot with color.
 </p>
@@ -1137,12 +1127,12 @@ <h3><span class="header-section-number">2.5.4</span> Scatter plot<a href="visual
 <p>Aha! Now we’ve got some color. Notice how in Figure <a href="visualization-1.html#fig:scatter-color">2.9</a> <code>ggplot2</code> added a legend for us, thanks! We’ll see later how to play around with legends. Form just eye-balling the plot, it looks like the diamonds with the best <code>color</code> (D) tended to have a lower <code>carat</code> value, and the ones with the worst <code>color</code> (J), tended to have the highest carat values.</p>
 <p>So this is why diamonds with better colors are less expensive – these diamonds have a lower carat value overall.</p>
 <p>There are many other things that we can define in <code>aes()</code>. Take a quick look at the vignette:</p>
-<div class="sourceCode" id="cb18"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb18-1"><a href="visualization-1.html#cb18-1" aria-hidden="true" tabindex="-1"></a><span class="fu">vignette</span>(<span class="st">&quot;ggplot2-specs&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb16"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb16-1"><a href="visualization-1.html#cb16-1" aria-hidden="true" tabindex="-1"></a><span class="fu">vignette</span>(<span class="st">&quot;ggplot2-specs&quot;</span>)</span></code></pre></div>
 <div id="practice-plot-1" class="section level4 hasAnchor" number="2.5.4.1">
 <h4><span class="header-section-number">2.5.4.1</span> Practice plot 1<a href="visualization-1.html#practice-plot-1" class="anchor-section" aria-label="Anchor link to header"></a></h4>
 <p>Make a scatter plot that shows the relationship between the variables <code>depth</code> (on the x-axis), and <code>table</code> (on the y-axis). Take a look at the description for the <code>diamonds</code> dataset so you know what these different variables mean. Your plot should look like the one shown in Figure <a href="visualization-1.html#fig:practice-plot1">2.10</a>.</p>
-<div class="sourceCode" id="cb19"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb19-1"><a href="visualization-1.html#cb19-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make practice plot 1 here</span></span></code></pre></div>
-<div class="sourceCode" id="cb20"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb20-1"><a href="visualization-1.html#cb20-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot1.png&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb17"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb17-1"><a href="visualization-1.html#cb17-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make practice plot 1 here</span></span></code></pre></div>
+<div class="sourceCode" id="cb18"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb18-1"><a href="visualization-1.html#cb18-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot1.png&quot;</span>)</span></code></pre></div>
 <div class="figure" style="text-align: center"><span style="display:block;" id="fig:practice-plot1"></span>
 <img src="figures/vis1_practice_plot1.png" alt="Practice plot 1." width="95%" />
 <p class="caption">
@@ -1150,8 +1140,8 @@ <h4><span class="header-section-number">2.5.4.1</span> Practice plot 1<a href="v
 </p>
 </div>
 <p><strong>Advanced</strong>: A neat trick to get a better sense for the data here is to add transparency. Your plot should look like the one shown in Figure <a href="visualization-1.html#fig:practice-plot1a">2.11</a>.</p>
-<div class="sourceCode" id="cb21"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb21-1"><a href="visualization-1.html#cb21-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make advanced practice plot 1 here</span></span></code></pre></div>
-<div class="sourceCode" id="cb22"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb22-1"><a href="visualization-1.html#cb22-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot1a.png&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb19"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb19-1"><a href="visualization-1.html#cb19-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make advanced practice plot 1 here</span></span></code></pre></div>
+<div class="sourceCode" id="cb20"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb20-1"><a href="visualization-1.html#cb20-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot1a.png&quot;</span>)</span></code></pre></div>
 <div class="figure" style="text-align: center"><span style="display:block;" id="fig:practice-plot1a"></span>
 <img src="figures/vis1_practice_plot1a.png" alt="Practice plot 1." width="95%" />
 <p class="caption">
@@ -1163,72 +1153,72 @@ <h4><span class="header-section-number">2.5.4.1</span> Practice plot 1<a href="v
 <div id="line-plot" class="section level3 hasAnchor" number="2.5.5">
 <h3><span class="header-section-number">2.5.5</span> Line plot<a href="visualization-1.html#line-plot" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>What else do we know about the diamonds? We actually know the quality of how they were cut. The <code>cut</code> variable ranges from “Fair” to “Ideal”. First, let’s take a look at the relationship between <code>cut</code> and <code>price</code>. This time, we’ll make a line plot instead of a bar plot (just because we can).</p>
-<div class="sourceCode" id="cb23"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb23-1"><a href="visualization-1.html#cb23-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb23-2"><a href="visualization-1.html#cb23-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> cut,</span>
-<span id="cb23-3"><a href="visualization-1.html#cb23-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb23-4"><a href="visualization-1.html#cb23-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb23-5"><a href="visualization-1.html#cb23-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;line&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb21"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb21-1"><a href="visualization-1.html#cb21-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb21-2"><a href="visualization-1.html#cb21-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> cut,</span>
+<span id="cb21-3"><a href="visualization-1.html#cb21-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb21-4"><a href="visualization-1.html#cb21-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb21-5"><a href="visualization-1.html#cb21-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;line&quot;</span>)</span></code></pre></div>
 <pre><code>geom_path: Each group consists of only one observation. Do you need to adjust
 the group aesthetic?</code></pre>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-21-1.png" width="672" /></p>
+<p><img src="psych252_files/figure-html/unnamed-chunk-21-1.png" width="672" /></p>
 <p>Oops! All we did is that we replaced <code>x = color</code> with <code>x = cut</code>, and <code>geom = "bar"</code> with <code>geom = "line"</code>. However, the plot doesn’t look like expected (i.e. there is no real plot). What happened here? The reason is that the line plot needs to know which points to connect. The error message tells us that each group consists of only one observation. Let’s adjust the group aesthetic to fix this.</p>
-<div class="sourceCode" id="cb25"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb25-1"><a href="visualization-1.html#cb25-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb25-2"><a href="visualization-1.html#cb25-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> cut,</span>
-<span id="cb25-3"><a href="visualization-1.html#cb25-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
-<span id="cb25-4"><a href="visualization-1.html#cb25-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> <span class="dv">1</span>)) <span class="sc">+</span></span>
-<span id="cb25-5"><a href="visualization-1.html#cb25-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb25-6"><a href="visualization-1.html#cb25-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;line&quot;</span>)</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-22-1.png" width="672" /></p>
+<div class="sourceCode" id="cb23"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb23-1"><a href="visualization-1.html#cb23-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb23-2"><a href="visualization-1.html#cb23-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> cut,</span>
+<span id="cb23-3"><a href="visualization-1.html#cb23-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
+<span id="cb23-4"><a href="visualization-1.html#cb23-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> <span class="dv">1</span>)) <span class="sc">+</span></span>
+<span id="cb23-5"><a href="visualization-1.html#cb23-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb23-6"><a href="visualization-1.html#cb23-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;line&quot;</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-22-1.png" width="672" /></p>
 <p>By adding the parameter <code>group = 1</code> to <code>mapping = aes()</code>, we specify that we would like all the levels in <code>x = cut</code> to be treated as coming from the same group. The reason for this is that <code>cut</code> (our x-axis variable) is a factor (and not a numeric variable), so, by default, <code>ggplot2</code> tries to draw a separate line for each factor level. We’ll learn more about grouping below (and about factors later).</p>
 <p>Interestingly, there is no simple relationship between the quality of the cut and the price of the diamond. In fact, “Ideal” diamonds tend to be cheapest.</p>
 </div>
 <div id="adding-error-bars" class="section level3 hasAnchor" number="2.5.6">
 <h3><span class="header-section-number">2.5.6</span> Adding error bars<a href="visualization-1.html#adding-error-bars" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>We often don’t just want to show the means but also give a sense for how much the data varies. <code>ggplot2</code> has some convenient ways of specifying error bars. Let’s take a look at how much <code>price</code> varies as a function of <code>clarity</code> (another variable in our <code>diamonds</code> data frame).</p>
-<div class="sourceCode" id="cb26"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb26-1"><a href="visualization-1.html#cb26-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb26-2"><a href="visualization-1.html#cb26-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> clarity,</span>
-<span id="cb26-3"><a href="visualization-1.html#cb26-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb26-4"><a href="visualization-1.html#cb26-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
-<span id="cb26-5"><a href="visualization-1.html#cb26-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;pointrange&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb24"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb24-1"><a href="visualization-1.html#cb24-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb24-2"><a href="visualization-1.html#cb24-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> clarity,</span>
+<span id="cb24-3"><a href="visualization-1.html#cb24-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb24-4"><a href="visualization-1.html#cb24-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
+<span id="cb24-5"><a href="visualization-1.html#cb24-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;pointrange&quot;</span>)</span></code></pre></div>
 <div class="figure"><span style="display:block;" id="fig:errorbars-normal"></span>
-<img src="02-visualization1_files/figure-html/errorbars-normal-1.png" alt="Relationship between diamond clarity and price. Error bars indicate 95% bootstrapped confidence intervals." width="672" />
+<img src="psych252_files/figure-html/errorbars-normal-1.png" alt="Relationship between diamond clarity and price. Error bars indicate 95% bootstrapped confidence intervals." width="672" />
 <p class="caption">
 Figure 2.12: Relationship between diamond clarity and price. Error bars indicate 95% bootstrapped confidence intervals.
 </p>
 </div>
 <p>Here we have it. The average price of our diamonds for different levels of <code>clarity</code> together with bootstrapped 95% confidence intervals. How do we know that we have 95% confidence intervals? That’s what <code>mean_cl_boot()</code> computes as a default. Let’s take a look at that function:</p>
-<div class="sourceCode" id="cb27"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb27-1"><a href="visualization-1.html#cb27-1" aria-hidden="true" tabindex="-1"></a><span class="fu">help</span>(mean_cl_boot)</span></code></pre></div>
+<div class="sourceCode" id="cb25"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb25-1"><a href="visualization-1.html#cb25-1" aria-hidden="true" tabindex="-1"></a><span class="fu">help</span>(mean_cl_boot)</span></code></pre></div>
 <p>Note that I had to use the <code>fun.data =</code> argument here instead of <code>fun =</code> because the <code>mean_cl_boot()</code> function produces three data points for each value of the x-axis (the mean, lower, and upper confidence interval).</p>
 </div>
 <div id="order-matters" class="section level3 hasAnchor" number="2.5.7">
 <h3><span class="header-section-number">2.5.7</span> Order matters<a href="visualization-1.html#order-matters" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>The order in which we add geoms to a ggplot matters! Generally, we want to plot error bars before the points that represent the means. To illustrate, let’s set the color in which we show the means to “red”.</p>
-<div class="sourceCode" id="cb28"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb28-1"><a href="visualization-1.html#cb28-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb28-2"><a href="visualization-1.html#cb28-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> clarity,</span>
-<span id="cb28-3"><a href="visualization-1.html#cb28-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb28-4"><a href="visualization-1.html#cb28-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
-<span id="cb28-5"><a href="visualization-1.html#cb28-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>) <span class="sc">+</span></span>
-<span id="cb28-6"><a href="visualization-1.html#cb28-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb28-7"><a href="visualization-1.html#cb28-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;point&quot;</span>,</span>
-<span id="cb28-8"><a href="visualization-1.html#cb28-8" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;red&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb26"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb26-1"><a href="visualization-1.html#cb26-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb26-2"><a href="visualization-1.html#cb26-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> clarity,</span>
+<span id="cb26-3"><a href="visualization-1.html#cb26-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb26-4"><a href="visualization-1.html#cb26-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
+<span id="cb26-5"><a href="visualization-1.html#cb26-5" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>) <span class="sc">+</span></span>
+<span id="cb26-6"><a href="visualization-1.html#cb26-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb26-7"><a href="visualization-1.html#cb26-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;point&quot;</span>,</span>
+<span id="cb26-8"><a href="visualization-1.html#cb26-8" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;red&quot;</span>)</span></code></pre></div>
 <div class="figure"><span style="display:block;" id="fig:good-figure"></span>
-<img src="02-visualization1_files/figure-html/good-figure-1.png" alt="This figure looks good. Error bars and means are drawn in the correct order." width="672" />
+<img src="psych252_files/figure-html/good-figure-1.png" alt="This figure looks good. Error bars and means are drawn in the correct order." width="672" />
 <p class="caption">
 Figure 2.13: This figure looks good. Error bars and means are drawn in the correct order.
 </p>
 </div>
 <p>Figure <a href="visualization-1.html#fig:good-figure">2.13</a> looks good.</p>
-<div class="sourceCode" id="cb29"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb29-1"><a href="visualization-1.html#cb29-1" aria-hidden="true" tabindex="-1"></a><span class="co"># I&#39;ve changed the order in which the means and error bars are drawn.</span></span>
-<span id="cb29-2"><a href="visualization-1.html#cb29-2" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(df.diamonds,</span>
-<span id="cb29-3"><a href="visualization-1.html#cb29-3" aria-hidden="true" tabindex="-1"></a>       <span class="fu">aes</span>(<span class="at">x =</span> clarity,</span>
-<span id="cb29-4"><a href="visualization-1.html#cb29-4" aria-hidden="true" tabindex="-1"></a>           <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb29-5"><a href="visualization-1.html#cb29-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb29-6"><a href="visualization-1.html#cb29-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;point&quot;</span>,</span>
-<span id="cb29-7"><a href="visualization-1.html#cb29-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;red&quot;</span>) <span class="sc">+</span></span>
-<span id="cb29-8"><a href="visualization-1.html#cb29-8" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
-<span id="cb29-9"><a href="visualization-1.html#cb29-9" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb27"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb27-1"><a href="visualization-1.html#cb27-1" aria-hidden="true" tabindex="-1"></a><span class="co"># I&#39;ve changed the order in which the means and error bars are drawn.</span></span>
+<span id="cb27-2"><a href="visualization-1.html#cb27-2" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(df.diamonds,</span>
+<span id="cb27-3"><a href="visualization-1.html#cb27-3" aria-hidden="true" tabindex="-1"></a>       <span class="fu">aes</span>(<span class="at">x =</span> clarity,</span>
+<span id="cb27-4"><a href="visualization-1.html#cb27-4" aria-hidden="true" tabindex="-1"></a>           <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb27-5"><a href="visualization-1.html#cb27-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb27-6"><a href="visualization-1.html#cb27-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;point&quot;</span>,</span>
+<span id="cb27-7"><a href="visualization-1.html#cb27-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;red&quot;</span>) <span class="sc">+</span></span>
+<span id="cb27-8"><a href="visualization-1.html#cb27-8" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
+<span id="cb27-9"><a href="visualization-1.html#cb27-9" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>)</span></code></pre></div>
 <div class="figure"><span style="display:block;" id="fig:bad-figure"></span>
-<img src="02-visualization1_files/figure-html/bad-figure-1.png" alt="This figure looks bad. Error bars and means are drawn in the incorrect order." width="672" />
+<img src="psych252_files/figure-html/bad-figure-1.png" alt="This figure looks bad. Error bars and means are drawn in the incorrect order." width="672" />
 <p class="caption">
 Figure 2.14: This figure looks bad. Error bars and means are drawn in the incorrect order.
 </p>
@@ -1240,8 +1230,8 @@ <h3><span class="header-section-number">2.5.7</span> Order matters<a href="visua
 <div id="practice-plot-2" class="section level4 hasAnchor" number="2.5.7.1">
 <h4><span class="header-section-number">2.5.7.1</span> Practice plot 2<a href="visualization-1.html#practice-plot-2" class="anchor-section" aria-label="Anchor link to header"></a></h4>
 <p>Make a bar plot that shows the average <code>price</code> of diamonds (on the y-axis) as a function of their <code>clarity</code> (on the x-axis). Also add error bars. Your plot should look like the one shown in Figure <a href="visualization-1.html#fig:practice-plot2">2.15</a>.</p>
-<div class="sourceCode" id="cb30"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb30-1"><a href="visualization-1.html#cb30-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make practice plot 2 here</span></span></code></pre></div>
-<div class="sourceCode" id="cb31"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb31-1"><a href="visualization-1.html#cb31-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot2.png&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb28"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb28-1"><a href="visualization-1.html#cb28-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make practice plot 2 here</span></span></code></pre></div>
+<div class="sourceCode" id="cb29"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb29-1"><a href="visualization-1.html#cb29-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot2.png&quot;</span>)</span></code></pre></div>
 <div class="figure" style="text-align: center"><span style="display:block;" id="fig:practice-plot2"></span>
 <img src="figures/vis1_practice_plot2.png" alt="Practice plot 2." width="95%" />
 <p class="caption">
@@ -1249,8 +1239,8 @@ <h4><span class="header-section-number">2.5.7.1</span> Practice plot 2<a href="v
 </p>
 </div>
 <p><strong>Advanced</strong>: Try to make the plot shown in Figure <a href="visualization-1.html#fig:practice-plot2a">2.16</a>.</p>
-<div class="sourceCode" id="cb32"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb32-1"><a href="visualization-1.html#cb32-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make advanced practice plot 2 here</span></span></code></pre></div>
-<div class="sourceCode" id="cb33"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb33-1"><a href="visualization-1.html#cb33-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot2a.png&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb30"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb30-1"><a href="visualization-1.html#cb30-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make advanced practice plot 2 here</span></span></code></pre></div>
+<div class="sourceCode" id="cb31"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb31-1"><a href="visualization-1.html#cb31-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot2a.png&quot;</span>)</span></code></pre></div>
 <div class="figure" style="text-align: center"><span style="display:block;" id="fig:practice-plot2a"></span>
 <img src="figures/vis1_practice_plot2a.png" alt="Practice plot 2." width="95%" />
 <p class="caption">
@@ -1263,64 +1253,64 @@ <h4><span class="header-section-number">2.5.7.1</span> Practice plot 2<a href="v
 <h3><span class="header-section-number">2.5.8</span> Grouping data<a href="visualization-1.html#grouping-data" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>Grouping in <code>ggplot2</code> is a very powerful idea. It allows us to plot subsets of the data – again without the need to make separate data frames first.</p>
 <p>Let’s make a plot that shows the relationship between <code>price</code> and <code>color</code> separately for the different qualities of <code>cut</code>.</p>
+<div class="sourceCode" id="cb32"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb32-1"><a href="visualization-1.html#cb32-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb32-2"><a href="visualization-1.html#cb32-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
+<span id="cb32-3"><a href="visualization-1.html#cb32-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
+<span id="cb32-4"><a href="visualization-1.html#cb32-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> cut)) <span class="sc">+</span></span>
+<span id="cb32-5"><a href="visualization-1.html#cb32-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb32-6"><a href="visualization-1.html#cb32-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;line&quot;</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-26-1.png" width="672" /></p>
+<p>Well, we got some separate lines here but we don’t know which line corresponds to which cut. Let’s add some color!</p>
+<div class="sourceCode" id="cb33"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb33-1"><a href="visualization-1.html#cb33-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb33-2"><a href="visualization-1.html#cb33-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
+<span id="cb33-3"><a href="visualization-1.html#cb33-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
+<span id="cb33-4"><a href="visualization-1.html#cb33-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> cut,</span>
+<span id="cb33-5"><a href="visualization-1.html#cb33-5" aria-hidden="true" tabindex="-1"></a>                     <span class="at">color =</span> cut)) <span class="sc">+</span></span>
+<span id="cb33-6"><a href="visualization-1.html#cb33-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb33-7"><a href="visualization-1.html#cb33-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;line&quot;</span>,</span>
+<span id="cb33-8"><a href="visualization-1.html#cb33-8" aria-hidden="true" tabindex="-1"></a>               <span class="at">size =</span> <span class="dv">2</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-27-1.png" width="672" /></p>
+<p>Nice! In addition to adding color, I’ve made the lines a little thicker here by setting the <code>size</code> argument to 2.</p>
+<p>Grouping is very useful for bar plots. Let’s take a look at how the average price of diamonds looks like taking into account both <code>cut</code> and <code>color</code> (I know – exciting times!). Let’s put the <code>color</code> on the x-axis and then group by the <code>cut</code>.</p>
 <div class="sourceCode" id="cb34"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb34-1"><a href="visualization-1.html#cb34-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
 <span id="cb34-2"><a href="visualization-1.html#cb34-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
 <span id="cb34-3"><a href="visualization-1.html#cb34-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
-<span id="cb34-4"><a href="visualization-1.html#cb34-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> cut)) <span class="sc">+</span></span>
-<span id="cb34-5"><a href="visualization-1.html#cb34-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb34-6"><a href="visualization-1.html#cb34-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;line&quot;</span>)</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-26-1.png" width="672" /></p>
-<p>Well, we got some separate lines here but we don’t know which line corresponds to which cut. Let’s add some color!</p>
+<span id="cb34-4"><a href="visualization-1.html#cb34-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> cut,</span>
+<span id="cb34-5"><a href="visualization-1.html#cb34-5" aria-hidden="true" tabindex="-1"></a>                     <span class="at">color =</span> cut)) <span class="sc">+</span></span>
+<span id="cb34-6"><a href="visualization-1.html#cb34-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb34-7"><a href="visualization-1.html#cb34-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-28-1.png" width="672" /></p>
+<p>That’s a fail! Several things went wrong here. All the bars are gray and only their outline is colored differently. Instead we want the bars to have a different color. For that we need to specify the <code>fill</code> argument rather than the <code>color</code> argument! But things are worse. The bars currently are shown on top of each other. Instead, we’d like to put them next to each other. Here is how we can do that:</p>
 <div class="sourceCode" id="cb35"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb35-1"><a href="visualization-1.html#cb35-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
 <span id="cb35-2"><a href="visualization-1.html#cb35-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
 <span id="cb35-3"><a href="visualization-1.html#cb35-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
 <span id="cb35-4"><a href="visualization-1.html#cb35-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> cut,</span>
-<span id="cb35-5"><a href="visualization-1.html#cb35-5" aria-hidden="true" tabindex="-1"></a>                     <span class="at">color =</span> cut)) <span class="sc">+</span></span>
+<span id="cb35-5"><a href="visualization-1.html#cb35-5" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> cut)) <span class="sc">+</span></span>
 <span id="cb35-6"><a href="visualization-1.html#cb35-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb35-7"><a href="visualization-1.html#cb35-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;line&quot;</span>,</span>
-<span id="cb35-8"><a href="visualization-1.html#cb35-8" aria-hidden="true" tabindex="-1"></a>               <span class="at">size =</span> <span class="dv">2</span>)</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-27-1.png" width="672" /></p>
-<p>Nice! In addition to adding color, I’ve made the lines a little thicker here by setting the <code>size</code> argument to 2.</p>
-<p>Grouping is very useful for bar plots. Let’s take a look at how the average price of diamonds looks like taking into account both <code>cut</code> and <code>color</code> (I know – exciting times!). Let’s put the <code>color</code> on the x-axis and then group by the <code>cut</code>.</p>
+<span id="cb35-7"><a href="visualization-1.html#cb35-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>,</span>
+<span id="cb35-8"><a href="visualization-1.html#cb35-8" aria-hidden="true" tabindex="-1"></a>               <span class="at">position =</span> <span class="fu">position_dodge</span>())</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-29-1.png" width="672" /></p>
+<p>Neato! We’ve changed the <code>color</code> argument to <code>fill</code>, and have added the <code>position = position_dodge()</code> argument to the <code>stat_summary()</code> call. This argument makes it such that the bars are nicely dodged next to each other. Let’s add some error bars just for kicks.</p>
 <div class="sourceCode" id="cb36"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb36-1"><a href="visualization-1.html#cb36-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
 <span id="cb36-2"><a href="visualization-1.html#cb36-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
 <span id="cb36-3"><a href="visualization-1.html#cb36-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
 <span id="cb36-4"><a href="visualization-1.html#cb36-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> cut,</span>
-<span id="cb36-5"><a href="visualization-1.html#cb36-5" aria-hidden="true" tabindex="-1"></a>                     <span class="at">color =</span> cut)) <span class="sc">+</span></span>
+<span id="cb36-5"><a href="visualization-1.html#cb36-5" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> cut)) <span class="sc">+</span></span>
 <span id="cb36-6"><a href="visualization-1.html#cb36-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb36-7"><a href="visualization-1.html#cb36-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>)</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-28-1.png" width="672" /></p>
-<p>That’s a fail! Several things went wrong here. All the bars are gray and only their outline is colored differently. Instead we want the bars to have a different color. For that we need to specify the <code>fill</code> argument rather than the <code>color</code> argument! But things are worse. The bars currently are shown on top of each other. Instead, we’d like to put them next to each other. Here is how we can do that:</p>
-<div class="sourceCode" id="cb37"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb37-1"><a href="visualization-1.html#cb37-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb37-2"><a href="visualization-1.html#cb37-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
-<span id="cb37-3"><a href="visualization-1.html#cb37-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
-<span id="cb37-4"><a href="visualization-1.html#cb37-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> cut,</span>
-<span id="cb37-5"><a href="visualization-1.html#cb37-5" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> cut)) <span class="sc">+</span></span>
-<span id="cb37-6"><a href="visualization-1.html#cb37-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb37-7"><a href="visualization-1.html#cb37-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>,</span>
-<span id="cb37-8"><a href="visualization-1.html#cb37-8" aria-hidden="true" tabindex="-1"></a>               <span class="at">position =</span> <span class="fu">position_dodge</span>())</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-29-1.png" width="672" /></p>
-<p>Neato! We’ve changed the <code>color</code> argument to <code>fill</code>, and have added the <code>position = position_dodge()</code> argument to the <code>stat_summary()</code> call. This argument makes it such that the bars are nicely dodged next to each other. Let’s add some error bars just for kicks.</p>
-<div class="sourceCode" id="cb38"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb38-1"><a href="visualization-1.html#cb38-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb38-2"><a href="visualization-1.html#cb38-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> color,</span>
-<span id="cb38-3"><a href="visualization-1.html#cb38-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
-<span id="cb38-4"><a href="visualization-1.html#cb38-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> cut,</span>
-<span id="cb38-5"><a href="visualization-1.html#cb38-5" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> cut)) <span class="sc">+</span></span>
-<span id="cb38-6"><a href="visualization-1.html#cb38-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb38-7"><a href="visualization-1.html#cb38-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>,</span>
-<span id="cb38-8"><a href="visualization-1.html#cb38-8" aria-hidden="true" tabindex="-1"></a>               <span class="at">position =</span> <span class="fu">position_dodge</span>(<span class="at">width =</span> <span class="fl">0.9</span>),</span>
-<span id="cb38-9"><a href="visualization-1.html#cb38-9" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;black&quot;</span>) <span class="sc">+</span></span>
-<span id="cb38-10"><a href="visualization-1.html#cb38-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
-<span id="cb38-11"><a href="visualization-1.html#cb38-11" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>,</span>
-<span id="cb38-12"><a href="visualization-1.html#cb38-12" aria-hidden="true" tabindex="-1"></a>               <span class="at">position =</span> <span class="fu">position_dodge</span>(<span class="at">width =</span> <span class="fl">0.9</span>))</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-30-1.png" width="672" /></p>
+<span id="cb36-7"><a href="visualization-1.html#cb36-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>,</span>
+<span id="cb36-8"><a href="visualization-1.html#cb36-8" aria-hidden="true" tabindex="-1"></a>               <span class="at">position =</span> <span class="fu">position_dodge</span>(<span class="at">width =</span> <span class="fl">0.9</span>),</span>
+<span id="cb36-9"><a href="visualization-1.html#cb36-9" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;black&quot;</span>) <span class="sc">+</span></span>
+<span id="cb36-10"><a href="visualization-1.html#cb36-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
+<span id="cb36-11"><a href="visualization-1.html#cb36-11" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>,</span>
+<span id="cb36-12"><a href="visualization-1.html#cb36-12" aria-hidden="true" tabindex="-1"></a>               <span class="at">position =</span> <span class="fu">position_dodge</span>(<span class="at">width =</span> <span class="fl">0.9</span>))</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-30-1.png" width="672" /></p>
 <p>Voila! Now with error bars. Note that we’ve added the <code>width = 0.9</code> argument to <code>position_dodge()</code>. Somehow R was complaining when this was not defined for geom “linerange”. I’ve also added some outline to the bars by including the argument <code>color = "black"</code>. I think it looks nicer this way.</p>
 <p>So, still somewhat surprisingly, diamonds with the worst color (J) are more expensive than dimanods with the best color (D), and diamonds with better cuts are not necessarily more expensive.</p>
 <div id="practice-plot-3" class="section level4 hasAnchor" number="2.5.8.1">
 <h4><span class="header-section-number">2.5.8.1</span> Practice plot 3<a href="visualization-1.html#practice-plot-3" class="anchor-section" aria-label="Anchor link to header"></a></h4>
 <p>Recreate the plot shown in Figure <a href="visualization-1.html#fig:practice-plot3">2.17</a>.</p>
-<div class="sourceCode" id="cb39"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb39-1"><a href="visualization-1.html#cb39-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make practice plot 3 here</span></span></code></pre></div>
-<div class="sourceCode" id="cb40"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb40-1"><a href="visualization-1.html#cb40-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot3.png&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb37"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb37-1"><a href="visualization-1.html#cb37-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make practice plot 3 here</span></span></code></pre></div>
+<div class="sourceCode" id="cb38"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb38-1"><a href="visualization-1.html#cb38-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot3.png&quot;</span>)</span></code></pre></div>
 <div class="figure" style="text-align: center"><span style="display:block;" id="fig:practice-plot3"></span>
 <img src="figures/vis1_practice_plot3.png" alt="Practice plot 3." width="95%" />
 <p class="caption">
@@ -1328,8 +1318,8 @@ <h4><span class="header-section-number">2.5.8.1</span> Practice plot 3<a href="v
 </p>
 </div>
 <p><strong>Advanced</strong>: Try to recreate the plot show in in Figure <a href="visualization-1.html#fig:practice-plot3a">2.18</a>.</p>
-<div class="sourceCode" id="cb41"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb41-1"><a href="visualization-1.html#cb41-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make advanced practice plot 3 here</span></span></code></pre></div>
-<div class="sourceCode" id="cb42"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb42-1"><a href="visualization-1.html#cb42-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot3a.png&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb39"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb39-1"><a href="visualization-1.html#cb39-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make advanced practice plot 3 here</span></span></code></pre></div>
+<div class="sourceCode" id="cb40"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb40-1"><a href="visualization-1.html#cb40-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot3a.png&quot;</span>)</span></code></pre></div>
 <div class="figure" style="text-align: center"><span style="display:block;" id="fig:practice-plot3a"></span>
 <img src="figures/vis1_practice_plot3a.png" alt="Practice plot 3." width="95%" />
 <p class="caption">
@@ -1342,62 +1332,62 @@ <h4><span class="header-section-number">2.5.8.1</span> Practice plot 3<a href="v
 <h3><span class="header-section-number">2.5.9</span> Making facets<a href="visualization-1.html#making-facets" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p>Having too much information in a single plot can be overwhelming. The previous plot is already pretty busy. Facets are a nice way of splitting up plots and showing information in separate panels.</p>
 <p>Let’s take a look at how wide these diamonds tend to be. The width in mm is given in the <code>y</code> column of the diamonds data frame. We’ll make a histogram first. To make a histogram, the only aesthetic we needed to specify is <code>x</code>.</p>
-<div class="sourceCode" id="cb43"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb43-1"><a href="visualization-1.html#cb43-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb43-2"><a href="visualization-1.html#cb43-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y)) <span class="sc">+</span></span>
-<span id="cb43-3"><a href="visualization-1.html#cb43-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_histogram</span>()</span></code></pre></div>
+<div class="sourceCode" id="cb41"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb41-1"><a href="visualization-1.html#cb41-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb41-2"><a href="visualization-1.html#cb41-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y)) <span class="sc">+</span></span>
+<span id="cb41-3"><a href="visualization-1.html#cb41-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_histogram</span>()</span></code></pre></div>
 <pre><code>`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.</code></pre>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-33-1.png" width="672" /></p>
+<p><img src="psych252_files/figure-html/unnamed-chunk-33-1.png" width="672" /></p>
 <p>That looks bad! Let’s pick a different value for the width of the bins in the histogram.</p>
+<div class="sourceCode" id="cb43"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb43-1"><a href="visualization-1.html#cb43-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb43-2"><a href="visualization-1.html#cb43-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y)) <span class="sc">+</span></span>
+<span id="cb43-3"><a href="visualization-1.html#cb43-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_histogram</span>(<span class="at">binwidth =</span> <span class="fl">0.1</span>)</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-34-1.png" width="672" /></p>
+<p>Still bad. There seems to be an outlier diamond that happens to be almost 60 mm wide, while most of the rest is much narrower. One option would be to remove the outlier from the data before plotting it. But generally, we don’t want to make new data frames. Instead, let’s just limit what data we show in the plot.</p>
+<div class="sourceCode" id="cb44"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb44-1"><a href="visualization-1.html#cb44-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb44-2"><a href="visualization-1.html#cb44-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y)) <span class="sc">+</span></span>
+<span id="cb44-3"><a href="visualization-1.html#cb44-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_histogram</span>(<span class="at">binwidth =</span> <span class="fl">0.1</span>) <span class="sc">+</span></span>
+<span id="cb44-4"><a href="visualization-1.html#cb44-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-35-1.png" width="672" /></p>
+<p>I’ve used the <code>coord_cartesian()</code> function to restrict the range of data to show by passing a minimum and maximum to the <code>xlim</code> argument. This looks better now.</p>
+<p>Instead of histograms, we can also plot a density fitted to the distribution.</p>
 <div class="sourceCode" id="cb45"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb45-1"><a href="visualization-1.html#cb45-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
 <span id="cb45-2"><a href="visualization-1.html#cb45-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y)) <span class="sc">+</span></span>
-<span id="cb45-3"><a href="visualization-1.html#cb45-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_histogram</span>(<span class="at">binwidth =</span> <span class="fl">0.1</span>)</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-34-1.png" width="672" /></p>
-<p>Still bad. There seems to be an outlier diamond that happens to be almost 60 mm wide, while most of the rest is much narrower. One option would be to remove the outlier from the data before plotting it. But generally, we don’t want to make new data frames. Instead, let’s just limit what data we show in the plot.</p>
+<span id="cb45-3"><a href="visualization-1.html#cb45-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_density</span>() <span class="sc">+</span></span>
+<span id="cb45-4"><a href="visualization-1.html#cb45-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-36-1.png" width="672" /></p>
+<p>Looks pretty similar to our histogram above! Just like we can play around with the binwidth of the histogram, we can change the smoothing bandwidth of the kernel that is used to create the histogram. Here is a histogram with a much wider bandwidth:</p>
 <div class="sourceCode" id="cb46"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb46-1"><a href="visualization-1.html#cb46-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
 <span id="cb46-2"><a href="visualization-1.html#cb46-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y)) <span class="sc">+</span></span>
-<span id="cb46-3"><a href="visualization-1.html#cb46-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_histogram</span>(<span class="at">binwidth =</span> <span class="fl">0.1</span>) <span class="sc">+</span></span>
+<span id="cb46-3"><a href="visualization-1.html#cb46-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_density</span>(<span class="at">bw =</span> <span class="fl">0.5</span>) <span class="sc">+</span></span>
 <span id="cb46-4"><a href="visualization-1.html#cb46-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-35-1.png" width="672" /></p>
-<p>I’ve used the <code>coord_cartesian()</code> function to restrict the range of data to show by passing a minimum and maximum to the <code>xlim</code> argument. This looks better now.</p>
-<p>Instead of histograms, we can also plot a density fitted to the distribution.</p>
-<div class="sourceCode" id="cb47"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb47-1"><a href="visualization-1.html#cb47-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb47-2"><a href="visualization-1.html#cb47-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y)) <span class="sc">+</span></span>
-<span id="cb47-3"><a href="visualization-1.html#cb47-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_density</span>() <span class="sc">+</span></span>
-<span id="cb47-4"><a href="visualization-1.html#cb47-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-36-1.png" width="672" /></p>
-<p>Looks pretty similar to our histogram above! Just like we can play around with the binwidth of the histogram, we can change the smoothing bandwidth of the kernel that is used to create the histogram. Here is a histogram with a much wider bandwidth:</p>
-<div class="sourceCode" id="cb48"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb48-1"><a href="visualization-1.html#cb48-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb48-2"><a href="visualization-1.html#cb48-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y)) <span class="sc">+</span></span>
-<span id="cb48-3"><a href="visualization-1.html#cb48-3" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_density</span>(<span class="at">bw =</span> <span class="fl">0.5</span>) <span class="sc">+</span></span>
-<span id="cb48-4"><a href="visualization-1.html#cb48-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-37-1.png" width="672" /></p>
+<p><img src="psych252_files/figure-html/unnamed-chunk-37-1.png" width="672" /></p>
 <p>We’ll learn more about how these densities are determined later in class.</p>
 <p>I promised that this section was about making facets, right? We’re getting there! Let’s first take a look at how wide diamonds of different <code>color</code> are. We can use grouping to make this happen.</p>
-<div class="sourceCode" id="cb49"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb49-1"><a href="visualization-1.html#cb49-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb49-2"><a href="visualization-1.html#cb49-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y,</span>
-<span id="cb49-3"><a href="visualization-1.html#cb49-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> color,</span>
-<span id="cb49-4"><a href="visualization-1.html#cb49-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> color)) <span class="sc">+</span></span>
-<span id="cb49-5"><a href="visualization-1.html#cb49-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_density</span>(<span class="at">bw =</span> <span class="fl">0.2</span>,</span>
-<span id="cb49-6"><a href="visualization-1.html#cb49-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">alpha =</span> <span class="fl">0.2</span>) <span class="sc">+</span></span>
-<span id="cb49-7"><a href="visualization-1.html#cb49-7" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-38-1.png" width="672" /></p>
+<div class="sourceCode" id="cb47"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb47-1"><a href="visualization-1.html#cb47-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb47-2"><a href="visualization-1.html#cb47-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y,</span>
+<span id="cb47-3"><a href="visualization-1.html#cb47-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">group =</span> color,</span>
+<span id="cb47-4"><a href="visualization-1.html#cb47-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> color)) <span class="sc">+</span></span>
+<span id="cb47-5"><a href="visualization-1.html#cb47-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_density</span>(<span class="at">bw =</span> <span class="fl">0.2</span>,</span>
+<span id="cb47-6"><a href="visualization-1.html#cb47-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">alpha =</span> <span class="fl">0.2</span>) <span class="sc">+</span></span>
+<span id="cb47-7"><a href="visualization-1.html#cb47-7" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-38-1.png" width="672" /></p>
 <p>OK! That’s a little tricky to tell apart. Notice that I’ve specified the <code>alpha</code> argument in the <code>geom_density()</code> function so that the densities in the front don’t completely hide the densities in the back. But this plot still looks too busy. Instead of grouping, let’s put the densities for the different colors, in separate panels. That’s what facetting allows you to do.</p>
-<div class="sourceCode" id="cb50"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb50-1"><a href="visualization-1.html#cb50-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb50-2"><a href="visualization-1.html#cb50-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y,</span>
-<span id="cb50-3"><a href="visualization-1.html#cb50-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> color)) <span class="sc">+</span></span>
-<span id="cb50-4"><a href="visualization-1.html#cb50-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_density</span>(<span class="at">bw =</span> <span class="fl">0.2</span>) <span class="sc">+</span></span>
-<span id="cb50-5"><a href="visualization-1.html#cb50-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">facet_grid</span>(<span class="at">cols =</span> <span class="fu">vars</span>(color)) <span class="sc">+</span></span>
-<span id="cb50-6"><a href="visualization-1.html#cb50-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-39-1.png" width="672" /></p>
+<div class="sourceCode" id="cb48"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb48-1"><a href="visualization-1.html#cb48-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb48-2"><a href="visualization-1.html#cb48-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y,</span>
+<span id="cb48-3"><a href="visualization-1.html#cb48-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> color)) <span class="sc">+</span></span>
+<span id="cb48-4"><a href="visualization-1.html#cb48-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_density</span>(<span class="at">bw =</span> <span class="fl">0.2</span>) <span class="sc">+</span></span>
+<span id="cb48-5"><a href="visualization-1.html#cb48-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">facet_grid</span>(<span class="at">cols =</span> <span class="fu">vars</span>(color)) <span class="sc">+</span></span>
+<span id="cb48-6"><a href="visualization-1.html#cb48-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-39-1.png" width="672" /></p>
 <p>Now we have the densities next to each other in separate panels. I’ve removed the <code>alpha</code> argument since the densities aren’t overlapping anymore. To make the different panels, I used the <code>facet_grid()</code> function and specified that I want separate columns for the different colors (<code>cols = vars(color)</code>). What’s the deal with <code>vars()</code>? Why couldn’t we just write <code>facet_grid(cols = color)</code> instead? The short answer is: that’s what the function wants. The long answer is: long. (We’ll learn more about this later in the course.)</p>
 <p>To show the facets in different rows instead of columns we simply replace <code>cols = vars(color)</code> with <code>rows = vars(color)</code>.</p>
-<div class="sourceCode" id="cb51"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb51-1"><a href="visualization-1.html#cb51-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb51-2"><a href="visualization-1.html#cb51-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y,</span>
-<span id="cb51-3"><a href="visualization-1.html#cb51-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> color)) <span class="sc">+</span></span>
-<span id="cb51-4"><a href="visualization-1.html#cb51-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_density</span>(<span class="at">bw =</span> <span class="fl">0.2</span>) <span class="sc">+</span></span>
-<span id="cb51-5"><a href="visualization-1.html#cb51-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">facet_grid</span>(<span class="at">rows =</span> <span class="fu">vars</span>(color)) <span class="sc">+</span></span>
-<span id="cb51-6"><a href="visualization-1.html#cb51-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-40-1.png" width="672" /></p>
+<div class="sourceCode" id="cb49"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb49-1"><a href="visualization-1.html#cb49-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb49-2"><a href="visualization-1.html#cb49-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> y,</span>
+<span id="cb49-3"><a href="visualization-1.html#cb49-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> color)) <span class="sc">+</span></span>
+<span id="cb49-4"><a href="visualization-1.html#cb49-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_density</span>(<span class="at">bw =</span> <span class="fl">0.2</span>) <span class="sc">+</span></span>
+<span id="cb49-5"><a href="visualization-1.html#cb49-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">facet_grid</span>(<span class="at">rows =</span> <span class="fu">vars</span>(color)) <span class="sc">+</span></span>
+<span id="cb49-6"><a href="visualization-1.html#cb49-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">coord_cartesian</span>(<span class="at">xlim =</span> <span class="fu">c</span>(<span class="dv">3</span>, <span class="dv">10</span>))</span></code></pre></div>
+<p><img src="psych252_files/figure-html/unnamed-chunk-40-1.png" width="672" /></p>
 <p>Several aspects about this plot should be improved:</p>
 <ul>
 <li>the y-axis text is overlapping</li>
@@ -1406,22 +1396,22 @@ <h3><span class="header-section-number">2.5.9</span> Making facets<a href="visua
 </ul>
 <p>So, what does this plot actually show us? Well, J-colored diamonds tend to be wider than D-colored diamonds. Fascinating!</p>
 <p>Of course, we could go completely overboard with facets and groups. So let’s do it! Let’s look at how the average <code>price</code> (somewhat more interesting) varies as a function of <code>color</code>, <code>cut</code>, and <code>clarity</code>. We’ll put color on the x-axis, and make separate rows for <code>cut</code> and columns for <code>clarity</code>.</p>
-<div class="sourceCode" id="cb52"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb52-1"><a href="visualization-1.html#cb52-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb52-2"><a href="visualization-1.html#cb52-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">y =</span> price,</span>
-<span id="cb52-3"><a href="visualization-1.html#cb52-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">x =</span> color,</span>
-<span id="cb52-4"><a href="visualization-1.html#cb52-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> color)) <span class="sc">+</span></span>
-<span id="cb52-5"><a href="visualization-1.html#cb52-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
-<span id="cb52-6"><a href="visualization-1.html#cb52-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>,</span>
-<span id="cb52-7"><a href="visualization-1.html#cb52-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;black&quot;</span>) <span class="sc">+</span></span>
-<span id="cb52-8"><a href="visualization-1.html#cb52-8" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
-<span id="cb52-9"><a href="visualization-1.html#cb52-9" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>) <span class="sc">+</span></span>
-<span id="cb52-10"><a href="visualization-1.html#cb52-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">facet_grid</span>(<span class="at">rows =</span> <span class="fu">vars</span>(cut),</span>
-<span id="cb52-11"><a href="visualization-1.html#cb52-11" aria-hidden="true" tabindex="-1"></a>             <span class="at">cols =</span> <span class="fu">vars</span>(clarity))</span></code></pre></div>
+<div class="sourceCode" id="cb50"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb50-1"><a href="visualization-1.html#cb50-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb50-2"><a href="visualization-1.html#cb50-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">y =</span> price,</span>
+<span id="cb50-3"><a href="visualization-1.html#cb50-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">x =</span> color,</span>
+<span id="cb50-4"><a href="visualization-1.html#cb50-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">fill =</span> color)) <span class="sc">+</span></span>
+<span id="cb50-5"><a href="visualization-1.html#cb50-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun =</span> <span class="st">&quot;mean&quot;</span>,</span>
+<span id="cb50-6"><a href="visualization-1.html#cb50-6" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;bar&quot;</span>,</span>
+<span id="cb50-7"><a href="visualization-1.html#cb50-7" aria-hidden="true" tabindex="-1"></a>               <span class="at">color =</span> <span class="st">&quot;black&quot;</span>) <span class="sc">+</span></span>
+<span id="cb50-8"><a href="visualization-1.html#cb50-8" aria-hidden="true" tabindex="-1"></a>  <span class="fu">stat_summary</span>(<span class="at">fun.data =</span> <span class="st">&quot;mean_cl_boot&quot;</span>,</span>
+<span id="cb50-9"><a href="visualization-1.html#cb50-9" aria-hidden="true" tabindex="-1"></a>               <span class="at">geom =</span> <span class="st">&quot;linerange&quot;</span>) <span class="sc">+</span></span>
+<span id="cb50-10"><a href="visualization-1.html#cb50-10" aria-hidden="true" tabindex="-1"></a>  <span class="fu">facet_grid</span>(<span class="at">rows =</span> <span class="fu">vars</span>(cut),</span>
+<span id="cb50-11"><a href="visualization-1.html#cb50-11" aria-hidden="true" tabindex="-1"></a>             <span class="at">cols =</span> <span class="fu">vars</span>(clarity))</span></code></pre></div>
 <pre><code>Warning: Removed 1 rows containing missing values (geom_segment).</code></pre>
 <pre><code>Warning: Removed 3 rows containing missing values (geom_segment).</code></pre>
 <pre><code>Warning: Removed 1 rows containing missing values (geom_segment).</code></pre>
 <div class="figure"><span style="display:block;" id="fig:stretching-it"></span>
-<img src="02-visualization1_files/figure-html/stretching-it-1.png" alt="A figure that is stretching it in terms of information." width="672" />
+<img src="psych252_files/figure-html/stretching-it-1.png" alt="A figure that is stretching it in terms of information." width="672" />
 <p class="caption">
 Figure 2.19: A figure that is stretching it in terms of information.
 </p>
@@ -1430,8 +1420,8 @@ <h3><span class="header-section-number">2.5.9</span> Making facets<a href="visua
 <div id="practice-plot-4" class="section level4 hasAnchor" number="2.5.9.1">
 <h4><span class="header-section-number">2.5.9.1</span> Practice plot 4<a href="visualization-1.html#practice-plot-4" class="anchor-section" aria-label="Anchor link to header"></a></h4>
 <p>Recreate the plot shown in Figure <a href="visualization-1.html#fig:practice-plot4">2.20</a>.</p>
-<div class="sourceCode" id="cb56"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb56-1"><a href="visualization-1.html#cb56-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make practice plot 4 here</span></span></code></pre></div>
-<div class="sourceCode" id="cb57"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb57-1"><a href="visualization-1.html#cb57-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot4.png&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb54"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb54-1"><a href="visualization-1.html#cb54-1" aria-hidden="true" tabindex="-1"></a><span class="co"># make practice plot 4 here</span></span></code></pre></div>
+<div class="sourceCode" id="cb55"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb55-1"><a href="visualization-1.html#cb55-1" aria-hidden="true" tabindex="-1"></a><span class="fu">include_graphics</span>(<span class="st">&quot;figures/vis1_practice_plot4.png&quot;</span>)</span></code></pre></div>
 <div class="figure" style="text-align: center"><span style="display:block;" id="fig:practice-plot4"></span>
 <img src="figures/vis1_practice_plot4.png" alt="Practice plot 4." width="95%" />
 <p class="caption">
@@ -1443,35 +1433,35 @@ <h4><span class="header-section-number">2.5.9.1</span> Practice plot 4<a href="v
 <div id="global-local-and-setting-aes" class="section level3 hasAnchor" number="2.5.10">
 <h3><span class="header-section-number">2.5.10</span> Global, local, and setting <code>aes()</code><a href="visualization-1.html#global-local-and-setting-aes" class="anchor-section" aria-label="Anchor link to header"></a></h3>
 <p><code>ggplot2</code> allows you to specify the plot aesthetics in different ways.</p>
-<div class="sourceCode" id="cb58"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb58-1"><a href="visualization-1.html#cb58-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb58-2"><a href="visualization-1.html#cb58-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> carat,</span>
-<span id="cb58-3"><a href="visualization-1.html#cb58-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
-<span id="cb58-4"><a href="visualization-1.html#cb58-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">color =</span> color)) <span class="sc">+</span></span>
-<span id="cb58-5"><a href="visualization-1.html#cb58-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>() <span class="sc">+</span></span>
-<span id="cb58-6"><a href="visualization-1.html#cb58-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>,</span>
-<span id="cb58-7"><a href="visualization-1.html#cb58-7" aria-hidden="true" tabindex="-1"></a>              <span class="at">se =</span> F)</span></code></pre></div>
+<div class="sourceCode" id="cb56"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb56-1"><a href="visualization-1.html#cb56-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb56-2"><a href="visualization-1.html#cb56-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> carat,</span>
+<span id="cb56-3"><a href="visualization-1.html#cb56-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
+<span id="cb56-4"><a href="visualization-1.html#cb56-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">color =</span> color)) <span class="sc">+</span></span>
+<span id="cb56-5"><a href="visualization-1.html#cb56-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>() <span class="sc">+</span></span>
+<span id="cb56-6"><a href="visualization-1.html#cb56-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>,</span>
+<span id="cb56-7"><a href="visualization-1.html#cb56-7" aria-hidden="true" tabindex="-1"></a>              <span class="at">se =</span> F)</span></code></pre></div>
 <pre><code>`geom_smooth()` using formula &#39;y ~ x&#39;</code></pre>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-42-1.png" width="672" /></p>
+<p><img src="psych252_files/figure-html/unnamed-chunk-42-1.png" width="672" /></p>
 <p>Here, I’ve drawn a scatter plot of the relationship between <code>carat</code> and <code>price</code>, and I have added the best-fitting regression lines via the <code>geom_smooth(method = "lm")</code> call. (We will learn more about what these regression lines mean later in class.)</p>
 <p>Because I have defined all the aesthetics at the top level (i.e. directly within the <code>ggplot()</code> function), the aesthetics apply to all the functions afterwards. Aesthetics defined in the <code>ggplot()</code> call are <strong>global</strong>. In this case, the <code>geom_point()</code> and the <code>geom_smooth()</code> functions. The <code>geom_smooth()</code> function produces separate best-fit regression lines for each different color.</p>
 <p>But what if we only wanted to show one regression line instead that applies to all the data? Here is one way of doing so:</p>
-<div class="sourceCode" id="cb60"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb60-1"><a href="visualization-1.html#cb60-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb60-2"><a href="visualization-1.html#cb60-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> carat,</span>
-<span id="cb60-3"><a href="visualization-1.html#cb60-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
-<span id="cb60-4"><a href="visualization-1.html#cb60-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>(<span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">color =</span> color)) <span class="sc">+</span></span>
-<span id="cb60-5"><a href="visualization-1.html#cb60-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb58"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb58-1"><a href="visualization-1.html#cb58-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb58-2"><a href="visualization-1.html#cb58-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> carat,</span>
+<span id="cb58-3"><a href="visualization-1.html#cb58-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price)) <span class="sc">+</span></span>
+<span id="cb58-4"><a href="visualization-1.html#cb58-4" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>(<span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">color =</span> color)) <span class="sc">+</span></span>
+<span id="cb58-5"><a href="visualization-1.html#cb58-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>)</span></code></pre></div>
 <pre><code>`geom_smooth()` using formula &#39;y ~ x&#39;</code></pre>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-43-1.png" width="672" /></p>
+<p><img src="psych252_files/figure-html/unnamed-chunk-43-1.png" width="672" /></p>
 <p>Here, I’ve moved the color aesthetic into the <code>geom_point()</code> function call. Now, the <code>x</code> and <code>y</code> aesthetics still apply to both the <code>geom_point()</code> and the <code>geom_smooth()</code> function call (they are <strong>global</strong>), but the <code>color</code> aesthetic applies only to <code>geom_point()</code> (it is <strong>local</strong>). Alternatively, we can simply overwrite global aesthetics within local function calls.</p>
-<div class="sourceCode" id="cb62"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb62-1"><a href="visualization-1.html#cb62-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
-<span id="cb62-2"><a href="visualization-1.html#cb62-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> carat,</span>
-<span id="cb62-3"><a href="visualization-1.html#cb62-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
-<span id="cb62-4"><a href="visualization-1.html#cb62-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">color =</span> color)) <span class="sc">+</span></span>
-<span id="cb62-5"><a href="visualization-1.html#cb62-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>() <span class="sc">+</span></span>
-<span id="cb62-6"><a href="visualization-1.html#cb62-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>,</span>
-<span id="cb62-7"><a href="visualization-1.html#cb62-7" aria-hidden="true" tabindex="-1"></a>              <span class="at">color =</span> <span class="st">&quot;black&quot;</span>)</span></code></pre></div>
+<div class="sourceCode" id="cb60"><pre class="sourceCode r"><code class="sourceCode r"><span id="cb60-1"><a href="visualization-1.html#cb60-1" aria-hidden="true" tabindex="-1"></a><span class="fu">ggplot</span>(<span class="at">data =</span> df.diamonds,</span>
+<span id="cb60-2"><a href="visualization-1.html#cb60-2" aria-hidden="true" tabindex="-1"></a>       <span class="at">mapping =</span> <span class="fu">aes</span>(<span class="at">x =</span> carat,</span>
+<span id="cb60-3"><a href="visualization-1.html#cb60-3" aria-hidden="true" tabindex="-1"></a>                     <span class="at">y =</span> price,</span>
+<span id="cb60-4"><a href="visualization-1.html#cb60-4" aria-hidden="true" tabindex="-1"></a>                     <span class="at">color =</span> color)) <span class="sc">+</span></span>
+<span id="cb60-5"><a href="visualization-1.html#cb60-5" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_point</span>() <span class="sc">+</span></span>
+<span id="cb60-6"><a href="visualization-1.html#cb60-6" aria-hidden="true" tabindex="-1"></a>  <span class="fu">geom_smooth</span>(<span class="at">method =</span> <span class="st">&quot;lm&quot;</span>,</span>
+<span id="cb60-7"><a href="visualization-1.html#cb60-7" aria-hidden="true" tabindex="-1"></a>              <span class="at">color =</span> <span class="st">&quot;black&quot;</span>)</span></code></pre></div>
 <pre><code>`geom_smooth()` using formula &#39;y ~ x&#39;</code></pre>
-<p><img src="02-visualization1_files/figure-html/unnamed-chunk-44-1.png" width="672" /></p>
+<p><img src="psych252_files/figure-html/unnamed-chunk-44-1.png" width="672" /></p>
 <p>Here, I’ve set <code>color = "black"</code> within the <code>geom_smooth()</code> function, and now only one overall regression line is displayed since the global color aesthetic was overwritten in the local function call.</p>
 </div>
 </div>
@@ -1575,9 +1565,9 @@ <h2><span class="header-section-number">2.7</span> Session info<a href="visualiz
 [76] broom_0.8.0         survival_3.3-1      viridisLite_0.4.0  
 [79] cluster_2.1.3       ellipsis_0.3.2     </code></pre>
 <div class="figure"><span style="display:block;" id="fig:unnamed-chunk-46"></span>
-<img src="figures/reproducibility_court.jpg" alt="Defense at the reproducibility court." width="95%" />
+<img src="figures/reproducibility_court.jpg" alt="Defense at the reproducibility court (graphic by [Allison Horst](https://github.com/allisonhorst))." width="95%" />
 <p class="caption">
-Figure 2.21: Defense at the reproducibility court.
+Figure 2.21: Defense at the reproducibility court (graphic by <a href="https://github.com/allisonhorst">Allison Horst</a>).
 </p>
 </div>