diff --git a/_quarto.yml b/_quarto.yml
index 560e94e..29ae34b 100644
--- a/_quarto.yml
+++ b/_quarto.yml
@@ -6,12 +6,12 @@ execute:
   cache: true
   
 book:
-  title: Advanced Data Science
+  title: Introduction to Data Science
   subtitle: Statistics and Prediction Algorithms Through Case Studies
   reader-mode: true
   page-footer:
     left: |
-      Advanced Data Science was written by Rafael A. Irizarry
+      Introduction to Data Science was written by Rafael A. Irizarry
     right: |
       This book was built with <a href="https://quarto.org/">Quarto</a>.
   site-url: http://rafalab.dfci.harvard.edu/dsbook-part-2
diff --git a/docs/highdim/dimension-reduction.html b/docs/highdim/dimension-reduction.html
index ca2c60b..d64ded5 100644
--- a/docs/highdim/dimension-reduction.html
+++ b/docs/highdim/dimension-reduction.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 22&nbsp; Dimension reduction</title>
+<title>Introduction to Data Science - 22&nbsp; Dimension reduction</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -748,7 +748,7 @@ <h1 class="title">
 <span><span class="fu"><a href="https://rdrr.io/r/graphics/hist.html">hist</a></span><span class="op">(</span><span class="va">z</span><span class="op">[</span>,<span class="fl">1</span><span class="op">]</span>, breaks <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="op">-</span><span class="fl">4</span>,<span class="fl">4</span>,<span class="fl">0.5</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>We can visualize these to see how the first component summarizes the data. In the plot below, red represents high values and blue negative values:</p>
-<div class="cell" data-layout-align="center" height="5" data-hash="dimension-reduction_cache/html/illustrate-pca-twin-heights_f45b9d5d17ca2b561c3de6d50b0cddfd">
+<div class="cell" data-layout-align="center" height="5" data-hash="dimension-reduction_cache/html/illustrate-pca-twin-heights_f66700fe465dd08f100048303c90bfdd">
 <div class="cell-output-display">
 <div class="quarto-figure quarto-figure-center">
 <figure class="figure"><p><img src="dimension-reduction_files/figure-html/illustrate-pca-twin-heights-1.png" class="img-fluid figure-img" style="width:70.0%"></p>
@@ -1213,7 +1213,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/highdim/dimension-reduction_files/figure-html/illustrate-pca-twin-heights-1.png b/docs/highdim/dimension-reduction_files/figure-html/illustrate-pca-twin-heights-1.png
index 0502606..a892810 100644
Binary files a/docs/highdim/dimension-reduction_files/figure-html/illustrate-pca-twin-heights-1.png and b/docs/highdim/dimension-reduction_files/figure-html/illustrate-pca-twin-heights-1.png differ
diff --git a/docs/highdim/intro-highdim.html b/docs/highdim/intro-highdim.html
index b3993de..add743d 100644
--- a/docs/highdim/intro-highdim.html
+++ b/docs/highdim/intro-highdim.html
@@ -8,7 +8,7 @@
 
 <meta name="author" content="Rafael A. Irizarry">
 
-<title>Advanced Data Science - High dimensional data</title>
+<title>Introduction to Data Science - High dimensional data</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -91,7 +91,7 @@
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto">
     <div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -662,7 +662,7 @@ <h1 class="title">High dimensional data</h1>
 </div> <!-- /content -->
 <footer class="footer">
   <div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     <div class="toc-actions"><div><i class="bi bi-github"></i></div><div class="action-links"><p><a href="https://github.com/rafalab/dsbook-part-2/blob/main/highdim/intro-highdim.qmd" class="toc-action">View source</a></p><p><a href="https://github.com/rafalab/dsbook-part-2/issues/new" class="toc-action">Report an issue</a></p></div></div></div>
diff --git a/docs/highdim/linear-algebra.html b/docs/highdim/linear-algebra.html
index 0ffd727..a30da4b 100644
--- a/docs/highdim/linear-algebra.html
+++ b/docs/highdim/linear-algebra.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 21&nbsp; Applied Linear Algebra</title>
+<title>Introduction to Data Science - 21&nbsp; Applied Linear Algebra</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -958,7 +958,7 @@ <h1 class="title"><span id="sec-matrix-algebra" class="quarto-section-identifier
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/highdim/matrices-in-R.html b/docs/highdim/matrices-in-R.html
index 5f7a33b..87d8b5c 100644
--- a/docs/highdim/matrices-in-R.html
+++ b/docs/highdim/matrices-in-R.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 20&nbsp; Matrices in R</title>
+<title>Introduction to Data Science - 20&nbsp; Matrices in R</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -1166,7 +1166,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/highdim/matrix-factorization.html b/docs/highdim/matrix-factorization.html
index 1345e8d..c12a078 100644
--- a/docs/highdim/matrix-factorization.html
+++ b/docs/highdim/matrix-factorization.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 24&nbsp; Matrix Factorization</title>
+<title>Introduction to Data Science - 24&nbsp; Matrix Factorization</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -652,7 +652,7 @@ <h1 class="title">
 <div class="cell" data-layout-align="center" data-hash="matrix-factorization_cache/html/unnamed-chunk-15_51dcda555208594fd5d18b0725d9acbe">
 <div class="sourceCode" id="cb17"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="http://factominer.free.fr/missMDA/index.html">missMDA</a></span><span class="op">)</span></span>
 <span><span class="va">ind</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/colSums.html">colSums</a></span><span class="op">(</span><span class="op">!</span><span class="fu"><a href="https://rdrr.io/r/base/NA.html">is.na</a></span><span class="op">(</span><span class="va">y</span><span class="op">)</span><span class="op">)</span> <span class="op">&gt;=</span> <span class="fl">25</span> <span class="op">|</span> <span class="fu"><a href="https://rdrr.io/r/base/colnames.html">colnames</a></span><span class="op">(</span><span class="va">y</span><span class="op">)</span> <span class="op">==</span> <span class="st">"3252"</span></span>
-<span><span class="va">imputed</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/missMDA/man/imputePCA.html">imputePCA</a></span><span class="op">(</span><span class="va">r</span><span class="op">[</span>,<span class="va">ind</span><span class="op">]</span>, ncp <span class="op">=</span> <span class="fl">2</span>, coeff.ridge <span class="op">=</span> <span class="va">lambda</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<span><span class="va">imputed</span> <span class="op">&lt;-</span> <span class="fu">imputePCA</span><span class="op">(</span><span class="va">r</span><span class="op">[</span>,<span class="va">ind</span><span class="op">]</span>, ncp <span class="op">=</span> <span class="fl">2</span>, coeff.ridge <span class="op">=</span> <span class="va">lambda</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>To see how much we improve our previous prediction, we construct a matrix with the ratings in the test set:</p>
 <div class="cell" data-layout-align="center" data-hash="matrix-factorization_cache/html/unnamed-chunk-16_859c6077f27749e51e39ea9980cac6e1">
@@ -1120,7 +1120,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/highdim/regularization.html b/docs/highdim/regularization.html
index 963e28b..769416b 100644
--- a/docs/highdim/regularization.html
+++ b/docs/highdim/regularization.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 23&nbsp; Regularization</title>
+<title>Introduction to Data Science - 23&nbsp; Regularization</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -110,7 +110,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -1161,7 +1161,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/index.html b/docs/index.html
index 8df4982..a547919 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -8,7 +8,7 @@
 
 <meta name="author" content="Rafael A. Irizarry">
 
-<title>Advanced Data Science</title>
+<title>Introduction to Data Science</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -88,7 +88,7 @@
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto">
     <div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="./">Advanced Data Science</a> 
+      <a href="./">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -396,7 +396,7 @@ <h2 id="toc-title">Table of contents</h2>
 
 <header id="title-block-header" class="quarto-title-block default">
 <div class="quarto-title">
-<h1 class="title">Advanced Data Science</h1>
+<h1 class="title">Introduction to Data Science</h1>
 <p class="subtitle lead">Statistics and Prediction Algorithms Through Case Studies</p>
 </div>
 
@@ -414,11 +414,11 @@ <h1 class="title">Advanced Data Science</h1>
 
 <section id="preface" class="level1 unnumbered">
 <h1 class="unnumbered">Preface</h1>
-<p>This is the website for the <strong>Advanced Data Science</strong>.</p>
-<p>The website for <strong>Introduction to Data Science</strong> is <a href="http://rafalab.dfci.harvard.edu/dsbook-part-1/">here</a>.</p>
+<p>This is the website for the <strong>Statistics and Prediction Algorithms Through Case Studies</strong> part of <strong>Introduction to Data Science</strong>.</p>
+<p>The website for the <strong>Data Wrangling and Visualization with R</strong> is <a href="http://rafalab.dfci.harvard.edu/dsbook-part-1/">here</a>.</p>
 <p>This book started out as part of the class notes used in the HarvardX <a href="https://www.edx.org/professional-certificate/harvardx-data-science">Data Science Series</a><a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a>.</p>
-<p>A hardcopy version of the first edition of the book, which combined both Introduction and Advanced parts, is available from <a href="https://www.routledge.com/Introduction-to-Data-Science-Data-Analysis-and-Prediction-Algorithms-with/Irizarry/p/book/9780367357986?utm_source=author&amp;utm_medium=shared_link&amp;utm_campaign=B043135_jm1_5ll_6rm_t081_1al_introductiontodatascienceauthorshare">CRC Press</a><a href="#fn2" class="footnote-ref" id="fnref2" role="doc-noteref"><sup>2</sup></a>.</p>
-<p>A free PDF of the October 24, 2019 version of the book, which combined both Introduction and Advanced parts, is available from <a href="https://leanpub.com/datasciencebook">Leanpub</a><a href="#fn3" class="footnote-ref" id="fnref3" role="doc-noteref"><sup>3</sup></a>.</p>
+<p>A hardcopy version of the first edition of the book, which combined both parts, is available from <a href="https://www.routledge.com/Introduction-to-Data-Science-Data-Analysis-and-Prediction-Algorithms-with/Irizarry/p/book/9780367357986?utm_source=author&amp;utm_medium=shared_link&amp;utm_campaign=B043135_jm1_5ll_6rm_t081_1al_introductiontodatascienceauthorshare">CRC Press</a><a href="#fn2" class="footnote-ref" id="fnref2" role="doc-noteref"><sup>2</sup></a>.</p>
+<p>A free PDF of the October 24, 2019 version of the book, which combined both parts, is available from <a href="https://leanpub.com/datasciencebook">Leanpub</a><a href="#fn3" class="footnote-ref" id="fnref3" role="doc-noteref"><sup>3</sup></a>.</p>
 <p>The Quarto code used to generate the book is available on <a href="https://github.com/rafalab/dsbook-part-2">GitHub</a><a href="#fn4" class="footnote-ref" id="fnref4" role="doc-noteref"><sup>4</sup></a>. Note that, the graphical theme used for plots throughout the book can be recreated using the <code>ds_theme_set()</code> function from <strong>dslabs</strong> package.</p>
 <p>This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International <a href="https://creativecommons.org/licenses/by-nc-sa/4.0">CC BY-NC-SA 4.0</a>.</p>
 <p>We make announcements related to the book on Twitter. For updates follow <a href="https://twitter.com/rafalab">@rafalab</a>.</p>
@@ -427,7 +427,7 @@ <h1 class="unnumbered">Preface</h1>
 <h1 class="unnumbered">Acknowledgments</h1>
 <p>A special thanks to my tidyverse guru David Robinson and Amy Gill for dozens of comments, edits, and suggestions. Also, many thanks to Stephanie Hicks who twice served as a co-instructor in my data science classes and Yihui Xie who patiently put up with my many questions about bookdown. Thanks also to Héctor Corrada-Bravo, for advice on how to best teach machine learning. Thanks to Alyssa Frazee for helping create the homework problem that became the Recommendation Systems case study. Also, many thanks to Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund for making the Quarto code for their R for Data Science book open. Finally, thanks to Alex Nones for proofreading the manuscript during its various stages.</p>
 <p>This book was conceived during the teaching of several applied statistics courses, starting over fifteen years ago. The teaching assistants working with me throughout the years made important indirect contributions to this book. The latest iteration of this course is a HarvardX series coordinated by Heather Sternshein and Zofia Gajdos. We thank them for their contributions. We are also grateful to all the students whose questions and comments helped us improve the book. The courses were partially funded by NIH grant R25GM114818. We are very grateful to the National Institutes of Health for its support.</p>
-<p>A special thanks goes to all those who edited the book via GitHub pull requests or made suggestions by creating an <em>issue</em> or sending an email: <code>nickyfoto</code> (Huang Qiang), <code>desautm</code> (Marc-André Désautels), <code>michaschwab</code> (Michail Schwab), <code>alvarolarreategui</code> (Alvaro Larreategui), <code>jakevc</code> (Jake VanCampen), <code>omerta</code> (Guillermo Lengemann), <code>espinielli</code> (Enrico Spinielli), <code>asimumba</code>(Aaron Simumba), <code>braunschweig</code> (Maldewar), <code>gwierzchowski</code> (Grzegorz Wierzchowski), <code>technocrat</code> (Richard Careaga), <code>atzakas</code>, <code>defeit</code> (David Emerson Feit), <code>shiraamitchell</code> (Shira Mitchell), <code>Nathalie-S</code>, <code>andreashandel</code> (Andreas Handel), <code>berkowitze</code> (Elias Berkowitz), <code>Dean-Webb</code> (Dean Webber), <code>mohayusuf</code>, <code>jimrothstein</code>, <code>mPloenzke</code> (Matthew Ploenzke), <code>NicholasDowand</code> (Nicholas Dow), <code>kant</code> (Darío Hereñú), <code>debbieyuster</code> (Debbie Yuster), <code>tuanchauict</code> (Tuan Chau), <code>phzeller</code>, <code>BTJ01</code> (BradJ), <code>glsnow</code> (Greg Snow), <code>mberlanda</code> (Mauro Berlanda), <code>wfan9</code>, <code>larswestvang</code> (Lars Westvang), <code>jj999</code> (Jan Andrejkovic), <code>Kriegslustig</code> (Luca Nils Schmid), <code>odahhani</code>, <code>aidanhorn</code> (Aidan Horn), <code>atraxler</code> (Adrienne Traxler), <code>alvegorova</code>,<code>wycheong</code> (Won Young Cheong), <code>med-hat</code> (Medhat Khalil), <code>kengustafson</code>, <code>Yowza63</code>, <code>ryan-heslin</code> (Ryan Heslin), <code>raffaem</code>, <code>tim8west</code>, David D. Kane, El Mustapha El Abbassi, Vadim Zipunnikov, Anna Quaglieri, Chris Dong, Rick Schoenberg, Isabella Grabski, and Doug Snyder.</p>
+<p>A special thanks goes to all those who edited the book via GitHub pull requests or made suggestions by creating an <em>issue</em> or sending an email: <code>nickyfoto</code> (Huang Qiang), <code>desautm</code> (Marc-André Désautels), <code>michaschwab</code> (Michail Schwab), <code>alvarolarreategui</code> (Alvaro Larreategui), <code>jakevc</code> (Jake VanCampen), <code>omerta</code> (Guillermo Lengemann), <code>espinielli</code> (Enrico Spinielli), <code>asimumba</code>(Aaron Simumba), <code>braunschweig</code> (Maldewar), <code>gwierzchowski</code> (Grzegorz Wierzchowski), <code>technocrat</code> (Richard Careaga), <code>atzakas</code>, <code>defeit</code> (David Emerson Feit), <code>shiraamitchell</code> (Shira Mitchell), <code>Nathalie-S</code>, <code>andreashandel</code> (Andreas Handel), <code>berkowitze</code> (Elias Berkowitz), <code>Dean-Webb</code> (Dean Webber), <code>mohayusuf</code>, <code>jimrothstein</code>, <code>mPloenzke</code> (Matthew Ploenzke), <code>NicholasDowand</code> (Nicholas Dow), <code>kant</code> (Darío Hereñú), <code>debbieyuster</code> (Debbie Yuster), <code>tuanchauict</code> (Tuan Chau), <code>phzeller</code>, <code>BTJ01</code> (BradJ), <code>glsnow</code> (Greg Snow), <code>mberlanda</code> (Mauro Berlanda), <code>wfan9</code>, <code>larswestvang</code> (Lars Westvang), <code>jj999</code> (Jan Andrejkovic), <code>Kriegslustig</code> (Luca Nils Schmid), <code>odahhani</code>, <code>aidanhorn</code> (Aidan Horn), <code>atraxler</code> (Adrienne Traxler), <code>alvegorova</code>,<code>wycheong</code> (Won Young Cheong), <code>med-hat</code> (Medhat Khalil), <code>biscotty666</code> (Brian Carey), <code>kengustafson</code>, <code>Yowza63</code>, <code>ryan-heslin</code> (Ryan Heslin), <code>raffaem</code>, <code>tim8west</code>, David D. Kane, El Mustapha El Abbassi, Vadim Zipunnikov, Anna Quaglieri, Chris Dong, Rick Schoenberg, Isabella Grabski, and Doug Snyder.</p>
 
 
 </section>
@@ -687,7 +687,7 @@ <h1 class="unnumbered">Acknowledgments</h1>
 </div> <!-- /content -->
 <footer class="footer">
   <div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/inference/bayes.html b/docs/inference/bayes.html
index 6d8823a..5ecd069 100644
--- a/docs/inference/bayes.html
+++ b/docs/inference/bayes.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 12&nbsp; Bayesian statistics</title>
+<title>Introduction to Data Science - 12&nbsp; Bayesian statistics</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -866,7 +866,7 @@ <h1 class="title"><span id="sec-bayesian-statistics" class="quarto-section-ident
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/inference/bootstrap.html b/docs/inference/bootstrap.html
index 18bfc5d..d0712b1 100644
--- a/docs/inference/bootstrap.html
+++ b/docs/inference/bootstrap.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 10&nbsp; Bootstrap</title>
+<title>Introduction to Data Science - 10&nbsp; Bootstrap</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -767,7 +767,7 @@ <h1 class="title"><span id="sec-bootstrap" class="quarto-section-identifier"><sp
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/inference/clt.html b/docs/inference/clt.html
index d9b1f7c..23de4a1 100644
--- a/docs/inference/clt.html
+++ b/docs/inference/clt.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 7&nbsp; Central Limit Theorem</title>
+<title>Introduction to Data Science - 7&nbsp; Central Limit Theorem</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -841,7 +841,7 @@ <h1 class="title"><span id="sec-clt" class="quarto-section-identifier"><span cla
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/inference/confidence-intervals.html b/docs/inference/confidence-intervals.html
index d596a23..2a671a9 100644
--- a/docs/inference/confidence-intervals.html
+++ b/docs/inference/confidence-intervals.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 8&nbsp; Confidence intervals</title>
+<title>Introduction to Data Science - 8&nbsp; Confidence intervals</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -814,7 +814,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/inference/hierarchical-models.html b/docs/inference/hierarchical-models.html
index 1208d75..a7dca77 100644
--- a/docs/inference/hierarchical-models.html
+++ b/docs/inference/hierarchical-models.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 13&nbsp; Hierarchichal Models</title>
+<title>Introduction to Data Science - 13&nbsp; Hierarchichal Models</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -110,7 +110,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -1139,7 +1139,7 @@ <h1 class="title"><span id="sec-election-forecasting" class="quarto-section-iden
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/inference/hypothesis-testing.html b/docs/inference/hypothesis-testing.html
index f4a7199..c4ae107 100644
--- a/docs/inference/hypothesis-testing.html
+++ b/docs/inference/hypothesis-testing.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 9&nbsp; Hypothesis testing</title>
+<title>Introduction to Data Science - 9&nbsp; Hypothesis testing</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -755,7 +755,7 @@ <h1 class="title"><span id="sec-hypothesis-testing" class="quarto-section-identi
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/inference/intro-inference.html b/docs/inference/intro-inference.html
index 84e9633..d29ab2f 100644
--- a/docs/inference/intro-inference.html
+++ b/docs/inference/intro-inference.html
@@ -8,7 +8,7 @@
 
 <meta name="author" content="Rafael A. Irizarry">
 
-<title>Advanced Data Science - Statistical inference</title>
+<title>Introduction to Data Science - Statistical inference</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -89,7 +89,7 @@
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto">
     <div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -678,7 +678,7 @@ <h1 class="title">Statistical inference</h1>
 </div> <!-- /content -->
 <footer class="footer">
   <div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     <div class="toc-actions"><div><i class="bi bi-github"></i></div><div class="action-links"><p><a href="https://github.com/rafalab/dsbook-part-2/blob/main/inference/intro-inference.qmd" class="toc-action">View source</a></p><p><a href="https://github.com/rafalab/dsbook-part-2/issues/new" class="toc-action">Report an issue</a></p></div></div></div>
diff --git a/docs/inference/models.html b/docs/inference/models.html
index 8f584c2..cd306fd 100644
--- a/docs/inference/models.html
+++ b/docs/inference/models.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 11&nbsp; Data-driven models</title>
+<title>Introduction to Data Science - 11&nbsp; Data-driven models</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -977,7 +977,7 @@ <h1 class="title"><span id="sec-models" class="quarto-section-identifier"><span
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/inference/parameters-estimates.html b/docs/inference/parameters-estimates.html
index dfc408a..58c5ea0 100644
--- a/docs/inference/parameters-estimates.html
+++ b/docs/inference/parameters-estimates.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 6&nbsp; Parameters and Estimates</title>
+<title>Introduction to Data Science - 6&nbsp; Parameters and Estimates</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -110,7 +110,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -951,7 +951,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/intro.html b/docs/intro.html
index 64561d7..e7702a1 100644
--- a/docs/intro.html
+++ b/docs/intro.html
@@ -8,7 +8,7 @@
 
 <meta name="author" content="Rafael A. Irizarry">
 
-<title>Advanced Data Science - Introduction</title>
+<title>Introduction to Data Science - Introduction</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -89,7 +89,7 @@
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto">
     <div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="./">Advanced Data Science</a> 
+      <a href="./">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -685,7 +685,7 @@ <h2 class="unnumbered anchored" data-anchor-id="what-is-not-covered-by-this-book
 </div> <!-- /content -->
 <footer class="footer">
   <div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/linear-models/association-not-causation.html b/docs/linear-models/association-not-causation.html
index 219bd1d..ffd4982 100644
--- a/docs/linear-models/association-not-causation.html
+++ b/docs/linear-models/association-not-causation.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 19&nbsp; Association is not causation</title>
+<title>Introduction to Data Science - 19&nbsp; Association is not causation</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -984,7 +984,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/linear-models/association-tests.html b/docs/linear-models/association-tests.html
index e7b30dd..770e2ce 100644
--- a/docs/linear-models/association-tests.html
+++ b/docs/linear-models/association-tests.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 18&nbsp; Association tests</title>
+<title>Introduction to Data Science - 18&nbsp; Association tests</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -110,7 +110,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -1008,7 +1008,7 @@ <h1 class="title"><span id="sec-association-tests" class="quarto-section-identif
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/linear-models/intro-to-linear-models.html b/docs/linear-models/intro-to-linear-models.html
index e567cc6..55fac9d 100644
--- a/docs/linear-models/intro-to-linear-models.html
+++ b/docs/linear-models/intro-to-linear-models.html
@@ -8,7 +8,7 @@
 
 <meta name="author" content="Rafael A. Irizarry">
 
-<title>Advanced Data Science - Linear Models</title>
+<title>Introduction to Data Science - Linear Models</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -89,7 +89,7 @@
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto">
     <div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -658,7 +658,7 @@ <h1 class="title">Linear Models</h1>
 </div> <!-- /content -->
 <footer class="footer">
   <div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     <div class="toc-actions"><div><i class="bi bi-github"></i></div><div class="action-links"><p><a href="https://github.com/rafalab/dsbook-part-2/blob/main/linear-models/intro-to-linear-models.qmd" class="toc-action">View source</a></p><p><a href="https://github.com/rafalab/dsbook-part-2/issues/new" class="toc-action">Report an issue</a></p></div></div></div>
diff --git a/docs/linear-models/measurement-error-models.html b/docs/linear-models/measurement-error-models.html
index f2284b2..90b713a 100644
--- a/docs/linear-models/measurement-error-models.html
+++ b/docs/linear-models/measurement-error-models.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 16&nbsp; Measurement error models</title>
+<title>Introduction to Data Science - 16&nbsp; Measurement error models</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -767,7 +767,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/linear-models/multivariate-regression.html b/docs/linear-models/multivariate-regression.html
index c4d4490..69685f6 100644
--- a/docs/linear-models/multivariate-regression.html
+++ b/docs/linear-models/multivariate-regression.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 15&nbsp; Multivariate Regression</title>
+<title>Introduction to Data Science - 15&nbsp; Multivariate Regression</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -110,7 +110,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -511,10 +511,10 @@ <h1 class="title"><span id="sec-multivariate-regression" class="quarto-section-i
 <div class="cell" data-layout-align="center">
 <div class="sourceCode" id="cb1"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://tidyverse.tidyverse.org">tidyverse</a></span><span class="op">)</span></span>
 <span><span class="co">#&gt; ── Attaching core tidyverse packages ──────────────── tidyverse 2.0.0 ──</span></span>
-<span><span class="co">#&gt; ✔ dplyr     1.1.1     ✔ readr     2.1.4</span></span>
-<span><span class="co">#&gt; ✔ forcats   1.0.0     ✔ stringr   1.5.0</span></span>
-<span><span class="co">#&gt; ✔ lubridate 1.9.2     ✔ tibble    3.2.1</span></span>
-<span><span class="co">#&gt; ✔ purrr     1.0.1     ✔ tidyr     1.3.0</span></span>
+<span><span class="co">#&gt; ✔ dplyr     1.1.4     ✔ readr     2.1.5</span></span>
+<span><span class="co">#&gt; ✔ forcats   1.0.0     ✔ stringr   1.5.1</span></span>
+<span><span class="co">#&gt; ✔ lubridate 1.9.3     ✔ tibble    3.2.1</span></span>
+<span><span class="co">#&gt; ✔ purrr     1.0.2     ✔ tidyr     1.3.1</span></span>
 <span><span class="co">#&gt; ── Conflicts ────────────────────────────────── tidyverse_conflicts() ──</span></span>
 <span><span class="co">#&gt; ✖ dplyr::filter() masks stats::filter()</span></span>
 <span><span class="co">#&gt; ✖ dplyr::lag()    masks stats::lag()</span></span>
@@ -1109,7 +1109,7 @@ <h1 class="title"><span id="sec-multivariate-regression" class="quarto-section-i
 <p>4. Now fit a linear model for each metric and use the <code>confint</code> function to compare the estimates.</p>
 <p>5. In a previous section, we computed the correlation between mothers and daughters, mothers and sons, fathers and daughters, and fathers and sons. We noticed that the highest correlation is between fathers and sons and the lowest is between mothers and sons. We can compute these correlations using:</p>
 <div class="cell" data-layout-align="center" data-hash="multivariate-regression_cache/html/unnamed-chunk-25_46ec94782090e423e19be4ff1804ed9d">
-<div class="sourceCode" id="cb31"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va">HistData</span><span class="op">)</span></span>
+<div class="sourceCode" id="cb31"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://friendly.github.io/HistData/">HistData</a></span><span class="op">)</span></span>
 <span><span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op">(</span><span class="fl">1</span><span class="op">)</span></span>
 <span><span class="va">galton_heights</span> <span class="op">&lt;-</span> <span class="va">GaltonFamilies</span> <span class="op">|&gt;</span></span>
 <span>  <span class="fu"><a href="https://dplyr.tidyverse.org/reference/group_by.html">group_by</a></span><span class="op">(</span><span class="va">family</span>, <span class="va">gender</span><span class="op">)</span> <span class="op">|&gt;</span></span>
@@ -1401,7 +1401,7 @@ <h1 class="title"><span id="sec-multivariate-regression" class="quarto-section-i
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/linear-models/regression.html b/docs/linear-models/regression.html
index 9922312..6fa9ff5 100644
--- a/docs/linear-models/regression.html
+++ b/docs/linear-models/regression.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 14&nbsp; Regression</title>
+<title>Introduction to Data Science - 14&nbsp; Regression</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -110,7 +110,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -461,7 +461,7 @@ <h1 class="title"><span id="sec-regression" class="quarto-section-identifier"><s
 <p>We have access to Galton’s family height data through the <strong>HistData</strong> package. This data contains heights on several dozen families: mothers, fathers, daughters, and sons. To imitate Galton’s analysis, we will create a dataset with the heights of fathers and a randomly selected son of each family:</p>
 <div class="cell" data-layout-align="center">
 <div class="sourceCode" id="cb1"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://tidyverse.tidyverse.org">tidyverse</a></span><span class="op">)</span></span>
-<span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va">HistData</span><span class="op">)</span></span>
+<span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://friendly.github.io/HistData/">HistData</a></span><span class="op">)</span></span>
 <span></span>
 <span><span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op">(</span><span class="fl">1983</span><span class="op">)</span></span>
 <span><span class="va">galton_heights</span> <span class="op">&lt;-</span> <span class="va">GaltonFamilies</span> <span class="op">|&gt;</span></span>
@@ -1076,7 +1076,7 @@ <h1 class="title"><span id="sec-regression" class="quarto-section-identifier"><s
 </table>
 </div>
 </div>
-<p>In fact, the proportion of players that have a lower batting average during their sophomore year is 0.6981132.</p>
+<p>In fact, the proportion of players that have a lower batting average during their sophomore year is 0.7037037.</p>
 <p>So is it “jitters” or “jinx”? To answer this question, let’s turn our attention to all the players that played the 2013 and 2014 seasons and batted more than 130 times (minimum to win Rookie of the Year).</p>
 <p>The same pattern arises when we look at the top performers: batting averages go down for most of the top performers.</p>
 <div class="cell" data-layout-align="center" data-hash="regression_cache/html/unnamed-chunk-26_4b146a2eddea6dfc183af2c2486c1384">
@@ -1436,7 +1436,7 @@ <h1 class="title"><span id="sec-regression" class="quarto-section-identifier"><s
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/linear-models/treatment-effect-models.html b/docs/linear-models/treatment-effect-models.html
index e7a8e22..1ceb098 100644
--- a/docs/linear-models/treatment-effect-models.html
+++ b/docs/linear-models/treatment-effect-models.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 17&nbsp; Treatment effect models</title>
+<title>Introduction to Data Science - 17&nbsp; Treatment effect models</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -645,7 +645,7 @@ <h1 class="title"><span id="sec-treatment-effect-models" class="quarto-section-i
 <p>The package <strong>emmeans</strong> simplifies the calculation and also calculates standard errors:</p>
 <div class="cell" data-layout-align="center" data-hash="treatment-effect-models_cache/html/unnamed-chunk-14_0100728b9937bf89d8a0de0ff0018ba0">
 <div class="sourceCode" id="cb17"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://github.com/rvlenth/emmeans">emmeans</a></span><span class="op">)</span></span>
-<span><span class="fu"><a href="https://rdrr.io/pkg/emmeans/man/emmeans.html">emmeans</a></span><span class="op">(</span><span class="va">fit</span>, <span class="op">~</span><span class="va">sex</span><span class="op">)</span></span>
+<span><span class="fu">emmeans</span><span class="op">(</span><span class="va">fit</span>, <span class="op">~</span><span class="va">sex</span><span class="op">)</span></span>
 <span><span class="co">#&gt;  sex emmean    SE  df lower.CL upper.CL</span></span>
 <span><span class="co">#&gt;  F     29.8 0.339 778     29.1     30.4</span></span>
 <span><span class="co">#&gt;  M     38.6 0.346 778     37.9     39.3</span></span>
@@ -668,7 +668,7 @@ <h1 class="title"><span id="sec-treatment-effect-models" class="quarto-section-i
 <p>We see that the intercept is now larger, reflecting the overall mean rather than just the mean for females. The other coefficient, <span class="math inline">\(\beta_1\)</span>, represents the contrast between females and the overall mean in our model. The coefficient for men is not shown because it is redundant: <span class="math inline">\(\beta_1= -\beta_2\)</span>.</p>
 <p>If we want to see all the estimates, the <strong>emmeans</strong> package also makes the calculations for us:</p>
 <div class="cell" data-layout-align="center" data-hash="treatment-effect-models_cache/html/unnamed-chunk-16_3e07836bc638d103d756aa5a7089b045">
-<div class="sourceCode" id="cb19"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/emmeans/man/contrast.html">contrast</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/pkg/emmeans/man/emmeans.html">emmeans</a></span><span class="op">(</span><span class="va">fit</span>, <span class="op">~</span><span class="va">sex</span><span class="op">)</span><span class="op">)</span></span>
+<div class="sourceCode" id="cb19"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu">contrast</span><span class="op">(</span><span class="fu">emmeans</span><span class="op">(</span><span class="va">fit</span>, <span class="op">~</span><span class="va">sex</span><span class="op">)</span><span class="op">)</span></span>
 <span><span class="co">#&gt;  contrast estimate    SE  df t.ratio p.value</span></span>
 <span><span class="co">#&gt;  F effect    -4.41 0.242 778 -18.200  &lt;.0001</span></span>
 <span><span class="co">#&gt;  M effect     4.41 0.242 778  18.200  &lt;.0001</span></span>
@@ -699,7 +699,7 @@ <h1 class="title"><span id="sec-treatment-effect-models" class="quarto-section-i
 <div class="cell" data-layout-align="center" data-hash="treatment-effect-models_cache/html/unnamed-chunk-18_3f9be1912ad25f4df716f24ada6290c5">
 <div class="sourceCode" id="cb21"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">fit</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/lm.html">lm</a></span><span class="op">(</span><span class="va">body_weight</span> <span class="op">~</span> <span class="va">gen</span>,  data <span class="op">=</span> <span class="va">mice_weights</span>, </span>
 <span>          contrasts <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op">(</span>gen <span class="op">=</span> <span class="va">contr.sum</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="fu"><a href="https://rdrr.io/pkg/emmeans/man/contrast.html">contrast</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/pkg/emmeans/man/emmeans.html">emmeans</a></span><span class="op">(</span><span class="va">fit</span>, <span class="op">~</span><span class="va">gen</span><span class="op">)</span><span class="op">)</span> </span>
+<span><span class="fu">contrast</span><span class="op">(</span><span class="fu">emmeans</span><span class="op">(</span><span class="va">fit</span>, <span class="op">~</span><span class="va">gen</span><span class="op">)</span><span class="op">)</span> </span>
 <span><span class="co">#&gt;  contrast     estimate    SE  df t.ratio p.value</span></span>
 <span><span class="co">#&gt;  gen4 effect    -0.122 0.705 775  -0.174  0.8620</span></span>
 <span><span class="co">#&gt;  gen7 effect    -0.812 0.542 775  -1.497  0.3370</span></span>
@@ -1109,7 +1109,7 @@ <h1 class="title"><span id="sec-treatment-effect-models" class="quarto-section-i
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/ml/algorithms.html b/docs/ml/algorithms.html
index fffbcfa..4463a82 100644
--- a/docs/ml/algorithms.html
+++ b/docs/ml/algorithms.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 30&nbsp; Examples of algorithms</title>
+<title>Introduction to Data Science - 30&nbsp; Examples of algorithms</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -702,7 +702,7 @@ <h1 class="title"><span id="sec-example-alogirhms" class="quarto-section-identif
 <span><span class="co">#&gt;    0.775</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>When we force this assumption, we can show mathematically that the boundary is a line, just as with logistic regression. For this reason, we call the method <em>linear</em> discriminant analysis (LDA). Similarly, for QDA, we can show that the boundary must be a quadratic function.</p>
-<div class="cell" data-layout-align="center" data-hash="algorithms_cache/html/lda-estimate_fe3abdf5a32c8bab96981387be2d0762">
+<div class="cell" data-layout-align="center" data-hash="algorithms_cache/html/lda-estimate_a5ecd21e8dd69a16f75b6c5cee77fac6">
 <div class="cell-output-display">
 <div class="quarto-figure quarto-figure-center">
 <figure class="figure"><p><img src="algorithms_files/figure-html/lda-estimate-1.png" class="img-fluid figure-img" style="width:100.0%"></p>
@@ -1404,7 +1404,7 @@ <h1 class="title"><span id="sec-example-alogirhms" class="quarto-section-identif
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/ml/algorithms_files/figure-html/lda-estimate-1.png b/docs/ml/algorithms_files/figure-html/lda-estimate-1.png
index 624292d..f141ea3 100644
Binary files a/docs/ml/algorithms_files/figure-html/lda-estimate-1.png and b/docs/ml/algorithms_files/figure-html/lda-estimate-1.png differ
diff --git a/docs/ml/clustering.html b/docs/ml/clustering.html
index 3cfd938..4f3af5b 100644
--- a/docs/ml/clustering.html
+++ b/docs/ml/clustering.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 32&nbsp; Clustering</title>
+<title>Introduction to Data Science - 32&nbsp; Clustering</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -108,7 +108,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -437,22 +437,7 @@ <h1 class="title"><span id="sec-clustering" class="quarto-section-identifier"><s
 <p>We will construct a simple example based on movie ratings. Here we quickly construct a matrix <code>x</code> that has ratings for the 50 movies with the most ratings.</p>
 <div class="cell" data-layout-align="center">
 <div class="sourceCode" id="cb1"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://tidyverse.tidyverse.org">tidyverse</a></span><span class="op">)</span></span>
-<span><span class="co">#&gt; ── Attaching core tidyverse packages ──────────────── tidyverse 2.0.0 ──</span></span>
-<span><span class="co">#&gt; ✔ dplyr     1.1.1     ✔ readr     2.1.4</span></span>
-<span><span class="co">#&gt; ✔ forcats   1.0.0     ✔ stringr   1.5.0</span></span>
-<span><span class="co">#&gt; ✔ lubridate 1.9.2     ✔ tibble    3.2.1</span></span>
-<span><span class="co">#&gt; ✔ purrr     1.0.1     ✔ tidyr     1.3.0</span></span>
-<span><span class="co">#&gt; ── Conflicts ────────────────────────────────── tidyverse_conflicts() ──</span></span>
-<span><span class="co">#&gt; ✖ dplyr::filter() masks stats::filter()</span></span>
-<span><span class="co">#&gt; ✖ dplyr::lag()    masks stats::lag()</span></span>
-<span><span class="co">#&gt; ℹ Use the conflicted package (&lt;http://conflicted.r-lib.org/&gt;) to force all conflicts to become errors</span></span>
 <span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://github.com/sfirke/janitor">janitor</a></span><span class="op">)</span></span>
-<span><span class="co">#&gt; </span></span>
-<span><span class="co">#&gt; Attaching package: 'janitor'</span></span>
-<span><span class="co">#&gt; </span></span>
-<span><span class="co">#&gt; The following objects are masked from 'package:stats':</span></span>
-<span><span class="co">#&gt; </span></span>
-<span><span class="co">#&gt;     chisq.test, fisher.test</span></span>
 <span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va">dslabs</span><span class="op">)</span></span>
 <span><span class="va">top_movies</span> <span class="op">&lt;-</span> <span class="va">movielens</span> <span class="op">|&gt;</span> <span class="fu"><a href="https://dplyr.tidyverse.org/reference/count.html">count</a></span><span class="op">(</span><span class="va">movieId</span><span class="op">)</span> <span class="op">|&gt;</span> <span class="fu"><a href="https://dplyr.tidyverse.org/reference/top_n.html">top_n</a></span><span class="op">(</span><span class="fl">50</span>, <span class="va">n</span><span class="op">)</span> <span class="op">|&gt;</span> <span class="fu"><a href="https://dplyr.tidyverse.org/reference/pull.html">pull</a></span><span class="op">(</span><span class="va">movieId</span><span class="op">)</span></span>
 <span><span class="va">x</span> <span class="op">&lt;-</span> <span class="va">movielens</span> <span class="op">|&gt;</span> <span class="fu"><a href="https://dplyr.tidyverse.org/reference/filter.html">filter</a></span><span class="op">(</span><span class="va">movieId</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">top_movies</span><span class="op">)</span> </span>
@@ -462,7 +447,8 @@ <h1 class="title"><span id="sec-clustering" class="quarto-section-identifier"><s
 <span>  <span class="fu"><a href="https://tidyr.tidyverse.org/reference/pivot_wider.html">pivot_wider</a></span><span class="op">(</span>names_from <span class="op">=</span> <span class="va">userId</span>, values_from <span class="op">=</span> <span class="va">rating</span><span class="op">)</span> <span class="op">|&gt;</span></span>
 <span>  <span class="fu"><a href="https://tibble.tidyverse.org/reference/rownames.html">column_to_rownames</a></span><span class="op">(</span><span class="st">"title"</span><span class="op">)</span> <span class="op">|&gt;</span></span>
 <span>  <span class="fu"><a href="https://rdrr.io/r/base/matrix.html">as.matrix</a></span><span class="op">(</span><span class="op">)</span></span>
-<span><span class="fu"><a href="https://rdrr.io/r/base/colnames.html">rownames</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://stringr.tidyverse.org/reference/str_remove.html">str_remove</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/colnames.html">rownames</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span>, <span class="st">": Episode"</span><span class="op">)</span> <span class="op">|&gt;</span> <span class="fu"><a href="https://stringr.tidyverse.org/reference/str_trunc.html">str_trunc</a></span><span class="op">(</span><span class="fl">20</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<span><span class="fu"><a href="https://rdrr.io/r/base/colnames.html">rownames</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://stringr.tidyverse.org/reference/str_remove.html">str_remove</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/colnames.html">rownames</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span>, <span class="st">": Episode"</span><span class="op">)</span> <span class="op">|&gt;</span> <span class="fu"><a href="https://stringr.tidyverse.org/reference/str_trunc.html">str_trunc</a></span><span class="op">(</span><span class="fl">20</span><span class="op">)</span></span>
+<span><span class="va">x</span> <span class="op">&lt;-</span> <span class="va">x</span> <span class="op">-</span> <span class="fu"><a href="https://rdrr.io/r/base/colSums.html">rowMeans</a></span><span class="op">(</span><span class="va">x</span>, na.rm <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>We want to use these data to find out if there are clusters of movies based on the ratings from 51 movie raters. A first step is to find the distance between each pair of movies using the <code>dist</code> function:</p>
 <div class="cell" data-layout-align="center" data-hash="clustering_cache/html/unnamed-chunk-2_23cf077f3aed45c923cc66e7043d812e">
@@ -811,7 +797,7 @@ <h1 class="title"><span id="sec-clustering" class="quarto-section-identifier"><s
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/ml/conditionals.html b/docs/ml/conditionals.html
index 25f6c40..2531478 100644
--- a/docs/ml/conditionals.html
+++ b/docs/ml/conditionals.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 27&nbsp; Conditional probabilities and expectations</title>
+<title>Introduction to Data Science - 27&nbsp; Conditional probabilities and expectations</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -740,7 +740,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/ml/evaluation-metrics.html b/docs/ml/evaluation-metrics.html
index 4b4eca2..4d66e28 100644
--- a/docs/ml/evaluation-metrics.html
+++ b/docs/ml/evaluation-metrics.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 26&nbsp; Evaluation metrics</title>
+<title>Introduction to Data Science - 26&nbsp; Evaluation metrics</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -110,7 +110,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -1070,7 +1070,7 @@ <h1 class="title"><span id="sec-evaluation-metrics" class="quarto-section-identi
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/ml/intro-ml.html b/docs/ml/intro-ml.html
index ba61918..48a85ec 100644
--- a/docs/ml/intro-ml.html
+++ b/docs/ml/intro-ml.html
@@ -8,7 +8,7 @@
 
 <meta name="author" content="Rafael A. Irizarry">
 
-<title>Advanced Data Science - Machine Learning</title>
+<title>Introduction to Data Science - Machine Learning</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -89,7 +89,7 @@
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto">
     <div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -660,7 +660,7 @@ <h1 class="title">Machine Learning</h1>
 </div> <!-- /content -->
 <footer class="footer">
   <div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     <div class="toc-actions"><div><i class="bi bi-github"></i></div><div class="action-links"><p><a href="https://github.com/rafalab/dsbook-part-2/blob/main/ml/intro-ml.qmd" class="toc-action">View source</a></p><p><a href="https://github.com/rafalab/dsbook-part-2/issues/new" class="toc-action">Report an issue</a></p></div></div></div>
diff --git a/docs/ml/ml-in-practice.html b/docs/ml/ml-in-practice.html
index 3db07c0..aab5837 100644
--- a/docs/ml/ml-in-practice.html
+++ b/docs/ml/ml-in-practice.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 31&nbsp; Machine learning in practice</title>
+<title>Introduction to Data Science - 31&nbsp; Machine learning in practice</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -410,22 +410,23 @@
   <ul class="collapse">
 <li><a href="#the-train-function" id="toc-the-train-function" class="nav-link" data-scroll-target="#the-train-function"><span class="header-section-number">31.1.1</span> The <code>train</code> function</a></li>
   <li><a href="#the-predict-function" id="toc-the-predict-function" class="nav-link" data-scroll-target="#the-predict-function"><span class="header-section-number">31.1.2</span> The <code>predict</code> function</a></li>
-  <li><a href="#sec-caret-cv" id="toc-sec-caret-cv" class="nav-link" data-scroll-target="#sec-caret-cv"><span class="header-section-number">31.1.3</span> Cross validation</a></li>
+  <li><a href="#sec-caret-cv" id="toc-sec-caret-cv" class="nav-link" data-scroll-target="#sec-caret-cv"><span class="header-section-number">31.1.3</span> Resampling</a></li>
   </ul>
 </li>
   <li><a href="#preprocessing" id="toc-preprocessing" class="nav-link" data-scroll-target="#preprocessing"><span class="header-section-number">31.2</span> Preprocessing</a></li>
+  <li><a href="#parallelization" id="toc-parallelization" class="nav-link" data-scroll-target="#parallelization"><span class="header-section-number">31.3</span> Parallelization</a></li>
   <li>
-<a href="#k-nearest-neighbors" id="toc-k-nearest-neighbors" class="nav-link" data-scroll-target="#k-nearest-neighbors"><span class="header-section-number">31.3</span> k-nearest neighbors</a>
+<a href="#knn-in-practice" id="toc-knn-in-practice" class="nav-link" data-scroll-target="#knn-in-practice"><span class="header-section-number">31.4</span> k-nearest neighbors</a>
   <ul class="collapse">
-<li><a href="#dimension-reduction-with-pca" id="toc-dimension-reduction-with-pca" class="nav-link" data-scroll-target="#dimension-reduction-with-pca"><span class="header-section-number">31.3.1</span> Dimension reduction with PCA</a></li>
+<li><a href="#dimension-reduction-with-pca" id="toc-dimension-reduction-with-pca" class="nav-link" data-scroll-target="#dimension-reduction-with-pca"><span class="header-section-number">31.4.1</span> Dimension reduction with PCA</a></li>
   </ul>
 </li>
-  <li><a href="#random-forest" id="toc-random-forest" class="nav-link" data-scroll-target="#random-forest"><span class="header-section-number">31.4</span> Random Forest</a></li>
-  <li><a href="#testing-and-improving-computation-time" id="toc-testing-and-improving-computation-time" class="nav-link" data-scroll-target="#testing-and-improving-computation-time"><span class="header-section-number">31.5</span> Testing and improving computation time</a></li>
-  <li><a href="#variable-importance" id="toc-variable-importance" class="nav-link" data-scroll-target="#variable-importance"><span class="header-section-number">31.6</span> Variable importance</a></li>
-  <li><a href="#diagnostics" id="toc-diagnostics" class="nav-link" data-scroll-target="#diagnostics"><span class="header-section-number">31.7</span> Diagnostics</a></li>
-  <li><a href="#ensembles" id="toc-ensembles" class="nav-link" data-scroll-target="#ensembles"><span class="header-section-number">31.8</span> Ensembles</a></li>
-  <li><a href="#exercises" id="toc-exercises" class="nav-link" data-scroll-target="#exercises"><span class="header-section-number">31.9</span> Exercises</a></li>
+  <li><a href="#random-forest" id="toc-random-forest" class="nav-link" data-scroll-target="#random-forest"><span class="header-section-number">31.5</span> Random Forest</a></li>
+  <li><a href="#testing-and-improving-computation-time" id="toc-testing-and-improving-computation-time" class="nav-link" data-scroll-target="#testing-and-improving-computation-time"><span class="header-section-number">31.6</span> Testing and improving computation time</a></li>
+  <li><a href="#variable-importance" id="toc-variable-importance" class="nav-link" data-scroll-target="#variable-importance"><span class="header-section-number">31.7</span> Variable importance</a></li>
+  <li><a href="#diagnostics" id="toc-diagnostics" class="nav-link" data-scroll-target="#diagnostics"><span class="header-section-number">31.8</span> Diagnostics</a></li>
+  <li><a href="#ensembles" id="toc-ensembles" class="nav-link" data-scroll-target="#ensembles"><span class="header-section-number">31.9</span> Ensembles</a></li>
+  <li><a href="#exercises" id="toc-exercises" class="nav-link" data-scroll-target="#exercises"><span class="header-section-number">31.10</span> Exercises</a></li>
   </ul><div class="toc-actions"><div><i class="bi bi-github"></i></div><div class="action-links"><p><a href="https://github.com/rafalab/dsbook-part-2/blob/main/ml/ml-in-practice.qmd" class="toc-action">View source</a></p><p><a href="https://github.com/rafalab/dsbook-part-2/issues/new" class="toc-action">Report an issue</a></p></div></div></nav>
     </div>
 <!-- main -->
@@ -478,6 +479,11 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 <span><span class="va">x_test</span> <span class="op">&lt;-</span> <span class="va">mnist</span><span class="op">$</span><span class="va">test</span><span class="op">$</span><span class="va">images</span><span class="op">[</span><span class="va">index</span>,<span class="op">]</span></span>
 <span><span class="va">y_test</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op">(</span><span class="va">mnist</span><span class="op">$</span><span class="va">test</span><span class="op">$</span><span class="va">labels</span><span class="op">[</span><span class="va">index</span><span class="op">]</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
+<p>When fitting models to large datasets, we recommend using matrices instead of data frames, as matrix operations tend to be faster. In the <strong>caret</strong> package, predictor matrices must have column names to track features accurately during prediction on the test set. If the matrices lack column names, you can assign names based on their position:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-7_324875282d13b73d7fc329f468a45048">
+<div class="sourceCode" id="cb6"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/colnames.html">colnames</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span> <span class="op">&lt;-</span> <span class="fl">1</span><span class="op">:</span><span class="fu"><a href="https://rdrr.io/r/base/nrow.html">ncol</a></span><span class="op">(</span><span class="va">mnist</span><span class="op">$</span><span class="va">train</span><span class="op">$</span><span class="va">images</span><span class="op">)</span></span>
+<span><span class="fu"><a href="https://rdrr.io/r/base/colnames.html">colnames</a></span><span class="op">(</span><span class="va">x_test</span><span class="op">)</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/colnames.html">colnames</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
 <section id="sec-caret" class="level2" data-number="31.1"><h2 data-number="31.1" class="anchored" data-anchor-id="sec-caret">
 <span class="header-section-number">31.1</span> The caret package</h2>
 <p>We have already learned about several machine learning algorithms. Many of these algorithms are implemented in R. However, they are distributed via different packages, developed by different authors, and often use different syntax. The <strong>caret</strong> package tries to consolidate these differences and provide consistency. It currently includes over 200 different methods which are summarized in the <strong>caret</strong> package manual<a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a>. Keep in mind that <strong>caret</strong> does not include the packages needed to run each possible algorithm. To apply a machine learning method through <strong>caret</strong> you still need to install the library that implement the method. The required packages for each method are described in the package manual.</p>
@@ -485,69 +491,69 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 <section id="the-train-function" class="level3" data-number="31.1.1"><h3 data-number="31.1.1" class="anchored" data-anchor-id="the-train-function">
 <span class="header-section-number">31.1.1</span> The <code>train</code> function</h3>
 <p>The R functions that fit machine algorithms are all slightly different. Functions such as <code>lm</code>, <code>glm</code>, <code>qda</code>, <code>lda</code>, <code>knn3</code>, <code>rpart</code> and <code>randomForrest</code> use different syntax, have different argument names and produce objects of different types.</p>
-<p>The <strong>caret</strong> <code>train</code> function lets us train different algorithms using similar syntax. So, for example, we can type:</p>
+<p>The <strong>caret</strong> <code>train</code> function lets us train different algorithms using similar syntax. So, for example, we can type the following to train three different models:</p>
 <div class="cell" data-layout-align="center">
-<div class="sourceCode" id="cb6"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://github.com/topepo/caret/">caret</a></span><span class="op">)</span></span>
-<span><span class="co">#&gt; Loading required package: lattice</span></span>
+<div class="sourceCode" id="cb7"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://github.com/topepo/caret/">caret</a></span><span class="op">)</span></span>
 <span><span class="va">train_glm</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">y</span> <span class="op">~</span> <span class="va">.</span>, method <span class="op">=</span> <span class="st">"glm"</span>, data <span class="op">=</span> <span class="va">mnist_27</span><span class="op">$</span><span class="va">train</span><span class="op">)</span></span>
 <span><span class="va">train_qda</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">y</span> <span class="op">~</span> <span class="va">.</span>, method <span class="op">=</span> <span class="st">"qda"</span>, data <span class="op">=</span> <span class="va">mnist_27</span><span class="op">$</span><span class="va">train</span><span class="op">)</span></span>
 <span><span class="va">train_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">y</span> <span class="op">~</span> <span class="va">.</span>, method <span class="op">=</span> <span class="st">"knn"</span>, data <span class="op">=</span> <span class="va">mnist_27</span><span class="op">$</span><span class="va">train</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
-<p>As we explain in more detail in <a href="#sec-caret-cv"><span>Section&nbsp;31.1.3</span></a>, the <code>train</code> function selects parameters for you using a resampling method, with boostrap as the default.</p>
+<p>As we explain in more detail in <a href="#sec-caret-cv"><span>Section&nbsp;31.1.3</span></a>, the <code>train</code> function selects parameters for you using a resampling method to estimate the MSE, with bootstrap as the default.</p>
 </section><section id="the-predict-function" class="level3" data-number="31.1.2"><h3 data-number="31.1.2" class="anchored" data-anchor-id="the-predict-function">
 <span class="header-section-number">31.1.2</span> The <code>predict</code> function</h3>
 <p>The <code>predict</code> function is very useful for machine learning applications. This function takes an object from a fitting function and a data frame with features <span class="math inline">\(\mathbf{x}\)</span> for which to predict, and returns predictions for these features.</p>
 <p>Here is an example with logistic regression:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-8_b9bcfd4786dadd581081457169ca78df">
-<div class="sourceCode" id="cb7"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">fit</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/glm.html">glm</a></span><span class="op">(</span><span class="va">y</span> <span class="op">~</span> <span class="va">.</span>, data <span class="op">=</span> <span class="va">mnist_27</span><span class="op">$</span><span class="va">train</span>, family <span class="op">=</span> <span class="st">"binomial"</span><span class="op">)</span></span>
-<span><span class="va">p_hat</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">fit</span>, <span class="va">mnist_27</span><span class="op">$</span><span class="va">test</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-9_53f31926b6b7a3877f06bd1bd86df48e">
+<div class="sourceCode" id="cb8"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">fit</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/glm.html">glm</a></span><span class="op">(</span><span class="va">y</span> <span class="op">~</span> <span class="va">.</span>, data <span class="op">=</span> <span class="va">mnist_27</span><span class="op">$</span><span class="va">train</span>, family <span class="op">=</span> <span class="st">"binomial"</span><span class="op">)</span></span>
+<span><span class="va">p_hat</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">fit</span>, newdata <span class="op">=</span> <span class="va">mnist_27</span><span class="op">$</span><span class="va">test</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
-<p>In this case, the function is simply computing:</p>
+<p>In this case, the function is simply computing</p>
 <p><span class="math display">\[
 \hat{p}(\mathbf{x}) = g^{-1}\left(\hat{\beta}_0 + \hat{\beta}_1 x_1 + \hat{\beta}_2 x_2 \right) \text{ with } g(p) = \log\frac{p}{1-p} \implies g^{-1}(\mu) = \frac{1}{1-e^{-\mu}}
 \]</span></p>
 <p>for the <code>x_1</code> and <code>x_2</code> in the test set <code>mnist_27$test</code>. With these estimates in place, we can make our predictions and compute our accuracy:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-9_7d31c8e6840b8cb4ada96f231070af67">
-<div class="sourceCode" id="cb8"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">y_hat</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op">(</span><span class="va">p_hat</span> <span class="op">&gt;</span> <span class="fl">0.5</span>, <span class="fl">7</span>, <span class="fl">2</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-10_540c3761ac8f6a88ae94387840360c0f">
+<div class="sourceCode" id="cb9"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">y_hat</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/ifelse.html">ifelse</a></span><span class="op">(</span><span class="va">p_hat</span> <span class="op">&gt;</span> <span class="fl">0.5</span>, <span class="fl">7</span>, <span class="fl">2</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
-<p>However, note that <code>predict</code> does not always return objects of the same types; it depends on what type of object it is applied to. To learn about the specifics, you need to look at the help file specific for the type of fit object that is being used. The <code>predict</code> is actually a special type of function in R (called a <em>generic function</em>) that calls other functions depending on what kind of object it receives. So if <code>predict</code> receives an object coming out of the <code>lm</code> function, it will call <code>predict.glm</code>. If it receives an object coming out of <code>glm</code>, it calls <code>predict.qda</code>. If the fit is from <code>knn3</code>, it calls <code>predict.knn3</code>, and so on. These functions are similar but not exactly. You can learn more about the differences by reading the help files:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-10_bd685df86558439e32162720bed08966">
-<div class="sourceCode" id="cb9"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="op">?</span><span class="va">predict.glm</span></span>
+<p>However, note that <code>predict</code> does not always return objects of the same type; it depends on what type of object it is applied to. To learn about the specifics, you need to look at the help file specific for the type of fit object that is being used.</p>
+<p><code>predict</code> is actually a special type of function in R called a <em>generic function</em>. Generic functions call other functions depending on what kind of object it receives. So if <code>predict</code> receives an object coming out of the <code>lm</code> function, it will call <code>predict.lm</code>. If it receives an object coming out of <code>glm</code>, it calls <code>predict.glm</code>. If the fit is from <code>knn3</code>, it calls <code>predict.knn3</code>, and so on. These functions are similar but not exactly. You can learn more about the differences by reading the help files:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-11_8495789dce729292b14be85d5bf637c6">
+<div class="sourceCode" id="cb10"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="op">?</span><span class="va">predict.glm</span></span>
 <span><span class="op">?</span><span class="va">predict.qda</span></span>
 <span><span class="op">?</span><span class="va">predict.knn3</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
-<p>There are many other versions of <code>predict</code> and many machine learning algorithms have a <code>predict</code> function.</p>
+<p>There are many other versions of <code>predict</code> and many machine learning algorithms define their own <code>predict</code> function.</p>
 <p>As with <code>train</code>, the <strong>caret</strong> packages unifies the use of <code>predict</code> with the function <code>predict.train</code>. This function takes the output of <code>train</code> and produces prediction of categories or estimates of <span class="math inline">\(p(\mathbf{x})\)</span>.</p>
 <p>The code looks the same for all methods:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-11_8683ddf64fbc93af1d792e622aefa098">
-<div class="sourceCode" id="cb10"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">y_hat_glm</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_glm</span>, <span class="va">mnist_27</span><span class="op">$</span><span class="va">test</span>, type <span class="op">=</span> <span class="st">"raw"</span><span class="op">)</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-12_fd36d29fd1d7d100bb99ccf66c6aa227">
+<div class="sourceCode" id="cb11"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">y_hat_glm</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_glm</span>, <span class="va">mnist_27</span><span class="op">$</span><span class="va">test</span>, type <span class="op">=</span> <span class="st">"raw"</span><span class="op">)</span></span>
 <span><span class="va">y_hat_qda</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_qda</span>, <span class="va">mnist_27</span><span class="op">$</span><span class="va">test</span>, type <span class="op">=</span> <span class="st">"raw"</span><span class="op">)</span></span>
 <span><span class="va">y_hat_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_knn</span>, <span class="va">mnist_27</span><span class="op">$</span><span class="va">test</span>, type <span class="op">=</span> <span class="st">"raw"</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>This permits us to quickly compare the algorithms. For example, we can compare the accuracy like this:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-12_a6cdb42608eca224555e43d589ce114d">
-<div class="sourceCode" id="cb11"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">fits</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op">(</span>glm <span class="op">=</span> <span class="va">y_hat_glm</span>, qda <span class="op">=</span> <span class="va">y_hat_qda</span>, knn <span class="op">=</span> <span class="va">y_hat_knn</span><span class="op">)</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-13_7c188684a946eeaa590f12f2669aa14b">
+<div class="sourceCode" id="cb12"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">fits</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op">(</span>glm <span class="op">=</span> <span class="va">y_hat_glm</span>, qda <span class="op">=</span> <span class="va">y_hat_qda</span>, knn <span class="op">=</span> <span class="va">y_hat_knn</span><span class="op">)</span></span>
 <span><span class="fu"><a href="https://rdrr.io/r/base/lapply.html">sapply</a></span><span class="op">(</span><span class="va">fits</span>, <span class="kw">function</span><span class="op">(</span><span class="va">fit</span><span class="op">)</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="va">fit</span>, <span class="va">mnist_27</span><span class="op">$</span><span class="va">test</span><span class="op">$</span><span class="va">y</span><span class="op">)</span><span class="op">$</span><span class="va">overall</span><span class="op">[[</span><span class="st">"Accuracy"</span><span class="op">]</span><span class="op">]</span><span class="op">)</span></span>
 <span><span class="co">#&gt;   glm   qda   knn </span></span>
 <span><span class="co">#&gt; 0.775 0.815 0.835</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 </section><section id="sec-caret-cv" class="level3" data-number="31.1.3"><h3 data-number="31.1.3" class="anchored" data-anchor-id="sec-caret-cv">
-<span class="header-section-number">31.1.3</span> Cross validation</h3>
-<p>When an algorithm includes a tuning parameter, <code>train</code> automatically uses cross validation to decide among a few default values. To find out what parameter or parameters are optimized, you can read the manual <a href="#fn2" class="footnote-ref" id="fnref2" role="doc-noteref"><sup>2</sup></a> or study the output of:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-13_3ab2878739ebd6258109ed2ee986d7a4">
-<div class="sourceCode" id="cb12"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/modelLookup.html">getModelInfo</a></span><span class="op">(</span><span class="st">"knn"</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
-<p>We can also use a quick lookup like this:</p>
+<span class="header-section-number">31.1.3</span> Resampling</h3>
+<p>When an algorithm includes a tuning parameter, <code>train</code> automatically uses a resampling method to estimate MSE and decide among a few default candidate values. To find out what parameter or parameters are optimized, you can read the <strong>caret</strong> manual<a href="#fn2" class="footnote-ref" id="fnref2" role="doc-noteref"><sup>2</sup></a> or study the output of:</p>
 <div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-14_1e612f72578b39cb7e9d4bf08f422d8f">
 <div class="sourceCode" id="cb13"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/modelLookup.html">modelLookup</a></span><span class="op">(</span><span class="st">"knn"</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
+<p>To obtain all the details of how <strong>caret</strong> implements kNN you can use:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-15_523448f2ccec7070944f95aeff1da130">
+<div class="sourceCode" id="cb14"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/modelLookup.html">getModelInfo</a></span><span class="op">(</span><span class="st">"knn"</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
 <p>If we run it with default values:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-15_d48c2f289af4f8c00b65791cf8a65c43">
-<div class="sourceCode" id="cb14"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">y</span> <span class="op">~</span> <span class="va">.</span>, method <span class="op">=</span> <span class="st">"knn"</span>, data <span class="op">=</span> <span class="va">mnist_27</span><span class="op">$</span><span class="va">train</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-16_57670ca94bedf31698c451567a822f2f">
+<div class="sourceCode" id="cb15"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">y</span> <span class="op">~</span> <span class="va">.</span>, method <span class="op">=</span> <span class="st">"knn"</span>, data <span class="op">=</span> <span class="va">mnist_27</span><span class="op">$</span><span class="va">train</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>you can quickly see the results of the cross validation using the <code>ggplot</code> function. The argument <code>highlight</code> highlights the max:</p>
 <div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/caret-highlight_d17c6c5125c1ffb3ed5ff21ef6c364a2">
-<div class="sourceCode" id="cb15"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggplot.html">ggplot</a></span><span class="op">(</span><span class="va">train_knn</span>, highlight <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="sourceCode" id="cb16"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggplot.html">ggplot</a></span><span class="op">(</span><span class="va">train_knn</span>, highlight <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 <div class="cell-output-display">
 <div class="quarto-figure quarto-figure-center">
 <figure class="figure"><p><img src="ml-in-practice_files/figure-html/caret-highlight-1.png" class="img-fluid figure-img" style="width:70.0%"></p>
@@ -555,13 +561,13 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 </div>
 </div>
 </div>
-<p>By default, the cross validation is performed by taking 25 bootstrap samples comprised of 25% of the observations. For the <code>kNN</code> method, the default is to try <span class="math inline">\(k=5,7,9\)</span>. We change this using the <code>tuneGrid</code> argument. The grid of values must be supplied by a data frame with the parameter names as specified in the <code>modelLookup</code> output.</p>
-<p>Here we present an example where we try out 30 values between 9 and 67. To do this with <strong>caret</strong>, we need to define a column named <code>k</code>, so we use this: <code>data.frame(k = seq(9, 67, 2))</code>. Note that when running this code, we are fitting 30 versions of kNN to 25 bootstrapped samples. Since we are fitting <span class="math inline">\(30 \times 25 = 750\)</span> kNN models, running this code will take several seconds. We set the seed because cross validation is a random procedure and we want to make sure the result here is reproducible.</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/train-knn-plot_97100927b08b3dc802b2fa12e13e2553">
-<div class="sourceCode" id="cb16"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op">(</span><span class="fl">2008</span><span class="op">)</span></span>
-<span><span class="va">train_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">y</span> <span class="op">~</span> <span class="va">.</span>, method <span class="op">=</span> <span class="st">"knn"</span>, </span>
+<p>By default, the resampling is performed by taking 25 bootstrap samples, each comprised of 25% of the observations.</p>
+<p>For the <code>kNN</code> method, the default is to try <span class="math inline">\(k=5,7,9\)</span>. We change this using the <code>tuneGrid</code> argument. The grid of values must be supplied by a data frame with the parameter names as specified in the <code>modelLookup</code> output.</p>
+<p>Here we present an example where we try out 38 values between 1 and 75. To do this with <strong>caret</strong>, we need to define a column named <code>k</code>, so we use this: <code>data.frame(k = seq(1, 75, 2))</code>. Note that when running this code, we are fitting 38 versions of kNN to 25 bootstrapped samples. Since we are fitting <span class="math inline">\(38 \times 25 = 950\)</span> kNN models, running this code will take several seconds.</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/train-knn-plot_a25a1280e64c94dccc3ed5882a50a7fa">
+<div class="sourceCode" id="cb17"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">y</span> <span class="op">~</span> <span class="va">.</span>, method <span class="op">=</span> <span class="st">"knn"</span>, </span>
 <span>                   data <span class="op">=</span> <span class="va">mnist_27</span><span class="op">$</span><span class="va">train</span>,</span>
-<span>                   tuneGrid <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>k <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="fl">9</span>, <span class="fl">71</span>, <span class="fl">2</span><span class="op">)</span><span class="op">)</span><span class="op">)</span></span>
+<span>                   tuneGrid <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>k <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="fl">1</span>, <span class="fl">75</span>, <span class="fl">2</span><span class="op">)</span><span class="op">)</span><span class="op">)</span></span>
 <span><span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggplot.html">ggplot</a></span><span class="op">(</span><span class="va">train_knn</span>, highlight <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 <div class="cell-output-display">
 <div class="quarto-figure quarto-figure-center">
@@ -570,57 +576,61 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 </div>
 </div>
 </div>
+<div class="calout-note">
+<p>Because resampling methods are random procedures, the same code can result in different results. To assure reproducible results you should set the seed, as we did at the start of this chapter.</p>
+</div>
 <p>To access the parameter that maximized the accuracy, you can use this:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-16_edb857e7aef1e8075e67b91207bbca4f">
-<div class="sourceCode" id="cb17"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_knn</span><span class="op">$</span><span class="va">bestTune</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-17_06fe39ee46cadbcf9932ce8a98519105">
+<div class="sourceCode" id="cb18"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_knn</span><span class="op">$</span><span class="va">bestTune</span></span>
 <span><span class="co">#&gt;     k</span></span>
-<span><span class="co">#&gt; 27 61</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<span><span class="co">#&gt; 37 73</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>and the best performing model like this:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-17_13e982acc3a5d60398ac6e1dee7a55b9">
-<div class="sourceCode" id="cb18"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_knn</span><span class="op">$</span><span class="va">finalModel</span></span>
-<span><span class="co">#&gt; 61-nearest neighbor model</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-18_02a6bf0286b3852914d29f4e58216311">
+<div class="sourceCode" id="cb19"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_knn</span><span class="op">$</span><span class="va">finalModel</span></span>
+<span><span class="co">#&gt; 73-nearest neighbor model</span></span>
 <span><span class="co">#&gt; Training set outcome distribution:</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt;   2   7 </span></span>
 <span><span class="co">#&gt; 401 399</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>The function <code>predict</code> will use this best performing model. Here is the accuracy of the best model when applied to the test set, which we have not yet used because the cross validation was done on the training set:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-18_c45eec151fe0bc13be46164139737734">
-<div class="sourceCode" id="cb19"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_knn</span>, <span class="va">mnist_27</span><span class="op">$</span><span class="va">test</span>, type <span class="op">=</span> <span class="st">"raw"</span><span class="op">)</span>,</span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-19_c2f8eda8e2ea2522e8fde4b6594e9b35">
+<div class="sourceCode" id="cb20"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_knn</span>, <span class="va">mnist_27</span><span class="op">$</span><span class="va">test</span>, type <span class="op">=</span> <span class="st">"raw"</span><span class="op">)</span>,</span>
 <span>                <span class="va">mnist_27</span><span class="op">$</span><span class="va">test</span><span class="op">$</span><span class="va">y</span><span class="op">)</span><span class="op">$</span><span class="va">overall</span><span class="op">[</span><span class="st">"Accuracy"</span><span class="op">]</span></span>
 <span><span class="co">#&gt; Accuracy </span></span>
 <span><span class="co">#&gt;    0.825</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
-<p>If we want to change how we perform cross validation, we can use the <code>trainControl</code> function. We can make the code above go a bit faster by using, for example, 10-fold cross validation. This means we have 10 samples using 10% of the observations each. We accomplish this using the following code:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/cv-10-fold-accuracy-estimate_d4212fede48073bbaf84f9b33a8cfec9">
-<div class="sourceCode" id="cb20"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">control</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/trainControl.html">trainControl</a></span><span class="op">(</span>method <span class="op">=</span> <span class="st">"cv"</span>, number <span class="op">=</span> <span class="fl">10</span>, p <span class="op">=</span> <span class="fl">.9</span><span class="op">)</span></span>
+<p>Bootstrapping is not always the best approach to resampling (see <span class="citation" data-cites="knn-in-practice">@knn-in-practice</span>, for an example). If we want to change our resampling method, we can use the <code>trainControl</code> function. For example, the code below runs 10-fold cross validation. This means we have 10 samples using 90% of the observations to train in each sample. We accomplish this using the following code:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/cv-10-fold-accuracy-estimate_3effe6c18d096818efc52bcd767f686c">
+<div class="sourceCode" id="cb21"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">control</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/trainControl.html">trainControl</a></span><span class="op">(</span>method <span class="op">=</span> <span class="st">"cv"</span>, number <span class="op">=</span> <span class="fl">10</span>, p <span class="op">=</span> <span class="fl">.9</span><span class="op">)</span></span>
 <span><span class="va">train_knn_cv</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">y</span> <span class="op">~</span> <span class="va">.</span>, method <span class="op">=</span> <span class="st">"knn"</span>, </span>
 <span>                   data <span class="op">=</span> <span class="va">mnist_27</span><span class="op">$</span><span class="va">train</span>,</span>
-<span>                   tuneGrid <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>k <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="fl">9</span>, <span class="fl">71</span>, <span class="fl">2</span><span class="op">)</span><span class="op">)</span>,</span>
-<span>                   trControl <span class="op">=</span> <span class="va">control</span><span class="op">)</span></span>
-<span><span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggplot.html">ggplot</a></span><span class="op">(</span><span class="va">train_knn_cv</span>, highlight <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-<div class="cell-output-display">
-<div class="quarto-figure quarto-figure-center">
-<figure class="figure"><p><img src="ml-in-practice_files/figure-html/cv-10-fold-accuracy-estimate-1.png" class="img-fluid figure-img" style="width:70.0%"></p>
-</figure>
-</div>
+<span>                   tuneGrid <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>k <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="fl">1</span>, <span class="fl">71</span>, <span class="fl">2</span><span class="op">)</span><span class="op">)</span>,</span>
+<span>                   trControl <span class="op">=</span> <span class="va">control</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
+<div class="callout callout-style-simple callout-note">
+<div class="callout-body d-flex">
+<div class="callout-icon-container">
+<i class="callout-icon"></i>
 </div>
-<p>We observe that the accuracy estimates are more variable, which is expected since we changed the number of samples used to estimate accuracy.</p>
-<p>Note that <code>results</code> component of the <code>train</code> output includes several summary statistics related to the variability of the cross validation estimates:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-19_afd39f491444a16266ca8312c20c2e8c">
-<div class="sourceCode" id="cb21"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/names.html">names</a></span><span class="op">(</span><span class="va">train_knn</span><span class="op">$</span><span class="va">results</span><span class="op">)</span></span>
+<div class="callout-body-container">
+<p>The <code>results</code> component of the <code>train</code> output includes several summary statistics related to the variability of the cross validation estimates:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-20_7de6158c08fe12eb5d0050f60061a5a3">
+<div class="sourceCode" id="cb22"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/names.html">names</a></span><span class="op">(</span><span class="va">train_knn</span><span class="op">$</span><span class="va">results</span><span class="op">)</span></span>
 <span><span class="co">#&gt; [1] "k"          "Accuracy"   "Kappa"      "AccuracySD" "KappaSD"</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
-<p>You can learn many more details about the <strong>caret</strong> package, from the manual <a href="#fn3" class="footnote-ref" id="fnref3" role="doc-noteref"><sup>3</sup></a>.</p>
+<p>You can learn many more details about the <strong>caret</strong> package, from the manual<a href="#fn3" class="footnote-ref" id="fnref3" role="doc-noteref"><sup>3</sup></a>.</p>
+</div>
+</div>
+</div>
 </section></section><section id="preprocessing" class="level2" data-number="31.2"><h2 data-number="31.2" class="anchored" data-anchor-id="preprocessing">
 <span class="header-section-number">31.2</span> Preprocessing</h2>
 <p>We often transform predictors before running the machine algorithm. We also remove predictors that are clearly not useful. We call these steps <em>preprocessing</em>.</p>
 <p>Examples of preprocessing include standardizing the predictors, taking the log transform of some predictors, removing predictors that are highly correlated with others, and removing predictors with very few non-unique values or close to zero variation.</p>
 <p>For example, we can run the <code>nearZero</code> function from the <strong>caret</strong> package to see that several features do not vary much from observation to observation. We can see that there is a large number of features with close to 0 variability:</p>
 <div class="cell" data-layout-align="center">
-<div class="sourceCode" id="cb22"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://github.com/HenrikBengtsson/matrixStats">matrixStats</a></span><span class="op">)</span></span>
+<div class="sourceCode" id="cb23"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://github.com/HenrikBengtsson/matrixStats">matrixStats</a></span><span class="op">)</span></span>
 <span><span class="va">sds</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/matrixStats/man/rowSds.html">colSds</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span></span>
 <span><span class="fu"><a href="https://rdrr.io/r/graphics/hist.html">hist</a></span><span class="op">(</span><span class="va">sds</span>, breaks <span class="op">=</span> <span class="fl">256</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 <div class="cell-output-display">
@@ -632,15 +642,15 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 </div>
 <p>This is expected because there are parts of the image that rarely contain writing (dark pixels).</p>
 <p>The <strong>caret</strong> packages includes a function that recommends features to be removed due to <em>near zero variance</em>:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-20_f94456d5577c664be9aaeb56fb3f8e28">
-<div class="sourceCode" id="cb23"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">nzv</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/nearZeroVar.html">nearZeroVar</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-21_a119d945d13a3562b9a76467da225d9b">
+<div class="sourceCode" id="cb24"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">nzv</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/nearZeroVar.html">nearZeroVar</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>We can see the columns recommended for removal are the near the edges:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-21_a8860d50f954b61c53b3397a9f737a08">
-<div class="sourceCode" id="cb24"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/graphics/image.html">image</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/matrix.html">matrix</a></span><span class="op">(</span><span class="fl">1</span><span class="op">:</span><span class="fl">784</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">nzv</span>, <span class="fl">28</span>, <span class="fl">28</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-22_7962e7138f73e94aa0b15e96b1a6ca78">
+<div class="sourceCode" id="cb25"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/graphics/image.html">image</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/matrix.html">matrix</a></span><span class="op">(</span><span class="fl">1</span><span class="op">:</span><span class="fl">784</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">nzv</span>, <span class="fl">28</span>, <span class="fl">28</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/near-zero-image_8b5146573c82c3b0b8098b5899086a71">
-<div class="sourceCode" id="cb25"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu">rafalib</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/rafalib/man/mypar.html">mypar</a></span><span class="op">(</span><span class="op">)</span></span>
+<div class="sourceCode" id="cb26"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu">rafalib</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/rafalib/man/mypar.html">mypar</a></span><span class="op">(</span><span class="op">)</span></span>
 <span><span class="fu"><a href="https://rdrr.io/r/graphics/image.html">image</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/matrix.html">matrix</a></span><span class="op">(</span><span class="fl">1</span><span class="op">:</span><span class="fl">784</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">nzv</span>, <span class="fl">28</span>, <span class="fl">28</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 <div class="cell-output-display">
 <div class="quarto-figure quarto-figure-center">
@@ -649,127 +659,153 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 </div>
 </div>
 </div>
-<p>So we end up keeping this number of columns:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-22_d31a4c09920d473f474883077d85e35e">
-<div class="sourceCode" id="cb26"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">col_index</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/sets.html">setdiff</a></span><span class="op">(</span><span class="fl">1</span><span class="op">:</span><span class="fu"><a href="https://rdrr.io/r/base/nrow.html">ncol</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span>, <span class="va">nzv</span><span class="op">)</span></span>
-<span><span class="fu"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op">(</span><span class="va">col_index</span><span class="op">)</span></span>
-<span><span class="co">#&gt; [1] 252</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<p>So we end up removing</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-23_22ad4bf75a0f6dfee9dcd94c2903a886">
+<div class="sourceCode" id="cb27"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/length.html">length</a></span><span class="op">(</span><span class="va">nzv</span><span class="op">)</span></span>
+<span><span class="co">#&gt; [1] 532</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>predictors.</p>
+<p>The <strong>caret</strong> package features the <code>preProcess</code> function, which allows users to establish a predefined set of preprocessing operations based on a training set. This function is designed to apply these operations to new datasets without recalculating anything on the test set, ensuring that all preprocessing steps are consistent and derived solely from the training data.</p>
+<p>Below is an example demonstrating how to remove predictors with near-zero variance and then center the remaining predictors:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-24_f0d170efcc1040ef6a3188e0bbd9b7ea">
+<div class="sourceCode" id="cb28"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">pp</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/preProcess.html">preProcess</a></span><span class="op">(</span><span class="va">x</span>, method <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op">(</span><span class="st">"nzv"</span>, <span class="st">"center"</span><span class="op">)</span><span class="op">)</span></span>
+<span><span class="va">centered_subsetted_x_test</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">pp</span>, newdata <span class="op">=</span> <span class="va">x_test</span><span class="op">)</span></span>
+<span><span class="fu"><a href="https://rdrr.io/r/base/dim.html">dim</a></span><span class="op">(</span><span class="va">centered_subsetted_x_test</span><span class="op">)</span></span>
+<span><span class="co">#&gt; [1] 1000  252</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>Additionally, the <code>train</code> function in caret includes a <code>preProcess</code> argument that allows users to specify which preprocessing steps to apply automatically during model training. We will explore this functionality further in the context of a k-Nearest Neighbors model <span class="citation" data-cites="knn-in-practice">@knn-in-practice</span>.</p>
+</section><section id="parallelization" class="level2" data-number="31.3"><h2 data-number="31.3" class="anchored" data-anchor-id="parallelization">
+<span class="header-section-number">31.3</span> Parallelization</h2>
+<p>During cross-validation or bootstrapping, the process of fitting models to different samples or using varying parameters can be performed independently. Imagine you are fitting 100 models; if you had access to 100 computers, you could theoretically speed up the process by a factor of 100 by fitting each model on a separate computer and then aggregating the results. In reality, most modern computers, including many personal computers, are equipped with multiple processors that allow for such parallel execution. This method, known as parallelization, leverages these multiple processors to conduct several computational tasks simultaneously, significantly accelerating the model training process. By distributing the workload across different processors, parallelization makes it feasible to manage large datasets and complex modeling procedures efficiently.</p>
+<p>The <strong>caret</strong> package is set up to run in parallel but you have to let R know that you are want to parallelize your work. To do this we can use the <strong>doParallel</strong> package:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-25_ce68a1f3604c0ad00e7299ddeb4fa3cd">
+<div class="sourceCode" id="cb29"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://github.com/RevolutionAnalytics/doparallel">doParallel</a></span><span class="op">)</span></span>
+<span><span class="co">#&gt; Loading required package: foreach</span></span>
+<span><span class="co">#&gt; Loading required package: iterators</span></span>
+<span><span class="co">#&gt; Loading required package: parallel</span></span>
+<span><span class="va">nc</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/parallel/detectCores.html">detectCores</a></span><span class="op">(</span><span class="op">)</span> <span class="op">-</span> <span class="fl">1</span></span>
+<span><span class="va">cl</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/parallel/makeCluster.html">makeCluster</a></span><span class="op">(</span><span class="va">nc</span><span class="op">)</span> <span class="co"># it is convention to leave 1 core for OS</span></span>
+<span><span class="fu"><a href="https://rdrr.io/pkg/doParallel/man/registerDoParallel.html">registerDoParallel</a></span><span class="op">(</span><span class="va">cl</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>If you do use parallelization, make sure to let R know you are done with the following lines of code:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-26_5aefab4a678229dc62f432f3602fab09">
+<div class="sourceCode" id="cb30"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/parallel/makeCluster.html">stopCluster</a></span><span class="op">(</span><span class="va">cl</span><span class="op">)</span></span>
+<span><span class="fu"><a href="https://rdrr.io/pkg/doParallel/man/registerDoParallel.html">stopImplicitCluster</a></span><span class="op">(</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
-<p>Now we are ready to fit some models. Before we start, we need to add column names to the feature matrices as these are required by <strong>caret</strong>:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-23_6b365df90dbda653edbd3e4b02d4c4d9">
-<div class="sourceCode" id="cb27"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/colnames.html">colnames</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span> <span class="op">&lt;-</span> <span class="fl">1</span><span class="op">:</span><span class="fu"><a href="https://rdrr.io/r/base/nrow.html">ncol</a></span><span class="op">(</span><span class="va">mnist</span><span class="op">$</span><span class="va">train</span><span class="op">$</span><span class="va">images</span><span class="op">)</span></span>
-<span><span class="fu"><a href="https://rdrr.io/r/base/colnames.html">colnames</a></span><span class="op">(</span><span class="va">x_test</span><span class="op">)</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/colnames.html">colnames</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
-</section><section id="k-nearest-neighbors" class="level2" data-number="31.3"><h2 data-number="31.3" class="anchored" data-anchor-id="k-nearest-neighbors">
-<span class="header-section-number">31.3</span> k-nearest neighbors</h2>
 <div class="callout callout-style-simple callout-warning">
 <div class="callout-body d-flex">
 <div class="callout-icon-container">
 <i class="callout-icon"></i>
 </div>
 <div class="callout-body-container">
-<p>Before starting this section, note that the first two calls to the <code>train</code> function in the code below can take several hours to run. This is a common challenge when training machine learning algorithms since we have to run the algorithm for each cross validation split and each set of tuning parameters being considered. In the next section, we will provide some suggestions on how to predict the duration of the process and ways to reduce.</p>
+<p>When parallelizing tasks across multiple processors, it’s important to consider the risk of running out of memory. Each processor might require a copy of the data or substantial portions of it, which can multiply overall memory demands. This is especially challenging if the data or models are large.</p>
 </div>
 </div>
 </div>
-<p>The first step is to optimize for <span class="math inline">\(k\)</span>.</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/mnist-knn-fit_6456da2f79a11665e0f69ed74ad8e6de">
-<div class="sourceCode" id="cb28"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">x</span><span class="op">[</span> ,<span class="va">col_index</span><span class="op">]</span>, <span class="va">y</span>, </span>
-<span>                   method <span class="op">=</span> <span class="st">"knn"</span>, </span>
-<span>                   tuneGrid <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>k <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="fl">3</span>, <span class="fl">13</span>, <span class="fl">2</span><span class="op">)</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
-<p>Once we optimize our algorithm, we can fit it to the entire dataset:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-24_0ebca7c1b3c7b4bd3300b097f71209ef">
-<div class="sourceCode" id="cb29"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">fit_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/knn3.html">knn3</a></span><span class="op">(</span><span class="va">x</span><span class="op">[</span>, <span class="va">col_index</span><span class="op">]</span>, <span class="va">y</span>,  k <span class="op">=</span> <span class="va">train_knn</span><span class="op">$</span><span class="va">bestTune</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
-<p>We achieve a high accuracy:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-26_b133fbe3cb13da6bf396cc724222fc90">
-<div class="sourceCode" id="cb30"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">y_hat_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">fit_knn</span>, <span class="va">x_test</span><span class="op">[</span>, <span class="va">col_index</span><span class="op">]</span>, type <span class="op">=</span> <span class="st">"class"</span><span class="op">)</span></span>
-<span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="va">y_hat_knn</span>, <span class="fu"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op">(</span><span class="va">y_test</span><span class="op">)</span><span class="op">)</span><span class="op">$</span><span class="va">overall</span><span class="op">[</span><span class="st">"Accuracy"</span><span class="op">]</span></span>
-<span><span class="co">#&gt; Accuracy </span></span>
-<span><span class="co">#&gt;    0.944</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
-<section id="dimension-reduction-with-pca" class="level3" data-number="31.3.1"><h3 data-number="31.3.1" class="anchored" data-anchor-id="dimension-reduction-with-pca">
-<span class="header-section-number">31.3.1</span> Dimension reduction with PCA</h3>
-<p>An alternative to removing low variance columns directly is to use dimension reduction on the feature matrix before applying the algorithms. It is important that we not use the test set when finding the PCs nor any summary of the data, as this could result in overtraining. So we start by applying <code>prcomp</code> to the training data:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-27_96ccfc06d2a8c6ac0c0f32b134600967">
-<div class="sourceCode" id="cb31"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">col_means</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/colSums.html">colMeans</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span></span>
-<span><span class="va">pca</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/prcomp.html">prcomp</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/sweep.html">sweep</a></span><span class="op">(</span><span class="va">x</span>, <span class="fl">2</span>, <span class="va">col_means</span><span class="op">)</span>, center <span class="op">=</span> <span class="cn">FALSE</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
-<p>Next, we run kNN on just a small number of dimensions. We try 36 dimensions since this explains about 80% of the data.</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-28_f25d49e9a9c4257f2f3f8dc935d1c233">
-<div class="sourceCode" id="cb32"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">k</span> <span class="op">&lt;-</span> <span class="fl">36</span></span>
-<span><span class="va">x_train</span> <span class="op">&lt;-</span> <span class="va">pca</span><span class="op">$</span><span class="va">x</span><span class="op">[</span>,<span class="fl">1</span><span class="op">:</span><span class="va">k</span><span class="op">]</span></span>
-<span><span class="va">train_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">x_train</span>, <span class="va">y</span>, </span>
-<span>                   method <span class="op">=</span> <span class="st">"knn"</span>, </span>
-<span>                   tuneGrid <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>k <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="fl">3</span>, <span class="fl">13</span>, <span class="fl">2</span><span class="op">)</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="va">fit_knn_pca</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/knn3.html">knn3</a></span><span class="op">(</span><span class="va">x_train</span>, <span class="va">y</span>, k <span class="op">=</span> <span class="va">train_knn</span><span class="op">$</span><span class="va">bestTune</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
-<p>Now we apply the transformation we learned with the training data to the test data, reduce the dimension, and then run predict. Note that we used the rotation and column means estimated from the training data.</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-30_a730fff03031462fdb3978f210950188">
-<div class="sourceCode" id="cb33"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">newdata</span> <span class="op">&lt;-</span>  <span class="fu"><a href="https://rdrr.io/r/base/sweep.html">sweep</a></span><span class="op">(</span><span class="va">x_test</span>, <span class="fl">2</span>, <span class="va">col_means</span><span class="op">)</span> <span class="op"><a href="https://rdrr.io/r/base/matmult.html">%*%</a></span> <span class="va">pca</span><span class="op">$</span><span class="va">rotation</span><span class="op">[</span>,<span class="fl">1</span><span class="op">:</span><span class="va">k</span><span class="op">]</span></span>
-<span><span class="va">y_hat_knn_pca</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">fit_knn_pca</span>, <span class="va">newdata</span>, type <span class="op">=</span> <span class="st">"class"</span><span class="op">)</span></span>
-<span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="va">y_hat_knn_pca</span>, <span class="fu"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op">(</span><span class="va">y_test</span><span class="op">)</span><span class="op">)</span><span class="op">$</span><span class="va">overall</span><span class="op">[</span><span class="st">"Accuracy"</span><span class="op">]</span></span>
-<span><span class="co">#&gt; Accuracy </span></span>
-<span><span class="co">#&gt;     0.96</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
-<p>We obtain an improvement in accuracy, while using only 36 dimensions.</p>
-<div class="callout callout-style-simple callout-note">
+</section><section id="knn-in-practice" class="level2" data-number="31.4"><h2 data-number="31.4" class="anchored" data-anchor-id="knn-in-practice">
+<span class="header-section-number">31.4</span> k-nearest neighbors</h2>
+<div class="callout callout-style-simple callout-warning">
 <div class="callout-body d-flex">
 <div class="callout-icon-container">
 <i class="callout-icon"></i>
 </div>
 <div class="callout-body-container">
-<p>Remember the entire algorithm needs to be developed on the training data. This is why the column means, subtracted from test set columns, and the rotation applied to obtained PCs, were both computed using the train data.</p>
+<p>Before starting this section, note that the first two calls to the <code>train</code> function in the code below can take several hours to run. This is a common challenge when training machine learning algorithms since we have to run the algorithm for each cross validation split and each set of tuning parameters being considered. In the next section, we will provide some suggestions on how to predict the duration of the process and ways to reduce.</p>
 </div>
 </div>
 </div>
-</section></section><section id="random-forest" class="level2" data-number="31.4"><h2 data-number="31.4" class="anchored" data-anchor-id="random-forest">
-<span class="header-section-number">31.4</span> Random Forest</h2>
+<p>As we will see soon, the optimal <span class="math inline">\(k\)</span> for the MNIST data is between 1 and 7. For small values of <span class="math inline">\(k\)</span>, bootstrapping can be problematic for estimating MSE for a k-Nearest Neighbors (kNN). This is because bootstrapping involves sampling with replacement from the original dataset which implies the closest neighbor will often appear twice but considered two independent observations by kNN. This is a unrealistic scenario that can distort the estimate MSE when, for example, <span class="math inline">\(k=3\)</span>. We therefore us cross-validation to estimate our MSE.</p>
+<p>The first step is to optimize for <span class="math inline">\(k\)</span>.</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/mnist-knn-fit_bc5e991862b218e14ec928604f3de51e">
+<div class="sourceCode" id="cb31"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">x</span>, <span class="va">y</span>, method <span class="op">=</span> <span class="st">"knn"</span>, </span>
+<span>                   preProcess <span class="op">=</span> <span class="st">"nzv"</span>,</span>
+<span>                   trControl <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/trainControl.html">trainControl</a></span><span class="op">(</span><span class="st">"cv"</span>, number <span class="op">=</span> <span class="fl">20</span>, p <span class="op">=</span> <span class="fl">0.95</span><span class="op">)</span>,</span>
+<span>                   tuneGrid <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>k <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="fl">1</span>, <span class="fl">7</span>, <span class="fl">2</span><span class="op">)</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>Once we optimize our algorithm, the <code>predict</code> function defaults to using the best performing algorithm fit with the entire training data:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-27_c7ecbe6ba9d3853a89f7c37d06b7ad80">
+<div class="sourceCode" id="cb32"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">y_hat_knn</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_knn</span>, <span class="va">x_test</span>, type <span class="op">=</span> <span class="st">"raw"</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>We achieve relatively high accuracy:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-28_d1bd89df5f12d19a9b361e727f6746a4">
+<div class="sourceCode" id="cb33"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="va">y_hat_knn</span>, <span class="fu"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op">(</span><span class="va">y_test</span><span class="op">)</span><span class="op">)</span><span class="op">$</span><span class="va">overall</span><span class="op">[</span><span class="st">"Accuracy"</span><span class="op">]</span></span>
+<span><span class="co">#&gt; Accuracy </span></span>
+<span><span class="co">#&gt;    0.953</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<section id="dimension-reduction-with-pca" class="level3" data-number="31.4.1"><h3 data-number="31.4.1" class="anchored" data-anchor-id="dimension-reduction-with-pca">
+<span class="header-section-number">31.4.1</span> Dimension reduction with PCA</h3>
+<p>An alternative to removing low variance columns directly is to use dimension reduction on the feature matrix before applying the algorithms.</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-29_85570926ab62cc37a8e82db4fa222970">
+<div class="sourceCode" id="cb34"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">pca</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/prcomp.html">prcomp</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>We can actually explain, say, 75% of the variability in the predictors with a small number of dimensions:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-30_af184b944f80a2ae3244f8ee3865a659">
+<div class="sourceCode" id="cb35"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">p</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/which.html">which</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/cumsum.html">cumsum</a></span><span class="op">(</span><span class="va">pca</span><span class="op">$</span><span class="va">sdev</span><span class="op">^</span><span class="fl">2</span><span class="op">)</span><span class="op">/</span><span class="fu"><a href="https://rdrr.io/r/base/sum.html">sum</a></span><span class="op">(</span><span class="va">pca</span><span class="op">$</span><span class="va">sdev</span><span class="op">^</span><span class="fl">2</span><span class="op">)</span> <span class="op">&gt;=</span> <span class="fl">0.75</span><span class="op">)</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>We can then re-run our algorithm on these 33 features:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-31_6f12768e93589e3a7f45c29fc89f2a6e">
+<div class="sourceCode" id="cb36"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">fit_knn_pca</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/knn3.html">knn3</a></span><span class="op">(</span><span class="va">pca</span><span class="op">$</span><span class="va">x</span><span class="op">[</span>,<span class="fl">1</span><span class="op">:</span><span class="va">p</span><span class="op">]</span>, <span class="va">y</span>, k <span class="op">=</span> <span class="va">train_knn</span><span class="op">$</span><span class="va">bestTune</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>When predicting, it is important that we not use the test set when finding the PCs nor any summary of the data, as this could result in overtraining. We therefrom compute the averages needed for centering and the rotation on the training set:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-32_f681fac6b38c412a85785c385e18bd43">
+<div class="sourceCode" id="cb37"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">newdata</span> <span class="op">&lt;-</span>  <span class="fu"><a href="https://rdrr.io/r/base/sweep.html">sweep</a></span><span class="op">(</span><span class="va">x_test</span>, <span class="fl">2</span>, <span class="fu"><a href="https://rdrr.io/r/base/colSums.html">colMeans</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span><span class="op">)</span> <span class="op"><a href="https://rdrr.io/r/base/matmult.html">%*%</a></span> <span class="va">pca</span><span class="op">$</span><span class="va">rotation</span><span class="op">[</span>,<span class="fl">1</span><span class="op">:</span><span class="va">p</span><span class="op">]</span></span>
+<span><span class="va">y_hat_knn_pca</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">fit_knn_pca</span>, <span class="va">newdata</span>, type <span class="op">=</span> <span class="st">"class"</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>We obtain similar accuracy, while using only 33 dimensions:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-33_a8aa15136fa4cacd6c7cf7d51dfdf5af">
+<div class="sourceCode" id="cb38"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="va">y_hat_knn_pca</span>, <span class="fu"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op">(</span><span class="va">y_test</span><span class="op">)</span><span class="op">)</span><span class="op">$</span><span class="va">overall</span><span class="op">[</span><span class="st">"Accuracy"</span><span class="op">]</span></span>
+<span><span class="co">#&gt; Accuracy </span></span>
+<span><span class="co">#&gt;    0.959</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>In this example, we used the <span class="math inline">\(k\)</span> optimized for the raw data, not the principal components. Note that to obtain an unbiased MSE estimate we have to recompute the PCA for each cross-validation sample and apply to the validation set. Because the <code>train</code> function includes PCA as one of the available preprocessing operations we can achieve this with this modification of the code above:</p>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-34_d82c6f1c0c7e40f767390b634155789a">
+<div class="sourceCode" id="cb39"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_knn_pca</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">x</span>, <span class="va">y</span>, method <span class="op">=</span> <span class="st">"knn"</span>, </span>
+<span>                       preProcess <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op">(</span><span class="st">"nzv"</span>, <span class="st">"pca"</span><span class="op">)</span>,</span>
+<span>                       trControl <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/trainControl.html">trainControl</a></span><span class="op">(</span><span class="st">"cv"</span>, number <span class="op">=</span> <span class="fl">20</span>, p <span class="op">=</span> <span class="fl">0.95</span>,</span>
+<span>                                                preProcOptions <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/list.html">list</a></span><span class="op">(</span>pcaComp <span class="op">=</span> <span class="va">p</span><span class="op">)</span><span class="op">)</span>,</span>
+<span>                       tuneGrid <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>k <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="fl">1</span>, <span class="fl">7</span>, <span class="fl">2</span><span class="op">)</span><span class="op">)</span><span class="op">)</span></span>
+<span><span class="va">y_hat_knn_pca</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_knn_pca</span>, <span class="va">x_test</span>, type <span class="op">=</span> <span class="st">"raw"</span><span class="op">)</span></span>
+<span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="va">y_hat_knn_pca</span>, <span class="fu"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op">(</span><span class="va">y_test</span><span class="op">)</span><span class="op">)</span><span class="op">$</span><span class="va">overall</span><span class="op">[</span><span class="st">"Accuracy"</span><span class="op">]</span></span>
+<span><span class="co">#&gt; Accuracy </span></span>
+<span><span class="co">#&gt;    0.964</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+</div>
+<p>A limitation of this approach is that we don’t get to optimize the number of PCs used in the analysis. To do this we need to write our own method. The <strong>caret</strong> manual<a href="#fn4" class="footnote-ref" id="fnref4" role="doc-noteref"><sup>4</sup></a> describe how to do this.</p>
+</section></section><section id="random-forest" class="level2" data-number="31.5"><h2 data-number="31.5" class="anchored" data-anchor-id="random-forest">
+<span class="header-section-number">31.5</span> Random Forest</h2>
 <p>With the random forest algorithm several parameters can be optimized, but the main one is <code>mtry</code>, the number of predictors that are randomly selected for each tree. This is also the only tuning parameter that the <strong>caret</strong> function <code>train</code> permits when using the default implementation from the <strong>randomForest</strong> package.</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-31_a9b34eb185ec94570edf1893e4bfb3f0">
-<div class="sourceCode" id="cb34"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://www.stat.berkeley.edu/~breiman/RandomForests/">randomForest</a></span><span class="op">)</span></span>
-<span><span class="va">grid</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>mtry <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="fl">3</span>, <span class="fl">24</span>, <span class="fl">3</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="va">train_rf</span> <span class="op">&lt;-</span>  <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">x</span><span class="op">[</span>, <span class="va">col_index</span><span class="op">]</span>, <span class="va">y</span>, </span>
-<span>                   method <span class="op">=</span> <span class="st">"rf"</span>, </span>
-<span>                   tuneGrid <span class="op">=</span> <span class="va">grid</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-35_0ad0e70babedb44ca14767c7f41f3280">
+<div class="sourceCode" id="cb40"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://www.stat.berkeley.edu/~breiman/RandomForests/">randomForest</a></span><span class="op">)</span></span>
+<span><span class="va">train_rf</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">x</span>, <span class="va">y</span>, method <span class="op">=</span> <span class="st">"rf"</span>, </span>
+<span>                  preProcess <span class="op">=</span> <span class="st">"nzv"</span>,</span>
+<span>                  tuneGrid <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>mtry <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/seq.html">seq</a></span><span class="op">(</span><span class="fl">5</span>, <span class="fl">15</span><span class="op">)</span><span class="op">)</span><span class="op">)</span></span>
+<span><span class="va">y_hat_rf</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_rf</span>, <span class="va">x_test</span>, type <span class="op">=</span> <span class="st">"raw"</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>Now that we have optimized our algorithm, we are ready to fit our final model:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-32_722be15713298270f2566c696ca8b09b">
-<div class="sourceCode" id="cb35"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">fit_rf</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/randomForest/man/randomForest.html">randomForest</a></span><span class="op">(</span><span class="va">x</span><span class="op">[</span>, <span class="va">col_index</span><span class="op">]</span>, <span class="va">y</span>, mtry <span class="op">=</span> <span class="va">train_rf</span><span class="op">$</span><span class="va">bestTune</span><span class="op">$</span><span class="va">mtry</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-36_26395e1bb32a2c1f4df9f92abb8610a9">
+<div class="sourceCode" id="cb41"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">y_hat_rf</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_rf</span>, <span class="va">x_test</span>, type <span class="op">=</span> <span class="st">"raw"</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>As with kNN, we also achieve high accuracy:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-35_6daf69f16bff72bb1a695eb57df5f3e5">
-<div class="sourceCode" id="cb36"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">y_hat_rf</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">fit_rf</span>, <span class="va">x_test</span><span class="op">[</span> ,<span class="va">col_index</span><span class="op">]</span><span class="op">)</span></span>
-<span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="va">y_hat_rf</span>, <span class="va">y_test</span><span class="op">)</span><span class="op">$</span><span class="va">overall</span><span class="op">[</span><span class="st">"Accuracy"</span><span class="op">]</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-38_65e346992f226fcba7ffc96c9fbb28a7">
+<div class="sourceCode" id="cb42"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="va">y_hat_rf</span>, <span class="va">y_test</span><span class="op">)</span><span class="op">$</span><span class="va">overall</span><span class="op">[</span><span class="st">"Accuracy"</span><span class="op">]</span></span>
 <span><span class="co">#&gt; Accuracy </span></span>
-<span><span class="co">#&gt;    0.955</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<span><span class="co">#&gt;    0.952</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>By optimizing some of the other algorithm parameters, we can achieve even higher accuracy.</p>
-</section><section id="testing-and-improving-computation-time" class="level2" data-number="31.5"><h2 data-number="31.5" class="anchored" data-anchor-id="testing-and-improving-computation-time">
-<span class="header-section-number">31.5</span> Testing and improving computation time</h2>
+</section><section id="testing-and-improving-computation-time" class="level2" data-number="31.6"><h2 data-number="31.6" class="anchored" data-anchor-id="testing-and-improving-computation-time">
+<span class="header-section-number">31.6</span> Testing and improving computation time</h2>
 <p>The default method for estimating accuracy used by the <code>train</code> function is to test prediction on 25 bootstrap samples. This can result in long compute times. For example, if we are considering several values, say 10, of the tuning parameters, we will fit the algorithm 250 times. We can use the <code>system.time</code> function to estimate how long it takes to run the algorithm once:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-36_0f8756ca9d2f7277fb54fe0000023a5a">
-<div class="sourceCode" id="cb37"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/system.time.html">system.time</a></span><span class="op">(</span><span class="op">{</span><span class="va">fit_rf</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/randomForest/man/randomForest.html">randomForest</a></span><span class="op">(</span><span class="va">x</span><span class="op">[</span>, <span class="va">col_index</span><span class="op">]</span>, <span class="va">y</span>,  mtry <span class="op">=</span> <span class="fl">9</span><span class="op">)</span><span class="op">}</span><span class="op">)</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-39_b35e98346f2ac8f94baed79385372aa1">
+<div class="sourceCode" id="cb43"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">nzv</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/nearZeroVar.html">nearZeroVar</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span></span>
+<span><span class="fu"><a href="https://rdrr.io/r/base/system.time.html">system.time</a></span><span class="op">(</span><span class="op">{</span><span class="va">fit_rf</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/randomForest/man/randomForest.html">randomForest</a></span><span class="op">(</span><span class="va">x</span><span class="op">[</span>, <span class="op">-</span><span class="va">nzv</span><span class="op">]</span>, <span class="va">y</span>,  mtry <span class="op">=</span> <span class="fl">9</span><span class="op">)</span><span class="op">}</span><span class="op">)</span></span>
 <span><span class="co">#&gt;    user  system elapsed </span></span>
-<span><span class="co">#&gt;  59.707   0.046  59.791</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<span><span class="co">#&gt;  61.656   0.053  62.368</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>and use this to estimate the total time for the 250 iterations. In this case it will be several hours.</p>
-<p>One way to reduce run time is to use k-fold cross validation with a smaller number of test sets. A popular choice is leaving out 10 test sets with 10% of the data:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/mnist-knn-fit-control_7d84fe9432f08fd5a97cb5a70aafd808">
-<div class="sourceCode" id="cb38"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">control</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/caret/man/trainControl.html">trainControl</a></span><span class="op">(</span>method <span class="op">=</span> <span class="st">"cv"</span>, number <span class="op">=</span> <span class="fl">10</span>, p <span class="op">=</span> <span class="fl">.9</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
-<p>and re-running the <code>train</code> function with this choice specified via the <code>trControl</code> argument:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-37_b017d8d2ef32d5f63818675e206d69a4">
-<div class="sourceCode" id="cb39"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">train_rf</span> <span class="op">&lt;-</span>  <span class="fu"><a href="https://rdrr.io/pkg/caret/man/train.html">train</a></span><span class="op">(</span><span class="va">x</span><span class="op">[</span>, <span class="va">col_index</span><span class="op">]</span>, <span class="va">y</span>, </span>
-<span>                   method <span class="op">=</span> <span class="st">"rf"</span>, </span>
-<span>                   tuneGrid <span class="op">=</span> <span class="va">grid</span>,</span>
-<span>                   trControl <span class="op">=</span> <span class="va">control</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
+<p>One way to reduce run time is to use k-fold cross validation with a smaller number of test sets. A popular choice is leaving out 5 test sets with 20% of the data. To use this we set the <code>trControl</code> argument in train to <code>trainControl(method = "cv", number = 5, p = .8)</code>.</p>
 <p>For random forest, we can also speed up the training step by running less trees per fit. After running the algorithm once, we can use the plot function to see how the error rate changes as the number of trees grows.</p>
 <p>Here we can see that error rate stabilizes after about 200 trees:</p>
 <div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/error-versus-number-of-trees_c3dbdd33d9ce465d12aeac6a65abed12">
-<div class="sourceCode" id="cb40"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/graphics/plot.default.html">plot</a></span><span class="op">(</span><span class="va">fit_rf</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="sourceCode" id="cb44"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/graphics/plot.default.html">plot</a></span><span class="op">(</span><span class="va">fit_rf</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 <div class="cell-output-display">
 <div class="quarto-figure quarto-figure-center">
 <figure class="figure"><p><img src="ml-in-practice_files/figure-html/error-versus-number-of-trees-1.png" class="img-fluid figure-img" style="width:70.0%"></p>
@@ -778,19 +814,19 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 </div>
 </div>
 <p>We can use this finding to speed up the cross validation procedure. Specifically, because the default is 500, by adding the argument <code>ntree = 200</code> to the call to <code>train</code> above, the procedure will finish 2.5 times faster.</p>
-</section><section id="variable-importance" class="level2" data-number="31.6"><h2 data-number="31.6" class="anchored" data-anchor-id="variable-importance">
-<span class="header-section-number">31.6</span> Variable importance</h2>
+</section><section id="variable-importance" class="level2" data-number="31.7"><h2 data-number="31.7" class="anchored" data-anchor-id="variable-importance">
+<span class="header-section-number">31.7</span> Variable importance</h2>
 <p>The following function computes the importance of each feature:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-38_1c9088636bd1bfcbfb2bad4dfd057f46">
-<div class="sourceCode" id="cb41"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">imp</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/randomForest/man/importance.html">importance</a></span><span class="op">(</span><span class="va">fit_rf</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-40_3c9f79bcb3818c4e90b0b79393d48149">
+<div class="sourceCode" id="cb45"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">imp</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/pkg/randomForest/man/importance.html">importance</a></span><span class="op">(</span><span class="va">fit_rf</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>We can see which features are being used most by plotting an image:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-39_b8bb34688fefaf9981c9c365ae9e26d1">
-<div class="sourceCode" id="cb42"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">mat</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op">(</span><span class="fl">0</span>, <span class="fu"><a href="https://rdrr.io/r/base/nrow.html">ncol</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span><span class="op">)</span></span>
-<span><span class="va">mat</span><span class="op">[</span><span class="va">col_index</span><span class="op">]</span> <span class="op">&lt;-</span> <span class="va">imp</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-41_9d98d4ee7084f05f313e0c2265ce23b7">
+<div class="sourceCode" id="cb46"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">mat</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/rep.html">rep</a></span><span class="op">(</span><span class="fl">0</span>, <span class="fu"><a href="https://rdrr.io/r/base/nrow.html">ncol</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span><span class="op">)</span></span>
+<span><span class="va">mat</span><span class="op">[</span><span class="op">-</span><span class="va">nzv</span><span class="op">]</span> <span class="op">&lt;-</span> <span class="va">imp</span></span>
 <span><span class="fu"><a href="https://rdrr.io/r/graphics/image.html">image</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/matrix.html">matrix</a></span><span class="op">(</span><span class="va">mat</span>, <span class="fl">28</span>, <span class="fl">28</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/importance-image_05260c3ca541817369d0578be7a25dfc">
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/importance-image_87afa3c7f3e0e4648444d0c68b1120f1">
 <div class="cell-output-display">
 <div class="quarto-figure quarto-figure-center">
 <figure class="figure"><p><img src="ml-in-practice_files/figure-html/importance-image-1.png" class="img-fluid figure-img" style="width:50.0%"></p>
@@ -798,10 +834,10 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 </div>
 </div>
 </div>
-</section><section id="diagnostics" class="level2" data-number="31.7"><h2 data-number="31.7" class="anchored" data-anchor-id="diagnostics">
-<span class="header-section-number">31.7</span> Diagnostics</h2>
+</section><section id="diagnostics" class="level2" data-number="31.8"><h2 data-number="31.8" class="anchored" data-anchor-id="diagnostics">
+<span class="header-section-number">31.8</span> Diagnostics</h2>
 <p>An important part of data analysis is visualizing results to determine why we are failing. How we do this depends on the application. Below we show the images of digits for which we made an incorrect prediction. Here are some errors for the random forest:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/rf-images,_c790a3692bde4d560c36bb262ca2859c">
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/rf-images,_f1b51c81f68864ae565b20b19ee930e7">
 <div class="cell-output-display">
 <div class="quarto-figure quarto-figure-center">
 <figure class="figure"><p><img src="ml-in-practice_files/figure-html/rf-images,-1.png" class="img-fluid figure-img" style="width:100.0%"></p>
@@ -810,39 +846,39 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 </div>
 </div>
 <p>By examining errors like this, we often find specific weaknesses to algorithms or parameter choices and can try to correct them.</p>
-</section><section id="ensembles" class="level2" data-number="31.8"><h2 data-number="31.8" class="anchored" data-anchor-id="ensembles">
-<span class="header-section-number">31.8</span> Ensembles</h2>
+</section><section id="ensembles" class="level2" data-number="31.9"><h2 data-number="31.9" class="anchored" data-anchor-id="ensembles">
+<span class="header-section-number">31.9</span> Ensembles</h2>
 <p>The idea of an ensemble is similar to the idea of combining data from different pollsters to obtain a better estimate of the true support for each candidate.</p>
 <p>In machine learning, one can usually greatly improve the final results by combining the results of different algorithms.</p>
 <p>Here is a simple example where we compute new class probabilities by taking the average of random forest and kNN. We can see that the accuracy improves:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-40_38c50a82eefb03eeee2280ece2f7bc19">
-<div class="sourceCode" id="cb43"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">p_rf</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">fit_rf</span>, <span class="va">x_test</span><span class="op">[</span>,<span class="va">col_index</span><span class="op">]</span>, type <span class="op">=</span> <span class="st">"prob"</span><span class="op">)</span>  </span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-42_86c88f5d79b96fbe04a166c56daedfbd">
+<div class="sourceCode" id="cb47"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">p_rf</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">fit_rf</span>, <span class="va">x_test</span><span class="op">[</span>,<span class="op">-</span><span class="va">nzv</span><span class="op">]</span>, type <span class="op">=</span> <span class="st">"prob"</span><span class="op">)</span>  </span>
 <span><span class="va">p_rf</span> <span class="op">&lt;-</span> <span class="va">p_rf</span> <span class="op">/</span> <span class="fu"><a href="https://rdrr.io/r/base/colSums.html">rowSums</a></span><span class="op">(</span><span class="va">p_rf</span><span class="op">)</span></span>
-<span><span class="va">p_knn_pca</span>  <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">fit_knn_pca</span>, <span class="va">newdata</span><span class="op">)</span></span>
+<span><span class="va">p_knn_pca</span>  <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/stats/predict.html">predict</a></span><span class="op">(</span><span class="va">train_knn_pca</span>, <span class="va">x_test</span>, type <span class="op">=</span> <span class="st">"prob"</span><span class="op">)</span></span>
 <span><span class="va">p</span> <span class="op">&lt;-</span> <span class="op">(</span><span class="va">p_rf</span> <span class="op">+</span> <span class="va">p_knn_pca</span><span class="op">)</span><span class="op">/</span><span class="fl">2</span></span>
 <span><span class="va">y_pred</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/factor.html">factor</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/apply.html">apply</a></span><span class="op">(</span><span class="va">p</span>, <span class="fl">1</span>, <span class="va">which.max</span><span class="op">)</span> <span class="op">-</span> <span class="fl">1</span><span class="op">)</span></span>
 <span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/confusionMatrix.html">confusionMatrix</a></span><span class="op">(</span><span class="va">y_pred</span>, <span class="va">y_test</span><span class="op">)</span><span class="op">$</span><span class="va">overall</span><span class="op">[</span><span class="st">"Accuracy"</span><span class="op">]</span></span>
 <span><span class="co">#&gt; Accuracy </span></span>
-<span><span class="co">#&gt;    0.967</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<span><span class="co">#&gt;    0.971</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>We have just built an ensemble with just two algorithms. By combing more similarly performing, but uncorrelated, algorithms we can improve accuracy further.</p>
-</section><section id="exercises" class="level2" data-number="31.9"><h2 data-number="31.9" class="anchored" data-anchor-id="exercises">
-<span class="header-section-number">31.9</span> Exercises</h2>
+</section><section id="exercises" class="level2" data-number="31.10"><h2 data-number="31.10" class="anchored" data-anchor-id="exercises">
+<span class="header-section-number">31.10</span> Exercises</h2>
 <p>1. In the exercises in <a href="algorithms.html"><span>Chapter&nbsp;30</span></a> we saw that changing <code>maxnodes</code> or <code>nodesize</code> in the <code>randomForest</code> function improved our estimate. Let’s use the <code>train</code> function to help us pick these values. From the <strong>caret</strong> manual we see that we can’t tune the <code>maxnodes</code> parameter or the <code>nodesize</code> argument with <code>randomForest</code>, so we will use the <strong>Rborist</strong> package and tune the <code>minNode</code> argument. Use the <code>train</code> function to try values <code>minNode &lt;- seq(5, 250, 25)</code>. See which value minimizes the estimated RMSE.</p>
 <p>2. This **dslabs* dataset includes a matrix <code>x</code>:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-41_962fcb10bedec3fb7d76f7dc6125e359">
-<div class="sourceCode" id="cb44"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va">dslabs</span><span class="op">)</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-44_b02fd6cc5dcb7aaeef70e20e84217c30">
+<div class="sourceCode" id="cb48"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va">dslabs</span><span class="op">)</span></span>
 <span><span class="fu"><a href="https://rdrr.io/r/base/dim.html">dim</a></span><span class="op">(</span><span class="va">tissue_gene_expression</span><span class="op">$</span><span class="va">x</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>with the gene expression measured on 500 genes for 189 biological samples representing seven different tissues. The tissue type is stored in <code>y</code>:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-42_1e1fa7362c89135b77a47007ca0c86ec">
-<div class="sourceCode" id="cb45"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/table.html">table</a></span><span class="op">(</span><span class="va">tissue_gene_expression</span><span class="op">$</span><span class="va">y</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-45_7134b09130449f5104732ed2fde6cf08">
+<div class="sourceCode" id="cb49"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/table.html">table</a></span><span class="op">(</span><span class="va">tissue_gene_expression</span><span class="op">$</span><span class="va">y</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>Split the data in training and test sets, then use kNN to predict tissue type and see what accuracy you obtain. Try it for <span class="math inline">\(k = 1, 3, \dots, 11\)</span>.</p>
 <p>3. We are going to apply LDA and QDA to the <code>tissue_gene_expression</code> dataset. We will start with simple examples based on this dataset and then develop a realistic example.</p>
 <p>Create a dataset with just the classes <code>cerebellum</code> and <code>hippocampus</code> (two parts of the brain) and a predictor matrix with 10 randomly selected columns. Estimate the accuracy of LDA.</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-43_73508bf9599eb65f639d16c4d0206635">
-<div class="sourceCode" id="cb46"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op">(</span><span class="fl">1993</span><span class="op">)</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-46_90c5e852fa27972cf6a44d878c0a8c0a">
+<div class="sourceCode" id="cb50"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op">(</span><span class="fl">1993</span><span class="op">)</span></span>
 <span><span class="va">tissues</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op">(</span><span class="st">"cerebellum"</span>, <span class="st">"hippocampus"</span><span class="op">)</span></span>
 <span><span class="va">ind</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/which.html">which</a></span><span class="op">(</span><span class="va">tissue_gene_expression</span><span class="op">$</span><span class="va">y</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">tissues</span><span class="op">)</span></span>
 <span><span class="va">y</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/droplevels.html">droplevels</a></span><span class="op">(</span><span class="va">tissue_gene_expression</span><span class="op">$</span><span class="va">y</span><span class="op">[</span><span class="va">ind</span><span class="op">]</span><span class="op">)</span></span>
@@ -855,8 +891,8 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 <p>7. One thing we see in the previous plot is that the value of predictors correlate in both groups: some predictors are low in both groups while others are high in both groups. The mean value of each predictor, <code>colMeans(x)</code>, is not informative or useful for prediction and often, for interpretation purposes, it is useful to center or scale each column. This can be achieved with the <code>preProcessing</code> argument in <code>train</code>. Re-run LDA with <code>preProcessing = "scale"</code>. Note that accuracy does not change but see how it is easier to identify the predictors that differ more between groups in the plot made in exercise 4.</p>
 <p>8. In the previous exercises, we saw that both approaches worked well. Plot the predictor values for the two genes with the largest differences between the two groups in a scatterplot to see how they appear to follow a bivariate distribution as assumed by the LDA and QDA approaches. Color the points by the outcome.</p>
 <p>9. Now we are going to increase the complexity of the challenge slightly: we will consider all the tissue types.</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-44_430da0f50908a8ba52b4a3573471f97b">
-<div class="sourceCode" id="cb47"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op">(</span><span class="fl">1993</span><span class="op">)</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-47_91ef8bebf85ee9140a4b0467bef8ec1f">
+<div class="sourceCode" id="cb51"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/Random.html">set.seed</a></span><span class="op">(</span><span class="fl">1993</span><span class="op">)</span></span>
 <span><span class="va">y</span> <span class="op">&lt;-</span> <span class="va">tissue_gene_expression</span><span class="op">$</span><span class="va">y</span></span>
 <span><span class="va">x</span> <span class="op">&lt;-</span> <span class="va">tissue_gene_expression</span><span class="op">$</span><span class="va">x</span></span>
 <span><span class="va">x</span> <span class="op">&lt;-</span> <span class="va">x</span><span class="op">[</span>, <span class="fu"><a href="https://rdrr.io/r/base/sample.html">sample</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/nrow.html">ncol</a></span><span class="op">(</span><span class="va">x</span><span class="op">)</span>, <span class="fl">10</span><span class="op">)</span><span class="op">]</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
@@ -872,8 +908,8 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 <p>17. We can see that with just six genes, we are able to predict the tissue type. Now let’s see if we can do even better with a random forest. Use the <code>train</code> function and the <code>rf</code> method to train a random forest. Try out values of <code>mtry</code> ranging from, at least, <code>seq(50, 200, 25)</code>. What <code>mtry</code> value maximizes accuracy? To permit small <code>nodesize</code> to grow as we did with the classification trees, use the following argument: <code>nodesize = 1</code>. This will take several seconds to run. If you want to test it out, try using smaller values with <code>ntree</code>. Set the seed to 1990.</p>
 <p>18. Use the function <code>varImp</code> on the output of <code>train</code> and save it to an object called <code>imp</code>.</p>
 <p>19. The <code>rpart</code> model we ran above produced a tree that used just six predictors. Extracting the predictor names is not straightforward, but can be done. If the output of the call to train was <code>fit_rpart</code>, we can extract the names like this:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-45_57bb3f0b3f411d5d8b70ce5c34c9c841">
-<div class="sourceCode" id="cb48"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">ind</span> <span class="op">&lt;-</span> <span class="op">!</span><span class="op">(</span><span class="va">fit_rpart</span><span class="op">$</span><span class="va">finalModel</span><span class="op">$</span><span class="va">frame</span><span class="op">$</span><span class="va">var</span> <span class="op">==</span> <span class="st">"&lt;leaf&gt;"</span><span class="op">)</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-48_87740bf2db4c21eba20df2d347200d27">
+<div class="sourceCode" id="cb52"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">ind</span> <span class="op">&lt;-</span> <span class="op">!</span><span class="op">(</span><span class="va">fit_rpart</span><span class="op">$</span><span class="va">finalModel</span><span class="op">$</span><span class="va">frame</span><span class="op">$</span><span class="va">var</span> <span class="op">==</span> <span class="st">"&lt;leaf&gt;"</span><span class="op">)</span></span>
 <span><span class="va">tree_terms</span> <span class="op">&lt;-</span> </span>
 <span>  <span class="va">fit_rpart</span><span class="op">$</span><span class="va">finalModel</span><span class="op">$</span><span class="va">frame</span><span class="op">$</span><span class="va">var</span><span class="op">[</span><span class="va">ind</span><span class="op">]</span> <span class="op">|&gt;</span></span>
 <span>  <span class="fu"><a href="https://rdrr.io/r/base/unique.html">unique</a></span><span class="op">(</span><span class="op">)</span> <span class="op">|&gt;</span></span>
@@ -883,8 +919,8 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 <p>What is the variable importance in the random forest call for these predictors? Where do they rank?</p>
 <p>20. Extract the top 50 predictors based on importance, take a subset of <code>x</code> with just these predictors and apply the function <code>heatmap</code> to see how these genes behave across the tissues. We will introduce the <code>heatmap</code> function in <a href="clustering.html"><span>Chapter&nbsp;32</span></a>.</p>
 <p>21. Previously, we compared the conditional probability <span class="math inline">\(p(\mathbf{x})\)</span> give two predictors <span class="math inline">\(\mathbf{x} = (x_1, x_2)^\top\)</span> to the fit <span class="math inline">\(\hat{p}(\mathbf{x})\)</span> obtained with a machine learning algorithm by making image plots. The following code can be used to make these images and include a curve at the values of <span class="math inline">\(x_1\)</span> and <span class="math inline">\(x_2\)</span> for which the function is <span class="math inline">\(0.5\)</span>:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-46_f942af32d2aa82377017395f332a08a5">
-<div class="sourceCode" id="cb49"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">plot_cond_prob</span> <span class="op">&lt;-</span> <span class="kw">function</span><span class="op">(</span><span class="va">x_1</span>, <span class="va">x_2</span>, <span class="va">p</span><span class="op">)</span><span class="op">{</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-49_cca09f850f651ffc581aa3b340e12b61">
+<div class="sourceCode" id="cb53"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">plot_cond_prob</span> <span class="op">&lt;-</span> <span class="kw">function</span><span class="op">(</span><span class="va">x_1</span>, <span class="va">x_2</span>, <span class="va">p</span><span class="op">)</span><span class="op">{</span></span>
 <span>  <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span>x_1 <span class="op">=</span> <span class="va">x_1</span>, x_2 <span class="op">=</span> <span class="va">x_2</span>, p <span class="op">=</span> <span class="va">p</span><span class="op">)</span> <span class="op">|&gt;</span></span>
 <span>    <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggplot.html">ggplot</a></span><span class="op">(</span><span class="fu"><a href="https://ggplot2.tidyverse.org/reference/aes.html">aes</a></span><span class="op">(</span><span class="va">x_1</span>, <span class="va">x_2</span><span class="op">)</span><span class="op">)</span> <span class="op">+</span></span>
 <span>    <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/geom_tile.html">geom_raster</a></span><span class="op">(</span><span class="fu"><a href="https://ggplot2.tidyverse.org/reference/aes.html">aes</a></span><span class="op">(</span>fill <span class="op">=</span> <span class="va">p</span><span class="op">)</span>, show.legend <span class="op">=</span> <span class="cn">FALSE</span><span class="op">)</span> <span class="op">+</span></span>
@@ -893,13 +929,13 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 <span><span class="op">}</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>We can see the true conditional probability for the 2 or 7 example like this:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-47_2f8860122f1634610295fa74463894db">
-<div class="sourceCode" id="cb50"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/with.html">with</a></span><span class="op">(</span><span class="va">mnist_27</span><span class="op">$</span><span class="va">true_p</span>, <span class="fu">plot_cond_prob</span><span class="op">(</span><span class="va">x_1</span>, <span class="va">x_2</span>, <span class="va">p</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-50_ba2d66e9ce1df278dbb7223194df39d6">
+<div class="sourceCode" id="cb54"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/r/base/with.html">with</a></span><span class="op">(</span><span class="va">mnist_27</span><span class="op">$</span><span class="va">true_p</span>, <span class="fu">plot_cond_prob</span><span class="op">(</span><span class="va">x_1</span>, <span class="va">x_2</span>, <span class="va">p</span><span class="op">)</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>Fit a kNN model and make this plot for the estimated conditional probability. Hint: Use the argument <code>newdata = mnist27$train</code> to obtain predictions for a grid points.</p>
 <p>22. Notice that, in the plot made in exercise 1, the boundary is somewhat wiggly. This is because kNN, like the basic bin smoother, does not use a kernel. To improve this we could try loess. By reading through the available models part of the <strong>caret</strong> manual, we see that we can use the <code>gamLoess</code> method. We need to install the <strong>gam</strong> package, if we have not done so already. We see that we have two parameters to optimize:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-48_39267f545710af73f3ee829176fda8fd">
-<div class="sourceCode" id="cb51"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/modelLookup.html">modelLookup</a></span><span class="op">(</span><span class="st">"gamLoess"</span><span class="op">)</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-51_0fffadc993582765971f8ed466262dda">
+<div class="sourceCode" id="cb55"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/caret/man/modelLookup.html">modelLookup</a></span><span class="op">(</span><span class="st">"gamLoess"</span><span class="op">)</span></span>
 <span><span class="co">#&gt;      model parameter  label forReg forClass probModel</span></span>
 <span><span class="co">#&gt; 1 gamLoess      span   Span   TRUE     TRUE      TRUE</span></span>
 <span><span class="co">#&gt; 2 gamLoess    degree Degree   TRUE     TRUE      TRUE</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
@@ -907,8 +943,8 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 <p>Use cross-validation to pick a span between 0.15 and 0.75. Keep <code>degree = 1</code>. What span does cross-validation select?</p>
 <p>23. Show an image plot of the estimate <span class="math inline">\(\hat{p}(x,y)\)</span> resulting from the model fit in the exercise 2. How does the accuracy compare to that of kNN? Comment on the difference between the estimate obtained with kNN.</p>
 <p>24. Use the <code>mnist_27</code> training set to build a model with several of the models available from the <strong>caret</strong> package. For example, you can try these:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-49_f7a0a41341c10f41394b6d8f3be3036a">
-<div class="sourceCode" id="cb52"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">models</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op">(</span><span class="st">"glm"</span>, <span class="st">"lda"</span>,  <span class="st">"naive_bayes"</span>,  <span class="st">"svmLinear"</span>, <span class="st">"gamboost"</span>,  </span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-52_f7f91cb388ba9f7896616b36ff6bab6a">
+<div class="sourceCode" id="cb56"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">models</span> <span class="op">&lt;-</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op">(</span><span class="st">"glm"</span>, <span class="st">"lda"</span>,  <span class="st">"naive_bayes"</span>,  <span class="st">"svmLinear"</span>, <span class="st">"gamboost"</span>,  </span>
 <span>            <span class="st">"gamLoess"</span>, <span class="st">"qda"</span>, <span class="st">"knn"</span>, <span class="st">"kknn"</span>, <span class="st">"loclda"</span>, <span class="st">"gam"</span>, <span class="st">"rf"</span>, </span>
 <span>            <span class="st">"ranger"</span>,<span class="st">"wsrf"</span>, <span class="st">"Rborist"</span>, <span class="st">"avNNet"</span>, <span class="st">"mlp"</span>, <span class="st">"monmlp"</span>, <span class="st">"gbm"</span>, </span>
 <span>            <span class="st">"adaboost"</span>, <span class="st">"svmRadial"</span>, <span class="st">"svmRadialCost"</span>, <span class="st">"svmRadialSigma"</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
@@ -922,8 +958,8 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 <p>30. Now let’s only consider the methods with an estimated accuracy of 0.8 when constructing the ensemble. What is the accuracy now?</p>
 <p>31. Note that if two machine algorithms methods predict the same outcome, ensembling them will not change the prediction. For each pair of algorithms compare the percent of observations for which they make the same prediction. Use this to define a function and then use the <code>heatmap</code> function to visualize the results. Hint: use the <code>method = "binary"</code> argument in the <code>dist</code> function.</p>
 <p>32. Note that each method can also produce an estimated conditional probability. Instead of majority vote, we can take the average of these estimated conditional probabilities. For most methods, we can the use the <code>type = "prob"</code> in the train function. Note that some of the methods require you to use the argument <code>trControl=trainControl(classProbs=TRUE)</code> when calling train. Also, these methods do not work if classes have numbers as names. Hint: change the levels like this:</p>
-<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-50_c9897365354e61e3b686a376cd4180cb">
-<div class="sourceCode" id="cb53"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">dat</span><span class="op">$</span><span class="va">train</span><span class="op">$</span><span class="va">y</span> <span class="op">&lt;-</span> <span class="fu">recode_factor</span><span class="op">(</span><span class="va">dat</span><span class="op">$</span><span class="va">train</span><span class="op">$</span><span class="va">y</span>, <span class="st">"2"</span><span class="op">=</span><span class="st">"two"</span>, <span class="st">"7"</span><span class="op">=</span><span class="st">"seven"</span><span class="op">)</span></span>
+<div class="cell" data-layout-align="center" data-hash="ml-in-practice_cache/html/unnamed-chunk-53_4e668a58b76fdbb5550103985ab00cca">
+<div class="sourceCode" id="cb57"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="va">dat</span><span class="op">$</span><span class="va">train</span><span class="op">$</span><span class="va">y</span> <span class="op">&lt;-</span> <span class="fu">recode_factor</span><span class="op">(</span><span class="va">dat</span><span class="op">$</span><span class="va">train</span><span class="op">$</span><span class="va">y</span>, <span class="st">"2"</span><span class="op">=</span><span class="st">"two"</span>, <span class="st">"7"</span><span class="op">=</span><span class="st">"seven"</span><span class="op">)</span></span>
 <span><span class="va">dat</span><span class="op">$</span><span class="va">test</span><span class="op">$</span><span class="va">y</span> <span class="op">&lt;-</span> <span class="fu">recode_factor</span><span class="op">(</span><span class="va">dat</span><span class="op">$</span><span class="va">test</span><span class="op">$</span><span class="va">y</span>, <span class="st">"2"</span><span class="op">=</span><span class="st">"two"</span>, <span class="st">"7"</span><span class="op">=</span><span class="st">"seven"</span><span class="op">)</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>33. In this chapter, we illustrated a couple of machine learning algorithms on a subset of the MNIST dataset. Try fitting a model to the entire dataset.</p>
@@ -934,6 +970,7 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 <li id="fn1"><p>https://topepo.github.io/caret/available-models.html<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
 <li id="fn2"><p>http://topepo.github.io/caret/available-models.html<a href="#fnref2" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
 <li id="fn3"><p>https://topepo.github.io/caret/available-models.html<a href="#fnref3" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
+<li id="fn4"><p>https://topepo.github.io/caret/using-your-own-model-in-train.html<a href="#fnref4" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
 </ol></section></main><!-- /main --><script id="quarto-html-after-body" type="application/javascript">
 window.document.addEventListener("DOMContentLoaded", function (event) {
   const toggleBodyColorMode = (bsSheetEl) => {
@@ -1179,7 +1216,7 @@ <h1 class="title"><span id="sec-ml-in-practice" class="quarto-section-identifier
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/ml/ml-in-practice_files/figure-html/cv-10-fold-accuracy-estimate-1.png b/docs/ml/ml-in-practice_files/figure-html/cv-10-fold-accuracy-estimate-1.png
deleted file mode 100644
index aebb49e..0000000
Binary files a/docs/ml/ml-in-practice_files/figure-html/cv-10-fold-accuracy-estimate-1.png and /dev/null differ
diff --git a/docs/ml/ml-in-practice_files/figure-html/importance-image-1.png b/docs/ml/ml-in-practice_files/figure-html/importance-image-1.png
index 93e171f..ed03fab 100644
Binary files a/docs/ml/ml-in-practice_files/figure-html/importance-image-1.png and b/docs/ml/ml-in-practice_files/figure-html/importance-image-1.png differ
diff --git a/docs/ml/ml-in-practice_files/figure-html/rf-images,-1.png b/docs/ml/ml-in-practice_files/figure-html/rf-images,-1.png
index 282e66e..6684450 100644
Binary files a/docs/ml/ml-in-practice_files/figure-html/rf-images,-1.png and b/docs/ml/ml-in-practice_files/figure-html/rf-images,-1.png differ
diff --git a/docs/ml/ml-in-practice_files/figure-html/train-knn-plot-1.png b/docs/ml/ml-in-practice_files/figure-html/train-knn-plot-1.png
index e980b2d..bc1b69f 100644
Binary files a/docs/ml/ml-in-practice_files/figure-html/train-knn-plot-1.png and b/docs/ml/ml-in-practice_files/figure-html/train-knn-plot-1.png differ
diff --git a/docs/ml/notation-and-terminology.html b/docs/ml/notation-and-terminology.html
index a8a9271..d61b171 100644
--- a/docs/ml/notation-and-terminology.html
+++ b/docs/ml/notation-and-terminology.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 25&nbsp; Notation and terminology</title>
+<title>Introduction to Data Science - 25&nbsp; Notation and terminology</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -75,7 +75,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -728,7 +728,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/ml/resampling-methods.html b/docs/ml/resampling-methods.html
index 2c16945..6abd2d8 100644
--- a/docs/ml/resampling-methods.html
+++ b/docs/ml/resampling-methods.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 29&nbsp; Resampling methods</title>
+<title>Introduction to Data Science - 29&nbsp; Resampling methods</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -965,7 +965,7 @@ <h1 class="title"><span id="sec-cross-validation" class="quarto-section-identifi
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/ml/smoothing.html b/docs/ml/smoothing.html
index 4b85541..e4ff45c 100644
--- a/docs/ml/smoothing.html
+++ b/docs/ml/smoothing.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 28&nbsp; Smoothing</title>
+<title>Introduction to Data Science - 28&nbsp; Smoothing</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -1032,7 +1032,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/prob/continuous-probability.html b/docs/prob/continuous-probability.html
index b34e777..444c4a8 100644
--- a/docs/prob/continuous-probability.html
+++ b/docs/prob/continuous-probability.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 4&nbsp; Continuous probability</title>
+<title>Introduction to Data Science - 4&nbsp; Continuous probability</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -883,7 +883,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/prob/discrete-probability.html b/docs/prob/discrete-probability.html
index b502bd0..c981ce2 100644
--- a/docs/prob/discrete-probability.html
+++ b/docs/prob/discrete-probability.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 3&nbsp; Discrete probability</title>
+<title>Introduction to Data Science - 3&nbsp; Discrete probability</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -773,9 +773,9 @@ <h1 class="title">
 <span><span class="co">#&gt; [1] "Queen Clubs"  "Seven Spades"</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>and then check if one card is an Ace and the other is a face card or a 10. Going forward, we include 10 when we refer to a <em>face card</em>. Now, we need to check both possibilities:</p>
-<div class="cell" data-layout-align="center" data-hash="discrete-probability_cache/html/unnamed-chunk-30_40c06d3f9efbfdd2ada4cf4de2077bee">
-<div class="sourceCode" id="cb29"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="op">(</span><span class="va">hands</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">aces</span> <span class="op">&amp;</span> <span class="va">hands</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">facecard</span><span class="op">)</span> <span class="op">|</span> </span>
-<span>  <span class="op">(</span><span class="va">hands</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">aces</span> <span class="op">&amp;</span> <span class="va">hands</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">facecard</span><span class="op">)</span></span>
+<div class="cell" data-layout-align="center" data-hash="discrete-probability_cache/html/unnamed-chunk-30_d73ad9f6742cfe55d76372633959e522">
+<div class="sourceCode" id="cb29"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="op">(</span><span class="va">hand</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">aces</span> <span class="op">&amp;</span> <span class="va">hand</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">facecard</span><span class="op">)</span> <span class="op">|</span> </span>
+<span>  <span class="op">(</span><span class="va">hand</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">aces</span> <span class="op">&amp;</span> <span class="va">hand</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">facecard</span><span class="op">)</span></span>
 <span><span class="co">#&gt; [1] FALSE</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>If we repeat this 10,000 times, we get a very good approximation of the probability of a Natural 21.</p>
@@ -1227,7 +1227,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/prob/intro-to-prob.html b/docs/prob/intro-to-prob.html
index 2c5aca4..0ab5b49 100644
--- a/docs/prob/intro-to-prob.html
+++ b/docs/prob/intro-to-prob.html
@@ -8,7 +8,7 @@
 
 <meta name="author" content="Rafael A. Irizarry">
 
-<title>Advanced Data Science - Probability</title>
+<title>Introduction to Data Science - Probability</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -89,7 +89,7 @@
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto">
     <div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -663,7 +663,7 @@ <h1 class="title">Probability</h1>
 </div> <!-- /content -->
 <footer class="footer">
   <div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     <div class="toc-actions"><div><i class="bi bi-github"></i></div><div class="action-links"><p><a href="https://github.com/rafalab/dsbook-part-2/blob/main/prob/intro-to-prob.qmd" class="toc-action">View source</a></p><p><a href="https://github.com/rafalab/dsbook-part-2/issues/new" class="toc-action">Report an issue</a></p></div></div></div>
diff --git a/docs/prob/random-variables-sampling-models-clt.html b/docs/prob/random-variables-sampling-models-clt.html
index 42b34cc..95a9190 100644
--- a/docs/prob/random-variables-sampling-models-clt.html
+++ b/docs/prob/random-variables-sampling-models-clt.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 5&nbsp; Random variables</title>
+<title>Introduction to Data Science - 5&nbsp; Random variables</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -1054,7 +1054,7 @@ <h1 class="title">
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/search.json b/docs/search.json
index 85d63c5..18379a1 100644
--- a/docs/search.json
+++ b/docs/search.json
@@ -2,14 +2,14 @@
   {
     "objectID": "index.html",
     "href": "index.html",
-    "title": "Advanced Data Science",
+    "title": "Introduction to Data Science",
     "section": "",
-    "text": "Preface\nThis is the website for the Advanced Data Science.\nThe website for Introduction to Data Science is here.\nThis book started out as part of the class notes used in the HarvardX Data Science Series1.\nA hardcopy version of the first edition of the book, which combined both Introduction and Advanced parts, is available from CRC Press2.\nA free PDF of the October 24, 2019 version of the book, which combined both Introduction and Advanced parts, is available from Leanpub3.\nThe Quarto code used to generate the book is available on GitHub4. Note that, the graphical theme used for plots throughout the book can be recreated using the ds_theme_set() function from dslabs package.\nThis work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International CC BY-NC-SA 4.0.\nWe make announcements related to the book on Twitter. For updates follow @rafalab.\nA special thanks to my tidyverse guru David Robinson and Amy Gill for dozens of comments, edits, and suggestions. Also, many thanks to Stephanie Hicks who twice served as a co-instructor in my data science classes and Yihui Xie who patiently put up with my many questions about bookdown. Thanks also to Héctor Corrada-Bravo, for advice on how to best teach machine learning. Thanks to Alyssa Frazee for helping create the homework problem that became the Recommendation Systems case study. Also, many thanks to Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund for making the Quarto code for their R for Data Science book open. Finally, thanks to Alex Nones for proofreading the manuscript during its various stages.\nThis book was conceived during the teaching of several applied statistics courses, starting over fifteen years ago. The teaching assistants working with me throughout the years made important indirect contributions to this book. The latest iteration of this course is a HarvardX series coordinated by Heather Sternshein and Zofia Gajdos. We thank them for their contributions. We are also grateful to all the students whose questions and comments helped us improve the book. The courses were partially funded by NIH grant R25GM114818. We are very grateful to the National Institutes of Health for its support.\nA special thanks goes to all those who edited the book via GitHub pull requests or made suggestions by creating an issue or sending an email: nickyfoto (Huang Qiang), desautm (Marc-André Désautels), michaschwab (Michail Schwab), alvarolarreategui (Alvaro Larreategui), jakevc (Jake VanCampen), omerta (Guillermo Lengemann), espinielli (Enrico Spinielli), asimumba(Aaron Simumba), braunschweig (Maldewar), gwierzchowski (Grzegorz Wierzchowski), technocrat (Richard Careaga), atzakas, defeit (David Emerson Feit), shiraamitchell (Shira Mitchell), Nathalie-S, andreashandel (Andreas Handel), berkowitze (Elias Berkowitz), Dean-Webb (Dean Webber), mohayusuf, jimrothstein, mPloenzke (Matthew Ploenzke), NicholasDowand (Nicholas Dow), kant (Darío Hereñú), debbieyuster (Debbie Yuster), tuanchauict (Tuan Chau), phzeller, BTJ01 (BradJ), glsnow (Greg Snow), mberlanda (Mauro Berlanda), wfan9, larswestvang (Lars Westvang), jj999 (Jan Andrejkovic), Kriegslustig (Luca Nils Schmid), odahhani, aidanhorn (Aidan Horn), atraxler (Adrienne Traxler), alvegorova,wycheong (Won Young Cheong), med-hat (Medhat Khalil), kengustafson, Yowza63, ryan-heslin (Ryan Heslin), raffaem, tim8west, David D. Kane, El Mustapha El Abbassi, Vadim Zipunnikov, Anna Quaglieri, Chris Dong, Rick Schoenberg, Isabella Grabski, and Doug Snyder."
+    "text": "Preface\nThis is the website for the Statistics and Prediction Algorithms Through Case Studies part of Introduction to Data Science.\nThe website for the Data Wrangling and Visualization with R is here.\nThis book started out as part of the class notes used in the HarvardX Data Science Series1.\nA hardcopy version of the first edition of the book, which combined both parts, is available from CRC Press2.\nA free PDF of the October 24, 2019 version of the book, which combined both parts, is available from Leanpub3.\nThe Quarto code used to generate the book is available on GitHub4. Note that, the graphical theme used for plots throughout the book can be recreated using the ds_theme_set() function from dslabs package.\nThis work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International CC BY-NC-SA 4.0.\nWe make announcements related to the book on Twitter. For updates follow @rafalab.\nA special thanks to my tidyverse guru David Robinson and Amy Gill for dozens of comments, edits, and suggestions. Also, many thanks to Stephanie Hicks who twice served as a co-instructor in my data science classes and Yihui Xie who patiently put up with my many questions about bookdown. Thanks also to Héctor Corrada-Bravo, for advice on how to best teach machine learning. Thanks to Alyssa Frazee for helping create the homework problem that became the Recommendation Systems case study. Also, many thanks to Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund for making the Quarto code for their R for Data Science book open. Finally, thanks to Alex Nones for proofreading the manuscript during its various stages.\nThis book was conceived during the teaching of several applied statistics courses, starting over fifteen years ago. The teaching assistants working with me throughout the years made important indirect contributions to this book. The latest iteration of this course is a HarvardX series coordinated by Heather Sternshein and Zofia Gajdos. We thank them for their contributions. We are also grateful to all the students whose questions and comments helped us improve the book. The courses were partially funded by NIH grant R25GM114818. We are very grateful to the National Institutes of Health for its support.\nA special thanks goes to all those who edited the book via GitHub pull requests or made suggestions by creating an issue or sending an email: nickyfoto (Huang Qiang), desautm (Marc-André Désautels), michaschwab (Michail Schwab), alvarolarreategui (Alvaro Larreategui), jakevc (Jake VanCampen), omerta (Guillermo Lengemann), espinielli (Enrico Spinielli), asimumba(Aaron Simumba), braunschweig (Maldewar), gwierzchowski (Grzegorz Wierzchowski), technocrat (Richard Careaga), atzakas, defeit (David Emerson Feit), shiraamitchell (Shira Mitchell), Nathalie-S, andreashandel (Andreas Handel), berkowitze (Elias Berkowitz), Dean-Webb (Dean Webber), mohayusuf, jimrothstein, mPloenzke (Matthew Ploenzke), NicholasDowand (Nicholas Dow), kant (Darío Hereñú), debbieyuster (Debbie Yuster), tuanchauict (Tuan Chau), phzeller, BTJ01 (BradJ), glsnow (Greg Snow), mberlanda (Mauro Berlanda), wfan9, larswestvang (Lars Westvang), jj999 (Jan Andrejkovic), Kriegslustig (Luca Nils Schmid), odahhani, aidanhorn (Aidan Horn), atraxler (Adrienne Traxler), alvegorova,wycheong (Won Young Cheong), med-hat (Medhat Khalil), biscotty666 (Brian Carey), kengustafson, Yowza63, ryan-heslin (Ryan Heslin), raffaem, tim8west, David D. Kane, El Mustapha El Abbassi, Vadim Zipunnikov, Anna Quaglieri, Chris Dong, Rick Schoenberg, Isabella Grabski, and Doug Snyder."
   },
   {
     "objectID": "index.html#footnotes",
     "href": "index.html#footnotes",
-    "title": "Advanced Data Science",
+    "title": "Introduction to Data Science",
     "section": "",
     "text": "https://www.edx.org/professional-certificate/harvardx-data-science↩︎\nhttps://www.routledge.com/Introduction-to-Data-Science-Data-Analysis-and-Prediction-Algorithms-with/Irizarry/p/book/9780367357986?utm_source=author&utm_medium=shared_link&utm_campaign=B043135_jm1_5ll_6rm_t081_1al_introductiontodatascienceauthorshare↩︎\nhttps://leanpub.com/datasciencebook↩︎\nhttps://github.com/rafalab/dsbook-part-2↩︎"
   },
@@ -221,7 +221,7 @@
     "href": "prob/discrete-probability.html#combinations-and-permutations",
     "title": "\n3  Discrete probability\n",
     "section": "\n3.8 Combinations and permutations",
-    "text": "3.8 Combinations and permutations\nIn our very first example, we imagined an urn with five beads. As a reminder, to compute the probability distribution of a single draw, we simply listed out all the possibilities, which amounted to 5. Subsequently, for each event, we counted how many of these possibilities were associated with the event. The resulting probability of choosing a blue bead is 3/5, as three of the five possible outcomes were blue.\nFor more complicated cases, the computations are not as straightforward. For instance, what is the probability that, if I draw five cards without replacement, I get all cards of the same suit, which is known as a “flush” in poker? In a discrete probability course, you learn theory on how to make these computations. Here, we focus on how to use R code to compute the answers.\nFirst, let’s construct a deck of cards. For this, we will use the expand.grid and paste functions. We use paste to create strings by joining smaller strings. To do this, we take the number and suit of a card, and create the card name like this:\n\nnumber &lt;- \"Three\"\nsuit &lt;- \"Hearts\"\npaste(number, suit)\n#&gt; [1] \"Three Hearts\"\n\npaste also works on pairs of vectors performing the operation element-wise:\n\npaste(letters[1:5], as.character(1:5))\n#&gt; [1] \"a 1\" \"b 2\" \"c 3\" \"d 4\" \"e 5\"\n\nThe function expand.grid gives us all the combinations of entries of two vectors. For example, if you have blue and black pants and white, grey, and plaid shirts, all your combinations are:\n\nexpand.grid(pants = c(\"blue\", \"black\"), shirt = c(\"white\", \"grey\", \"plaid\"))\n#&gt;   pants shirt\n#&gt; 1  blue white\n#&gt; 2 black white\n#&gt; 3  blue  grey\n#&gt; 4 black  grey\n#&gt; 5  blue plaid\n#&gt; 6 black plaid\n\nHere is how we generate a deck of cards:\n\nsuits &lt;- c(\"Diamonds\", \"Clubs\", \"Hearts\", \"Spades\")\nnumbers &lt;- c(\"Ace\", \"Deuce\", \"Three\", \"Four\", \"Five\", \"Six\", \"Seven\", \n             \"Eight\", \"Nine\", \"Ten\", \"Jack\", \"Queen\", \"King\")\ndeck &lt;- expand.grid(number = numbers, suit = suits)\ndeck &lt;- paste(deck$number, deck$suit)\n\nWith the deck constructed, we can double-check that the probability of a King as the first card is 1/13 by computing the proportion of possible outcomes that satisfy our condition:\n\nkings &lt;- paste(\"King\", suits)\nmean(deck %in% kings)\n#&gt; [1] 0.0769\n\nNow, what about the conditional probability of the second card being a King, given that the first card was a King? Earlier, we deduced that if one King is already drawn from the deck, leaving 51 cards, then this probability is 3/51. Let’s confirm by listing out all possible outcomes.\nTo do this, we can use the permutations function from the gtools package. For any list of size n, this function computes all the different combinations we can obtain when we select r items. Here are all the ways we can choose two numbers from a list consisting of 1,2,3:\n\nlibrary(gtools)\npermutations(3, 2)\n#&gt;      [,1] [,2]\n#&gt; [1,]    1    2\n#&gt; [2,]    1    3\n#&gt; [3,]    2    1\n#&gt; [4,]    2    3\n#&gt; [5,]    3    1\n#&gt; [6,]    3    2\n\nObserve that the order matters here: 3,1 is different than 1,3. Also, note that (1,1), (2,2), and (3,3) do not appear because once we pick a number, it can’t appear again.\nOptionally, we can add a vector. If you want to see five random seven digit phone numbers out of all possible phone numbers (without repeats), you can type:\n\nall_phone_numbers &lt;- permutations(10, 7, v = 0:9)\nn &lt;- nrow(all_phone_numbers)\nindex &lt;- sample(n, 5)\nall_phone_numbers[index,]\n#&gt;      [,1] [,2] [,3] [,4] [,5] [,6] [,7]\n#&gt; [1,]    1    3    8    0    6    7    5\n#&gt; [2,]    2    9    1    6    4    8    0\n#&gt; [3,]    5    1    6    0    9    8    2\n#&gt; [4,]    7    4    6    0    2    8    1\n#&gt; [5,]    4    6    5    9    2    8    0\n\nInstead of using the numbers 1 through 10, the default, it uses what we provided through v: the digits 0 through 9.\nTo compute all possible ways we can choose two cards when the order matters, we type:\n\nhands &lt;- permutations(52, 2, v = deck)\n\nThis is a matrix with two columns and 2652 rows. With a matrix, we can obtain the first and second cards like this:\n\nfirst_card &lt;- hands[,1]\nsecond_card &lt;- hands[,2]\n\nNow, the cases for which the first hand was a King can be computed as follows:\n\nkings &lt;- paste(\"King\", suits)\nsum(first_card %in% kings)\n#&gt; [1] 204\n\nTo get the conditional probability, we compute what fraction of these have a King as the second card:\n\nsum(first_card %in% kings & second_card %in% kings) / \n  sum(first_card %in% kings)\n#&gt; [1] 0.0588\n\nwhich is exactly 3/51, as we had already deduced. Notice that the code above is equivalent to:\n\nmean(first_card %in% kings & second_card %in% kings) / \n  mean(first_card %in% kings)\n#&gt; [1] 0.0588\n\nwhich uses mean instead of sum and is an R version of:\n\\[\n\\frac{\\mbox{Pr}(A \\mbox{ and } B)}{ \\mbox{Pr}(A)}\n\\]\nWhat about if the order does not matter? For example, in Blackjack, if you obtain an Ace and a face card in the first draw, it is called a Natural 21, and you win automatically. If we wanted to compute the probability of this happening, we would enumerate the combinations, not the permutations, since the order does not matter.\n\ncombinations(3,2)\n#&gt;      [,1] [,2]\n#&gt; [1,]    1    2\n#&gt; [2,]    1    3\n#&gt; [3,]    2    3\n\nIn the second line, the outcome does not include (2,1) because (1,2) already was enumerated. The same applies to (3,1) and (3,2).\nSo to compute the probability of a Natural 21 in Blackjack, we can do this:\n\naces &lt;- paste(\"Ace\", suits)\n\nfacecard &lt;- c(\"King\", \"Queen\", \"Jack\", \"Ten\")\nfacecard &lt;- expand.grid(number = facecard, suit = suits)\nfacecard &lt;- paste(facecard$number, facecard$suit)\n\nhands &lt;- combinations(52, 2, v = deck)\nmean(hands[,1] %in% aces & hands[,2] %in% facecard)\n#&gt; [1] 0.0483\n\nIn the last line, we assume the Ace comes first. This assumption is made based on our knowledge of how combination enumerates possibilities, and it will list this case first. However, to be safe, we could have written this and produced the same answer:\n\nmean((hands[,1] %in% aces & hands[,2] %in% facecard) |\n       (hands[,2] %in% aces & hands[,1] %in% facecard))\n#&gt; [1] 0.0483\n\n\n3.8.1 Monte Carlo example\nInstead of using combinations to deduce the exact probability of a Natural 21, we can use a Monte Carlo to estimate this probability. In this case, we draw two cards over and over and keep track of how many 21s we obtain. We can use the function sample to draw two cards without replacement:\n\nhand &lt;- sample(deck, 2)\nhand\n#&gt; [1] \"Queen Clubs\"  \"Seven Spades\"\n\nand then check if one card is an Ace and the other is a face card or a 10. Going forward, we include 10 when we refer to a face card. Now, we need to check both possibilities:\n\n(hands[1] %in% aces & hands[2] %in% facecard) | \n  (hands[2] %in% aces & hands[1] %in% facecard)\n#&gt; [1] FALSE\n\nIf we repeat this 10,000 times, we get a very good approximation of the probability of a Natural 21.\nLet’s start by writing a function that draws a hand and returns TRUE if we get a 21. The function does not need any arguments because it uses objects defined in the global environment.\n\nblackjack &lt;- function(){\n   hand &lt;- sample(deck, 2)\n  (hand[1] %in% aces & hand[2] %in% facecard) | \n    (hand[2] %in% aces & hand[1] %in% facecard)\n}\n\nHere. we do have to check both possibilities: Ace first or Ace second, because we are not using the combinations function. The function returns TRUE if we get a 21 and FALSE otherwise:\n\nblackjack()\n#&gt; [1] FALSE\n\nNow we can play this game, say, 10,000 times:\n\nB &lt;- 10000\nresults &lt;- replicate(B, blackjack())\nmean(results)\n#&gt; [1] 0.0475"
+    "text": "3.8 Combinations and permutations\nIn our very first example, we imagined an urn with five beads. As a reminder, to compute the probability distribution of a single draw, we simply listed out all the possibilities, which amounted to 5. Subsequently, for each event, we counted how many of these possibilities were associated with the event. The resulting probability of choosing a blue bead is 3/5, as three of the five possible outcomes were blue.\nFor more complicated cases, the computations are not as straightforward. For instance, what is the probability that, if I draw five cards without replacement, I get all cards of the same suit, which is known as a “flush” in poker? In a discrete probability course, you learn theory on how to make these computations. Here, we focus on how to use R code to compute the answers.\nFirst, let’s construct a deck of cards. For this, we will use the expand.grid and paste functions. We use paste to create strings by joining smaller strings. To do this, we take the number and suit of a card, and create the card name like this:\n\nnumber &lt;- \"Three\"\nsuit &lt;- \"Hearts\"\npaste(number, suit)\n#&gt; [1] \"Three Hearts\"\n\npaste also works on pairs of vectors performing the operation element-wise:\n\npaste(letters[1:5], as.character(1:5))\n#&gt; [1] \"a 1\" \"b 2\" \"c 3\" \"d 4\" \"e 5\"\n\nThe function expand.grid gives us all the combinations of entries of two vectors. For example, if you have blue and black pants and white, grey, and plaid shirts, all your combinations are:\n\nexpand.grid(pants = c(\"blue\", \"black\"), shirt = c(\"white\", \"grey\", \"plaid\"))\n#&gt;   pants shirt\n#&gt; 1  blue white\n#&gt; 2 black white\n#&gt; 3  blue  grey\n#&gt; 4 black  grey\n#&gt; 5  blue plaid\n#&gt; 6 black plaid\n\nHere is how we generate a deck of cards:\n\nsuits &lt;- c(\"Diamonds\", \"Clubs\", \"Hearts\", \"Spades\")\nnumbers &lt;- c(\"Ace\", \"Deuce\", \"Three\", \"Four\", \"Five\", \"Six\", \"Seven\", \n             \"Eight\", \"Nine\", \"Ten\", \"Jack\", \"Queen\", \"King\")\ndeck &lt;- expand.grid(number = numbers, suit = suits)\ndeck &lt;- paste(deck$number, deck$suit)\n\nWith the deck constructed, we can double-check that the probability of a King as the first card is 1/13 by computing the proportion of possible outcomes that satisfy our condition:\n\nkings &lt;- paste(\"King\", suits)\nmean(deck %in% kings)\n#&gt; [1] 0.0769\n\nNow, what about the conditional probability of the second card being a King, given that the first card was a King? Earlier, we deduced that if one King is already drawn from the deck, leaving 51 cards, then this probability is 3/51. Let’s confirm by listing out all possible outcomes.\nTo do this, we can use the permutations function from the gtools package. For any list of size n, this function computes all the different combinations we can obtain when we select r items. Here are all the ways we can choose two numbers from a list consisting of 1,2,3:\n\nlibrary(gtools)\npermutations(3, 2)\n#&gt;      [,1] [,2]\n#&gt; [1,]    1    2\n#&gt; [2,]    1    3\n#&gt; [3,]    2    1\n#&gt; [4,]    2    3\n#&gt; [5,]    3    1\n#&gt; [6,]    3    2\n\nObserve that the order matters here: 3,1 is different than 1,3. Also, note that (1,1), (2,2), and (3,3) do not appear because once we pick a number, it can’t appear again.\nOptionally, we can add a vector. If you want to see five random seven digit phone numbers out of all possible phone numbers (without repeats), you can type:\n\nall_phone_numbers &lt;- permutations(10, 7, v = 0:9)\nn &lt;- nrow(all_phone_numbers)\nindex &lt;- sample(n, 5)\nall_phone_numbers[index,]\n#&gt;      [,1] [,2] [,3] [,4] [,5] [,6] [,7]\n#&gt; [1,]    1    3    8    0    6    7    5\n#&gt; [2,]    2    9    1    6    4    8    0\n#&gt; [3,]    5    1    6    0    9    8    2\n#&gt; [4,]    7    4    6    0    2    8    1\n#&gt; [5,]    4    6    5    9    2    8    0\n\nInstead of using the numbers 1 through 10, the default, it uses what we provided through v: the digits 0 through 9.\nTo compute all possible ways we can choose two cards when the order matters, we type:\n\nhands &lt;- permutations(52, 2, v = deck)\n\nThis is a matrix with two columns and 2652 rows. With a matrix, we can obtain the first and second cards like this:\n\nfirst_card &lt;- hands[,1]\nsecond_card &lt;- hands[,2]\n\nNow, the cases for which the first hand was a King can be computed as follows:\n\nkings &lt;- paste(\"King\", suits)\nsum(first_card %in% kings)\n#&gt; [1] 204\n\nTo get the conditional probability, we compute what fraction of these have a King as the second card:\n\nsum(first_card %in% kings & second_card %in% kings) / \n  sum(first_card %in% kings)\n#&gt; [1] 0.0588\n\nwhich is exactly 3/51, as we had already deduced. Notice that the code above is equivalent to:\n\nmean(first_card %in% kings & second_card %in% kings) / \n  mean(first_card %in% kings)\n#&gt; [1] 0.0588\n\nwhich uses mean instead of sum and is an R version of:\n\\[\n\\frac{\\mbox{Pr}(A \\mbox{ and } B)}{ \\mbox{Pr}(A)}\n\\]\nWhat about if the order does not matter? For example, in Blackjack, if you obtain an Ace and a face card in the first draw, it is called a Natural 21, and you win automatically. If we wanted to compute the probability of this happening, we would enumerate the combinations, not the permutations, since the order does not matter.\n\ncombinations(3,2)\n#&gt;      [,1] [,2]\n#&gt; [1,]    1    2\n#&gt; [2,]    1    3\n#&gt; [3,]    2    3\n\nIn the second line, the outcome does not include (2,1) because (1,2) already was enumerated. The same applies to (3,1) and (3,2).\nSo to compute the probability of a Natural 21 in Blackjack, we can do this:\n\naces &lt;- paste(\"Ace\", suits)\n\nfacecard &lt;- c(\"King\", \"Queen\", \"Jack\", \"Ten\")\nfacecard &lt;- expand.grid(number = facecard, suit = suits)\nfacecard &lt;- paste(facecard$number, facecard$suit)\n\nhands &lt;- combinations(52, 2, v = deck)\nmean(hands[,1] %in% aces & hands[,2] %in% facecard)\n#&gt; [1] 0.0483\n\nIn the last line, we assume the Ace comes first. This assumption is made based on our knowledge of how combination enumerates possibilities, and it will list this case first. However, to be safe, we could have written this and produced the same answer:\n\nmean((hands[,1] %in% aces & hands[,2] %in% facecard) |\n       (hands[,2] %in% aces & hands[,1] %in% facecard))\n#&gt; [1] 0.0483\n\n\n3.8.1 Monte Carlo example\nInstead of using combinations to deduce the exact probability of a Natural 21, we can use a Monte Carlo to estimate this probability. In this case, we draw two cards over and over and keep track of how many 21s we obtain. We can use the function sample to draw two cards without replacement:\n\nhand &lt;- sample(deck, 2)\nhand\n#&gt; [1] \"Queen Clubs\"  \"Seven Spades\"\n\nand then check if one card is an Ace and the other is a face card or a 10. Going forward, we include 10 when we refer to a face card. Now, we need to check both possibilities:\n\n(hand[1] %in% aces & hand[2] %in% facecard) | \n  (hand[2] %in% aces & hand[1] %in% facecard)\n#&gt; [1] FALSE\n\nIf we repeat this 10,000 times, we get a very good approximation of the probability of a Natural 21.\nLet’s start by writing a function that draws a hand and returns TRUE if we get a 21. The function does not need any arguments because it uses objects defined in the global environment.\n\nblackjack &lt;- function(){\n   hand &lt;- sample(deck, 2)\n  (hand[1] %in% aces & hand[2] %in% facecard) | \n    (hand[2] %in% aces & hand[1] %in% facecard)\n}\n\nHere. we do have to check both possibilities: Ace first or Ace second, because we are not using the combinations function. The function returns TRUE if we get a 21 and FALSE otherwise:\n\nblackjack()\n#&gt; [1] FALSE\n\nNow we can play this game, say, 10,000 times:\n\nB &lt;- 10000\nresults &lt;- replicate(B, blackjack())\nmean(results)\n#&gt; [1] 0.0475"
   },
   {
     "objectID": "prob/discrete-probability.html#examples",
@@ -718,7 +718,7 @@
     "href": "linear-models/regression.html#the-regression-fallacy",
     "title": "14  Regression",
     "section": "\n14.15 The regression fallacy",
-    "text": "14.15 The regression fallacy\nWikipedia defines the sophomore slump as:\n\nA sophomore slump or sophomore jinx or sophomore jitters refers to an instance in which a second, or sophomore, effort fails to live up to the standards of the first effort. It is commonly used to refer to the apathy of students (second year of high school, college or university), the performance of athletes (second season of play), singers/bands (second album), television shows (second seasons) and movies (sequels/prequels).\n\nIn Major League Baseball, the rookie of the year (ROY) award is given to the first-year player who is judged to have performed the best. The sophomore slump phrase is used to describe the observation that ROY award winners don’t do as well during their second year. For example, this Fox Sports article2 asks “Will MLB’s tremendous rookie class of 2015 suffer a sophomore slump?”\nDoes the data confirm the existence of a sophomore slump? Let’s take a look. Examining the data for a widely used measure of success, the batting average, we see that this observation holds true for the top performing ROYs:\n\n\n\n\nnameFirst\nnameLast\nrookie_year\nrookie\nsophomore\n\n\n\nWillie\nMcCovey\n1959\n0.354\n0.238\n\n\nIchiro\nSuzuki\n2001\n0.350\n0.321\n\n\nAl\nBumbry\n1973\n0.337\n0.233\n\n\nFred\nLynn\n1975\n0.331\n0.314\n\n\nAlbert\nPujols\n2001\n0.329\n0.314\n\n\n\n\n\nIn fact, the proportion of players that have a lower batting average during their sophomore year is 0.6981132.\nSo is it “jitters” or “jinx”? To answer this question, let’s turn our attention to all the players that played the 2013 and 2014 seasons and batted more than 130 times (minimum to win Rookie of the Year).\nThe same pattern arises when we look at the top performers: batting averages go down for most of the top performers.\n\n\n\n\nnameFirst\nnameLast\n2013\n2014\n\n\n\nMiguel\nCabrera\n0.348\n0.313\n\n\nHanley\nRamirez\n0.345\n0.283\n\n\nMichael\nCuddyer\n0.331\n0.332\n\n\nScooter\nGennett\n0.324\n0.289\n\n\nJoe\nMauer\n0.324\n0.277\n\n\n\n\n\nBut these are not rookies! Also, look at what happens to the worst performers of 2013:\n\n\n\n\nnameFirst\nnameLast\n2013\n2014\n\n\n\nDanny\nEspinosa\n0.158\n0.219\n\n\nDan\nUggla\n0.179\n0.149\n\n\nJeff\nMathis\n0.181\n0.200\n\n\nB. J.\nUpton\n0.184\n0.208\n\n\nAdam\nRosales\n0.190\n0.262\n\n\n\n\n\nTheir batting averages mostly go up! Is this some sort of reverse sophomore slump? It is not. There is no such thing as a sophomore slump. This is all explained with a simple statistical fact: the correlation for performance in two separate years is high, but not perfect:\n\n\n\n\n\n\n\n\nThe correlation is 0.460254 and the data look very much like a bivariate normal distribution, which means we predict a 2014 batting average \\(Y\\) for any given player that had a 2013 batting average \\(X\\) with:\n\\[ \\frac{Y - .255}{.032} = 0.46 \\left( \\frac{X - .261}{.023}\\right) \\]\nBecause the correlation is not perfect, regression tells us that, on average, expect high performers from 2013 to do a bit worse in 2014. It’s not a jinx; it’s just due to chance. The ROY are selected from the top values of \\(X\\), so it is expected that \\(Y\\) will regress to the mean."
+    "text": "14.15 The regression fallacy\nWikipedia defines the sophomore slump as:\n\nA sophomore slump or sophomore jinx or sophomore jitters refers to an instance in which a second, or sophomore, effort fails to live up to the standards of the first effort. It is commonly used to refer to the apathy of students (second year of high school, college or university), the performance of athletes (second season of play), singers/bands (second album), television shows (second seasons) and movies (sequels/prequels).\n\nIn Major League Baseball, the rookie of the year (ROY) award is given to the first-year player who is judged to have performed the best. The sophomore slump phrase is used to describe the observation that ROY award winners don’t do as well during their second year. For example, this Fox Sports article2 asks “Will MLB’s tremendous rookie class of 2015 suffer a sophomore slump?”\nDoes the data confirm the existence of a sophomore slump? Let’s take a look. Examining the data for a widely used measure of success, the batting average, we see that this observation holds true for the top performing ROYs:\n\n\n\n\nnameFirst\nnameLast\nrookie_year\nrookie\nsophomore\n\n\n\nWillie\nMcCovey\n1959\n0.354\n0.238\n\n\nIchiro\nSuzuki\n2001\n0.350\n0.321\n\n\nAl\nBumbry\n1973\n0.337\n0.233\n\n\nFred\nLynn\n1975\n0.331\n0.314\n\n\nAlbert\nPujols\n2001\n0.329\n0.314\n\n\n\n\n\nIn fact, the proportion of players that have a lower batting average during their sophomore year is 0.7037037.\nSo is it “jitters” or “jinx”? To answer this question, let’s turn our attention to all the players that played the 2013 and 2014 seasons and batted more than 130 times (minimum to win Rookie of the Year).\nThe same pattern arises when we look at the top performers: batting averages go down for most of the top performers.\n\n\n\n\nnameFirst\nnameLast\n2013\n2014\n\n\n\nMiguel\nCabrera\n0.348\n0.313\n\n\nHanley\nRamirez\n0.345\n0.283\n\n\nMichael\nCuddyer\n0.331\n0.332\n\n\nScooter\nGennett\n0.324\n0.289\n\n\nJoe\nMauer\n0.324\n0.277\n\n\n\n\n\nBut these are not rookies! Also, look at what happens to the worst performers of 2013:\n\n\n\n\nnameFirst\nnameLast\n2013\n2014\n\n\n\nDanny\nEspinosa\n0.158\n0.219\n\n\nDan\nUggla\n0.179\n0.149\n\n\nJeff\nMathis\n0.181\n0.200\n\n\nB. J.\nUpton\n0.184\n0.208\n\n\nAdam\nRosales\n0.190\n0.262\n\n\n\n\n\nTheir batting averages mostly go up! Is this some sort of reverse sophomore slump? It is not. There is no such thing as a sophomore slump. This is all explained with a simple statistical fact: the correlation for performance in two separate years is high, but not perfect:\n\n\n\n\n\n\n\n\nThe correlation is 0.460254 and the data look very much like a bivariate normal distribution, which means we predict a 2014 batting average \\(Y\\) for any given player that had a 2013 batting average \\(X\\) with:\n\\[ \\frac{Y - .255}{.032} = 0.46 \\left( \\frac{X - .261}{.023}\\right) \\]\nBecause the correlation is not perfect, regression tells us that, on average, expect high performers from 2013 to do a bit worse in 2014. It’s not a jinx; it’s just due to chance. The ROY are selected from the top values of \\(X\\), so it is expected that \\(Y\\) will regress to the mean."
   },
   {
     "objectID": "linear-models/regression.html#exercises",
@@ -739,7 +739,7 @@
     "href": "linear-models/multivariate-regression.html#case-study-moneyball",
     "title": "15  Multivariate Regression",
     "section": "\n15.1 Case study: Moneyball",
-    "text": "15.1 Case study: Moneyball\nMoneyball: The Art of Winning an Unfair Game by Michael Lewis focuses on the Oakland Athletics (A’s) baseball team and its general manager, Billy Beane, the person tasked with building the team.\nTraditionally, baseball teams use scouts to help them decide what players to hire. These scouts evaluate players by observing them perform, tending to favor athletic players with observable physical abilities. For this reason, scouts generally agree on who the best players are and, as a result, these players are often in high demand. This in turn drives up their salaries.\nFrom 1989 to 1991, the A’s had one of the highest payrolls in baseball. They were able to buy the best players and, during that time, were one of the best teams. However, in 1995, the A’s team owner changed and the new management cut the budget drastically, leaving then general manager, Sandy Alderson, with one of the lowest payrolls in baseball. He could no longer afford the most sought-after players. As a result, Alderson began using a statistical approach to find inefficiencies in the market. Alderson was a mentor to Billy Beane, who succeeded him in 1998 and fully embraced data science, as opposed to relying exclusively on scouts, as a method for finding low-cost players that data predicted would help the team win. Today, this strategy has been adapted by most baseball teams. As we will see, regression plays a significant role in this approach.\nAs motivation for this part of the book, let’s imagine it is 2002, and attempt to build a baseball team with a limited budget, just like the A’s had to do. To appreciate what you are up against, note that in 2002 the Yankees’ payroll of $125,928,583 more than tripled the Oakland A’s payroll of $39,679,746:\n\n\n\n\n\n\n\n\nStatistics have been used in baseball since its beginnings. The dataset we will be using, included in the Lahman library, goes back to the 19th century. For example, a summary statistics we will describe soon, the batting average, has been used for decades to summarize a batter’s success. Other statistics1, such as home runs (HR), runs batted in (RBI), and stolen bases (SB), are reported for each player in the game summaries included in the sports section of newspapers, with players rewarded for high numbers. Although summary statistics such as these were widely used in baseball, data analysis per se was not. These statistics were arbitrarily chosen without much thought as to whether they actually predicted anything or were related to helping a team win.\nThis changed with Bill James2. In the late 1970s, this aspiring writer and baseball fan started publishing articles describing more in-depth analysis of baseball data. He named the approach of using data to determine which outcomes best predicted if a team would win sabermetrics3. Yet until Billy Beane made sabermetrics the center of his baseball operation, Bill James’ work was mostly ignored by the baseball world. Currently, sabermetrics popularity is no longer limited to just baseball, with other sports also adopting this approach.\nTo simplify the exercise, we will focus on scoring runs and ignore the two other important aspects of the game: pitching and fielding. We will see how regression analysis can help develop strategies to build a competitive baseball team with a constrained budget. The approach can be divided into two separate data analyses. In the first, we determine which recorded player-specific statistics predict runs. In the second, we examine if players were undervalued based on the predictions from our first analysis.\n\n15.1.1 Baseball basics\nTo understand how regression helps us find undervalued players, we don’t need to delve into all the details of the game of baseball, which has over 100 rules. Here, we distill the sport to the basic knowledge necessary for effectively addressing the data science problem.\nThe goal of a baseball game is to score more runs (points) than the other team. Each team has 9 batters that have an opportunity to hit a ball with a bat in a predetermined order. After the 9th batter has had their turn, the first batter bats again, then the second, and so on. Each time a batter has an opportunity to bat, we call it a plate appearance (PA). At each PA, the other team’s pitcher throws the ball and the batter tries to hit it. The PA ends with an binary outcome: the batter either makes an out (failure) and returns to the bench, or the batter doesn’t (success) and can run around the bases, potentially scoring a run (reaching all 4 bases). Each team gets nine tries, referred to as innings, to score runs, and each inning ends after three outs (three failures).\nHere is a video showing a success: https://www.youtube.com/watch?v=HL-XjMCPfio. And here is one showing a failure: https://www.youtube.com/watch?v=NeloljCx-1g. In these videos, we see how luck is involved in the process. When at bat, the batter wants to hit the ball hard. If the batter hits it hard enough, it is a HR, the best possible outcome as the batter gets at least one automatic run. But sometimes, due to chance, the batter hits the ball very hard and a defender catches it, resulting in an out. In contrast, sometimes the batter hits the ball softly, but it lands in just the right place. The fact that there is chance involved hints at why probability models will be involved.\nNow, there are several ways to succeed. Understanding this distinction will be important for our analysis. When the batter hits the ball, the batter wants to pass as many bases as possible. There are four bases, with the fourth one called home plate. Home plate is where batters start by trying to hit, so the bases form a cycle.\n\n\n\n\n\n\n\n\n(Courtesy of Cburnett4. CC BY-SA 3.0 license5.) \nA batter who goes around the bases and arrives home, scores a run.\nWe are simplifying a bit, but there are five ways a batter can succeed, meaning not make an out:\n\nBases on balls (BB): The pitcher fails to throw the ball through a predefined area considered to be hittable (the strike zone), so the batter is permitted to go to first base.\nSingle: The batter hits the ball and gets to first base.\nDouble (2B): The batter hits the ball and gets to second base.\nTriple (3B): The batter hits the ball and gets to third base.\nHome Run (HR): The batter hits the ball and goes all the way home and scores a run.\n\nHere is an example of a HR: https://www.youtube.com/watch?v=xYxSZJ9GZ-w. If a batter reaches a base, the batter still has a chance of reaching home and scoring a run if the next batter succeeds with a hit. While the batter is on base, the batter can also try to steal a base (SB). If a batter runs fast enough, the batter can try to advance from one base to the next without the other team tagging the runner. Here is an example of a stolen base: https://www.youtube.com/watch?v=JSE5kfxkzfk.\nAll these events are tracked throughout the season and are available to us through the Lahman package. Now, we can begin discussing how data analysis can help us determine how to use these statistics to evaluate players.\n\n15.1.2 No awards for BB\nHistorically, the batting average has been considered the most important offensive statistic. To define this average, we define a hit (H) and an at bat (AB). Singles, doubles, triples, and home runs are hits. The fifth way to be successful, BB, is not a hit. An AB is the number of times in which you either get a hit or make an out; BBs are excluded. The batting average is simply H/AB and is considered the main measure of a success rate. Today, this success rate ranges from 20% to 38%. We refer to the batting average in thousands so, for example, if your success rate is 28%, we call it batting 280.\n\n\n\n\n\n\n\n\n(Picture courtesy of Keith Allison6. CC BY-SA 2.0 license7.)\nOne of Bill James’ first important insights is that the batting average ignores BB, but a BB is a success. Instead of batting average, James proposed the use of the on-base percentage (OBP), which he defined as (H+BB)/(AB+BB), or simply the proportion of plate appearances that don’t result in an out, a very intuitive measure. He noted that a player that accumulates many more BB than the average player might go unrecognized if the batter does not excel in batting average. But is this player not helping produce runs? No award is given to the player with the most BB. However, bad habits are hard to break and baseball did not immediately adopt OBP as an important statistic. In contrast, total stolen bases were considered important and an award8 given to the player with the most. But players with high totals of SB also made more outs as they did not always succeed. Does a player with high SB total help produce runs? Can we use data science to determine if it’s better to pay for players with high BB or SB?\n\n15.1.3 Base on balls or stolen bases?\nOne of the challenges in this analysis is that it is not obvious how to determine if a player produces runs because so much depends on his teammates. Although we keep track of the number of runs scored by a player, remember that if player X bats right before someone who hits many HRs, batter X will score many runs. Note these runs don’t necessarily happen if we hire player X, but not his HR hitting teammate.\nHowever, we can examine team-level statistics. How do teams with many SB compare to teams with few? How about BB? We have data! Let’s examine some. We start by creating a data frame with statistics from 1962 (the first year all teams played 162 games, like today, instead of 154) to 2001 (the year before the year for which we will construct a team). We convert the data to a per game rate, because a small proportion of seasons had less games than usual due to strikes, and some teams played extra games due to tie breakers.\n\nlibrary(tidyverse)\n#&gt; ── Attaching core tidyverse packages ──────────────── tidyverse 2.0.0 ──\n#&gt; ✔ dplyr     1.1.1     ✔ readr     2.1.4\n#&gt; ✔ forcats   1.0.0     ✔ stringr   1.5.0\n#&gt; ✔ lubridate 1.9.2     ✔ tibble    3.2.1\n#&gt; ✔ purrr     1.0.1     ✔ tidyr     1.3.0\n#&gt; ── Conflicts ────────────────────────────────── tidyverse_conflicts() ──\n#&gt; ✖ dplyr::filter() masks stats::filter()\n#&gt; ✖ dplyr::lag()    masks stats::lag()\n#&gt; ℹ Use the conflicted package (&lt;http://conflicted.r-lib.org/&gt;) to force all conflicts to become errors\nlibrary(Lahman)\ndat &lt;- Teams |&gt; filter(yearID %in% 1962:2002) |&gt;\n  mutate(team = teamID, year = yearID, r = R/G, \n         singles = (H - X2B - X3B - HR)/G, \n         doubles = X2B/G, triples = X3B/G, hr = HR/G,\n         sb = SB/G, bb = BB/G) |&gt;\n  select(team, year, r, singles, doubles, triples, hr, sb, bb)\n\nNow let’s start with a obvious question: do teams that hit more home runs score more runs? When exploring the relationship between two variables, The visualization of choice is a scatterplot:\n\np &lt;- dat |&gt; ggplot(aes(hr, r)) + geom_point(alpha = 0.5)\np \n\n\n\n\n\n\n\nWe defined p because we will add it to this plot latter. The plot shows a strong association: teams with more HRs tend to score more runs. Now let’s examine the relationship between stolen bases and runs:\n\ndat |&gt; ggplot(aes(sb, r)) + geom_point(alpha = 0.5)\n\n\n\n\n\n\n\nHere the relationship is not as clear. Finally, let’s examine the relationship between BB and runs:\n\ndat |&gt; ggplot(aes(bb, r)) + geom_point(alpha = 0.5)\n\n\n\n\n\n\n\nHere again we see a clear association. But does this mean that increasing a team’s BBs causes an increase in runs? One of the most important lessons you learn in this book is that association is not causation. In fact, it looks like BBs and HRs are also associated:\n\ndat |&gt; ggplot(aes(hr, bb)) + geom_point(alpha = 0.5)\n\n\n\n\n\n\n\nWe know that HRs cause runs because when a player hits a HR, they are guaranteed at least one run. Could it be that HRs also cause BB and this makes it appear as if BB cause runs? When this happens, we say there is confounding, an important concept we will learn more about throughout this section.\nLinear regression will help us parse out the information and quantify the associations. This, in turn, will aid us in determining what players to recruit. Specifically, we will try to predict things like how many more runs will a team score if we increase the number of BBs, but keep the HRs fixed? Regression will help us answer questions like this one.\n\n15.1.4 Regression applied to baseball statistics\nCan we use regression with these data? First, notice that the HR and Run data, shown above, appear to be bivariate normal. Specifically, the qqplots confirm that the normal approximation for each HR strata is useful here:\n\ndat |&gt; mutate(z_hr = round(scale(hr))) |&gt;\n  filter(z_hr %in% -2:3) |&gt;\n  ggplot() +  \n  stat_qq(aes(sample = r)) +\n  facet_wrap(~z_hr) \n\n\n\n\n\n\n\nNow we are ready to use linear regression to predict the number of runs a team will score, if we know how many home runs the team hits using regression:\n\nhr_fit  &lt;- lm(r ~ hr, data = dat)$coef\np + geom_abline(intercept = hr_fit[[1]], slope = hr_fit[[2]])\n\n\n\n\n\n\n\nNote that we can obtain the same plot more quickly by using the ggplot2 function geom_smooth, which computes and adds a regression line to plot along with confidence intervals. We use the argument method = \"lm\", which stands for linear model, the title of an upcoming section. We simplify the code above like this:\n\np + geom_smooth(method = \"lm\")\n\n\n\n\n\n\n\nIn the example above, the slope is 1.8517449. This tells us that teams that hit 1 more HR per game than the average team, score 1.8517449 more runs per game than the average team. Given that the most common final score is a difference of one run, this can certainly lead to a large increase in wins. Not surprisingly, HR hitters are very expensive. Because we are working on a budget, we will need to find some other way to increase wins. In the next chapter, we introduce linear models, which provide an framework for performing this analysis. In @ref{@moneyball}, we apply what have learned to build a baseball team."
+    "text": "15.1 Case study: Moneyball\nMoneyball: The Art of Winning an Unfair Game by Michael Lewis focuses on the Oakland Athletics (A’s) baseball team and its general manager, Billy Beane, the person tasked with building the team.\nTraditionally, baseball teams use scouts to help them decide what players to hire. These scouts evaluate players by observing them perform, tending to favor athletic players with observable physical abilities. For this reason, scouts generally agree on who the best players are and, as a result, these players are often in high demand. This in turn drives up their salaries.\nFrom 1989 to 1991, the A’s had one of the highest payrolls in baseball. They were able to buy the best players and, during that time, were one of the best teams. However, in 1995, the A’s team owner changed and the new management cut the budget drastically, leaving then general manager, Sandy Alderson, with one of the lowest payrolls in baseball. He could no longer afford the most sought-after players. As a result, Alderson began using a statistical approach to find inefficiencies in the market. Alderson was a mentor to Billy Beane, who succeeded him in 1998 and fully embraced data science, as opposed to relying exclusively on scouts, as a method for finding low-cost players that data predicted would help the team win. Today, this strategy has been adapted by most baseball teams. As we will see, regression plays a significant role in this approach.\nAs motivation for this part of the book, let’s imagine it is 2002, and attempt to build a baseball team with a limited budget, just like the A’s had to do. To appreciate what you are up against, note that in 2002 the Yankees’ payroll of $125,928,583 more than tripled the Oakland A’s payroll of $39,679,746:\n\n\n\n\n\n\n\n\nStatistics have been used in baseball since its beginnings. The dataset we will be using, included in the Lahman library, goes back to the 19th century. For example, a summary statistics we will describe soon, the batting average, has been used for decades to summarize a batter’s success. Other statistics1, such as home runs (HR), runs batted in (RBI), and stolen bases (SB), are reported for each player in the game summaries included in the sports section of newspapers, with players rewarded for high numbers. Although summary statistics such as these were widely used in baseball, data analysis per se was not. These statistics were arbitrarily chosen without much thought as to whether they actually predicted anything or were related to helping a team win.\nThis changed with Bill James2. In the late 1970s, this aspiring writer and baseball fan started publishing articles describing more in-depth analysis of baseball data. He named the approach of using data to determine which outcomes best predicted if a team would win sabermetrics3. Yet until Billy Beane made sabermetrics the center of his baseball operation, Bill James’ work was mostly ignored by the baseball world. Currently, sabermetrics popularity is no longer limited to just baseball, with other sports also adopting this approach.\nTo simplify the exercise, we will focus on scoring runs and ignore the two other important aspects of the game: pitching and fielding. We will see how regression analysis can help develop strategies to build a competitive baseball team with a constrained budget. The approach can be divided into two separate data analyses. In the first, we determine which recorded player-specific statistics predict runs. In the second, we examine if players were undervalued based on the predictions from our first analysis.\n\n15.1.1 Baseball basics\nTo understand how regression helps us find undervalued players, we don’t need to delve into all the details of the game of baseball, which has over 100 rules. Here, we distill the sport to the basic knowledge necessary for effectively addressing the data science problem.\nThe goal of a baseball game is to score more runs (points) than the other team. Each team has 9 batters that have an opportunity to hit a ball with a bat in a predetermined order. After the 9th batter has had their turn, the first batter bats again, then the second, and so on. Each time a batter has an opportunity to bat, we call it a plate appearance (PA). At each PA, the other team’s pitcher throws the ball and the batter tries to hit it. The PA ends with an binary outcome: the batter either makes an out (failure) and returns to the bench, or the batter doesn’t (success) and can run around the bases, potentially scoring a run (reaching all 4 bases). Each team gets nine tries, referred to as innings, to score runs, and each inning ends after three outs (three failures).\nHere is a video showing a success: https://www.youtube.com/watch?v=HL-XjMCPfio. And here is one showing a failure: https://www.youtube.com/watch?v=NeloljCx-1g. In these videos, we see how luck is involved in the process. When at bat, the batter wants to hit the ball hard. If the batter hits it hard enough, it is a HR, the best possible outcome as the batter gets at least one automatic run. But sometimes, due to chance, the batter hits the ball very hard and a defender catches it, resulting in an out. In contrast, sometimes the batter hits the ball softly, but it lands in just the right place. The fact that there is chance involved hints at why probability models will be involved.\nNow, there are several ways to succeed. Understanding this distinction will be important for our analysis. When the batter hits the ball, the batter wants to pass as many bases as possible. There are four bases, with the fourth one called home plate. Home plate is where batters start by trying to hit, so the bases form a cycle.\n\n\n\n\n\n\n\n\n(Courtesy of Cburnett4. CC BY-SA 3.0 license5.) \nA batter who goes around the bases and arrives home, scores a run.\nWe are simplifying a bit, but there are five ways a batter can succeed, meaning not make an out:\n\nBases on balls (BB): The pitcher fails to throw the ball through a predefined area considered to be hittable (the strike zone), so the batter is permitted to go to first base.\nSingle: The batter hits the ball and gets to first base.\nDouble (2B): The batter hits the ball and gets to second base.\nTriple (3B): The batter hits the ball and gets to third base.\nHome Run (HR): The batter hits the ball and goes all the way home and scores a run.\n\nHere is an example of a HR: https://www.youtube.com/watch?v=xYxSZJ9GZ-w. If a batter reaches a base, the batter still has a chance of reaching home and scoring a run if the next batter succeeds with a hit. While the batter is on base, the batter can also try to steal a base (SB). If a batter runs fast enough, the batter can try to advance from one base to the next without the other team tagging the runner. Here is an example of a stolen base: https://www.youtube.com/watch?v=JSE5kfxkzfk.\nAll these events are tracked throughout the season and are available to us through the Lahman package. Now, we can begin discussing how data analysis can help us determine how to use these statistics to evaluate players.\n\n15.1.2 No awards for BB\nHistorically, the batting average has been considered the most important offensive statistic. To define this average, we define a hit (H) and an at bat (AB). Singles, doubles, triples, and home runs are hits. The fifth way to be successful, BB, is not a hit. An AB is the number of times in which you either get a hit or make an out; BBs are excluded. The batting average is simply H/AB and is considered the main measure of a success rate. Today, this success rate ranges from 20% to 38%. We refer to the batting average in thousands so, for example, if your success rate is 28%, we call it batting 280.\n\n\n\n\n\n\n\n\n(Picture courtesy of Keith Allison6. CC BY-SA 2.0 license7.)\nOne of Bill James’ first important insights is that the batting average ignores BB, but a BB is a success. Instead of batting average, James proposed the use of the on-base percentage (OBP), which he defined as (H+BB)/(AB+BB), or simply the proportion of plate appearances that don’t result in an out, a very intuitive measure. He noted that a player that accumulates many more BB than the average player might go unrecognized if the batter does not excel in batting average. But is this player not helping produce runs? No award is given to the player with the most BB. However, bad habits are hard to break and baseball did not immediately adopt OBP as an important statistic. In contrast, total stolen bases were considered important and an award8 given to the player with the most. But players with high totals of SB also made more outs as they did not always succeed. Does a player with high SB total help produce runs? Can we use data science to determine if it’s better to pay for players with high BB or SB?\n\n15.1.3 Base on balls or stolen bases?\nOne of the challenges in this analysis is that it is not obvious how to determine if a player produces runs because so much depends on his teammates. Although we keep track of the number of runs scored by a player, remember that if player X bats right before someone who hits many HRs, batter X will score many runs. Note these runs don’t necessarily happen if we hire player X, but not his HR hitting teammate.\nHowever, we can examine team-level statistics. How do teams with many SB compare to teams with few? How about BB? We have data! Let’s examine some. We start by creating a data frame with statistics from 1962 (the first year all teams played 162 games, like today, instead of 154) to 2001 (the year before the year for which we will construct a team). We convert the data to a per game rate, because a small proportion of seasons had less games than usual due to strikes, and some teams played extra games due to tie breakers.\n\nlibrary(tidyverse)\n#&gt; ── Attaching core tidyverse packages ──────────────── tidyverse 2.0.0 ──\n#&gt; ✔ dplyr     1.1.4     ✔ readr     2.1.5\n#&gt; ✔ forcats   1.0.0     ✔ stringr   1.5.1\n#&gt; ✔ lubridate 1.9.3     ✔ tibble    3.2.1\n#&gt; ✔ purrr     1.0.2     ✔ tidyr     1.3.1\n#&gt; ── Conflicts ────────────────────────────────── tidyverse_conflicts() ──\n#&gt; ✖ dplyr::filter() masks stats::filter()\n#&gt; ✖ dplyr::lag()    masks stats::lag()\n#&gt; ℹ Use the conflicted package (&lt;http://conflicted.r-lib.org/&gt;) to force all conflicts to become errors\nlibrary(Lahman)\ndat &lt;- Teams |&gt; filter(yearID %in% 1962:2002) |&gt;\n  mutate(team = teamID, year = yearID, r = R/G, \n         singles = (H - X2B - X3B - HR)/G, \n         doubles = X2B/G, triples = X3B/G, hr = HR/G,\n         sb = SB/G, bb = BB/G) |&gt;\n  select(team, year, r, singles, doubles, triples, hr, sb, bb)\n\nNow let’s start with a obvious question: do teams that hit more home runs score more runs? When exploring the relationship between two variables, The visualization of choice is a scatterplot:\n\np &lt;- dat |&gt; ggplot(aes(hr, r)) + geom_point(alpha = 0.5)\np \n\n\n\n\n\n\n\nWe defined p because we will add it to this plot latter. The plot shows a strong association: teams with more HRs tend to score more runs. Now let’s examine the relationship between stolen bases and runs:\n\ndat |&gt; ggplot(aes(sb, r)) + geom_point(alpha = 0.5)\n\n\n\n\n\n\n\nHere the relationship is not as clear. Finally, let’s examine the relationship between BB and runs:\n\ndat |&gt; ggplot(aes(bb, r)) + geom_point(alpha = 0.5)\n\n\n\n\n\n\n\nHere again we see a clear association. But does this mean that increasing a team’s BBs causes an increase in runs? One of the most important lessons you learn in this book is that association is not causation. In fact, it looks like BBs and HRs are also associated:\n\ndat |&gt; ggplot(aes(hr, bb)) + geom_point(alpha = 0.5)\n\n\n\n\n\n\n\nWe know that HRs cause runs because when a player hits a HR, they are guaranteed at least one run. Could it be that HRs also cause BB and this makes it appear as if BB cause runs? When this happens, we say there is confounding, an important concept we will learn more about throughout this section.\nLinear regression will help us parse out the information and quantify the associations. This, in turn, will aid us in determining what players to recruit. Specifically, we will try to predict things like how many more runs will a team score if we increase the number of BBs, but keep the HRs fixed? Regression will help us answer questions like this one.\n\n15.1.4 Regression applied to baseball statistics\nCan we use regression with these data? First, notice that the HR and Run data, shown above, appear to be bivariate normal. Specifically, the qqplots confirm that the normal approximation for each HR strata is useful here:\n\ndat |&gt; mutate(z_hr = round(scale(hr))) |&gt;\n  filter(z_hr %in% -2:3) |&gt;\n  ggplot() +  \n  stat_qq(aes(sample = r)) +\n  facet_wrap(~z_hr) \n\n\n\n\n\n\n\nNow we are ready to use linear regression to predict the number of runs a team will score, if we know how many home runs the team hits using regression:\n\nhr_fit  &lt;- lm(r ~ hr, data = dat)$coef\np + geom_abline(intercept = hr_fit[[1]], slope = hr_fit[[2]])\n\n\n\n\n\n\n\nNote that we can obtain the same plot more quickly by using the ggplot2 function geom_smooth, which computes and adds a regression line to plot along with confidence intervals. We use the argument method = \"lm\", which stands for linear model, the title of an upcoming section. We simplify the code above like this:\n\np + geom_smooth(method = \"lm\")\n\n\n\n\n\n\n\nIn the example above, the slope is 1.8517449. This tells us that teams that hit 1 more HR per game than the average team, score 1.8517449 more runs per game than the average team. Given that the most common final score is a difference of one run, this can certainly lead to a large increase in wins. Not surprisingly, HR hitters are very expensive. Because we are working on a budget, we will need to find some other way to increase wins. In the next chapter, we introduce linear models, which provide an framework for performing this analysis. In @ref{@moneyball}, we apply what have learned to build a baseball team."
   },
   {
     "objectID": "linear-models/multivariate-regression.html#the-broom-package",
@@ -1502,70 +1502,77 @@
     "href": "ml/ml-in-practice.html#sec-caret",
     "title": "31  Machine learning in practice",
     "section": "\n31.1 The caret package",
-    "text": "31.1 The caret package\nWe have already learned about several machine learning algorithms. Many of these algorithms are implemented in R. However, they are distributed via different packages, developed by different authors, and often use different syntax. The caret package tries to consolidate these differences and provide consistency. It currently includes over 200 different methods which are summarized in the caret package manual1. Keep in mind that caret does not include the packages needed to run each possible algorithm. To apply a machine learning method through caret you still need to install the library that implement the method. The required packages for each method are described in the package manual.\nThe caret package also provides a function that performs cross validation for us. Here we provide some examples showing how we use this helpful package. We will first use the 2 or 7 example to illustrate and, in later sections, we use the package to run algorithms on the larger MNIST dataset.\n\n31.1.1 The train function\nThe R functions that fit machine algorithms are all slightly different. Functions such as lm, glm, qda, lda, knn3, rpart and randomForrest use different syntax, have different argument names and produce objects of different types.\nThe caret train function lets us train different algorithms using similar syntax. So, for example, we can type:\n\nlibrary(caret)\n#&gt; Loading required package: lattice\ntrain_glm &lt;- train(y ~ ., method = \"glm\", data = mnist_27$train)\ntrain_qda &lt;- train(y ~ ., method = \"qda\", data = mnist_27$train)\ntrain_knn &lt;- train(y ~ ., method = \"knn\", data = mnist_27$train)\n\nAs we explain in more detail in Section 31.1.3, the train function selects parameters for you using a resampling method, with boostrap as the default.\n\n31.1.2 The predict function\nThe predict function is very useful for machine learning applications. This function takes an object from a fitting function and a data frame with features \\(\\mathbf{x}\\) for which to predict, and returns predictions for these features.\nHere is an example with logistic regression:\n\nfit &lt;- glm(y ~ ., data = mnist_27$train, family = \"binomial\")\np_hat &lt;- predict(fit, mnist_27$test)\n\nIn this case, the function is simply computing:\n\\[\n\\hat{p}(\\mathbf{x}) = g^{-1}\\left(\\hat{\\beta}_0 + \\hat{\\beta}_1 x_1 + \\hat{\\beta}_2 x_2 \\right) \\text{ with } g(p) = \\log\\frac{p}{1-p} \\implies g^{-1}(\\mu) = \\frac{1}{1-e^{-\\mu}}\n\\]\nfor the x_1 and x_2 in the test set mnist_27$test. With these estimates in place, we can make our predictions and compute our accuracy:\n\ny_hat &lt;- factor(ifelse(p_hat &gt; 0.5, 7, 2))\n\nHowever, note that predict does not always return objects of the same types; it depends on what type of object it is applied to. To learn about the specifics, you need to look at the help file specific for the type of fit object that is being used. The predict is actually a special type of function in R (called a generic function) that calls other functions depending on what kind of object it receives. So if predict receives an object coming out of the lm function, it will call predict.glm. If it receives an object coming out of glm, it calls predict.qda. If the fit is from knn3, it calls predict.knn3, and so on. These functions are similar but not exactly. You can learn more about the differences by reading the help files:\n\n?predict.glm\n?predict.qda\n?predict.knn3\n\nThere are many other versions of predict and many machine learning algorithms have a predict function.\nAs with train, the caret packages unifies the use of predict with the function predict.train. This function takes the output of train and produces prediction of categories or estimates of \\(p(\\mathbf{x})\\).\nThe code looks the same for all methods:\n\ny_hat_glm &lt;- predict(train_glm, mnist_27$test, type = \"raw\")\ny_hat_qda &lt;- predict(train_qda, mnist_27$test, type = \"raw\")\ny_hat_knn &lt;- predict(train_knn, mnist_27$test, type = \"raw\")\n\nThis permits us to quickly compare the algorithms. For example, we can compare the accuracy like this:\n\nfits &lt;- list(glm = y_hat_glm, qda = y_hat_qda, knn = y_hat_knn)\nsapply(fits, function(fit) confusionMatrix(fit, mnist_27$test$y)$overall[[\"Accuracy\"]])\n#&gt;   glm   qda   knn \n#&gt; 0.775 0.815 0.835\n\n\n31.1.3 Cross validation\nWhen an algorithm includes a tuning parameter, train automatically uses cross validation to decide among a few default values. To find out what parameter or parameters are optimized, you can read the manual 2 or study the output of:\n\ngetModelInfo(\"knn\")\n\nWe can also use a quick lookup like this:\n\nmodelLookup(\"knn\")\n\nIf we run it with default values:\n\ntrain_knn &lt;- train(y ~ ., method = \"knn\", data = mnist_27$train)\n\nyou can quickly see the results of the cross validation using the ggplot function. The argument highlight highlights the max:\n\nggplot(train_knn, highlight = TRUE)\n\n\n\n\n\n\n\nBy default, the cross validation is performed by taking 25 bootstrap samples comprised of 25% of the observations. For the kNN method, the default is to try \\(k=5,7,9\\). We change this using the tuneGrid argument. The grid of values must be supplied by a data frame with the parameter names as specified in the modelLookup output.\nHere we present an example where we try out 30 values between 9 and 67. To do this with caret, we need to define a column named k, so we use this: data.frame(k = seq(9, 67, 2)). Note that when running this code, we are fitting 30 versions of kNN to 25 bootstrapped samples. Since we are fitting \\(30 \\times 25 = 750\\) kNN models, running this code will take several seconds. We set the seed because cross validation is a random procedure and we want to make sure the result here is reproducible.\n\nset.seed(2008)\ntrain_knn &lt;- train(y ~ ., method = \"knn\", \n                   data = mnist_27$train,\n                   tuneGrid = data.frame(k = seq(9, 71, 2)))\nggplot(train_knn, highlight = TRUE)\n\n\n\n\n\n\n\nTo access the parameter that maximized the accuracy, you can use this:\n\ntrain_knn$bestTune\n#&gt;     k\n#&gt; 27 61\n\nand the best performing model like this:\n\ntrain_knn$finalModel\n#&gt; 61-nearest neighbor model\n#&gt; Training set outcome distribution:\n#&gt; \n#&gt;   2   7 \n#&gt; 401 399\n\nThe function predict will use this best performing model. Here is the accuracy of the best model when applied to the test set, which we have not yet used because the cross validation was done on the training set:\n\nconfusionMatrix(predict(train_knn, mnist_27$test, type = \"raw\"),\n                mnist_27$test$y)$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;    0.825\n\nIf we want to change how we perform cross validation, we can use the trainControl function. We can make the code above go a bit faster by using, for example, 10-fold cross validation. This means we have 10 samples using 10% of the observations each. We accomplish this using the following code:\n\ncontrol &lt;- trainControl(method = \"cv\", number = 10, p = .9)\ntrain_knn_cv &lt;- train(y ~ ., method = \"knn\", \n                   data = mnist_27$train,\n                   tuneGrid = data.frame(k = seq(9, 71, 2)),\n                   trControl = control)\nggplot(train_knn_cv, highlight = TRUE)\n\n\n\n\n\n\n\nWe observe that the accuracy estimates are more variable, which is expected since we changed the number of samples used to estimate accuracy.\nNote that results component of the train output includes several summary statistics related to the variability of the cross validation estimates:\n\nnames(train_knn$results)\n#&gt; [1] \"k\"          \"Accuracy\"   \"Kappa\"      \"AccuracySD\" \"KappaSD\"\n\nYou can learn many more details about the caret package, from the manual 3."
+    "text": "31.1 The caret package\nWe have already learned about several machine learning algorithms. Many of these algorithms are implemented in R. However, they are distributed via different packages, developed by different authors, and often use different syntax. The caret package tries to consolidate these differences and provide consistency. It currently includes over 200 different methods which are summarized in the caret package manual1. Keep in mind that caret does not include the packages needed to run each possible algorithm. To apply a machine learning method through caret you still need to install the library that implement the method. The required packages for each method are described in the package manual.\nThe caret package also provides a function that performs cross validation for us. Here we provide some examples showing how we use this helpful package. We will first use the 2 or 7 example to illustrate and, in later sections, we use the package to run algorithms on the larger MNIST dataset.\n\n31.1.1 The train function\nThe R functions that fit machine algorithms are all slightly different. Functions such as lm, glm, qda, lda, knn3, rpart and randomForrest use different syntax, have different argument names and produce objects of different types.\nThe caret train function lets us train different algorithms using similar syntax. So, for example, we can type the following to train three different models:\n\nlibrary(caret)\ntrain_glm &lt;- train(y ~ ., method = \"glm\", data = mnist_27$train)\ntrain_qda &lt;- train(y ~ ., method = \"qda\", data = mnist_27$train)\ntrain_knn &lt;- train(y ~ ., method = \"knn\", data = mnist_27$train)\n\nAs we explain in more detail in Section 31.1.3, the train function selects parameters for you using a resampling method to estimate the MSE, with bootstrap as the default.\n\n31.1.2 The predict function\nThe predict function is very useful for machine learning applications. This function takes an object from a fitting function and a data frame with features \\(\\mathbf{x}\\) for which to predict, and returns predictions for these features.\nHere is an example with logistic regression:\n\nfit &lt;- glm(y ~ ., data = mnist_27$train, family = \"binomial\")\np_hat &lt;- predict(fit, newdata = mnist_27$test)\n\nIn this case, the function is simply computing\n\\[\n\\hat{p}(\\mathbf{x}) = g^{-1}\\left(\\hat{\\beta}_0 + \\hat{\\beta}_1 x_1 + \\hat{\\beta}_2 x_2 \\right) \\text{ with } g(p) = \\log\\frac{p}{1-p} \\implies g^{-1}(\\mu) = \\frac{1}{1-e^{-\\mu}}\n\\]\nfor the x_1 and x_2 in the test set mnist_27$test. With these estimates in place, we can make our predictions and compute our accuracy:\n\ny_hat &lt;- factor(ifelse(p_hat &gt; 0.5, 7, 2))\n\nHowever, note that predict does not always return objects of the same type; it depends on what type of object it is applied to. To learn about the specifics, you need to look at the help file specific for the type of fit object that is being used.\npredict is actually a special type of function in R called a generic function. Generic functions call other functions depending on what kind of object it receives. So if predict receives an object coming out of the lm function, it will call predict.lm. If it receives an object coming out of glm, it calls predict.glm. If the fit is from knn3, it calls predict.knn3, and so on. These functions are similar but not exactly. You can learn more about the differences by reading the help files:\n\n?predict.glm\n?predict.qda\n?predict.knn3\n\nThere are many other versions of predict and many machine learning algorithms define their own predict function.\nAs with train, the caret packages unifies the use of predict with the function predict.train. This function takes the output of train and produces prediction of categories or estimates of \\(p(\\mathbf{x})\\).\nThe code looks the same for all methods:\n\ny_hat_glm &lt;- predict(train_glm, mnist_27$test, type = \"raw\")\ny_hat_qda &lt;- predict(train_qda, mnist_27$test, type = \"raw\")\ny_hat_knn &lt;- predict(train_knn, mnist_27$test, type = \"raw\")\n\nThis permits us to quickly compare the algorithms. For example, we can compare the accuracy like this:\n\nfits &lt;- list(glm = y_hat_glm, qda = y_hat_qda, knn = y_hat_knn)\nsapply(fits, function(fit) confusionMatrix(fit, mnist_27$test$y)$overall[[\"Accuracy\"]])\n#&gt;   glm   qda   knn \n#&gt; 0.775 0.815 0.835\n\n\n31.1.3 Resampling\nWhen an algorithm includes a tuning parameter, train automatically uses a resampling method to estimate MSE and decide among a few default candidate values. To find out what parameter or parameters are optimized, you can read the caret manual2 or study the output of:\n\nmodelLookup(\"knn\")\n\nTo obtain all the details of how caret implements kNN you can use:\n\ngetModelInfo(\"knn\")\n\nIf we run it with default values:\n\ntrain_knn &lt;- train(y ~ ., method = \"knn\", data = mnist_27$train)\n\nyou can quickly see the results of the cross validation using the ggplot function. The argument highlight highlights the max:\n\nggplot(train_knn, highlight = TRUE)\n\n\n\n\n\n\n\nBy default, the resampling is performed by taking 25 bootstrap samples, each comprised of 25% of the observations.\nFor the kNN method, the default is to try \\(k=5,7,9\\). We change this using the tuneGrid argument. The grid of values must be supplied by a data frame with the parameter names as specified in the modelLookup output.\nHere we present an example where we try out 38 values between 1 and 75. To do this with caret, we need to define a column named k, so we use this: data.frame(k = seq(1, 75, 2)). Note that when running this code, we are fitting 38 versions of kNN to 25 bootstrapped samples. Since we are fitting \\(38 \\times 25 = 950\\) kNN models, running this code will take several seconds.\n\ntrain_knn &lt;- train(y ~ ., method = \"knn\", \n                   data = mnist_27$train,\n                   tuneGrid = data.frame(k = seq(1, 75, 2)))\nggplot(train_knn, highlight = TRUE)\n\n\n\n\n\n\n\n\nBecause resampling methods are random procedures, the same code can result in different results. To assure reproducible results you should set the seed, as we did at the start of this chapter.\n\nTo access the parameter that maximized the accuracy, you can use this:\n\ntrain_knn$bestTune\n#&gt;     k\n#&gt; 37 73\n\nand the best performing model like this:\n\ntrain_knn$finalModel\n#&gt; 73-nearest neighbor model\n#&gt; Training set outcome distribution:\n#&gt; \n#&gt;   2   7 \n#&gt; 401 399\n\nThe function predict will use this best performing model. Here is the accuracy of the best model when applied to the test set, which we have not yet used because the cross validation was done on the training set:\n\nconfusionMatrix(predict(train_knn, mnist_27$test, type = \"raw\"),\n                mnist_27$test$y)$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;    0.825\n\nBootstrapping is not always the best approach to resampling (see @knn-in-practice, for an example). If we want to change our resampling method, we can use the trainControl function. For example, the code below runs 10-fold cross validation. This means we have 10 samples using 90% of the observations to train in each sample. We accomplish this using the following code:\n\ncontrol &lt;- trainControl(method = \"cv\", number = 10, p = .9)\ntrain_knn_cv &lt;- train(y ~ ., method = \"knn\", \n                   data = mnist_27$train,\n                   tuneGrid = data.frame(k = seq(1, 71, 2)),\n                   trControl = control)\n\n\n\n\n\n\n\nThe results component of the train output includes several summary statistics related to the variability of the cross validation estimates:\n\nnames(train_knn$results)\n#&gt; [1] \"k\"          \"Accuracy\"   \"Kappa\"      \"AccuracySD\" \"KappaSD\"\n\nYou can learn many more details about the caret package, from the manual3."
   },
   {
     "objectID": "ml/ml-in-practice.html#preprocessing",
     "href": "ml/ml-in-practice.html#preprocessing",
     "title": "31  Machine learning in practice",
     "section": "\n31.2 Preprocessing",
-    "text": "31.2 Preprocessing\nWe often transform predictors before running the machine algorithm. We also remove predictors that are clearly not useful. We call these steps preprocessing.\nExamples of preprocessing include standardizing the predictors, taking the log transform of some predictors, removing predictors that are highly correlated with others, and removing predictors with very few non-unique values or close to zero variation.\nFor example, we can run the nearZero function from the caret package to see that several features do not vary much from observation to observation. We can see that there is a large number of features with close to 0 variability:\n\nlibrary(matrixStats)\nsds &lt;- colSds(x)\nhist(sds, breaks = 256)\n\n\n\n\n\n\n\nThis is expected because there are parts of the image that rarely contain writing (dark pixels).\nThe caret packages includes a function that recommends features to be removed due to near zero variance:\n\nnzv &lt;- nearZeroVar(x)\n\nWe can see the columns recommended for removal are the near the edges:\n\nimage(matrix(1:784 %in% nzv, 28, 28))\n\n\nrafalib::mypar()\nimage(matrix(1:784 %in% nzv, 28, 28))\n\n\n\n\n\n\n\nSo we end up keeping this number of columns:\n\ncol_index &lt;- setdiff(1:ncol(x), nzv)\nlength(col_index)\n#&gt; [1] 252\n\nNow we are ready to fit some models. Before we start, we need to add column names to the feature matrices as these are required by caret:\n\ncolnames(x) &lt;- 1:ncol(mnist$train$images)\ncolnames(x_test) &lt;- colnames(x)"
+    "text": "31.2 Preprocessing\nWe often transform predictors before running the machine algorithm. We also remove predictors that are clearly not useful. We call these steps preprocessing.\nExamples of preprocessing include standardizing the predictors, taking the log transform of some predictors, removing predictors that are highly correlated with others, and removing predictors with very few non-unique values or close to zero variation.\nFor example, we can run the nearZero function from the caret package to see that several features do not vary much from observation to observation. We can see that there is a large number of features with close to 0 variability:\n\nlibrary(matrixStats)\nsds &lt;- colSds(x)\nhist(sds, breaks = 256)\n\n\n\n\n\n\n\nThis is expected because there are parts of the image that rarely contain writing (dark pixels).\nThe caret packages includes a function that recommends features to be removed due to near zero variance:\n\nnzv &lt;- nearZeroVar(x)\n\nWe can see the columns recommended for removal are the near the edges:\n\nimage(matrix(1:784 %in% nzv, 28, 28))\n\n\nrafalib::mypar()\nimage(matrix(1:784 %in% nzv, 28, 28))\n\n\n\n\n\n\n\nSo we end up removing\n\nlength(nzv)\n#&gt; [1] 532\n\npredictors.\nThe caret package features the preProcess function, which allows users to establish a predefined set of preprocessing operations based on a training set. This function is designed to apply these operations to new datasets without recalculating anything on the test set, ensuring that all preprocessing steps are consistent and derived solely from the training data.\nBelow is an example demonstrating how to remove predictors with near-zero variance and then center the remaining predictors:\n\npp &lt;- preProcess(x, method = c(\"nzv\", \"center\"))\ncentered_subsetted_x_test &lt;- predict(pp, newdata = x_test)\ndim(centered_subsetted_x_test)\n#&gt; [1] 1000  252\n\nAdditionally, the train function in caret includes a preProcess argument that allows users to specify which preprocessing steps to apply automatically during model training. We will explore this functionality further in the context of a k-Nearest Neighbors model @knn-in-practice."
   },
   {
-    "objectID": "ml/ml-in-practice.html#k-nearest-neighbors",
-    "href": "ml/ml-in-practice.html#k-nearest-neighbors",
+    "objectID": "ml/ml-in-practice.html#parallelization",
+    "href": "ml/ml-in-practice.html#parallelization",
     "title": "31  Machine learning in practice",
-    "section": "\n31.3 k-nearest neighbors",
-    "text": "31.3 k-nearest neighbors\n\n\n\n\n\n\nBefore starting this section, note that the first two calls to the train function in the code below can take several hours to run. This is a common challenge when training machine learning algorithms since we have to run the algorithm for each cross validation split and each set of tuning parameters being considered. In the next section, we will provide some suggestions on how to predict the duration of the process and ways to reduce.\n\n\n\nThe first step is to optimize for \\(k\\).\n\ntrain_knn &lt;- train(x[ ,col_index], y, \n                   method = \"knn\", \n                   tuneGrid = data.frame(k = seq(3, 13, 2)))\n\nOnce we optimize our algorithm, we can fit it to the entire dataset:\n\nfit_knn &lt;- knn3(x[, col_index], y,  k = train_knn$bestTune)\n\nWe achieve a high accuracy:\n\ny_hat_knn &lt;- predict(fit_knn, x_test[, col_index], type = \"class\")\nconfusionMatrix(y_hat_knn, factor(y_test))$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;    0.944\n\n\n31.3.1 Dimension reduction with PCA\nAn alternative to removing low variance columns directly is to use dimension reduction on the feature matrix before applying the algorithms. It is important that we not use the test set when finding the PCs nor any summary of the data, as this could result in overtraining. So we start by applying prcomp to the training data:\n\ncol_means &lt;- colMeans(x)\npca &lt;- prcomp(sweep(x, 2, col_means), center = FALSE)\n\nNext, we run kNN on just a small number of dimensions. We try 36 dimensions since this explains about 80% of the data.\n\nk &lt;- 36\nx_train &lt;- pca$x[,1:k]\ntrain_knn &lt;- train(x_train, y, \n                   method = \"knn\", \n                   tuneGrid = data.frame(k = seq(3, 13, 2)))\nfit_knn_pca &lt;- knn3(x_train, y, k = train_knn$bestTune)\n\nNow we apply the transformation we learned with the training data to the test data, reduce the dimension, and then run predict. Note that we used the rotation and column means estimated from the training data.\n\nnewdata &lt;-  sweep(x_test, 2, col_means) %*% pca$rotation[,1:k]\ny_hat_knn_pca &lt;- predict(fit_knn_pca, newdata, type = \"class\")\nconfusionMatrix(y_hat_knn_pca, factor(y_test))$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;     0.96\n\nWe obtain an improvement in accuracy, while using only 36 dimensions.\n\n\n\n\n\n\nRemember the entire algorithm needs to be developed on the training data. This is why the column means, subtracted from test set columns, and the rotation applied to obtained PCs, were both computed using the train data."
+    "section": "\n31.3 Parallelization",
+    "text": "31.3 Parallelization\nDuring cross-validation or bootstrapping, the process of fitting models to different samples or using varying parameters can be performed independently. Imagine you are fitting 100 models; if you had access to 100 computers, you could theoretically speed up the process by a factor of 100 by fitting each model on a separate computer and then aggregating the results. In reality, most modern computers, including many personal computers, are equipped with multiple processors that allow for such parallel execution. This method, known as parallelization, leverages these multiple processors to conduct several computational tasks simultaneously, significantly accelerating the model training process. By distributing the workload across different processors, parallelization makes it feasible to manage large datasets and complex modeling procedures efficiently.\nThe caret package is set up to run in parallel but you have to let R know that you are want to parallelize your work. To do this we can use the doParallel package:\n\nlibrary(doParallel)\n#&gt; Loading required package: foreach\n#&gt; Loading required package: iterators\n#&gt; Loading required package: parallel\nnc &lt;- detectCores() - 1\ncl &lt;- makeCluster(nc) # it is convention to leave 1 core for OS\nregisterDoParallel(cl)\n\nIf you do use parallelization, make sure to let R know you are done with the following lines of code:\n\nstopCluster(cl)\nstopImplicitCluster()\n\n\n\n\n\n\n\nWhen parallelizing tasks across multiple processors, it’s important to consider the risk of running out of memory. Each processor might require a copy of the data or substantial portions of it, which can multiply overall memory demands. This is especially challenging if the data or models are large."
+  },
+  {
+    "objectID": "ml/ml-in-practice.html#knn-in-practice",
+    "href": "ml/ml-in-practice.html#knn-in-practice",
+    "title": "31  Machine learning in practice",
+    "section": "\n31.4 k-nearest neighbors",
+    "text": "31.4 k-nearest neighbors\n\n\n\n\n\n\nBefore starting this section, note that the first two calls to the train function in the code below can take several hours to run. This is a common challenge when training machine learning algorithms since we have to run the algorithm for each cross validation split and each set of tuning parameters being considered. In the next section, we will provide some suggestions on how to predict the duration of the process and ways to reduce.\n\n\n\nAs we will see soon, the optimal \\(k\\) for the MNIST data is between 1 and 7. For small values of \\(k\\), bootstrapping can be problematic for estimating MSE for a k-Nearest Neighbors (kNN). This is because bootstrapping involves sampling with replacement from the original dataset which implies the closest neighbor will often appear twice but considered two independent observations by kNN. This is a unrealistic scenario that can distort the estimate MSE when, for example, \\(k=3\\). We therefore us cross-validation to estimate our MSE.\nThe first step is to optimize for \\(k\\).\n\ntrain_knn &lt;- train(x, y, method = \"knn\", \n                   preProcess = \"nzv\",\n                   trControl = trainControl(\"cv\", number = 20, p = 0.95),\n                   tuneGrid = data.frame(k = seq(1, 7, 2)))\n\nOnce we optimize our algorithm, the predict function defaults to using the best performing algorithm fit with the entire training data:\n\ny_hat_knn &lt;- predict(train_knn, x_test, type = \"raw\")\n\nWe achieve relatively high accuracy:\n\nconfusionMatrix(y_hat_knn, factor(y_test))$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;    0.953\n\n\n31.4.1 Dimension reduction with PCA\nAn alternative to removing low variance columns directly is to use dimension reduction on the feature matrix before applying the algorithms.\n\npca &lt;- prcomp(x)\n\nWe can actually explain, say, 75% of the variability in the predictors with a small number of dimensions:\n\np &lt;- which(cumsum(pca$sdev^2)/sum(pca$sdev^2) &gt;= 0.75)[1]\n\nWe can then re-run our algorithm on these 33 features:\n\nfit_knn_pca &lt;- knn3(pca$x[,1:p], y, k = train_knn$bestTune)\n\nWhen predicting, it is important that we not use the test set when finding the PCs nor any summary of the data, as this could result in overtraining. We therefrom compute the averages needed for centering and the rotation on the training set:\n\nnewdata &lt;-  sweep(x_test, 2, colMeans(x)) %*% pca$rotation[,1:p]\ny_hat_knn_pca &lt;- predict(fit_knn_pca, newdata, type = \"class\")\n\nWe obtain similar accuracy, while using only 33 dimensions:\n\nconfusionMatrix(y_hat_knn_pca, factor(y_test))$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;    0.959\n\nIn this example, we used the \\(k\\) optimized for the raw data, not the principal components. Note that to obtain an unbiased MSE estimate we have to recompute the PCA for each cross-validation sample and apply to the validation set. Because the train function includes PCA as one of the available preprocessing operations we can achieve this with this modification of the code above:\n\ntrain_knn_pca &lt;- train(x, y, method = \"knn\", \n                       preProcess = c(\"nzv\", \"pca\"),\n                       trControl = trainControl(\"cv\", number = 20, p = 0.95,\n                                                preProcOptions = list(pcaComp = p)),\n                       tuneGrid = data.frame(k = seq(1, 7, 2)))\ny_hat_knn_pca &lt;- predict(train_knn_pca, x_test, type = \"raw\")\nconfusionMatrix(y_hat_knn_pca, factor(y_test))$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;    0.964\n\nA limitation of this approach is that we don’t get to optimize the number of PCs used in the analysis. To do this we need to write our own method. The caret manual4 describe how to do this."
   },
   {
     "objectID": "ml/ml-in-practice.html#random-forest",
     "href": "ml/ml-in-practice.html#random-forest",
     "title": "31  Machine learning in practice",
-    "section": "\n31.4 Random Forest",
-    "text": "31.4 Random Forest\nWith the random forest algorithm several parameters can be optimized, but the main one is mtry, the number of predictors that are randomly selected for each tree. This is also the only tuning parameter that the caret function train permits when using the default implementation from the randomForest package.\n\nlibrary(randomForest)\ngrid &lt;- data.frame(mtry = seq(3, 24, 3))\ntrain_rf &lt;-  train(x[, col_index], y, \n                   method = \"rf\", \n                   tuneGrid = grid)\n\nNow that we have optimized our algorithm, we are ready to fit our final model:\n\nfit_rf &lt;- randomForest(x[, col_index], y, mtry = train_rf$bestTune$mtry)\n\nAs with kNN, we also achieve high accuracy:\n\ny_hat_rf &lt;- predict(fit_rf, x_test[ ,col_index])\nconfusionMatrix(y_hat_rf, y_test)$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;    0.955\n\nBy optimizing some of the other algorithm parameters, we can achieve even higher accuracy."
+    "section": "\n31.5 Random Forest",
+    "text": "31.5 Random Forest\nWith the random forest algorithm several parameters can be optimized, but the main one is mtry, the number of predictors that are randomly selected for each tree. This is also the only tuning parameter that the caret function train permits when using the default implementation from the randomForest package.\n\nlibrary(randomForest)\ntrain_rf &lt;- train(x, y, method = \"rf\", \n                  preProcess = \"nzv\",\n                  tuneGrid = data.frame(mtry = seq(5, 15)))\ny_hat_rf &lt;- predict(train_rf, x_test, type = \"raw\")\n\nNow that we have optimized our algorithm, we are ready to fit our final model:\n\ny_hat_rf &lt;- predict(train_rf, x_test, type = \"raw\")\n\nAs with kNN, we also achieve high accuracy:\n\nconfusionMatrix(y_hat_rf, y_test)$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;    0.952\n\nBy optimizing some of the other algorithm parameters, we can achieve even higher accuracy."
   },
   {
     "objectID": "ml/ml-in-practice.html#testing-and-improving-computation-time",
     "href": "ml/ml-in-practice.html#testing-and-improving-computation-time",
     "title": "31  Machine learning in practice",
-    "section": "\n31.5 Testing and improving computation time",
-    "text": "31.5 Testing and improving computation time\nThe default method for estimating accuracy used by the train function is to test prediction on 25 bootstrap samples. This can result in long compute times. For example, if we are considering several values, say 10, of the tuning parameters, we will fit the algorithm 250 times. We can use the system.time function to estimate how long it takes to run the algorithm once:\n\nsystem.time({fit_rf &lt;- randomForest(x[, col_index], y,  mtry = 9)})\n#&gt;    user  system elapsed \n#&gt;  59.707   0.046  59.791\n\nand use this to estimate the total time for the 250 iterations. In this case it will be several hours.\nOne way to reduce run time is to use k-fold cross validation with a smaller number of test sets. A popular choice is leaving out 10 test sets with 10% of the data:\n\ncontrol &lt;- trainControl(method = \"cv\", number = 10, p = .9)\n\nand re-running the train function with this choice specified via the trControl argument:\n\ntrain_rf &lt;-  train(x[, col_index], y, \n                   method = \"rf\", \n                   tuneGrid = grid,\n                   trControl = control)\n\nFor random forest, we can also speed up the training step by running less trees per fit. After running the algorithm once, we can use the plot function to see how the error rate changes as the number of trees grows.\nHere we can see that error rate stabilizes after about 200 trees:\n\nplot(fit_rf)\n\n\n\n\n\n\n\nWe can use this finding to speed up the cross validation procedure. Specifically, because the default is 500, by adding the argument ntree = 200 to the call to train above, the procedure will finish 2.5 times faster."
+    "section": "\n31.6 Testing and improving computation time",
+    "text": "31.6 Testing and improving computation time\nThe default method for estimating accuracy used by the train function is to test prediction on 25 bootstrap samples. This can result in long compute times. For example, if we are considering several values, say 10, of the tuning parameters, we will fit the algorithm 250 times. We can use the system.time function to estimate how long it takes to run the algorithm once:\n\nnzv &lt;- nearZeroVar(x)\nsystem.time({fit_rf &lt;- randomForest(x[, -nzv], y,  mtry = 9)})\n#&gt;    user  system elapsed \n#&gt;  61.656   0.053  62.368\n\nand use this to estimate the total time for the 250 iterations. In this case it will be several hours.\nOne way to reduce run time is to use k-fold cross validation with a smaller number of test sets. A popular choice is leaving out 5 test sets with 20% of the data. To use this we set the trControl argument in train to trainControl(method = \"cv\", number = 5, p = .8).\nFor random forest, we can also speed up the training step by running less trees per fit. After running the algorithm once, we can use the plot function to see how the error rate changes as the number of trees grows.\nHere we can see that error rate stabilizes after about 200 trees:\n\nplot(fit_rf)\n\n\n\n\n\n\n\nWe can use this finding to speed up the cross validation procedure. Specifically, because the default is 500, by adding the argument ntree = 200 to the call to train above, the procedure will finish 2.5 times faster."
   },
   {
     "objectID": "ml/ml-in-practice.html#variable-importance",
     "href": "ml/ml-in-practice.html#variable-importance",
     "title": "31  Machine learning in practice",
-    "section": "\n31.6 Variable importance",
-    "text": "31.6 Variable importance\nThe following function computes the importance of each feature:\n\nimp &lt;- importance(fit_rf)\n\nWe can see which features are being used most by plotting an image:\n\nmat &lt;- rep(0, ncol(x))\nmat[col_index] &lt;- imp\nimage(matrix(mat, 28, 28))"
+    "section": "\n31.7 Variable importance",
+    "text": "31.7 Variable importance\nThe following function computes the importance of each feature:\n\nimp &lt;- importance(fit_rf)\n\nWe can see which features are being used most by plotting an image:\n\nmat &lt;- rep(0, ncol(x))\nmat[-nzv] &lt;- imp\nimage(matrix(mat, 28, 28))"
   },
   {
     "objectID": "ml/ml-in-practice.html#diagnostics",
     "href": "ml/ml-in-practice.html#diagnostics",
     "title": "31  Machine learning in practice",
-    "section": "\n31.7 Diagnostics",
-    "text": "31.7 Diagnostics\nAn important part of data analysis is visualizing results to determine why we are failing. How we do this depends on the application. Below we show the images of digits for which we made an incorrect prediction. Here are some errors for the random forest:\n\n\n\n\n\n\n\n\nBy examining errors like this, we often find specific weaknesses to algorithms or parameter choices and can try to correct them."
+    "section": "\n31.8 Diagnostics",
+    "text": "31.8 Diagnostics\nAn important part of data analysis is visualizing results to determine why we are failing. How we do this depends on the application. Below we show the images of digits for which we made an incorrect prediction. Here are some errors for the random forest:\n\n\n\n\n\n\n\n\nBy examining errors like this, we often find specific weaknesses to algorithms or parameter choices and can try to correct them."
   },
   {
     "objectID": "ml/ml-in-practice.html#ensembles",
     "href": "ml/ml-in-practice.html#ensembles",
     "title": "31  Machine learning in practice",
-    "section": "\n31.8 Ensembles",
-    "text": "31.8 Ensembles\nThe idea of an ensemble is similar to the idea of combining data from different pollsters to obtain a better estimate of the true support for each candidate.\nIn machine learning, one can usually greatly improve the final results by combining the results of different algorithms.\nHere is a simple example where we compute new class probabilities by taking the average of random forest and kNN. We can see that the accuracy improves:\n\np_rf &lt;- predict(fit_rf, x_test[,col_index], type = \"prob\")  \np_rf &lt;- p_rf / rowSums(p_rf)\np_knn_pca  &lt;- predict(fit_knn_pca, newdata)\np &lt;- (p_rf + p_knn_pca)/2\ny_pred &lt;- factor(apply(p, 1, which.max) - 1)\nconfusionMatrix(y_pred, y_test)$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;    0.967\n\nWe have just built an ensemble with just two algorithms. By combing more similarly performing, but uncorrelated, algorithms we can improve accuracy further."
+    "section": "\n31.9 Ensembles",
+    "text": "31.9 Ensembles\nThe idea of an ensemble is similar to the idea of combining data from different pollsters to obtain a better estimate of the true support for each candidate.\nIn machine learning, one can usually greatly improve the final results by combining the results of different algorithms.\nHere is a simple example where we compute new class probabilities by taking the average of random forest and kNN. We can see that the accuracy improves:\n\np_rf &lt;- predict(fit_rf, x_test[,-nzv], type = \"prob\")  \np_rf &lt;- p_rf / rowSums(p_rf)\np_knn_pca  &lt;- predict(train_knn_pca, x_test, type = \"prob\")\np &lt;- (p_rf + p_knn_pca)/2\ny_pred &lt;- factor(apply(p, 1, which.max) - 1)\nconfusionMatrix(y_pred, y_test)$overall[\"Accuracy\"]\n#&gt; Accuracy \n#&gt;    0.971\n\nWe have just built an ensemble with just two algorithms. By combing more similarly performing, but uncorrelated, algorithms we can improve accuracy further."
   },
   {
     "objectID": "ml/ml-in-practice.html#exercises",
     "href": "ml/ml-in-practice.html#exercises",
     "title": "31  Machine learning in practice",
-    "section": "\n31.9 Exercises",
-    "text": "31.9 Exercises\n1. In the exercises in Chapter 30 we saw that changing maxnodes or nodesize in the randomForest function improved our estimate. Let’s use the train function to help us pick these values. From the caret manual we see that we can’t tune the maxnodes parameter or the nodesize argument with randomForest, so we will use the Rborist package and tune the minNode argument. Use the train function to try values minNode &lt;- seq(5, 250, 25). See which value minimizes the estimated RMSE.\n2. This **dslabs* dataset includes a matrix x:\n\nlibrary(dslabs)\ndim(tissue_gene_expression$x)\n\nwith the gene expression measured on 500 genes for 189 biological samples representing seven different tissues. The tissue type is stored in y:\n\ntable(tissue_gene_expression$y)\n\nSplit the data in training and test sets, then use kNN to predict tissue type and see what accuracy you obtain. Try it for \\(k = 1, 3, \\dots, 11\\).\n3. We are going to apply LDA and QDA to the tissue_gene_expression dataset. We will start with simple examples based on this dataset and then develop a realistic example.\nCreate a dataset with just the classes cerebellum and hippocampus (two parts of the brain) and a predictor matrix with 10 randomly selected columns. Estimate the accuracy of LDA.\n\nset.seed(1993)\ntissues &lt;- c(\"cerebellum\", \"hippocampus\")\nind &lt;- which(tissue_gene_expression$y %in% tissues)\ny &lt;- droplevels(tissue_gene_expression$y[ind])\nx &lt;- tissue_gene_expression$x[ind, ]\nx &lt;- x[, sample(ncol(x), 10)]\n\n4. In this case, LDA fits two 10-dimensional normal distributions. Look at the fitted model by looking at the finalModel component of the result of train. Notice there is a component called means that includes the estimate means of both distributions. Plot the mean vectors against each other and determine which predictors (genes) appear to be driving the algorithm.\n5. Repeat exercises 3 with QDA. Does it have a higher accuracy than LDA?\n6. Are the same predictors (genes) driving the algorithm? Make a plot as in exercise 3.\n7. One thing we see in the previous plot is that the value of predictors correlate in both groups: some predictors are low in both groups while others are high in both groups. The mean value of each predictor, colMeans(x), is not informative or useful for prediction and often, for interpretation purposes, it is useful to center or scale each column. This can be achieved with the preProcessing argument in train. Re-run LDA with preProcessing = \"scale\". Note that accuracy does not change but see how it is easier to identify the predictors that differ more between groups in the plot made in exercise 4.\n8. In the previous exercises, we saw that both approaches worked well. Plot the predictor values for the two genes with the largest differences between the two groups in a scatterplot to see how they appear to follow a bivariate distribution as assumed by the LDA and QDA approaches. Color the points by the outcome.\n9. Now we are going to increase the complexity of the challenge slightly: we will consider all the tissue types.\n\nset.seed(1993)\ny &lt;- tissue_gene_expression$y\nx &lt;- tissue_gene_expression$x\nx &lt;- x[, sample(ncol(x), 10)]\n\nWhat accuracy do you get with LDA?\n10. We see that the results are slightly worse. Use the confusionMatrix function to learn what type of errors we are making.\n11. Plot an image of the centers of the seven 10-dimensional normal distributions.\n12. Make a scatterplot along with the prediction from the best fitted model.\n13. Use the rpart function to fit a classification tree to the tissue_gene_expression dataset. Use the train function to estimate the accuracy. Try out cp values of seq(0, 0.05, 0.01). Plot the accuracy to report the results of the best model.\n14. Study the confusion matrix for the best fitting classification tree. What do you observe happening for placenta?\n15. Notice that placentas are called endometrium more often than placenta. Note also that the number of placentas is just six, and that, by default, rpart requires 20 observations before splitting a node. Thus it is not possible with these parameters to have a node in which placentas are the majority. Rerun the above analysis, but this time permit rpart to split any node by using the argument control = rpart.control(minsplit = 0). Does the accuracy increase? Look at the confusion matrix again.\n16. Plot the tree from the best fitting model obtained in exercise 11.\n17. We can see that with just six genes, we are able to predict the tissue type. Now let’s see if we can do even better with a random forest. Use the train function and the rf method to train a random forest. Try out values of mtry ranging from, at least, seq(50, 200, 25). What mtry value maximizes accuracy? To permit small nodesize to grow as we did with the classification trees, use the following argument: nodesize = 1. This will take several seconds to run. If you want to test it out, try using smaller values with ntree. Set the seed to 1990.\n18. Use the function varImp on the output of train and save it to an object called imp.\n19. The rpart model we ran above produced a tree that used just six predictors. Extracting the predictor names is not straightforward, but can be done. If the output of the call to train was fit_rpart, we can extract the names like this:\n\nind &lt;- !(fit_rpart$finalModel$frame$var == \"&lt;leaf&gt;\")\ntree_terms &lt;- \n  fit_rpart$finalModel$frame$var[ind] |&gt;\n  unique() |&gt;\n  as.character()\ntree_terms\n\nWhat is the variable importance in the random forest call for these predictors? Where do they rank?\n20. Extract the top 50 predictors based on importance, take a subset of x with just these predictors and apply the function heatmap to see how these genes behave across the tissues. We will introduce the heatmap function in Chapter 32.\n21. Previously, we compared the conditional probability \\(p(\\mathbf{x})\\) give two predictors \\(\\mathbf{x} = (x_1, x_2)^\\top\\) to the fit \\(\\hat{p}(\\mathbf{x})\\) obtained with a machine learning algorithm by making image plots. The following code can be used to make these images and include a curve at the values of \\(x_1\\) and \\(x_2\\) for which the function is \\(0.5\\):\n\nplot_cond_prob &lt;- function(x_1, x_2, p){\n  data.frame(x_1 = x_1, x_2 = x_2, p = p) |&gt;\n    ggplot(aes(x_1, x_2)) +\n    geom_raster(aes(fill = p), show.legend = FALSE) +\n    stat_contour(aes(z = p), breaks = 0.5, color = \"black\") +\n    scale_fill_gradientn(colors = c(\"#F8766D\", \"white\", \"#00BFC4\"))\n}\n\nWe can see the true conditional probability for the 2 or 7 example like this:\n\nwith(mnist_27$true_p, plot_cond_prob(x_1, x_2, p))\n\nFit a kNN model and make this plot for the estimated conditional probability. Hint: Use the argument newdata = mnist27$train to obtain predictions for a grid points.\n22. Notice that, in the plot made in exercise 1, the boundary is somewhat wiggly. This is because kNN, like the basic bin smoother, does not use a kernel. To improve this we could try loess. By reading through the available models part of the caret manual, we see that we can use the gamLoess method. We need to install the gam package, if we have not done so already. We see that we have two parameters to optimize:\n\nmodelLookup(\"gamLoess\")\n#&gt;      model parameter  label forReg forClass probModel\n#&gt; 1 gamLoess      span   Span   TRUE     TRUE      TRUE\n#&gt; 2 gamLoess    degree Degree   TRUE     TRUE      TRUE\n\nUse cross-validation to pick a span between 0.15 and 0.75. Keep degree = 1. What span does cross-validation select?\n23. Show an image plot of the estimate \\(\\hat{p}(x,y)\\) resulting from the model fit in the exercise 2. How does the accuracy compare to that of kNN? Comment on the difference between the estimate obtained with kNN.\n24. Use the mnist_27 training set to build a model with several of the models available from the caret package. For example, you can try these:\n\nmodels &lt;- c(\"glm\", \"lda\",  \"naive_bayes\",  \"svmLinear\", \"gamboost\",  \n            \"gamLoess\", \"qda\", \"knn\", \"kknn\", \"loclda\", \"gam\", \"rf\", \n            \"ranger\",\"wsrf\", \"Rborist\", \"avNNet\", \"mlp\", \"monmlp\", \"gbm\", \n            \"adaboost\", \"svmRadial\", \"svmRadialCost\", \"svmRadialSigma\")\n\nWe have not explained many of these, but apply them anyway using train with all the default parameters. Keep the results in a list. You might need to install some packages. Keep in mind that you will likely get some warnings.\n25. Now that you have all the trained models in a list, use sapply or map to create a matrix of predictions for the test set. You should end up with a matrix with length(mnist_27$test$y) rows and length(models) columns.\n26. Compute accuracy for each model on the test set.\n27. Build an ensemble prediction by majority vote and compute its accuracy.\n28. Earlier we computed the accuracy of each method on the training set and noticed they varied. Which individual methods do better than the ensemble?\n29. It is tempting to remove the methods that do not perform well and re-do the ensemble. The problem with this approach is that we are using the test data to make a decision. However, we could use the accuracy estimates obtained from cross validation with the training data. Obtain these estimates and save them in an object.\n30. Now let’s only consider the methods with an estimated accuracy of 0.8 when constructing the ensemble. What is the accuracy now?\n31. Note that if two machine algorithms methods predict the same outcome, ensembling them will not change the prediction. For each pair of algorithms compare the percent of observations for which they make the same prediction. Use this to define a function and then use the heatmap function to visualize the results. Hint: use the method = \"binary\" argument in the dist function.\n32. Note that each method can also produce an estimated conditional probability. Instead of majority vote, we can take the average of these estimated conditional probabilities. For most methods, we can the use the type = \"prob\" in the train function. Note that some of the methods require you to use the argument trControl=trainControl(classProbs=TRUE) when calling train. Also, these methods do not work if classes have numbers as names. Hint: change the levels like this:\n\ndat$train$y &lt;- recode_factor(dat$train$y, \"2\"=\"two\", \"7\"=\"seven\")\ndat$test$y &lt;- recode_factor(dat$test$y, \"2\"=\"two\", \"7\"=\"seven\")\n\n33. In this chapter, we illustrated a couple of machine learning algorithms on a subset of the MNIST dataset. Try fitting a model to the entire dataset."
+    "section": "\n31.10 Exercises",
+    "text": "31.10 Exercises\n1. In the exercises in Chapter 30 we saw that changing maxnodes or nodesize in the randomForest function improved our estimate. Let’s use the train function to help us pick these values. From the caret manual we see that we can’t tune the maxnodes parameter or the nodesize argument with randomForest, so we will use the Rborist package and tune the minNode argument. Use the train function to try values minNode &lt;- seq(5, 250, 25). See which value minimizes the estimated RMSE.\n2. This **dslabs* dataset includes a matrix x:\n\nlibrary(dslabs)\ndim(tissue_gene_expression$x)\n\nwith the gene expression measured on 500 genes for 189 biological samples representing seven different tissues. The tissue type is stored in y:\n\ntable(tissue_gene_expression$y)\n\nSplit the data in training and test sets, then use kNN to predict tissue type and see what accuracy you obtain. Try it for \\(k = 1, 3, \\dots, 11\\).\n3. We are going to apply LDA and QDA to the tissue_gene_expression dataset. We will start with simple examples based on this dataset and then develop a realistic example.\nCreate a dataset with just the classes cerebellum and hippocampus (two parts of the brain) and a predictor matrix with 10 randomly selected columns. Estimate the accuracy of LDA.\n\nset.seed(1993)\ntissues &lt;- c(\"cerebellum\", \"hippocampus\")\nind &lt;- which(tissue_gene_expression$y %in% tissues)\ny &lt;- droplevels(tissue_gene_expression$y[ind])\nx &lt;- tissue_gene_expression$x[ind, ]\nx &lt;- x[, sample(ncol(x), 10)]\n\n4. In this case, LDA fits two 10-dimensional normal distributions. Look at the fitted model by looking at the finalModel component of the result of train. Notice there is a component called means that includes the estimate means of both distributions. Plot the mean vectors against each other and determine which predictors (genes) appear to be driving the algorithm.\n5. Repeat exercises 3 with QDA. Does it have a higher accuracy than LDA?\n6. Are the same predictors (genes) driving the algorithm? Make a plot as in exercise 3.\n7. One thing we see in the previous plot is that the value of predictors correlate in both groups: some predictors are low in both groups while others are high in both groups. The mean value of each predictor, colMeans(x), is not informative or useful for prediction and often, for interpretation purposes, it is useful to center or scale each column. This can be achieved with the preProcessing argument in train. Re-run LDA with preProcessing = \"scale\". Note that accuracy does not change but see how it is easier to identify the predictors that differ more between groups in the plot made in exercise 4.\n8. In the previous exercises, we saw that both approaches worked well. Plot the predictor values for the two genes with the largest differences between the two groups in a scatterplot to see how they appear to follow a bivariate distribution as assumed by the LDA and QDA approaches. Color the points by the outcome.\n9. Now we are going to increase the complexity of the challenge slightly: we will consider all the tissue types.\n\nset.seed(1993)\ny &lt;- tissue_gene_expression$y\nx &lt;- tissue_gene_expression$x\nx &lt;- x[, sample(ncol(x), 10)]\n\nWhat accuracy do you get with LDA?\n10. We see that the results are slightly worse. Use the confusionMatrix function to learn what type of errors we are making.\n11. Plot an image of the centers of the seven 10-dimensional normal distributions.\n12. Make a scatterplot along with the prediction from the best fitted model.\n13. Use the rpart function to fit a classification tree to the tissue_gene_expression dataset. Use the train function to estimate the accuracy. Try out cp values of seq(0, 0.05, 0.01). Plot the accuracy to report the results of the best model.\n14. Study the confusion matrix for the best fitting classification tree. What do you observe happening for placenta?\n15. Notice that placentas are called endometrium more often than placenta. Note also that the number of placentas is just six, and that, by default, rpart requires 20 observations before splitting a node. Thus it is not possible with these parameters to have a node in which placentas are the majority. Rerun the above analysis, but this time permit rpart to split any node by using the argument control = rpart.control(minsplit = 0). Does the accuracy increase? Look at the confusion matrix again.\n16. Plot the tree from the best fitting model obtained in exercise 11.\n17. We can see that with just six genes, we are able to predict the tissue type. Now let’s see if we can do even better with a random forest. Use the train function and the rf method to train a random forest. Try out values of mtry ranging from, at least, seq(50, 200, 25). What mtry value maximizes accuracy? To permit small nodesize to grow as we did with the classification trees, use the following argument: nodesize = 1. This will take several seconds to run. If you want to test it out, try using smaller values with ntree. Set the seed to 1990.\n18. Use the function varImp on the output of train and save it to an object called imp.\n19. The rpart model we ran above produced a tree that used just six predictors. Extracting the predictor names is not straightforward, but can be done. If the output of the call to train was fit_rpart, we can extract the names like this:\n\nind &lt;- !(fit_rpart$finalModel$frame$var == \"&lt;leaf&gt;\")\ntree_terms &lt;- \n  fit_rpart$finalModel$frame$var[ind] |&gt;\n  unique() |&gt;\n  as.character()\ntree_terms\n\nWhat is the variable importance in the random forest call for these predictors? Where do they rank?\n20. Extract the top 50 predictors based on importance, take a subset of x with just these predictors and apply the function heatmap to see how these genes behave across the tissues. We will introduce the heatmap function in Chapter 32.\n21. Previously, we compared the conditional probability \\(p(\\mathbf{x})\\) give two predictors \\(\\mathbf{x} = (x_1, x_2)^\\top\\) to the fit \\(\\hat{p}(\\mathbf{x})\\) obtained with a machine learning algorithm by making image plots. The following code can be used to make these images and include a curve at the values of \\(x_1\\) and \\(x_2\\) for which the function is \\(0.5\\):\n\nplot_cond_prob &lt;- function(x_1, x_2, p){\n  data.frame(x_1 = x_1, x_2 = x_2, p = p) |&gt;\n    ggplot(aes(x_1, x_2)) +\n    geom_raster(aes(fill = p), show.legend = FALSE) +\n    stat_contour(aes(z = p), breaks = 0.5, color = \"black\") +\n    scale_fill_gradientn(colors = c(\"#F8766D\", \"white\", \"#00BFC4\"))\n}\n\nWe can see the true conditional probability for the 2 or 7 example like this:\n\nwith(mnist_27$true_p, plot_cond_prob(x_1, x_2, p))\n\nFit a kNN model and make this plot for the estimated conditional probability. Hint: Use the argument newdata = mnist27$train to obtain predictions for a grid points.\n22. Notice that, in the plot made in exercise 1, the boundary is somewhat wiggly. This is because kNN, like the basic bin smoother, does not use a kernel. To improve this we could try loess. By reading through the available models part of the caret manual, we see that we can use the gamLoess method. We need to install the gam package, if we have not done so already. We see that we have two parameters to optimize:\n\nmodelLookup(\"gamLoess\")\n#&gt;      model parameter  label forReg forClass probModel\n#&gt; 1 gamLoess      span   Span   TRUE     TRUE      TRUE\n#&gt; 2 gamLoess    degree Degree   TRUE     TRUE      TRUE\n\nUse cross-validation to pick a span between 0.15 and 0.75. Keep degree = 1. What span does cross-validation select?\n23. Show an image plot of the estimate \\(\\hat{p}(x,y)\\) resulting from the model fit in the exercise 2. How does the accuracy compare to that of kNN? Comment on the difference between the estimate obtained with kNN.\n24. Use the mnist_27 training set to build a model with several of the models available from the caret package. For example, you can try these:\n\nmodels &lt;- c(\"glm\", \"lda\",  \"naive_bayes\",  \"svmLinear\", \"gamboost\",  \n            \"gamLoess\", \"qda\", \"knn\", \"kknn\", \"loclda\", \"gam\", \"rf\", \n            \"ranger\",\"wsrf\", \"Rborist\", \"avNNet\", \"mlp\", \"monmlp\", \"gbm\", \n            \"adaboost\", \"svmRadial\", \"svmRadialCost\", \"svmRadialSigma\")\n\nWe have not explained many of these, but apply them anyway using train with all the default parameters. Keep the results in a list. You might need to install some packages. Keep in mind that you will likely get some warnings.\n25. Now that you have all the trained models in a list, use sapply or map to create a matrix of predictions for the test set. You should end up with a matrix with length(mnist_27$test$y) rows and length(models) columns.\n26. Compute accuracy for each model on the test set.\n27. Build an ensemble prediction by majority vote and compute its accuracy.\n28. Earlier we computed the accuracy of each method on the training set and noticed they varied. Which individual methods do better than the ensemble?\n29. It is tempting to remove the methods that do not perform well and re-do the ensemble. The problem with this approach is that we are using the test data to make a decision. However, we could use the accuracy estimates obtained from cross validation with the training data. Obtain these estimates and save them in an object.\n30. Now let’s only consider the methods with an estimated accuracy of 0.8 when constructing the ensemble. What is the accuracy now?\n31. Note that if two machine algorithms methods predict the same outcome, ensembling them will not change the prediction. For each pair of algorithms compare the percent of observations for which they make the same prediction. Use this to define a function and then use the heatmap function to visualize the results. Hint: use the method = \"binary\" argument in the dist function.\n32. Note that each method can also produce an estimated conditional probability. Instead of majority vote, we can take the average of these estimated conditional probabilities. For most methods, we can the use the type = \"prob\" in the train function. Note that some of the methods require you to use the argument trControl=trainControl(classProbs=TRUE) when calling train. Also, these methods do not work if classes have numbers as names. Hint: change the levels like this:\n\ndat$train$y &lt;- recode_factor(dat$train$y, \"2\"=\"two\", \"7\"=\"seven\")\ndat$test$y &lt;- recode_factor(dat$test$y, \"2\"=\"two\", \"7\"=\"seven\")\n\n33. In this chapter, we illustrated a couple of machine learning algorithms on a subset of the MNIST dataset. Try fitting a model to the entire dataset."
   },
   {
     "objectID": "ml/ml-in-practice.html#footnotes",
     "href": "ml/ml-in-practice.html#footnotes",
     "title": "31  Machine learning in practice",
     "section": "",
-    "text": "https://topepo.github.io/caret/available-models.html↩︎\nhttp://topepo.github.io/caret/available-models.html↩︎\nhttps://topepo.github.io/caret/available-models.html↩︎"
+    "text": "https://topepo.github.io/caret/available-models.html↩︎\nhttp://topepo.github.io/caret/available-models.html↩︎\nhttps://topepo.github.io/caret/available-models.html↩︎\nhttps://topepo.github.io/caret/using-your-own-model-in-train.html↩︎"
   },
   {
     "objectID": "ml/clustering.html#hierarchical-clustering",
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index e83cb39..a6460b2 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -2,166 +2,166 @@
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/index.html</loc>
-    <lastmod>2024-01-04T22:27:36.646Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.890Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/intro.html</loc>
-    <lastmod>2024-01-04T22:27:36.652Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.894Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/summaries/intro-summaries.html</loc>
-    <lastmod>2024-01-04T22:27:36.657Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.897Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/summaries/distributions.html</loc>
-    <lastmod>2024-01-04T22:27:36.680Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.922Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/summaries/robust-summaries.html</loc>
-    <lastmod>2024-01-04T22:27:36.689Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.929Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/prob/intro-to-prob.html</loc>
-    <lastmod>2024-01-04T22:27:36.692Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.933Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/prob/discrete-probability.html</loc>
-    <lastmod>2024-01-04T22:27:36.713Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.954Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/prob/continuous-probability.html</loc>
-    <lastmod>2024-01-04T22:27:36.724Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.965Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/prob/random-variables-sampling-models-clt.html</loc>
-    <lastmod>2024-01-04T22:27:36.737Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.980Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/inference/intro-inference.html</loc>
-    <lastmod>2024-01-04T22:27:36.741Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.984Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/inference/parameters-estimates.html</loc>
-    <lastmod>2024-01-04T22:27:36.749Z</lastmod>
+    <lastmod>2024-04-22T20:06:36.991Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/inference/clt.html</loc>
-    <lastmod>2024-01-04T22:27:36.757Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.000Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/inference/confidence-intervals.html</loc>
-    <lastmod>2024-01-04T22:27:36.766Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.008Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/inference/hypothesis-testing.html</loc>
-    <lastmod>2024-01-04T22:27:36.772Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.014Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/inference/bootstrap.html</loc>
-    <lastmod>2024-01-04T22:27:36.779Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.022Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/inference/models.html</loc>
-    <lastmod>2024-01-04T22:27:36.792Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.036Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/inference/bayes.html</loc>
-    <lastmod>2024-01-04T22:27:36.802Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.044Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/inference/hierarchical-models.html</loc>
-    <lastmod>2024-01-04T22:27:36.832Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.077Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/linear-models/intro-to-linear-models.html</loc>
-    <lastmod>2024-01-04T22:27:36.836Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.081Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/linear-models/regression.html</loc>
-    <lastmod>2024-01-04T22:27:36.859Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.105Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/linear-models/multivariate-regression.html</loc>
-    <lastmod>2024-01-04T22:27:36.883Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.130Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/linear-models/measurement-error-models.html</loc>
-    <lastmod>2024-01-04T22:27:36.890Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.136Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/linear-models/treatment-effect-models.html</loc>
-    <lastmod>2024-01-04T22:27:36.904Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.151Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/linear-models/association-tests.html</loc>
-    <lastmod>2024-01-04T22:27:36.918Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.165Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/linear-models/association-not-causation.html</loc>
-    <lastmod>2024-01-04T22:27:36.931Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.178Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/highdim/intro-highdim.html</loc>
-    <lastmod>2024-01-04T22:27:36.935Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.181Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/highdim/matrices-in-R.html</loc>
-    <lastmod>2024-01-04T22:27:36.950Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.196Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/highdim/linear-algebra.html</loc>
-    <lastmod>2024-01-04T22:27:36.961Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.208Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/highdim/dimension-reduction.html</loc>
-    <lastmod>2024-01-04T22:27:36.976Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.225Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/highdim/regularization.html</loc>
-    <lastmod>2024-01-04T22:27:36.994Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.243Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/highdim/matrix-factorization.html</loc>
-    <lastmod>2024-01-04T22:27:37.016Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.264Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/ml/intro-ml.html</loc>
-    <lastmod>2024-01-04T22:27:37.020Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.267Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/ml/notation-and-terminology.html</loc>
-    <lastmod>2024-01-04T22:27:37.024Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.273Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/ml/evaluation-metrics.html</loc>
-    <lastmod>2024-01-04T22:27:37.040Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.289Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/ml/conditionals.html</loc>
-    <lastmod>2024-01-04T22:27:37.045Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.294Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/ml/smoothing.html</loc>
-    <lastmod>2024-01-04T22:27:37.059Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.307Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/ml/resampling-methods.html</loc>
-    <lastmod>2024-01-04T22:27:37.071Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.319Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/ml/algorithms.html</loc>
-    <lastmod>2024-01-04T22:27:37.095Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.343Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/ml/ml-in-practice.html</loc>
-    <lastmod>2024-01-04T22:27:37.118Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.369Z</lastmod>
   </url>
   <url>
     <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/ml/clustering.html</loc>
-    <lastmod>2024-01-04T22:27:37.127Z</lastmod>
+    <lastmod>2024-04-22T20:06:37.377Z</lastmod>
   </url>
   <url>
-    <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/Advanced-Data-Science.pdf</loc>
-    <lastmod>2024-01-04T22:27:36.466Z</lastmod>
+    <loc>http://rafalab.dfci.harvard.edu/dsbook-part-2/Introduction-to-Data-Science.pdf</loc>
+    <lastmod>2024-04-22T20:06:36.682Z</lastmod>
   </url>
 </urlset>
diff --git a/docs/summaries/distributions.html b/docs/summaries/distributions.html
index cbf42fb..ecb2897 100644
--- a/docs/summaries/distributions.html
+++ b/docs/summaries/distributions.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 1&nbsp; Distributions</title>
+<title>Introduction to Data Science - 1&nbsp; Distributions</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -1218,7 +1218,7 @@ <h1 class="title"><span id="sec-summaries-distributions" class="quarto-section-i
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/docs/summaries/intro-summaries.html b/docs/summaries/intro-summaries.html
index 14c92ad..f3c8b03 100644
--- a/docs/summaries/intro-summaries.html
+++ b/docs/summaries/intro-summaries.html
@@ -8,7 +8,7 @@
 
 <meta name="author" content="Rafael A. Irizarry">
 
-<title>Advanced Data Science - Summary statistics</title>
+<title>Introduction to Data Science - Summary statistics</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -89,7 +89,7 @@
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto">
     <div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -659,7 +659,7 @@ <h1 class="title">Summary statistics</h1>
 </div> <!-- /content -->
 <footer class="footer">
   <div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     <div class="toc-actions"><div><i class="bi bi-github"></i></div><div class="action-links"><p><a href="https://github.com/rafalab/dsbook-part-2/blob/main/summaries/intro-summaries.qmd" class="toc-action">View source</a></p><p><a href="https://github.com/rafalab/dsbook-part-2/issues/new" class="toc-action">Report an issue</a></p></div></div></div>
diff --git a/docs/summaries/robust-summaries.html b/docs/summaries/robust-summaries.html
index 6fc50bd..3d7323a 100644
--- a/docs/summaries/robust-summaries.html
+++ b/docs/summaries/robust-summaries.html
@@ -5,7 +5,7 @@
 <meta name="generator" content="quarto-1.3.353">
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 <meta name="author" content="Rafael A. Irizarry">
-<title>Advanced Data Science - 2&nbsp; Robust summaries</title>
+<title>Introduction to Data Science - 2&nbsp; Robust summaries</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -109,7 +109,7 @@
 <!-- sidebar -->
   <nav id="quarto-sidebar" class="sidebar collapse collapse-horizontal sidebar-navigation floating overflow-auto"><div class="pt-lg-2 mt-2 text-left sidebar-header">
     <div class="sidebar-title mb-0 py-0">
-      <a href="../">Advanced Data Science</a> 
+      <a href="../">Introduction to Data Science</a> 
         <div class="sidebar-tools-main">
     <a href="https://github.com/rafalab/dsbook-part-2" title="Source Code" class="quarto-navigation-tool px-1" aria-label="Source Code"><i class="bi bi-github"></i></a>
   <a href="" class="quarto-reader-toggle quarto-navigation-tool px-1" onclick="window.quartoToggleReader(); return false;" title="Toggle reader mode">
@@ -527,7 +527,7 @@ <h1 class="title"><span id="sec-robust-summaries" class="quarto-section-identifi
 <span class="header-section-number">2.6</span> Exercises</h2>
 <p>We are going to use the <strong>HistData</strong> package. Load the height data set and create a vector <code>x</code>, consisting solely of the male heights used in Galton’s data on the heights of parents and their children, part of his historic research on heredity.</p>
 <div class="cell" data-layout-align="center" data-hash="robust-summaries_cache/html/unnamed-chunk-10_496c326c146d88c3c5b448202c47c020">
-<div class="sourceCode" id="cb10"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va">HistData</span><span class="op">)</span></span>
+<div class="sourceCode" id="cb10"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://friendly.github.io/HistData/">HistData</a></span><span class="op">)</span></span>
 <span><span class="va">x</span> <span class="op">&lt;-</span> <span class="va">Galton</span><span class="op">$</span><span class="va">child</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>1. Compute the average and median of these data.</p>
@@ -810,7 +810,7 @@ <h1 class="title"><span id="sec-robust-summaries" class="quarto-section-identifi
 </nav>
 </div> <!-- /content -->
 <footer class="footer"><div class="nav-footer">
-    <div class="nav-footer-left">Advanced Data Science was written by Rafael A. Irizarry</div>   
+    <div class="nav-footer-left">Introduction to Data Science was written by Rafael A. Irizarry</div>   
     <div class="nav-footer-center">
       &nbsp;
     </div>
diff --git a/highdim/dimension-reduction.qmd b/highdim/dimension-reduction.qmd
index f1b5c06..6ef1c93 100644
--- a/highdim/dimension-reduction.qmd
+++ b/highdim/dimension-reduction.qmd
@@ -519,7 +519,7 @@ illustrate_pca <- function(x, flip=1,
   image(cols, rows, nz[,rev(1:ncol(nz))],  xaxt = "n", yaxt = "n", bty = "n", xlab="",ylab="", col = colors)
   abline(h = pad+0:ncol(z)+1/2)
   lines(c(ncol(z)/2+0.5,ncol(z)/2+1/2),c(pad,pad+ncol(z))+0.5)
-  text(ncol(z)/2+0.5, pad+ncol(z)+2 , expression(bold(A^T)), font=2)
+  text(ncol(z)/2+0.5, pad+ncol(z)+2 , expression(bold(V^T)), font=2)
 }
 rafalib::mypar(1,1)
 illustrate_pca(x, flip = -1)
diff --git a/index.qmd b/index.qmd
index 3415775..3a59dbf 100644
--- a/index.qmd
+++ b/index.qmd
@@ -1,15 +1,15 @@
 # Preface {.unnumbered}
 
-This is the website for the  **Advanced Data Science**. 
+This is the website for the **Statistics and Prediction Algorithms Through Case Studies** part of **Introduction to Data Science**. 
 
-The website for **Introduction to Data Science** is [here](http://rafalab.dfci.harvard.edu/dsbook-part-1/).
+The website for the **Data Wrangling and Visualization with R** is [here](http://rafalab.dfci.harvard.edu/dsbook-part-1/).
 
 This book started out as part of the class notes used in the
  HarvardX [Data Science Series](https://www.edx.org/professional-certificate/harvardx-data-science)^[https://www.edx.org/professional-certificate/harvardx-data-science].
 
-A hardcopy version of the first edition of the book, which combined both Introduction and Advanced parts, is available from [CRC Press](https://www.routledge.com/Introduction-to-Data-Science-Data-Analysis-and-Prediction-Algorithms-with/Irizarry/p/book/9780367357986?utm_source=author&utm_medium=shared_link&utm_campaign=B043135_jm1_5ll_6rm_t081_1al_introductiontodatascienceauthorshare)^[https://www.routledge.com/Introduction-to-Data-Science-Data-Analysis-and-Prediction-Algorithms-with/Irizarry/p/book/9780367357986?utm_source=author&utm_medium=shared_link&utm_campaign=B043135_jm1_5ll_6rm_t081_1al_introductiontodatascienceauthorshare].
+A hardcopy version of the first edition of the book, which combined both parts, is available from [CRC Press](https://www.routledge.com/Introduction-to-Data-Science-Data-Analysis-and-Prediction-Algorithms-with/Irizarry/p/book/9780367357986?utm_source=author&utm_medium=shared_link&utm_campaign=B043135_jm1_5ll_6rm_t081_1al_introductiontodatascienceauthorshare)^[https://www.routledge.com/Introduction-to-Data-Science-Data-Analysis-and-Prediction-Algorithms-with/Irizarry/p/book/9780367357986?utm_source=author&utm_medium=shared_link&utm_campaign=B043135_jm1_5ll_6rm_t081_1al_introductiontodatascienceauthorshare].
 
-A free PDF of the October 24, 2019 version of the book, which combined both Introduction and Advanced parts, is available from [Leanpub](https://leanpub.com/datasciencebook)^[https://leanpub.com/datasciencebook].
+A free PDF of the October 24, 2019 version of the book, which combined both parts, is available from [Leanpub](https://leanpub.com/datasciencebook)^[https://leanpub.com/datasciencebook].
 
 The Quarto code used to generate the book is available on  [GitHub](https://github.com/rafalab/dsbook-part-2)^[https://github.com/rafalab/dsbook-part-2]. Note that, the graphical theme used for plots throughout the book can be recreated using the `ds_theme_set()` function from __dslabs__ package.
 
@@ -25,7 +25,7 @@ A special thanks to my tidyverse guru David Robinson and Amy Gill for dozens of
 This book was conceived during the teaching of several applied statistics courses, starting over fifteen years ago. The teaching assistants working with me throughout the years made important indirect contributions to this book. The latest iteration of this course is a HarvardX series coordinated by Heather Sternshein and Zofia Gajdos. We thank them for their contributions. We are also grateful to all the students whose questions and comments helped us improve the book. The courses were partially funded by NIH grant R25GM114818. We are very grateful to the National Institutes of Health for its support.
 
 A special thanks goes to all those who edited the book via GitHub pull requests or made suggestions by creating an _issue_ or sending an email:  `nickyfoto` (Huang Qiang), `desautm` (Marc-André Désautels), `michaschwab` (Michail Schwab), `alvarolarreategui` (Alvaro Larreategui), `jakevc` (Jake VanCampen), `omerta` (Guillermo Lengemann), `espinielli` (Enrico Spinielli), `asimumba`(Aaron Simumba), `braunschweig` (Maldewar), `gwierzchowski` (Grzegorz Wierzchowski), `technocrat` (Richard Careaga), `atzakas`, `defeit` (David Emerson Feit), `shiraamitchell` (Shira Mitchell),  `Nathalie-S`, `andreashandel` (Andreas Handel), `berkowitze` (Elias Berkowitz), `Dean-Webb` (Dean Webber), `mohayusuf`, `jimrothstein`, `mPloenzke` (Matthew Ploenzke), `NicholasDowand` (Nicholas Dow), `kant` (Darío Hereñú), `debbieyuster` (Debbie Yuster), `tuanchauict` (Tuan Chau), `phzeller`, `BTJ01` (BradJ), `glsnow` (Greg Snow), `mberlanda` (Mauro Berlanda), `wfan9`, `larswestvang` (Lars Westvang), `jj999` (Jan Andrejkovic), `Kriegslustig` (Luca Nils Schmid), `odahhani`, `aidanhorn` (Aidan Horn), `atraxler` (Adrienne Traxler), `alvegorova`,`wycheong` (Won Young Cheong), 
-`med-hat` (Medhat Khalil), `kengustafson`, `Yowza63`, `ryan-heslin` (Ryan Heslin), `raffaem`, `tim8west`,
+`med-hat` (Medhat Khalil), `biscotty666` (Brian Carey), `kengustafson`, `Yowza63`, `ryan-heslin` (Ryan Heslin), `raffaem`, `tim8west`,
 David D. Kane, El Mustapha El Abbassi, Vadim Zipunnikov, Anna Quaglieri, Chris Dong, Rick Schoenberg, Isabella Grabski, and Doug Snyder.
  
 
diff --git a/ml/algorithms.qmd b/ml/algorithms.qmd
index fcf4280..7eb4071 100644
--- a/ml/algorithms.qmd
+++ b/ml/algorithms.qmd
@@ -346,7 +346,7 @@ confusionMatrix(y_hat, mnist_27$test$y)$overall["Accuracy"]
 When we force this assumption, we can show mathematically that the boundary is a line, just as with logistic regression. For this reason, we call the method _linear_ discriminant analysis (LDA). Similarly, for QDA, we can show that the boundary must be a quadratic function.
 
 ```{r lda-estimate, echo = FALSE, out.width = "100%"}
-train_lda <- train(y ~ ., method = "lda", data = mnist_27$train)
+train_lda <- MASS::lda(y ~ ., data = mnist_27$train)
 
 p1 <- plot_cond_prob() + ggtitle("True conditional probability")
 
diff --git a/ml/clustering.qmd b/ml/clustering.qmd
index 2f54a5f..252d566 100644
--- a/ml/clustering.qmd
+++ b/ml/clustering.qmd
@@ -10,7 +10,7 @@ A first step in any clustering algorithm is defining a distance between observat
 
 We will construct a simple example based on movie ratings. Here we quickly construct a matrix `x` that has ratings for the 50 movies with the most ratings.
 
-```{r, cache=FALSE}
+```{r, cache=FALSE, message=FALSE, warning=FALSE}
 library(tidyverse)
 library(janitor)
 library(dslabs)
@@ -23,6 +23,7 @@ x <- x |> filter(userId %in% top_users) |>
   column_to_rownames("title") |>
   as.matrix()
 rownames(x) <- str_remove(rownames(x), ": Episode") |> str_trunc(20)
+x <- x - rowMeans(x, na.rm = TRUE)
 ```
 
 We want to use these data to find out if there are clusters of movies based on the ratings from  `r ncol(x)` movie raters. A first step is to find the distance between each pair of movies using the `dist` function: 
diff --git a/ml/ml-in-practice.qmd b/ml/ml-in-practice.qmd
index e7d5c2e..ff62f90 100644
--- a/ml/ml-in-practice.qmd
+++ b/ml/ml-in-practice.qmd
@@ -1,6 +1,6 @@
 # Machine learning in practice {#sec-ml-in-practice}
 
-Now that we have learned several methods and explored them with simple examples, we will try them out on a real example: the MNIST digits.
+Now that we have learned several methods and explored them with simple examples, we will try them out on a real example: the MNIST digits. 
 
 We can load this data using the following **dslabs** package:
 
@@ -46,6 +46,13 @@ x_test <- mnist$test$images[index,]
 y_test <- factor(mnist$test$labels[index])
 ```
 
+When fitting models to large datasets, we recommend using matrices instead of data frames, as matrix operations tend to be faster. In the **caret** package, predictor matrices must have column names to track features accurately during prediction on the test set. If the matrices lack column names, you can assign names based on their position:
+
+```{r}
+colnames(x) <- 1:ncol(mnist$train$images)
+colnames(x_test) <- colnames(x)
+```
+
 ## The caret package {#sec-caret}
 
 We have already learned about several machine learning algorithms. Many of these algorithms are implemented in R. However, they are distributed via different packages, developed by different authors, and often use different syntax. The __caret__ package tries to consolidate these differences and provide consistency. It currently includes over 200 different methods which are summarized in the __caret__ package manual^[https://topepo.github.io/caret/available-models.html]. Keep in mind that __caret__ does not include the packages needed to run each possible algorithm. To apply a machine learning method through __caret__ you still need to install the library that implement the method. The required packages for each method are described in the package manual. 
@@ -56,7 +63,7 @@ The __caret__ package also provides a function that performs cross validation fo
 
 The R functions that fit machine algorithms are all slightly different. Functions such as `lm`, `glm`, `qda`, `lda`, `knn3`, `rpart` and `randomForrest` use different syntax, have different argument names and produce objects of different types.
 
-The __caret__ `train` function lets us train different algorithms using similar syntax. So, for example, we can type:
+The __caret__ `train` function lets us train different algorithms using similar syntax. So, for example, we can type the following to train three different models:
 
 ```{r}
 #| cache: false
@@ -66,7 +73,7 @@ train_qda <- train(y ~ ., method = "qda", data = mnist_27$train)
 train_knn <- train(y ~ ., method = "knn", data = mnist_27$train)
 ```
 
-As we explain in more detail in @sec-caret-cv, the `train` function selects parameters for you using a resampling method, with boostrap as the default.
+As we explain in more detail in @sec-caret-cv, the `train` function selects parameters for you using a resampling method to estimate the MSE, with bootstrap as the default.
 
 ### The `predict` function
 
@@ -76,10 +83,10 @@ Here is an example with logistic regression:
 
 ```{r}
 fit <- glm(y ~ ., data = mnist_27$train, family = "binomial")
-p_hat <- predict(fit, mnist_27$test)
+p_hat <- predict(fit, newdata = mnist_27$test)
 ```
 
-In this case, the function is simply computing: 
+In this case, the function is simply computing 
 
 $$
 \hat{p}(\mathbf{x}) = g^{-1}\left(\hat{\beta}_0 + \hat{\beta}_1 x_1 + \hat{\beta}_2 x_2 \right) \text{ with } g(p) = \log\frac{p}{1-p} \implies g^{-1}(\mu) = \frac{1}{1-e^{-\mu}}
@@ -91,7 +98,9 @@ for the `x_1` and `x_2` in the test set `mnist_27$test`. With these estimates in
 y_hat <- factor(ifelse(p_hat > 0.5, 7, 2))
 ```
 
-However, note that `predict` does not always return objects of the same types; it depends on what type of object it is applied to. To learn about the specifics, you need to look at the help file specific for the type of fit object that is being used. The `predict` is actually a special type of function in R (called a _generic function_) that calls other functions depending on what kind of object it receives. So if `predict` receives an object coming out of the `lm` function, it will call `predict.glm`. If it receives an object coming out of `glm`, it calls `predict.qda`. If the fit is from `knn3`, it calls `predict.knn3`, and so on. These functions are similar but not exactly. You can learn more about the differences by reading the help files:
+However, note that `predict` does not always return objects of the same type; it depends on what type of object it is applied to. To learn about the specifics, you need to look at the help file specific for the type of fit object that is being used. 
+
+`predict` is actually a special type of function in R called a _generic function_. Generic functions call other functions depending on what kind of object it receives. So if `predict` receives an object coming out of the `lm` function, it will call `predict.lm`. If it receives an object coming out of `glm`, it calls `predict.glm`. If the fit is from `knn3`, it calls `predict.knn3`, and so on. These functions are similar but not exactly. You can learn more about the differences by reading the help files:
 
 ```{r, eval = FALSE}
 ?predict.glm
@@ -99,7 +108,7 @@ However, note that `predict` does not always return objects of the same types; i
 ?predict.knn3
 ```
 
-There are many other versions of `predict` and many machine learning algorithms have a `predict` function.
+There are many other versions of `predict` and many machine learning algorithms define their own `predict` function.
 
 As with `train`, the **caret** packages unifies the use of `predict` with the function `predict.train`. This function takes the output of `train` and produces prediction of categories or estimates of $p(\mathbf{x})$.
 
@@ -119,18 +128,18 @@ sapply(fits, function(fit) confusionMatrix(fit, mnist_27$test$y)$overall[["Accur
 ```
 
 
-### Cross validation {#sec-caret-cv}
+### Resampling {#sec-caret-cv}
 
-When an algorithm includes a tuning parameter, `train` automatically uses cross validation to decide among a few default values. To find out what parameter or parameters are optimized, you can read the manual ^[http://topepo.github.io/caret/available-models.html] or study the output of: 
+When an algorithm includes a tuning parameter, `train` automatically uses a resampling method to estimate MSE and decide among a few default candidate values. To find out what parameter or parameters are optimized, you can read the **caret** manual^[http://topepo.github.io/caret/available-models.html] or study the output of: 
 
 ```{r, eval=FALSE}
-getModelInfo("knn")
+modelLookup("knn")
 ```
 
-We can also use a quick lookup like this:
+To obtain all the details of how **caret** implements kNN you can use:
 
 ```{r, eval=FALSE}
-modelLookup("knn")
+getModelInfo("knn")
 ```
 
 If we run it with default values: 
@@ -145,18 +154,25 @@ you can quickly see the results of the cross validation using the `ggplot` funct
 ggplot(train_knn, highlight = TRUE)
 ```
 
-By default, the cross validation is performed by taking 25 bootstrap samples comprised of 25% of the observations. For the `kNN` method, the default is to try $k=5,7,9$. We change this using the `tuneGrid` argument. The grid of values must be supplied by a data frame with the parameter names as specified in the `modelLookup` output. 
+By default, the resampling is performed by taking 25 bootstrap samples, each comprised of 25% of the observations. 
+
+For the `kNN` method, the default is to try $k=5,7,9$. We change this using the `tuneGrid` argument. The grid of values must be supplied by a data frame with the parameter names as specified in the `modelLookup` output. 
+
+Here we present an example where we try out 38 values between 1 and 75. To do this with __caret__, we need to define a column named `k`, so we use this:  `data.frame(k = seq(1, 75, 2))`. Note that when running this code, we are fitting 38 versions of kNN to 25 bootstrapped samples. Since we are fitting $38 \times 25 = 950$ kNN models, running this code will take several seconds.
 
-Here we present an example where we try out 30 values between 9 and 67. To do this with __caret__, we need to define a column named `k`, so we use this:  `data.frame(k = seq(9, 67, 2))`. Note that when running this code, we are fitting 30 versions of kNN to 25 bootstrapped samples. Since we are fitting $30 \times 25 = 750$ kNN models, running this code will take several seconds. We set the seed because cross validation is a random procedure and we want to make sure the result here is reproducible.
 
 ```{r train-knn-plot}
-set.seed(2008)
 train_knn <- train(y ~ ., method = "knn", 
                    data = mnist_27$train,
-                   tuneGrid = data.frame(k = seq(9, 71, 2)))
+                   tuneGrid = data.frame(k = seq(1, 75, 2)))
 ggplot(train_knn, highlight = TRUE)
 ```
 
+:::{.calout-note}
+Because resampling methods are random procedures, the same code can result in different
+results. To assure reproducible results you should set the seed, as we did at the start of this chapter.
+:::
+
 To access the parameter that maximized the accuracy, you can use this:
 
 ```{r}
@@ -176,26 +192,25 @@ confusionMatrix(predict(train_knn, mnist_27$test, type = "raw"),
                 mnist_27$test$y)$overall["Accuracy"]
 ```
 
-If we want to change how we perform cross validation, we can use the `trainControl` function. We can make the code above go a bit faster by using, for example, 10-fold cross validation. This means we have 10 samples using 10% of the observations each. We accomplish this using the following code:
+Bootstrapping is not always the best approach to resampling (see @knn-in-practice, for an example). If we want to change our resampling method, we can use the `trainControl` function. For example, the code below runs 10-fold cross validation. This means we have 10 samples using 90% of the observations to train in each sample. We accomplish this using the following code:
 
 ```{r cv-10-fold-accuracy-estimate}
 control <- trainControl(method = "cv", number = 10, p = .9)
 train_knn_cv <- train(y ~ ., method = "knn", 
                    data = mnist_27$train,
-                   tuneGrid = data.frame(k = seq(9, 71, 2)),
+                   tuneGrid = data.frame(k = seq(1, 71, 2)),
                    trControl = control)
-ggplot(train_knn_cv, highlight = TRUE)
 ```
 
-We observe that the accuracy estimates are more variable, which is expected since we changed the number of samples used to estimate accuracy.
-
-Note that `results` component of the `train` output includes several summary statistics related to the variability of the cross validation estimates:
+:::{.callout-note}
+The `results` component of the `train` output includes several summary statistics related to the variability of the cross validation estimates:
 
 ```{r}
 names(train_knn$results)
 ```
 
-You can learn many more details about the **caret** package, from the manual ^[https://topepo.github.io/caret/available-models.html].
+You can learn many more details about the **caret** package, from the manual^[https://topepo.github.io/caret/available-models.html].
+:::
 
 ## Preprocessing
 
@@ -230,94 +245,127 @@ rafalib::mypar()
 image(matrix(1:784 %in% nzv, 28, 28))
 ```
 
-So we end up keeping this number of columns:
+So we end up removing 
 
 ```{r}
-col_index <- setdiff(1:ncol(x), nzv)
-length(col_index)
+length(nzv)
 ```
 
-Now we are ready to fit some models. Before we start, we need to add column names to the feature matrices as these are required by **caret**:
+predictors.
+
+The **caret** package features the `preProcess` function, which allows users to establish a predefined set of preprocessing operations based on a training set. This function is designed to apply these operations to new datasets without recalculating anything on the test set, ensuring that all preprocessing steps are consistent and derived solely from the training data.
+
+Below is an example demonstrating how to remove predictors with near-zero variance and then center the remaining predictors:
 
 ```{r}
-colnames(x) <- 1:ncol(mnist$train$images)
-colnames(x_test) <- colnames(x)
+pp <- preProcess(x, method = c("nzv", "center"))
+centered_subsetted_x_test <- predict(pp, newdata = x_test)
+dim(centered_subsetted_x_test)
+```
+
+Additionally, the `train` function in caret includes a `preProcess` argument that allows users to specify which preprocessing steps to apply automatically during model training. We will explore this functionality further in the context of a k-Nearest Neighbors model @knn-in-practice.
+
+## Parallelization 
+
+During cross-validation or bootstrapping, the process of fitting models to different samples or using varying parameters can be performed independently. Imagine you are fitting 100 models; if you had access to 100 computers, you could theoretically speed up the process by a factor of 100 by fitting each model on a separate computer and then aggregating the results. In reality, most modern computers, including many personal computers, are equipped with multiple processors that allow for such parallel execution. This method, known as parallelization, leverages these multiple processors to conduct several computational tasks simultaneously, significantly accelerating the model training process. By distributing the workload across different processors, parallelization makes it feasible to manage large datasets and complex modeling procedures efficiently.
+
+
+The **caret** package is set up to run in parallel but you have to let R know that you are want to parallelize your work. To do this we can use the **doParallel** package:
+ 
+```{r}
+library(doParallel)
+nc <- detectCores() - 1
+cl <- makeCluster(nc) # it is convention to leave 1 core for OS
+registerDoParallel(cl)
 ```
 
-## k-nearest neighbors
+If you do use parallelization, make sure to let R know you are done with the following lines of code:
+
+```{r}
+#| eval: false
+stopCluster(cl)
+stopImplicitCluster()
+```
+
+:::{.callout-warning}
+When parallelizing tasks across multiple processors, it's important to consider the risk of running out of memory. Each processor might require a copy of the data or substantial portions of it, which can multiply overall memory demands. This is especially challenging if the data or models are large.
+:::
+
+## k-nearest neighbors {#knn-in-practice}
 
 :::{.callout-warning}
 Before starting this section, note that the first two calls to the `train` function in the code below can take several hours to run. This is a common challenge when training machine learning algorithms since we have to run the algorithm for each cross validation split and each set of tuning parameters being considered. In the next section, we will provide some suggestions on how to predict the duration of the process and ways to reduce. 
 :::
 
+As we will see soon, the optimal $k$ for the MNIST data is between 1 and 7. For small values of $k$, bootstrapping can be problematic for estimating MSE for a k-Nearest Neighbors (kNN). This is because bootstrapping involves sampling with replacement from the original dataset which implies the closest neighbor will often appear twice but considered two independent observations by kNN. This is a unrealistic scenario that can distort the estimate MSE when, for example, $k=3$. We therefore us cross-validation to estimate our MSE.
+
 The first step is to optimize for $k$.
 
-```{r mnist-knn-fit, eval=FALSE}
-train_knn <- train(x[ ,col_index], y, 
-                   method = "knn", 
-                   tuneGrid = data.frame(k = seq(3, 13, 2)))
+```{r mnist-knn-fit}
+train_knn <- train(x, y, method = "knn", 
+                   preProcess = "nzv",
+                   trControl = trainControl("cv", number = 20, p = 0.95),
+                   tuneGrid = data.frame(k = seq(1, 7, 2)))
 ```
 
-Once we optimize our algorithm, we can fit it to the entire dataset:
+Once we optimize our algorithm, the `predict` function defaults to using the best performing algorithm fit with the entire training data:
 
-```{r, eval = FALSE}
-fit_knn <- knn3(x[, col_index], y,  k = train_knn$bestTune)
-```
-
-```{r, echo = FALSE}
-fit_knn <- knn3(x[, col_index], y,  k = 7)
+```{r}
+y_hat_knn <- predict(train_knn, x_test, type = "raw")
 ```
 
-We achieve a high accuracy:
+We achieve relatively high accuracy:
 
 ```{r}
-y_hat_knn <- predict(fit_knn, x_test[, col_index], type = "class")
 confusionMatrix(y_hat_knn, factor(y_test))$overall["Accuracy"]
 ```
 
 ### Dimension reduction with PCA
 
-An alternative to removing low variance columns directly is to use dimension reduction on the feature matrix before applying the algorithms. It is important that we not use the test set when finding the PCs nor any summary of the data, as this could result in overtraining. So we start by applying `prcomp` to the training data:
+An alternative to removing low variance columns directly is to use dimension reduction on the feature matrix before applying the algorithms. 
 
 ```{r}
-col_means <- colMeans(x)
-pca <- prcomp(sweep(x, 2, col_means), center = FALSE)
+pca <- prcomp(x)
 ```
 
-Next, we run kNN on just a small number of dimensions. We try 36 dimensions since this explains about 80% of the data. 
+We can actually explain, say, 75% of the variability in the predictors with a small number of dimensions:
 
 ```{r}
-#| eval: false
-
-k <- 36
-x_train <- pca$x[,1:k]
-train_knn <- train(x_train, y, 
-                   method = "knn", 
-                   tuneGrid = data.frame(k = seq(3, 13, 2)))
-fit_knn_pca <- knn3(x_train, y, k = train_knn$bestTune)
+p <- which(cumsum(pca$sdev^2)/sum(pca$sdev^2) >= 0.75)[1]
 ```
 
-```{r}
-#| echo: false
+We can then re-run our algorithm on these `r p` features:
 
-k <- 36
-fit_knn_pca <- knn3(pca$x[,1:k], y, k = 7)
+```{r}
+fit_knn_pca <- knn3(pca$x[,1:p], y, k = train_knn$bestTune)
 ```
 
-Now we apply the transformation we learned with the training data to the test data, reduce the dimension, and then run predict. Note that we used the rotation and column means estimated from the training data.
+When predicting, it is important that we not use the test set when finding the PCs nor any summary of the data, as this could result in overtraining. We therefrom compute the averages needed for centering and the rotation on the training set:
 
 ```{r}
-newdata <-  sweep(x_test, 2, col_means) %*% pca$rotation[,1:k]
+newdata <-  sweep(x_test, 2, colMeans(x)) %*% pca$rotation[,1:p]
 y_hat_knn_pca <- predict(fit_knn_pca, newdata, type = "class")
+```
+
+We obtain similar accuracy, while using only `r p` dimensions:
+
+```{r}
 confusionMatrix(y_hat_knn_pca, factor(y_test))$overall["Accuracy"]
 ```
 
-We obtain an improvement in accuracy, while using only 36 dimensions. 
+In this example, we used the $k$ optimized for the raw data, not the principal components. Note that to obtain an unbiased MSE estimate we have to recompute the PCA for each cross-validation sample and apply to the validation set. Because the `train` function includes PCA as one of the available preprocessing operations we can achieve this with this modification of the code above:
 
-:::{.callout-note}
-Remember the entire algorithm needs to be developed on the training data. 
-This is why the column means, subtracted from test set columns, and the rotation applied to obtained PCs, were both computed using the train data. 
-:::
+```{r}
+train_knn_pca <- train(x, y, method = "knn", 
+                       preProcess = c("nzv", "pca"),
+                       trControl = trainControl("cv", number = 20, p = 0.95,
+                                                preProcOptions = list(pcaComp = p)),
+                       tuneGrid = data.frame(k = seq(1, 7, 2)))
+y_hat_knn_pca <- predict(train_knn_pca, x_test, type = "raw")
+confusionMatrix(y_hat_knn_pca, factor(y_test))$overall["Accuracy"]
+```
+
+A limitation of this approach is that we don't get to optimize the number of PCs used in the analysis. To do this we need to write our own method. The **caret** manual^[https://topepo.github.io/caret/using-your-own-model-in-train.html] describe how to do this.
 
 
 ## Random Forest
@@ -326,35 +374,32 @@ With the random forest algorithm several parameters can be optimized, but the ma
 
 ```{r, eval=FALSE}
 library(randomForest)
-grid <- data.frame(mtry = seq(3, 24, 3))
-train_rf <-  train(x[, col_index], y, 
-                   method = "rf", 
-                   tuneGrid = grid)
+train_rf <- train(x, y, method = "rf", 
+                  preProcess = "nzv",
+                  tuneGrid = data.frame(mtry = seq(5, 15)))
+y_hat_rf <- predict(train_rf, x_test, type = "raw")
 ```
 
 Now that we have optimized our algorithm, we are ready to fit our final model:
 
 ```{r, eval=FALSE}
-fit_rf <- randomForest(x[, col_index], y, mtry = train_rf$bestTune$mtry)
+y_hat_rf <- predict(train_rf, x_test, type = "raw")
 ```
 
 ```{r}
 #| echo: false
 #| cache: false
 #| message: false
-library(randomForest)
-```
-
-```{r, echo=FALSE}
 ## we hard wire to make book compilation faster.
 ## we ran previous code once to determine 9 was the best
-fit_rf <- randomForest(x[, col_index], y,  mtry = 9)
+nzv <- nearZeroVar(x)
+fit_rf <- randomForest(x[, -nzv], y,  mtry = 9)
+y_hat_rf <- predict(fit_rf, x_test[,-nzv])
 ```
 
 As with kNN, we also achieve high accuracy:
 
 ```{r}
-y_hat_rf <- predict(fit_rf, x_test[ ,col_index])
 confusionMatrix(y_hat_rf, y_test)$overall["Accuracy"]
 ```
 
@@ -365,25 +410,13 @@ By optimizing some of the other algorithm parameters, we can achieve even higher
 The default method for estimating accuracy used by the `train` function is to test prediction on 25 bootstrap samples. This can result in long compute times. For example, if we are considering several values, say 10, of the tuning parameters, we will fit the algorithm 250 times. We can use the `system.time` function to estimate how long it takes to run the algorithm once:
 
 ```{r}
-system.time({fit_rf <- randomForest(x[, col_index], y,  mtry = 9)})
+nzv <- nearZeroVar(x)
+system.time({fit_rf <- randomForest(x[, -nzv], y,  mtry = 9)})
 ```
 
 and use this to estimate the total time for the 250 iterations. In this case it will be several hours.
 
-One way to reduce run time is to use k-fold cross validation with a smaller number of test sets. A popular choice is leaving out 10 test sets with 10% of the data:
-
-```{r mnist-knn-fit-control, eval=FALSE}
-control <- trainControl(method = "cv", number = 10, p = .9)
-```
-
-and re-running the `train` function with this choice specified via the `trControl` argument:
-
-```{r, eval=FALSE}
-train_rf <-  train(x[, col_index], y, 
-                   method = "rf", 
-                   tuneGrid = grid,
-                   trControl = control)
-```
+One way to reduce run time is to use k-fold cross validation with a smaller number of test sets. A popular choice is leaving out 5 test sets with 20% of the data. To use this we set the `trControl` argument in train to `trainControl(method = "cv", number = 5, p = .8)`.
 
 For random forest, we can also speed up the training step by running less trees per fit. After running the algorithm once, we can use the plot function to see how the error rate changes as the number of trees grows. 
 
@@ -407,14 +440,14 @@ We can see which features are being used most by plotting an image:
 
 ```{r eval=FALSE}
 mat <- rep(0, ncol(x))
-mat[col_index] <- imp
+mat[-nzv] <- imp
 image(matrix(mat, 28, 28))
 ```
 
 ```{r importance-image, echo=FALSE, fig.width = 4, fig.height = 4, out.width="50%"}
 rafalib::mypar()
 mat <- rep(0, ncol(x))
-mat[col_index] <- imp
+mat[-nzv] <- imp
 image(matrix(mat, 28, 28))
 ```
 
@@ -423,7 +456,7 @@ image(matrix(mat, 28, 28))
 An important part of data analysis is visualizing results to determine why we are failing. How we do this depends on the application. Below we show the images of digits for which we made an incorrect prediction. Here are some errors for the random forest:
 
 ```{r rf-images,, echo=FALSE, out.width="100%", fig.width=6, fig.height=1.65}
-p_max <- predict(fit_rf, x_test[,col_index], type = "prob") 
+p_max <- predict(fit_rf, x_test[,-nzv], type = "prob") 
 p_max <- p_max / rowSums(p_max)
 p_max <- apply(p_max, 1, max)
 
@@ -449,9 +482,9 @@ In machine learning, one can usually greatly improve the final results by combin
 Here is a simple example where we compute new class probabilities by taking the average of random forest and kNN. We can see that the accuracy improves:
 
 ```{r}
-p_rf <- predict(fit_rf, x_test[,col_index], type = "prob")  
+p_rf <- predict(fit_rf, x_test[,-nzv], type = "prob")  
 p_rf <- p_rf / rowSums(p_rf)
-p_knn_pca  <- predict(fit_knn_pca, newdata)
+p_knn_pca  <- predict(train_knn_pca, x_test, type = "prob")
 p <- (p_rf + p_knn_pca)/2
 y_pred <- factor(apply(p, 1, which.max) - 1)
 confusionMatrix(y_pred, y_test)$overall["Accuracy"]
@@ -459,6 +492,12 @@ confusionMatrix(y_pred, y_test)$overall["Accuracy"]
 
 We have just built an ensemble with just two algorithms. By combing more similarly performing, but uncorrelated, algorithms we can improve accuracy further.
 
+```{r}
+#| echo: false
+stopCluster(cl)
+stopImplicitCluster()
+```
+
 ## Exercises
 
 1\. In the exercises in @sec-example-alogirhms we saw that changing `maxnodes` or `nodesize` in the `randomForest` function improved our estimate. Let's use the `train` function to help us pick these values. From the **caret** manual we see that we can't tune the `maxnodes` parameter or the `nodesize` argument with `randomForest`, so we will use the **Rborist** package and tune the `minNode` argument. Use the `train` function to try values `minNode <- seq(5, 250, 25)`. See which value minimizes the estimated RMSE.
diff --git a/prob/discrete-probability.qmd b/prob/discrete-probability.qmd
index 6421086..2049430 100644
--- a/prob/discrete-probability.qmd
+++ b/prob/discrete-probability.qmd
@@ -369,8 +369,8 @@ hand
 and then check if one card is an Ace and the other is a face card or a 10. Going forward, we include 10 when we refer to a *face card*. Now, we need to check both possibilities:
 
 ```{r}
-(hands[1] %in% aces & hands[2] %in% facecard) | 
-  (hands[2] %in% aces & hands[1] %in% facecard)
+(hand[1] %in% aces & hand[2] %in% facecard) | 
+  (hand[2] %in% aces & hand[1] %in% facecard)
 ```
 
 If we repeat this 10,000 times, we get a very good approximation of the probability of a Natural 21.