DS105-Intermediate-Statistics/DS105-L9-Power-Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1d71ce4b",
   "metadata": {},
   "source": [
    "# DS105 Intermediate Statistics : Lesson Nine - Power Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecb308c1",
   "metadata": {},
   "source": [
    "### Table of Contents <a class=\"anchor\" id=\"DS105L9_toc\"></a>\n",
    "\n",
    "* [Table of Contents](#DS105L9_toc)\n",
    "    * [Page 1 - Introduction to Power Analysis](#DS105L9_page_1)\n",
    "    * [Page 2 - Type I and Type II Error Review](#DS105L9_page_2)\n",
    "    * [Page 3 - Installing G*Power for Windows](#DS105L9_page_3)\n",
    "    * [Page 4 - Installing G*Power for Mac / Linux](#DS105L9_page_4)\n",
    "    * [Page 5 - Using G*Power](#DS105L9_page_5)\n",
    "    * [Page 6 - Power Analysis for t Tests](#DS105L9_page_6)\n",
    "    * [Page 7 - Power Analysis for F Tests](#DS105L9_page_7)\n",
    "    * [Page 8 - Between-Subjects Power Analysis Variants](#DS105L9_page_8)\n",
    "    * [Page 9 - Input Parameters for Repeated Measures Designs](#DS105L9_page_9)\n",
    "    * [Page 10 - Key Terms](#DS105L9_page_10)\n",
    "    * [Page 11 - Lesson 9 Hands-On](#DS105L9_page_11)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fe6f92d",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 1 - Introduction to Power Analysis<a class=\"anchor\" id=\"DS105L9_page_1\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ac6a9697",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/388343721\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7f2a944e7690>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Introduction to Power Analysis\n",
    "VimeoVideo('388343721', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6bf806f",
   "metadata": {},
   "source": [
    "The transcript for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L09overview.zip)**.\n",
    "\n",
    "# Introduction \n",
    "\n",
    "In this lesson, you will brush up on your Type I and Type II error rates, and learn how they interact with power, or the ability to detect effects.  You'll learn about the different types of power, the typical values for power, and then you'll install a program called G*Power that will allow you to calculate power quickly and easily at the click of a button! \n",
    "\n",
    "By the end of this lesson, you should be able to: \n",
    "\n",
    "* Understand power\n",
    "* Install G*Power\n",
    "* Calculate power for *t*-tests, *F*-family tests, and regressions\n",
    "\n",
    "This lesson will culminate in a hands-on that will test your knowledge of all the statistics you have learned and will require you to calculate power for those statistics you have identified.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15493858",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 2 - Type I and Type II Error Review<a class=\"anchor\" id=\"DS105L9_page_2\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a149d4d",
   "metadata": {},
   "source": [
    "# Type I and Type II Error Review\n",
    "\n",
    "You have been working with hypothesis testing for a while, so it's time to review Type I and Type II error rates, and then build upon those concepts. *Type I Error* is when you reject the null hypothesis when you shouldn't have.  This means you reported finding something in your data when there was really nothing.  *Type II Error* is when you accept the null hypothesis when you shouldn't have. This means that you reported finding zilch in your data, but really, there was something interesting of note. \n",
    "\n",
    "![A table has two columns and two rows. The column headings are labeled reject null and accept null. The row headings are labeled null true and null false. The row entries are as follows. Row 1, Type I alpha, correct 1 alpha. Row 2, correct, type II.](Media/power1.png)\n",
    "\n",
    "---\n",
    "\n",
    "# What is Power? \n",
    "\n",
    "Type I error is also known as *alpha*, while Type II error is also known as *beta*. If you've gotten the correct answer by accepting the null correctly, then this is one minus alpha.  But if you've gotten to the correct answer by rejecting the null correctly, then this is conceptually one minus beta, which is also known as *power*.  Power is important, because it determines whether you have enough data to detect an effect.  Just like your phone won't charge without power, a statistical test won't find anything if it doesn't have enough power.  Said another way, power is the probability of rejecting the null hypothesis when the null hypothesis is false (correct). \n",
    "\n",
    "Here are some things to remember about power: \n",
    "\n",
    "* As the *n*, or your sample size, increases, so does power.\n",
    "* As the standard deviation of the population increases, the power decreases. \n",
    "* Power is greater in a one-tailed test than a two-tailed test.\n",
    "\n",
    "---\n",
    "\n",
    "## Types of Power\n",
    "\n",
    "* **A priori power:** Examine your power ahead of time, so you can gather enough data to get a satisfactory result with your statistical analysis.\n",
    "* **Post hoc / observed power:** Examine your power after the fact, to see if your conclusions are valid.\n",
    "\n",
    "---\n",
    "\n",
    "## Typical Power Value\n",
    "\n",
    "Just like you can assume that alpha, your significance level benchmark, will always be .05 unless otherwise stated, it is typically accepted that power will be .8 unless otherwise stated.  This means that you have an 80% chance of detecting an effect if it's there. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56b64e9b",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 3 - Installing G*Power for Windows<a class=\"anchor\" id=\"DS105L9_page_3\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff80c30d",
   "metadata": {},
   "source": [
    "# Installing G*Power for Windows\n",
    "\n",
    "These directions are for how to install G*Power on a Windows computer.  If you have a Mac or a Linux machine, please proceed to the next page.\n",
    "\n",
    "In order to calculate a priori power, you will use a program called G*Power. It is **[available for free here](https://www.gpower.hhu.de/)**.  The only thing that the creators of this program ask of you is that you reference their publications in any formal presentations or reports: \n",
    "\n",
    "Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.\n",
    "\n",
    "Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160.\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>For those of you who are suffering from insomnia, or really, really love statistics, those publications are available for download on the website above as well. </p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "Scroll down to the ```Download``` section, and then choose the Windows download option.\n",
    "\n",
    "![A screen displays the terms of use for downloading G asterisk power. The terms of use are as follows. 1. G asterisk Power is free for everyone. Commercial distribution is strictly prohibited. 2. G asterisk Power is distributed from this website. If you wish to distribute G asterisk Power in some other way, then you need to seek permission from the authors. Please send us an e-mail in which you specify how and for what purpose you intend to distribute G asterisk Power. 3. You may use screenshots of G asterisk Power without asking for permission. 4. Considerable effort has been put into program development and evaluation, but there is no warranty whatsoever. Two links are provided to download the G asterisk Power for windows and Mac OS.](Media/power3.png)\n",
    "\n",
    "Once you do so, you will then need to open the zip file.  It should look something like this: \n",
    "\n",
    "![A snapshot of a window explorer displays the files in the folder for G asterisk Power. The files are displayed in a format as follows. filename, type, compressed size, protected, size, ratio, and date modified.](Media/power4.png)\n",
    "\n",
    "Go ahead and click on the ```GPowerSetup``` Windows Installer option.  When this pops up, you'll then want to click ```Next```.\n",
    "\n",
    "![A dialog box is labeled G asterisk Power 3.1.9.4 on the title bar. A welcome message reads welcome to the G asterisk Power 3.1.9.4 setup wizard. It also displays a message that reads, the installer will guide you through the steps required to install G asterisk Power 3.1.9.4 on your computer. Cancel, next, and back buttons are placed at the bottom of the window. The back button is disabled. The screen also displays a warning message.](Media/power5.png)\n",
    "\n",
    "Then you will see this screen, which has some choices about where and for whom to install this program. Typically the defaults will work just fine for you. So click ```Next``` again.\n",
    "\n",
    "![A wizard is labeled G asterisk Power 3.1.9.4 on the title bar. This wizard is for selecting installation folder. A text field labeled folder is placed and this text field depicts the location of the folder. Two buttons labeled browse and disk cost are placed after the text field. Two radio buttons labeled everyone and just me are placed at the bottom of the wizard. Everyone radio button is selected. Cancel, back, and next buttons are placed at the bottom of the wizard.](Media/power6.png)\n",
    "\n",
    "And then ```Next``` twice more!\n",
    "\n",
    "You may have to give your computer permission to install G*Power as well, so if a flashing question about that comes up, make sure you allow it.\n",
    "\n",
    "And that's all there is to it! You should now have G*Power installed on your computer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59339cb5",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 4 - Installing G*Power for Mac / Linux<a class=\"anchor\" id=\"DS105L9_page_4\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31c6d7ad",
   "metadata": {},
   "source": [
    "# Installing G*Power for Mac / Linux\n",
    "\n",
    "These directions are for how to install G*Power on a Mac / Linux computer.  If you have a Windows machine, please proceed to the next page.\n",
    "\n",
    "In order to calculate a priori power, you will use a program called G*Power. It is **[available for free here](https://www.gpower.hhu.de/)**.  The only thing that the creators of this program ask of you is that you reference their publications in any formal presentations or reports: \n",
    "\n",
    "Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.\n",
    "\n",
    "Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160.\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>For those of you who are suffering from insomnia, or really, really love statistics, those publications are available for download on the website above as well. </p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "Scroll down to the ```Download``` section, and then choose the Mac OS download option.\n",
    "\n",
    "First, your computer may ask you if you're sure you want to open it. Yes, you do!\n",
    "\n",
    "![A prompt box has a question that reads, are you sure you want to open it? The box has three buttons labeled cancel, show web page, and open are placed at the bottom of the box.](Media/power22.png)\n",
    "\n",
    "After you download it, you will need to unzip it so that it no longer looks like a file, but now has an application to click on: \n",
    "\n",
    "![A window is labeled downloads on the title bar. The left panel of the window shows a list of locations for the favorites group. The downloads option is selected. The right panel lists the data in three columns labeled name, size, and kind. The entry labeled G asterisk power, 5.5 MB, and application is selected.](Media/power23.png)\n",
    "\n",
    "And that's all there is to it! You should now have G*Power installed on your computer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50dd29c1",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 5 - Using G*Power<a class=\"anchor\" id=\"DS105L9_page_5\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e8d4f0a",
   "metadata": {},
   "source": [
    "# Using G*Power\n",
    "\n",
    "Although it has a plethora of capabilities, you will mainly use G*Power for determining how large a sample size you need to detect effects.  This is probably most important if / when you are collecting your own data, and you need to determine a target for which to shoot.  However, power is an important thing to pay attention to, even if you aren't collecting your own data.  It's important to make sure you have enough data from your source, otherwise, you risk not being able to actually find significant differences when there may be - if only you had more data! No one wants to conclude that a new method isn't working, when really it is, you just didn't have enough data to determine it's validity yet.\n",
    "\n",
    "---\n",
    "\n",
    "## Protocol of Power Analyses\n",
    "\n",
    "At the very top of your G*Power application, you will see that there are two tabs.  You will always want to use the ```Protocol of power analyses``` tab, so ensure you're in that section. \n",
    "\n",
    "![A window is labeled G asterisk Power 3.1.9.4 on the title bar. The next bar has six menus labeled file, edit, view, tests, calculator, and help. The next bar has two tabs labeled central and non central distributions and protocol of power analyses. The protocol of power analyses bar is selected.](Media/power8.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Type of Power Analysis\n",
    "\n",
    "Under the ```Type of power analysis``` heading, you will find these menu options: \n",
    "\n",
    "![A box labeled type of power analysis lists six options in which the second option is selected. The selected option reads, a priori: compute required sample size - given alpha, power, and effect size.](Media/power9.png)\n",
    "\n",
    "The one you will make the most use of is the ```A priori``` option, but you may also utilize the ```Post hoc``` option.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73e6f1eb",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 6 - Power Analysis for t Tests<a class=\"anchor\" id=\"DS105L9_page_6\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0911504f",
   "metadata": {},
   "source": [
    "# Power Analysis for t Tests\n",
    "\n",
    "There are a wide variety of different tests in the *t*-test family. You will select *t* tests from the drop down of ```Test family``` on the left, and then a list of the different tests will populate in the ```Statistical test``` menu on the right. There are many more tests in the *t*-test family than you might have realized or learned; nor do you need to learn all of them.  \n",
    "\n",
    "![A window has two panels. The left panel displays two dropdown list boxes labeled test family and type of power analysis. A box labeled determine is placed at the bottom of the left panel. The right panel is labeled statistical test on top of a list from which the sixth option is selected. The sixth option reads, means: difference between two dependent means open bracket matched pairs close bracket.](Media/power10.png)\n",
    "\n",
    "The ones that align with what you have learned for *t*-tests start with ```Means```:\n",
    "\n",
    "* **Means: Difference between two dependent means (matched pairs):** Dependent *t*-test\n",
    "* **Means: Difference between two independent means (two groups):** Independent *t*-test\n",
    "* **Means: Difference from constant (one sample case):** Single sample *t*-test\n",
    "\n",
    "You will also find the information for a simple linear regression (only one IV) under the name ```Linear bivariate regression: One group, size of slope```. \n",
    "\n",
    "All of the options that have in them ```Wilcoxon``` are *t* tests that can be done when you do not meet the assumption of normality for a regular *t*-test. \n",
    "\n",
    "---\n",
    "\n",
    "## Input Parameters\n",
    "\n",
    "Once you have chosen the appropriate test, you will then need to define input parameters.  These are the specifications about the analysis you'd like to run. For instance, if you'd like to run an independent *t*-test and have selected the option to do so, you will need to provide the following input parameters, as shown below:\n",
    "\n",
    "* Tail(s) for your test\n",
    "* Effect size using Cohen's D\n",
    "* Alpha level\n",
    "* Power level\n",
    "* Ratio of sample size per group\n",
    "\n",
    "![A window displays three dropdown list boxes labeled test family, statistical test, and type of power analysis. A few text fields are placed on the left and right side of the box labeled input parameters and output parameters. The input parameters field has five fields labeled tail, effect size d, alpha err prob, power of 1, minus beta err prob, and allocation ration N2 by N1. The output parameters field has seven text fields.](Media/power11.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Tail(s)\n",
    "\n",
    "Although using a one-tailed test will provide you with slightly more power, two-tailed tests are used much more frequently.  This is because they require less prior knowledge (you can just guess that your data differs, not in which direction it differs), and they are a little more practical for real-world data science.\n",
    "\n",
    "---\n",
    "\n",
    "### Effect Size\n",
    "\n",
    "You have been briefly introduced to effect size using Cohen's D for *t*-tests.  The important thing to remember about effect size is that they are a better measure of your statistical test than *p* value because they are not based on sample size.  The *p* value is inherently flawed, because it will get smaller the more data you add! By the time you get to a dataset of a thousand or more, you are almost guaranteed to find significance sometimes because of the calculations involved! \n",
    "\n",
    "You'll want to plug in a number here for effect size that makes sense for the information you want to analyze.  The standard is typically a medium effect size, but you may want to have enough data to hit someone over the head with your results, which would require a large effect size, or you may be doing something where even a small effect makes a difference - in healthcare, for instance, where even the very small preventive effects of one aspirin a day can help prevent heart attacks! \n",
    "\n",
    "If you have forgotten what makes a small, medium, or large effect size for Cohen's D, you're in luck - G*Power has your back! Simply hover over the ```Effect size d``` input box and you will get a reminder: \n",
    "\n",
    "![A window displays three dropdown list boxes labeled test family, statistical test, and type of power analysis. A few text fields are placed on the left and right side of the box labeled input parameters and output parameters. The input parameters field has five fields labeled tail, effect size d, alpha err prob, power of 1, minus beta err prob, and allocation ration N2 by N1. 0.5 is entered on the effect size d. A popup message indicating the effect size d box and the message reads, effect size conventions d equals .20 minus small, d equals .50 minus medium, d equals .80 minus large.](Media/power12.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Alpha\n",
    "\n",
    "Just like with anything else, your alpha level will typically be .05 unless otherwise stated or decided. G*Power's default is .05, so you won't have to worry about it.\n",
    "\n",
    "---\n",
    "\n",
    "### Power\n",
    "\n",
    "Generally, you will go with a standard power level of .8.  You will need to change this manually, as G*Power sets the default as .95, which is pretty high. \n",
    "\n",
    "---\n",
    "\n",
    "### Allocation Ratio\n",
    "\n",
    "The last input parameter is for the ratio of your sample sizes between groups in your independent variable. If they are relatively equal, then you can leave the allocation ratio as the default of 1.  But if you anticipate having very unequal sample sizes, you'll need to change it, so that the ratio mimics the ratio in your data. For instance, if you are comparing differences between men and women in data science, you know that there are far more men than women, so you might want to consider changing this ratio to 2, meaning you expect to collect data from only one women to every two men.\n",
    "\n",
    "---\n",
    "\n",
    "## Output\n",
    "\n",
    "The output you receive from your power analysis will show up in two places: the top section, and the right hand side in the bottom under ```Output Parameters```.  When doing an ```A priori``` test, the most important output you can receive is the sample size.  It typically will provide you with both the total sample size needed and the sample size needed for each group, in case you haven't chosen to have them evenly distributed with an allocation ratio of 1.\n",
    "\n",
    "![A window is labeled G asterisk Power 3.1.9.4 on the title bar. The next bar has six menus labeled file, edit, view, tests, calculator, and help. The next bar has two tabs labeled central and non central distributions and protocol of power analyses. The protocol of power analyses bar is selected. The screen displays the details of analysis, input, and output. Three dropdown list boxes labeled test family, statistical test, and type of power analysis. A few text fields are placed on the left and right side of the box labeled input parameters and output parameters. The input parameters field has five fields labeled tail, effect size d, alpha err prob, power of 1, minus beta err prob, and allocation ration N2 by N1. The output parameters field has seven text fields that are labeled noncentrality parameter lambda, critical t, Df, sample size group 1, sample size group 2, total sample size, and actual power. Two buttons labeled X-Y plot for a range of values and calculate are placed at the bottom.](Media/power13.png)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "075fa063",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 7 - Power Analysis for F Tests<a class=\"anchor\" id=\"DS105L9_page_7\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb026915",
   "metadata": {},
   "source": [
    "# Power Analysis for F Tests\n",
    "\n",
    "There are a myriad of ways in which the *F* distribution gets used, including, but not limited to: ANOVAs, ANCOVAs, MANOVAs, MANCOVAs, and multiple regression! It really gets around! When you choose the ```F tests``` option under the ```Test family``` drop down, you will get this list: \n",
    "\n",
    "![A window has two panels. The left panel displays two dropdown list boxes labeled test family and type of power analysis. A box labeled determine is placed at the bottom of the left panel. The right panel is labeled statistical test on top of a list from which the first option is selected. The first option reads, ANCOVA: fixed effects, main effects and interactions.](Media/power14.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Input Parameters for Between-Subjects Designs\n",
    "\n",
    "Take a look at the ```ANOVA: Fixed effects, special, main effects, and interactions```. There are five input parameters here that you will go over: \n",
    "\n",
    "* Effect size f\n",
    "* alpha err prob\n",
    "* Power\n",
    "* Numerator df\n",
    "* Number of groups \n",
    "\n",
    "---\n",
    "\n",
    "### Effect Size f\n",
    "\n",
    "Although there are many different effect sizes you can calculate for the ANOVA family, one of the more common is ```f```. G*Power kindly tells you what the small, medium, and large parameters are for f if you hover over the cell: \n",
    "\n",
    "![A window displays three dropdown list boxes labeled test family, statistical test, and type of power analysis. A few text fields are placed on the left and right side of the box labeled input parameters and output parameters. The input parameters field has five fields labeled tail, effect size d, alpha err prob, power of 1, minus beta err prob, and allocation ration N2 by N1. 0.5 is entered on the effect size d. A popup message indicating the effect size d box and the message reads, effect size conventions f equals .10 minus small, f equals .25 minus medium, f equals .40 minus large.](Media/power15.png)\n",
    "\n",
    "As with *t*-tests, you will want to pick an effect size range that makes sense for the task at hand. The default medium effect size of .25 should do just fine for most things, but if it's really important that you detect a large effect, you'll probably want to go with something a little larger, and if even a small effect can have a large difference, like a reduction in violence or in the prevention of disease, then you'll want to make your f value smaller.\n",
    "\n",
    "---\n",
    "\n",
    "### Alpha Error Probability\n",
    "\n",
    "Like with all tests, you can keep this default at alpha = .05 unless otherwise specified.\n",
    "\n",
    "---\n",
    "\n",
    "### Power\n",
    "\n",
    "Just like with the *t*-tests, don't forget to change your power from G*Power's default of .95 to a more reasonable .8, which is typically the standard for power.\n",
    "\n",
    "---\n",
    "\n",
    "### Numerator Degrees of Freedom\n",
    "\n",
    "The numerator degrees of freedom is just the number of variables you plan to have in your model minus one.  So, if you were going to have a two-way ANOVA, this would be 1.  If you were going to have a 4 way ANOVA, the value you would place in ```Numerator df``` would be 3.  You'll note that you won't see this box pop up if you select the ```ANOVA: Fixed effects, omnibus, one-way``` because by default, you are only including one variable.\n",
    "\n",
    "---\n",
    "\n",
    "### Number of Groups\n",
    "\n",
    "This is simply the number of levels of the main independent variable you intend to test.  Trying to determine how five age groups differ? Then you'll place 5 here.  Examining the difference between three dog breeds? Then go ahead and enter a 3 instead.\n",
    "\n",
    "---\n",
    "\n",
    "## Output Parameters for Between Subjects\n",
    "\n",
    "Here is what the output will look like for an example: \n",
    "\n",
    "![A window is labeled G asterisk Power 3.1.9.4 on the title bar. The next bar has six menus labeled file, edit, view, tests, calculator, and help. The next bar has two tabs labeled central and non central distributions and protocol of power analyses. The protocol of power analyses bar is selected. The screen displays the details of analysis, input, and output. Three dropdown list boxes labeled test family, statistical test, and type of power analysis. A few text fields are placed on the left and right side of the box labeled input parameters and output parameters. The input parameters field has five fields labeled tail, effect size d, alpha err prob, power of 1, minus beta err prob, and allocation ration N2 by N1. The entries of the input parameters are 0.25, 0.05, 0.8, 10, and 5. The output parameters field has seven text fields that are labeled noncentrality parameter lambda, critical t, Df, sample size group 1, sample size group 2, total sample size, and actual power. The entries of the output parameters are 16.8125000, 1.8666726, 264, 269, and 0.8..1280. Two buttons labeled X-Y plot for a range of values and calculate are placed at the bottom.](Media/power16.png)\n",
    "\n",
    "The most important thing you'll derive from this output is the ```Total sample size``` - make sure you have at least that much data, or you may not find the effect you're looking for.  Is that a Jedi mind trick? \"This is not the effect you're looking for?\" Possibly.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8d66e53",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 8 - Between-Subjects Power Analysis Variants<a class=\"anchor\" id=\"DS105L9_page_8\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d4391d5",
   "metadata": {},
   "source": [
    "# Between-Subjects Power Analysis Variants\n",
    "\n",
    "The input parameters for between-subjects measures by and large apply to regressions and analyses of covariance as well. On this page, you'll take a look at how these variants differ in their input parameters. \n",
    "\n",
    "---\n",
    "\n",
    "## Input Parameters for Regression\n",
    "\n",
    "When running a power analysis for ```Linear multiple regression```, there are two things that will differ from the between-subjects parameters. First, you will need to change the effect size from ```f``` to ```f squared```. Luckily, G*Power provides the estimates for a small, medium, and large effect size for you if you hover over the cell: \n",
    "\n",
    "![A window displays three dropdown list boxes labeled test family, statistical test, and type of power analysis. A few text fields are placed on the left and right side of the box labeled input parameters and output parameters. The input parameters field has five fields labeled tail, effect size d, alpha err prob, power of 1, minus beta err prob, and allocation ration N2 by N1. 0.15 is entered on the effect size d. A popup message indicating the effect size d box and the message reads, effect size conventions f square equals .02 minus small, f square equals .15 minus medium, f square equals .35 minus large.](Media/power18.png)\n",
    "\n",
    "Second, you will now need to specify the ```Number of predictors``` instead of the ```Number of groups```.  The ```Number of predictors``` is simply the number of independent variables you have in your regression model.  If you were trying to predict power outages across the city as your dependent variable, using age of power lines and the amount of shelter from the wind as your independent variables, then you would place a two here for the number of predictors.\n",
    "\n",
    "---\n",
    "\n",
    "## Output Parameters for Regression\n",
    "\n",
    "Here is an example of the typical output you may receive for output regression parameters. Per usual, you're focused on the sample size requirements.\n",
    "\n",
    "![A window is labeled G asterisk Power 3.1.9.4 on the title bar. The next bar has six menus labeled file, edit, view, tests, calculator, and help. The next bar has two tabs labeled central and non central distributions and protocol of power analyses. The protocol of power analyses bar is selected. The screen displays the details of input and output parameters. Three dropdown list boxes labeled test family, statistical test, and type of power analysis. A few text fields are placed on the left and right side of the box labeled input parameters and output parameters. The input parameters field has five fields labeled tail, effect size d, alpha err prob, power of 1, minus beta err prob, and allocation ration N2 by N1. The entries of the input parameters are 0.15, 0.05, 0.8, and 2. The output parameters field has seven text fields that are labeled noncentrality parameter lambda, critical t, Df, sample size group 1, sample size group 2, total sample size, and actual power. The entries of the output parameters are 10.2000000, 3.1381419, 2, 65, 68, and 0.8044183. Two buttons labeled X-Y plot for a range of values and calculate are placed at the bottom.](Media/power21.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Input Parameters for Covariates\n",
    "\n",
    "The only thing that differs for a covariate model, ```ANCOVA: Fixed effects, main effects and interactions``` is that you will need to specify the number of covariates.  Looking at how gender and age covary? Well then you have two covariates.  Simple!\n",
    "\n",
    "![A window has three dropdown list boxes labeled test family, statistical test, and type of power analysis. A few text fields are placed on the left and right side of the box labeled input parameters and output parameters. The input parameters field has six fields labeled effect size f, alpha err prob, power of 1 minus beta err prob, numerator df, number of groups, and number of covariates. The entries of the input parameters are 0.25, 0.05, 0.8, 10, 5and 1. The output parameters field has seven text fields that are labeled noncentrality parameter lambda, critical t, Df, sample size group 1, sample size group 2, total sample size, and actual power. The entries of the output parameters are labeled with a question mark. Two buttons labeled X-Y plot for a range of values and calculate are placed at the bottom.](Media/power19.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Output Parameters for Covariates\n",
    "\n",
    "Here is what an example might look like if you ran it in G*Power: \n",
    "\n",
    "![A window is labeled G asterisk Power 3.1.9.4 on the title bar. The next bar has six menus labeled file, edit, view, tests, calculator, and help. The next bar has two tabs labeled central and non central distributions and protocol of power analyses. The protocol of power analyses bar is selected. The screen displays the details of input and output parameters. Three dropdown list boxes labeled test family, statistical test, and type of power analysis. A few text fields are placed on the left and right side of the box labeled input parameters and output parameters. The input parameters field has six fields labeled effect size f, alpha err prob, power of 1 minus beta err prob, numerator df, number of groups, and number of covariates. The entries of the input parameters are 0.25, 0.05, 0.8, 10, 5and 1. The output parameters field has seven text fields that are labeled noncentrality parameter lambda, critical t, Df, sample size group 1, sample size group 2, total sample size, and actual power. The entries of the output parameters are labeled 16.8125000, 1.8668102, 263, 269, and 0.8000648. Two buttons labeled X-Y plot for a range of values and calculate are placed at the bottom.](Media/power20.png)\n",
    "\n",
    "As usual, sample size is the most important part here. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3da47d1f",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 9 - Input Parameters for Repeated Measures Designs<a class=\"anchor\" id=\"DS105L9_page_9\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef688615",
   "metadata": {},
   "source": [
    "# Input Parameters for Repeated Measures Designs\n",
    "\n",
    "With a more complex design comes even more input parameters! In addition to those you just learned about, for ```ANOVA: Repeated measures``` you will also need to include information about: \n",
    "\n",
    "* Number of measurements\n",
    "* Corr among rep measures\n",
    "* Nonsphericity correction\n",
    "\n",
    "These parameter guidelines apply to anything with ```Repeated measures``` in its name, including ANOVAs and MANOVAs.\n",
    "\n",
    "---\n",
    "\n",
    "## Number of Measurements\n",
    "\n",
    "The number of measurements is simply the number of timepoints (or other repeated measurement) you'll be examining.  If you have a pre and a post test, then this value will be 2.  If you are looking at sales data every quarter for a year, this number would be 4.\n",
    "\n",
    "---\n",
    "\n",
    "## Correlation Among Repeated Measures\n",
    "\n",
    "The correlation among repeated measures is the average correlation value, *r* of how each repeated measure relates to each other.  If you remember back from your correlation interpretation values, .5 is a moderate correlation, exactly in the middle.  You typically will need to use your best guess here to determine what an appropriate *r* value might be, since most likely you don't have access to your data yet. If thinking through the issue says that your measures will be really closely related, then you'll probably want to bump these values up, but if you're thinking they might not relate much, then you'll probably want to bump that default .5 value downwards.\n",
    "\n",
    "---\n",
    "\n",
    "## Nonsphericity Correction\n",
    "\n",
    "You can leave this default as 1. \n",
    "\n",
    "---\n",
    "\n",
    "# Output Parameters\n",
    "\n",
    "Here is an example of what the output might look like for a repeated measures design.\n",
    "\n",
    "![A window is labeled G asterisk Power 3.1.9.4 on the title bar. The next bar has six menus labeled file, edit, view, tests, calculator, and help. The next bar has two tabs labeled central and non central distributions and protocol of power analyses. The protocol of power analyses bar is selected. The screen displays the details of input and output parameters. Three dropdown list boxes labeled test family, statistical test, and type of power analysis. A few text fields are placed on the left and right side of the box labeled input parameters and output parameters. The input parameters field has seven fields labeled effect size f, alpha err prob, power of 1 minus beta err prob, numerator df, number of groups, and number of covariates. The entries of the input parameters are 0.25, 0.05, 0.8, 2, 4, 0.5, and 1. The output parameters field has seven text fields that are labeled noncentrality parameter lambda, critical t, Df, sample size group 1, sample size group 2, total sample size, and actual power. The entries of the output parameters are labeled 12.0000000, 2.7437108, 3.0000000, 66.0000000, 24, 08157454. Three buttons labeled X-Y plot for a range of values, options, and calculate are placed at the bottom.](Media/power17.png)\n",
    "\n",
    "You'll notice that the total sample size is really small - only 24! And this may surprise you at first.  However, this is a great moment to emphasize how much stronger repeated measures designs are than within subjects designs, because they naturally reduce variation - you are measuring the same people or things over and over again, so you don't have to deal with the individual variation from person to person or object to object.  Thus it is much more likely to determine effects. \n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Want to go into even more detail with G*Power? Check out this <a href=\"https://www.gpower.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch-Naturwissenschaftliche_Fakultaet/Psychologie/AAP/gpower/GPowerShortTutorial.pdf\">Short Tutorial</a></p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "Statistical power is important, because it provides you the necessary oomph to ensure you find the correct results when doing analyses. You have now learned how to calculate power for most tests you will run \"in the wild.\" Typically, you will most often calculate power before you begin analyzing data, rather than after you have already analyzed the data.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38e2104f",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 10 - Key Terms<a class=\"anchor\" id=\"DS105L9_page_10\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d32a596",
   "metadata": {},
   "source": [
    "# Key Terms\n",
    "\n",
    "Below is a list and short description of the important keywords learned in this lesson. Please read through and go back and review any concepts you do not fully understand. Great Work!\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Type I Error</td>\n",
    "        <td>Rejecting the null in error.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Type II Error</td>\n",
    "        <td>Accepting the null in error.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Power</td>\n",
    "        <td>The ability to detect an effect; also referred to as 1 minus beta.  Value is typically .8.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>a priori power</td>\n",
    "        <td>Testing power before you analyze data.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>post hoc power</td>\n",
    "        <td>Testing power after you analyze data.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>G*Power</td>\n",
    "        <td>Software used to calculate power.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Allocation ratio</td>\n",
    "        <td>The sample size ratio of groups for t-test power testing. </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Numerator df</td>\n",
    "        <td>The number of variables minus one for ANOVA power testing.</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dea2671",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 11 - Lesson 9 Hands-On<a class=\"anchor\" id=\"DS105L9_page_11\"></a>\n",
    "\n",
    "[Back to Top](#DS105L9_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ead9713",
   "metadata": {},
   "source": [
    "For this Hands On, you will be performing power analyses in G*Power for a variety of different tests.  \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements\n",
    "\n",
    "For each scenario, please complete the following:\n",
    "\n",
    "* Determine what analysis you would run \n",
    "* Conduct an a priori power analysis to determine the sample size for that analysis\n",
    "\n",
    "Then report your chosen test and the required sample size in a text document or presentation.\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>You may want to watch <a href=\"https://vimeo.com/447003939\">this recorded live workshop </a> before beginning the hands-on, which goes over a similar example.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "### Scenario 1\n",
    "\n",
    "A company is hoping to collect data about the different marketing strategies they have undertake via social media.  They want to measure the number of people who follow their posts on Facebook, Twitter, and LinkedIn to determine if one site works better than the others.  \n",
    "\n",
    "---\n",
    "\n",
    "### Scenario 2\n",
    "\n",
    "You have been hired to predict how roofing companies will fare in the upcoming years.  There are several predictors: yearly hurricanes, winter storms, shingle prices, and GDP.  \n",
    "\n",
    "---\n",
    "\n",
    "### Scenario 3\n",
    "\n",
    "A hospital has contracted with you to determine how to improve patient care, as measured continuously by both pain level and disability level.  They are examining these metrics upon admission to the hospital and at discharge from the hospital, and they are comparing their current standard of care to one where they check on the patients every hour.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}