probTheory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Theories of Probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've been looking at the mathematical side of probability, which starts from some axioms\n",
    "that probability is assumed to satisfy, and draws conclusions (i.e., theorems).\n",
    "But what _is_ probability?\n",
    "What does probability _mean_?\n",
    "What connects the mathematics to the real world?\n",
    "\n",
    "This is a knotty philosophical problem that scholars still disagree about.\n",
    "We will look at several attempts to make sense of statements such as \"the probability that a fair coin lands\n",
    "heads when you toss it is 1/2.\"\n",
    "\n",
    "To tie this to scientific questions, we will also think about how to interpret statements such as \"there is a 70% chance of a magnitude 7.5 or greater earthquake in the San Francisco Bay Area in the next 25 years,\" \"there is a 90% chance global temperature will increase 3&deg; in the next 50 years,\" \"there is a 99.99% chance that the Higgs boson exists (or that it was detected),\" and similar statements. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Equally Likely Outcomes\n",
    "\n",
    "This theory of probability is the oldest.\n",
    "It originated in the study of games of chance, such as dice games and card games.\n",
    "In the Theory of Equally Likely Outcomes, probability assignments depend on the\n",
    "assertion that no particular outcome is preferred over any other by Nature; generally,\n",
    "arguments that Nature should show no preference among the outcomes appeal to the\n",
    "symmetry of the system studied (such as a &quot;fair&quot; coin or die).\n",
    "\n",
    "If a given experiment or trial has $n$ possible outcomes among\n",
    "which&mdash;it is assumed&mdash;Nature should show no preference, this\n",
    "theory defines them to be _equally likely_.\n",
    "The probability of each outcome is then $100\\%/n$.\n",
    "\n",
    "Applying this theory of probability involves arguing that no particular outcome should occur\n",
    "in preference to another, typically by appeal to physical symmetries\n",
    "or other considerations&mdash;e.g., that there is no reason one marble would be predisposed\n",
    "to turn up rather than another in drawing a marble from a well-stirred bowl of marbles.\n",
    "Laplace's _Principle of Insufficient Reason_ asserts that if there \n",
    "is no reason to think that a set of outcomes is not equally likely, they should \n",
    "be taken to be equally likely.  This is poor logic: no evidence of a difference \n",
    "is not the same as evidence of no difference.  The fallacy is called \n",
    "_Appeal to Ignorance_. See also \n",
    "Stark and Freedman, 2003.\n",
    "\n",
    "For example, if a coin is balanced well, there is no reason for it to\n",
    "land heads in preference to tails when it is tossed vigorously,\n",
    "so according to the Theory of Equally Likely Outcomes,\n",
    "the probability that the coin lands heads is equal to the\n",
    "probability that the coin lands tails, and both are 100%/2 = 50%.\n",
    "(This ignores the nearly impossible outcomes that the coin does not\n",
    "land at all or lands balanced on its edge.)\n",
    "Similarly, if a die is fair\n",
    "(properly balanced) the chance that when it is rolled vigorously\n",
    "it lands with the side with one spot on top (the chance that the die _shows_\n",
    "one spot) is the same as the chance that it shows two spots or three\n",
    "spots or four spots or five spots or six spots: 100%/6, about 16.7%.\n",
    "\n",
    "If an event consists of more than one possible\n",
    "outcome, the chance of the event is the number of ways it can occur, divided by the total\n",
    "number of things that could occur.\n",
    "For example, the chance that a die lands showing an even number of spots\n",
    "is the number of ways it could land showing an even number of spots\n",
    "(3, namely, landing showing 2, 4, or 6 spots), divided by the total\n",
    "number of things that could occur\n",
    "(6, namely, landing showing 1, 2, 3, 4, 5, or 6 spots).\n",
    "Since the total number of possible outcomes is $n$, the maximum possible\n",
    "probability of any event is $100% \\times n/n = 100\\%$.\n",
    "Thus, in the Theory of Equally Likely Outcomes, probabilities are between 0%\n",
    "and 100%, as claimed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frequency Theory\n",
    "\n",
    "In the Frequency Theory of Probability,\n",
    "probability is the limit of the relative frequency\n",
    "with which an event occurs in repeated trials.\n",
    "(The trials must be _independent_.)\n",
    "\n",
    "Relative frequencies are always between 0% (the event essentially never happens)\n",
    "and 100% (the event essentially always happens), so in this theory as well, probabilities\n",
    "are between 0% and 100%.\n",
    "According to the Frequency Theory of Probability, what it means to\n",
    "say that &quot;the probability that $A$ occurs is\n",
    "$p$%&quot; is that\n",
    "if you repeat the experiment over and over again, independently and\n",
    "under essentially identical conditions, the percentage of the time\n",
    "that $A$ occurs will converge to $p$.\n",
    "For example, under the Frequency Theory, to say that the chance\n",
    "that a coin lands heads is 50% _means_ that if you toss the coin over and over again,\n",
    "independently, the ratio of the number of times the coin lands heads to the\n",
    "total number of tosses approaches a limiting value of 50% as the number\n",
    "of tosses grows.\n",
    "Because the ratio of heads to tosses is always between 0% and 100%, when the\n",
    "probability exists it must be between 0% and 100%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Subjective Theory\n",
    "\n",
    "In the Subjective Theory of Probability, probability measures the speaker's\n",
    "&quot;degree of belief&quot; that the event will occur, on a scale of 0%\n",
    "(complete disbelief that the event will happen) to 100%\n",
    "(certainty that the event will happen).\n",
    "According to the Subjective Theory, what it means for me to say that\n",
    "&quot;the probability that $A$ occurs is 2/3&quot; is that\n",
    "I believe that $A$ will happen twice as strongly as I believe\n",
    "that $A$ will not happen.\n",
    "The Subjective Theory is particularly useful in assigning meaning to the\n",
    "probability of events that in principle can occur only once.\n",
    "For example, how might one assign meaning to a statement like\n",
    "&quot;there is a 25% chance of an earthquake on the San Andreas fault\n",
    "with magnitude 8 or larger before 2050?&quot;\n",
    "(See Stark and Freedman, 2003, for more discussion\n",
    "of theories of probability and their application to earthquakes.)\n",
    "It is very hard to use either the Theory of Equally Likely Outcomes\n",
    "or the Frequency Theory to make sense of the assertion.\n",
    "Can you think of other examples?\n",
    "\n",
    "These three theories of probability assign different meanings to the statement\n",
    "&quot;the chance that $A$ occurs is $p%$.&quot;\n",
    "Each theory has situations in which it is most natural, and each\n",
    "theory has shortcomings.\n",
    "We will use the Frequency Theory primarily."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model-Based Probability\n",
    "\n",
    "[To do]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shortcomings of the Theories\n",
    "\n",
    "While each of the theories has attractive elements, all have shortcomings as well.\n",
    "The shortcomings involve hidden assumptions, limited domains of applicability, and\n",
    "changes of subject.\n",
    "(The Subjective Theory changes the subject more than the others do, as \n",
    "elaborated below.  Arguing for a conclusion superficially related to the \n",
    "desired conclusion and pretending you have established the latter is \n",
    "_The Red Herring_ fallacy. In contrast, arguing against \n",
    "a hypothesis superficially like one you wish to refute then claiming you have \n",
    "refuted the latter is _The Straw Man_ fallacy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Theory of Equally Likely Outcomes\n",
    "\n",
    "Even when there are only a few possible outcomes, it is not always clear whether\n",
    "they should be deemed equally likely.\n",
    "For example, consider tossing two coins at the same time.\n",
    "The possible outcomes could be {two heads, not two heads} or\n",
    "{two heads, one head and one tail, two tails} or\n",
    "{two heads, head on coin 1 and tail on coin 2, tail on coin 1 and head on\n",
    "coin 2, two tails}.\n",
    "\n",
    "The last of these assigns the same probabilities the\n",
    "Frequency Theory does.\n",
    "For instance, if the equally likely outcomes are taken to be\n",
    "{two heads, head on coin 1 and tail on coin 2, tail on coin 1 and head on\n",
    "coin 2, two tails}, both the Theory of Equally Likely Outcomes and the Frequency Theory\n",
    "would say that the chance of two heads is 25%.\n",
    "If one is using probability to bet on games of chance, long-term relative\n",
    "frequencies&mdash;which the Frequency Theory contemplates&mdash;are perhaps the most\n",
    "important consideration, because they determine how much one wins or loses in\n",
    "the long run.\n",
    "It seems rather artificial to introduce a distinction between two otherwise\n",
    "identical coins in order to make the probabilities calculated using the\n",
    "Theory of Equally Likely Outcomes agree with the probabilities calculated using\n",
    "the Frequency Theory.\n",
    "Perhaps a more serious limitation of the Theory of Equally Likely Outcomes is that\n",
    "many situations do not have natural symmetries to exploit to decide which outcomes\n",
    "are equally likely.\n",
    "For example, what is the chance that a thumbtack lands with its point up when it\n",
    "is tossed vigorously? What is the chance that a die that has been &quot;loaded&quot;\n",
    "(modified to be unbalanced) lands showing one spot?\n",
    "Neither of these problems has a natural symmetry from which to argue that the\n",
    "outcomes are equally likely.\n",
    "Moreover, in many situations there are an infinite number of possible outcomes;\n",
    "dividing 100% by infinity yields zero."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Frequency Theory\n",
    "\n",
    "The Frequency Theory requires an assumption about how the world works:\n",
    "The relative frequency with which an event occurs in repeated trials\n",
    "is _assumed_ to converge to a limit.\n",
    "What is a limit?\n",
    "In the case of coin tossing, the theory says that for any positive\n",
    "number $\\epsilon$, no matter how small, there is some number\n",
    "$M$, which can depend\n",
    "on $\\epsilon$, such that\n",
    "\n",
    "\\begin{equation*}  | \\mbox{(#heads in n tosses)}/n - 50\\% | < \\epsilon \\end{equation*}\n",
    "\n",
    "whenever the number of tosses $n > M$.\n",
    "Not all sequences of heads and tails satisfy this assumption.\n",
    "For example, suppose the first toss gives a head.\n",
    "The relative frequency of heads is then 100%.\n",
    "Suppose the next 3 tosses give tails.\n",
    "The relative frequency of heads is then 25%.\n",
    "Suppose the next 100 tosses give heads.\n",
    "The relative frequency of heads is then over 97%.\n",
    "Suppose the next 5000 tosses give tails.\n",
    "The relative frequency of heads is then about 20%.\n",
    "If we continue in this way, with ever longer runs of heads and of tails,\n",
    "the relative frequency of heads never approaches a limit.\n",
    "\n",
    "The _Empirical Law of Averages_\n",
    "says this never happens:\n",
    "The world works in such a way that the relative frequency with which a random\n",
    "event occurs in repeated trials always settles down to a limit.\n",
    "This &quot;law&quot; is an _assumption_ about how the world works.\n",
    "It is not a mathematical fact, and it is not an observation because no one can continue\n",
    "tossing coins forever to see whether the relative frequency of heads starts to vary\n",
    "again after, say, 100,000,000,000,000 tosses.\n",
    "The Empirical Law of Averages is essential to the Frequency Theory.\n",
    "\n",
    "The second limitation of the Frequency Theory is that many events to which\n",
    "we might like to assign probabilities are not the outcomes of repeatable experiments.\n",
    "For example, what is the probability that the universe will end in a &quot;big crunch?&quot;\n",
    "What is the probability that my 2020 tax return will be audited?\n",
    "What is the probability that in 2020 more online textbooks than paper\n",
    "textbooks will be sold?\n",
    "What is the probability that the Dow Jones Industrial Average reaches 20,000\n",
    "before the year 2020?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Subjective Theory\n",
    "\n",
    "The principal shortcoming of the Subjective Theory is that colloquially we think\n",
    "of probability as being a property of an event in the external (objective) world,\n",
    "not merely a reflection of our state of mind.\n",
    "When I say &quot;this thumbtack has probability 66% of landing point up when I toss it,&quot;\n",
    "you probably think I am talking about the tack, not about my state of mind with\n",
    "respect to the tack: this theory changes the subject.\n",
    "Similarly, under the Subjective Theory, you and I can disagree about the probability\n",
    "of an event and both be correct, which seems unsatisfactory in many scientific\n",
    "settings.\n",
    "\n",
    "There are a variety of technical difficulties in the Subjective Theory regarding how to\n",
    "measure the probability of an event.\n",
    "One possible resolution is to study the bets you are willing to take.\n",
    "Would you be indifferent between a bet that a coin lands heads and a bet with the\n",
    "same stakes that it lands tails?\n",
    "If so, some theorists would conclude that your subjective probability that the coin\n",
    "lands heads is 50%.\n",
    "Some factors can complicate this approach.\n",
    "For example, even though you know that buying a lottery ticket is almost certainly\n",
    "throwing your money away, you might buy a ticket anyway, reasoning that you would\n",
    "not particularly miss the \\$1 cost of the ticket, while you would definitely notice\n",
    "winning \\$20,000,000.\n",
    "In this scenario, the _probability_ of winning is less of an issue than the\n",
    "_possibility_ of winning.\n",
    "\n",
    "Here is another example: I will bet you \\$1,000,000 against \\$500 that there will\n",
    "not be a nuclear bomb dropped on Berkeley, California, by the year 2020.\n",
    "Even if I am confident that a nuclear bomb will be dropped, if it is dropped,\n",
    "I won't have to pay off the lost bet (I live in Berkeley), but if it is not dropped,\n",
    "I could use the \\$500 you would owe me.\n",
    "\n",
    "Another problem with the Subjective Theory has to do with\n",
    "scientific method.\n",
    "Some philosophers of science maintain that unless an hypothesis can,\n",
    "in principle, be shown to be false, it is not scientific.\n",
    "An hypothesis that in principle can be disproved is called _falsifiable_.\n",
    "In the Frequency Theory, one can collect evidence against the statement that\n",
    "&quot;the probability that $A$ occurs\n",
    "is $p$%&quot; by repeating an experiment\n",
    "over and over and looking at the fraction of times the event $A$ occurs.\n",
    "In the Subjective Theory, evidence against the hypothesis that\n",
    "&quot;the probability that $A$ occurs is $p$%&quot;\n",
    "is found by psychological testing to see whether the individual making the\n",
    "statement is telling the truth and is internally consistent in his assignments\n",
    "of probability.\n",
    "Running the real-world experiment over would not be relevant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model-based Probability\n",
    "\n",
    "[To Do]\n",
    "Calibrating/testing the model, availability of relevant data, ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previous: [Sets, Combinatorics, & Probability](prob.ipynb)\n",
    "Next: [Random Variables, Expectation, & Random Vectors](rv.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}