This unit introduces the core concepts of classical hypothesis testing. It builds on the knowledge that you built about sampling theory and random varaibles to ask (and answer) questions of the following form.
- Suppose you have observed a difference in sample averages between two groups of a certain magnitude, d. How likely is it to see a difference of this size or larger, if in fact the two groups have the same expected value?
The first test that we introduce -- the t-test -- is just one of a family of tests that all share the same general testing framework. These tests -- the frequentist tests -- all compute a test statistic (which is a random variable) that has a known probability distribution under iid sampling. To conduct a test, this statistic is compared to the known, referece distribution and the data scientist reads off a p-value, which is quite literally the total probability of the reference distribution that is more extreme than the statistic generated by the data.
The t-test is a workhorse, and also sets a pattern for every other frequentist, statistical test.