Skip to content

Latest commit

 

History

History
54 lines (24 loc) · 3.41 KB

12. Multiple Comparisons.md

File metadata and controls

54 lines (24 loc) · 3.41 KB

Quizz

1.Recall that a "discovery" occurs when a test rejects the null hypothesis. In the medical literature a discovery is called a "positive result". So a "false positive" is a "false discovery". What is the false discovery proportion (FDP) of the procedure that yielded the following results:

image

9/(9+36) , The FDP is the fraction of all positive results that are false.


  1. A medical study examines whether there is a significant correlation between any of 12 lifestyle choices and high blood pressure. It doesn't find any significant correlation, but upon further examination the researchers find a highly significant (pp-value <0.5%) correlation between two of the lifestyle choices. This correlation seems not to have been noticed before. Which of the following three statements is an appropriate summary of these findings? Select all that apply.

  • The correlation between these two lifestyle choices is highly significant and should be reported as such.

  • The seemingly significant correlation was found as a consequence of data snooping and therefore the pp-value is not valid. The researchers shouldn't report anything.

  • The seemingly significant correlation was found as a consequence of data snooping and therefore the pp-value is not valid. However, this could potentially be a significant new finding. The researchers can report it as such, pointing out that they cannot attach a valid pp-value to this finding. It can serve as a hypothesis for a future study with new data, which would then allow for statistically valid conclusions.


  1. 1,000 tests were evaluated with the Bonferroni correction. 31 tests had corrected pp-values smaller than 5%. Which of the following three statements is an appropriate conclusion?

  • If we reject these 31 null hypotheses then we can expect that about 5% of them are rejected in error.

  • This is sufficient evidence to reject all of these 31 null hypotheses, because there is only a 5% chance that any of these 31 pp-values would be this small if the null hypotheses were true.

  • There is a 95% probability that all of these 31 null hypotheses are false.

The Bonferroni correction ensures that if all the null hypotheses are true, the probability that the 1000 tests will lead to a discovery is less than 5%, and we consider true any discovery that arises from these tests. That is, we reject any null hypothesis associated with a corrected pp-value of 5% (i.e., an uncorrected pp-value of 0.005%).


  1. 1,000 tests were evaluated with the FDR at the 5% level, which resulted in 31 discoveries. Which of the following three statements is an appropriate conclusion?

  • There is a 95% probability that all of these 31 null hypotheses are false.

  • This is sufficient evidence to reject all of these 31 null hypotheses, because there is only a 5% chance that any of these 31 pp-values would be this small if the null hypothesis were true.

  • If we reject these 31 null hypotheses then we can expect that about 5% of them are rejected in error.

The FDR is designed to constrain the FDP (the proportion of discoveries that are false discoveries) to be at most \alphaα%, so in this case we can expect that about 5% of the discoveries we find using FDR to be false. That is, we can expect that 5% of the null hypotheses that FDR leads us to reject are rejected in error.