g2g

declaration-diagnosis-redesign/choosing-answer-strategy.qmd

@@ -378,7 +378,7 @@ More formally, @aronow2019foundations describe a plug-in estimator as:
 
 where $\widehat F$ is an estimate of $F$.
 
-### Illustration of estimates using the plug-in principle 
+#### Illustration of estimates using the plug-in principle 
 
 To illustrate the plug-in principle, suppose that our inquiry is the average treatment effect among the $N$ units in the population:
 
@@ -394,7 +394,7 @@ where units 1 through $m$ reveal their treated potential outcomes and the remain
 
 We could do the same thing for other functions, such as quantiles of the distribution or the variance of the distribution. In general, plug-in estimators are not guaranteed to be unbiased, but they can have nice asymptotic properties, converging to targets as the data increases [for conditions, see @van2000asymptotic]. 
 
-### Illustration of estimates of uncertainty using the plug-in principle 
+#### Illustration of estimates of uncertainty using the plug-in principle 
 
 The plug-in principle can be used also for generating estimates of uncertainty. For instance if we are interested in understanding the variance of the sampling distribution of estimates that we get from our procedure, we can use the bootstrap. With the bootstrap, we "plug-in" the sample for the population data, then repeatedly *re*sample from our existing data. We can then approximate the true sampling distribution by calculating estimates on each resampled dataset, from which we can estimate the variance. (We use this nonparametric bootstrapping approach when generating estimates of uncertainty of diagnosands, see @sec-ch10s3ss3).
 
@@ -461,7 +461,7 @@ When the data strategy introduces distortions like differential probabilities of
 This idea can be summarized as "analyze as you randomize," a dictum attributed to R. A. Fisher. We use known features of the data strategy to adjust the answer strategy. We can undo the distortion introduced by differential probabilities of assignment by weighting units by the inverse of the probability of being in the condition that they are in. If we use an inverse-probability weighted (IPW) estimator, we restore parallelism because even though *A* no longer equals *I*, the relationship of *D* to *A* once again parallels the relationship of *M* to *I*.
 
 
-### Illustration of estimates using the AAYR principle {#sec-ch9s3ss5}
+#### Illustration of estimates using the AAYR principle {#sec-ch9s3ss5}
 
 @def-ch9num6 illustrates this idea. We declare the theoretical half of the design as `MI` then consider the intersection of two data strategies with two answer strategies. `D1` has constant probabilities of assignment and `D2` has differential probabilities of assignment. `A1` is the plug-in estimator, applied to unweighted data, and `A2` is the IPW estimator with the inverse probability weights generated by the `D2` randomization protocol.
 
@@ -499,7 +499,7 @@ Understanding whether the choices over answer strategies---logit or probit or OL
 
 Designing agnostically has something in common with robustness checks: both share the motivation that we have fundamental uncertainty about the true model. A robustness check is an *alternative* answer strategy that changes some model assumption that the main answer strategy depends on. Presenting three estimates of the same parameter under different answer strategies (logit, probit, and OLS) and making a joint decision based on the set of estimates about whether the main analysis is "robust" is a procedure for assessing "model dependence" --- meaning, dependence on *statistical* models. But robustness checks are just answer strategies themselves, and we should declare them and diagnose them to understand whether they are good answer strategies. We want to understand the *properties* of the robustness check, e.g., under what models and how frequently does it correctly describe the main answer strategy as "robust."
 
-### Illustration of estimates of uncertainty using the AAYR principle
+#### Illustration of estimates of uncertainty using the AAYR principle
 
 We can illustrate the AAYR principle using the idea of "randomization inference." Randomization inference describes a large class of procedures for generating $p$-values that merit special attention. Randomization inference leverages known features of the randomization procedure to simulate trials under a null model (see @Gerber2012, chapter 3, for an introduction to randomization inference). In a common case, a randomization inference test proceeds by stipulating a null model under which the counterfactual outcomes of each unit are exactly equal to the observed outcomes, the so-called "sharp null hypothesis of no effect."" Under this null hypothesis, the treated and untreated potential outcomes are exactly equal for each unit, reflecting a model in which the treatment has exactly zero effect for each unit.
 

lifecycle/integration.qmd

@@ -4,6 +4,7 @@
 #| echo: false
 #| include: false
 #| purl:  false
+#| cache: false
 source("scripts/before_chapter_script.R")
 ```
 
@@ -84,15 +85,70 @@ The reanalyst has in mind a different model. In this second model, $X$ confounds
 ```{r, file = "scripts_declarations/declaration_23.1b.R"}
 ```
 
+
+
 The original answer strategy `A` is a regression of the outcome $Y$ on the treatment $Z$. The reanalyst collects the covariate $X$ and proposes to control for it in a linear regression; call that strategy `A_prime`. 
 
 ```{r, file = "scripts_declarations/declaration_23.1c.R"}
 ```
+
+
 Applying the two answer strategies, we get differing results. The treatment effect estimate is significant under `A` but not under `A_prime`. Commonly, reanalysts would infer from this that the answer strategy `A_prime` is preferred and that the original result was incorrect.
 
+```{r, eval = FALSE}
+draw_estimates(model_2 + A + A_prime)
+```
+
+```{r}
+#| tbl-cap: "Results under A and A'"
+#| echo: false
+#| label: tbl-reanalysisresults
+set.seed(3)
+est <- draw_estimates(model_2 + A + A_prime)
+est |>
+  select(estimator, estimate, std.error, p.value) |>
+  kable(booktabs = TRUE,
+        align = "c",
+        digits = 3,
+        caption = "Analysis and reanalysis estiamtes")
+```
+
+
 As we show now, these claims depend on the validity of the model and should be assessed with design diagnosis. Consider a third model in which *X* is affected by *Z* and *Y*. (In many observational settings, which variables are causally prior or posterior to others can be difficult to know with certainty.) We now diagnose both answer strategies under all three models.
 
-{What we see in the diagnosis below is that `A_prime` is only preferred if we know for sure that *X* is measured pretreatment. In design 3, where *X* is measured posttreatment, `A` is preferred, because controlling for *X* leads to posttreatment bias. This diagnosis indicates that the reanalyst needs to justify their beliefs about the causal ordering of *X* and *Z* to claim that `A_prime` is preferred to `A`. The reanalyst should not conclude on the basis of the realized estimates only that their answer strategy is preferred.}
+::: {#def-ch23num1 .declaration} 
+
+Reanalysis declaration.
+
+```{r, file = "scripts_declarations/declaration_23.1d.R", eval = FALSE}
+```
+
+:::
+
+
+```{r, echo = FALSE, eval = FALSE}
+diagnosis_23.1 <- diagnose_designs(declaration_23.1)
+```
+
+::: {#lem-ch23num1} 
+
+Reanalysis diagnosis.
+
+What we see in the diagnosis below is that `A_prime` is only preferred if we know for sure that $X$ is measured pretreatment. In design 3, where $X$ is measured posttreatment, `A` is preferred, because controlling for $X$ leads to posttreatment bias. This diagnosis indicates that the reanalyst needs to justify their beliefs about the causal ordering of $X$ and $Z$ to claim that `A_prime` is preferred to `A`. The reanalyst should not conclude on the basis of the realized estimates only that their answer strategy is preferred.
+
+```{r renalysisdiagnosis, echo = FALSE}
+diagnosis_23.1 <- get_rdddr_file_local("diagnosis_23.1")
+get_diagnosands(diagnosis_23.1) |>
+  select(design, estimator, bias) |>
+  kable(
+    booktabs = TRUE,
+    align = "c",
+    digits = 3,
+    caption = "Diagnosis of the reanalysis design under alternative models"
+  )
+```
+
+:::
 
 Three principles emerge from the idea that changing *A* to $A'$ should be justified by diagnosis, not by the comparison of the realized results of the two answer strategies.
 
@@ -154,3 +210,4 @@ For the second kind of synthesis---literature-wide progress on a full causal mod
 Design then with a view to integration. In the ideal case the meta-analytic design will already exist and your job will be to design a new study that can demonstrably add value to the collective design.
 
 
+