Rm analyses that shouldn't be run (BT random effects models)

[egouldo]

Hey Tim,

I think this may potentially be a result of the way I repeated the analyses over each of the different data subsets, i'll look into it.

So, the models may have been fitted automatically, but I never extracted the results of the cases where we had < 5 analyses with random effects included.

1. I’ll make sure these cases are removed from the manuscript first,
2. And then can remove these from the analysis pipeline.

Elliot.

On 1 Jun 2024, at 6:14 am, Tim Parker [email protected] wrote:

Hi Elliot,

In the supplement
C.6.1 Effect Sizes Zr
also
C.6.2 Out of sample predictions yi

It appears that the presence of mixed effects (whether or not the analyst included a random effect) was included in both the BT and the Euc models, but I thought that we only included this effect for Euc since there were too few BT analysts that did not include a random effect.

-Tim

[egouldo]

Sections C.6.1 and C.6.2 refer to the multivariate models for both Zr and yi, not the deviation from the analytic mean explained by inclusion of random effect models.

• Were we supposed to remove this fixed term from the multivariate models if we didn’t meet the preregistered threshold?

I may have misinterpreted the original preregistration if so.

Depending on Tim's response I’ll update the code or leave as is.

[egouldo]

From Tim:

Got it. Yes, the registration stated that we would include the 'random effect presence' variable in the univariate and the multivariate analysis only if it crossed the threshold:

"Thus if there are at least 5 analyses that do and 5 analyses that do not include random effects, we will add a binary predictor variable “random effects
included (yes/no)” to our set of univariate analyses and will add this predictor variable to our multivariate model described below".

• @egouldo to remove predictor term from models and re-run
• Check random effects need to add effectID, we said we would run with analysis ID, but not included in reported model.
• Check convergence / singularity, if fails, then return back to ReviewerID only.

[egouldo]

be0ae12 fixes this I think

[egouldo]

Yep:

library(report)
library(ManyEcoEvo)
#> Loading required package: rmarkdown
#> Loading required package: bookdown
#> Registered S3 method overwritten by 'parsnip':
#>   method          from 
#>   print.nullmodel vegan
#> Registered S3 method overwritten by 'lava':
#>   method         from    
#>   print.estimate EnvStats

library(tidyverse)

ManyEcoEvo_yi_results %>%
  dplyr::filter(dataset == "blue tit") %>%
  pluck("MA_mod_mv", 1) %>%
  format_formula()
#> [1] "formula: box_cox_abs_deviation_score_estimate ~ RateAnalysis + PublishableAsIs + mean_diversity_index"

^{Created on 2024-09-05 with reprex v2.1.0}

library(report)
library(ManyEcoEvo)
#> Loading required package: rmarkdown
#> Loading required package: bookdown
#> Registered S3 method overwritten by 'parsnip':
#>   method          from 
#>   print.nullmodel vegan
#> Registered S3 method overwritten by 'lava':
#>   method         from    
#>   print.estimate EnvStats

library(tidyverse)

ManyEcoEvo_results %>%
  dplyr::filter(dataset == "blue tit") %>%
  pluck("MA_mod_mv", 1) %>%
  format_formula()
#> [1] "formula: box_cox_abs_deviation_score_estimate ~ RateAnalysis + PublishableAsIs + mean_diversity_index"

^{Created on 2024-09-05 with reprex v2.1.0}

And now check univariate mixed effects models

library(ManyEcoEvo)
#> Loading required package: rmarkdown
#> Loading required package: bookdown
#> Registered S3 method overwritten by 'parsnip':
#>   method          from 
#>   print.nullmodel vegan
#> Registered S3 method overwritten by 'lava':
#>   method         from    
#>   print.estimate EnvStats

library(tidyverse)

ManyEcoEvo_results %>%
  dplyr::filter(dataset == "blue tit") %>%
  pluck("uni_mixed_effects", 1)
#> [1] NA

ManyEcoEvo_yi_results %>%
  dplyr::filter(dataset == "blue tit") %>%
  pluck("uni_mixed_effects", 1)
#> [1] NA

^{Created on 2024-09-05 with reprex v2.1.0}