Accumulate setting in EpiNow2

[nlinton]

Hello, I noticed there have been a lot of updates around the accumulate setting in EpiNow2, which is very exciting to see. We've tried to run a version of the code using fill_missing(missing_dates="accumulate", initial_accumulate=7) (pulled from github, since it was not showing up in the CRAN install of EpiNow2 1.6.1) to set up the data, which then look like this:

However, trying to use this as the data for EpiNow2::epinow() results in the following error, which seems to tell me I cannot have the accumulate column in the input dataset.

Error in data.table::rbindlist(list(data.table::data.table(date = seq(min(reported_cases$date) - : Item 2 has 4 columns, inconsistent with item 1 which has 3 columns. To fill missing columns use fill=TRUE. Error in data.table::rbindlist(list(data.table::data.table(date = seq(min(reported_cases$date) - : Item 2 has 4 columns, inconsistent with item 1 which has 3 columns. To fill missing columns use fill=TRUE.

Any guidance on how to get the accumulate setting to accurately work in EpiNow2 1.6.1?

[jamesmbaazam]

Hi @nlinton It's great to see that you are already trying out the new accumulate feature. I tried to reconstruct your data and reproduce your error but I didn't get that (below). Could you provide a reprex to help us diagnose your issue?

library(EpiNow2)
#> 
#> Attaching package: 'EpiNow2'
#> The following object is masked from 'package:stats':
#> 
#>     Gamma
library(data.table)

packageVersion("EpiNow2")
#> [1] '1.6.1.9000'

data <- data.table::data.table(
  date = as.Date(c("2024-08-24", "2024-08-31")),
  confirm = c(2531, 2306)
)
# View the data
data
#>          date confirm
#>        <Date>   <num>
#> 1: 2024-08-24    2531
#> 2: 2024-08-31    2306

data <- fill_missing(
  data,
  missing_dates = "accumulate",
  initial_accumulate = 7
)

# View the data
data
#> Key: <date>
#>           date confirm accumulate
#>         <Date>   <num>     <lgcl>
#>  1: 2024-08-18      NA       TRUE
#>  2: 2024-08-19      NA       TRUE
#>  3: 2024-08-20      NA       TRUE
#>  4: 2024-08-21      NA       TRUE
#>  5: 2024-08-22      NA       TRUE
#>  6: 2024-08-23      NA       TRUE
#>  7: 2024-08-24    2531      FALSE
#>  8: 2024-08-25      NA       TRUE
#>  9: 2024-08-26      NA       TRUE
#> 10: 2024-08-27      NA       TRUE
#> 11: 2024-08-28      NA       TRUE
#> 12: 2024-08-29      NA       TRUE
#> 13: 2024-08-30      NA       TRUE
#> 14: 2024-08-31    2306      FALSE

# Specify other inputs
generation_time <- Gamma(
  shape = Normal(1.3, 0.3),
  rate = Normal(0.37, 0.09),
  max = 14
)

incubation_period <- LogNormal(
  meanlog = Normal(1.6, 0.06),
  sdlog = Normal(0.4, 0.07),
  max = 14
)
# set an example reporting delay. In practice this should use an estimate
# from the literature or be estimated from data
reporting_delay <- LogNormal(mean = 2, sd = 1, max = 10)

# estimate Rt and nowcast/forecast cases by date of infection
out <- estimate_infections(
  data = data,
  generation_time = gt_opts(generation_time),
  rt = rt_opts(prior = LogNormal(mean = 2, sd = 0.1)),
  delays = delay_opts(incubation_period + reporting_delay)
)
#> ℹ As of version 1.5.0 missing dates or dates with `NA` cases are treated as
#>   missing. This is in contrast to previous versions where these were
#>   interpreted as dates with zero cases.
#> ℹ In order to treat missing or `NA` cases as zeroes, see solutions in
#>   <https://github.com/epiforecasts/EpiNow2/issues/767#issuecomment-2348805272>
#> ℹ If the data is reported at non-daily intervals (for example weekly), consider
#>   using `obs_opts(na="accumulate")`.
#> ℹ For more information on these options, see `?obs_opts`.
#> This message is displayed once every 8 hours.
#> Warning: There were 16 divergent transitions after warmup. See
#> https://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup
#> to find out why this is a problem and how to eliminate them.
#> Warning: Examine the pairs() plot to diagnose sampling problems
#> Warning: Tail Effective Samples Size (ESS) is too low, indicating posterior variances and tail quantiles may be unreliable.
#> Running the chains for more iterations may help. See
#> https://mc-stan.org/misc/warnings.html#tail-ess

out$summarised
#>            date                 variable  strat                           type
#>          <Date>                   <char> <char>                         <char>
#>   1: 2024-08-18                        R   <NA> estimate based on partial data
#>   2: 2024-08-19                        R   <NA> estimate based on partial data
#>   3: 2024-08-20                        R   <NA> estimate based on partial data
#>   4: 2024-08-21                        R   <NA> estimate based on partial data
#>   5: 2024-08-22                        R   <NA> estimate based on partial data
#>  ---                                                                          
#> 126: 2024-09-04           reported_cases   <NA>                       forecast
#> 127: 2024-09-05           reported_cases   <NA>                       forecast
#> 128: 2024-09-06           reported_cases   <NA>                       forecast
#> 129: 2024-09-07           reported_cases   <NA>                       forecast
#> 130:       <NA> reporting_overdispersion   <NA>                           <NA>
#>            median        mean           sd    lower_90   lower_50     lower_20
#>             <num>       <num>        <num>       <num>      <num>        <num>
#>   1:    1.9531002    1.957395 9.860917e-02   1.8060208   1.888709    1.9279919
#>   2:    1.9511507    1.956226 9.875206e-02   1.8054405   1.888070    1.9270143
#>   3:    1.9515471    1.954867 9.958806e-02   1.8015232   1.885639    1.9260132
#>   4:    1.9513628    1.953424 1.013485e-01   1.7962150   1.883408    1.9256043
#>   5:    1.9500809    1.951989 1.041336e-01   1.7857012   1.881186    1.9241806
#>  ---                                                                          
#> 126: 1507.0000000 2710.223500 4.015238e+03 102.9000000 595.750000 1086.2000000
#> 127: 1618.5000000 2972.081500 4.510027e+03 108.9000000 656.500000 1196.6000000
#> 128: 1801.0000000 3532.740000 6.239869e+03 123.8500000 679.750000 1312.2000000
#> 129: 2176.5000000 4148.669500 6.060557e+03 120.9500000 893.000000 1546.2000000
#> 130:    0.3560215    0.371260 1.453547e-01   0.1602343   0.266294    0.3187301
#>          upper_20     upper_50     upper_90
#>             <num>        <num>        <num>
#>   1:    1.9798608    2.0259564     2.118147
#>   2:    1.9804259    2.0254079     2.119610
#>   3:    1.9794639    2.0248136     2.118680
#>   4:    1.9793869    2.0238450     2.119698
#>   5:    1.9800565    2.0222187     2.123985
#>  ---                                       
#> 126: 2069.4000000 3311.2500000  9155.900000
#> 127: 2139.6000000 3511.2500000  9734.850000
#> 128: 2470.4000000 4082.2500000 11683.200000
#> 129: 2917.4000000 5071.7500000 14354.700000
#> 130:    0.3955891    0.4640018     0.632445

^{Created on 2024-12-10 with reprex v2.1.1}

[seabbs]

@jamesmbaazam the difference I see between what you have and what @nlinton is reporting is that they are reporting having used epinow() and you have used estimate_infections(). I can imagine that there is an integration test missing here and that the data ingestion in epinow is the problem (if this is the case it suggests that there is a DRY problem here as if epinow() is passing to estimate_infections() they should have the same check.

[jamesmbaazam]

Sorry, that's actually an artifact of me checking if there was a difference between estimate_infections() and epinow(). The epinow() version has no issue as well

library(EpiNow2)
#> 
#> Attaching package: 'EpiNow2'
#> The following object is masked from 'package:stats':
#> 
#>     Gamma
library(data.table)

packageVersion("EpiNow2")
#> [1] '1.6.1.9000'

data <- data.table::data.table(
  date = as.Date(c("2024-08-24", "2024-08-31")),
  confirm = c(2531, 2306)
)
# View the data
data
#>          date confirm
#>        <Date>   <num>
#> 1: 2024-08-24    2531
#> 2: 2024-08-31    2306

data <- fill_missing(
  data,
  missing_dates = "accumulate",
  initial_accumulate = 7
)

# View the data
data
#> Key: <date>
#>           date confirm accumulate
#>         <Date>   <num>     <lgcl>
#>  1: 2024-08-18      NA       TRUE
#>  2: 2024-08-19      NA       TRUE
#>  3: 2024-08-20      NA       TRUE
#>  4: 2024-08-21      NA       TRUE
#>  5: 2024-08-22      NA       TRUE
#>  6: 2024-08-23      NA       TRUE
#>  7: 2024-08-24    2531      FALSE
#>  8: 2024-08-25      NA       TRUE
#>  9: 2024-08-26      NA       TRUE
#> 10: 2024-08-27      NA       TRUE
#> 11: 2024-08-28      NA       TRUE
#> 12: 2024-08-29      NA       TRUE
#> 13: 2024-08-30      NA       TRUE
#> 14: 2024-08-31    2306      FALSE

# Specify other inputs
generation_time <- Gamma(
  shape = Normal(1.3, 0.3),
  rate = Normal(0.37, 0.09),
  max = 14
)

incubation_period <- LogNormal(
  meanlog = Normal(1.6, 0.06),
  sdlog = Normal(0.4, 0.07),
  max = 14
)
# set an example reporting delay. In practice this should use an estimate
# from the literature or be estimated from data
reporting_delay <- LogNormal(mean = 2, sd = 1, max = 10)

# estimate Rt and nowcast/forecast cases by date of infection
out <- epinow(
  data = data,
  generation_time = gt_opts(generation_time),
  rt = rt_opts(prior = LogNormal(mean = 2, sd = 0.1)),
  delays = delay_opts(incubation_period + reporting_delay)
)
#> Logging threshold set at INFO for the name logger
#> Writing EpiNow2 logs to the console and:
#> '/var/folders/vr/dn4r1_zj417drd1zr9301trw0000gp/T//RtmpqTcVBG/regional-epinow/2024-08-31.log'.
#> Logging threshold set at INFO for the name logger
#> Writing EpiNow2.epinow logs to the console and:
#> '/var/folders/vr/dn4r1_zj417drd1zr9301trw0000gp/T//RtmpqTcVBG/epinow/2024-08-31.log'.
#> WARN [2024-12-11 10:56:39] epinow: The `filter_leading_zeros` argument of `estimate_infections()` is deprecated as of EpiNow2 1.7.0. - 
#> WARN [2024-12-11 10:56:39] epinow: The `zero_threshold` argument of `estimate_infections()` is deprecated as of EpiNow2 1.7.0. -
#> ℹ As of version 1.5.0 missing dates or dates with `NA` cases are treated as
#>   missing. This is in contrast to previous versions where these were
#>   interpreted as dates with zero cases.
#> ℹ In order to treat missing or `NA` cases as zeroes, see solutions in
#>   <https://github.com/epiforecasts/EpiNow2/issues/767#issuecomment-2348805272>
#> ℹ If the data is reported at non-daily intervals (for example weekly), consider
#>   using `obs_opts(na="accumulate")`.
#> ℹ For more information on these options, see `?obs_opts`.
#> This message is displayed once every 8 hours.
#> WARN [2024-12-11 10:56:45] epinow: There were 8 divergent transitions after warmup. See
#> https://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup
#> to find out why this is a problem and how to eliminate them. - 
#> WARN [2024-12-11 10:56:45] epinow: Examine the pairs() plot to diagnose sampling problems
#>  -

out$plots$summary

^{Created on 2024-12-11 with reprex v2.1.1}

[seabbs]

Ah nice - no worries. Good idea to check that and good it isn't the issue!

[nlinton]

My suspicion is perhaps it's a package version issue? I tried running the same code as @jamesmbaazam with packageVersion("EpiNow2") printing ‘1.6.1’ from CRAN, (code below, but not reprex).

`print(packageVersion("EpiNow2"))

data <- data.table::data.table(
date = as.Date(c("2024-08-24", "2024-08-31")),
confirm = c(2531, 2306)
)

data <- fill_missing(
data,
missing_dates = "accumulate",
initial_accumulate = 7
)

generation_time <- Gamma(
shape = Normal(1.3, 0.3),
rate = Normal(0.37, 0.09),
max = 14
)

incubation_period <- LogNormal(
meanlog = Normal(1.6, 0.06),
sdlog = Normal(0.4, 0.07),
max = 14
)

reporting_delay <- LogNormal(mean = 2, sd = 1, max = 10)

out <- estimate_infections(
data = data,
generation_time = gt_opts(generation_time),
rt = rt_opts(prior = list(mean = 2, sd = 0.1)),
delays = delay_opts(incubation_period + reporting_delay)
)

out$summarised`

I received the error
Error in data.table::rbindlist(list(data.table::data.table(date = seq(min(reported_cases$date) - : Item 2 has 4 columns, inconsistent with item 1 which has 3 columns. To fill missing columns use fill=TRUE. Error in data.table::rbindlist(list(data.table::data.table(date = seq(min(reported_cases$date) - : Item 2 has 4 columns, inconsistent with item 1 which has 3 columns. To fill missing columns use fill=TRUE.

It seems from the posts by @jamesmbaazam this is succeeding on '1.6.1.9000' but not for ‘1.6.1’ on CRAN. Is there a way to get '1.6.1.9000' from GitHub? devtools::install_github("epiforecasts/[email protected]") does not work.

[jamesmbaazam]

The current functionality is not yet on CRAN, so you will need to install the development version from GitHub with

# Remove current installation
remove.packages("EpiNow2")
# Install dev version from GitHub
if (!require("pak")) {
  install.packages("pak")
}
pak::pkg_install("epiforecasts/EpiNow2")

[seabbs]

Ah interesting. We should have a new CRAN release soon with this functionality but in the meantime if you need a pinned release you can use devtools::install_github("epiforecasts/EpiNow2@<commit-hash>").

[nlinton]

Works great with the Github development version! Thanks for developing this functionality. Unfortunately, we can't operationalize it for our current workflow until it is on CRAN. What is the eta on the CRAN release?

[nlinton]

Also, just FYI, the pak code was "installing" for two hours before I gave up on it. devtools::install_github("epiforecasts/[email protected]") also did not work for whatever reason.

A simple remotes::install_github("epiforecasts/EpiNow2") got me to '1.6.1.9000' though.

[seabbs]

the pak code was "installing" for two hours before I gave up on it.

Super weird.

also did not work for whatever reason.

So this is because there isn't a fixed release for the development version (as its evolving) so nothing exists at v1..6.1.9000. If you did need a fixed version though the thing to do is use the git commit hash vs a tag (but this is quite an edge case use).

I can't really speak as to the planned release schedule but I would think soon. @sbfnk?

[nlinton]

Hello, following up on this thread again. I am trying to install the latest version of EpiNow2 (as we need to do every week, since we work on Databricks), however remotes::install_github("epiforecasts/EpiNow2") will run for an hour or more and the package never finishes installing (zero exit status on second attempt after 56 minutes). Given our past experience has been 15-25 minutes for installation there seems to be some issue for the current github version. I am unsure if it is an EpiNow2 issue or a depends issue, but the installation appears to go off the rails around installation of Eigen/Rstan. Any ideas on how to follow up on this?

[jamesmbaazam]

• I see a lot of ^~~~~~ warnings around Eigen and EigenRcpp installing and have been wondering if the issue stems from Eigen being called in by rstan being called in by EpiNow2 (i.e. not an EpiNow2 issue directly, but an issue nonetheless due to dependencies). I am curious to hear if any of you see the same errors if you try a fresh install.

Yes, those are expected.

[jamesmbaazam]

I tried installing EpiNow2 from CRAN on Posit workbench and it failed.

Do you have any security issues that might be blocking the installation (firewall, sudo rights, admin, etc)? without a log or full details of your system, it is hard to know what's causing this failure.

Can you successfully install and run the code below on Posit/DataBricks

install.packages("rstan", repos = c('https://stan-dev.r-universe.dev', getOption("repos")))

library(rstan)
example(stan_model, package = "rstan", run.dontrun = TRUE)

[sbfnk]

@nlinton how frustrating - do you get any specific error when installation fails? There's none shown in the output you pasted above.

[seabbs]

urgh this sounds annoying. I just did an install from CRAN into a fresh install and a install from github into a fresh install both with no problems (on a mac) so it does sometimes work. note that the cran one used the binary and so no compilation whilst the github one built from source and so had all the warnings etc that it likes to throw

[nlinton]

@jamesmbaazam That rstan code worked great on a fresh Posit workbench. No issues. However, after uninstalling that rstan install and a subsequent failed EpiNow2 install I tried to run install.packages("rstan", repos = c('https://stan-dev.r-universe.dev', getOption("repos"))) again and it failed. I restarted my session and then the rstan install worked. With rstan installed, I tried to install EpiNow2 from github and it failed.

@sbfnk I have links to google docs below that have as much of the install history as my console displayed. (I uninstalled rstan from the check suggested by @jamesmbaazam before trying the first two installs of EpiNow2.

Here is the full text from an attempted install from CRAN on Posit workbench.
Here is the full text from an attempted install from Github on Posit workbench.
Here is the full text from an attempted install from Github on Posit workbench with rstan alread installed.

[jamesmbaazam]

Hi @nlinton, the logs seem to suggest an issue with your compiler engine (gcc) and the packages that use it. I wonder if the compiler is outdated? Please do the following and retry installing EpiNow2:

Step 1:

1. Update gcc (this is platform specific but find linux example here.
2. Update eigen. If linux, you can do

sudo apt update
sudo apt install libeigen3-dev

3. Update the following R packages

install.packages(c("rstan", "StanHeaders", "RcppEigen"))

4. Reinstall EpiNow2

[nlinton]

@jamesmbaazam thanks so much for helping parse the error codes. We don't have sudo permission on posit (and don't use EpiNow2 there anyways, I'd just looked into it for troubleshooting) so I'm going to skip verifying on that platform, but this week the install now seems to be working again Databricks. If I have trouble again though I'll try the Databricks version of your recommended troubleshooting. Thank you so much for helping me look into this. Hopefully it will stay working.

[jamesmbaazam]

@nlinton I'm glad to hear that it worked. Feel free to reach out if you encounter any issues as they help us to improve our documentation for a better user experience.

[nlinton]

Thank you!

[sbfnk]

Just noting that version 1.7.0 (which includes the accumulation feature) is now on CRAN - noting that the interface has evolved slightly from previously and you're expected to use the fill_missing() function to get accumulated counts.

[nlinton]

Thanks for letting me know! We'll give it a go soon.