Missing daily precipitation data in winter months on ncdc() calls

[kross0926]

When executing an NCDC call to get daily precipitation data, it appears the data from January to March is now missing. This has just started happening recently. I have downloaded the matching data from NOAA's online application to verify. Any ideas on what is happening? My code is below. I have attached the NCDC online download as well.

precip_tmp <- ncdc(datasetid='GHCND', datatypeid=c('PRCP'), stationid="GHCND:US1LAWC0001",
                      startdate = "2015-10-01", enddate = "2016-09-30", limit=500,add_units = TRUE)

# convert to precip to inches
precip_tmp$data$precip_mm <- round(precip_tmp$data$value/10,2) #GHCN data retrieve is in 10ths of mm
precip_tmp$data$precip_IN <- round(precip_tmp$data$precip_mm / 25.4,2)
  

Session Info

Session info ────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.2.1 (2022-06-23 ucrt)
 os       Windows 10 x64 (build 19044)
 system   x86_64, mingw32
 ui       RStudio
 language (EN)
 collate  English_United States.utf8
 ctype    English_United States.utf8
 tz       America/Chicago
 date     2022-09-01
 rstudio  2022.07.0+548 Spotted Wakerobin (desktop)
 pandoc   NA

─ Packages ────────────────────────────────────────────────────────────────────────────────────────────────────────────
 ! package       * version   date (UTC) lib source
   AOI             0.2.1     2022-06-16 [1] Github (mikejohnson51/AOI@b6ae1d4)
   assertthat      0.2.1     2019-03-21 [1] CRAN (R 4.2.0)
   bit             4.0.4     2020-08-04 [1] CRAN (R 4.2.0)
   bit64           4.0.5     2020-08-30 [1] CRAN (R 4.2.0)
   brio            1.1.3     2021-11-30 [1] CRAN (R 4.2.0)
   cachem          1.0.6     2021-08-19 [1] CRAN (R 4.2.0)
   callr           3.7.0     2021-04-20 [1] CRAN (R 4.2.0)
   class           7.3-20    2022-01-16 [2] CRAN (R 4.2.1)
   classInt        0.4-7     2022-06-10 [1] CRAN (R 4.2.0)
   cli             3.3.0     2022-04-25 [1] CRAN (R 4.2.0)
   codetools       0.2-18    2020-11-04 [2] CRAN (R 4.2.1)
   colorspace      2.0-3     2022-02-21 [1] CRAN (R 4.2.0)
   crayon          1.5.1     2022-03-26 [1] CRAN (R 4.2.0)
   crul            1.2.0     2021-11-22 [1] CRAN (R 4.2.0)
   curl            4.3.2     2021-06-23 [1] CRAN (R 4.2.0)
   dataRetrieval * 2.7.11    2022-02-18 [1] CRAN (R 4.2.0)
   DBI             1.1.2     2021-12-20 [1] CRAN (R 4.2.0)
   desc            1.4.1     2022-03-06 [1] CRAN (R 4.2.0)
   devtools        2.4.3     2021-11-30 [1] CRAN (R 4.2.0)
   digest          0.6.29    2021-12-01 [1] CRAN (R 4.2.0)
   dplyr         * 1.0.9     2022-04-28 [1] CRAN (R 4.2.0)
   e1071           1.7-11    2022-06-07 [1] CRAN (R 4.2.0)
   EflowStats    * 5.0.1     2022-09-01 [1] Github (USGS-R/EflowStats@883fbd5)
   ellipsis        0.3.2     2021-04-29 [1] CRAN (R 4.2.0)
   fansi           1.0.3     2022-03-24 [1] CRAN (R 4.2.0)
   farver          2.1.0     2021-02-28 [1] CRAN (R 4.2.0)
   fastmap         1.1.0     2021-01-25 [1] CRAN (R 4.2.0)
   fipio           1.1.1     2022-03-15 [1] CRAN (R 4.2.0)
   forecast        8.17.0    2022-07-25 [1] CRAN (R 4.2.1)
   fracdiff        1.5-1     2020-01-24 [1] CRAN (R 4.2.1)
   fs              1.5.2     2021-12-08 [1] CRAN (R 4.2.0)
 V generics        0.1.2     2022-07-05 [1] CRAN (R 4.2.1) (on disk 0.1.3)
   ggplot2       * 3.3.6     2022-05-03 [1] CRAN (R 4.2.0)
   ggrepel         0.9.1     2021-01-15 [1] CRAN (R 4.2.0)
   ggsflabel     * 0.0.1     2022-06-16 [1] Github (yutannihilation/ggsflabel@a489481)
   ggspatial     * 1.1.5     2021-01-04 [1] CRAN (R 4.2.0)
   ggtext          0.1.1     2020-12-17 [1] CRAN (R 4.2.1)
   glue            1.6.2     2022-02-24 [1] CRAN (R 4.2.0)
   gridExtra       2.3       2017-09-09 [1] CRAN (R 4.2.0)
   gridtext        0.1.4     2020-12-10 [1] CRAN (R 4.2.1)
   gtable        * 0.3.0     2019-03-25 [1] CRAN (R 4.2.0)
   hms             1.1.1     2021-09-26 [1] CRAN (R 4.2.0)
   hoardr          0.5.2     2018-12-02 [1] CRAN (R 4.2.0)
   httpcode        0.3.0     2020-04-10 [1] CRAN (R 4.2.0)
   httr          * 1.4.3     2022-05-04 [1] CRAN (R 4.2.0)
   imputeTS        3.2       2021-01-16 [1] CRAN (R 4.2.1)
   jpeg            0.1-9     2021-07-24 [1] CRAN (R 4.2.0)
   jsonlite        1.8.0     2022-02-22 [1] CRAN (R 4.2.0)
   KernSmooth      2.23-20   2021-05-03 [2] CRAN (R 4.2.1)
   labeling        0.4.2     2020-10-20 [1] CRAN (R 4.2.0)
   lattice         0.20-45   2021-09-22 [2] CRAN (R 4.2.1)
   lifecycle       1.0.1     2021-09-24 [1] CRAN (R 4.2.0)
   lmom            2.9       2022-05-29 [1] CRAN (R 4.2.0)
   lmtest          0.9-40    2022-03-21 [1] CRAN (R 4.2.1)
   lubridate     * 1.8.0     2021-10-07 [1] CRAN (R 4.2.0)
   magrittr        2.0.3     2022-03-30 [1] CRAN (R 4.2.0)
   memoise         2.0.1     2021-11-26 [1] CRAN (R 4.2.0)
   munsell         0.5.0     2018-06-12 [1] CRAN (R 4.2.0)
   nlme            3.1-157   2022-03-25 [2] CRAN (R 4.2.1)
   nnet            7.3-17    2022-01-16 [2] CRAN (R 4.2.1)
   padr          * 0.6.0     2021-10-01 [1] CRAN (R 4.2.0)
 V pillar          1.7.0     2022-08-19 [1] CRAN (R 4.2.1) (on disk 1.8.1)
   pkgbuild        1.3.1     2021-12-20 [1] CRAN (R 4.2.0)
   pkgconfig       2.0.3     2019-09-22 [1] CRAN (R 4.2.0)
   pkgload         1.2.4     2021-11-30 [1] CRAN (R 4.2.0)
   plyr            1.8.7     2022-03-24 [1] CRAN (R 4.2.0)
   png             0.1-7     2013-12-03 [1] CRAN (R 4.2.0)
   prettyunits     1.1.1     2020-01-24 [1] CRAN (R 4.2.0)
   processx        3.6.0     2022-06-10 [1] CRAN (R 4.2.0)
   proxy           0.4-27    2022-06-09 [1] CRAN (R 4.2.0)
   ps              1.7.0     2022-04-23 [1] CRAN (R 4.2.0)
   purrr         * 0.3.4     2020-04-17 [1] CRAN (R 4.2.0)
   quadprog        1.5-8     2019-11-20 [1] CRAN (R 4.2.0)
   quantmod        0.4.20    2022-04-29 [1] CRAN (R 4.2.1)
   R6              2.5.1     2021-08-19 [1] CRAN (R 4.2.0)
   rappdirs        0.3.3     2021-01-31 [1] CRAN (R 4.2.0)
   raster        * 3.5-15    2022-01-22 [1] CRAN (R 4.2.0)
   Rcpp            1.0.8.3   2022-03-17 [1] CRAN (R 4.2.0)
   RcppRoll        0.3.0     2018-06-05 [1] CRAN (R 4.2.1)
   readr           2.1.2     2022-01-30 [1] CRAN (R 4.2.0)
   remotes         2.4.2     2021-11-30 [1] CRAN (R 4.2.0)
   reshape2      * 1.4.4     2020-04-09 [1] CRAN (R 4.2.0)
   rlang           1.0.2     2022-03-04 [1] CRAN (R 4.2.0)
   rnaturalearth   0.1.0     2017-03-21 [1] CRAN (R 4.2.0)
   rnoaa         * 1.3.8     2021-12-01 [1] CRAN (R 4.2.0)
   rprojroot       2.0.3     2022-04-02 [1] CRAN (R 4.2.0)
   rstudioapi      0.13      2020-11-12 [1] CRAN (R 4.2.0)
   rvest           1.0.2     2021-10-16 [1] CRAN (R 4.2.0)
   s2              1.0.7     2021-09-28 [1] CRAN (R 4.2.0)
 V scales          1.2.0     2022-08-20 [1] CRAN (R 4.2.1) (on disk 1.2.1)
   sessioninfo     1.2.2     2021-12-06 [1] CRAN (R 4.2.0)
   sf            * 1.0-7     2022-03-07 [1] CRAN (R 4.2.0)
   sp            * 1.5-0     2022-06-05 [1] CRAN (R 4.2.0)
   stinepack       1.4       2018-07-30 [1] CRAN (R 4.2.0)
   stringi         1.7.6     2021-11-29 [1] CRAN (R 4.2.0)
 V stringr         1.4.0     2022-08-20 [1] CRAN (R 4.2.1) (on disk 1.4.1)
   terra           1.5-34    2022-06-09 [1] CRAN (R 4.2.0)
   testthat        3.1.4     2022-04-26 [1] CRAN (R 4.2.0)
   tibble          3.1.7     2022-05-03 [1] CRAN (R 4.2.0)
   tidyr         * 1.2.0     2022-02-01 [1] CRAN (R 4.2.0)
   tidyselect      1.1.2     2022-02-21 [1] CRAN (R 4.2.0)
   timeDate        4021.104  2022-07-19 [1] CRAN (R 4.2.1)
   triebeard       0.3.0     2016-08-04 [1] CRAN (R 4.2.0)
   tseries         0.10-51   2022-05-01 [1] CRAN (R 4.2.1)
   TTR             0.24.3    2021-12-12 [1] CRAN (R 4.2.1)
   tzdb            0.3.0     2022-03-28 [1] CRAN (R 4.2.0)
   units           0.8-0     2022-02-05 [1] CRAN (R 4.2.0)
   urca            1.3-3     2022-08-29 [1] CRAN (R 4.2.1)
   urltools        1.7.3     2019-04-14 [1] CRAN (R 4.2.0)
   usethis         2.1.6     2022-05-25 [1] CRAN (R 4.2.0)
   utf8            1.2.2     2021-07-24 [1] CRAN (R 4.2.0)
   vctrs           0.4.1     2022-04-13 [1] CRAN (R 4.2.0)
   vroom           1.5.7     2021-11-30 [1] CRAN (R 4.2.0)
   withr           2.5.0     2022-03-03 [1] CRAN (R 4.2.0)
   wk              0.6.0     2022-01-03 [1] CRAN (R 4.2.0)
   XML             3.99-0.10 2022-06-09 [1] CRAN (R 4.2.0)
   xml2            1.3.3     2021-11-30 [1] CRAN (R 4.2.0)
   xts           * 0.12.1    2020-09-09 [1] CRAN (R 4.2.0)
   zoo           * 1.8-10    2022-04-15 [1] CRAN (R 4.2.0)

 [1] C:/Users/kstrauch/AppData/Local/R/win-library/4.2
 [2] C:/Program Files/R/R-4.2.1/library

 V ── Loaded and on-disk version mismatch.

[3068272.csv](https://github.com/ropensci/rnoaa/files/9473093/3068272.csv)

[kross0926]

It appears this is an issue with the data coming from the api. However while investigating the issue I noticed the library uses the old api url of https://www.ncdc.noaa.gov/cdo-web/api/v2/. According to Climate Data Online: Web Services Documentation It has been changed to https://www.ncei.noaa.gov/cdo-web/api/v2/

[kross0926]

I appologize after emailing with NCEI, NCEI is ending support for API v2
https://www.ncei.noaa.gov/support/access-data-service-api-user-documentation

The correct api url is https://www.ncei.noaa.gov/access/services/data/v1
This api has all the data available.

[djhocking]

Thanks so much, there have been a couple of this API switches lately and I definitely was unaware of this one. I've been lagging on this project lately but I'll get this incorporated soon. Thanks for tracking down the issue!

[kross0926]

Any update on this? I have a due date coming up and using this library and data is a major component of the code.

[djhocking]

Unfortunately, unlike the last API update they did, not only did they change the base URL but the entire structure and naming scheme of the data. All the R functions related to NCDC are completely broken currently. I am now actively working on it but I can't honestly give a timeframe for completion because I haven't gotten any of it working yet. I will keep you posted. Thanks for bumming this to the front again!

[djhocking]

@kross0926 - I figured out at least the data pull part of the new API so I should be able to at least hack together something quickly for your due date depending what info or metadata you need. For example a new query might look like this:

precip_tmp <- ncdc2(datasetid='daily-summaries', datatypeid=c('PRCP'), stationid="US1LAWC0001",  
          startdate = "2015-10-01", enddate = "2016-09-30", limit=500,add_units = TRUE)

Which would currently look like:

> precip_tmp
$meta
NULL

$data
# A tibble: 112 × 3
   station     date        prcp
   <chr>       <date>     <dbl>
 1 US1LAWC0001 2015-10-10  12.4
 2 US1LAWC0001 2015-10-25  47  
 3 US1LAWC0001 2015-10-26 127. 
 4 US1LAWC0001 2015-10-27   1  
 5 US1LAWC0001 2015-10-28   5.1
 6 US1LAWC0001 2015-11-01  35.6
 7 US1LAWC0001 2015-11-02  11.4
 8 US1LAWC0001 2015-11-03   1.3
 9 US1LAWC0001 2015-11-06   6.4
10 US1LAWC0001 2015-11-07  48.5
# … with 102 more rows

attr(,"class")
[1] "ncdc_data"

Then you could do your conversion, notice the chance of value to prcp:

# convert to precip to inches
precip_tmp$data$precip_mm <- round(precip_tmp$data$prcp/10,2) #GHCN data retrieve is in 10ths of mm
precip_tmp$data$precip_IN <- round(precip_tmp$data$precip_mm / 25.4,2)

If this would work for your immediate purposes, I could get a version pushed to a github branch that you could install tomorrow. It would be fragile (e.g. prone to break without good errors if the data query was invalid) and without the metadata for now.

[kross0926]

Thanks for your response and hard work on getting something functional. I think just the data pull capability would be helpful, I don't need the metadata right away. Thanks!

[djhocking]

@kross0926 Okay I've pushed the minimal data pull option to github. You'll have to install it via:

remotes::install_github("ropensci/rnoaa@ncdc_api")

Then run this to see if it works:

precip_tmp <- rnoaa:::ncdc2(datasetid='daily-summaries', datatypeid=c('PRCP'), stationid="US1LAWC0001", startdate = "2015-10-01", enddate = "2016-09-30", limit=500,add_units = TRUE)

Notice the rnoaa::: which is currently needed because the ncdc2 function is not yet exporting even in this branch. Let me know if this works and if there are any other immediate needs while I slowly go through all the NCDC-related functions (and likely a number of the other datasets).

[kross0926]

This looks like it is working. As far as next needs if you could work on ncdc_stations() functions next. I was hoping they would still work but doesn't seem to be the case. I need the datasetid = NORMAL_DLY and 'daily-summaries' (used to be 'GHCND'), I usually retrieve stations within a bounding box. Thanks!