Adding functions and scripts for downloading, extracting, and processing observations, initial conditions, land cover types, ERA5 drivers for anchor sites within NA. #3278

DongchenZ · 2024-03-12T23:51:44Z

Description

This PR includes:

script for preparing required data sets (ERA5; AGB, LAI, SMAP, and SOC; land cover) for anchor sites.
script for preparing initial conditions (AGB, LAI, Soil moisture, SOC) for anchor sites.
updated function for searching ecoregion within NA.
functions for downloading and extracting soil moisture from the CDS server.
function for downloading and extracting MODIS land cover products.
function for extracting AGB initial condition from the GeoTIFF files.
function for extracting ISCN SOC from existing Rdata file.

Motivation and Context

Review Time Estimate

Immediately
Within one week
When possible

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My change requires a change to the documentation.
My name is in the list of CITATION.cff
I have updated the CHANGELOG.md.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

…ecoregion codes (L1 and L2).

…develop

infotroph · 2024-03-13T00:15:25Z

As currently structured this PR adds more than 100 megabytes of data files to the package, which is far outside the norm for R package size -- CRAN wants special justification for anything with more than 5 MB of data or more then 10 MB for the whole package. Do the shapefiles and Rdata files really need to be distributed as part of data.land or can they be stored elsewhere and read in as needed?

mdietze

For the process of data prep for large-scale SDA runs, there needs to be better overall documentation of how one does that. Even if someone else found these functions and scripts, they wouldn't know how to run them without some sort of README.md. Indeed, for prep scripts that only make sense to run manually (as opposed to cron jobs or some other automation) you might consider writing them as Rmd files.

modules/assim.sequential/inst/anchor/anchorSites_data_prep.R

modules/data.land/R/ISCN_extract.R

modules/data.land/R/download.SM_CDS.R

modules/data.land/inst/IC_prep_anchorSites.R

modules/data.remote/R/MODIS_LAI_prep.R

mdietze · 2024-03-13T01:16:51Z

modules/data.remote/R/Prep_AGB_IC_from_geotiff.R

@@ -0,0 +1,41 @@
+#' Extract above ensemble ground biomass density from pre-existing GeoTIFF file for the SDA workflow.


Ok, so presumably this isn't a generic approch to reading any AGB geotiff (of which there are many in the world), but something for working with some specific product? That needs to be well documented, including being clear in the function name.

modules/data.land/R/download.SM_CDS.R

modules/assim.sequential/inst/anchor/anchorSites_data_prep.R

… download is problematic.

…r MODIS LAI extraction.

…elop

modules/assim.sequential/inst/anchor/IC_prep_anchorSites.Rmd

modules/assim.sequential/R/sda_plotting.R

infotroph · 2024-05-21T18:33:32Z

modules/assim.sequential/R/sda_plotting.R

@@ -918,20 +918,29 @@ post.analysis.multisite.ggplot <- function(settings, t, obs.times, obs.mean, obs
 ##' @param CI range of confidence interval.
 ##' @param unit list of unit used for y axis label.
 ##' @param style color option.
+##' @param PDF_w width of exported PDF file.


What units? I'm guessing probably whatever pdf() defaults to?

For cases like this I like to leave a breadcrumb trail:

Suggested change

##' @param PDF_w width of exported PDF file.

##' @param PDF_w width of exported PDF file, passed on to `base::pdf()`.

(and similar for PDF_h)

modules/data.land/R/download.SM_CDS.R

Dongchen Zhang added 15 commits March 12, 2024 19:23

Add shapefiles of level 1 and 2 NA eco-region.

eddb3ac

Add a script for processing anchor site data preparations.

0370079

Automatic updated namespace.

0ac8aee

Update codes for grabbing site info from a multi-settings object.

76fa0f2

Add a function for extracting ISCN SOC initial conditions.

f4102be

Add a function for downloading soil moisture from the CDS server.

80de614

Add a function for extracting downloaded soil moisture netcdf files.

2c7d181

Add rdata file for the ISCN SOC records along with the corresponding …

d31f478

…ecoregion codes (L1 and L2).

Add a script for processing initial conditions for the anchor site.

0fb6a4d

Tweak ecoregion finder function to adjust the NA area.

2ecace0

Automatic updated namespace.

fa445ce

Update MODIS_LAI_prep function.

f8025bb

Add a function to download, extract and filter MODIS LC data.

b2ebe9b

Add a function for extracting AGB initial conditions from GeoTIFF files.

c9eb19d

Merge branch 'develop' of https://github.com/PecanProject/pecan into …

dace7a2

…develop

github-actions bot added Tests Modules labels Mar 12, 2024

mdietze requested changes Mar 13, 2024

View reviewed changes

Dongchen Zhang and others added 11 commits March 17, 2024 11:39

Updated previous documentation.

a75ab0b

Update progress bar for the qsub_parallel function.

d919d12

Correct the unit for using the soil moisture initial condition in SDA.

d89d66a

Correct the indexing for block based analysis function.

d27e7dd

Correct logic for the analysis run.

a854ed9

Add arguments in the plotting function to control plot size.

4cfc6be

Update create a multi-site script to include the NA anchor sites.

a103d7b

Fix the bug where we write ERA5 paths into the settings.xml file.

8d0bc46

Update the script to download North America ERA5 data.

9c1dbf4

Provide another option for downloading SoilGrids data if the parallel…

ca4fa97

… download is problematic.

Add argument to decide if we want to skip the high sd observations fo…

68a1471

…r MODIS LAI extraction.

Dongchen Zhang and others added 4 commits May 16, 2024 16:33

Update documentation.

9b09744

Merge branch 'PecanProject:develop' into develop

c91660b

Merge branch 'develop' of https://github.com/DongchenZ/pecan into dev…

fda90e3

…elop

Update documentation.

569da69

DongchenZ requested a review from mdietze May 16, 2024 20:52

Dongchen Zhang and others added 8 commits May 16, 2024 16:56

Update change log file.

bbb304f

Bug fixes.

9e121ff

Merge branch 'PecanProject:develop' into develop

2eaddc8

Update book source.

e5bae23

Update IC prep script

87e75fe

Update documentation.

10d6307

Merge branch 'develop' of https://github.com/DongchenZ/pecan into dev…

9b911cd

…elop

Update documentation

aaf1a93

github-actions bot added the Documentation label May 21, 2024

mdietze reviewed May 21, 2024

View reviewed changes

modules/assim.sequential/inst/anchor/IC_prep_anchorSites.Rmd Outdated Show resolved Hide resolved

modules/assim.sequential/inst/anchor/IC_prep_anchorSites.Rmd Outdated Show resolved Hide resolved

modules/assim.sequential/inst/anchor/IC_prep_anchorSites.Rmd Outdated Show resolved Hide resolved

Fix ifelse.

dda87a1

mdietze approved these changes May 21, 2024

View reviewed changes

mdietze enabled auto-merge May 21, 2024 17:57

Dongchen Zhang added 3 commits May 21, 2024 14:08

Remove the level 1 ecoregion map.

5b82269

Remove ecoregion map from assimsequential package.

fce4f1b

Change the package name.

0c6224d

infotroph reviewed May 21, 2024

View reviewed changes

modules/assim.sequential/R/sda_plotting.R Show resolved Hide resolved

infotroph reviewed May 21, 2024

View reviewed changes

mdietze added this pull request to the merge queue May 21, 2024

infotroph reviewed May 21, 2024

View reviewed changes

modules/data.land/R/download.SM_CDS.R Outdated Show resolved Hide resolved

Update documentation.

bffb27e

Merged via the queue into PecanProject:develop with commit 6f82227 May 21, 2024
13 checks passed

dlebauer added Type: Documentation and removed Documentation labels Jul 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding functions and scripts for downloading, extracting, and processing observations, initial conditions, land cover types, ERA5 drivers for anchor sites within NA. #3278

Adding functions and scripts for downloading, extracting, and processing observations, initial conditions, land cover types, ERA5 drivers for anchor sites within NA. #3278

DongchenZ commented Mar 12, 2024 •

edited

Loading

infotroph commented Mar 13, 2024 •

edited

Loading

mdietze left a comment

mdietze Mar 13, 2024

DongchenZ Apr 25, 2024

infotroph May 21, 2024

DongchenZ May 21, 2024

infotroph May 21, 2024

		@@ -0,0 +1,41 @@
		#' Extract above ensemble ground biomass density from pre-existing GeoTIFF file for the SDA workflow.

	##' @param PDF_w width of exported PDF file.
	##' @param PDF_w width of exported PDF file, passed on to `base::pdf()`.

Adding functions and scripts for downloading, extracting, and processing observations, initial conditions, land cover types, ERA5 drivers for anchor sites within NA. #3278

Adding functions and scripts for downloading, extracting, and processing observations, initial conditions, land cover types, ERA5 drivers for anchor sites within NA. #3278

Conversation

DongchenZ commented Mar 12, 2024 • edited Loading

Description

Motivation and Context

Review Time Estimate

Types of changes

Checklist:

infotroph commented Mar 13, 2024 • edited Loading

mdietze left a comment

Choose a reason for hiding this comment

mdietze Mar 13, 2024

Choose a reason for hiding this comment

DongchenZ Apr 25, 2024

Choose a reason for hiding this comment

infotroph May 21, 2024

Choose a reason for hiding this comment

DongchenZ May 21, 2024

Choose a reason for hiding this comment

infotroph May 21, 2024

Choose a reason for hiding this comment

DongchenZ commented Mar 12, 2024 •

edited

Loading

infotroph commented Mar 13, 2024 •

edited

Loading