Merge pull request #1712 from PecanProject/develop

release 1.5.1

.gitignore

@@ -75,3 +75,5 @@ shiny/BenchmarkReport/*
 .check/
 .test/
 .doc/
+# files generated by tests
+base/qaqc/tests/testthat/Rplots.pdf

.travis.yml

@@ -4,19 +4,19 @@
 language: r
 
 # use containers
+dist: trusty
 sudo: false
 
 cache: 
   - directories:
-    - $HOME/R/Library
     - .install
     - .check
     - .test
     - .doc
   - packages
 
 addons:
-  postgresql: 9.4
+  postgresql: 9.6
   apt:
     packages:
       - libnetcdf-dev
@@ -25,15 +25,17 @@ addons:
       - jags
       - libudunits2-dev
       - python-dev 
-      - postgresql-9.4-postgis-2.1
-      - postgresql-9.4-postgis-2.1-scripts
+      - postgresql-9.6-postgis-2.3
+      - postgresql-9.6-postgis-2.3-scripts
       - netcdf-bin
       - bc
       - curl
       - udunits-bin 
       - libgmp-dev
       - libproj-dev
       - pandoc
+      - tcl
+      - tcl-dev
       # need to compile documentation
       # BROKEN MISSING FONT
       #- texinfo

CHANGELOG.md

@@ -7,13 +7,82 @@ For more information about this file see also [Keep a Changelog](http://keepacha
 
 ## [Unreleased]
 
-## [1.5.10] - Prerelease
+### Fixes
+- `PEcAn.utils` now lazy-loads data for faster execution of functions that consult lookup tables, especially `to_ncvar`.
+
+### Added
+
+### Removed
+- Removed `PEcAn.utils::model2netcdf`, which has been deprecated since PEcAn 1.3.7. Use `model2netcdf.<YOURMODEL>` in the appropriate model package instead.
+
+### Changed
+- Moved `listToXml` and `SafeList` ouf the `PEcAn.utils` package and into the `PEcAn.settings` package. The `PEcAn.utils` versions still work with a deprecation warning, but will be removed in a future release.
+- Major namespace cleanup in the `PEcAn.utils` package. It now loads more quietly and is much less likely to mask functions in a package you loaded earlier.
+
+
+## [1.5.1] - 2017-09-??
+
+### Fixes
+- Fixed hyperparameter draws in PDA
+- Show workflowid in the URL when run is finshed and user clicks results (#1659)
+- `PEcAn.BIOCRO` now uses PEcAn-standard variable names. As a result, two output variables have been renamed but keep their existing units and definitions:
+	- `StemBiom` renamed to `AbvGrndWood`
+	- `RootBiom` renamed to `root_carbon_content`
+- Improved make install logic (#1558)
+- Fixed remote code execution #1545
+- Added check for NA end/start year in read.output
+- Fixed jagify bug for raw field data
+- Fixed bug (order of dims in nc_create) introduced in model2netcdf.DALEC by standard_vars changes
+- Cleaned up NAMESPACE and source code of `PEcAn.DB` (#1520)
+- Debugged python script in call_MODIS in data.remote to allow MODIS downloads
+- Fixed FATES build script to work on ubuntu
+ SIPNET output netcdf now includes LAI; some variable names changed to match standard
+- Cleanup of leap year logic, using new `PEcAn.utils::days_in_year(year)` function (#801).
+- Replace many hard-coded unit conversions with `udunits2::ud.convert` for consistency, readability, and clarity
+- Refactored extract_soil_nc to create soil2netcdf, which will write soil data out in PEcAn standard.
+- Added a new retry.func() to base/utils to provide ability to re-try a function X times before stopping.  Currently using this function in the download.CRUNCEP() function to handle slow responses from THREDDS.
+- Reformatted call_MODIS netcdf output to reflect the orientation of the MODIS grid
+- Remote execution is more robust to errors in the submission process, not just the actual model execution
+- PRELES model run script bug fix 
+
+### Added
+- Functionality to pass hyperparameters via settings
+- Created new (and very rudimentary) web interface for downloading data from the dataone federation into the PEcAn database. More updates to come.
+- Expanded initial conditions workflow for pool-based models, including PEcAn.data.land::prepare_pools to calculate pools from IC file (to be coupled with write.configs)
+- New `PEcAn.utils::days_in_year(year)` function that should make it easier to work with leap years.
+- New `PEcAn.data.atmosphere::solar_angle` function that replaces math that occurs in some models.
+- New `PEcAn.benchmarking::align_pft` fucntion that aligns data assosiated with two different plant functional types
+- #1594 shiny/workflowPlot Adding interactiveness using ggploltly
+- #1594 shiny/workflowPlot Load outputs from multiple runs of the model
+- #1594 shiny/workflowPlot Ways to toggle geometries (e.g. geom_point vs. geom_line).
+- #1594 shiny/workflowPlot Smoothing using geom_smooth (Slider for specifying moving window width)
+- #1594 shiny/workflowPlot Comparing model output vs loaded data according to [tutorial](https://github.com/PecanProject/pecan/blob/develop/documentation/tutorials/AnalyzeOutput/modelVSdata.Rmd) 
+- Allow SIPNET and DALEC met files and model2netcdf to start or end mid year
+- A Pre-release database clean up script that deletes unused/unassosiated entries from the database
+
+### Changed
+- Clean up directory structure:
+    * Move `base` packages (`utils`, `settings`, `db`, `visualizaton`) to a `base` directory, for consistency with `modules` and `models`
+    * Move `logger.*` functions out of the `PEcAn.utils` package and into the `PEcAn.logger` package
+    * Move `remote` functions out of the `PEcAn.utils` package and into their own `PEcAn.remote` package.
+- #1594 shiny/workflowPlot Refactoring of code. `get_workflow_ids` in db/R/query.dplyr.R changed with `ensemble = FALSE`. Also allowing to load all workflow IDs. `load_data_single_run` and `var_names_all` also moved from shiny/workflowPlot/server.R to query.dplyr.R
+- `PEcAn.remote::start.model.runs` has been significantly refactored to be less redundant and more robust
+- `betyConnect` function in `query.dplyr.R` is now refactored into `read_web_config` so that the the Data-Ingest app can leverage `read_web_config` and provide it with a machine specific filepath for `.../dbfiles`
+- Rpreles and Maeswrap package moved to suggest checked for within package function.
+
+
+## [1.5.0] - 2017-07-13
 ### Added
+- Added cron job and script for the sync of the database.
 - Added PEcAn.utils::download.file() to allow for use of alternative FTP programs
 - Updated downloadAmeriflux and downloadNARR to make use of PEcAn.utils::download.file()
 - Added -w flag to load.bety.sh script to specify the URL to fetch the data from
-- add new table sites_cultivars to betydb sync scripts (dump and load)
-- thredds server documentation and catlog generating script
+- Add new table sites_cultivars to betydb sync scripts (dump and load)
+- Added docker container scrips (.yml) to create docker container for PEcAn
+- Added the configuration edit page to allow easy modification of config via web interface
+- Thredds server documentation and catlog generating script
+- Added new standard variables table (standard_vars.csv) and to_ncvar and to_ncdim functions in PEcAn.utils
+- Added initial conditions file io functions for pool-based models in data.land
 
 ### Changed
 - upscale_met now accepts ~any valid CF file (not just full years), retains correct time units, and respects the previously ignored `overwrite` parameter

Dockerfile

@@ -0,0 +1,36 @@
+FROM ubuntu:16.04
+MAINTAINER Aman Kumar ([email protected])
+
+# updated ppa's
+RUN echo "deb http://cran.rstudio.com/bin/linux/ubuntu xenial/" > /etc/apt/sources.list.d/R.list &&\
+    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
+
+# copy the installation script inside the container
+ADD docker/ /build
+
+# Set script mod +x for preprocessors
+RUN chmod 750 /build/*.sh
+
+# Run the OS System setup script
+RUN /build/system_services.sh
+
+# run update machine to update machine
+RUN /build/update_machine.sh
+
+# run install packages to install required packages
+RUN /build/install_packages.sh
+
+# run install R to install R packages
+RUN /build/install_R.sh
+
+# run install pecan to install pecan cores
+RUN /build/install_pecan.sh
+
+# run install sipnet to install SIPNET (default testing Model)
+RUN /build/install_sipnet.sh
+
+# Clean up APT when done.
+RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /build/*
+
+# startup
+CMD ["/sbin/my_init"]

Makefile

@@ -1,70 +1,73 @@
 NCPUS ?= 1
 
-BASE := utils db settings visualization
+BASE := logger utils db settings visualization qaqc remote
 
 MODELS := biocro clm45 dalec ed fates gday jules linkages \
-				lpjguess maat maespa preles sipnet
+				lpjguess maat maespa preles sipnet template
 
 MODULES := allometry assim.batch assim.sequential benchmark \
 				 data.atmosphere data.hydrology data.land \
 				 data.mining data.remote emulator meta.analysis \
 				 photosynthesis priors rtm uncertainty
 
+BASE := $(BASE:%=base/%)
 MODELS := $(MODELS:%=models/%)
 MODULES := $(MODULES:%=modules/%)
-ALL_PKGS := $(BASE) $(MODELS) $(MODULES) models/template
+ALL_PKGS := $(BASE) $(MODULES) $(MODELS)
 
 BASE_I := $(BASE:%=.install/%)
 MODELS_I := $(MODELS:%=.install/%)
 MODULES_I := $(MODULES:%=.install/%)
-ALL_PKGS_I := $(BASE_I) $(MODELS_I) $(MODULES_I) .install/models/template
+ALL_PKGS_I := $(BASE_I) $(MODULES_I) $(MODELS_I)
 
 BASE_C := $(BASE:%=.check/%)
 MODELS_C := $(MODELS:%=.check/%)
 MODULES_C := $(MODULES:%=.check/%)
-ALL_PKGS_C := $(BASE_C) $(MODELS_C) $(MODULES_C) .check/models/template
+ALL_PKGS_C := $(BASE_C) $(MODULES_C) $(MODELS_C)
 
 BASE_T := $(BASE:%=.test/%)
 MODELS_T := $(MODELS:%=.test/%)
 MODULES_T := $(MODULES:%=.test/%)
-ALL_PKGS_T := $(BASE_T) $(MODELS_T) $(MODULES_T) .test/models/template
+ALL_PKGS_T := $(BASE_T) $(MODULES_T) $(MODELS_T)
 
 BASE_D := $(BASE:%=.doc/%)
 MODELS_D := $(MODELS:%=.doc/%)
 MODULES_D := $(MODULES:%=.doc/%)
-ALL_PKGS_D := $(BASE_D) $(MODELS_D) $(MODULES_D) .doc/models/template
+ALL_PKGS_D := $(BASE_D) $(MODULES_D) $(MODELS_D)
 
 .PHONY: all install check test document
 
-all: install
+all: install document
 
-document: .doc/all
-install: .install/all
-check: .check/all
-test: .test/all 
+document: $(ALL_PKGS_D) .doc/base/all
+install: $(ALL_PKGS_I) .install/base/all
+check: $(ALL_PKGS_C) .check/base/all
+test: $(ALL_PKGS_T) .test/base/all
 
 ### Dependencies
-.doc/all: $(ALL_PKGS_D)
-.install/all: $(ALL_PKGS_I)
-.check/all: $(ALL_PKGS_C)
-.test/all: $(ALL_PKGS_T)
-
-depends = .install/$(1) .doc/$(1) .check/$(1) .test/$(1)
-
-$(call depends,db): .install/utils
-$(call depends,settings): .install/utils .install/db
-$(call depends,visualization): .install/db
-$(call depends,modules/data.atmosphere): .install/utils
-$(call depends,modules/data.land): .install/db .install/utils
-$(call depends,modules/meta.analysis): .install/utils .install/db
-$(call depends,modules/priors): .install/utils
-$(call depends,modules/assim.batch): .install/utils .install/db .install/modules/meta.analysis 
-$(call depends,modules/rtm): .install/modules/assim.batch
-$(call depends,models/template): .install/utils
-$(call depends,models/biocro): .install/utils .install/modules/data.atmosphere .install/modules/data.land
-
-$(MODELS_I): .install/models/template
-
+.doc/base/all: $(ALL_PKGS_D)
+.install/base/all: $(ALL_PKGS_I)
+.check/base/all: $(ALL_PKGS_C)
+.test/base/all: $(ALL_PKGS_T)
+
+$(subst .doc/models/template,,$(MODELS_D)): .install/models/template # for models that import Roxygen docs from template
+$(subst .install/base/logger,,$(ALL_PKGS_I)): .install/base/logger
+
+depends = .doc/$(1) .install/$(1) .check/$(1) .test/$(1)
+
+$(call depends,base/utils): .install/base/remote
+$(call depends,base/db): .install/base/utils
+$(call depends,base/settings): .install/base/utils .install/base/db
+$(call depends,base/visualization): .install/base/db
+$(call depends,modules/data.atmosphere): .install/base/utils .install/base/remote
+$(call depends,modules/data.land): .install/base/db .install/base/utils .install/base/remote
+$(call depends,modules/meta.analysis): .install/base/utils .install/base/db .install/base/remote
+$(call depends,modules/priors): .install/base/utils .install/base/remote
+$(call depends,modules/assim.batch): .install/base/utils .install/base/db .install/modules/meta.analysis .install/base/remote
+$(call depends,modules/rtm): .install/modules/assim.batch .install/base/remote
+$(call depends,modules/uncertainty): .install/base/utils .install/modules/priors .install/base/remote
+$(call depends,models/template): .install/base/utils .install/base/remote
+$(call depends,models/biocro): .install/base/utils .install/base/settings .install/base/db .install/modules/data.atmosphere .install/modules/data.land .install/base/remote
 
 clean:
 	rm -rf .install .check .test .doc
@@ -85,6 +88,7 @@ clean:
 	mkdir -p $(@D)
 	echo `date` > $@
 
+depends_R_pkg = Rscript -e "devtools::install_deps('$(strip $(1))', threads = ${NCPUS});"
 install_R_pkg = Rscript -e "devtools::install('$(strip $(1))', Ncpus = ${NCPUS});"
 check_R_pkg = Rscript scripts/check_with_errors.R $(strip $(1))
 test_R_pkg = Rscript -e "devtools::test('"$(strip $(1))"', reporter = 'stop')"
@@ -94,11 +98,12 @@ $(ALL_PKGS_I) $(ALL_PKGS_C) $(ALL_PKGS_T) $(ALL_PKGS_D): .install/devtools .inst
 
 .SECONDEXPANSION:
 .doc/%: $$(wildcard %/**/*) $$(wildcard %/*)
+	$(call depends_R_pkg, $(subst .doc/,,$@))
 	$(call doc_R_pkg, $(subst .doc/,,$@))
 	mkdir -p $(@D)
 	echo `date` > $@
 
-.install/%: $$(wildcard %/**/*) $$(wildcard %/*)
+.install/%: $$(wildcard %/**/*) $$(wildcard %/*) .doc/%
 	$(call install_R_pkg, $(subst .install/,,$@))
 	mkdir -p $(@D)
 	echo `date` > $@

README.md

@@ -46,6 +46,7 @@ The demo instance only allows for runs at pecan.ncsa.illinois.edu. Once you have
 * Dietze, Michael C., Shawn P. Serbin, Carl Davidson, Ankur R. Desai, Xiaohui Feng, Ryan Kelly, Rob Kooper et al. "A quantitative assessment of a terrestrial biosphere model's data needs across North American biomes." Journal of Geophysical Research: Biogeosciences 119, no. 3 (2014): 286-300.
 * Viskari, Toni, Brady Hardiman, Ankur R. Desai, and Michael C. Dietze. "Model-data assimilation of multiple phenological observations to constrain and predict leaf area index." (2015) [doi:10.1890/14-0497.1](http://dx.doi.org/10.1890/14-0497.1)
 * Shiklomanov. A, MC Dietze, T Viskari, PA Townsend, SP Serbin. 2016 "Quantifying the influences of spectral resolution on uncertainty in leaf trait estimates through a Bayesian approach to RTM inversion" Remote Sensing of the Environment 183: 226-238
+* LeBauer, David, Rob Kooper, Patrick Mulrooney, Scott Rohde, Dan Wang, Stephen P. Long, and Michael C. Dietze. "BETYdb: a yield, trait, and ecosystem service database applied to second‐generation bioenergy feedstock production." GCB Bioenergy (2017).
 
 ## Acknowledgements
 

all/DESCRIPTION → base/all/DESCRIPTION

@@ -2,8 +2,8 @@ Package: PEcAn.all
 Type: Package
 Title: PEcAn functions used for ecological forecasts and
     reanalysis
-Version: 1.5.0
-Date: 2017-07-14
+Version: 1.5.1
+Date: 2017-09-25
 Author: David LeBauer, Mike Dietze, Xiaohui Feng, Dan Wang,
     Mike Dietze, Carl Davidson, Rob Kooper, Shawn Serbin
 Maintainer: David LeBauer <[email protected]>
@@ -18,6 +18,7 @@ Depends:
     PEcAn.DB,
     PEcAn.settings,
     PEcAn.MA,
+    PEcAn.logger,
     PEcAn.utils,
     PEcAn.uncertainty,
     PEcAn.data.atmosphere,
@@ -26,7 +27,8 @@ Depends:
     PEcAn.assim.batch,
     PEcAn.emulator,
     PEcAn.priors,
-    PEcAn.benchmark
+    PEcAn.benchmark,
+    PEcAn.remote
 Suggests:
     PEcAn.ed,
     PEcAn.sipnet,

all/tests/testthat.R → base/all/tests/testthat.R

@@ -9,5 +9,5 @@
 library(testthat)
 library(PEcAn.utils)
 
-logger.setQuitOnSevere(FALSE)
+PEcAn.logger::logger.setQuitOnSevere(FALSE)
 test_check("PEcAn.all")

db/DESCRIPTION → base/db/DESCRIPTION

@@ -1,8 +1,8 @@
 Package: PEcAn.DB
 Type: Package
 Title: PEcAn functions used for ecological forecasts and reanalysis
-Version: 1.5.0
-Date: 2017-07-14
+Version: 1.5.1
+Date: 2017-09-25
 Author: David LeBauer, Mike Dietze, Xiaohui Feng, Dan Wang, Carl
     Davidson, Rob Kooper, Shawn Serbin
 Maintainer: David LeBauer <[email protected]>
@@ -11,10 +11,10 @@ Description: The Predictive Ecosystem Carbon Analyzer (PEcAn) is a scientific
     model parameterization, execution, and analysis. The goal of PECAn is to
     streamline the interaction between data and models, and to improve the
     efficacy of scientific investigation.
-Depends:
-    DBI,
-    PEcAn.utils
 Imports:
+    DBI,
+    PEcAn.logger,
+    PEcAn.utils,
     plyr (>= 1.8.4)
 Suggests:
     RPostgreSQL,

db/NAMESPACE → base/db/NAMESPACE

@@ -19,6 +19,7 @@ export(dbfile.input.insert)
 export(dbfile.insert)
 export(dbfile.posterior.check)
 export(dbfile.posterior.insert)
+export(default_hostname)
 export(derive.trait)
 export(derive.traits)
 export(dplyr.count)
@@ -30,10 +31,8 @@ export(get_run_ids)
 export(get_users)
 export(get_var_names)
 export(get_workflow_ids)
+export(load_data_single_run)
 export(ncdays2date)
-export(query.base)
-export(query.base.con)
-export(query.close)
 export(query.file.path)
 export(query.format.vars)
 export(query.pft_species)
@@ -45,5 +44,6 @@ export(rename.jags.columns)
 export(runModule.get.trait.data)
 export(runs)
 export(take.samples)
+export(var_names_all)
 export(workflow)
 export(workflows)