Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Thredds download function #2458

Draft
wants to merge 12 commits into
base: develop
Choose a base branch
from

Conversation

bailsofhay
Copy link
Collaborator

This function is the start of a general function that should be able to download data from THREDDS.

Description

The user can download any THREDDS datasets as long as they have site information, dates, and the base URLS needed to grab the datasets. Base URLS need the dataset name, but not the dates or locations. There will likely be issues pulling data sets in the future since they are not all stored in similarly structured directory formats like AVHRR LAI (which this function is based on). Updates to the function will occur as we start playing with more datasets from THREDDS.

Motivation and Context

We need an easy way to grab more datasets and THREDDS has many already, and is a great place to store future datasets that we can use.

Review Time Estimate

  • Immediately
  • Within one week
  • When possible

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My change requires a change to the documentation.
  • I have updated the CHANGELOG.md.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@bailsofhay
Copy link
Collaborator Author

@ankurdesai I had to remove the earlier PR I made last week with the SDA file changes attached. this is the new PR for the same Thredds function, but without non-related file changes.

@bailsofhay
Copy link
Collaborator Author

Turns out there was already a branch named thredds that was messing up my pr....... :/

Copy link
Member

@infotroph infotroph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that this is still WIP so I hope my review isn't premature, but a few interface-level comments that are probably easier to address sooner rather than later.

Might also be useful to add a fully updated example that can double as a test case while you're working on the details.

modules/data.remote/R/download.thredds.R Outdated Show resolved Hide resolved
modules/data.remote/R/download.thredds.R Outdated Show resolved Hide resolved
run_parallel = TRUE) {
# require("XML")
# require("RCurl")
require("foreach")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functions should never explicitly load other packages. If it's not possible to call everything in namespaced form (foreach::fn_name(...)), consider formally importing the functions you need from foreach.

modules/data.remote/R/download.thredds.R Outdated Show resolved Hide resolved
modules/data.remote/R/download.thredds.R Outdated Show resolved Hide resolved
modules/data.remote/R/download.thredds.R Outdated Show resolved Hide resolved
modules/data.remote/R/download.thredds.R Outdated Show resolved Hide resolved
modules/data.remote/R/download.thredds.R Outdated Show resolved Hide resolved
@infotroph
Copy link
Member

Once merged this will close #2415, correct?

modules/data.remote/R/download.thredds.R Outdated Show resolved Hide resolved
modules/data.remote/R/download.thredds.R Outdated Show resolved Hide resolved
@bailsofhay
Copy link
Collaborator Author

@infotroph, @ashiklom or @mdietze. What is the standard way to code in warnings. For example, I want a warning to pop up and kill data extraction if the user puts in a date range that the data doesn't exist during.

@mdietze
Copy link
Member

mdietze commented Oct 16, 2019

Warning, errors, and logs should be handled through the logger functions in the PEcAn.utils package

@robkooper robkooper marked this pull request as draft September 9, 2020 21:37
@github-actions
Copy link

This PR is stale because it has been open 365 days with no activity.

@infotroph
Copy link
Member

Status: @serbinsh will replace this with an updated version as part of #2861, then close this when done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants